Search index updates that leave unchanged postings alone

A small code change used to replace every path and trigram posting for its repository in the Hub search index. Repeating that work multiplied SQLite row writes even when almost all of the index stayed the same. The publisher now compares the complete incoming descriptor with its stored postings, then changes only added, edited and removed entries in one transaction.

Background indexing also converges on the newest automatic head. An explicit historical request remains required, and jobs created before the upgrade remain protected. The search interface, exact matcher and current access checks keep their existing behavior.

These demonstrations use actual compiled Rust in workerd with native SQLite and isolated Git history. The repositories and identities are synthetic. The visible controls are an inspection adapter for the backend, and the measurements are local SQL counters. They establish neither production dollar savings nor a deployed outcome.

Publish the changed postings atomically

In the isolated compiled backend, a new head changed the first path size from 1 to 2 while retaining 1,000 paths and 88,699 grams; the publication wrote seven SQLite rows, and its identical-head retry wrote zero.

The publisher still receives a complete validated descriptor. It preserves duplicate-path and duplicate-gram semantics, rejects older generations, and rolls metadata and postings back together if a SQL statement fails. Empty and partial descriptors remain valid conservative states.

Let background work converge on the current head

After eight automatic fixture heads were queued, one real alarm step left only the latest generation at phase 1; six further steps completed and published that head. The separate pinned-history/restart comparison retained required history and reduced SQL writes from 216 to 63.

The marker table distinguishes automatic work from explicitly requested history. The alarm rereads current metadata after Hub I/O and rechecks demand after Git reads. The completed active snapshot remains available while newer work is pending. Existing bounded chunks and retry limits are retained.

Keep exact matching behind current access checks

The exact fixture query returned needle7 café Ω from src/search.txt on the ready current head; a private Trunk refused anonymous content despite stale public Hub metadata, allowed the current Alice approver, and returned HTTP 403 immediately after her access was revoked.

The runtime comparison covers filename rename/deletion, content changes, Unicode, regular expressions, short queries, filters and incomplete hints. Hub candidates are a shortlist; the selected Trunk still applies current read checks and the exact matcher.

Measure row mutations against the prior implementation

The source-bound workerd comparison measured a small edit at 357,798 SQL row writes before and seven after, with all 36 shared posting, candidate and query observations identical; initial publication remained at 268,110 writes in both builds.

The representative publication contains 1,000 paths and 88,699 distinct grams. The edit changes one path size and exchanges one gram. Initial population writes 268,110 rows in both builds; a new head with identical postings drops from 357,798 writes to two. A burst with pinned history and a restart drops from 216 to 63 writes. Comparison reads remain proportional to the stored set, and wholesale replacement can read more rows. SQL counters exclude KV writes, production traffic, CPU limits and billing aggregation.

Retain the original source and observations

This article binds every task requirement to landed commit f45d2eb9 and the original controlled-fixture capture hashes; source anchors for the separate private architecture explanation were prepared against the same frozen contract and landed source.

The product change landed in PR #323 as f45d2eb9bc751d740113bf1d546f9a4b3c0a3d02. The original runtime measurements and reproducible commands are in cf/evidence/hub-search-writes-20261026.json and cf/scale/hub-search-writes.md at that commit. Private architecture review and release signatures remain outside the public article source.