Repository search that shares unchanged Git trees

A small code change should not require paying to rewrite an index of the whole repository. T-296 repairs that write amplification in Nerudo's repository indexer and stops completed indexing from repeatedly scheduling another alarm. Exact commit search and current access checks remain part of the contract.

These are original captures of scripted local executions of the actual Rust Worker in workerd and SQLite. The inspection adapter displays captured backend response records with synthetic Git and identities. It is not the production interface, a recorded coding-provider outcome or a production savings claim.

A small edit should not copy the repository

Repository search has to retain the exact commit it searched. Previously that meant another flattened set of filename and directory rows for each head. An unchanged subtree was copied into the new generation and its old rows were eventually deleted. Both operations incur writes, including secondary-index changes.

The index now retains the commit's Git root and stores immutable directory entries once by object ID. Entries are packed into roughly 32 KiB SQLite rows. A rename can reuse a directory under its new path. Changing a wide directory still writes its new encoded entries, but unrelated unchanged subtrees need no copy.

The normal durable-landing handoff remains in place. For measurement, the isolated harness advances real local Git heads directly into the compiled indexer, rather than claiming to record a hosted sequencer landing. The screenshot pairs the actual exact search response with native SQLite counters from that same controlled change. At 10,000 files, an ordinary small edit fell from 30,027–40,034 to 22–30 row writes; editing inside a wide directory took 77 instead of 40,037. Initial population still has to index new content.

In the scripted 10,000-file Git fixture, the current3 query returned src/changed.rs at its ready commit while the same small-change indexing sequence wrote 30 SQLite rows, compared with 40,034 in the previous implementation.

History and access still decide which content appears

Sharing a Git directory does not share repository permission. Each Durable Object retains its own derived data, and the existing current read gate runs before a response can return content. Hub shortlist hints remain conservative candidates, never an access grant.

The matcher still verifies literals, regular expressions, Unicode, case and path filters against the requested indexed commit. Historical snapshots are retained or rebuilt on explicit demand. The scripted screenshot shows an exact historical result after restart alongside private search before and after revoking the same approver. The original measurement record also retains the rename, deletion, regex and filter observations.

After restart, the scripted historical needle0 query returned src/search.txt at its pinned commit; a current private approver saw needle7, and the same viewer received no accessible repositories and no results after revocation.

A failed write must remain incomplete

A directory becomes readable only after all its bounded chunks have been persisted and its completion marker exists. If a write fails first, the chunks are retained for retry and traversal cannot treat them as a complete directory. The next execution resumes from durable SQLite data. Activation is also retried if its pointer write fails after content completes.

The paired screenshot follows the same query and commit through an injected directory-completion failure and a workerd restart. A separate real-binary fixture runs the previous Worker, the new Worker, the previous Worker again, then the new Worker against the same persisted store. Legacy snapshots remain readable on rollback. A later upgrade detects intervening legacy writes and restores shared blob coverage before reporting readiness.

Automatic superseded work can still be coalesced, while explicit historical and pre-upgrade demand survives. A directory that remains unavailable after the existing read retry limit produces generation-scoped partial coverage. That gap does not incorrectly masquerade as a byte-scan limit.

An injected directory-completion failure returned building with pending and partial coverage for needle7; after restarting workerd, the same query and commit returned src/search.txt in ready state.

Measure useful work and stop when it finishes

The completed-head path previously requested another alarm even when no indexing remained. It now leaves the object idle once indexing and bounded cleanup finish. Landing and recovery retain priority over one background indexing chunk.

The existing authenticated admin status reports aggregate SQL helper counters by operation, along with steps, failures and completions. These values live in memory and reset when the isolate restarts. They do not add a durable telemetry row per SQL statement, and they omit KV operations and direct streaming body-read cursors. The screenshot shows both the actual counters and the native next-alarm state after invoking a completed index.

This changes the cost shape, not the definition of a complete bill. At current published marginal SQL rates, 30 writes and roughly 151,000 reads are about $0.000181, or $0.91 for 5,000 instances of this specific 10,000-file small-edit workload before shared allowances. Initial indexing uses more reads and took longer locally. New content, wholesale rewrites, historical demand, retained storage, CPU and provider operations remain billable. There is no new per-organization dollar cap.

The original source/build hashes and exact scripted responses are retained with the landed engineering measurement. Production deployment and post-deploy Cloudflare measurements must reconcile before any account-wide saving is claimed. The screenshots establish the visible controlled results; owner review remains independent.

The scripted admin response exposed operation-level SQL reads and writes, 14 steps, one failure and two completions since isolate startup; the completed index alarm wrote zero SQL rows and left no next alarm.

Published rate references: Durable Objects pricing and SQLite index accounting. The engineering record and reproduction commands are in the landed product source at cf/scale/trunk-search-writes.md.