Source-linked AI summary
RAG Deserves an Index: Why Ingest-Time Compilation Beats Query-Time Interpretation
Kyle Wild, Yusuke Takahashi, Asako Uraki
TL;DR
The paper addresses the repeated cost and unreliability of reconstructing corpus meaning from raw text at query time. It proposes ingest-time semantic compilation, which maintains embeddings and provenance-validated atomic claims as a database-like substrate. On held-out transcripts, compiled claims won all 32 budget-by-model cells, while the only baseline that kept pace used roughly 21× more query-path tokens; the paper also identifies scope and cost boundaries for compilation.
Problem
RAG systems repeatedly ask language models to reconstruct meaning from raw corpus text, while rising context volume increases inference spend and can degrade reliability.
Method
The paper proposes ingest-time semantic compilation: a maintained substrate combining incrementally updated embeddings with atomic claims whose provenance is validated at compile time.
Results
Facts won all 32 budget-by-model cells; at 2,048 tokens, they reached 85.2% accuracy from roughly 2.2k reader tokens versus 72.5% from roughly 16.3k for the best chunk configuration.
Takeaways & Limitations
The results support treating compiled semantic structures as first-class database objects and choosing compilation per corpus or document rather than by default.
Takeaways & Limitations
Compilation may be unsuitable for small, highly volatile, or rarely queried corpora, and the maintenance pilot is synthetic with idealized incremental updates.
Abstract
from arXiv · showhide
Nearly every retrieval-augmented question-answering system in production ships with a hidden interpreter: on each query a language model re-derives the meaning of raw corpus text and then throws that work away. Cheaper models do not close the gap: per-token prices have fallen by orders of magnitude while inference spend has risen, because context volume grows faster than prices fall. This is the modern equivalent of the full-table scan, and the remedy is the one databases found fifty years ago: do the expensive work once, at write time, into a maintained structure that makes reads cheap. A corpus whose read pattern is known before it ever meets a user can and should be indexed too. We call the paradigm ingest-time semantic compilation (ISC): compile a corpus's meaning into a queryable substrate with two coupled layers - incrementally maintained embeddings, and atomic claims whose provenance is validated at compile time - and treat that substrate as a first-class database object with its own DDL, maintenance contract, migration contract, and cost model. Two existence proofs support it. Substrate upkeep scales with change rather than corpus size: incremental updates run 33.7x cheaper than reconstruction while tracking it to floating-point precision. And on a held-out sample of 500 broadcast-interview transcripts, compiled claims as the retrieval payload win all 32 budget-by-model cells: 85.2% correct from roughly 2.2k reader tokens against 72.5% from 16.3k for the best chunk configuration anywhere. The only baseline that keeps pace is a contextualized-chunk pipeline with hybrid retrieval and reranking, statistically indistinguishable from compiled claims at roughly twenty-one times the query-path tokens - and it reaches that parity, we argue, precisely because it has itself begun to compile. We close with the systems agenda this opens, from compilation planners to read planning.
1 The Interpreter We Ship on Every Query
Query-time retrieval repeatedly asks a language model to reconstruct meaning from raw text, paying the same interpretive cost on every query. The paper frames this as an unindexed semantic scan whose economic, behavioral, and epistemic costs compound.
- Every query makes the model locate relevant statements, resolve references, distinguish assertions from quotations, and recover meaning from surrounding transcript text.
- Query-time semantic reconstruction repeats interpretive work, whereas indexing moves that cost to write time so reads can reuse the result.
- Orders-of-magnitude token-price declines have coincided with rising aggregate inference spend because context volume grew faster than prices fell.
- Longer inputs measurably reduce reliability, lose relevant material in the middle, and let irrelevant material distract the model.
- Query-time reconstruction also lacks a source-versus-inference boundary, allowing meaning to be silently re-guessed and extraction errors to propagate into answers.
2 The Vision: A Compiled Semantic Substrate
Ingest-time semantic compilation persists a corpus’s meaning in a maintained substrate with geometric and symbolic layers, governed like a database object. The design includes provenance validation, maintenance and migration contracts, and a cost-based decision about how deeply to compile.
- Ingest-time semantic compilation performs semantic work when documents arrive and persists the result for later consultation in a two-layer substrate.
- Embeddings and incrementally maintained index structures answer where meaning is similar, while compiled claims answer what was said and by whom.
- Compiled claims carry verbatim evidence and attribution, making the payload itself readable while retaining raw transcripts for inspection and missing-context cases.
- The substrate is governed by compilation, maintenance, migration, and cost contracts, with expected read count compared against break-even statistic R* to choose compilation depth.
- Exact-quote validation rejects claims whose supporting spans cannot be located in the canonical transcript before insertion into the substrate.
- A PostgreSQL implementation represents claims, evidence, hashes, lifecycle state, and pipeline provenance as linked rows, with idempotent recompilation and index propagation.
3 Two Existence Proofs
Two existence proofs support ingest-time semantic compilation: incremental maintenance tracks full recomputation at much lower cost, while compiled claims outperform raw-text chunking on held-out dialogue question answering.
- 3.1 Maintenance scales with change, not size: 33.7× cheaper per update and 23.8× cheaper cumulatively, incremental low-rank updates tracked the fully recomputed subspace to floating-point precision.The pilot measured 8.4 ms per incremental update versus 283 ms for full re-decomposition; maximum principal-angle drift stayed below 10^-11 degrees with recall@10 of 1.0.
- 3.2 Compiled payloads dominate the read-time frontier: Facts won every one of 32 budget-by-model cells, with all 24 facts-versus-chunk comparisons surviving Holm correction.The held-out evaluation used 500 transcripts and 499 questions sharing no document with extraction development.
- 3.2 Compiled payloads dominate the read-time frontier: 85.2% correct from roughly 2.2k reader tokens beat the best chunk configuration’s 72.5% from roughly 16.3k tokens at a 2,048-token budget.Facts at a 256-token budget reached 69.5% and beat every chunk policy at 2,048 tokens.
- 3.2 Compiled payloads dominate the read-time frontier: Fact payloads carried gold evidence in 98–99% of cases, while chunk reading accuracy declined from 81% to 73% as more text was supplied.The decomposition attributes the shape to coverage and reading accuracy, not only retrieval placement.
- 3.2 Compiled payloads dominate the read-time frontier: The strongest contextualized-chunk stack was statistically indistinguishable from facts, but used roughly 21× the query-path tokens.At 2,048 tokens, the stack reached 88.0% versus 85.2% for facts, with McNemar p=0.202; at 16,384, it reached 87.4% versus 83.6%, p=0.076.
- 3.2 Compiled payloads dominate the read-time frontier: The contextualized stack keeps pace precisely insofar as it performs compilation, while still lacking validated provenance against the source span.Its vector and hybrid arms reached 73.9% and 75.6%, losing to facts after Holm correction.
4 The Research Agenda
The paper proposes a database-style agenda for semantic compilation, centered on planners, integrity constraints, maintenance, shared substrates, and read planning. It also cautions that compiled-payload benefits may not automatically survive agent-mediated retrieval.
- The compilation planner: ISC planners should decide per document what to compile and to what depth using query popularity, volatility, compilation cost, and R∗.Possible compilation depths include embeddings, facts, or enriched relations; rarely queried material may remain interpreted.
- Provenance as an integrity constraint: Exact-quote validation makes provenance an integrity constraint that can mechanically reject hallucinated compiled claims at write time.The proposed constraint language includes span containment, speaker attribution, temporal validity, and revision lineage.
- Maintenance and migration at production scale: Production-scale maintenance remains open beyond the controlled pilot, including streaming revisions, migration anchors, bounded staleness, and redo logging.The paper reports incremental subspace tracking and Procrustes alignment only in a controlled pilot.
- The substrate as shared infrastructure: A compiled substrate could serve multiple agents and applications, raising questions about isolation, versioning, and claim-level access control.The substrate is framed as shared infrastructure accessed through standard interfaces.
- Reading as a planned operation: Read planning can treat context quantity as a physical-plan choice because more context sometimes reduced accuracy.The reader model’s degradation curve is proposed as an input to a substrate-aware cost model.
- How to evaluate a compiler: End-to-end answer accuracy is insufficient to evaluate compilers because it conflates compilation, retrieval, and reading.The paper proposes separately measuring coverage, provenance validity, self-containment, duplication, staleness, and validation-gate rejection.
- Honest boundaries: In agent-mediated retrieval, the compiled-payload advantage did not automatically survive, leaving agent–substrate interaction as an open boundary.The paper specifically flags when agents should request siblings, lineage, or raw source.
5 Related Visions
Related approaches move some work to ingest time, but they differ in whether they compile retrieval pointers or reader payloads and whether they validate outputs against sources. ISC combines payload compilation with source validation within a database-governed substrate.
- Prior ingest-time approaches: Document expansion, proposition indexing, and contextualized chunking improve retrieval by compiling pointers while the reader still receives raw text.These approaches differ in their specific transformations, but the cited comparison identifies the pointer-versus-payload distinction.
- ISC’s distinction: ISC compiles the payload consumed by the reader and validates it against its supporting source span.This is the stated distinction from query prediction, proposition indexing, and contextualized chunking.
- Comparison framework: Table 2 compares what each approach compiles, source validation, and the reading model’s input; only ISC fills both middle columns.The caption defines the table’s comparison dimensions without providing additional row-level details.
- Payload compilation and validation: Agent-memory systems compile payloads but lack a validation gate, allowing extraction errors to propagate into downstream answers.GraphRAG-family systems are described as addressing a different multi-hop query class.
- Database lineage: The database lineage supplies maintenance contracts, integrity constraints, and cost models for governing derived semantic structures.ISC applies this discipline to meaning as a maintained database object.
6 Conclusion
The conclusion argues that semantic work can move from query time to ingest time into an affordably maintained, mechanically validated substrate. Controlled studies support feasibility, while the paper leaves a broad database-style systems agenda open.
- Conclusion: Ingest-time semantic compilation treats meaning as compilable, affordably maintained, mechanically validated, and cheaper to read than its source text.The conclusion frames this as the paper’s central claim about governing derived semantic structures.
- Conclusion: Two controlled studies support the feasibility of moving semantic work to write time and governing the resulting substrate.The conclusion does not enumerate the studies’ individual results here.
- Conclusion: The open agenda includes planners, constraints, migration, shared substrates, and read planning.The paper explicitly characterizes this agenda as resembling database research.
A The Break-Even Read Count
The break-even read count R∗ compares one-time compilation and ongoing maintenance with repeated query-time reconstruction. In the held-out study’s fixed corpus, the estimate is about 580 reads, placing payback within the first thousand questions.
- Definition: R∗ is the finite read count at which compilation and query-time semantic reconstruction have equal cost, requiring c_q > c_r.It becomes a planner statistic when read cost is lower on the compiled path and maintenance scales with changes rather than corpus size.
- Planner interpretation: R∗ is practically evaluated per document, where expected read count is compared with that document’s compilation cost.The paper argues for estimating read demand per document rather than per corpus.
- Held-out estimate: R∗≈580 reads in the 500-document held-out corpus, based on roughly 26.3M compilation tokens and the contextualized-chunk stack’s per-read gap.The compilation estimate used a 20-document replay and extrapolation, with no corpus changes during the study.
- Interpretation: The estimate places compilation payback inside the first thousand questions for this corpus.The equivalence is in token counts rather than dollars because compile-time and query-time work use different models.
- Assumption: With W=0 in the fixed-corpus study, the maintenance term vanishes, whereas a live corpus would add change-dependent maintenance cost.The paper links this missing term to the separate maintenance study.