Source-linked AI summary
Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Stores
Yu Pan, Hongfeng Yu
TL;DR
Existing RAG systems treat the document store as fixed, while offline structure-building pipelines construct it without question-specific supervision. This paper trains a non-parametric knowledge base from supervised question–answer pairs and evaluates the frozen result on held-out questions, finding that gains increase with key overlap and corpus coverage.
Problem
RAG and offline structure-building systems do not use supervised question–answer signals to decide which store structure future questions need.
Method
A curator answers supervised questions from the current store, sees the gold answer, edits its structure, and passes the resulting store to a fixed independent reader for held-out evaluation.
Results
+0.294 F1 and 25% fewer actions occur on trained questions, while the advantage decreases as key overlap falls to none.
Takeaways & Limitations
Accuracy rises with the share of the corpus covered by indexes, indicating that more training questions can widen coverage and generalization.
Takeaways & Limitations
The evaluation uses a single seed and agent, while the four-point key-coverage gradient is KBGym-only.
Abstract
from arXiv · showhide
Retrieval-augmented generation treats the document store as a frozen input, and the offline pipelines that do build structure over it build it unsupervised -- a whole corpus indexed at uniform effort, with no signal about which structure a question will need. We instead treat the knowledge base as a non-parametric model trained on (question, answer) pairs: a curator agent answers a supervised question against the current store, is shown the gold answer, then edits the store. The store carries forward, and we evaluate the curated store with a test set, on two contamination-free benchmarks: KBGym, a fictional-universe generator we release, and PhantomWiki. Generalization is probed with four question groups of decreasing overlap with the training set: the trained questions themselves, and unseen questions sharing both of their keys with training, one key, or neither. The curated store's advantage grows with overlap -- from parity where no key was shared, through +0.176 F1 where both keys were, to 25% fewer actions at +0.294 F1 on the trained questions, the one cell significant on both benchmarks -- while matching HippoRAG's gains with 1,913 links against its 196,112: per point of corpus covered, 1.5x the action saving and 2.1x the accuracy gain. Accuracy rises steadily with the share of the corpus the indexes cover, so training on more questions widens coverage, and with it the generalization.
I. INTRODUCTION
The paper treats the knowledge base as a trainable non-parametric model: supervised QA pairs guide a curator to edit store structure, which is then evaluated on held-out questions. This directly addresses the gap that existing store-building systems construct structure unsupervised and usually measure only downstream answers.
- Agent-trained knowledge bases: Supervised curation trains the knowledge base by answering each question, revealing the gold, and editing the store for future readers.The store is evaluated after freezing, using an independent reader without gold access and a fixed action budget.
- Agent-trained knowledge bases: The trained store constructs access structure by turning verified reasoning paths into traversable links and key-specific index documents.This lets future readers perform lookup instead of re-deriving the reasoning path.
- Agent-trained knowledge bases: 1,913 links deliver 1.5× the action saving and 2.1× the accuracy of an unsupervised entity index with 196,112 links.The comparison is framed as greater efficiency per point of corpus indexed, not a higher performance ceiling.
- Relation to retrieval: Unlike classic and agentic RAG, which improve retrieval or reasoning over a fixed corpus, this approach trains the store while keeping the reader policy fixed.The paper presents the two axes as orthogonal and composable.
- Relation to prior structure building: GraphRAG, RAPTOR, LightRAG, and HippoRAG build structure before questions arrive, whereas this method incrementally follows the questions actually asked.The contrast is supervised, query-distribution-aware structure versus unsupervised construction blind to future queries.
- Evaluation gap: Published curation systems generally report downstream task scores without a train/test split over questions or direct measurement of the store.The paper positions its protocol as a way to measure store learning without contamination.
III. PROBLEM FORMULATION
The formulation makes the knowledge base a directed document graph whose learned structure is evaluated by reader accuracy and cost. Training edits the graph from supervised pairs, with the intended benefit being reusable access structure that amortizes curation cost.
- Store and reader: A knowledge base is a directed graph of self-contained sentence documents whose outgoing links expose learned structure.The agent sees and writes document text and links; index documents are ordinary documents whose value lies in their links.
- Store and reader: Reader quality is defined by expected F1 accuracy and expected action cost under a fixed policy, question distribution, and action budget.The reader can search, read, and answer while retaining only the current question and recent action-result pairs.
- Training: Training consumes supervised pairs (q, a∗) and edits the initial store K0 to produce a trained store KT.This establishes the store as the object changed by the learning procedure.
- What the store learns: Access structure generalizes across questions, whereas content documents that record verified answers mainly serve the instance that produced them.Indexes over named keys can serve every question whose search names that key.
- Amortization: Curation is intended to amortize its token cost after enough queries benefit from the per-question action saving.The break-even query volume is determined by training cost divided by the per-question saving.
- Learning interpretation: The supervised-learning correspondence maps store configuration to parameters, curation to optimization, and held-out reader performance to loss.Matching tools, budgets, and answer rules make the generalization gap well defined.
A. Universe, Store and Questions
KBGym is a contamination-free fictional-person environment built from atomic fact documents and structured question categories. Its action design makes exhaustive search expensive while allowing compact linked indexes to reduce reader effort.
- Universe and questions: KBGym samples 500 fictional people with family, social, attribute, birthdate, and name-distractor facts.The generated universe follows the PhantomWiki style and supports controlled relational questions.
- Universe and questions: 5,864 atomic single-sentence documents form an initially link-free store, with ten question categories spanning entity chains, aggregation, abstention, and joins.The first three categories are retrieval-friendly; later categories target relations between documents.
- Action separation: The reader uses only search, read, and answer, while editing actions are reserved for the curator.The reader’s toolset is identical during training phase 1 and examination.
- Action costs: Twenty-nine residents of a city cost six search actions to enumerate, whereas reading a complete index costs one action.This deliberate price gap is the efficiency target of trained store structure.
- Action costs: link_many attaches up to forty targets in one action, and batches above forty are rejected.The largest genuine key has thirty-six members, so larger batches would represent search results rather than valid sets.
- Measurement: Provenance fields track original facts, authored or edited status, and absorbed origins, enabling exact coverage and duplication measurement.These environment-maintained fields are invisible to the agent.
C. Design Rationale
The protocol is designed to preserve a clean comparison between training and testing while constraining curation to realistic, uniform feedback. Its design emphasizes reproducibility, separation of roles, and disciplined index construction.
- Methodological controls: The measurement pipeline uses exact golds and provenance counts rather than an LLM judge, making reported store metrics reproducible.The sole judge vetoes near-duplicate writes and does not score outcomes.
- Methodological controls: Fig. 1 defines one training iteration followed by an examination of the resulting frozen store.The figure provides the protocol shape for both curation and evaluation.
- Two-phase training: Phase 1 predicts before seeing the gold, then exposes the gold for consolidation, with leftover forward budget forfeited.A budget-exhausted forward receives F1 0 but still supplies the gold for updating.
- Two-phase training: Every intermediate store state can be reconstructed by replaying the validated edit trace against epoch snapshots.This supports per-iteration store trajectories without storing every state separately.
- Curation feedback: Backward feedback is uniform: the agent receives its trajectory, gold answer, F1, and one coarse outcome before identifying named keys and missing indexes.The protocol provides no oracle localization or failure-specific branching.
- Curation feedback: The curation constraints favor one key at a time, links over recited membership, and precision before completeness.The design prioritizes useful complete indexes while allowing unreached keys to be addressed by later questions.
- Curation feedback: Gold-revealed answers let the agent re-search after errors and identify repairs from the difference between searched locations and answer locations.This supplies credit assignment without oracle localization.
VI. EXPERIMENTAL DESIGN
The experiments compare question-authored and corpus-only store preparation under a shared evaluation protocol. PhantomWiki supplies an external-validity arm, while the baselines construct structure without question or answer supervision.
- PhantomWiki uses 16 family trees, 405 person articles, and 480 generated QA items spanning eight templates and four composition shapes.Hop depth ranges from 1–5, with two reserved templates forming test_out.
- Both benchmark arms use the same split scheme, reader, action set, and budgets, differing only in who authored the questions.
- All baselines induce structure from the corpus alone, without access to questions or answers, while reader, test sets, and budgets remain identical.
C. Implementation
The implementation represents the store as linked document records with a derived vector index and uses fixed-model curator and reader budgets. Four experiments examine dynamics, key-coverage generalization, retention, and the store itself.
- Implementation: The store is a dictionary of document records with outgoing ids; edits trigger batched re-embedding of dirty documents in a separate vector index.Links are ids rather than copies, and deletes cascade to prevent dangling edges.
- Implementation: Curator and reader agents use gpt-5-mini with temperature 0.3, low reasoning effort, static schemas, and tool-forced decisions.The recorded model snapshot is gpt-5-mini-2025-08-07.
- Implementation: The deduplication guard compares new or edited documents with their three nearest neighbors and invokes an LLM judge above cosine similarity 0.90.The judge receives 500 completion tokens at minimal reasoning effort.
- Experiment Suite: Four experiments test training dynamics, the key-coverage gradient, retention by question class, and store structure, followed by break-even analysis.
A. E1: Training Dynamics
E1 shows that training builds store structure progressively: performance changes with training exposure and corpus coverage, while the chosen configuration remains undertrained. The first epoch reaches 24.8% coverage, and the second epoch’s curve flattens.
- Configuration: The 500-person configuration leaves relation-level headroom, with untrained F1 falling from 0.83 at 120 people to 0.67 at 500.Mean steps increase from 6.7 to 9.2, leaving roughly 3× room above the approximately 3-step floor.
- Configuration: The main run trains on 100 questions for two epochs, totaling 200 iterations, 18.8M tokens, 6.8 hours, and approximately $13.
- Training Dynamics: Coverage reaches 24.8% by the end of epoch 1, while Fig. 2 shows key-covered groups improving and uncovered groups remaining unchanged.
- Training Dynamics: Cost falls and accuracy rises as the share of corpus reached by indexes increases.The same runs are pooled against corpus coverage, with the final point representing the epoch-2 store.
B. E2: The Key-Coverage Gradient
The key-coverage gradient shows that supervised structure transfers most strongly to questions whose keys overlap with training, while broader corpus coverage determines overall performance. The store remains undertrained, so additional questions are expected to widen coverage and improve generalization.
- Key-Coverage Gradient: +0.294 F1 and 25% fewer actions occur on trained questions, while accuracy gains decay to nothing when neither key was covered.With both keys covered, F1 improves by +0.176; with one key, +0.059.
- Training Dynamics: The first curation pass mainly lowers retrieval cost, while the second improves accuracy by adding more indexes aimed at the same questions.On trained questions, actions move 10.35 → 8.59 in epoch 1 and 8.59 → 8.00 in epoch 2, while F1 moves 0.735 → 0.765 and 0.765 → 0.912.
- Cross-Benchmark Transfer: Accuracy rises with key overlap on both benchmarks, while significant action savings appear only for questions trained exactly.The trained group is the only cell significant on both KBGym and PhantomWiki.
- Coverage and Efficiency: The trained store matches B3’s accuracy where both keys were covered and falls back to the flat store where neither was covered.B3 remains uniformly effective because it indexes the whole corpus, whereas the trained store follows the keys touched during training.
- Coverage and Efficiency: Our store reaches 27.6% corpus coverage versus 100% for offline baselines, explaining its lower pooled performance despite stronger gains per covered point.Against B1, the store saves 0.035 actions and gains +0.0045 F1 per coverage point, versus 0.023 and +0.0021 for HippoRAG-style B3.
- Training Dynamics: Coverage grows almost linearly at 0.25 points per question through epoch 1, but repeats flatten from 24.8% to 27.6%, indicating undertraining rather than saturation.Extrapolating the initial slope suggests roughly 400 distinct questions for full coverage, stated as a lower-bound projection.
C. E3: Retention by Question Class
Retention analysis separates accuracy gains from action savings across question classes. Repeated curation improves relation-level reasoning, while forward-chain classes were already solved by document retrieval.
- Class-Level Retention: +0.19 forward F1 is gained on the same 100 questions when seen a second time, with improvements concentrated in relation-level classes.Forward chains remain at 1.00, while counts, joins, reverse lookup, intersection, deep counts, and abstention improve.
- Efficiency: 7.4 steps and 70k tokens replace 8.4 steps and 118k tokens on the repeated questions, while budget exhaustion falls from 15/100 to 8/100.Among questions resolved within budget in both epochs, lookup length falls 7.16 → 6.37 steps as F1 rises 0.759 → 0.892.
- Interpretation: The two-epoch comparison is not a controlled store contrast because the store evolves during epoch 1 and the epoch label mixes store state with question order.The controlled comparison is the trained column of Table IV, where the larger saving is ρ = 0.747.
D. E4: What the Agent Built
The agent primarily builds a sparse, mostly additive indexing layer rather than rewriting source documents. The resulting indexes are accurate but narrow, with coverage concentrated on keys named by training questions.
- Curation shape and cost: +287 documents (+4.9%) and 1,913 links were added, nearly all during epoch 1.The backward pass used 307 add and 284 link_many calls, versus only 3 edits, 12 deletes, and 9 unlinks.
- Index classification: 242 of 287 authored documents (84.3%) were genuine indexes, while 43 (15.0%) were empty stubs and 2 were materialized answers.An index required a link, a resolvable key, and text that did not enumerate the key’s members.
- Index quality: Index quality reached 94% precision and 91% member recall over the 220 indexes whose keys resolved.Attribute indexes were near-perfect, while relational indexes were smaller and noisier; two-hop relations were not worse than one-hop.
- Coverage and structure: 1,896 of 1,913 links attached indexes, but only 1,621 of 5,864 source documents (27.6%) were reachable from an index in one read.The structure is therefore flat and provides entry points more readily than layered navigation.
- Break-even: On trained questions, the store saved 5,466 answering tokens per question, repaying its 18.8M-token construction cost after approximately 3,400 questions.This break-even estimate prices repeated queries within the covered population rather than generalization.
- Network visualization: The shared t-SNE layout shows document neighborhoods under the same embedding function used for retrieval, so only local proximity should be interpreted.One layout, computed on the final store, is reused across training snapshots.
VIII. DISCUSSION AND LIMITATIONS
The curated indexes are accurate, but their coverage is limited by question-specific construction and incomplete expansion. The paper also identifies scope boundaries involving harmful edits, online evaluation, and benchmark validity.
- Coverage boundary: The main limitation is coverage, not construction: accurate indexes cover only the quarter of the corpus named by training questions.The backward budget may not populate a large index in one iteration, and the agent is not asked to revisit it.
- Coverage boundary: The agent indexes the specific key named by a question rather than the broader class to which that key belongs.For example, asking about one person’s friends produces a person-specific index rather than an index for friends generally.
- Editing boundary: Adding GraphRAG-style summaries to an already-searchable store reduced accuracy, making selective non-editing part of the curation boundary.Downstream-only evaluation may miss gradual store degradation while individual answers remain acceptable.
- Online evaluation: With the store changing during deployment, held-out accuracy is no longer well defined and prequential evaluation is required instead.The online variant needs a different measurement, not a different curator.
- Threats to validity: Validity is constrained by template-rendered language, a shared reader and curator model family, a single seed, and KBGym-only gradient analysis.The PhantomWiki generator is not controlled, and the external arm lacks support-set diagnostics.