PaperlayerView exact revision on arXiv

Source-linked AI summary

ExperienceIndex: Artifact-Grounded Memory

Peter Baile Chen, Geoffrey X. Yu, Xinming Liu, Samuel Madden, Dan Roth, Jacob Andreas, Doug Downey, Michael Cafarella

arXiv:2610.10091v1cs.CLcs.AIcs.IRcs.LG

TL;DR

Agents answering knowledge-intensive questions over shared artifact corpora lack memory that retains artifact-specific experience from earlier tasks. ExperienceIndex stores single-artifact and artifact-pair experiences and injects retrieved knowledge into agent workflows; across diverse settings, it reports higher answer quality and lower online cost, with cross-task transfer and teacher-student benefits.

  • Problem

    Knowledge-intensive tasks require agents to answer questions over shared artifact corpora, but existing memory mechanisms focus on preferences, facts, or general reasoning patterns rather than persistent artifact-specific knowledge.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    ExperienceIndex stores artifact contribution summaries and artifact-pair relationships from prior reasoning traces, then retrieves and injects relevant experiences as middleware.

    Source: §1 · p. 2, §3 · p. 3, §3.1 · p. 4

  • Results

    Across diverse tasks and search frameworks, ExpIdx improves answer quality by up to 11.0 points and reduces online cost by up to 50.5%, with cross-task transfer and teacher-student benefits also reported.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Artifact-centered experiences can help agents assess artifact relevance and surface related artifacts that might otherwise be missed.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Knowledge-intensive tasks require answering many questions by reasoning about a shared corpus of artifacts (e.g., court cases, or scientific literature). As humans interact with these corpora, they naturally accumulate experiential knowledge about artifacts, enabling them to quickly identify the complete set of relevant artifacts for each new task. However, existing AI agents lack appropriate memory solutions to build or reuse such artifact-grounded experience, leading to lower answer quality and higher online cost. Existing memory solutions extract and reuse information from prior task-solving traces, but they primarily focus on user preferences, factual attributes, or abstract reasoning patterns rather than persistent artifact-specific knowledge. We introduce ExperienceIndex, a novel experience layer for AI agents that captures and reuses knowledge about artifacts based on prior reasoning traces. ExperienceIndex stores two complementary forms of experience: (i) single-artifact experiences that summarize an artifact's contribution to prior tasks and (ii) artifact-pair experiences that encode structural relationships discovered during past reasoning. Integrated as lightweight middleware, ExperienceIndex uses an experience retrieval mechanism to guide agents toward the complete set of relevant artifacts for new tasks, improving both answer quality and efficiency. Across diverse corpora and agentic solutions with different search frameworks, ExperienceIndex delivers consistent gains, raising answer quality by up to 11.0 points and reducing online dollar cost by up to 50.5%. We further demonstrate two benefits: (i) cross-task generalization, where experiences accumulated from text-to-SQL tasks transfer to factoid QA tasks over the same artifact corpus, and (ii) teacher-student learning, where experiences from a stronger model enable a weaker model to reach comparable performance.

1 INTRODUCTION

Repeated work over shared artifact corpora gives people experience that helps them find relevant information, but existing AI memory focuses on preferences, facts, or general reasoning patterns. ExperienceIndex stores artifact-grounded experience and uses it to guide agents toward relevant artifacts, with reported quality and cost gains across tasks and search frameworks.

1 INTRODUCTION

  • Repeated interaction with shared corpora builds artifact-specific mental anchors that help people identify complete relevant information more quickly.Initial corpus exploration can involve extensive reading and cross-reference tracing.Source: §1 · p. 1
  • Existing memory mechanisms store preferences or facts, or reuse general reasoning patterns, leaving artifact-specific knowledge unaddressed.Examples of abstracted memory include strategies extracted from prior math or web-navigation tasks.Source: §1 · pp. 1–2
  • ExperienceIndex records single-artifact contributions and artifact-pair relationships, then retrieves them to filter irrelevant artifacts and identify relevant ones for future tasks.The approach is grounded in artifacts involved in prior task reasoning.Source: §1 · p. 2
  • Across a broad suite of knowledge-intensive tasks and heterogeneous search frameworks, ExpIdx consistently improved answer quality by up to 11.0 points and reduced online cost by up to 50.5% relative to existing agentic solutions.Source: §1 · p. 2
  • Experiences from text-to-SQL tasks can benefit factoid QA over the same table corpus, while a weaker evaluation model with stronger-model experiences can reach comparable performance.The teacher-student comparison is reported in Table 4.Source: §1 · pp. 2–3

2 RELATED WORK

Existing memory and corpus-enrichment approaches target different anchors or rely on processing artifacts in advance. ExperienceIndex instead anchors memory in artifacts and updates it incrementally from prior task traces.

2 RELATED WORK

  • Profile-based memory stores user preferences, conversation history, or facts, whereas ExperienceIndex anchors experience in artifacts across shared-corpus workloads.The cited profile-memory work includes OpenAI and Chhikara et al. (2025); Shu et al. (2026) adds a temporal organization of user information.Source: §2 · p. 3
  • Abstracted memory reuses general reasoning patterns, while ExperienceIndex captures artifact analyses and relationships; the paper describes the approaches as complementary.The cited abstracted-memory work includes Suzgun et al. (2026), Chen et al. (2025b), and Ouyang et al. (2025).Source: §2 · p. 3
  • Offline corpus enrichment processes artifacts without downstream-task visibility and may require substantial upfront cost and full-corpus access; ExperienceIndex updates incrementally from executed-task traces.The cited enrichment work includes Chen et al. (2025a), Edge et al. (2024), Gospodinov et al. (2023), and Gutiérrez et al. (2024).Source: §2 · p. 3

3 EXPERIENCEINDEX

ExperienceIndex treats artifacts as persistent-memory anchors for agents searching a shared corpus. It retrieves prior artifact experiences, injects them into agent context and search outputs, and indexes new experiences from reasoning traces.

3 EXPERIENCEINDEX

  • ExpIdx adds a memory store indexed by artifact IDs, retrieves experiences for task descriptions or search-returned artifact IDs, injects them into context, and stores new trace-derived experiences.Its supported artifact modalities follow the solver model, and it can work with search frameworks that return artifact IDs.Source: §3 · pp. 3–4

3.1 EXPERIENCE REPRESENTATION

  • Single-artifact experiences record task descriptions, contribution summaries, and pertinent snippets, while artifact-pair experiences record structural relationships between artifacts.Single-artifact experience supports relevance assessment; pair experience can surface related artifacts that might otherwise be missed.Source: §3.1 · p. 4

3.2 EXPERIENCE RETRIEVAL

  • Retrieval finds candidate artifacts from prior single-artifact experiences, assesses their relevance to the query, then checks pair relationships to find relevant neighbors.Candidates may be retrieved from the query or artifact IDs; irrelevant artifacts and experiences are pruned.Source: §3.2 · p. 5

3.3 EXPERIENCE INJECTION

  • ExpIdx enriches both search outputs and the model prompt with retrieved experiences, and prompt reflection checks relevance and verifies experience against current artifacts.Search-output enrichment can prune irrelevant artifacts and introduce relevant ones before the model receives the tool output.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments compare ExpIdx with baseline solutions across diverse datasets and examine its performance, resource costs, cross-task transfer, and teacher–student setting.

4.1 SETUP

  • Evaluation covered multiple domains, modalities, and task types, using ScholarQA for SQA and ReAct elsewhere, alongside ReasoningBank and EnrichIndex baselines.EnrichIndex was not evaluated on SQA because it requires offline access to the full corpus.Source: §4.1 · pp. 6–7
  • The default protocol used 80% of each dataset for experience tasks and the remainder for evaluation, with a static memory store built from experience-task traces.Methods were evaluated with and without memory; the experience and evaluation models could be varied.Source: §4.1 · p. 7
  • Online metrics were answer score and dollars per thousand evaluation tasks; indexing metrics were build cost and added storage footprint.Online cost includes runtime expenses, including memory-store or enriched-index retrieval.Source: §4.1 · p. 7

4.2 RESULTS

  • ExpIdx achieved the highest average answer score, improving 3.9% relative to EnrichIndex, and the lowest average online cost, reducing it 20.8%, averaged across evaluation models.Table 2 reports performance across files, codebases, documents, and tables.Source: §4.2 · p. 7, Table 2 · p. 8
  • ExpIdx reduced average indexing cost by 115.0× and storage footprint by 73.0× versus EnrichIndex.EnrichIndex scans every document, while ExpIdx costs scale primarily with the number of tasks stored.Source: §4.2 · p. 7, Table 3 · p. 8
  • In teacher-student learning, experiences from the stronger Gemini-3-Flash model enable the weaker GPT-5-Mini model to reach performance comparable to the stronger model.Source: Table 4 · p. 8
  • Compared with ReasoningBank, ExpIdx improved average answer score by 5.01% and reduced average online cost by 19.8%.Its indexing cost was comparable to ReasoningBank’s, and its storage footprint was of the same order of magnitude.Source: §4.2 · p. 8, Table 2 · p. 8

5 ANALYSIS

The analysis tests how experience-store size, experience-model strength, artifact-relevance models, and lightweight reflection affect ExpIdx. It reports benefits from larger stores and stronger experience models, comparable relevance-model performance, and a drop in answer score without reflection.

Qualitative examples

  • Manually reviewed task-solving traces provide qualitative examples of benefits from both single-artifact and artifact-pair experiences.The examples are selected as representative cases.Source: §5 · p. 8

Experience as teacher-student learning

  • A weaker GPT-5-Mini evaluation model paired with Gemini-3-Flash experiences reaches performance comparable to a stronger evaluation model at substantially lower cost.For the same evaluation model, performance improves as the experience model improves.Source: §5 · p. 9

Performance varying size of memory store

  • At 20%, 40%, and 60% memory-store sizes, ExpIdx outperforms the 0% no-memory setting, and larger stores improve answer quality and cost efficiency.The experiments vary the store from the 80% dataset setting used in Section 4.Source: §5 · p. 9

Sensitivity to the artifact-relevance model

  • Nemotron-3-Ultra and Mimo-V2.5 achieve performance comparable to GPT-5-Mini for artifact-relevance analysis.Table 5 reports scores using different relevance models.Source: Table 5 · p. 9, §5 · p. 9

Ablation of lightweight reflection

  • Removing lightweight reflection lowers answer score; the step checks relevance and verifies experiences against current artifacts to mitigate noise and errors.Current artifacts take priority over past experiences when discrepancies arise from hallucinations, errors, or outdated information.Source: §5 · p. 9, Table 6 · p. 9

6 CONCLUSION

ExperienceIndex addresses the gap in memory for knowledge-intensive tasks over shared artifact corpora by capturing and reusing artifact-grounded experience from prior reasoning traces. The paper concludes that this approach improves answer quality while reducing computational cost, and reports cross-task generalization and teacher-student learning.

6 CONCLUSION

  • ExperienceIndex represents experience at the single-artifact and artifact-pair levels, retrieves it to surface relevant artifacts, and consistently improves answer quality while reducing computational cost across tasks and corpora.The conclusion also reports cross-task generalization and teacher-student learning as benefits.Source: §6 · p. 9

A IMPLEMENTATION DETAILS OF ExpIdx

The implementation injects experience-informed artifacts into the model's working set, then extracts and indexes experiences from completed tasks for future retrieval. Its indexing process derives artifact contributions and pairwise relationships from past reasoning traces.

A.1 EXPERIENCE INJECTION

  • Experience injection starts with search results and retrieved artifact experiences, then removes irrelevant artifacts, retains or adds relevant ones with experience snippets, and leaves inconclusive artifacts unchanged.The model receives the resulting A′ instead of the original artifact set A.Source: §A.1 · p. 13

A.2 EXPERIENCE INDEXING

  • Experience indexing extracts single-artifact and artifact-pair experiences from completed-task traces, then stores them with embeddings of task descriptions and contribution summaries.The trace combines reasoning, explicit thinking, final answers, and artifact citations; an LLM summarizes each artifact's contribution and infers relationships between artifact pairs.Source: §A.2 · p. 13

B.2 PROMPT FOR INJECTING EXPERIENCES IN MODEL SYSTEM PROMPT

  • The appendix identifies a system prompt for injecting retrieved experiences into the model's system prompt.The prompt is provided in Table 8.Source: §B.2 · p. 15, Table 8 · p. 15

B.3 PROMPT FOR EXTRACTING EXPERIENCES FROM REASONING TRACES

  • The extraction prompt asks for direct relationships between artifact pairs mentioned in a response and excludes pairs without a direct relationship.Table 9 is identified as the prompt for generating contribution summaries and relationships.Source: §B.3 · p. 16, Table 9 · p. 16

B.1 PROMPT FOR ARTIFACT RELEVANCE ANALYSIS

The appendix identifies a prompt for artifact-relevance analysis, provided in Table 7.

B.1 PROMPT FOR ARTIFACT RELEVANCE ANALYSIS

  • Table 7 contains the prompt used to analyze artifact relevance.The passage identifies the prompt and its table, but supplies no prompt contents.Source: §B.1 · p. 14, Table 7 · p. 14

C.1 OVERVIEW

A qualitative analysis illustrates how artifact-grounded experiences help agents find relevant files and snippets or recover tables missed by retrieval. The examples show single-artifact experience improving task efficiency and artifact-pair experience improving information coverage.

C.1 OVERVIEW

  • The MMQA example centers on a table that vector search missed but ExpIdx surfaces to help reach the correct answer.Source: Figure 4 · p. 17
  • With single-artifact experience, ExpIdx surfaces specific files and snippets, helping the agent answer correctly with less context load and greater efficiency.Without ExpIdx, generic keyword search returns irrelevant files, and the agent inspects only a subset of relevant files.Source: §C.1 · p. 17, Figure 3 · p. 17
  • With artifact-pair experience, ExpIdx adds neighbor tables when retrieval returns only a subset, enabling the agent to answer correctly.The MMQA example shows a table missed by vector search being surfaced through artifact connections.Source: Figure 4 · p. 17, §C.1 · p. 18

C.2 SINGLE-ARTIFACT EXPERIENCE

The Locomo example contrasts a standard agentic solution with ExpIdx on the same task, showing different answers about when Jolene started yoga.

C.2 SINGLE-ARTIFACT EXPERIENCE

  • The standard solution's grep results include several yoga-related files, while its answer says the start year is not specified.Source: §C.2 · p. 19, Table 10 · p. 19
  • Table 11 presents the ExpIdx version of the same Locomo example shown as a standard solution in Table 10.Source: Table 10 · p. 19, Table 11 · p. 20
  • ExpIdx's answer places Jolene's yoga start at about 2020, based on her saying she had practiced sporadically for about three years.Source: §C.2 · p. 20, Table 11 · p. 20

C.3 ARTIFACT-PAIR EXPERIENCE

The MMQA example illustrates artifact-pair experience: incorporating tables introduced through ExpIdx edge connections changes the answer to correct.

C.3 ARTIFACT-PAIR EXPERIENCE

  • Table 12 illustrates an example of ExpIdx leveraging artifact-pair experience.Source: §C.3 · p. 21
  • On MMQA, the answer becomes correct once tables introduced through ExpIdx edge connections are incorporated.Source: Table 12 · p. 21, §C.3 · p. 21

D EVALUATION DETAILS

The evaluation details specify task sampling, scoring protocols for the benchmarks, and the judge model used for LLM-judged scores.

D EVALUATION DETAILS

Loading 2610.10091v1…