Source-linked AI summary
Beyond Context Windows: Persistent Discovery Context for Data-Centric Agents
Jalal Mahmud
TL;DR
Data-centric agents repeatedly identify relevant data objects, but successful discovery outcomes are usually discarded rather than reused. The paper introduces persistent discovery context, a memory layer that stores intent-to-object mappings and augments retrieval. Across three structured data environments, it consistently improves retrieval, remains useful with LLM-generated memories, and exhibits interference from incorrect memories.
Problem
Data-centric agents must discover relevant data objects before planning or execution, yet successful discovery outcomes are typically discarded instead of preserved across tasks.
Method
Persistent discovery context stores prior intent-to-object mappings and retrieves them to augment registry-based object rankings.
Results
Across three domains, persistent discovery context consistently improves retrieval over metadata-only search and remains effective with automatically generated memories.
Takeaways & Limitations
Successful discovery outcomes constitute a reusable memory artifact that complements conversation, experience, and workflow memory for data-centric agents.
Takeaways & Limitations
Semantically similar but incorrect memories can degrade retrieval below the no-memory baseline through cross-family interference.
Abstract
from arXiv · showhide
Data-centric agents repeatedly perform a discovery step before planning or execution: identifying the data objects relevant to a task. Yet successful discovery outcomes are typically discarded rather than reused. We introduce persistent discovery context, a lightweight memory layer that stores prior intent-to-object mappings and reuses them to augment future retrieval. Across three structured data environments, persistent discovery context consistently improves retrieval quality over metadata-only search, remains effective with automatically generated memories, and exposes a reproducible interference failure mode. In lexically sparse domains, memory-only retrieval can even outperform metadata-based retrieval. These findings suggest that discovery outcomes constitute a useful form of reusable context for data-centric agents.
1 Introduction
Data-centric agents must discover relevant data objects before acting, but successful discovery is usually discarded instead of reused. The paper asks whether these intent-to-object mappings should persist across tasks and introduces persistent discovery context to address this gap.
- Discovery is a prerequisite for data-centric agents before planning or execution.
- Existing agent memory systems retain conversations, observations, skills, and workflows, whereas data discovery is typically handled per task.
- A discovery outcome maps task intent to the data objects that proved useful for completing the task.
- Persistent discovery context stores prior intent-to-object mappings and reuses them to augment future retrieval across three structured data environments.
2 Persistent Discovery Context
Persistent discovery context adds a registry-level memory of prior intent-to-object mappings to standard metadata retrieval. Retrieved memories augment object rankings, with α controlling their influence and α = 0 recovering registry-only search.
- The registry contains tables and attributes with natural-language metadata, and standard text retrieval produces candidate objects.
- A discovery memory records a task description and the data objects previously found relevant, then accumulates across tasks within one data environment.
- For query q, registry retrieval scores candidate objects while similar discovery memories are retrieved to augment their rankings.
- The memory weight α controls memory influence; α = 0 gives standard registry search, while setting r(q, oi) = 0 produces memory-only retrieval.
3 Experiment
The evaluation spans three structured data environments and held-out task families, comparing metadata, registry, memory-only, and combined retrieval. Combined persistent context improves retrieval broadly, but gains vary and incorrect memories can cause interference.
- Datasets and Tasks: The study evaluates three structurally distinct environments: Finance, NYC Collisions, and Northwind.They cover banking, crash records, and trading data, respectively.
- Datasets and Tasks: Tasks are organized into 15 families, with one seed task initializing memory and 5–12 held-out evaluation tasks per family.
- Experimental Conditions: The four conditions compare raw metadata, registry descriptions, memory-only ranking, and registry retrieval augmented with discovery memory.
- Main Results: F1@5 rises with TF-IDF from 0.396 → 0.499 in finance, 0.299 → 0.482 in NYC collisions, and 0.446 → 0.528 in Northwind when memory augments registry retrieval.
- Main Results: Neural retrieval gains are +0.167 in finance, +0.119 in Northwind, and +0.051 in NYC collisions.The NYC neural baseline is already strong, with S1⋆MRR = 0.848.
- Main Results: Memory-only retrieval exceeds registry retrieval in NYC collisions under TF-IDF, with F1@5 of 0.412 versus 0.299.In finance and Northwind under TF-IDF, registry retrieval remains stronger than memory-only retrieval.
- Main Results: Families with distinctive object vocabularies benefit most, while families whose objects recur across task types show near-zero or negative gains.
- Main Results: LLM-generated memories remain useful despite imperfect construction, outperforming S1 in finance and NYC and remaining above S0 in all three domains.Their object-level F1 agreement is 0.698, 0.762, and 0.735 for finance, NYC, and Northwind.
4 Related Work
The paper relates persistent discovery context to agent memory, schema linking, Text-to-SQL, metadata catalogs, and data-discovery systems. It distinguishes discovery memory by reusing agent-generated object selections across related tasks and structured-object mappings.
- Prior agent-memory systems retain conversational context, observations, executable skills, workflows, and execution experience for data-centric tasks.
- Related data systems focus on schema linking, reasoning, query generation, metadata management, or improved object representations.
- Discovery memory is conceptually closest to relevance feedback but reuses agent-generated object selections across semantically related, lexically distinct tasks.
5 Discussion and Conclusion
Across three domains, persistent discovery context improves retrieval over metadata-only search and remains useful with automatically generated memories, while memory interference remains a primary limitation.
- Discussion and Conclusion: Persistent discovery context consistently improves retrieval quality over metadata-only search across three structurally distinct domains.The evaluation covers 125 held-out tasks organized into 15 task families, plus seed tasks for memory initialization.
- Discussion and Conclusion: The approach remains effective when initialized with automatically generated memories, although improvements vary by domain.Finance and NYC Collisions improve over registry-only search, while Northwind does not improve over S1.
- Discussion and Conclusion: Memory interference can cause semantically similar but incorrect memories to degrade retrieval below the no-memory baseline.The paper identifies interference-aware memory selection, memory aging, and broader evaluation as future work.
B Full Retrieval Results
Table 3 reports retrieval results across multiple metrics and retrieval conditions for held-out evaluation tasks.
- Full Retrieval Results: Table 3 reports R@5, R@10, MRR, and F1@5 for all retrieval conditions.The F1@5 column matches Table 1 in the main paper.
- Full Retrieval Results: The table includes results for all evaluated retrieval conditions on held-out evaluation tasks.The passage identifies the table as a full retrieval-results report.
- Full Retrieval Results: F1@5 results in Table 3 correspond to the F1@5 results reported in Table 1.This links the appendix table's F1@5 column to the main-paper presentation.
C LLM-Generated Discovery Memories
Automatically generated discovery memories are evaluated for object-level agreement and then tested for downstream retrieval performance across three domains.
- LLM-Generated Discovery Memories: Generated memories are created by asking Claude Haiku to identify relevant data objects from each seed task and registry contents.This replaces manually annotated discovery memories for robustness evaluation.
- LLM-Generated Discovery Memories: Memory agreement F1 is 0.698 for Finance, 0.762 for NYC Collisions, and 0.735 for Northwind.Agreement is measured against manually annotated discovery memories using object-level F1.
- LLM-Generated Discovery Memories: Finance improves from S1 F1@5 = 0.396 to S3 F1@5 = 0.433, while NYC Collisions improves from 0.299 to 0.452.These results use automatically generated memories.
- LLM-Generated Discovery Memories: Northwind changes from S1 F1@5 = 0.446 to S3 F1@5 = 0.435, showing no improvement over registry-only retrieval.The passage attributes this to noisier object selections amid higher cross-entity vocabulary overlap.
- LLM-Generated Discovery Memories: Across all three domains, retrieval quality remains above the raw-metadata baseline S0 despite imperfect memory construction.This supports robustness to automatically generated memories relative to raw metadata.
D Per-Family Analysis
Per-family gains vary substantially: distinctive object vocabularies benefit most, whereas overlapping object sets can produce little or negative improvement.
- Per-Family Analysis: Figure 1 compares F1@5 improvement from S1 to S3 across task families.The comparison is organized by task family.
- Per-Family Analysis: Temporal analysis gains +0.34, while supply chain and transaction behavior each gain +0.26.These are the three largest reported family-level gains.
- Per-Family Analysis: Loan risk gains +0.19 and account activity gains +0.17, indicating more moderate improvement.These families improve less than the top three families.
- Per-Family Analysis: Customer demographics, regional economics, and product catalog decline by −0.077, −0.102, and −0.074, respectively.Their objects appear across multiple task types, and seed memories can overlap with unrelated queries.
- Per-Family Analysis: Discovery memory is most effective when task-intent and object relationships are specific and stable, and least effective when object sets are shared.Shared object sets can make memory mildly harmful.
E Memory Weight Ablation
Memory-augmented retrieval performs best when memory contributes moderately rather than dominating registry evidence. Blending registry and memory signals also outperforms naive direct injection across all three domains.
- Memory Weight: Both retrievers peak at memory weight α = 0.25 across all three domains.Setting α = 0 recovers registry-only retrieval, while α = 1 gives memory full control.
- Memory Weight: Over-weighting memory at α = 1.0 degrades retrieval performance in all settings.The main conclusions remain unchanged when using the ablation-optimal α = 0.25 instead of the default α = 0.5.
- Blending vs. Direct Injection: S3 blended registry-plus-memory retrieval outperforms S4 naive injection in both MRR and F1@5 across all three domains.S4 promotes objects from the highest-scoring memory without combining memory and registry scores.
- Blending vs. Direct Injection: Blending preserves retrieval precision because direct injection can elevate irrelevant objects when a retrieved memory only partially matches the query.S3 combines registry and memory signals while still promoting objects associated with relevant prior discovery outcomes.
G Leave-One-Out Seed Robustness
Leave-one-out seed rotation tests whether retrieval gains depend on a particular seed task, while additional retriever comparisons examine robustness across retrieval methods. The reported method ordering remains stable under seed rotation, and memory augmentation continues to help with BM25.
- Leave-One-Out Seed Robustness: The relative ordering of methods remains consistent across leave-one-out seed rotation and fixed-seed results.The evaluation includes 56 Finance, 51 NYC Collisions, and 36 Northwind trials.
- BM25 Retriever Comparison: S3 still improves over S1 in all three domains when using BM25 instead of TF-IDF.This supports memory boosting across the evaluated underlying retrievers.
- BM25 Retriever Comparison: BM25 underperforms TF-IDF at the S1 and S3 levels across all three domains, while S0 differences are negligible.The registry-level gap is attributed to BM25 term-frequency saturation being poorly matched to long, semantically rich descriptions.
- Downstream Planning: In a downstream planning example, S3 retrieves exact transaction-related objects whereas S1 retrieves mostly demographic objects.The S3 plan references correct operational fields and produces a more executable analysis strategy for unusual cash withdrawals.