PaperlayerView exact revision on arXiv

Source-linked AI summary

From Intent to Improvement: Agentic Dataset Search for Better Model Training

Qiaosheng Chen, Kaijia Huang, Fanqi Zhou, Gong Cheng

arXiv:2610.39608v1cs.AIcs.IR

TL;DR

Dataset search must account for training intent, structured constraints, sparse relational evidence, and whether retrieved data help model training. AgenticDS uses typed plans and deterministic retrieval with evidence verification; it leads the benchmark pool and yields the best downstream accuracy across the three studied domains.

  • Problem

    Dataset search must combine semantic training goals with constraints and heterogeneous evidence, while retrieval relevance alone does not establish downstream training utility.

    Source: §1 · p. 1

  • Method

    AgenticDS uses a Planner, Engineer, and Critic to translate queries into typed plans, execute deterministic retrieval and bounded graph expansion, and verify provenance-grounded evidence.

    Source: §1 · p. 1

  • Results

    AgenticDS outranks every system in the benchmark pool and achieves the best downstream accuracy in mathematics, code, and medicine.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Graph expansion is conditional: unverified candidates lower NDCG@10, while entity-neighbor queries needing one- or two-hop traversal gain +0.197 NDCG@10.

    Source: §6 · p. 8

    The benchmark shares a graph with the retrieval engine, and the utility study has narrow coverage.

    Source: §7 · p. 9

Abstract

from arXiv · show

Finding training data in million-scale machine learning repositories is a Web search problem where text relevance is not enough. A query couples a semantic training goal with constraints such as task, license, and training stage, and the metadata that would verify them are often missing. Text retrieval ranks dataset cards by similarity but cannot enforce hard constraints or follow the relations that reach a sparsely described dataset. Free-form tool-calling agents are more flexible, but the same model that picks a dataset also declares it a match, and nothing checks that the query's constraints are actually met. Neither approach establishes that the data it returns will improve model training. To address it, we propose AgenticDS, a contract-governed agentic framework in which every language model decision instantiates a typed object before deterministic code consumes it. The framework compiles a query into a typed search plan, executes it through admissible deterministic operators over hybrid retrieval and bounded graph expansion, and verifies the provenance-grounded evidence before accepting a ranking or emitting a typed diagnosis that permits only local repairs. On HuDS-Bench, a new benchmark of 200 queries with 24,908 graded judgments pooled over 15 systems, AgenticDS improves NDCG@10 by 25.7\% over the strongest dense--sparse hybrid and by 18.9\% over a commercial LLM agent. A controlled search-to-train study further shows that the datasets it retrieves give the best downstream accuracy in mathematics, code, and medicine. Code and data are publicly available at https://github.com/nju-websoft/AgenticDS.

1 Introduction

Dataset search must reconcile training intent, hard constraints, sparse relational evidence, and actual training utility. AgenticDS separates language-model reasoning from deterministic retrieval and evaluates both ranking and downstream training outcomes.

1 Introduction

  • Dataset acquisition combines semantic goals with typed constraints, heterogeneous evidence, and the need to verify whether retrieved data improve training.Text retrieval can miss hard constraints and sparse relational evidence, while an LLM that selects a dataset can also declare it a match without verification.Source: §1 · p. 1
  • AgenticDS compiles queries into typed plans and runs admissible deterministic operators over hybrid retrieval and bounded graph expansion.A Critic checks a fixed-format evidence package and can accept the result or issue a typed diagnosis for local plan repair.Source: §1 · pp. 1–2
  • AgenticDS outranks every pooled system and gives the best downstream accuracy in mathematics, code, and medicine, while graph expansion helps only when paired with verification.Removing the Critic or Planner costs 0.149 and 0.086 NDCG@10, respectively.Source: §1 · p. 2
  • HuDS-Bench and a search-to-train protocol evaluate 15 retrieval systems and test retrieved datasets through fine-tuning of three base models.The benchmark contains 200 queries and 24,908 graded judgments; the protocol covers mathematics, code, and medicine.Source: §1 · p. 2

2 Related Work

Related work spans Web dataset search, agentic information retrieval, and data-centric recursive self-improvement. AgenticDS combines these lines by acquiring external datasets under typed constraints and testing their utility through controlled fine-tuning.

2.1 Dataset Search on the Web

  • Dataset-search systems include metadata portals, dense representations, query-focused snippets, and natural-language recommendation, but DataFinder lacks typed constraints, relational evidence, and training-utility checks.AgenticDS uses HuggingKG as auxiliary infrastructure and makes natural-language data needs measurable through HuDS-Bench.Source: §2.1 · p. 2

2.2 Agentic Information Retrieval

  • Agentic information-retrieval systems plan and refine searches, but direct LLM control of tools entangles query interpretation with index operations; AgenticDS keeps retrieval operations deterministic.This trades open-ended tool flexibility for reproducibility, provenance-transparent ranking, and bounded token cost.Source: §2.2 · p. 2

2.3 Data-Centric Recursive Self-Improvement

  • Data-centric RSI systems optimize data within a known corpus, whereas AgenticDS acquires external datasets and verifies their utility through controlled fine-tuning.The paper positions this search-to-train step as an upstream complement to in-corpus optimization.Source: §2.3 · p. 2

3 AgenticDS: Framework and Mechanism

AgenticDS turns dataset acquisition into a typed, contract-governed search process over incomplete metadata and heterogeneous graph relations. Its Planner, Engineer, and Critic constrain retrieval, preserve evidence provenance, and use diagnoses for local plan repair.

3.1 Task and Design Principles

  • The Planner’s typed contract, separately inspectable provenance, and bounded autonomy constrain what language-model decisions can change during search.AgenticDS retains the round with maximal utility, balancing ranking quality against joint intent coverage, and stops on deterministic criteria.Source: §3.1 · pp. 2–3, §3.5 · p. 4

3.2 Planner: Compiling Intent into a Typed Contract

  • The Planner compiles the query into typed branches, graph paths, hard constraints, soft preferences, intent requirements, pool budgets, and acceptance thresholds.It retains the original query branch and adds branches for direct retrieval or graph expansion; later repairs preserve the original requirements.Source: §3.2 · p. 3

3.3 Engineer: Hybrid Retrieval and Typed Relation Expansion

  • Hybrid retrieval combines dense and BM25 ranks, while bounded graph expansion follows only declared paths from retrieved anchors to candidate datasets.Relation scoring aggregates permitted paths, discounts indirect paths, and penalizes widely connected entities, allowing sparsely described datasets to be reached through supported anchors.Source: §3.3 · pp. 3–4

3.4 Constraint Handling and Provenance-Preserving Pre-Ranking

  • The Engineer removes candidates with explicit hard-constraint violations, keeps missing metadata unknown, and adjusts within satisfactory and unknown groups for soft preferences.It retains branch ranks, fusion contributions, constraint status, anchors, and paths so users can inspect whether support comes from a dataset card or a relation path.Source: §3.4 · p. 4

3.5 Critic: Evidence Compression, Reranking, and Diagnosis

  • The Critic scores fixed-format evidence packages for joint query satisfaction, then combines semantic judgment with the deterministic pre-score to rank candidates.It accepts only when ranking quality, intent coverage, minimum result count, and unresolved critical diagnoses meet the stated criteria; otherwise, typed diagnoses guide local repair.Source: §3.5 · p. 4

4 Evaluation Methodology

The evaluation measures both dataset-search quality and whether retrieved datasets improve models under controlled training conditions. It introduces a KG-grounded benchmark and annotation process, compares retrieval systems, and fixes the training setup so retrieval method is the varying factor.

4 Evaluation Methodology

  • The evaluation asks whether AgenticDS retrieves datasets for complex natural-language training needs and whether those datasets improve models when used as training interventions.For the training question, training budgets and optimization settings are held constant so differences arise from dataset acquisition rather than the training recipe.Source: §4 · p. 5

4.1 HuDS-Bench

  • HuDS-Bench contains 200 English queries across semantic and relational search types, with most entity-neighbor queries requiring multi-hop graph evidence.Its KG-grounded construction uses conditions from gold dataset metadata and, for relational queries, HuggingKG anchors; the query writer does not see the gold identity or card text for those types.Source: §4.1 · p. 5

4.2 Pooling and Relevance Annotation

  • The annotation pool merges top-20 results from 15 systems into 24,908 query–candidate pairs and assigns graded labels from per-candidate evidence packages.Disputed labels receive an independent re-check and residual disagreements go to human adjudication; agreement with a 2,490-pair human-annotated sample is weighted Cohen’s κ=0.754.Source: §4.2 · p. 5

4.3 Baselines, Backbone, and Metrics

  • The comparison includes LLM agents, a dense–sparse hybrid, a retrieval-then-rerank baseline, and additional platform and retrieval systems, with NDCG and MAP measured at cutoffs 5, 10, and 20.AgenticDS uses GLM-5.2 as its backbone and Qwen3-Embedding-0.6B as its embedding model; significance tests use paired per-query scores with Holm correction.Source: §4.3 · p. 5

4.4 Utility-Grounded Search-to-Train Protocol

  • The search-to-train protocol varies retrieval method while holding the base model, data budget, trainer, and evaluation suite fixed.It evaluates three base models across mathematics, code, and medicine, using a shared benchmark grid and development-based checkpoint selection; tests are interpreted as a consistency check because there are no independent training seeds.Source: §4.4 · pp. 5–6

5 Results

AgenticDS leads the reported retrieval and training comparisons, while downstream ablation does not resolve individual component effects. Its cost analysis reports lower evaluation costs than the agent baselines.

5.1 Dataset Retrieval (RQ1)

  • AgenticDS beats the strongest hybrid by 25.7% and the strongest LLM agent by 18.9% on NDCG@10, with margins surviving Holm correction.The advantage is in graded ranking; MAP margins over the agent baselines do not survive correction.Source: §5.1 · p. 6

5.3 Training Utility (RQ2)

  • AgenticDS leads four other acquisition methods across the three domains, with a grand mean of 52.81 versus 49.75 for the second-best method.The advantage is clearest in math (+3.86 over the next best); code and medicine have narrower three-model margins of +0.41 and +0.49.Source: Table 3 · p. 7, §5.3 · p. 7

5.2 Component Ablation

  • Removing the Critic, Planner, or KG expansion reduces NDCG@10 by 0.149, 0.086, and 0.084, respectively.The Critic has the largest single ablation effect; removing it with the Planner is less damaging than removing it alone because the Engineer falls back to wider retrieval.Source: §5.2 · p. 7

5.5 Cost Analysis

  • AgenticDS costs $19.86 for 200 queries versus $48.78 and $112.40 for the agent baselines, saving 59% and 82% while matching Claude Code’s wall-clock time.The Critic accounts for 83.0% of tokens but 28% of wall-clock time; deterministic retrieval and fusion use no tokens.Source: Figure 3 · p. 8, §5.5 · p. 8

5.4 Downstream Component Ablation

  • The downstream ablation cannot attribute effects to individual components: its nine-block standard errors are 0.6–1.4 accuracy points, and even the largest effect falls below the separation threshold.Removing the Critic has the largest downstream effect at −1.85, just under the roughly two-point threshold.Source: Table 4 · p. 7, §5.4 · p. 8

6 Discussion

AgenticDS treats dataset search as acquisition for model training, separating interpretation, execution, and verification. Its graph evidence helps most when queries require traversing relations, but only when candidates are semantically verified.

From retrieval to acquisition

  • Separating interpretation, execution, and verification gives the same plan the same corpus and candidate set, unlike unconstrained tool-calling agents.The framework frames dataset acquisition as training-oriented search rather than description matching alone.Source: §6 · p. 8

The role of KG evidence

  • Entity-neighbor queries needing one- or two-hop traversal gain +0.197 NDCG@10, versus +0.075 for general queries answerable from card text.The reported gains show that graph evidence is especially valuable for relation-dependent queries.Source: §6 · p. 8
  • Adding graph candidates without semantic verification lowers NDCG@10 by 0.025–0.029 across three backbones.Graph expansion and verification are therefore not separable features in these experiments.Source: §6 · p. 8

7 Conclusion

AgenticDS uses typed exchanges among a Planner, Engineer, and Critic to keep semantic reasoning from silently changing retrieval behavior. The paper reports retrieval improvements and stronger downstream accuracy, while bounding its claims by benchmark construction and study coverage.

7 Conclusion

  • AgenticDS improves retrieval over commercial LLM agents and strong dense–sparse pipelines, while its retrieved datasets yield the best downstream accuracy in all three studied domains.The conclusion frames dataset acquisition as an upstream stage of a data-centric improvement loop judged by the model it trains.Source: §7 · p. 9
  • The claims are bounded because benchmark construction shares a graph with the retrieval engine and the utility study has narrow coverage.The authors identify a KG-free gold-label pipeline and a distilled Critic as future work.Source: §7 · p. 9

A Updated HuggingKG Snapshot

  • The March 12, 2026 HuggingKG snapshot contains 4,928,681 entities and 3,951,507 relations; the update changes coverage, not schema or method.The snapshot includes 910,119 datasets, with graph traversal limited to one- or two-hop paths under per-branch budgets.Source: A Updated HuggingKG Snapshot · p. 10

B Contracts, Parameters, and Prompts

  • Contracts restrict Engineer execution to admissible operators and declared paths, discard explicit hard-constraint violations, and allow only bounded local plan repairs.Missing constraint fields remain unknown; acceptance also checks quality, candidate count, and critical diagnoses.Source: B Contracts, Parameters, and Prompts · p. 10

C HuDS-Bench Construction Details

  • HuDS-Bench labels candidates from evidence packages containing card excerpts, normalized metadata, sample fields, and retrieval and graph provenance.Label 2 requires a strong intent match, satisfied hard constraints, a matched dataset role, and direct evidence; uncertain or conflicting cases receive additional review.Source: C HuDS-Bench Construction Details · p. 10
  • On 2,490 sampled query–dataset pairs, automated and human labels achieved 84.1% exact agreement and linearly weighted Cohen’s κ = 0.754.The sample included 861 items flagged for review, which were not excluded.Source: C HuDS-Bench Construction Details · p. 10

D Extended Retrieval Results

Extended retrieval analyses show that relational query types benefit more than semantic types, while backbone choice bounds the architecture’s ranking performance.

Per-query-type retrieval gains

  • The four query-type gains weight-average to a +0.126 overall margin over the Qwen3-Emb+BM25 hybrid, with relational types gaining twice as much as semantic types.The relational types have near-binary labels and only 40 and 30 queries, making their individual margins more volatile.Source: D Extended Retrieval Results · p. 11

Agent-backbone sweep

  • With Qwen3.5-35B-A3B, AgenticDS reaches 0.5244 NDCG@10; with GLM-4-9B, it falls to 0.4706, level with BM25 at 0.4717 and below the hybrid at 0.4893.The retrieval stack and contracts are held fixed across this backbone sweep.Source: D Extended Retrieval Results · p. 11
  • The contracts constrain what the Planner and Critic may emit, but do not substitute for their reasoning capability.The paper characterizes backbone capability as a prerequisite for ranking quality under the contract-governed loop.Source: D Extended Retrieval Results · p. 11

E Significance Testing

The paper reports significant retrieval and training comparisons, alongside sensitivity to the training comparison unit and specific benchmark and intervention settings.

E Significance Testing

  • All fourteen retrieval comparisons are significant on NDCG@5, NDCG@10, and NDCG@20; the four training comparisons against AgenticDS remain significant after Holm correction.The NDCG@10 comparison with the bge-m3 variant is the exception after joint metric correction (p = 0.063).Source: E Significance Testing · p. 11
  • Collapsing training comparisons to three domain-level units leaves only 23 sign assignments and a smallest two-sided p of 0.25.The authors also note single runs per system, graph-anchored queries that are not fully independent, silver labels, and no independent training seeds.Source: E Significance Testing · p. 11

F.1 Benchmark configuration

  • The evaluation uses the 272-problem OCWCourses set and a 1,784-item MMLU-Medical union of nine published subject test splits.Scores are normalized to the 0–100 range.Source: §F.1 · p. 11

F.2 Training grid and search queries

  • Training interventions use 2,000 examples per dataset, identical optimization settings, and a three-epoch learning-rate grid selected on a held-out development split.The three search queries target instruction-tuning data for advanced mathematics, competitive programming, and medical question answering.Source: §F.2 · p. 11

F.3 Per-model scores

  • In the training study, AgenticDS denotes the Qwen3-Emb variant, and Table 10 expands Table 3 means into nine blocks.The supplied passage does not state the per-model scores.Source: §F.3 · p. 11

F.4 Ablation grid

The ablation grid compares component removals across base models and domains; its block-level variability makes aggregate means more informative than isolated wins.

F.4 Ablation grid

  • Figure 4 presents five ablations across three base models and three domains, averaging benchmark scores within each domain.The grid uses each benchmark’s best learning rate.Source: §F.4 · pp. 11–12, Figure 4 · p. 12
  • Removal helps in 21 of 45 base-model/domain/component blocks, with changes varying widely across blocks.Block changes have standard deviations of 1.7–4.1 points, and nine-block mean standard errors of 0.6–1.4 points.Source: §F.4 · p. 12
  • Neither KG expansion nor the adaptive budget clears two standard errors, leaving their +0.10 and +0.43 means unresolved.The passage cautions that these means are not evidence against either module.Source: §F.4 · p. 12

G Implementation and Cost Comparison

The section details the evaluated systems and implementation settings, then uses two cases to show how evidence supports rejection, local repair, or an unresolvable constraint.

G Implementation and Cost Comparison

  • The nine additional baselines span platform search, sparse, dense, and hybrid retrieval, with hybrid branches fused by RRF.The inventory includes Google Dataset Search and HuggingFace full-text search, BM25 variants, dense encoders, and three additional hybrid systems.Source: G Implementation and Cost Comparison · p. 12
  • Dense retrieval uses separate FAISS indexes for descriptions and concatenated metadata, while sparse retrieval uses Lucene BM25.RRF smoothing, fusion weight, LLM weight, and acceptance thresholds were tuned on a pre-release internal pilot set, not the 200 released queries.Source: G Implementation and Cost Comparison · p. 12
  • Table 11 prices all systems using the same GLM-5.2 rates, while AgenticDS’s no-cache cost is an upper bound.The listed rates are $1.19 per million uncached input tokens, $0.30 per million cached input tokens, and $4.17 per million output tokens.Source: Table 11 · p. 12, G Implementation and Cost Comparison · p. 12
  • Both LLM-agent baselines use GLM-5.2 with shared HuggingFace tools and were evaluated in a single run per query.Each was asked for 30 ranked identifiers; outputs were normalized, deduplicated, and snapshot-filtered to retain up to 20 datasets, leaving short lists unfilled.Source: G Implementation and Cost Comparison · p. 12

H Case Studies

  • The commit-provenance case rejects two top-10 corpora whose sample fields show synthetic rather than real-commit prompts, guiding a local repair hypothesis.The Planner separates code generation, supervision format, and commit provenance into complementary hypotheses.Source: H Case Studies · p. 12
  • For the Hungarian-only medical query, none of the top five results is relevant, and broader Finno-Ugric or English candidates cannot satisfy the language constraint.The case contrasts this constraint-preserving diagnosis with commercial agents’ near-miss English or Spanish returns.Source: H Case Studies · pp. 12–13

I Limitations and Future Work

The authors bound the interpretation of their results by describing benchmark overlap, utility-study scope, and limits of the agreement analysis, then propose targeted extensions.

I Limitations and Future Work

  • The two relational query types use HuggingKG anchors traversable by the Engineer, but 130 of 200 queries lack KG-derived gold labels and non-KG baselines receive full-text search.Without KG expansion, AgenticDS still reaches 0.5307 NDCG@10, above every non-agent baseline; KG-free gold-label construction would remove the overlap entirely.Source: I Limitations and Future Work · p. 13
  • The utility study uses one query per domain across three base models, and its scope is English queries on one ML hub at offline-acquisition latency.It is powered to separate acquisition methods in aggregate rather than individual components.Source: I Limitations and Future Work · p. 13
  • A single κ across all four query types mainly reflects semantic types because relational types rarely use the middle grade.The authors say per-type coefficients would be more informative.Source: I Limitations and Future Work · p. 13
  • Future work calls for more queries per domain and repeated seeds, KG-independent gold labels, a smaller Critic, and testing beyond HuggingFace through other hubs or federation.The authors propose swapping the hub-tied Engineer for ecosystems with typed metadata and navigable relation graphs.Source: I Limitations and Future Work · p. 13

Related papers

Loading 2610.39608v1…