PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoDataBench: A Data-centric Testbed for Accelerating Auto Research

Ruifeng Yuan, Yizhi Li, Yaxin Du, Fengyu Cai, Yiqi Liu, Hou Pong Chan, Chenghua Lin, Yun Chen, Jian Yang, Bryan Dai, Pinyan Lu, Chenghao Xiao

arXiv:2609.40097v1cs.CL

TL;DR

AutoDataBench addresses whether LLMs can understand and improve training data, a capability not established by other benchmark domains and costly to evaluate through training experiments. It evaluates data interventions across three task types under controlled pipelines, finding varied optimization and generalization outcomes and improved downstream coding performance when trajectories are reused for mid-training.

  • Problem

    Existing benchmarks do not directly establish whether LLMs can understand and improve training data, even though data quality and composition influence model performance and evaluating alternatives can require costly training experiments.

    Source: §1 · p. 1

  • Method

    AutoDataBench evaluates frontier LLMs’ iterative data interventions across tool use, retrieval, and knowledge injection under prescribed pipelines, and probes pre-training predictions and trajectory reuse.

    Source: §1 · p. 2

  • Results

    Agent-curated data approaches expert references on tool use and retrieval and improves knowledge acquisition, but strong in-distribution performance does not consistently transfer to hidden evaluations; trajectory reuse improves downstream coding performance.

    Source: §5 · p. 10

  • Takeaways & Limitations

    AutoDataBench trajectories show potential as mid-training material, with larger reported coding gains on CRUXEval input prediction and SWE-bench Multilingual than on MBPP and LiveCodeBench.

    Source: §1 · p. 2

Abstract

from arXiv · show

Existing auto-research benchmarks often entangle multiple sources of improvement, including training frameworks, hyperparameters, compute budgets, and data, making it difficult to attribute why one frontier agent outperforms another to specific research capabilities. In this work, we isolate and systematically evaluate Data Intelligence: an agent's ability to understand, manipulate, and improve the data that shapes model capabilities. We introduce AutoDataBench, a controlled testbed built on a conceptual framework of data intelligence spanning data diagnosis, data organization, and data construction, instantiated through three highly curated optimization tasks while holding non-data factors fixed. Across tool use, retrieval, and knowledge injection, we evaluate frontier LLMs' ability to improve training data through iterative experimentation under task-specific resource budgets. Beyond optimization performance, we ask: do LLMs understand what their data interventions do? We compare predictions made before training with observed outcomes to seek evidence of data-effect reasoning beyond trial and error, and explore whether iterative feedback helps LLMs better understand how changes to training data affect model performance. Finally, we show that reusing AutoDataBench trajectories for mid-training improves downstream coding performance, highlighting its value in both evaluating data intelligence and generating high-quality training data. Code and resources are available at https://github.com/AutoDataBench/AutoDataBench.

1 INTRODUCTION

AutoDataBench asks whether frontier LLMs can understand and improve training data, a capability not established by performance on other benchmark domains. It evaluates data interventions under controlled training procedures and also examines predictions about intervention effects and reuse of optimization trajectories.

1 INTRODUCTION

  • Existing capability benchmarks do not establish whether LLMs can improve training data, whose quality and composition affect model performance and whose alternatives can be costly to evaluate.Effective optimization also requires diagnosing examples, choosing what to select, revise, or construct, and using experimental feedback.Source: §1 · pp. 1–2
  • AutoDataBench evaluates data intelligence in a shared agent framework across data diagnosis and repair, organization, and construction, instantiated through tool use, retrieval, and knowledge injection.The benchmark uses prescribed training pipelines and task-specific resource budgets.Source: §1 · p. 2
  • Agents iteratively inspect data, implement interventions, submit datasets for training, and use evaluation feedback; assessments cover target-task performance and generalization beyond feedback distributions.Additional evaluation data remain withheld during optimization and are used to assess checkpoints selected with target-task feedback.Source: §1 · p. 2
  • Predictions recorded before training are compared with observed outcomes to probe LLMs’ understanding of data effects and how it changes with experimental feedback.The evaluation retains full optimization trajectories for behavioral analysis.Source: §1 · p. 2
  • Reusing AutoDataBench optimization trajectories for mid-training improves downstream coding performance, with larger gains on CRUXEval input prediction and SWE-bench Multilingual than on MBPP and LiveCodeBench.Matched-token-budget mid-training runs with and without trajectories are followed by identical supervised fine-tuning.Source: §1 · p. 2

2 RELATED WORK

Prior benchmarks assess general capabilities, data curation, or extended research workflows, but do not directly evaluate how agents improve training data across distinct learning tasks. AutoDataBench connects data decisions to learning outcomes through iterative experimentation and held-out evaluation.

2 RELATED WORK

  • Capability benchmarks such as SWE-bench, KernelBench, and FrontierMath do not directly assess how effectively agents improve training data.DataComp, DCA-Bench, PostTrainBench, and Curation-Bench address data curation or bounded agent optimization from different perspectives.Source: §2 · p. 2
  • AutoDataBench compares frontier LLMs in one framework across tool-use data repair, retrieval-data organization, and knowledge-data construction, using fixed pipelines spanning supervised fine-tuning, contrastive learning, and distillation.Hidden evaluations assess generalization, while pre-training predictions probe expectations about data effects.Source: §2 · p. 2
  • Figure 1 organizes data intelligence into diagnosis and repair, organization, and construction, with iterative interventions evaluated for target-task improvement and held-out generalization.Agents receive feedback from a fixed training and evaluation pipeline.Source: Figure 1 · p. 3
  • AutoDataBench adopts the extended experimental loop of auto-research benchmarks to examine the specific capability of understanding and improving training data.Keeping the agent framework and task-specific learning pipelines fixed links data decisions to measurable learning outcomes and shows how decisions evolve with feedback.Source: §2 · p. 3

3 AUTODATABENCH: DESIDERATA AND SETUP

AutoDataBench fixes non-data components to isolate data decisions, while probing generalization and supporting extensible task configurations. Its three tasks test data repair, organization, and construction under fixed learning pipelines and bounded resources.

3.1 DESIDERATA

  • The benchmark’s extensible, task-agnostic framework allows tasks, training frameworks, and tools to be added in a plug-and-play manner.It is designed to cover diverse task types, training paradigms, and datasets.Source: §3.1 · p. 3
  • Fixing each task’s training framework, augmentation tools, and source datasets isolates agents’ data decisions and their effects on model outcomes.This is the benchmark’s data-centrality design.Source: §3.1 · p. 3
  • Hidden out-of-distribution evaluation tests whether strategies selected using target-task feedback generalize beyond the optimization target and helps identify benchmaxxing.The agent does not see OOD evaluation during iterative optimization.Source: §3.1 · p. 3

3.2 BENCHMARK FRAMEWORK

  • Agents inspect data, implement interventions, and submit datasets to a fixed pipeline; target-task feedback guides subsequent decisions, while hidden data assess the selected checkpoint’s generalization.Each task specifies its source data, base model, training procedure, and resource budget, and the full optimization trajectory is retained.Source: §3.2 · p. 4

3.3.1 TASK 1: TRAINING TOOL-USE MODELS (WITH POLLUTED DATA)

  • Tool-use data repair asks agents to discover undisclosed quality issues in a deliberately corrupted function-calling dataset and improve training data through scalable curation.Permitted interventions include filtering or repairing examples, removing duplicates, synthesizing supervision, and adjusting data mixtures under a limited data budget.Source: §3.3.1 · p. 4

3.3.2 TASK 2: TRAINING EMBEDDING MODELS

  • Embedding-data organization tests whether agents can mine informative hard negatives without mislabeling relevant passages and organize examples under a fixed contrastive-learning pipeline.The agent receives anchors and positives, is expected to mine negatives, and allocates a cumulative budget equivalent to ten trials of at most 600k pairs each.Source: §3.3.2 · pp. 4–5

3.3.3 TASK 3: KNOWLEDGE INJECTION

  • Knowledge injection asks agents to construct context–question–answer examples from facts beyond the target model’s cutoff for a fixed offline privileged-context distillation pipeline.The frozen teacher sees supporting context while the student learns without it; Novel accuracy guides optimization, while Retention is reserved for hidden evaluation.Source: §3.3.3 · pp. 5–6

3.4 IMPLEMENTATION DETAILS

  • Runs share external-API access, one NVIDIA A800 GPU, a 24-hour wall-clock limit, and at most 20 training-and-evaluation rounds; trials restart from the same checkpoint under fixed training settings.The cumulative submitted training-set size is capped at ten times the task-specific maximum per-round dataset size.Source: §3.4 · p. 6

4 EXPERIMENTAL RESULTS

The benchmark compares frontier LLMs’ data interventions across three tasks, then examines optimization paths, forecasts, and trajectory reuse. Results show strong but task-dependent performance, a gap between in-distribution and hidden-evaluation performance, and downstream coding gains from reused trajectories.

4.1 REPORTING PROTOCOL

  • The reporting protocol averages three standard runs per LLM–task combination, selects checkpoints by highest ID score, and reports hidden-evaluation scores at those checkpoints.Baselines include random sampling for tool use, a fixed random retrieval subset, and an unadapted Talkie model for knowledge injection.Source: §4.1 · p. 6

4.2 LLM ABILITY ON AUTODATABENCH

  • 60.68: Kimi-3 leads Overall; tool-use mean ID accuracies of 80.82–81.98 approach the oracle-clean 81.65 from random 70.45, while all LLMs remain below the BFCL oracle-clean score of 60.23.Overall is the unweighted mean of the three task-wise ID means; the tool-use ID range is across LLMs.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • Retrieval reverses the ID ranking under hidden evaluation: GPT-5.6-Sol leads mean ID at 40.46 but ranks lowest on OOD at 26.15, while Qwen-3.7-Max leads OOD at 30.18.All LLMs fall below the expert OOD score of 34.76.Source: §4.2 · p. 7
  • Kimi-3 reaches 60.60 Novel accuracy, 12.20 points above the expert reference of 48.40; Claude-4.7 leads Retention at 50.37, while GLM-5.2 scores below the unadapted model on Retention.GLM-5.2 scores 47.15 on Retention versus 47.84 for the unadapted model.Source: §4.2 · p. 7

4.3 BEHAVIORAL ANALYSIS

  • 61 of 63 standard runs improve over their initial datasets, with mean first-to-best gains of 8.11 points for tool use, 6.46 for retrieval, and 4.91 for knowledge injection.The gains combine experimentation and checkpoint selection; targeted examples include schema-aware repairs, retrieval sampling and negative mining, and QA-pair paraphrasing.Source: §4.3 · pp. 7–8

4.4 CAN LLMS PREDICT THE EFFECTS OF THEIR DATA INTERVENTIONS?

  • On retrieval, GPT-5.6-Sol and Kimi-3 forecast observed scores with mean absolute errors of 0.64 and 1.55 points, respectively, but prediction-enabled runs score below standard-run means in four of six comparisons.Forecasts move closer to observed scores after early evaluations, while explicit forecasting may impose an additional optimization burden.Source: §4.4 · p. 9

4.5 BENCHMARK AS DATA ENGINE

  • 72.00 → 74.12 on CRUXEval input prediction and 27.67 → 33.33 on SWE-bench Multilingual follow mid-training with auto-research trajectories, which account for 1.6% of tokens.The comparison uses matched 5B-token mid-training budgets followed by identical SFT; all five evaluation scores improve, with larger gains on those two tasks.Source: §4.5 · p. 9, Table 2 · p. 9

5 CONCLUSION

AutoDataBench evaluates data intelligence through controlled data repair, organization, and construction tasks. Its results show task-dependent optimization and transfer patterns, while reused optimization trajectories improve downstream coding performance.

5 CONCLUSION

  • Agent-curated data approaches expert references on tool use and retrieval and improves knowledge acquisition, although strong ID performance does not consistently transfer to hidden evaluations.The conclusion frames evaluation around optimization outcomes, generalization, and predictions of data effects.Source: §5 · p. 10
  • Prediction-enabled runs broadly track retrieval scores but achieve lower best scores than standard-run means in four of six comparisons.The conclusion presents this as a reason to evaluate both data-effect predictions and optimization outcomes.Source: §5 · p. 10
  • Reusing auto-research trajectories during mid-training improves downstream coding performance, with the largest gains on CRUXEval input prediction and SWE-bench Multilingual.The paper concludes that AutoDataBench can serve both as a data-intelligence testbed and as a source of training data.Source: §5 · p. 10

A.1 TOOL-USE DATA CORRUPTION

Tool-use pool corruptions were applied sequentially and deterministically, with examples sometimes receiving multiple operations; corruption tags were removed from the agent-facing pool.

A.1 TOOL-USE DATA CORRUPTION

  • Operators are attempted in Table 3 order with deterministic example- and seed-derived randomness, and skipped when prerequisites are not met.An example may receive multiple corruption operations.Source: §A.1 · p. 13, Table 3 · p. 14
  • Of 40,001 pool examples, 16,877 (42.19%) receive at least one corruption, including 4,176 receiving multiple operations.Source: §A.1 · p. 13
  • Table 3 reports attempt probabilities for callable examples and application counts for operations passing applicability checks; counts overlap when examples receive multiple corruptions.Source: §A.1 · p. 13, Table 3 · p. 14

A.2 ADDITIONAL EXPERIMENTAL RESULTS

The additional results section specifies how individual-run scores, checkpoint selection, and aggregate statistics are reported across the three tasks.

A.2 ADDITIONAL EXPERIMENTAL RESULTS

  • Across seven LLMs, each task has three scored standard runs (63 total); prediction-enabled runs are reported separately in Figure 3.Source: §A.2 · p. 14
  • First, Best, and Last denote ID scores at the first, best, and last successful evaluations; At marks the first evaluation attaining Best.Source: §A.2 · p. 14
  • OOD is measured at the ID-selected checkpoint: BFCL for tool use, five held-out retrieval datasets, and hidden Retention for knowledge injection.Source: §A.2 · p. 14
  • Table 1 summarizes three Best ID scores per LLM and task, averages corresponding selected-checkpoint OOD scores, and defines Overall as the unweighted mean of task-wise ID means.Source: §A.2 · p. 14
  • Individual-run Tables 4–6 report tool-use macro AST accuracy, retrieval mean nDCG@10, and knowledge-injection Novel ID and hidden Retention OOD scores, all multiplied by 100.Source: Table 4 · p. 15, Table 5 · p. 15, Table 6 · p. 16

A.3 REFERENCE RESULTS

The references provide task-specific baselines and expert comparisons, with different construction procedures and uncertainty reporting.

A.3 REFERENCE RESULTS

  • The retrieval baseline samples 600k query–positive pairs with seed 42, retains in-batch negatives without explicit hard negatives, and scores 34.38 ID and 31.13 OOD mean nDCG@10.Source: §A.3 · p. 16
  • The tool-use oracle-clean reference applies schema validation, deduplication, and category-proportional selection to the pre-corruption pool.Source: §A.3 · p. 16
  • Unadapted Talkie is the knowledge-injection reference, scoring 39.10 Novel and 47.84 Retention accuracy.Source: §A.3 · p. 16
  • The retrieval expert uses experience-based hard-negative mining and reorganization; the knowledge expert uses keyword- and rule-based mining without LLM-assisted construction, and neither reference has repeated-run uncertainty estimates.Source: §A.3 · p. 16

A.4 OPTIMIZATION TRAJECTORIES AND REFINEMENT STATISTICS

Most scored standard runs improved over their initial datasets, while selected-run curves and a retrieval case illustrate distinct optimization paths and revisions.

A.4 OPTIMIZATION TRAJECTORIES AND REFINEMENT STATISTICS

  • Figure 2 plots cumulative maxima for each LLM/task’s highest-ID run, and the selected curves do not imply equal compute use.An individual trial may decrease even when the displayed cumulative-maximum curve does not.Source: §A.4 · p. 16
  • 21/21 tool-use and 21/21 retrieval runs, and 19/21 knowledge-injection runs, improved; mean first-to-best gains were 8.11, 6.46, and 4.91 points, respectively.These gains compare each run’s best ID score with its first successful evaluation, not with the fixed Baseline row.Source: §A.4 · p. 16, Table 7 · p. 17
  • Qwen-3.8-Max’s selected retrieval run progresses from 31.22 to 34.24 after negative filtering, then 32.70 with relevance judgments, and 41.66 after later mixture, size, and negative-count revisions.These sequential comparisons do not isolate each operation’s causal contribution.Source: §A.4 · pp. 16–17

A.5 DATA ENGINE TRAINING DETAILS

The appendix specifies the controlled mid-training and supervised fine-tuning settings, separately from benchmark optimization runs. Both arms use matched training settings apart from the mid-training corpus.

Mid-training

  • Each mid-training arm consumes exactly 5.0B source tokens once; the auto-research arm includes 80M trajectory tokens, or 1.6% of the mixture.Training starts from Qwen2.5-Coder-14B, extends the context window from 32K to 128K, and runs for 150 optimization steps per arm.Source: §A.5 · p. 17
  • Mid-training uses 256 packed sequences per batch, Adam, bf16 precision, and a learning rate that decays from 5 × 10−5 to 10−5.The schedule warms up over 5 steps and uses cosine decay; other stated settings include weight decay 0.1 and gradient clipping at 1.0.Source: §A.5 · p. 17

Supervised fine-tuning

  • Fine-tuning uses the same 202,800 multi-turn tool conversations for both arms, with assistant responses and end-of-turn tokens as the only supervised targets.The 4 conversations exceeding 128K are split into overlapping windows, with each assistant target supervised once per epoch.Source: §A.5 · p. 17
  • Fine-tuning runs for 3 epochs, and all settings are identical across arms apart from the mid-training corpus, including random seed 42.The optimizer state is re-initialized at fine-tuning start; training uses a global batch size of 512 packed sequences and 60 total steps.Source: §A.5 · p. 17
Loading 2609.40097v1…