PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Generalist Agents Automate Data Curation?

Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, Ruoxi Jia

arXiv:2606.04261v1cs.AIcs.CLcs.CVcs.ETcs.LG

TL;DR

Training-data curation requires repeated policy design and revision, motivating the question of whether generalist coding agents can automate the process. The paper introduces CURATION-BENCH, a fixed-pipeline benchmark with interactive agent search, and finds that agents execute curation well but need method-adaptation scaffolds for reliable research. Scaffolded adaptation yields a stronger policy at one-tenth the data budget, while longer searches continue improving through 50 iterations.

  • Problem

    Practitioners must repeatedly propose, implement, evaluate, and revise data policies, but prior benchmarks do not evaluate how data-policy discovery is conducted.

  • Method

    CURATION-BENCH fixes the model, training recipe, and evaluation suite while recording interactive agent trajectories as agents inspect data, implement policies, submit datasets, and revise from feedback.

  • Results

    Method-adaptation scaffolding shifts agents toward method-guided exploration and produces a hybrid loss-based policy that outperforms compared published non-agent baselines using one-tenth as much data.

  • Takeaways & Limitations

    Generalist agents can execute the curation loop, but reliable data research requires scaffolded method adaptation rather than open-ended prompting alone.

  • Takeaways & Limitations

    Experiments focus mainly on vision-language instruction tuning, and scaffold comparisons do not isolate a single causal ingredient because heavy scaffolds combine multiple constraints.

Abstract

from arXiv · show

Curating training data is among the most consequential yet labor-intensive parts of modern AI development: practitioners iteratively propose, implement, evaluate, and revise data policies against noisy benchmark feedback. We ask whether generalist coding agents can automate this data-curation loop. We introduce *Curation-Bench*, an agent-centric benchmark that fixes the model, training recipe, and evaluation suite while giving agents command-line access to inspect data, implement policies, submit them to a fixed training/evaluation pipeline, and revise. In a vision-language instruction-tuning instantiation, out-of-the-box agents reach strong published data-selection baselines within ten iterations. However, trajectory analysis reveals a persistent *execution-research gap*: agents mainly tune local policy variants rather than explore new policy families, even when given strategy guides and paper references. Scaffolds requiring each iteration to cite, instantiate, and adapt a prior method shift agents toward method-guided exploration. The scaffolded agent autonomously composes -- without human design input -- a data-selection policy that outperforms strong published baselines at one-tenth their data budget. Overall, current agents can run the curation loop, but reliable data research requires scaffolded method adaptation, not open-ended prompting alone. Code and benchmark are open-sourced.

1 Introduction

The paper asks whether generalist coding agents can automate iterative data-policy discovery and introduces CURATION-BENCH to evaluate that process. Agents can execute curation effectively, but method-adaptation scaffolds are needed to move beyond local policy tuning toward stronger exploration.

  • Motivation: Training-data curation remains labor-intensive because practitioners must choose, adapt, and revise policies after noisy benchmark feedback.The paper frames this iterative search process as the part of curation where human effort accumulates.
  • Research gap: Prior benchmarks isolate data quality or evaluate iterative research, but do not assess iterative data-policy discovery.Data-centric benchmarks use fixed models, recipes, and evaluations, while research-agent benchmarks typically optimize training procedures under fixed data.
  • Benchmark: CURATION-BENCH fixes the model, training recipe, and evaluation suite while giving agents an interactive loop to inspect data, implement policies, submit datasets, observe feedback, and revise.It records policy scripts, manifests, training outputs, and evaluation logs to assess both final performance and search behavior.
  • Findings: Generalist agents reach strong published data-selection baselines within ten iterations under open-ended prompting in the main multimodal instruction-tuning setting.The benchmark primarily uses vision-language instruction tuning and also includes a smaller CLIP-style pretraining setting.
  • Findings: Method-adaptation scaffolding produces a hybrid policy combining EL2N-style top-loss selection with an assistant-loss noise filter, outperforming compared published non-agent baselines at one-tenth their data budget.The scaffold requires agents to cite, instantiate, adapt, and validate prior methods rather than merely receive strategy guidance.
  • Findings: Beyond ten iterations, average outcomes continue improving through 50 iterations without a clear plateau, making agent search iterations a meaningful form of curation compute.The paper connects this additional compute to searching over how to select, adapt, and validate already available data.

2 CURATION-BENCH

CURATION-BENCH evaluates executable data-policy search under a fixed training and evaluation pipeline. It scores both curated-data outcomes and the quality of the agent’s trajectory, including exploration, grounding, effectiveness, and shallow search behavior.

  • Task contract: Each CURATION-BENCH task gives an agent a candidate pool, fixed training–evaluation pipeline, constraints, and a budgeted executable data-policy objective.The task contract separates final curated-data quality from the quality of the search process.
  • Design principles: The harness fixes the model, optimizer, schedule, and evaluation suite, leaving curated data as the agent’s controlled variable.This data-isolation principle makes the curated policy the object of optimization.
  • Environment: Agents work through a terminal workspace where they inspect files, write scripts, run commands, debug failures, and read logs.They may edit curation scripts and materialize datasets but may not modify training, evaluation, or contamination checks.
  • Submission pipeline: Every submission passes schema, budget, and contamination checks before accepted datasets enter training.Failed submissions return to the agent before training; accepted submissions are evaluated and their feedback can guide the next policy.
  • Scoring and diagnostics: Sessions are evaluated by best downstream score and trajectory diagnostics labeled new policy family, grounded, effective, and shallow.Diagnostics are reported at both iteration and full-session levels to make search behavior auditable.

3 Default Agents: Useful Executors, Narrow Researchers

Generalist coding agents reliably execute the curation loop and improve training outcomes across several settings, but open-prompt search largely remains within narrow local policy variants.

  • Default Agent Setup: 100% of iterations were successfully executed, and agents produced fewer than 10 non-incident crashed iterations across more than 500 iterations in 50+ sessions.The crashes did not derail the corresponding sessions.
  • Default Agent Setup: Agents inspect data, implement policies, materialize manifests, train with a fixed recipe, evaluate, and log outcomes across iterative sessions.The benchmark interface leaves policy choice and revision to the agent while recording trajectory metrics.
  • Outcome: Across multiple fine-tuning tasks, agents consistently improved over random selection and reached expert-baseline performance within a few iterations.The evaluated settings include LLaVA-1.5-7B and SmolVLM-Base instruction tuning.
  • Outcome: 33.7 was Claude Code’s average score on the primary LLaVA task, versus 32.5 for the best random run and 33.3/33.2 for ICONS/ARDS.This recovered 59% of the full-data gain using approximately 1.5% of the 665k pool.
  • Outcome: On DataComp Small CLIP pretraining, the agent established a clear gain over top-30% CLIP L/14-score filtering, extending the pattern beyond instruction tuning.Figure 3 reports average scores across 38 tasks for this setting.
  • Trajectory: Open-prompt agents often justify changes with generic goals and usually make low-cost edits within an existing policy family rather than switching families.Examples include source-ratio, response-length, metadata, or random-seed changes; once a workable family is found, agents rarely pivot.

4 Scaffolds Convert Execution into Exploration

Scaffolds alter how agents search for data policies: light guidance broadens stated options, while heavy protocols force evidence- or method-grounded execution and can reach higher-upside policy families.

  • Scaffold Design: Light scaffolds expose more policy families or paper-derived guidance, but they do not require a specific decision protocol.Data-strategies surfaces families such as source balancing and diversity sampling; research-papers provides paper-derived skill cards.
  • Scaffold Design: Heavy scaffolds require structured observations or prior-method citation, adaptation to available fields and budget, and manifest validation before training.These protocols constrain the research loop’s structure while leaving policy content to the agent.
  • Heavy Scaffolds: 34.9 average score: adapt-papers produced the strongest 10k policy, exceeding open-prompt and the evaluated 100k ARDS baseline with one-tenth as many examples.The policy combines an EL2N-style high-loss selector with an assistant-loss noise filter.
  • Light Scaffolds: 27% to 43%: data-strategies increased new-policy-family moves, while research-papers raised grounding from 57% to 70% and reduced shallow moves from 47% to 37%.Neither light scaffold improved the open-prompt maximum of 34.0; data-strategies matched it, while research-papers remained below it.
  • Heavy Scaffolds: 67% new-policy-family moves: adapt-papers eliminated shallow operations and grounded all iterations, changing executed trajectories rather than only written rationales.Its immediate-improvement effective-iteration rate was only 20%, showing that larger breakthroughs need not maximize local gains.
  • Interpretation: The comparison does not establish that heavier scaffolds are always better because self-research improves grounding while lowering final score versus open-prompt.Heavy conditions also combine method grounding, structured logging, category rotation, and procedural constraints.

5 Broader Investigations

Longer agent sessions continue improving average curation performance under a fixed data budget, and the benchmark also supports richer actions such as tool-assisted rewriting.

  • Scaling the Curation Compute Budget: 10 to 50 iterations: average performance continues improving without a clear plateau when the candidate pool, budget, training recipe, and evaluation suite remain fixed.Under open-prompting, gains accumulate gradually; under Heavy II, extra iterations reduce variance and raise the average after the best score appears early.
  • Scaling the Curation Compute Budget: Additional compute can be spent searching over data policies when more raw data is unavailable, costly, or lower quality.The search may select different subsets, test policy families, adapt prior methods, or rewrite examples.
  • Richer Data Actions: CURATION-BENCH extends beyond subset selection to tool-assisted rewriting through the same validation, contamination, training, and evaluation loop.The agent selects examples, rewrites them with an external MLLM, verifies the results, and submits the transformed dataset.
  • Richer Data Actions: The rewriting results indicate that the choice of rewriting model substantially affects outcomes, with Qwen3.5-9B consistently outperforming smaller alternatives.Table 6 evaluates 10k selected and rewritten examples against a Template Rewriting baseline.

6 Conclusion

CURATION-BENCH evaluates generalist agents as iterative data-policy researchers by isolating curated data as the optimization target and recording their full search trajectories.

  • Conclusion: CURATION-BENCH fixes the model, training recipe, and evaluation suite while recording the full trajectory of agent decisions.This isolates the curated data policy as the object of optimization.
  • Conclusion: Open-prompt agents execute curation effectively and match strong human-designed baselines at small data budgets, but often rely on local heuristic edits instead of distinct policy families.Awareness scaffolds broaden plans without reliably changing execution, whereas method-adaptation scaffolds can open higher-upside regions.
  • Conclusion: Future evaluations should report final scores together with scaffold strength and trajectory diagnostics because open-ended instructions and paper-grounded protocols answer different scientific questions.The recommendation follows the paper’s distinction between execution quality and the quality of policy search.

Limitations

The experiments primarily cover vision-language instruction tuning, with smaller extensions, and the scaffold comparison does not isolate individual heavy-scaffold components.

  • Limitations: The conclusions may differ for pretraining mixtures, code, math, domain-specific instruction tuning, or settings with less reliable evaluation.The experiments center on vision-language instruction tuning, with one smaller CLIP-style pretraining instantiation and a rewriting extension.
  • Limitations: The scaffold comparison is not a full factorial study because heavy scaffolds combine method grounding, structured logging, and stronger procedural constraints.The paper therefore identifies a useful scaffold configuration without isolating a single causal component.

Broader Impacts

Agentic data curation may lower fine-tuning costs by finding smaller, higher-value subsets, but it also creates oversight risks. Without review, curated datasets may amplify bias, remove underrepresented examples, or overfit narrow evaluations.

  • Agentic data curation could reduce fine-tuning costs by identifying smaller, higher-value subsets.
  • Unreviewed agent-curated datasets may amplify biases, remove underrepresented examples, or optimize toward narrow evaluation targets.
  • Paper-grounded scaffolds could accelerate harmful applications by lowering the engineering barrier for data optimization.
  • The paper recommends human or programmatic review, scaffold and trajectory reporting, contamination checks, and held-out evaluations.

Appendices

Prior work benchmarks interactive agents, data-centric methods, and data-science workflows separately. The paper positions its contribution at their intersection: iterative data-policy discovery as the central research object.

  • Data-centric AI treats dataset quality, coverage, labeling, and selection as first-class levers for model improvement.
  • Data-science-agent benchmarks mainly assess dataset analysis, notebooks, predictive modeling, and tabular machine-learning tasks.
  • Agent benchmarks evaluate interactive coding, experimentation, and reporting capabilities across environments such as AgentBench, MLAgentBench, MLE-bench, and PostTrainBench.
  • The remaining gap is an agentic benchmark centered on iterative discovery of data-curation policies.

B.1 Source data and models.

The benchmark uses public visual-instruction mixtures and a CLIP pretraining pool, with fixed model choices and evaluation safeguards. Agents optimize only which training examples are retained under a budget.

  • The benchmark draws training pools from two public visual-instruction mixtures and fine-tunes one of four base vision-language models.
  • The LLaVA-665K pool contains 665,298 rows partitioned across six source groups, including COCO, VG, OCR-VQA, GQA, text-only, and TextVQA.
  • Vision-Flan contains more than 191 task types with approximately 1,000 samples per task.
  • The four model configurations include LLaVA-1.5-7B, Qwen2-VL-2B, Qwen2.5-VL-3B-Instruct, and SmolVLM-2.2B-Base.
  • The CLIP instantiation uses DataComp-Small’s 12.8M-pair filtering pool, of which 8.77M pairs were recoverable after link rot.
  • Agents may audit submissions for exact and near-duplicate contamination, while accepted datasets must satisfy schema and target-row budget constraints.

B.4 Training

CURATION-BENCH fixes training and evaluation while measuring both final curated-data quality and the agent’s search process. Experiments show that open-prompt agents execute effectively but often fail to operationalize broader research ideas without stronger scaffolding.

  • B.4 Training: CURATION-BENCH fixes the fine-tuning recipe so the curated subset is the only variable across iterations.
  • B.4 Training: The DataComp setting fixes 12.8M processed pairs, uses the published Small-track recipe, and exposes no training hyperparameter to the agent.
  • B.4 Training: Evaluation is fixed through a VLMEvalKit wrapper, eight target benchmarks, a shared judge endpoint, and a specified aggregation rule.
  • B.4 Training: Strategy families are partially disconnected, so moving between them often requires new tools, proxies, and assumptions.
  • C.2 Light Scaffolds: Light scaffolds broaden written plans but rarely change executed trajectories; agents typically choose the easiest implementable heuristic.
  • Heavy-scaffolding I: Self-research: Self-research requires observations, hypotheses, minimal changes, and logged findings to structure experimentation across iterations.
  • Heavy-scaffolding II: Adapt Research Papers: Adapt Research Papers requires each iteration to cite and adapt a specific method, rotate across skill categories, and avoid low-effort heuristics.

D.5 Additional Results and Ablation Studies

Additional ablations show that scaffold design, agent backbone, and task setting affect curation outcomes, while open-prompt agents generally outperform random selection across budgets and tasks. Heavy scaffolding improves grounded exploration, but the strongest adaptation scaffold also changes the autonomy being measured.

  • Scaffold ablations: Heavy Scaffold II significantly increases new-policy-family moves and eliminates shallow operations, producing visibly improved outcomes.Both heavy scaffolds increase grounded iterations, but only Adapt papers changes exploration breadth and shallow behavior.
  • Trajectory analysis: Trajectory diagnostics evaluate whether iterations introduce new policy families, use evidence, improve accuracy, or remain shallow or trivial.The benchmark records commits, retained or discarded policies, task scores, and qualitative trajectory annotations.
  • Backbone ablations: Different agent backbones moderately affect final outcomes without changing the general patterns relative to random baselines or human solutions.The backbone comparison averages results over eight benchmarks.
  • Budget ablations: Open-prompt agents improve over random baselines and reach or exceed evaluated human baselines when selecting 20k or 50k examples.The comparison uses LLaVA-665K instruction-tuning data for LLaVA-1.5-7B and averages scores across eight benchmarks.
  • Task generalization: Open-prompt agents achieve general improvements over random baselines across multimodal instruction-tuning tasks with different models and datasets.The task ablation varies target models and data pools while retaining the core selection setting.
  • Interpretation: Scaffold strength is an experimental-design variable: stronger scaffolds can produce different agent behavior and make scores less directly comparable.The paper recommends reporting scaffold details alongside model details and distinguishing execution from autonomous method discovery.

F.1 Broader Societal Impacts

The paper presents agentic data curation as potentially reducing training costs and improving auditability, while also creating risks from opaque selection, misuse, and unequal access. It recommends review, trajectory reporting, and attention to scaffold strength and released artifacts.

  • Potential positive impacts: Agent-curated 10k subsets recover up to 71% of the full-data fine-tuning gain on 665k examples.This suggests lower fine-tuning barriers for users with limited compute, within the reported setting.
  • Potential negative impacts: Unreviewed agent-curated subsets may silently amplify harmful biases, unsafe content, or underrepresented groups through opaque scoring functions.The paper recommends treating curated datasets as candidate artifacts subject to representation, safety, and license review.
  • Potential negative impacts: Paper-adaptation scaffolds reduce engineering friction for optimizing data pools whose downstream models support harmful tasks.The concern applies beyond beneficial uses and includes disinformation, targeted persuasion, and surveillance-related vision tasks.
  • Potential negative impacts: The strongest results require capable backbone agents and non-trivial compute for repeated training and evaluation, creating concentration risks.The paper identifies unequal access to the required agents and compute as a potential negative impact.
  • Release scope: The released artifacts are a benchmark harness, scaffolds, trace logs, and dataset manifests rather than new pretrained models or raw multimodal datasets.The manifests index existing public datasets used under their original licenses.
  • Potential positive impacts: Persisted scripts, manifests, audit results, outputs, scores, and notes support post-hoc inspection and correction of individual curation iterations.Each iteration is stored under a commit hash, improving traceability of agent behavior.
Loading 2606.04261v1…