PaperlayerView exact revision on arXiv

Source-linked AI summary

Autodata: An agentic data scientist to create high quality synthetic data

Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, Jason Weston

arXiv:2606.25996v3cs.AIcs.CLcs.LG

TL;DR

As models strengthen, existing tasks and synthetic-data methods may not yield sufficiently challenging data. Autodata uses an iterative agentic data-scientist process, and experiments with Agentic Self-Instruct report gains across three task areas, with a larger uplift after meta-optimization.

  • Problem

    As models strengthen, existing tasks and synthetic-data methods may not produce sufficiently challenging tasks and data for further progress.

    Source: §1 · p. 2

  • Method

    Autodata has an agent create, evaluate, and iteratively refine training or evaluation data, and its data-scientist agent can be meta-optimized.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    Agentic Self-Instruct improved results over classical synthetic-data methods across computer science research, legal reasoning, and mathematical-object reasoning, with an even larger uplift from meta-optimization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support using inference-time compute to generate higher-quality, more challenging synthetic data for model training.

    Source: §1 · p. 2, §3.3.1 · p. 10

    In the computer science task, some generated questions and rubrics were overly tied to specific experimental numbers rather than testing generalizable reasoning.

    Source: §6 · p. 15

Abstract

from arXiv · show

We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data. We show how to train (meta-optimize) such a data scientist agent, so that it learns to create even stronger data. We describe the overall formulation, and a specific practical implementation, Agentic Self-Instruct. We conduct experiments on computer science research tasks, legal reasoning tasks and reasoning with mathematical objects, where we obtain improved results compared to classical synthetic dataset creation methods. Further, meta-optimizing the data scientist agent itself delivers an even larger performance uplift. Agentic data creation provides a way to convert increased inference compute into higher quality model training. Overall, we believe this direction has the potential to change the way we build AI data.

1 Introduction

Autodata treats data creation as an agentic data-science process that iteratively creates, evaluates, and improves training or benchmark data, and can be meta-optimized. Its Agentic Self-Instruct implementation improves results across computer science research, legal reasoning, and mathematical-object reasoning tasks.

1 Introduction

  • Synthetic data can generate underrepresented edge cases and long-tail scenarios, reduce manual labeling effort and latency, and potentially be more challenging than human-generated data.The paper motivates synthetic data as a response to the growing importance of high-quality training data and challenging benchmarks.Source: §1 · pp. 1–2
  • Self-Instruct [Wang et al., 2023], grounded methods [Lupidi et al., 2024; Yuan et al., 2025], CoT Self-Instruct [Yu et al., 2025], and self-challenging [Zhou et al., 2025] extend synthetic task generation, but do not directly control difficulty or quality.Filtering [Yu et al., 2025], evolution [Xu et al., 2024], and refinement [Shah et al., 2024] address this gap.Source: §1 · p. 2
  • Autodata assigns an agent the data scientist’s role, iterating through data creation, curation, analysis, performance measurement, and improved recipes for training or benchmark data.The agent first creates data, then “eyeballs” it, measures its performance, constructs learnings, and iterates with an improved recipe.Source: §1 · p. 2
  • Meta-optimized Agentic Self-Instruct improves results over classical synthetic dataset creation methods across computer science research, legal reasoning, and mathematical-object reasoning tasks.Meta-optimizing the data scientist agent itself delivers an even larger performance uplift.Source: §1 · p. 2
  • Agentic data creation offers a way to convert increased inference compute into higher-quality model training and create challenging tasks as models become stronger.The paper argues this direction could change how new tasks and benchmarks are created to advance the frontier.Source: §1 · p. 2

2 Autodata

Autodata treats data creation as an iterative data-scientist loop: an agent creates data from task-relevant grounding, analyzes its quality, and feeds learnings into further creation. Agentic Self-Instruct implements this approach with challenger, weak and strong solver, and judge agents; the data-scientist agent itself can also be optimized.

2 Autodata

  • The agent iterates between creating grounded training or evaluation data and analyzing example- or dataset-level quality, feeding its learnings into revisions until it produces a final dataset or benchmark.The outer loop can include guardrails to prevent hacking.Source: §2 · p. 2

2.1 A specific implementation: Agentic Self-Instruct

  • Agentic Self-Instruct uses a main orchestrator with a challenger, weak solver, strong solver, and verifier/judge to create and assess candidate examples.The challenger generates examples from the orchestrator’s prompt; the solvers attempt them, and the judge checks example quality and returns its learnings to the main agent.Source: Figure 2 · p. 3, §2.1 · p. 3

2 Autodata

  • The data-scientist agent itself can be optimized using the inner-loop criterion of creating better data to guide outer-loop agent optimization.The paper describes autoresearch or meta-harness-style optimization.Source: §2 · p. 3

2.1 A specific implementation: Agentic Self-Instruct

  • For verifiable tasks, the method can require strong-solver majority correctness and weak-solver majority error; for non-verifiable tasks, it uses a judge-rated quality gap.The agent analyzes verifier reports and revises the challenger prompt until the criteria are met; for non-verifiable tasks, the task should be neither too easy nor too hard for the weak solver.Source: §2.1 · pp. 3–4
  • The weak and strong solvers may be the same LLM in different modes, with the strong solver using more inference-time compute, scaffolding, aggregation, or privileged information.This process aims to create challenging, high-quality examples specifically for training the weak solver.Source: §2.1 · p. 4

3 Experiments

Across computer science, legal reasoning, and scientific reasoning, Agentic Self-Instruct improves downstream training results over standard CoT Self-Instruct. Its loop adapts questions and reward signals to task-specific failure modes, while scientific-reasoning gains transfer across difficulty levels and benchmarks.

3.1 Computer Science Research tasks

  • For CS research questions, the agent generates a context, question, reference answer, and weighted rubric, then iterates using solver and verifier feedback until acceptance or budget exhaustion.Acceptance required strong-solver average ≥0.65, weak-solver average <0.5, and a strong–weak gap ≥20 percentage points; 1.3k filtered examples per data source were used for the RL comparison.Source: §3.1 · pp. 4–5

3.1.2 RL Training Results

  • 0.774 versus 0.727 mean@3 on the CoT test set and 0.632 versus 0.500 on the harder Agentic test set favored Agentic-trained Qwen3.5-4B over CoT-trained.Agentic-trained models matched or exceeded CoT-trained models on both test sets at every checkpoint, including the CoT in-distribution test.Source: Table 2 · p. 5, Figure 3 · p. 5, §3.1.2 · p. 6

3.1.1 Agentic Self-Instruct Loop Analysis

  • On a 10k-paper corpus, Agentic Self-Instruct averaged 6.59 rounds per accepted item; across 880 pre-acceptance rounds, 80% of failed rounds were rejected because questions were too easy and the weak solver scored too high.The loop produced 2.8k accepted examples.Source: §3.1 · p. 4, §3.1.1 · p. 5

3.2.2 RL Training Results

  • 0.441 (GPT-5) and 0.393 (Kimi) for Agentic-trained Qwen3.5-4B exceeded CoT-trained scores of 0.377 and 0.343 and the 397B baseline’s 0.404 and 0.358 on PRBench-Legal.Agentic also led on PRBench-Legal-Hard; the comparison used the same 2.8k-example budget, and training curves showed it ahead at every evaluated checkpoint.Source: Table 4 · p. 8, §3.2.2 · pp. 8–9

3.2.1 Agentic Self-Instruct Loop Analysis

  • In legal tasks, the loop raised weak-rollout mean from 15.9% to 28.3% and per-prompt standard deviation from 7.93 to 12.63, making reward variation more usable for GRPO.Strong mean remained roughly unchanged (71.7% to 69.8%); accepted questions had median four rounds, and 52% of Agentic-pool questions received high GRPO-suitability verdicts versus 4.8% for CoT.Source: §3.2.1 · p. 8

“More Challenging” vs “Just Right”

  • In CS, the loop widened the weak–strong solver gap, while in legal reasoning it narrowed the gap; both changes improved downstream RL, showing that useful data must be learnable, not merely harder.CoT questions were too easy for the weak solver in CS but too hard in legal reasoning, where many rollouts scored zero.Source: “More Challenging” vs “Just Right” · p. 9

3.3.1 RL training results

  • +3.20% avg@8 on combined validation and +1.04% avg@8 on the out-of-distribution Principia benchmark were the largest overall gains with Agentic Self-Instruct.Agentic data also improved the CoT validation subset by +3.05%, versus +1.86% for CoT data; on Principia, gains included +1.75% on RealMath and +0.82% on SuperGPQA.Source: §3.3.1 · pp. 9–10, Table 5 · p. 10, Table 6 · p. 10
  • Agentic scientific-reasoning data improved average performance, but Combined data had pass@8 advantages on ARB and RealMath, indicating a trade-off between average reliability and occasional broader coverage.Training also reduced reasoning truncation from 23.75% to 4.09% for Agentic Self-Instruct with a 65,536-token budget.Source: §3.3.1 · p. 10

4 Meta Optimization of the Data Scientist

The data scientist agent can itself be meta-optimized: an outer loop uses solver evaluations to improve its prompt strategy, and the evolved prompt raises validation pass rate.

Method

  • Meta-optimization treats the data scientist scaffold as code, evolving candidate prompt diffs using solver feedback and held-out validation.The optimizer samples parents with Boltzmann selection, diagnoses solver trajectories, proposes prompt edits, and accepts a mutant only when its validation score strictly exceeds its parent’s; re-evaluations help average noisy scores.Source: §4 · pp. 11–12

Setup

  • The experiment meta-optimizes CS research QA using Kimi-K2.6, with 50 training papers, 25 validation papers, and rubric-based weak/strong solver criteria.A QA pair succeeds when the weak solver scores ≤65%, its best attempt scores ≤75%, the strong solver scores 60–95%, and the strong–weak gap is at least 20 percentage points.Source: §4 · p. 12

Results

  • 79.6% validated pass rate, up from ∼62.1% at baseline, shows gains across 233 iterations of prompt evolution.The reported baseline and final evolved harness results use a 6-hour per-session timeout.Source: Table 7 · p. 12, §4 · p. 12
  • Trajectory analysis exposed generic answers and rubric-format errors as dominant failures, prompting paper-specific questions, context-leak checks, positive-only capped rubrics, and strict JSON formatting.The changes included a test that questions require paper-specific knowledge, rules barring the proposed solution from context, positive integer rubric weights capped at 7, and JSON-formatted criteria; negative weights had misfired and damaged strong-model scores without improving discrimination.Source: §4 · p. 12

5 Related Work

Autodata frames synthetic data creation as an iterative data-science process that evaluates usefulness, analyzes failures, revises generation recipes, and can meta-optimize the agent. Its key distinction is optimizing for learning value to the target model, not merely generating or filtering examples.

Synthetic instruction and alignment data

  • Synthetic instruction methods established LLMs as data generators, but typically use fixed prompting or filtering rather than Autodata’s iterative evaluation and recipe revision.The methods span seed-based instruction bootstrapping, teacher distillation, synthetic conversations, preference data, instruction evolution, and large-scale alignment data from minimal prompting.Source: §5 · p. 13

Grounded, verifiable and reasoning-based synthetic data

  • Grounded and reasoning-aware methods use domain-specific tasks, source documents, and reasoning traces; Autodata adds solver behavior and evaluator feedback to adapt data to the target model.Synthetic mathematical reasoning data, document- or table-grounded examples, and chain-of-thought planning and filtering are examples of this line of work.Source: §5 · p. 13

Agentic data generation and automated data-science systems

  • Autodata differs from other data-oriented agents by making another model’s training or evaluation data—not web collection or data-science competition solving—the agent’s objective.AgentInstruct generates diverse synthetic post-training data; Khan et al. (2025) targets student errors; AutoData (Ma et al., 2026) collects web data; DS-Agent and Data Interpreter automate data-science workflows.Source: §5 · p. 13

Self-improvement, self-play, and challenger–solver data

  • Autodata’s challenger–solver setup resembles self-play methods, but its data-scientist loop optimizes examples for learning usefulness rather than difficulty alone.The loop analyzes solver failures, judges example quality, and adjusts difficulty.Source: §5 · pp. 13–14

LLM judges, filtering, and data selection

  • Judge feedback steers generation rather than filtering a static pool: it raises difficulty and discrimination in CS tasks but reduces degeneracy and all-zero weak rollouts for legal GRPO.The goal is data that provides an effective learning signal for the target model, not simply high quality or high difficulty.Source: §5 · p. 14

Autoresearch and optimization of agent scaffolds

  • Autodata applies iterative prompt and scaffold optimization to data creation, using the data-quality criteria of its inner generation loop to improve the data scientist agent.Related work ranges from prompt optimization to broader automated research systems and end-to-end harness optimization.Source: §5 · p. 14

Positioning

  • Autodata unifies prior synthetic-data ideas in a loop of generation, evaluation, failure analysis, recipe revision, and meta-optimization, targeting learning value for another model.The framework aims to convert stronger inference-time models and larger agentic compute budgets into higher-quality training and evaluation data.Source: §5 · p. 14

6 Conclusion and Discussion

Autodata uses an autonomous agent to generate, evaluate, and improve synthetic data, with Agentic Self-Instruct and meta-optimization producing quality gains across the studied tasks. The authors outline future work on broader task coverage, safeguards, dataset-level analysis, and human-agent co-improvement.

Conclusion

  • Autodata makes an agent generate, evaluate, and refine synthetic data; Agentic Self-Instruct improved results across three task families, and meta-optimizing the agent added substantial gains.Agentic Self-Instruct optimizes for examples that separate weak and strong solvers.Source: §6 · p. 14

More tasks, models and baselines

  • Future work should test more tasks, models, and baselines, aiming for a general Autodata agent spanning diverse data types and verifiable or non-verifiable tasks.The envisioned scope includes mathematics, code, instruction following, safety, and single-turn, multi-turn, and agentic tasks.Source: §6 · p. 14

Hacking & limitations

  • Agents sometimes tried to evade the task, so the authors constrained the pipeline and plan stronger safeguards while ensuring generated questions test generalizable reasoning.Some computer-science questions and rubrics were overly tied to specific experimental numbers rather than generalizable reasoning.Source: §6 · p. 15

Full dataset analysis iteration

  • The experiments improve data quality at the example level; future work would analyze whole datasets, including diversity and interactions with existing datasets.Iterative batched analysis could generate N examples, learn from the batch, and use those learnings to generate the next batch.Source: §6 · p. 15

From Self-Improvement to Co-improvement

  • Future co-improvement could train the challenger and solver together, while incorporating human feedback and co-research rather than removing humans from the loop.The authors explored autoresearch-like meta-training and describe human-agent co-research as co-improvement (Weston and Foerster, 2025).Source: §6 · p. 15

A Token Efficiency and Truncation in Principa Experiments

The section examines whether training improves token efficiency by analyzing response truncation and distinguishing accuracy gains from reduced truncation versus improved reasoning.

A Token Efficiency and Truncation in Principa Experiments

A.1 Truncation Rates

With a 65,536-token reasoning budget, training lowers truncation, and Agentic Self-Instruct achieves the lowest reported rates.

A.1 Truncation Rates

  • Agentic Self-Instruct records 4.09% truncation on combined validation and 1.85% on Principia, versus 23.75% and 17.06% for base Qwen3.5-4B and 10.00% and 6.62% for Grounding.The results suggest training teaches more concise, efficient reasoning within the token budget.Source: Table 8 · p. 19, §A.1 · p. 19

A.2 Attribution of Accuracy Improvements

Attribution analysis finds that Agentic Self-Instruct gains arise from both fixing truncation and improving reasoning, with more than half of its incorrect-to-correct flips attributed to truncation fixes. The findings also point to more concise reasoning as a way to solve more problems within a fixed token budget.

A.2 Attribution of Accuracy Improvements

  • The analysis classifies flipped generations as truncation-fixed, improved reasoning on non-truncated examples, or other cases.It covers 816 agentic validation QA items with 8 generations each, totaling 6,528 paired generations.Source: §A.2 · p. 19
  • 54.81% of Agentic Self-Instruct’s 945 incorrect-to-correct flips came from truncation fixes, and 41.06% from improved reasoning on non-truncated examples.Across training configurations, approximately 50% of accuracy improvements came from fixing truncation issues.Source: Table 9 · p. 19, §A.2 · p. 19
  • The findings suggest models may fail because they run out of tokens before completing reasoning, and Agentic Self-Instruct teaches more concise reasoning within a 65,536-token budget.The passage frames this as improving reasoning efficiency alongside reasoning quality.Source: §A.2 · p. 19
  • Synthetic data training can improve reasoning efficiency, helping models solve more problems within fixed computational budgets.The passage distinguishes this benefit from improvements in reasoning quality.Source: §A.2 · p. 20

B Question Type Analysis for Principia Grounded Agentic Self-Instruct Data · B.1 Annotation Procedure · B.2 Question Types

Autodata questions were characterized by annotating verified QA pairs with a two-phase LLM pipeline, then organizing the resulting question types into Reasoning, Knowledge, and Mixed categories.

B.1 Annotation Procedure

  • A random sample of 1,000 verified QA pairs from the full agentic data was selected for question-type annotation using Kimi-K2.6.Source: §B.1 · p. 20
  • Taxonomy discovery used 200 items stratified by challenge score, yielding 98 raw category proposals consolidated into 11 non-overlapping types.The items were presented in batches of 20, with definitions and examples requested.Source: §B.1 · p. 20
  • Each item was assigned exactly one of the 11 types; 687 items received valid annotations after parsing failures were filtered.Source: §B.1 · p. 20

B.2 Question Types

  • The 11 question types are organized into three categories: Reasoning, Knowledge, and Mixed.Source: §B.2 · p. 20
  • Reasoning types span symbolic derivation, combinatorial analysis, probabilistic or dynamical analysis, spectral or optimization analysis, asymptotic analysis, and proof or verification.Source: §B.2 · p. 20
  • Knowledge types cover formula or theorem application and factual recall, while Mixed types cover physical modeling, procedural computation, and data-driven inference.Source: §B.2 · p. 20

B.3 Distribution · C Subagent System Prompts

The annotated Principia sample is weighted toward reasoning questions, consistent with the pipeline’s aim of generating challenging data. The appendix documents the distinct CS, legal, and scientific-reasoning subagent prompts and workflows.

B.3 Distribution

  • About half of the 687 annotated Principia questions are reasoning-dominant, with about a quarter mixed and one-fifth knowledge-oriented.The authors say this distribution indicates the pipeline generates questions emphasizing multi-step reasoning over simple recall, to help separate weak and strong solvers.Source: §B.3 · pp. 20–21, Table 10 · p. 21

C Subagent System Prompts

  • The appendix reproduces the CS and legal subagent system prompts, preserving each setting’s shared role, workflow, and output-schema wrapper format.The CS pipeline has three subagents, while the legal pipeline has four.Source: C Subagent System Prompts · p. 21

C.1 CS subagent prompts

  • In the CS pipeline, the main agent runs a challenger–quality-verifier–rubric-evaluation loop, while the challenger drafts questions, answers, and weighted rubrics.A quality verifier checks for answer leakage, recall versus reasoning, and rubric well-formedness; the main agent decides when to accept a question.Source: §C.1 · p. 21, Figure 7 · p. 22, Figure 8 · p. 23

C.2 Legal subagent prompts

  • The legal pipeline’s main agent orchestrates an extractor, a question-and-rubric writer, and a loop-judge to iteratively develop questions from one legal document.The writer treats the document as a SOURCE OF LAW and targets specified legal-reasoning capabilities; the judge returns an accept/improve verdict and, when needed, a suggestion_for_writer.Source: §C.2 · p. 21, Figure 10 · p. 24, Figure 11 · p. 25

C.3 Scientific reasoning prompts

  • Table 12 reports Legal RL results on PRBench with normalized scores, which explicitly credit avoiding negative criteria across a worst-to-best denominator.The caption says the models, training setup, and test sets match Table 4; only the scoring formula changes, with details attributed to Akyürek et al. (2025).Source: Table 12 · p. 22
  • The scientific-reasoning prompt figures include a challenger and an excerpt from the main orchestrator’s prompt.The appendix also lists legal RL training results evaluated on PRBench using normalized scores, with the scoring formula as the only change from Table 4.Source: Table 12 · p. 22, Figure 14 · p. 27, Figure 15 · p. 28

Related papers

Loading 2606.25996v3…