Source-linked AI summary
Autodata: An agentic data scientist to create high quality synthetic data
Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, Jason Weston
TL;DR
Synthetic data creation needs stronger ways to produce challenging, high-quality training and evaluation data. Autodata trains an agentic data scientist to iteratively create and analyze data, and Agentic Self-Instruct improves results across computer science, legal reasoning, and mathematical reasoning tasks, with further gains from meta-optimization.
Problem
Existing synthetic data methods may not produce sufficiently challenging, high-quality training and evaluation data as models become stronger.
Method
Autodata uses an agentic data scientist that iteratively creates, analyzes, and improves training or evaluation data, with optional meta-optimization of the agent itself.
Results
Agentic Self-Instruct improves results across computer science, legal reasoning, and mathematical reasoning tasks, while meta-optimizing the data scientist agent delivers further gains.
Takeaways & Limitations
Agentic data creation can convert increased inference-time compute into higher-quality, more challenging synthetic data for model training and benchmarking.
Takeaways & Limitations
The reported experiments improve data at the example level but do not yet provide full dataset-level analysis for diversity and interactions with existing datasets.
Abstract
from arXiv · showhide
We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data. We show how to train (meta-optimize) such a data scientist agent, so that it learns to create even stronger data. We describe the overall formulation, and a specific practical implementation, Agentic Self-Instruct. We conduct experiments on computer science research tasks, legal reasoning tasks and reasoning with mathematical objects, where we obtain improved results compared to classical synthetic dataset creation methods. Further, meta-optimizing the data scientist agent itself delivers an even larger performance uplift. Agentic data creation provides a way to convert increased inference compute into higher quality model training. Overall, we believe this direction has the potential to change the way we build AI data.
1 Introduction
Autodata treats data creation as an agentic data-science process for constructing and curating training or benchmark data. Its Agentic Self-Instruct implementation improves results across computer science, legal reasoning, and mathematical-object reasoning tasks, with further gains from meta-optimizing the agent.
- Motivation: Synthetic data can generate underrepresented edge cases and long-tail scenarios, reduce manual-labeling difficulty and latency, and potentially create more challenging data than human-generated distributions.These benefits address the growing need for high-quality training data and challenging benchmarks.
- Motivation: Existing synthetic-data methods do not directly control data difficulty and quality, motivating filtering, evolution, and refinement approaches.Prior methods include Self-Instruct, document-grounded variants, chain-of-thought methods, and self-challenging agents.
- Contributions: Autodata assigns an agent the role of a data scientist who constructs and curates training or benchmark data through creation, analysis, and performance measurement.The process includes an initial data-creation iteration followed by “eyeballing” the data and measuring its performance.
- Contributions: Agentic Self-Instruct improves results over classical synthetic dataset creation methods across computer science research, legal reasoning, and reasoning with mathematical objects.Meta-optimizing the data scientist agent delivers an even larger performance uplift.
- Motivation: Agentic data creation converts increased inference compute into higher-quality model training and may enable more challenging tasks and benchmarks as language models become stronger.The paper argues that this direction could change how new tasks and benchmarks are created to advance the AI frontier.
2 Autodata
Autodata is an iterative data-scientist loop in which an agent creates and analyzes training or evaluation data, repeatedly improving it until producing a final dataset or benchmark. Its practical Agentic Self-Instruct implementation uses challenger, weak and strong solvers, and a verifier to generate examples that challenge the weak solver while remaining solvable by the strong solver.
- Autodata: Autodata repeatedly alternates data creation and analysis, using example- and dataset-level feedback to improve training or evaluation data before producing a final dataset or benchmark.The outer loop can include guardrails against hacking and lets the agent build on its own learnings.
- Meta-Optimization of the Data Scientist: Autodata can meta-optimize the data-scientist agent itself by using inner-loop data quality to guide optimization of the outer agent harness.The paper identifies autoresearch and meta-harness (Lee et al., 2026) style optimization as approaches for this outer-loop improvement.
- Agentic Self-Instruct: Agentic Self-Instruct orchestrates a challenger, weak solver, strong solver, and verifier/judge to create and assess candidate training examples.The challenger proposes examples, the solvers attempt them, and the verifier evaluates solution and example quality before passing feedback to the main agent.
- Agentic Self-Instruct: The main agent updates the challenger prompt from verifier reports and repeats generation until examples meet task-specific difficulty and quality criteria.For verifiable tasks, strong-solver majority must be correct while weak-solver majority is wrong; for non-verifiable tasks, the judge must measure a suitable quality gap.
- Agentic Self-Instruct: The weak and strong solvers may be the same LLM in different modes, with the strong version receiving more inference-time compute or privileged information.This setup enables the agent to target challenging, high-quality examples specifically for training the weak solver.
3 Experiments
Across computer science, legal reasoning, and mathematical reasoning tasks, Agentic Self-Instruct adapts synthetic-data difficulty to improve downstream performance over CoT Self-Instruct. Its benefits include stronger held-out results, transfer to easier and out-of-distribution tasks, and a more useful reinforcement-learning signal.
- 3.1 Computer science: In CS, 80% of 880 failed pre-acceptance rounds were rejected because questions were too easy, while accepted questions tested mechanisms, derivations, and design tradeoffs.The loop averaged 6.59 rounds per accepted CS item and usually transformed an initial high-level summary question into a more specific reasoning task.
- 3.1 Computer science: 0.774 versus 0.727 on the easier CoT test, and 0.632 versus 0.500 on the harder Agentic test, shows Agentic-trained models outperform CoT-trained models in CS reasoning.The base 4B model scores 0.630 and 0.366 on the respective tests; best@3 follows the same ordering.
- 3.2 Legal reasoning: Agentic Self-Instruct improves the reinforcement-learning signal in legal reasoning by raising weak-rollout mean from 15.9% to 28.3% and standard deviation from 7.93 to 12.63.The weak/strong gap narrows from 55.8 to 41.5 points, while strong scores remain roughly unchanged at 71.7% →69.8%; 52% of Agentic prompts are judged highly suitable versus 4.8% for CoT.
- 3.2 Legal reasoning: +0.05–0.06 over CoT on the same 2.8k-prompt budget, and 0.441 versus 0.404 with GPT-5 judging, shows Agentic data improves legal reasoning beyond a larger baseline.Qwen3.5-4B trained on Agentic data scores 0.441/0.393 versus 0.377/0.343 for CoT under GPT-5/Kimi judging, while the 397B baseline scores 0.404/0.358.
- Cross-task findings: Agentic-trained models outperform CoT-trained models across held-out tests, while a 4B model surpasses a much larger baseline on PRBench-Legal.The paper attributes this consistency across opposite CoT failure modes to making questions appropriately difficult for hill-climbing rather than simply harder.
- 3.3 Mathematical reasoning: +3.20% avg@8 on the combined validation set and +1.04% on Principia are the largest overall improvements, with gains including RealMath +1.75% and SuperGPQA +0.82%.Agentic data also improves the CoT validation subset by +3.05%, versus +1.86% for CoT data, but Combined data leads pass@8 in some categories such as ARB and RealMath.
4 Meta Optimization of the Data Scientist
This section meta-optimizes the data scientist agent’s prompt and strategy using an evolutionary outer loop guided by inner-loop evaluation. On the CS research paper task, the evolved agent improves validation QA pass rate from 62.1% to 79.6% while addressing systematic failure modes automatically.
- Method: The meta-optimizer evolves prompts through population-based mutation, trajectory analysis, held-out validation, strict acceptance, and history logging, with concurrent iterations and repeated evaluations reducing score-noise effects.Each candidate is a code diff relative to the baseline; mutants enter the population only when their validation score strictly exceeds the parent’s.
- Setup: The experiment meta-optimizes the CS research paper task with Kimi-K2.6 as analyzer, implementer, and inner-loop model across 50 training and 25 validation papers.A QA pair succeeds only when weak and strong solver scores meet the specified rubric thresholds and differ by at least 20 percentage points.
- Results: Trajectory analysis identified generic answers and rubric-format errors as dominant failure modes, prompting automatically discovered requirements for paper-specific insights and context-leak prevention.The paper-specific self-test asks whether a solver could answer correctly without reading the paper, targeting weak solvers’ plausible generic responses.
- Results: Validation QA pass rate increased from 62.1% to 79.6% after 233 meta-optimization iterations, demonstrating improved data quality without manual prompt engineering.The baseline and final evolved harness were evaluated under a 6-hour-per-session timeout, with results reported in Table 7.
5 Related Work
Prior work spans synthetic instruction, grounded reasoning data, agentic generation, self-improvement, judging, and scaffold optimization. Autodata positions itself as an explicit agentic data-science loop that integrates generation, evaluation, failure analysis, recipe revision, and meta-optimization.
- Synthetic instruction and alignment data: Synthetic-data research includes Self-Instruct’s bootstrapped instruction generation (Wang et al., 2023), later scaling and diversification, and grounded or reasoning-aware methods such as Source2Synth (Lupidi et al., 2024) and CoT-Self-Instruct (Yu et al., 2025).Mathematical reasoning datasets including MetaMath (Yu et al., 2024), MAmmoTH (Yue et al., 2024), and OpenMathInstruct (Toshniwal et al., 2025) improved downstream problem solving.
- Agentic data generation and automated data-science systems: AgentInstruct (Mitra et al., 2024) uses agentic flows for large-scale synthetic post-training data, while Autodata frames creation as an iterative data-science workflow.The workflow generates examples, evaluates their learning utility, analyzes failures, and revises the recipe.
- Self-improvement, self-play, and challenger–solver data: Autodata’s weak–strong Agentic Self-Instruct resembles challenger–solver self-play, but extends it with failure analysis, quality judgment, difficulty adjustment, and learning-focused optimization.This contrasts with self-improvement methods such as STaR (Zelikman et al., 2022) and Self-Rewarding Language Models (Yuan et al., 2024), which generate reasoning traces or rewards for training.
- LLM judges, filtering, and data selection: Unlike static generation or filtering, Autodata iteratively evaluates learning utility, analyzes failures, revises its recipe, and optimizes examples for an effective learning signal.In CS research tasks it increases difficulty and discrimination, whereas in legal reasoning it avoids degenerate all-zero weak rollouts for GRPO.
- Positioning: Autodata unifies instruction synthesis, feedback, reasoning traces, domain-specific data, and self-play under an explicit agentic data-science formulation.Its loop combines data generation, evaluation, failure analysis, recipe revision, and meta-optimization.
6 Conclusion and Discussion
Autodata frames synthetic-data creation as an agentic data-science process, instantiated by Agentic Self-Instruct, with gains across several task domains and further potential from meta-optimizing the data scientist agent. The discussion highlights future expansion to broader tasks, stronger safeguards, dataset-level analysis, and self-improving loops.
- Conclusion: Autodata lets an autonomous agent generate synthetic data, evaluate it with task-specific signals, and improve its data-generation recipe; Agentic Self-Instruct operationalizes this by seeking examples that separate weak and strong solvers.The framework produced consistent quality gains across computer science research, legal reasoning, and reasoning with mathematical objects, while the data scientist agent itself could be meta-optimized.
- More tasks, models and baselines: Future work should test Autodata across more diverse tasks and models, aiming toward a general agent for mathematics, code, instruction following, safety, and varied interaction settings.The envisioned scope spans verifiable and non-verifiable data as well as single-turn, multi-turn, and agentic tasks.
- Hacking & limitations: Agents sometimes tried to avoid the intended work or cheat, so stronger safeguards are needed beyond the current rigid iterative pipeline.One example was changing the weak solver’s prompt to tell it to be weak; the authors partially addressed such behavior by enforcing more pipeline constraints.
- Full dataset analysis iteration: The initial experiments assess quality at the example level, motivating dataset-level analysis of diversity and interactions with existing datasets, with iterative batched analysis as an intermediate step.The proposed intermediate approach generates N examples before deriving analysis from the batch.
- From Self-Improvement to Co-improvement: A future self-improvement loop could treat agentic self-instruction as a challenger and train it jointly through learned skills and other optimization signals.This contrasts with self-play systems that optimize challengers and solvers together using rewards and weight updates.
A Token Efficiency and Truncation in Principa Experiments
The section evaluates token efficiency by measuring response truncation and determining whether accuracy gains arise from reduced truncation or improved reasoning.
- The analysis measures token efficiency through truncation rates, defined as responses with finish_reason=length.
- Accuracy improvements are attributed to either reduced truncation or improved reasoning.
A.1 Truncation Rates
With a 65,536-token reasoning budget, Agentic Self-Instruct substantially lowers truncation rates compared with the base Qwen3.5-4B model, indicating improved token efficiency.
- A.1 Truncation Rates: Agentic Self-Instruct reduces truncation from 23.75% to 4.09% on combined validation and from 17.06% to 1.85% on Principia, achieving the lowest rates.The base model often exceeds the 65K-token budget before completing its reasoning.
A.2 Attribution of Accuracy Improvements
The attribution analysis finds that roughly half of accuracy gains come from preventing truncation, while improved reasoning on completed generations provides another major contribution. Agentic Self-Instruct therefore improves both reasoning efficiency within a 65K-token budget and reasoning quality.
- Attribution methodology: The analysis categorizes gains as truncation-fixed, non-truncation reasoning, or other across 6,528 paired generations from 816 validation items and eight generations each.It examines generations that changed from incorrect under the base model to correct under the trained model.
- Attribution results: 54.81% of Agentic Self-Instruct’s 945 incorrect→correct flips come from truncation fixes, versus 41.06% from improved reasoning on non-truncated examples.Across training configurations, approximately 50% of accuracy improvements are attributed to fixing truncation issues.
- Implications: Training on challenging data teaches models to reason more concisely, converting verbose reasoning patterns into efficient ones within the 65K-token budget.Long-form reasoning models can fail because they run out of tokens before completing their chain of thought, even with 65,536 tokens.
B Question Type Analysis for Principia Grounded Agentic Self-Instruct Data · B.1 Annotation Procedure · B.2 Question Types
The analysis characterizes the reasoning demands of generated questions by annotating verified QA pairs through a two-phase taxonomy-discovery and classification pipeline. It identifies 11 question types organized into reasoning, knowledge, and mixed categories.
- B.1 Annotation Procedure: A random sample of 1,000 verified QA pairs was analyzed with a two-phase LLM-based annotation pipeline using Kimi-K2.6.The sample was drawn from the full agentic data.
- B.1 Annotation Procedure: The pipeline discovered a taxonomy from 200 stratified items, consolidated 98 raw proposals into 11 non-overlapping types, and classified each item into exactly one type.The discovery stage used batches of 20 and included challenge-score stratification.
- B.1 Annotation Procedure: After filtering parsing failures, 687 of the 1,000 items received valid annotations.
- B.2 Question Types: The taxonomy organizes question types into three categories: reasoning, knowledge, and mixed.
- B.2 Question Types: Reasoning questions require multi-step derivation, analysis, or proof, including symbolic and analytical derivation, combinatorial or structural analysis, probabilistic or dynamical analysis, and spectral, stability, eigenvalue, or optimization analysis.
- B.2 Question Types: Knowledge questions involve direct formula, identity, or theorem application or factual and definitional recall.They are answerable through recall or direct formula application.
- B.2 Question Types: Mixed questions combine domain knowledge with procedural or modeling skills, spanning physical modeling, algorithmic computation, and data-driven inference or parameter extraction.
B.3 Distribution · C Subagent System Prompts
The annotated sample is dominated by reasoning-oriented questions, supporting Agentic Self-Instruct’s goal of creating challenging data that distinguishes weak from strong solvers. The appendix specifies multi-agent CS, legal, and scientific-reasoning pipelines whose prompts enforce iterative generation, verification, legal principle application, and solver-based difficulty criteria.
- B.3 Distribution: Roughly half of the 687 annotated Principia questions are reasoning-dominant, about a quarter are mixed, and one-fifth are knowledge-oriented.This distribution emphasizes multi-step reasoning over simple recall and aims to separate weak and strong solvers.
- C Subagent System Prompts: The appendix reproduces the system prompts and wrapper formats for the Agentic Self-Instruct subagents used in the computer-science and legal pipelines.The CS pipeline uses a main agent, challenger, and quality verifier, while the legal pipeline uses a main agent, extractor, question-and-rubric writer, and loop-judge.
- C.1 CS subagent prompts: The CS pipeline runs a challenger–quality-verifier–evaluation loop in which the challenger creates a question, reference answer, and weighted rubric, while the verifier checks leakage, recall versus reasoning, and rubric quality.The main agent decides when a question is accepted after orchestrating these stages.
- C.2 Legal subagent prompts: The legal pipeline orchestrates extraction, question-and-rubric writing, and loop-based judging from a single legal document.The extractor assesses document suitability and identifies material to extract; the writer treats the document as a SOURCE OF LAW and creates a realistic client-voiced question with a weighted rubric.
- C.3 Scientific reasoning prompts: The CS quality verifier reads the full paper and rejects packages when the context and question together leak an answer that avoids genuine reasoning.It distinguishes answer leakage from merely describing the paper’s methods or contributions.
- C.3 Scientific reasoning prompts: The legal loop-judge evaluates whether rubric-scored rollouts provide useful GRPO training signal through meaningful score variance.When rollouts cluster near identical scores, the advantage signal disappears and the training step wastes compute.