Source-linked AI summary
Autodata: An agentic data scientist to create high quality synthetic data
Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, Jason Weston
TL;DR
Synthetic data creation needs to produce increasingly challenging, high-quality training and evaluation data as models improve. Autodata treats an AI agent as a data scientist that iteratively creates and evaluates data, and Agentic Self-Instruct improves results across computer science, legal reasoning, and mathematical reasoning tasks, with further gains from meta-optimizing the agent.
Problem
Existing synthetic data methods may struggle to produce sufficiently challenging, high-quality training and evaluation data as models become stronger.
Method
Autodata uses an agentic data scientist to create, evaluate, and iteratively improve training or evaluation data, with optional meta-optimization of the agent itself.
Results
Agentic Self-Instruct improves results across computer science, legal reasoning, and mathematical reasoning tasks, while meta-optimizing the data scientist agent delivers additional gains.
Takeaways & Limitations
Agentic data creation can convert inference-time compute into higher-quality, more challenging synthetic data for model training and benchmarking.
Takeaways & Limitations
Agents sometimes try to evade the intended task, and generated questions or rubrics may be overly tied to specific experimental numbers rather than testing generalizable reasoning.
Abstract
from arXiv · showhide
We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data. We show how to train (meta-optimize) such a data scientist agent, so that it learns to create even stronger data. We describe the overall formulation, and a specific practical implementation, Agentic Self-Instruct. We conduct experiments on computer science research tasks, legal reasoning tasks and reasoning with mathematical objects, where we obtain improved results compared to classical synthetic dataset creation methods. Further, meta-optimizing the data scientist agent itself delivers an even larger performance uplift. Agentic data creation provides a way to convert increased inference compute into higher quality model training. Overall, we believe this direction has the potential to change the way we build AI data.
1 Introduction
Autodata frames data construction as an agentic data-science process that creates and curates training and benchmark data through iterative generation, analysis, and performance measurement. Its Agentic Self-Instruct implementation improves results across computer science, legal reasoning, and mathematical-object tasks, with further gains from meta-optimizing the data scientist agent.
- Motivation: Synthetic data can generate underrepresented edge cases and long-tail scenarios, reduce manual-labeling difficulty and latency, and potentially be more challenging than human-generated data.These benefits motivate synthetic data as performance improvements increasingly depend on high-quality training data and challenging benchmarks.
- Problem: Prior synthetic-data methods use prompting, grounding, chain-of-thought, self-challenging, filtering, evolution, or refinement, but do not directly control data difficulty and quality.Autodata generalizes these approaches by making an agent responsible for data construction and curation.
- Contribution: Autodata assigns an agent the human data scientist’s role of constructing and curating high-quality training and benchmark data through creation, analysis, and performance measurement.The process includes an initial data-creation iteration followed by eyeballing and measuring performance.
- Results: Agentic Self-Instruct improves results over classical synthetic dataset creation methods across computer science research, legal reasoning, and reasoning with mathematical objects.Meta-optimizing the data scientist agent delivers an even larger performance uplift.
- Significance: Agentic data creation converts increased inference compute into higher-quality training data and may enable sufficiently challenging tasks and benchmarks for increasingly capable LLMs.The paper argues that this direction could change how new tasks and benchmarks are created to advance the AI frontier.
2 Autodata
Autodata is an iterative data-scientist loop in which an agent creates, analyzes, and refines training or evaluation data until quality criteria are met. Agentic Self-Instruct implements this design with challenger, weak and strong solvers, and a verifier, while the agent itself can also be meta-optimized.
- Autodata: Autodata repeatedly creates and analyzes data, feeding example- and dataset-level learnings back into creation before producing a final training dataset or benchmark.The outer loop can include guardrails against hacking and continues until the agent is satisfied with data quality.
- Agentic Self-Instruct: The system targets examples that strong solving succeeds on while weak solving struggles, using verifier feedback to revise the challenger prompt until criteria are met.For verifiable tasks, strong and weak majority votes should respectively be correct and incorrect; for non-verifiable tasks, the judge should identify a quality gap.
- Meta-Optimization of the Data Scientist: The data-scientist agent can itself be optimized with autoresearch or meta-harness (Lee et al., 2026) methods using inner-loop data quality to guide outer-loop optimization.This extends the iterative data-creation objective to improving the agent harness itself.
- Agentic Self-Instruct: Agentic Self-Instruct uses a main orchestrator with a Challenger, Weak solver, Strong solver, and Verifier/judge to generate and assess training examples.The challenger creates examples, the solvers attempt them, and the verifier checks solution quality and passes its learnings to the main agent.
- Agentic Self-Instruct: Weak and strong solvers may be the same LLM in different modes, with the strong version receiving more inference-time computation or privileged information.This supports constructing challenging examples specifically for training the weak solver.
3 Experiments
Across computer science, legal reasoning, and mathematical reasoning tasks, Agentic Self-Instruct produces training data that improves downstream performance over CoT Self-Instruct. Its iterative loop adapts question difficulty to create more useful learning signals and supports transfer beyond the targeted difficulty level.
- Mathematical reasoning: +3.20% avg@8 is achieved on the combined validation set with Agentic data, versus +2.42% for CoT data and +2.70% for combined data.Agentic data also improves the CoT validation subset by +3.05%, compared with +1.86% for CoT data.
- Data creation process: The loop typically requires multiple refinement rounds: accepted questions average 6.59 rounds in CS, while the legal pool has 52% high-suitability versus 4.8% for CoT.In CS, 80% of failed rounds are rejected because questions are too easy for the weak solver; accepted questions test technical mechanisms, derivations, and design tradeoffs.
- CS research tasks: On the harder Agentic test set, performance rises from 0.366 for the base model to 0.500 with CoT data and 0.632 with Agentic data; the same ordering holds on the easier CoT test.On the CoT test, scores are 0.630, 0.727, and 0.774 respectively, while Agentic remains at least as strong throughout training checkpoints.
- PRBench-Legal: On PRBench-Legal, Agentic-trained Qwen3.5-4B scores 0.441/0.393 versus 0.377/0.343 for CoT-trained models and 0.404/0.358 for the larger untrained baseline.The Agentic advantage is +0.05–0.06 on the same 2.8k-prompt budget, challenger, and source corpus, and the ordering also holds on PRBench-Legal-Hard.
- Overall findings: Agentic Self-Instruct data outperforms CoT-trained models on every held-out test, while a trained 4B model also surpasses a much larger baseline on PRBench-Legal.The key is making questions appropriately difficult for hill-climbing rather than simply making them harder.
- Mathematical reasoning: +1.04% overall avg@8 improvement is achieved on the out-of-distribution Principia benchmark, including +1.75% on RealMath and +0.82% on SuperGPQA.Pass@8 has trade-offs: Combined data leads on ARB (+2.13% vs. +0.00% for Agentic and Grounding) and RealMath (+2.37% vs. +1.74% for Agentic).
4 Meta Optimization of the Data Scientist
Meta-optimizing the data scientist agent’s prompt and strategy improved CS research QA generation from 62.1% to 79.6% validation pass rate over 233 iterations. The optimizer used trajectory analysis to identify and address systematic failures such as generic answers and rubric-format errors.
- Method: The meta-optimizer evolved prompts and strategies by sampling candidate code-diff mutations, analyzing solver trajectories, editing prompts, and accepting mutants only when held-out validation scores strictly improved.Accepted candidates received additional evaluations when resampled because solver scores were noisy.
- Setup: The experiment used Kimi-K2.6 for analysis, implementation, and the inner multi-agent data scientist, with 50 training papers and 25 validation papers.A QA pair succeeded when weak solvers scored at most 65%, their best attempt at most 75%, strong solvers between 60% and 95%, and the solver gap was at least 20 percentage points.
- Results: 62.1% to 79.6%: Meta-optimization improved validation QA pass rate over 233 iterations on the CS research paper task.The final comparison used a 6-hour-per-session timeout, and validation pass rate was re-evaluated to reduce score noise.
- Results: Trajectory analysis found generic answers and rubric-format errors as dominant failure modes, prompting paper-specific insight requirements and context-leak prevention rules.The paper-specific self-test required questions to be answerable only by reading the specific paper, preventing plausible generic responses from scoring highly.
5 Related Work
Autodata unifies synthetic-data, reasoning-aware, agentic, self-improvement, judging, and scaffold-optimization approaches under an explicit agentic data-science loop. Its distinguishing feature is integrating generation, evaluation, failure analysis, recipe revision, and meta-optimization.
- Grounded, verifiable and reasoning-based synthetic data: Autodata builds on grounded and reasoning-aware methods that generate verifiable, domain-specific examples from documents, tables, or planned reasoning traces.MetaMath (Yu et al., 2024), MAmmoTH (Yue et al., 2024), OpenMathInstruct (Toshniwal et al., 2025), Source2Synth (Lupidi et al., 2024), NaturalReasoning (Yuan et al., 2025), and CoT-Self-Instruct (Yu et al., 2025) exemplify this line.
- Agentic data generation and automated data-science systems: Autodata extends agentic generation by treating data creation as an iterative data-science loop that evaluates learning utility, analyzes failures, and revises the generation recipe.AgentInstruct (Mitra et al., 2024) uses agentic flows for large-scale, diverse synthetic post-training data, whereas Autodata emphasizes iterative evaluation and revision.
- Self-improvement, self-play, and challenger–solver data: Its weak–strong Agentic Self-Instruct instantiation resembles challenger–solver self-play but additionally analyzes failures, judges quality, adjusts difficulty, and optimizes learning utility.This differs from methods such as STaR (Zelikman et al., 2022), Self-Challenging Language Model Agents (Zhou et al., 2025), Absolute Zero (Zhao et al., 2025a), and SPICE (Liu et al., 2025), which generate reasoning traces or self-play tasks.
- LLM judges, filtering, and data selection: Unlike static-pool filtering, Autodata incorporates the judge’s feedback into generation so examples become harder and more discriminative for CS research or less degenerate for legal GRPO.The target is an effective learning signal rather than simply high quality or high difficulty.
- Positioning: Autodata unifies prior approaches to synthetic instructions, conversations, feedback, reasoning traces, domain-specific datasets, and self-play tasks under an explicit agentic data-science formulation.The loop integrates data generation, evaluation, failure analysis, recipe revision, and meta-optimization.
6 Conclusion and Discussion
Autodata frames synthetic data creation as an agentic process of generation, evaluation, and recipe improvement, instantiated by Agentic Self-Instruct with gains across multiple task domains. The discussion identifies meta-optimization, broader applicability, safeguards, dataset-level analysis, and co-improvement as key directions.
- Conclusion: Autodata treats an autonomous agent as a data scientist that generates synthetic data, evaluates it with task-specific signals, and improves its generation recipe.Agentic Self-Instruct optimizes for examples that separate weak and strong solvers, yielding consistent gains across computer science research, legal reasoning, and mathematical-object reasoning tasks; the data scientist agent can also be meta-optimized.
- More tasks, models and baselines: Future work should test Autodata across more diverse tasks and models, toward a general agent spanning mathematics, code, instruction following, safety, and agentic settings.The envisioned system would support verifiable and non-verifiable data, as well as single-turn, multi-turn, and agentic tasks.
- Hacking & limitations: Agents sometimes attempted to hack the objective, such as weakening the weak solver through prompt changes, motivating stronger safeguards beyond the current rigid iterative loop.The authors partially addressed this behavior by enforcing more pipeline constraints and plan safeguards that permit greater freedom to act and use tools.
- Full dataset analysis iteration: The initial experiments improve data at the example level, while future work should analyze full datasets for diversity and interactions with existing data.Iterative batched analysis—generating N examples before deriving lessons—offers an intermediate step toward full dataset-level analysis.
- From Self-Improvement to Co-improvement: Autodata’s agentic self-instruction could serve as a challenger in a self-improving loop, with joint optimization of its learned skills and weights.This direction extends prior self-play approaches that jointly optimize challenger-generated training examples, rewards, and model updates.
A Token Efficiency and Truncation in Principa Experiments
The section analyzes training’s impact on token efficiency through truncation rates and separates accuracy gains from reduced truncation versus improved reasoning.
- Training impact is evaluated using truncation rates, defined as responses with finish_reason=length.
- Accuracy improvements are attributed to either reduced truncation or improved reasoning.
A.1 Truncation Rates
With a 65,536-token reasoning budget, the base Qwen3.5-4B model frequently truncated responses, while training substantially improved token efficiency. Agentic Self-Instruct achieved the lowest reported truncation rates on both evaluated datasets.
- A.1 Truncation Rates: The base Qwen3.5-4B model truncated 23.75% of responses on combined validation and 17.06% on Principia.Many responses exceeded the 65,536-token budget before completing their reasoning.
- A.1 Truncation Rates: Experiments used a 65,536-token reasoning budget to measure truncation across different training configurations.Table 8 reports truncation rates defined by finish_reason=length.
- A.1 Truncation Rates: Agentic Self-Instruct achieved the lowest truncation rates, reducing them to 4.09% on combined validation and 1.85% on Principia.These rates were measured with finish_reason=length under a 65,536-token reasoning budget.
A.2 Attribution of Accuracy Improvements
The attribution analysis finds that roughly half of accuracy improvements come from fixing truncation, while substantial gains also arise from better reasoning on non-truncated examples. These results indicate that synthetic-data training improves both reasoning quality and efficiency under fixed token budgets.
- Attribution results: Across all training configurations, approximately 50% of accuracy improvements came from resolving truncation issues.Improvements were categorized as truncation-fixed, non-truncation reasoning, or other cases.
- Implications: Long-form reasoning models often fail because they exhaust even a 65,536-token budget before completing their chain of thought, rather than because they lack reasoning ability.Training on challenging data, particularly Agentic Self-Instruct data, teaches more concise reasoning.
- Implications: Synthetic-data training therefore improves reasoning efficiency, enabling models to solve more problems within fixed computational budgets.The benefit extends beyond improving reasoning quality itself.
B Question Type Analysis for Principia Grounded Agentic Self-Instruct Data · B.1 Annotation Procedure · B.2 Question Types
The analysis characterized the reasoning demands of agentic data through a two-phase annotation pipeline and organized the resulting question types into reasoning, knowledge, and mixed categories. From 1,000 sampled verified QA pairs, 687 received valid classifications across 11 non-overlapping types.
- B.1 Annotation Procedure: The study annotated a random sample of 1,000 verified QA pairs from the full agentic data using a two-phase LLM-based pipeline with Kimi-K2.6.The pipeline was designed to characterize the reasoning demands of generated questions.
- B.1 Annotation Procedure: The taxonomy-discovery phase sampled 200 items stratified by challenge score and consolidated 98 raw proposals into 11 non-overlapping question types.Items were presented in batches of 20 with requests for categories, definitions, and examples.
- B.1 Annotation Procedure: Each of the 1,000 items was classified into exactly one of the 11 types, with 687 items receiving valid annotations after parsing failures were filtered.
- B.2 Question Types: The 11 question types were organized into three categories: Reasoning, Knowledge, and Mixed.
- B.2 Question Types: Reasoning questions require multi-step derivation, analysis, or proof, including symbolic and analytical, combinatorial and structural, probabilistic and dynamical, and spectral, stability, eigenvalue, and optimization analysis.
- B.2 Question Types: Knowledge questions are answerable through recall or direct formula application, including theorem and identity use as well as factual and definitional recall.
- B.2 Question Types: Mixed questions combine domain knowledge with procedural or modeling skills, covering physical first-principles synthesis, algorithmic computation, and data-driven parameter extraction.
B.3 Distribution · C Subagent System Prompts
The annotated sample emphasizes reasoning-heavy questions, while the Agentic Self-Instruct pipelines use specialized subagents and prompts to generate, verify, and select challenging CS, legal, and scientific-reasoning data.
- B.3 Distribution: Roughly half of the 687 annotated Principia questions are reasoning-dominant, about a quarter are mixed, and one-fifth are knowledge-oriented.This distribution emphasizes multi-step reasoning over simple recall and aims to separate weak and strong solvers.
- C Subagent System Prompts: The reproduced system prompts preserve each subagent’s role, workflow, and output schema across the CS, legal, and scientific-reasoning pipelines.The appendix states that these prompts come from the repository’s .opencode/prompts/ directories.
- C.1 CS subagent prompts: The CS pipeline uses a main agent, challenger, and quality verifier in an iterative loop that accepts questions only after evaluation.The challenger creates a question, reference answer, and weighted rubric, while the verifier checks answer leakage, recall versus reasoning, and rubric well-formedness.
- C.2 Legal subagent prompts: The legal pipeline uses a main orchestrator, extractor, question-and-rubric writer, and loop-judge to transform one legal document into challenging training data.The extractor assesses suitability and extracts principles; the writer invents a realistic client scenario and produces a weighted rubric targeting legal-reasoning capabilities.
- C.3 Scientific reasoning prompts: The CS challenger prompt requires full-paper reading and generates question types, reasoning-skill tags, non-leaking context, and deep-reasoning questions.The CS quality verifier tests whether context and question together leak an answer without genuine reasoning.
- C.3 Scientific reasoning prompts: The legal orchestrator caps improvement at 15 rounds and requires a question whose answer demands non-trivial legal analysis beyond the weak solver’s current ability.The loop-judge evaluates whether GRPO rollouts vary enough to provide an advantage signal rather than clustering at identical scores.
- C.3 Scientific reasoning prompts: The scientific-reasoning orchestrator accepts a question only when the weak solver gets at most 1 correct answer out of 4 attempts.This hard gate is presented as a mandatory success criterion for generating discriminative question–answer pairs.