Autodata: An agentic data scientist to create high quality synthetic data

1 Introduction

• Synthetic data can cover underrepresented edge cases, reduce manual-labeling difficulty and latency, and potentially produce more challenging examples than human-generated data.High-quality training data and benchmarks that continue challenging models increasingly underpin progress at the AI frontier.Source: §1 · pp. 1–2
• Earlier synthetic-data methods lack direct difficulty and quality control, motivating filtering (Yu et al., 2025), evolution (Xu et al., 2024), and refinement (Shah et al., 2024).These earlier approaches include Self-Instruct (Wang et al., 2023), document-grounded generation (Lupidi et al., 2024; Yuan et al., 2025), Chain-of-Thought generation (Yu et al., 2025), and tool-using challenger agents (Zhou et al., 2025).Source: §1 · p. 2
• Autodata agents iteratively create and curate training or benchmark data, inspect examples, measure performance, extract lessons, and revise generation recipes.An outer meta-optimization loop trains the agentic system to perform better as a data scientist in the inner loop.Source: §1 · p. 2
• Agentic Self-Instruct improves on classical synthetic-data methods for computer science research, legal reasoning, and mathematical-object reasoning; meta-optimizing its data scientist yields further gains.The introduction reports these improvements qualitatively, without numerical results.Source: §1 · p. 2
• Agentic data creation converts increased inference compute into higher-quality model training by producing challenging data as existing tasks become insufficiently difficult.The authors position this approach as a way to create new tasks and benchmarks that continue challenging stronger models.Source: §1 · p. 2

2 Autodata

• Autodata grounds data creation in supplied sources, using tools, previously acquired skills, and inference-time compute to create training or evaluation data.Creation can be repeated after analysis and new learnings to improve the data further.Source: §2 · p. 2
• Autodata alternates data creation and analysis until quality is satisfactory, then produces a final training dataset or benchmark.Analysis can assess individual-example correctness, quality, and difficulty, or dataset diversity and training usefulness, feeding learnings into subsequent iterations. The outer loop can include guardrails against hacking.Source: §2 · p. 2
• Autodata can meta-optimize the data-scientist agent’s harness, using better-data criteria from the inner loop to guide outer-loop agent optimization.Suggested approaches include autoresearch and meta-harness (Lee et al., 2026) style optimization.Source: §2 · p. 3

2.1 A specific implementation: Agentic Self-Instruct

• Agentic Self-Instruct gives an orchestrator four LLM subagents: a challenger generating examples, weak and strong solvers, and a verifier/judge returning quality feedback.The weak solver is expected to struggle with generated examples, while the strong solver is expected to succeed.Source: §2.1 · p. 3
• Agentic Self-Instruct seeks examples strong solvers solve but weak solvers struggle with, revising challenger prompts using verifier feedback until its criteria are met.For verifiable tasks, one approach requires correct strong-solver and incorrect weak-solver majority votes; non-verifiable tasks require a judge-measured quality gap calibrated to the weak solver. The judge also checks the question, reference answer, or generated rubric.Source: §2.1 · pp. 3–4
• Weak and strong solvers can be the same LLM operating in different modes rather than different models.The strong mode can receive additional inference-time compute, including scaffolding or aggregation (Zhao et al., 2025b), or access to privileged information.Source: §2.1 · p. 4

3.1.1 Agentic Self-Instruct Loop Analysis

• 0.677 →0.458 weak-solver scores and 0.696 →0.772 strong-solver scores show that accepted CS questions discriminate more effectively than CoT-generated questions.Accepted items require 6.59 rounds on average; among 880 pre-acceptance rounds, 80% fail because questions are too easy and 13% because the strong solver struggles. Feedback redirects generic summaries toward technical mechanisms, multi-step derivations, numerical claims, and paper-specific design tradeoffs.Source: Table 1 · p. 4, §3.1.1 · pp. 5–6

3.1 Computer Science Research tasks

• CS generation iteratively revises paper-grounded, open-ended questions and weighted rubrics, accepting questions that remain solvable for the strong solver but challenge the weak solver.Acceptance requires strong scores ≥0.65, weak scores <0.5, and a strong−weak gap ≥20 percentage points across three attempts per solver. A verifier checks leakage, rubric coverage, and question quality; rejected candidates receive targeted feedback prompting a different reasoning angle.Source: §3.1 · p. 4

3.1.2 RL Training Results

• At step 200, Agentic-trained Qwen3.5-4B achieves mean@3 of 0.774 versus CoT-trained 0.727 on the CoT CS test and 0.632 versus 0.500 on the Agentic CS test.Both models receive GRPO training on 1.3k examples from their respective data sources, with Kimi-K2.6 scoring responses against generated rubrics. Agentic matches or exceeds CoT at every validation checkpoint on both test sets, and best@3 follows the same ordering.Source: Table 2 · p. 5, §3.1.2 · p. 6

3.2.1 Agentic Self-Instruct Loop Analysis

• 7.93 →12.63 per-prompt weak-rollout standard deviation and 15.9% →28.3% weak mean scores make legal rewards more useful for GRPO.The weak/strong gap narrows from 55.8 to 41.5 points, reversing the CS pattern while avoiding near-zero learning signals. High GRPO-suitability verdicts rise from 4.8% to 52%; accepted questions require a median four rounds and become shorter, application-oriented questions without reduced rubric density.Source: Table 3 · p. 7, §3.2.1 · pp. 7–8

3.2 Legal Reasoning Tasks / “More Challenging” vs “Just Right”

• Legal generation replaces fixed CS thresholds with a judge assessing data quality and GRPO suitability, addressing overly difficult questions rather than maximizing challenge.An extractor structures legal documents before question-and-rubric generation; candidates receive five weak-solver and three strong-solver rollouts. The judge considers rollout variance, solver separation, and rubric concerns, then accepts or supplies concrete revision feedback without fixed acceptance thresholds.Source: §3.2 · p. 7, “More Challenging” vs “Just Right” · p. 9

3.2.2 RL Training Results

• 0.441 on GPT-5-judged PRBench-Legal puts Agentic-trained Qwen3.5-4B above CoT-trained Qwen3.5-4B (0.377) and the untrained Qwen3.5-397B-A17B baseline (0.404).Kimi-judged scores follow the same ordering: 0.393, 0.343, and 0.358; both graders also agree on PRBench-Legal-Hard. Agentic and CoT use matching 2.8k-prompt training budgets, and Agentic leads throughout the evaluated training checkpoints.Source: Table 4 · p. 8, §3.2.2 · pp. 8–9, Figure 5 · p. 8

3.3 Scientific reasoning / 3.3.1 RL training results

• +3.20% avg@8 on scientific combined validation favors Agentic data over CoT (+2.42%) and Combined (+2.70%), despite Combined using twice the training data.The experiment generates mathematical-object problems grounded in the Principia collection (Aggarwal et al., 2026), using 9k training examples per individual source versus 18k for Combined. Agentic also achieves the best overall out-of-distribution Principia avg@8 improvement (+1.04%) and improves the CoT validation subset by +3.05% versus CoT training’s +1.86%.Source: §3.3.1 · pp. 9–10, §3.3 · p. 9, Table 5 · p. 10

3.3.1 RL training results

• +2.37% pass@8 on RealMath favors Combined over Agentic (+1.74%), showing that Agentic’s average-performance advantage does not extend to every coverage metric.Combined also improves ARB pass@8 by +2.13%, versus +0.00% for both Agentic and Grounding. Agentic training reduces reasoning truncation from 23.75% to 4.09% under a 65,536-token budget, with approximately half its accuracy improvement attributed to more token-efficient reasoning.Source: §3.3.1 · p. 10

Method

• Evolutionary meta-optimization edits the agent’s scaffold as code, using inner-loop evaluation criteria to optimize its prompts and strategy.Boltzmann sampling with T=0.1 selects parents for training-minibatch evaluation, trajectory-based failure analysis, and code-editing mutations. Concurrent iterations log outcomes and accept mutants only when they strictly outperform their parents on held-out validation papers; repeated parent evaluations are averaged to reduce noise.Source: §4 · pp. 11–12

Setup

• 50 training papers and 25 validation papers support CS-task meta-optimization, with Kimi-K2.6 analyzing failures, editing prompts, and running challenger, main-agent, and verifier roles.Rubric-based success requires weak solver Qwen3.5-4B to score ≤65%, its best attempt ≤75%, and strong solver Qwen3.5-397B-A17B to score ≥60% and ≤95%. The strong–weak score gap must be ≥20 percentage points.Source: §4 · p. 12

Results

• 79.6% validated pass rate, up from 62.1%, was achieved by automatically evolving the data scientist’s instructions on the CS research paper task.The search ran for 233 iterations; baseline and final evolved harness results used a six-hour timeout per session on 100 validation samples.Source: Table 7 · p. 12, §4 · p. 12
• Automatically discovered prompts require paper-specific questions and prevent contexts from revealing the paper’s solution, addressing generic answers that weak solvers could exploit.Self-tests reject questions answerable without reading the specific paper and require rewriting contexts when answers can be obtained by rephrasing them.Source: §4 · p. 12
• Automatically discovered rubrics use positive integer weights capped at 7 and strict JSON, avoiding harmful negative criteria and parsing failures.Negative-weight criteria had reduced strong-model scores without improving discrimination; structured formatting eliminated errors such as string-valued weights.Source: §4 · p. 12

Synthetic instruction and alignment data

• Self-Instruct (Wang et al., 2023) and subsequent synthetic-instruction methods largely use fixed pipelines; Autodata instead evaluates examples, analyzes failures, and revises generation.Prior extensions encompass teacher distillation, synthetic conversations, preference feedback, instruction evolution, and large-scale alignment data from minimal prompting.Source: §5 · p. 13

Grounded, verifiable and reasoning-based synthetic data

• Autodata builds on grounded, domain-specific, reasoning-aware synthesis, adding an explicit agentic loop that uses solver behavior and evaluator feedback to adapt data to the target model.Prior approaches include synthetic mathematical reasoning datasets and document- or table-grounded generation, while CoT-Self-Instruct (Yu et al., 2025) uses chain-of-thought planning and filtering.Source: §5 · p. 13

Agentic data generation and automated data-science systems

• Unlike AgentInstruct (Mitra et al., 2024), Autodata explicitly evaluates learning utility and revises its generation recipe; Agentic Self-Instruct also supports meta-optimizing the data scientist.Weak–strong solver behavior and judge feedback tune difficulty. Unlike web-collection and general data-science agents, its output is training or evaluation data for another model.Source: §5 · p. 13

Self-improvement, self-play, and challenger–solver data

• Agentic Self-Instruct shares challenger–solver task generation with self-improvement methods but analyzes failures, judges quality, and adjusts difficulty to prioritize learning usefulness over difficulty alone.Related approaches include rationale bootstrapping in STaR (Zelikman et al., 2022), self-generated verifiable tasks in Absolute Zero (Zhao et al., 2025a), and corpus-grounded challenger–reasoner training in SPICE (Liu et al., 2025).Source: §5 · pp. 13–14

LLM judges, filtering, and data selection

• Autodata incorporates judge feedback into generation rather than merely filtering examples, adapting difficulty to provide an effective learning signal for the target model.CS research examples become harder and more discriminative; legal reasoning examples become less degenerate and more suitable for GRPO by avoiding all-zero weak rollouts.Source: §5 · p. 14

Autoresearch and optimization of agent scaffolds

• Autodata extends prompt and agent-scaffold optimization to data creation: an outer loop improves the data scientist’s prompt and strategy using inner-loop data-quality criteria.Related approaches include iterative optimization through GEPA (Agrawal et al., 2025) and end-to-end harness optimization through Meta-Harness (Lee et al., 2026).Source: §5 · p. 14

Positioning

• Autodata unifies synthetic-data approaches within an agentic loop, using stronger inference-time models and larger agentic compute budgets to improve training and evaluation data.Generation, evaluation, failure analysis, recipe revision, and meta-optimization are all part of this formulation.Source: §5 · p. 14

6 Conclusion and Discussion

• Agentic Self-Instruct delivered consistent quality gains across computer science research, legal reasoning, and mathematical-object reasoning; meta-optimizing the data scientist yielded substantial additional improvements.The method explicitly targets examples that separate weak and strong solvers. Meta-optimization improved the agent without manual prompt engineering.Source: §6 · p. 14

More tasks, models and baselines

• Autodata’s autonomous data scientist generates, evaluates, and refines synthetic data; future work seeks broader tasks, models, verifiability settings, and interaction formats.Evaluation uses task-specific signals to improve the generation recipe. The envisioned general agent would cover verifiable and non-verifiable data, spanning single-turn, multi-turn, and agentic tasks.Source: §6 · p. 14

Hacking & limitations

• Agents sometimes attempted to cheat the objective, while some generated questions tested paper-specific experimental numbers rather than generalizable reasoning.Additional pipeline constraints partially addressed cheating, including instructions telling the weak solver to perform poorly. Stronger safeguards are needed to permit greater agent freedom and tool use.Source: §6 · p. 15

Full dataset analysis iteration

• Current experiments optimize individual examples; future dataset-level analysis would assess diversity and interactions with existing datasets, with iterative batch analysis as an intermediate step.In the batched approach, the agent generates N examples and derives lessons from that batch to guide the next.Source: §6 · p. 15

From Self-Improvement to Co-improvement

• Human–agent co-improvement (Weston and Foerster, 2025) is a main future direction, alongside jointly training the agentic challenger and solver.A joint loop could train the challenger’s learned skills and weights alongside the solver, extending the autoresearch-like meta-training explored here. The authors favor retaining human feedback and co-research rather than removing humans entirely from current training pipelines.Source: §6 · p. 15

• Training’s impact on token efficiency is examined using response truncation rates.Source: A Token Efficiency and Truncation in… · p. 19
• The analysis distinguishes accuracy improvements attributable to reduced truncation from those attributable to improved reasoning.Source: A Token Efficiency and Truncation in… · p. 19
• Truncated responses are identified by finish_reason=length.Source: A Token Efficiency and Truncation in… · p. 19

A.1 Truncation Rates

• 10.00% on combined validation and 6.62% on Principia were Grounding’s truncation rates, improving over the base model but remaining above Agentic Self-Instruct.All configurations used the same 65,536-token reasoning budget.Source: §A.1 · p. 19
• Lower truncation after training suggests the model learns to reason more concisely and efficiently within the token budget.This is an interpretation of the truncation results rather than a direct measurement of reasoning concision.Source: §A.1 · p. 19
• 4.09% and 1.85% truncation on combined validation and Principia, respectively, followed Agentic Self-Instruct training, versus 23.75% and 17.06% for base Qwen3.5-4B.Truncation means a response reached the token limit before completing its reasoning.Source: Table 8 · p. 19, §A.1 · p. 19

A.2 Attribution of Accuracy Improvements

• 6,528 paired generations (816 QA items × 8 generations) underpin an attribution analysis of base-model errors corrected by the trained model.Flips were classified as truncation-fixed (base truncated, trained completed), non-truncation reasoning (neither truncated), or other, including cases where both truncated but the trained model was correct.Source: §A.2 · p. 19
• 54.81% of Agentic Self-Instruct’s 945 incorrect-to-correct flips came from truncation fixes, while 41.06% came from improved reasoning on non-truncated examples.Across all training configurations, approximately 50% of accuracy improvements came from fixing truncation.Source: Table 9 · p. 19, §A.2 · p. 19
• Training on challenging synthetic data, particularly Agentic Self-Instruct data, appears to improve reasoning concision as well as quality under fixed computational budgets.The findings suggest Qwen3.5-4B often fails by exhausting its 65,536-token budget rather than lacking reasoning ability.Source: §A.2 · pp. 19–20

B.1 Annotation Procedure

• 1,000 verified QA pairs were randomly sampled from the full agentic dataset for a two-phase annotation pipeline using Kimi-K2.6.The phases were taxonomy discovery and question-type annotation.Source: §B.1 · p. 20
• 98 proposed categories were consolidated into 11 non-overlapping question types.Taxonomy discovery used 200 items stratified by challenge score, presented in batches of 20 to elicit categories, definitions, and examples.Source: §B.1 · p. 20
• 687 of 1,000 items received valid annotations after filtering parsing failures, with classification assigning each item exactly one of 11 types.The subsequent type analysis therefore uses fewer items than the original random sample.Source: §B.1 · p. 20

B.2 Question Types

• Reasoning types cover symbolic derivation, combinatorial and structural analysis, probabilistic and dynamical analysis, spectral and optimization analysis, asymptotics, and formal proof.These include chaining algebraic or calculus manipulations, analyzing stability, extracting limiting behavior, and proving or disproving claims.Source: §B.2 · p. 20
• Knowledge questions involve direct formula, identity, or theorem application, or factual and definitional recall.Formula application requires selecting a known formula or theorem and substituting parameters in one or two steps; recall requires no derivation.Source: §B.2 · p. 20
• Mixed questions combine domain knowledge with physical modeling, procedural computation, or data-driven inference.Tasks include deriving and solving governing equations, executing defined multi-step procedures, and extracting quantities from data, fits, or observations.Source: §B.2 · p. 20

B.3 Distribution

• Roughly half of the annotated questions are reasoning-dominant, about a quarter are mixed, and one-fifth are knowledge-oriented.The sample contains 687 Principia questions. This distribution supports the goal of emphasizing multi-step reasoning over recall to distinguish weak and strong solvers.Source: §B.3 · pp. 20–21, Table 10 · p. 21

C Subagent System Prompts

• The appendix reproduces repository system prompts, preserving the role, workflow, and output-schema wrapper across subagents within each CS or legal setting.The prompts come from the repository’s .opencode/prompts/ directories and correspond to the pipelines reported in Sections 3.1 and 3.2.Source: C Subagent System Prompts · p. 21

C.1 CS subagent prompts

• The CS main agent runs a challenger → quality-verifier → evaluate-rubric loop on a single paper and decides when to accept a question.The pipeline uses three subagents: the main agent, challenger, and quality verifier.Source: §C.1 · p. 21
• The CS challenger produces a question, reference answer, and weighted rubric; the quality verifier checks answer leakage, recall versus reasoning, and rubric well-formedness.The challenger reads the source paper before generating these outputs.Source: §C.1 · p. 21

C.2 Legal subagent prompts

• The legal pipeline coordinates document extraction, client-voiced question and weighted-rubric writing, and solver-pattern judging through a four-subagent loop.The extractor assesses document suitability and selects material; the writer treats the document as a source of law and declares targeted legal-reasoning capabilities. The loop-judge uses per-rollout solver patterns and the rubric to return an accept/improve verdict, supplying a writer suggestion when improvement is needed.Source: §C.2 · p. 21

C.3 Scientific reasoning prompts

• Scientific-reasoning prompt documentation includes a challenger system prompt and an excerpt from the main orchestrator agent’s prompt.These appear in Figures 14 and 15, respectively.Source: Figure 14 · p. 27, Figure 15 · p. 28