PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenThoughts-Agent: Data Recipes for Agentic Models

Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie F. Ruan, Tyler Griggs, Alexander Glenn Shaw, Hritik Bansal, E. Kelly Buchanan, Artem Gazizov, Reinhard Heckel, Chinmay Hegde, Sankalp Jajee, Daanish Khazi, Emmanouil Koukoumidis, Xiangyi Li, Hange Liu, Shlok Natarajan, Harsh Raj, Nicholas Roberts, Ethan Shen, Nishad Singhi, Michael Siu, Ashima Suvarna, Hanwen Xing, Patrick Yubeaton, Robert Zhang, Leon Liangyu Chen, Xiaokun Chen, Steven Dillmann, Saadia Gabriel, Xunyi Jiang, Anurag Kashyap, Boxuan Li, Yein Park, Minh Pham, Sujay Sanghavi, Lin Shi, Ke Sun, Yixin Wang, Zhiwei Xu, Erica Zhang, Siyan Zhao, Wanjia Zhao, Jenia Jitsev, Alex Dimakis, Benjamin Feuer, Ludwig Schmidt

arXiv:2606.24855v1cs.AI

TL;DR

Open research has limited information about curating training data for agents that generalize across diverse tasks. OpenThoughts-Agent studies an open SFT and RL curation pipeline through controlled ablations, then produces strong cross-benchmark performance and scaling results with released data and models.

  • Problem

    Existing open agent-training efforts provide limited guidance for broadly capable models because they typically focus on individual benchmarks.

  • Method

    OpenThoughts-Agent ablates stages of an SFT data-curation pipeline, studies agentic RL data, and selects strategies using downstream benchmark performance.

  • Results

    OpenThinker-Agent-32B is the strongest open-data <=32B model across seven agentic benchmarks, while the 100K-data model achieves 54.0% on SWE-Bench Verified and 26.2% on Terminal-Bench 2.0.

  • Takeaways & Limitations

    The training data shows strong compute-controlled scaling across training-set sizes, and the released datasets, pipeline, experimental data, and models support open agentic-model research.

  • Takeaways & Limitations

    The RL investigation is limited to the 8B scale, so whether the same recipe transfers to 32B remains an open question.

Abstract

from arXiv · show

Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents. Existing open efforts such as SWE-Smith, SERA, and Nemotron-Terminal typically target a single benchmark, leaving open the question of how to train models that generalize across diverse agentic tasks. The OpenThoughts-Agent (OT-Agent) project addresses this gap with a fully open data curation pipeline for training agentic models. We conduct more than 100 controlled ablation experiments to systematically investigate each stage of the pipeline, yielding insights on the importance of task sources and diversity. We then assemble a training set of 100K examples from our pipeline and fine-tune Qwen3-32B on this dataset, which yields an average accuracy of 44.8% across seven agentic benchmarks and a 3.9 percentage point improvement over the strongest existing open data agentic model (Nemotron-Terminal-32B, 40.9%). Moreover, our training data exhibits strong scaling properties, outperforming alternative open datasets at every training set size in compute-controlled comparisons. We publicly release our training sets, data pipeline, experimental data, and models at openthoughts.ai to support future open research on agentic model training.

1 Introduction

OpenThoughts-Agent addresses limited public knowledge about curating broadly capable agentic-model training data through an open SFT and RL data-curation effort. Its resulting datasets and models achieve strong multi-benchmark performance, scaling behavior, and reproducibility-oriented openness.

  • Public literature provides little information about training data for state-of-the-art agentic models, while open curation efforts usually target one benchmark.
  • More than 100 controlled ablations investigate the OpenThoughts-Agent SFT data-curation pipeline and identify instruction choice, teacher selection, trace length, and source diversity as important factors.
  • 54.0% on SWE-Bench Verified and 26.2% on Terminal-Bench 2.0 exceed Nemotron-Terminal-32B’s 41.9% and 25.1%, respectively, for the reported benchmarks.
  • Training data outperforms other open datasets at every training-set size in compute-controlled comparisons, indicating strong scaling trends beyond dataset size alone.
  • An SFT-plus-RL 8B pipeline outperforms the best single-stage 8B model and strongest existing <=8B baselines across seven agentic benchmarks.
  • The authors publicly release training sets, the data pipeline, experimental data, and models to support future open research.

2 Related Work

Related work spans agentic data curation, benchmarks, and public agent-building frameworks. The paper positions OpenThoughts-Agent against efforts that typically study isolated training stages or narrow benchmark clusters.

  • Public data-curation research for agents remains scarce despite broader advances in curation for vision-language, language, and reasoning models.
  • Agentic benchmarks evaluate discrete tasks such as GitHub issue resolution, with standardized platforms improving affordability and reproducibility.
  • Prior agentic data-curation works usually focus on either SFT or RL and on a single benchmark or small cluster of related benchmarks.
  • OpenThoughts-Agent instead studies generalization across benchmarks, SFT–RL interaction, multiple model scales, and a broader benchmark range.
  • The project uses a Llama-Factory fork for SFT, an extended SkyRL framework for RL, and public tooling for environments, benchmarks, and harnesses.

3 SFT Data Pipeline

The SFT pipeline independently ablates task sources, mixing, task filtering, teacher models, and rollout filtering, selecting strategies by average performance across three benchmarks. Task-source choice and teacher selection produce especially large differences, while balanced mixing, difficulty-based filtering, and longer traces improve results.

  • Pipeline: The pipeline evaluates each data-curation stage independently and selects strategies using average z-score across three benchmarks.Each ablation generates 10,000 trajectories and fine-tunes Qwen3-8B under fixed training settings.
  • 3.1 Sourcing Tasks: Task-source choice has the largest spread among pipeline stages, with issue-resolution and human-written infrastructure tasks ranking highly but varying across benchmarks.The study evaluates 95 task-generation strategies spanning sources, generation modes, and knowledge domains.
  • 3.2 Mixing Tasks: Mixing the top 4–8 task-generation strategies outperforms the unmixed top-1 baseline by improving performance across benchmarks rather than specializing in SWE-Bench.The mix uses the highest-ranked strategies and samples 10,000 task descriptions from each.
  • 3.2 Mixing Tasks: LLM-based difficulty filtering improves average performance by approximately 3 percentage points, whereas task-description augmentation does not improve the untouched baseline.Selecting tasks requiring longer GPT-5 responses produces roughly a 3-point gain across benchmarks.
  • 3.5 Teacher Model: GPT-5.3-Codex is a worse teacher than GLM-4.7-AWQ despite being the stronger benchmark model, reducing Terminal-Bench 2.0 performance by roughly 5%.Teacher quality therefore does not follow the teacher model’s standalone benchmark strength.
  • Rollout Filtering: Filtering out rollouts with fewer than 5 turns yields the largest improvement among the tested rollout filters, supporting longer agentic traces.Other filters remove timeout or subagent traces.

4 Scaling Up SFT Data

Scaling the SFT dataset requires more task-description diversity rather than simply generating more rollouts from the same tasks. Synthetic augmentation overcomes this bottleneck and supports continued gains through the 100K scale.

  • Scaling Strategies: Scaling dataset size is tested through more rollouts per task, more original task descriptions, synthetic task augmentation, and additional initial sources.The post-ablation pipeline begins with a 10K dataset.
  • Upsampling Rollouts: Performance plateaus from 31.6K to 100K when scaling rollouts per task, with changes of +3pp on SWE-Bench Verified-100 and −2pp on Terminal-Bench 2.0 within standard error.The result suggests task-description diversity is the bottleneck, while the initial Tezos source has only 997 unique task descriptions.
  • Source Expansion: Adding sources beyond Top-4 does not reliably help, and broadening to Top-16 hurts on every benchmark at the 100K scale.The final experiments retain the original Top-4 source mix.
  • Synthetic Augmentation: Synthetic augmentation replaces the limited Tezos task set with rewritten variants, addressing the task-description-diversity bottleneck.The augmentation operates within the four highest-scoring sources and targets Tezos because it contributes the fewest unique tasks.
  • Final 100K Dataset: At 100K, OpenThoughts-Agent-v2 reaches 55.7% on SWE-Bench Verified-100, 41.3% on OT-TBLite, and 26.2% on Terminal-Bench 2.0.Relative to 31.6K, performance improves monotonically by +7.7pp on SWE-Bench Verified-100 and +5.0pp on Terminal-Bench 2.0.

5 Reinforcement Learning

The RL study controls the training pipeline while varying data sources, showing that source choice materially affects agentic RL performance. A complete SFT-plus-RL pipeline outperforms relevant baselines, but RL benefits depend on the SFT starting point.

  • 5.1 Experimental Details: The RL investigation focuses on 8B models to control compute and uses async RLOO training with binary verifier-success rewards.The hero run uses 24 A100 80GB GPUs, batch size 64, and approximately 46 hours of wall-clock time.
  • 5.2 Sourcing Tasks in RL: The RL source ablation varies only the training dataset while holding hyperparameters and evaluation criteria fixed across six sources.The sources span competitive programming, repository bug fixing, code contests, code-oracle data, freelancer tasks, and natural-language-to-Bash tasks.
  • 5.2 Sourcing Tasks in RL: RL source choice spans a 7.6-point range in raw average accuracy, exceeding the 2.0-point run-to-run reproducibility variance.The source effect is smaller than the SFT source-ablation variance reported elsewhere.
  • 5.2 Sourcing Tasks in RL: Pymethods2test is the strongest RL source, using single-function Python contracts with synthesized descriptions and generated unittest suites.Its tasks emphasize code correctness without multi-file editing, repository navigation, or accumulated shell state.
  • 5.2 Sourcing Tasks in RL: Synthetic-and-competitive sources lead on in-distribution benchmarks, while heterogeneous tool-use sources are more competitive out of distribution at some in-distribution cost.The observed ID/OOD decoupling links single-function correctness with core benchmarks and broader tool use with OOD performance.
  • 5.3 Results: The complete SFT + RL pipeline improves average performance by 18 points over the Qwen3-8B base model across seven agentic benchmarks.RL provides the most gains when the SFT model is selected with RL in mind; a weak Qwen3-8B SFT starting point cannot benefit from agentic RL.

6 Conclusion

OpenThoughts-Agent combines controlled SFT ablations with focused RL data curation to improve open agentic-model training across diverse benchmarks. The project releases its data, pipeline, and models, while identifying limits around RL scale, base-model choice, and dataset-size extrapolation.

  • Contributions: OpenThoughts-Agent studies a six-stage SFT curation pipeline through controlled ablations and complements it with a focused investigation of agentic RL data.The project uses the resulting dataset to fine-tune Qwen3-32B and evaluates performance across seven agentic benchmarks.
  • Contributions: At 8B, combining SFT data with the pymethods2test RL dataset outperforms the strongest existing baselines on average across seven agentic benchmarks.The paper presents this as initial evidence that SFT and RL stages can compose.
  • Open release: The paper releases its data, curation pipeline, and models to support broader open research on agentic models.The release is intended to lower barriers to further investigation of agentic-model training.
  • Limitations: RL evaluation is limited to the 8B scale, so transfer of the same recipe to 32B remains an open question.The authors attribute this scope boundary to compute constraints.
  • Limitations: All SFT runs begin from the Qwen3 family, leaving the contribution of base-model pretraining unisolated.The paper also leaves extrapolation beyond its largest 100K-trajectory training set untested.

A.2 Mixing Strategies (Full Results)

The full mixing results show that mixing several top task-generation strategies provides the strongest balanced performance, while longer-episode filtering retains an advantage even when token budgets are matched.

  • Mixing strategies: The full-results table reports both random shuffling within tasks and sequential round-robin presentation across all values of N.The random-shuffle Top-4 result is reproduced in the main text.
  • Mixing strategies: Mixing the top-4 to top-8 task-generation strategies yields the strongest balanced performance under both random-shuffle and sequential round-robin presentation.The sweep evaluates N ∈{1, 2, 4, 8, 16, 32} with 10,000/N tasks sampled from each of the top-N sources.
  • Longer-episode filtering: The min-turns filter outperforms a random subsample at an equal approximately 145M-token budget.Both compared models are Qwen3-8B fine-tuned on subsets with matched token budgets.
  • Longer-episode filtering: +5.4pp on SWE-bench Verified-100 and +3.8pp on Terminal-Bench 2.0 are achieved by selecting episodes with at least five turns.OT-TBLite remains within noise, while the average gain is +3.5pp.
  • Longer-episode filtering: The matched-budget result indicates that the benefit reflects multi-turn supervision rather than additional training compute.The token budget control addresses the fact that longer episodes contain more tokens per example.

B Scaling at 8B

At 8B, OpenThoughts-Agent-v2 improves as the dataset scales from 316 to 100K rows and generally leads the Nemotron-Terminal-Corpus baseline on both evaluated benchmarks. At 100K, it surpasses that baseline on SWE-bench Verified-100 and Terminal-Bench 2.0.

  • Setup: OpenThoughts-Agent-v2 is evaluated at 8B across dataset sizes from 316 to 100K rows on SWE-bench Verified-100 and Terminal-Bench 2.0.The comparison includes the Nemotron-Terminal-Corpus baseline and the base Qwen3-8B model.
  • Scaling results: At 10K rows, OpenThoughts-Agent reaches 24.3% versus 9.3% for Nemotron-Terminal-Corpus on SWE-bench Verified-100.This is an example of OpenThoughts-Agent leading at a matched dataset size.
  • Scaling results: At 100K rows, the 8B model reaches 39.7% on SWE-bench Verified-100 and 10.9% on Terminal-Bench 2.0.These values rise from 26.3% and 7.9%, respectively, at 31.6K rows.
  • Scaling results: At 100K rows, OpenThoughts-Agent surpasses the Nemotron-Terminal-Corpus baseline on both benchmarks.The figure presents accuracy across dataset sizes with standard errors from three stochastic reruns per task.

C.1 SFT training hyperparameters per data scale

The training setup uses fixed shared infrastructure and optimizer settings within each model scale, while data-scale experiments vary selected hyperparameters for 32B runs and hold 8B hyperparameters constant.

  • 32B settings: All 32B SFT runs start from Qwen3-32B and share optimization settings, with only the items in Table 17 varying across data scales.The 32B runs use the qwen3 thinking chat template.
  • 32B settings: Smaller 32B datasets use more epochs and looser gradient clipping, whereas larger datasets use fewer epochs and tighter clipping.The 3.16K and 10K runs use the 32b_small recipe; 31.6K and 100K use 32b_large.
  • 8B settings: All 8B SFT runs use a non-thinking chat template, a seven-epoch horizon, and shared optimizer and infrastructure settings.The 8B per-scale configuration holds all hyperparameters constant while data volume changes.
  • RL settings: The RL configuration uses RLOO with per-prompt standard-deviation normalization, token-mean loss reduction, and a PPO clip range of [0.2, 0.2].The setup also specifies no KL control, entropy regularization, or per-batch advantage normalization.
  • Optimization: Training uses AdamW optimization with bf16 autocasting, fp32 gradient accumulation, global gradient clipping at 1.0, and a 5 × 10^-6 learning rate.The policy and critic use different weight-decay settings.
  • Infrastructure: The distributed system uses FSDP2 with intra-node sharding, CPU parameter offload, asynchronous generation, and vLLM inference engines.Reference and policy models are colocated, with asynchronous generation allowing up to 16 steps of staleness.
  • Evaluation environment: Agent trials use the terminus-2 Harbor environment with 1,800-second agent timeouts, 120-second verifier timeouts, and three retries.Each trial receives 1 vCPU, 2,048 MB RAM, and 2,048 MB storage.

E RL Run-to-Run Reproducibility

Near-replicate RL runs show that the pipeline is reproducible at approximately the 2-point level, with run-to-run variation smaller than the reported gains. The study separates training-run noise from evaluation noise using repeated runs and repeated evaluations.

  • Three near-replicate runs share the same checkpoint, RLOO recipe, environment, and 24×A100 setup, differing only in minor training choices.The headline checkpoint was evaluated twice on every benchmark to distinguish evaluation noise from training-run noise.
  • ≈1.6 points on ID and ≈2.0 points on OOD separate the replicate RL runs.Core means range from 20.2–21.8%, while OOD means range from 26.5–28.5%.
  • ≈5-point RL-specific gains and an ≈18-point full-pipeline gain exceed the ≈1.6/2.0-point ID/OOD run-to-run spread.The authors conclude that these gains are unlikely to be artifacts of training-run noise.
  • The reproducibility analysis compares the strongest RL data source with a baseline to contextualize behavioral changes and downstream ranking.The analysis combines replicate-run evidence with behavioral and evaluation comparisons.

F.1 Legitimate exploration, not reward hacking

The reported RL improvement is accompanied by expanded exploratory behavior rather than evidence of parser-focused reward hacking. The same pipeline can instead produce a more compact policy when trained on a different data source.

  • F.1 Legitimate exploration, not reward hacking: 116% more think tokens, 81% more self-correction, and 31% more tool calls characterize the post-RL exploratory shift.Think tokens rise from 30.3 to 65.4, self-correction phrases from 0.63 to 1.14, and tool calls from 31.3 to 40.9 per trace.
  • F.1 Legitimate exploration, not reward hacking: 18 tasks flip from fail to pass against one regression on the shared SWE-Bench-Verified tasks.The behavioral expansion therefore coincides with held-out evaluation gains.
  • F.1 Legitimate exploration, not reward hacking: The post-RL policy emits more tool errors in absolute terms while its per-call error rate rises only 4.1 percentage points.Mark_task_complete calls also increase, which the authors use to argue against parser gaming or early termination.
  • F.1 Legitimate exploration, not reward hacking: A medium, non-saturated reward around 0.47–0.51 is associated with pressure to think more, call more tools, and attempt more fixes.The same exploration pressure later produces over-extension and reward collapse.
  • F.1 Legitimate exploration, not reward hacking: The alternative RL run moves in the opposite direction, reducing turns, tool calls, think tokens, and self-correction while reward rises from ≈0.54 to ≈0.73.This contrast comes from changing the RL data source while keeping the broader pipeline identical.
  • F.1 Legitimate exploration, not reward hacking: Post-RL Harbor Mean accuracy on SWE-Bench-Verified rises from 0.19 to 0.33 before collapse.This metric counts errored and timed-out trials as zero, and the checkpoint flips 18 tasks from fail to pass.

G Eval Setup

The evaluation uses isolated sandboxes, repeated stochastic trials, multiple agentic benchmarks, and timeout adjustments for model throughput. The paper also documents benchmark coverage, reporting conventions, and comparability limitations.

  • Terminal-Bench 2.0 contains 89 hand-crafted, human-verified tasks spanning software engineering, biology, security, system administration, and machine learning.
  • SWE-Bench-Verified-100 is a 100-task repository-stratified subsample, while SWE-Bench-Verified contains the full 500-task split.Both evaluate patches against Python repositories at fixed commits.
  • Figure 6 shows hero-run reward peaking near 0.51 before collapsing to approximately 0.13 as exploration increases and productive tool calls fall.
  • Figure 7 shows the baseline reward rising from approximately 0.54 to 0.73 without collapse as the policy compacts.
  • The benchmark suite additionally includes Aider Polyglot, BFCL-Parity, OpenThoughts-TBLite, and other agentic evaluations.Aider Polyglot covers six programming languages, while OpenThoughts-TBLite is a 100-task proxy for Terminal-Bench 2.0.
  • 3 repeated runs per model–benchmark–harness combination provide average pass@1 accuracy and standard error.Trained models use a 32K context window, 16K output cap, and proactive summarization.
  • Timeout multiplier m=2 compensates for slower decoding in larger models, followed by a two-hour execution cap.Without adjustment, fixed timeouts confound accuracy across model sizes.
  • OpenSWE is excluded from the main comparison because its reported 62.4% uses the full 500-task split, whereas the reproduction measures 44.0% on a 100-task subset.The authors also note unreleased scaffold and evaluation-harness components prevent guaranteeing an exact match.
Loading 2606.24855v1…