PaperlayerView exact revision on arXiv

Source-linked AI summary

On Data Engineering for Scaling LLM Terminal Capabilities

Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, Wei Ping

arXiv:2602.21193v1cs.CL

TL;DR

Terminal-agent training faces data scarcity, while effective data curation must balance breadth, depth, and scalability. The paper combines dataset adaptation with targeted synthetic task generation and reports that Nemotron-Terminal-32B scores 27.4±2.4 versus 23.9±2.8 for Qwen3-Coder-480B on Terminal-Bench 2.0.

  • Problem

    Terminal-agent training faces a data-scarcity bottleneck, and data curation must balance breadth, depth, and scalability.

    Source: §4 · p. 4, §6 · p. 11

  • Method

    The paper combines dataset adaptation for broad foundational coverage with synthetic task generation for targeted skill refinement in a two-stage SFT data framework.

    Source: §4 · p. 4

  • Results

    27.4±2.4 vs. 23.9±2.8: Nemotron-Terminal-32B outperforms Qwen3-Coder-480B on Terminal-Bench 2.0.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that high-quality, diverse trajectories are more pivotal than sheer parameter scale for terminal-agent performance.

    Source: §6 · p. 11

    Generated tasks rely on solution isolation: prompts must exclude solution approaches, while reference solutions serve only to derive test expectations.

    Source: §4.2.3 · p. 7

Abstract

from arXiv · show

Despite rapid recent progress in the terminal capabilities of large language models, the training data strategies behind state-of-the-art terminal agents remain largely undisclosed. We address this gap through a systematic study of data engineering practices for terminal agents, making two key contributions: (1) Terminal-Task-Gen, a lightweight synthetic task generation pipeline that supports seed-based and skill-based task construction, and (2) a comprehensive analysis of data and training strategies, including filtering, curriculum learning, long context training, and scaling behavior. Our pipeline yields Terminal-Corpus, a large-scale open-source dataset for terminal tasks. Using this dataset, we train Nemotron-Terminal, a family of models initialized from Qwen3(8B, 14B, 32B) that achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B improves from 2.5% to 13.0% Nemotron-Terminal-14B improves from 4.0% to 20.2%, and Nemotron-Terminal-32B improves from 3.4% to 27.4%, matching the performance of significantly larger models. To accelerate research in this domain, we open-source our model checkpoints and most of our synthetic datasets at https://huggingface.co/collections/nvidia/nemotron-terminal.

1. Introduction

The introduction identifies a lack of practical, scalable terminal-agent training data and presents a coarse-to-fine approach combining dataset adaptation with synthetic task generation. It reports that Nemotron-Terminal models improve on their Qwen3 baselines, with the 32B model exceeding Qwen3-Coder-480B on Terminal-Bench 2.0.

1. Introduction

  • The introduction identifies undisclosed training mixtures, scarce task and environment resources, and costly multi-turn trajectory collection as barriers to effective terminal-agent data engineering.It also notes that dataset adapters may inherit assumptions unsuited to sequential interaction, while multi-agent generation frameworks add complexity that scales poorly.Source: §1 · p. 2
  • Dataset adaptation scales terminal-task data from existing math, code, and software-engineering datasets, while synthetic generation targets specific skills, difficulty levels, and capability gaps.The introduction frames these strategies as complementary: adaptation builds foundational capabilities at scale, and synthetic tasks provide finer-grained control.Source: §1 · p. 2
  • Terminal-Task-Gen enables scalable synthetic task generation, rapid exploration of task design, and targeted tasks for specific skills at different difficulty levels.The pipeline is part of the paper’s proposed response to costly trajectory collection and scarce foundational resources.Source: §1 · p. 2
  • The paper systematically studies adapter and synthetic-task filtering, curriculum learning for data mixing, long-context training, and scaling trends.It introduces Nemotron-Terminal models fine-tuned from Qwen3 models and releases the models and Terminal-Corpus dataset.Source: §1 · pp. 2–3
  • 27.4±2.4 vs. 23.9±2.8: Nemotron-Terminal-32B outperforms Qwen3-Coder-480B on Terminal-Bench 2.0.The paper also reports substantial improvements over the initial Qwen3 baselines and competitive performance with significantly larger models.Source: Table 1 · p. 2, §1 · p. 3

2. Related Work

Related work spans agent scaffolding, terminal datasets built from existing prompts, and synthetic instruction generation. This paper positions its contribution as simplifying terminal-task generation while systematically studying data and training strategies.

Agent Design

  • Leading terminal agents illustrate how scaffolding can improve performance, but scaffolds are often model-specific and engineering-intensive; this work instead targets model capabilities through supervised fine-tuning.The authors expect complex scaffolding's marginal benefit to decrease as base models improve.Source: §2 · p. 3

Dataset Adapters

  • Dataset adapters reuse prompts from existing datasets to collect terminal execution traces, but their effects on downstream training had not been formally analyzed.The authors study adapter strengths and weaknesses using custom datasets and adapters.Source: §2 · p. 3

Synthetic Task Generation

Terminal Task Generation

  • Prior work uses multi-agent systems for terminal-task and Docker-environment generation and validation; this work simplifies coordination and optimizes validation.The authors say their systematic study and ablations provide actionable insights for scaling terminal-capable models.Source: §2 · p. 3

3. Background

Terminal-Bench evaluates end-to-end terminal workflows in verified, containerized tasks, while Terminus 2 provides a model-agnostic agent framework for evaluation. The background establishes the benchmark’s task structure and the agent’s interactive command-line interface.

3.1. Terminal-Bench

  • Terminal-Bench comprises 89 human-verified tasks across six domains, requiring end-to-end workflows rather than isolated function generation.The domains include scientific computing, software engineering, machine learning, security, system administration, and data science.Source: §3.1 · p. 3
  • Each Terminal-Bench task pairs a natural-language objective and containerized environment with verification tests and an oracle solution; this work evaluates with Terminal-Bench 2.0 and Terminus 2.Terminus 2 is the benchmark’s model-agnostic reference agent for consistent checkpoint evaluation.Source: §3.1 · p. 4

3.2. Terminus 2 Agent Framework

  • Terminus 2 (Team, 2025) acts through keystrokes in a sandboxed Docker container’s interactive tmux session, receiving terminal output and returning structured JSON actions.The response format includes analysis, plan, commands, and task_complete.Source: §3.2 · p. 4, Figure 3 · p. 4

4. Synthetic Data Generation · 4.1. Dataset Adapters · 4.2. Synthetic Task Generation

The paper uses a coarse-to-fine data pipeline: adapters turn existing prompts into broad terminal-task coverage, while synthetic generation creates executable tasks with targeted skills, difficulty, and environment constraints.

4. Synthetic Data Generation

  • The two-stage framework separates broad foundational coverage through dataset adaptation from targeted skill refinement through synthetic task generation.This decouples data-volume scaling from task-design iteration while aiming to cover operational and computational terminal skills.Source: §4 · p. 4

4.1.1. Prompt Datasets

  • Prompt sources span math, code, and SWE; filtering and deduplication yield 35K code prompts and 32K unique SWE prompts.The math set contains 163K unique prompts; code begins with 79K prompts, while the SWE source comprises 127K instances. The math prompts exclude questions with DeepSeek-R1 responses shorter than 2K tokens.Source: §4.1.1 · pp. 4–5

4.1.2. Adapter Format

  • Adapters map each source prompt into the Terminus 2 instruction template, add a dataset-specific suffix, and instantiate SWE code files in the environment.Because the source datasets provide prompts only, adapted tasks have instructions and environments but no test cases.Source: §4.1.2 · p. 5

4.2. Synthetic Task Generation

  • Terminal-Task-Gen supplements source-limited adapters with executable tasks built from structured seeds and primitive skills, controlling skill complexity and environment constraints.Both LLM-based methods target diverse tasks that reflect terminal operations and multi-step tool interactions.Source: §4.2 · p. 5

4.2.1. Generation from Seed Data

  • Seed-based generation uses existing problems as inspiration for new terminal tasks, rather than wrapping source prompts in a terminal scaffold.Seeds can include a problem description, optional domain label, and optional reference solution; any reference solution guides test expectations but is not shown to the agent.Source: §4.2.1 · pp. 5–6
  • The seed adapter adds software requirements, realistic input files, and pytest checks for output existence, format, numerical accuracy, and edge cases.Conversion guidelines decompose complex problems when needed, specify practical input and precision constraints, and make outputs programmatically verifiable.Source: §4.2.1 · p. 6

4.2.2. Generation from Primitive Skills

  • Skill-based generation recombines primitive terminal skills into novel tasks, typically integrating 3–5 skills per task across nine domain-specific prompt categories.The skill dimensions include algorithmic, systems, data processing, mathematical, testing, and web/security capabilities.Source: §4.2.2 · p. 6

4.2.3. Task Format and Execution Environment

  • Generated tasks combine a prompt, weighted pytest cases, input files, and a domain-specific Docker environment, while keeping solution information out of the agent-visible prompt.The pipeline omits oracle solutions because generating ground-truth code without human verification is difficult; instead it uses tests to check solutions. Pre-built domain images use 9 shared base images, avoiding per-task Dockerfile validation and decoupling task from environment generation.Source: §4.2.3 · p. 7

4.3. Teacher Model · 4.4. Data Filtering

The sections select DeepSeek-V3.2 as the teacher model and describe filtering the SFT data for contamination and quality, alongside experiments on trajectory completeness and test success.

4.3. Teacher Model

  • DeepSeek-V3.2 (Liu et al., 2025) is selected to generate synthetic tasks and trajectories because of its strong Terminal-Bench 2.0 performance.The paper also evaluates its suitability for producing dataset-adapter trajectories using Terminus 2.Source: §4.3 · p. 7
  • The teacher is evaluated in Terminal-Bench format on AIME 2024 and AIME 2025.These are among the standard benchmarks adapted using the Terminus 2 agent framework.Source: §4.3 · p. 7
  • The Terminal-Bench-format teacher evaluation also includes LiveCodeBench v6 (Jain et al., 2024) and SWE-bench Verified (Jimenez et al., 2023).The evaluation uses the Terminus 2 agent framework.Source: §4.3 · p. 7

4.4. Data Filtering

  • The SFT dataset is decontaminated by removing prompts with a 14-gram overlap with Terminal-Bench 2.0 test samples.Source: §4.4 · p. 8
  • Quality filters remove identity leaks and responses containing Chinese characters.Source: §4.4 · p. 8
  • The authors experiment with removing incomplete teacher-generated trajectories to discourage overly verbose fine-tuned models.Source: §4.4 · p. 8
  • When tests are available, they also experiment with retaining only trajectories that pass them.Source: §4.4 · p. 8

5. Experiments

Experiments show that Nemotron-Terminal improves Terminal-Bench 2.0 performance, while ablations examine data composition, filtering, context length, curriculum, and training-data scale.

5.1. Experimental Setup

  • Experiments use Qwen3-8B for ablations and Qwen3-14B and 32B to test scaling.Unless otherwise specified, training uses a 32,768-token maximum sequence length and two epochs; 8B and 14B run on 32 GPUs, while 32B runs on 128 GPUs.Source: §5.1 · p. 8

5.2. Main Results

  • 13.0 ± 2.2 vs 2.47±0.5 for Nemotron-Terminal-8B and Qwen3-8B on TB2.0; Nemotron-Terminal-32B scores 27.4 ± 2.4.Nemotron-Terminal-14B scores 20.2 ± 2.7, exceeding 120B GPT-OSS (18.7 ± 2.7) and Gemini 2.5 Flash (16.9 ± 2.4); 32B exceeds 480B Qwen3-Coder (23.9 ± 2.8).Source: Table 3 · p. 8, §5.2 · p. 8
  • For Qwen3-14B and 32B, Nemotron-Terminal-32B raises Data Querying from 0.0 to 60.0 and Model Training from 0.0 to 50.0.The 32B model also improves Security (2.5 to 27.5), Data Processing (5.0 to 50.0), Software Engineering (5.0 to 31.7), System Administration (6.7 to 31.1), and Debugging (0.0 to 33.3).Source: Table 4 · p. 9, §5.2 · p. 9

5.3. Ablation on Dataset Components

  • Mixed dataset adapters reach 9.66%, while skill-based synthetic data drives 12.4%; adding seed-based data reduces variance without raising the mean.Among adapter subsets, Math scores 5.39%, Code 6.29%, and SWE 7.02%; combining them outperforms any single source.Source: §5.3 · p. 9, Table 5 · p. 10

5.4. Filtering Strategies

  • 12.4% with no synthetic-task filtering exceeds complete-only (6.74%) and success-only (5.06%); no filtering also yields the highest adapter full-set score, 9.66%.Strict filtering discards over half the available training data, while unsuccessful trajectories can teach realistic error states and recovery patterns.Source: §5.4 · pp. 9–10, Table 6 · p. 10, Table 7 · p. 10

5.5. Long Context Training and Evaluation

  • Extending Qwen3-8B context to 65,536 tokens yields no significant gain; evaluation with a 40,960-token context window performs better than the tested alternatives.Most high-quality supervision fits within the default 32,768-token window, while longer-tail trajectories tend to be noisy and less informative.Source: Table 8 · p. 10, §5.5 · p. 10

5.6. Curriculum Learning

  • Single-stage mixed training performs at least as well as the two-stage dataset-adapter-then-synthetic curriculum, so the authors use mixed training thereafter.The compared strategies train on all datasets concurrently or in two sequential stages.Source: §5.6 · pp. 10–11, Table 9 · p. 11

5.7. Scaling Experiments

  • As synthetic training data increases from 0% to 100%, Qwen3-8B and Qwen3-14B both improve; 14B has higher absolute performance and larger gains.The results indicate that both model capacity and training-data scale matter for performance.Source: Figure 4 · p. 11, §5.7 · p. 11

6. Conclusion

The paper addresses data scarcity in terminal-agent training with Terminal-Task-Gen and concludes that precise data engineering enables Nemotron-Terminal to outperform its Qwen3 base and rival larger frontier models on Terminal-Bench 2.0. It also identifies reinforcement learning and released models and datasets as directions and resources for future research.

6. Conclusion

  • Terminal-Task-Gen combines large-scale dataset adaptation with targeted synthetic task generation to address data scarcity in terminal-agent training.The authors conclude that high-quality, diverse trajectories matter more than sheer parameter scale.Source: §6 · p. 11
  • Precise data engineering enables Nemotron-Terminal to significantly outperform its Qwen3 base and rival larger frontier models on Terminal-Bench 2.0.The paper frames these results as evidence for the importance of high-quality, diverse trajectories over sheer parameter scale.Source: §6 · p. 11
  • The authors propose extending the work with reinforcement learning that uses verifiable execution feedback for self-correction and optimal planning on long-horizon tasks.Source: §6 · p. 11
  • The authors release their models and most synthetic datasets, including adapter and skill-based task subsets, to support research in autonomous terminal agents.Source: §6 · p. 11

A. Appendix

The appendix includes a summary of skills organized by domain.

A.1. Synthetic Trajectory Analysis

The section analyzes trajectory lengths by token count and turn count, separating synthetic tasks from dataset adapters.

A.1. Synthetic Trajectory Analysis

  • The analysis measures token- and turn-count distributions separately for synthetic tasks and dataset adapters; Figure 5 shows token counts in generated trajectories.Source: §A.1 · p. 15, Figure 5 · p. 15

A.2. Details for Trajectory Generation

Trajectory generation uses Terminus 2 with prompts tailored to synthetic tasks or dataset adapters, while the current shell state is supplied as terminal output.

A.2. Details for Trajectory Generation

  • Terminus 2 generates all SFT trajectories, using explicit task instructions or adapter-specific prompts and the latest terminal output to populate the prompt’s terminal_state placeholder.The full system prompt template is provided in Figure 7; dataset-adapter suffixes are shown in Figures 8–10.Source: §A.2 · p. 15

A.3. Details for Synthetic Task Generation

The section describes broad primitive-skill coverage for terminal tasks and a modular prompting strategy that combines general task logic with domain-specific requirements.

Curation of Primitive Skills

  • Primitive skills span terminal domains from filesystem manipulation to algorithmic reasoning, covering security, software engineering, file operations, administration, data science, scientific computing, and data processing.Examples include exploit-payload construction, dependency-resolution graph traversals, structured-format parsing, permission management, and statistical transformations.Source: §A.3 · p. 16

Prompts for Synthetic Task Generation

  • Synthetic task prompts combine a system template for general task logic with domain-specific modules that specify each vertical’s preconditions and objectives.Examples include security requirements for crafting exploit payloads and data-processing requirements for transformation pipelines with interpolation.Source: §A.3 · p. 16, Figure 11 · p. 19

Related papers

Loading 2602.21193v1…