Source-linked AI summary
On Data Engineering for Scaling LLM Terminal Capabilities
Renjie Pi, Grace Lam, Mohammad Shoeybi, Pooya Jannaty, Bryan Catanzaro, Wei Ping
TL;DR
Terminal-agent training strategies are largely undisclosed, limiting systematic understanding of how data engineering affects terminal capability. The paper introduces a scalable adaptation and synthetic-generation framework, then trains Nemotron-Terminal models that substantially improve over Qwen3 baselines and rival larger systems. It also releases models and synthetic datasets to support further research.
Problem
Training-data strategies for state-of-the-art terminal agents remain largely undisclosed, motivating a systematic study of terminal-agent data engineering.
Method
The paper combines dataset adaptation with seed-based and skill-based synthetic task generation, while analyzing filtering, curriculum, long-context training, and scaling.
Results
Nemotron-Terminal-32B achieves 27.4±2.4 on Terminal-Bench 2.0, outperforming Qwen3-Coder-480B at 23.9±2.8.
Takeaways & Limitations
The study concludes that high-quality, diverse trajectories are more pivotal for terminal capability than sheer parameter scale.
Takeaways & Limitations
Generated tasks isolate solution information and do not expose oracle solutions to agents, relying on synthesized tests for evaluation.
Abstract
from arXiv · showhide
Despite rapid recent progress in the terminal capabilities of large language models, the training data strategies behind state-of-the-art terminal agents remain largely undisclosed. We address this gap through a systematic study of data engineering practices for terminal agents, making two key contributions: (1) Terminal-Task-Gen, a lightweight synthetic task generation pipeline that supports seed-based and skill-based task construction, and (2) a comprehensive analysis of data and training strategies, including filtering, curriculum learning, long context training, and scaling behavior. Our pipeline yields Terminal-Corpus, a large-scale open-source dataset for terminal tasks. Using this dataset, we train Nemotron-Terminal, a family of models initialized from Qwen3(8B, 14B, 32B) that achieve substantial gains on Terminal-Bench 2.0: Nemotron-Terminal-8B improves from 2.5% to 13.0% Nemotron-Terminal-14B improves from 4.0% to 20.2%, and Nemotron-Terminal-32B improves from 3.4% to 27.4%, matching the performance of significantly larger models. To accelerate research in this domain, we open-source our model checkpoints and most of our synthetic datasets at https://huggingface.co/collections/nvidia/nemotron-terminal.
1. Introduction
Terminal interaction is becoming important for practical software engineering, but the data strategies behind strong terminal agents remain underexplored. The paper studies these strategies and reports substantial Terminal-Bench gains from data engineering.
- Terminal interaction has emerged as a critical capability for practical software engineering applications.
- Nemotron-Terminal-32B reaches 27.4±2.4 on Terminal-Bench 2.0, exceeding Qwen3-Coder-480B at 23.9±2.8.
2. Related Work
Prior work scales terminal data through agent scaffolding, dataset adaptation, and synthetic generation, but these approaches leave efficiency and systematic analysis gaps. This paper simplifies generation and studies data strategies through systematic experiments and ablations.
- Sophisticated agent scaffolds can improve terminal-agent performance but are often model-specific and require extensive engineering.
- Dataset adapters reuse prompts from existing domains to collect terminal execution traces, but their effects on downstream training have lacked formal analysis.
- Synthetic-data research has developed iterative evolution, seed-based, taxonomy-driven, and seed-free instruction-generation methods.
- The paper removes unnecessary multi-agent coordination and optimizes environment validation to make terminal-task scaling more efficient.
3. Background
Terminal-Bench evaluates end-to-end agent workflows in containerized environments using programmatic verification. Terminus 2 provides a standardized interactive tmux interface and structured JSON actions for consistent evaluation.
- Terminal-Bench contains 89 human-verified tasks spanning scientific computing, software engineering, machine learning, security, system administration, and data science.
- Each benchmark task combines an instruction, Docker environment, verification suite, and oracle solution.
- Terminus 2 gives agents an interactive tmux session inside a sandboxed Docker container rather than multiple specialized tools.
- At each step, terminal output is converted into a structured JSON response specifying the next environment action.
4. Synthetic Data Generation
The paper uses a two-stage generation framework: dataset adaptation supplies broad coverage, while synthetic task generation targets specific terminal skills. Tasks are produced from seeds or primitive skills and executed in standardized, scalable environments with solution information isolated.
- Framework: The two-stage framework combines broad dataset adaptation with targeted synthetic task generation for skill refinement.
- Dataset Adaptation: Dataset adaptation converts math, code, and software-engineering prompts into Terminal-Bench environments without an LLM in the loop.
- Seed-Based Generation: Seed-based generation transforms structured problem records, optionally including domains and reference solutions, into terminal tasks.
- Seed-Based Generation: The adapter adds installation, file-based inputs and outputs, generated test data, practical constraints, and programmatically verifiable formats.
- Skill-Based Generation: Skill-based generation recombines primitive terminal skills across nine domains, including data processing, debugging, security, and software engineering.
- Task Format and Environment: Generated tasks prevent solution leakage, using reference solutions only for test expectations so agents must solve rather than retrieve problems.
- Task Format and Environment: Nine shared pre-built Docker images reduce validation overhead and resource use while separating environment generation from task generation.
4.3. Teacher Model
The study uses DeepSeek-V3.2 as its teacher model and applies decontamination and quality filtering to generated training data.
- DeepSeek-V3.2 is selected as the teacher model for generating synthetic tasks and trajectories because of its Terminal-Bench 2.0 performance.The authors additionally validate it on AIME 2024, AIME 2025, LiveCodeBench v6, and SWE-bench Verified in the Terminus 2 framework.
- The SFT dataset is decontaminated by removing prompts with 14-gram overlap against Terminal-Bench 2.0 test samples.Additional filters remove identity leaks and responses containing Chinese characters.
- The study also tests filtering incomplete trajectories and retaining only trajectories that pass available tests.These filtering choices are evaluated as alternatives for controlling trajectory quality and verbosity.
5. Experiments
Experiments evaluate Qwen3-based Nemotron-Terminal models on Terminal-Bench 2.0 and analyze data sources, filtering, context length, curriculum, and training-data scaling. The models substantially improve over their bases and compete with much larger systems, while several stricter training strategies provide no advantage.
- Main Results: 27.4 ± 2.4 is the Terminal-Bench 2.0 score of Nemotron-Terminal-32B, exceeding Qwen3-Coder-480B at 23.9 ± 2.8.Nemotron-Terminal-8B and 14B score 13.0 ± 2.2 and 20.2 ± 2.7, respectively.
- Main Results: 13.0 ± 2.2 is Nemotron-Terminal-8B’s score, a five-fold increase over Qwen3-8B at 2.47±0.5.
- Main Results: 60.0 and 50.0 are Nemotron-Terminal-32B’s scores in Data Querying and Model Training, where Qwen3-14B and 32B both scored 0.0.The 32B model also improves in Security, Data Processing, Software Engineering, System Administration, and Debugging.
- Ablation on Dataset Components: 9.66% is the combined dataset-adapter score, exceeding Math at 5.39%, Code at 6.29%, and SWE at 7.02%.For synthetic tasks, skill-based data drives the primary gains at 12.4%, while seed-based data reduces variance.
- Filtering Strategies: 12.4% with no synthetic-trajectory filtering exceeds complete-only at 6.74% and success-only at 5.06%.The authors suggest unsuccessful trajectories provide supervision about error states and recovery patterns.
- Curriculum and Context: Mixed single-stage training yields no performance disadvantage relative to the two-stage curriculum, so it is adopted thereafter.Training and evaluation with extended 65,536-token context also shows no significant advantage, while 40,960 tokens performs more strongly.
- Scaling Experiments: Performance consistently increases as synthetic training data rises from 0% to 100% for Qwen3-8B and Qwen3-14B.The 14B model achieves higher absolute performance and larger gains from additional data.
6. Conclusion
The paper concludes that scalable data engineering can improve efficient terminal models and support competition with larger frontier systems. It releases models and most synthetic datasets while identifying reinforcement learning as future work.
- Terminal-Task-Gen addresses terminal-agent data scarcity by combining large-scale dataset adaptation with targeted synthetic task generation.
- Precise data engineering enables Nemotron-Terminal models to outperform their Qwen3 bases and rival larger frontier models on Terminal-Bench 2.0.The conclusion emphasizes high-quality, diverse trajectories over sheer parameter scale.
- The authors release their models and most synthetic datasets, including adapter and skill-based subsets, to support research in autonomous terminal agents.They identify reinforcement learning with verifiable execution feedback as a potential extension.
A. Appendix
The appendix includes a skills summary organized by domain.
- Table 10 summarizes skills by domain for terminal interaction tasks.
A.1. Synthetic Trajectory Analysis
The analysis characterizes generated trajectories by token and turn counts, separating synthetic tasks from dataset-adapter tasks.
- The pipeline measures trajectory length using both token counts and interaction turns.These distributions are reported separately for synthetic tasks and dataset adapters.
- Synthetic tasks and dataset adapters are treated as separate sources when analyzing trajectory distributions.
- Figure 5 reports the distribution of the number of tokens in generated trajectories.
A.2. Details for Trajectory Generation
Trajectory Generation uses Terminus 2 with task-specific prompt insertion and terminal outputs supplied through the terminal-state placeholder.
- Terminus 2 generates all trajectories used for supervised fine-tuning.Its system prompt includes instruction and terminal_state placeholders.
- Explicit instructions from Terminal-Task-Gen replace the instruction placeholder for generated tasks.
- The analysis reports the distribution of interaction turns in generated trajectories in Figure 6.
A.3. Details for Synthetic Task Generation
Synthetic task generation organizes terminal-task construction around primitive skills and modular prompts spanning multiple technical domains.
- Primitive Skills: Primitive skills span domains including security, software engineering, data processing, querying, data science, debugging, file operations, scientific computing, and system administration.Security examples include crafting exploit payloads to bypass authentication, while software engineering combines algorithmic and systems skills.
- Prompt Construction: A modular prompting strategy combines a structural backbone with domain-specific requirement modules defining each task’s preconditions and objectives.The domain modules are injected into a common system-prompt template.
- Domain Coverage: The task-generation materials provide dedicated modules and builders for data processing, data querying, data science, debugging, file operations, scientific computing, security, software engineering, and system administration.