PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal-World: Scaling Terminal-Agent Environments via Agent Skills

Zihao Cheng, Hongru Wang, Zeming Liu, Xinyi Wang, Xiangrong Zhu, Yuhang Guo, Wei Lin, Jeff Z. Pan, Yunhong Wang

arXiv:2605.20876v1cs.CLcs.AI

TL;DR

Terminal agents lack scalable, high-quality training data because existing synthesis methods rely on partial sources and can produce narrow tasks, misaligned environments, and inefficient trajectories. Terminal-World uses agent skills to co-derive task instructions, executable environments, and teacher trajectories, then scales synthesis through skill teams and graphs. Across 6 benchmarks, its model family consistently outperforms terminal-agent baselines, including a +4.5 Pass@1 gain for Terminal-World-32B over Nemotron-Terminal-32B on Terminal-Bench 2.0 with 1.2% of the training data.

  • Problem

    Terminal-agent training data is scarce and expensive to curate because each example jointly specifies a task instruction, executable environment, and high-quality trajectory; existing methods also have narrow tasks, misaligned environments, and inefficient trajectories.

  • Method

    Terminal-World uses agent skills to jointly drive task instruction synthesis, environment construction, and teacher trajectory collection, extending skills into teams and graphs for broader synthesis.

  • Results

    Across 6 benchmarks, Terminal-World consistently outperforms terminal-agent baselines; with 1.2% of the training data, Terminal-World-32B surpasses Nemotron-Terminal-32B by +4.5 Pass@1 (31.5) on Terminal-Bench 2.0 and achieves 43.8 Pass@3.

  • Takeaways & Limitations

    The findings demonstrate the effectiveness of skill-grounded synthesis and suggest a practical path toward building more capable terminal agents.

  • Takeaways & Limitations

    Existing synthesis methods remain constrained by seed-derived task distributions, post-hoc environment construction, and teacher trajectories produced without explicit procedural guidance.

Abstract

from arXiv · show

Terminal agents extend Large Language Models with the ability to execute tasks directly in command-line environments, but their progress is bottlenecked by the scarcity of high-quality training data. Existing approaches bootstrap from partial sources such as human-defined seeds or GitHub repositories to instantiate one component and then complete the rest, producing tasks confined to narrow seed distributions, environments misaligned with task semantics, and inefficient trajectories from unguided exploration. To address these limitations, we introduce Terminal-World, a fully automated pipeline that uses agent skills as the central synthesis primitive, which jointly encode what to accomplish, when to apply (preconditions and environment state), and how to execute, enabling task instructions, environments, and teacher trajectories to be co-derived. To further broaden the synthesis space, Terminal-World composes skills into skill teams and skill graphs for multi-role and cross-domain task synthesis. Using this pipeline, we construct 5,723 training environments and train Terminal-World-8B/14B/32B, evaluated across 6 benchmarks where the Terminal-World series consistently outperforms terminal-agent baselines. Notably, using the same teacher model and only 1.2% of the training data, Terminal-World-32B surpasses Nemotron-Terminal-32B on Terminal-Bench 2.0 by +4.5 Pass@1 (31.5) and achieves 43.8 Pass@3.

1 Introduction

Terminal-World addresses the scarcity and high cost of terminal-agent training data by using agent skills to co-specify tasks, environments, and execution procedures. Its automated pipeline scales this synthesis through skill compositions and produces models that outperform existing terminal-agent baselines.

  • Terminal-agent training data is difficult to scale because each example jointly requires an instruction, executable environment, and high-quality multi-turn trajectory.
  • Existing synthesis methods produce narrow task distributions, misaligned environments, and inefficient trajectories from unguided exploration.
  • Agent skills encode what to accomplish, when to apply a workflow, and how to execute it, aligning task semantics, environmental constraints, and procedures.
  • Terminal-World composes individual skills into skill teams and graphs for more complex multi-role and cross-domain task synthesis.
  • 5,723 training environments were constructed, and Terminal-World-8B/14B/32B were trained on this data.
  • Across 6 benchmarks, Terminal-World outperforms existing terminal-agent baselines; Terminal-World-32B reaches +4.5 Pass@1 (31.5) against Nemotron-Terminal-32B on Terminal-Bench 2.0 and 43.8 Pass@3.

2 Related Work

Related work has developed tool-using agents and terminal-based agents by expanding tool coverage, interaction complexity, and synthesized terminal-agent training data. Terminal-agent datasets commonly begin with human-defined seeds before constructing tasks, environments, verification scripts, and teacher trajectories.

  • Tool-using-agent research expanded from API selection and argument filling toward broader tool coverage, longer interactions, and more complex orchestration.
  • Terminal-based-agent work synthesizes training data for direct CLI interaction, with existing systems starting from human-defined seeds and generating tasks, environments, verification scripts, and teacher trajectories.

3 Terminal-World

Terminal-World uses agent skills to co-derive task specifications, executable environments, and teacher trajectories in a fully automated synthesis pipeline. It scales task diversity through personas, skill teams, and skill graphs while validating artifacts and trajectories.

  • Task Generation: Skill teams compose skills within subcategories for multi-role workflows, while skill graphs connect skills across subcategories into end-to-end pipelines.
  • Agent Skill Collection: Agent skills encode what to accomplish, when to apply a capability, and how to execute it, providing anchors for task instructions, execution contexts, and guidelines.
  • Task Generation: The pipeline pairs skills with user personas to synthesize instructions, environment blueprints, evaluation criteria, and execution guidelines, retaining only high-quality task specifications.The generated specification is represented as (I, E, V, G), where V supports verifiable completion and G guides trajectory collection.
  • Environment Building: Environment building instantiates each blueprint as initial files, setup scripts, and pytest verifiers through generate-verify-repair cycles, discarding artifacts unrepaired after 3 iterations.The resulting sandbox is executable and automatically checked for task completion.
  • Trajectory Collection: Skill-derived guidance steers teacher actions instead of free exploration, and the guidance is removed before SFT so students learn from verified terminal interactions.Successful and failed trajectories are retained and annotated with verifier outcomes.
  • Dataset Statistics: 5,723 tasks are synthesized from 1,000 single skills, 76 skill teams, and 237 skill graphs, with sandboxes averaging 2.25 initial files and 4.27 pytest tests across 104 file types.

4 Experiments

Terminal-World is evaluated against terminal-agent baselines on six benchmarks spanning coding, mathematics, table analysis, and SQL data analysis. Its models show strong performance and sample efficiency, with the largest reported gains at 32B scale.

  • Experiment Setting: The evaluation covers six benchmarks: Terminal-Bench 2.0, AIME24, AIME25, DABench, TableBench, and BIRD.These measure terminal coding, mathematical reasoning, table-based analysis, and SQL-based data analysis.
  • Main Results: Terminal-World-32B achieves 69.3 Avg. Pass@1 and 77.9 Avg. Pass@3 across benchmarks, using 5.7K training trajectories.
  • Main Results: 31.5 Pass@1 and 43.8 Pass@3 on Terminal-Bench 2.0 place Terminal-World-32B above all open-source models below 100B parameters.
  • Generalization: Terminal-World-32B matches or exceeds Nemotron-Terminal-32B on all five non-terminal benchmarks without auxiliary domain-specific training data.Nemotron-Terminal supplements terminal data with 226.3K auxiliary samples, whereas Terminal-World uses 5.7K terminal-style trajectories.
  • Scale Effects: +4.5 Pass@1 and +6.7 Pass@3 are the largest margins over Nemotron-Terminal-32B on Terminal-Bench 2.0, occurring at the 32B scale.

5 Analysis

Terminal-World exhibits efficient and reliable execution, benefits from careful SFT data construction, and broadens task diversity through persona grounding at low generation cost.

  • RQ1: Behavior Analysis: Terminal-World-32B uses 10.2 steps and 40.3 commands per task on average, with a lower command failure rate than its comparison model.On the same correctly solved task intersection, it also outperforms DeepSeek-V3.2 across all four behavioral metrics.
  • RQ1: Behavior Analysis: Skill-guided trajectories yield more concise and goal-directed execution strategies, even without guideline assistance.Terminal-World-32B outperforms its teacher model across Avg Steps, Command Error Rate, Commands / Step, and Avg Total Commands.
  • RQ2: Impact of SFT Data Strategies: Penalizing complete failure trajectories suppresses correct intermediate behaviors because unsuccessful trajectories still contain many correct steps.The ablation compares retaining guidelines, reducing data, training only on successful trajectories, and suppressing failure trajectories with negative SFT loss.
  • RQ3: Task Diversity and Cost: Persona grounding substantially improves scenario coverage in independently generated task sets.The diversity study compares skills without personas against skills paired with five or ten personas.
  • RQ3: Task Diversity and Cost: 83.1% of 6,884 accepted tasks become valid executable environments at a total cost of $999.59, or $0.17 per trajectory.The pipeline produces 5,723 valid environments using automated construction.

6 Conclusion

Terminal-World uses skill-grounded synthesis to construct terminal-agent training data and expands coverage through skill taxonomies, teams, and graphs. The resulting models consistently outperform existing terminal-agent baselines across six benchmarks.

  • Conclusion: Terminal-World jointly constructs task instructions, executable environments, and teacher trajectories from agent skills.The pipeline further extends individual skills into skill teams and skill graphs to scale synthesis coverage.
  • Conclusion: Terminal-World constructs 5,723 training environments, and its 8B, 14B, and 32B models consistently outperform existing terminal-agent baselines across six benchmarks.The conclusion presents skill-grounded synthesis as a practical path toward more capable terminal agents.
  • Conclusion: The skill taxonomy contains 12 major categories and 63 subcategories spanning system operations, data engineering, and scientific computing.Each skill represents a core terminal capability required by an agent.
  • Conclusion: Skill teams use same-subcategory Compose-with relations for depth extension, while skill graphs use cross-subcategory Depend-on relations for breadth extension.Greedy longest-path extraction is used to construct skill graphs.

B.1 Benchmark Details

The evaluation covers six benchmarks spanning terminal coding, mathematics, data analysis, table reasoning, and database tasks, converted into executable terminal environments with task-specific metrics and reproducible sampling.

  • Benchmark Overview: Six benchmarks evaluate terminal agents across coding, mathematics, data analysis, table reasoning, and database tasks.The latter five benchmarks are converted into executable terminal tasks following Nemotron-Terminal.
  • Terminal-Bench 2.0: Terminal-Bench 2.0 contains 89 native terminal-agent tasks covering command-line navigation, system administration, file manipulation, and coding in Bash.Success is measured by exact matching of the final environment state.
  • Domain Benchmarks: AIME tasks require Python execution for exact-match mathematical answers, while DABench and TableBench require terminal-based data processing and are evaluated with LLM-as-a-Judge.DABench uses a randomly sampled 500-instance test set.
  • Terminalization: Terminalized tasks place agents in open-ended Bash sandboxes where they manipulate files, execute code or queries, debug from output, and save results to designated files.Original benchmark questions are wrapped with file paths, tool constraints, and output-format requirements.
  • Evaluation Protocol: Model comparisons use provider-recommended sampling configurations for APIs and local SGLang deployments.The sampling settings are documented separately for API-based and locally deployed models.

B.4 Training Compute Details

Terminal-World-32B is fine-tuned with a 32,768-token sequence length for two epochs on 32 NVIDIA H20 GPUs, completing a full run in approximately 80 hours.

  • Training Compute: Terminal-World-32B training uses 32 GPUs for approximately 80 hours per full run.The setup uses four nodes with eight NVIDIA H20 141 GB GPUs each.
  • Training Configuration: The training configuration includes a 2 × 10^-5 peak learning rate, cosine decay, 10% warmup, 1 × 10^-4 weight decay, gradient clipping of 1.0, and two epochs.Training uses a sequence length of 32,768 tokens.

C.1 Error Analysis

Terminal-World-32B failures were analyzed across four recurring error types, while many verifier-failed trajectories were judged semantically complete by independent evaluators.

  • Failure categories: Four error types were identified from unsuccessful Terminal-World-32B trajectories: context window overflow, execution deadlock, premature completion, and task substitution.The analysis used three independent evaluation runs in Terminal-World 2.0.
  • Failure mechanisms: Context window overflow causes repeated failed approaches after history summarization removes fine-grained memory of prior actions, leading to execution timeouts.
  • Failure mechanisms: Execution deadlock occurs when agents repeatedly issue identical interruption commands after recognizing abnormal interactive terminal states.
  • Failure mechanisms: Premature completion reflects declaring success after resolving superficial triggers without verifying implicit or secondary semantic constraints.
  • Failure mechanisms: Task substitution replaces a required analytical procedure with a superficial heuristic, creating apparent progress while bypassing the task’s core requirements.
  • Semantic correctness: 67.7% of verifier-failed trajectories were judged task-completing by majority vote, with Fleiss’ κ = 0.742 across four judges.Individual judge completion rates ranged from 63.3% to 70.3%.

C.3 Difficulty of Synthesized Tasks

Terminal-World environments were evaluated for difficulty using the same DeepSeek-V3.2 configuration, and they produced a substantially lower pass rate than three comparison environments.

  • Difficulty comparison: 39.8% Pass@1 was achieved on Terminal-World by DeepSeek-V3.2, only 4.2% relatively higher than on Terminal-Bench 2.0.
  • Difficulty comparison: DeepSeek-V3.2 exceeded 50% pass rates on the other three terminal environments, with Nemotron-Terminal-Corpus reaching 79.8%.
  • Implication: The results suggest that Terminal-World environments are more challenging and therefore provide higher-quality supervision signals for terminal-task solving.

C.4 Quality of Task and Environment

Terminal-World was assessed for dataset quality and guideline-supported trajectory collection, showing strong task–environment alignment and more reliable, efficient teacher demonstrations with execution guidelines.

  • Quality assessment: Terminal-World quality was evaluated across terminal nativeness, environment-task consistency, environment quality, and verifier robustness.The assessment used Claude Code with three judge models.
  • Quality assessment: Terminal-World achieved the highest quality across all four dimensions and all three judge models.
  • Execution guidelines: 39.6% success was achieved with execution guidelines versus 27.6% without them, a 30.3% relative drop when guidelines were removed.
  • Execution guidelines: Successful trajectories averaged 12.66 steps with guidelines versus 14.27 steps without guidelines.The comparison used the same DeepSeek-V3.2 teacher on 500 sampled tasks.
  • Execution guidelines: Skill-derived guidelines help teachers avoid redundant exploration and produce more concise successful demonstrations.

C.6 LLM-as-a-Judge Consistency Analysis

The paper evaluates LLM-as-a-Judge reliability through multi-judge agreement and human comparisons, covering task filtering and benchmark evaluation procedures.

  • Evaluation scope: The pipeline uses LLM-as-a-Judge for task quality filtering and evaluation on DABench and TableBench.
  • Task filtering: Task quality filtering was assessed on five criteria using four judge models and human annotations on a random subset.The filtering sample contained 500 generated task specifications, with 200 also annotated by humans.
  • Task filtering: ICC(2,1) ranged from 0.708 to 0.812 across filtering criteria, with an overall ICC of 0.770.Human scores were close to model-judge averages.
  • Benchmark evaluation: Audit-judge agreement was Fleiss’ κ = 0.873 on DABench and 0.881 on TableBench.The audit judges were GPT-4.1, GLM-5, and Doubao-2.0-Pro.
  • Prompt templates: The prompt templates define task instructions, environment blueprints, evaluation criteria, and skill-derived execution guidance as components of task synthesis.The broader prompt set also specifies skills, personas, generated goals, and guideline-generation requirements.

E.1 Example 1: Task Generation — ELF Binary Parsing (Astrophysics)

This example instantiates an ELF-binary analysis task for an astrophysics persona, with files, dependencies, execution steps, and verification criteria derived into a structured environment. The task extracts 4-byte values from the RW LOAD segment and validates the resulting JSON map.

  • Task Generation: An astrophysicist persona receives a task to parse an ELF binary and extract calibration constants from its memory mapping.The instruction targets the RW PT_LOAD segment, includes BSS zero-padding, and saves decimal virtual-address keys with 32-bit integer values.
  • Environment Building: The environment includes a compiled ELF64 executable whose initialized data and BSS are represented in the RW LOAD segment.The binary contains calibration, initialized readings, and an uninitialized photon buffer; the RW segment has MemSiz 0x1400 and FileSiz 0x400.
  • Evaluation Scope: 1280 key-value pairs are expected because the RW segment has MemSiz 0x1400, or 5120 bytes, divided into 4-byte values.The extraction begins at virtual address 0x601000, whose decimal key is 6295552.
  • Evaluation Scope: The generated verifier checks that the output file exists, contains valid JSON, has the expected key count, and stores 271828 at the segment start.The expected calibration value is associated with decimal virtual address 6295552.
  • Execution Guideline: The execution guideline uses readelf to identify the RW segment, Node.js to extract aligned values, and JSON checks to validate the output.The workflow verifies ELF64 little-endian properties, confirms the segment metadata, creates a BSS-aware extraction script, and checks the resulting JSON.

E.3 Example 3: Trajectory Collection — Video OCR Extraction

This example collects a trajectory for extracting treatment methods from a video by installing tooling, sampling frames, applying OCR, and filtering the recognized text. The resulting list contains four treatments and passes six downstream checks.

  • Trajectory Collection: The trajectory begins by installing ffmpeg and then extracts video frames at one frame per second.The video is processed into PNG files under /app/frames for subsequent OCR.
  • Trajectory Collection: The OCR script reads sorted frame images with OpenCV and pytesseract, accumulating recognized text in /app/all_text.txt.The trajectory records successful extraction of 15 frames and writes the combined OCR output to a text file.
  • Trajectory Collection: The parser removes slide headings, filters lines using treatment-related keywords, deduplicates matches, and writes the cleaned treatment list.The keyword whitelist includes therapy, treatment, medication, CBT, SSRI, and antidepressant.
  • Trajectory Collection: Four treatments are extracted: Cognitive Behavioral Therapy, Family-Based Treatment, Interpersonal Therapy, and SSRIs such as fluoxetine.The output contains four unique lines and explicitly includes CBT.
  • Trajectory Collection: 6/6 downstream tests pass, confirming file presence, minimum length, and inclusion of CBT, SSRI, IPT, and family-based treatment.The reported reward is 1.0 after 11 steps.
Loading 2605.20876v1…