PaperlayerView exact revision on arXiv

Source-linked AI summary

Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments

Siwei Wu, Yizhi Li, Yuyang Song, Wei Zhang, Yang Wang, Riza Batista-Navarro, Xian Yang, Mingjie Tang, Bryan Dai, Jian Yang, Chenghua Lin

arXiv:2602.01244v3cs.CL

TL;DR

Terminal-agent training needs executable environments and verifiable trajectories, but existing scalable methods remain insufficiently grounded in real execution. TerminalTraj addresses this with repository filtering, Docker-aligned task generation, and executable validation, producing verified data that improves TerminalBench performance and test-time scaling.

  • Problem

    Existing methods struggle to scale execution-based terminal-agent training because they rely on heuristic or simulated environments and verification rather than actual execution.

  • Method

    TerminalTraj filters repositories, constructs Docker execution environments, generates environment-aligned tasks, and verifies trajectories with instance-specific executable validation.

  • Results

    TerminalTraj builds 32K Docker images and 50,733 trajectories, improving Qwen2.5-Coder performance by 20% on TB 1.0 and 10% on TB 2.0; TerminalTraj-32B reaches 35.30% and 22.00%.

  • Takeaways & Limitations

    TerminalTraj provides execution-grounded training data that supports strong terminal-agent performance and test-time scaling across realistic terminal benchmarks.

  • Takeaways & Limitations

    Automated privacy and security filtering may not achieve perfect recall when removing sensitive information from environments and trajectories.

Abstract

from arXiv · show

Training agentic models for terminal-based tasks critically depends on high-quality terminal trajectories that capture realistic long-horizon interactions across diverse domains. However, constructing such data at scale remains challenging due to two key requirements: \textbf{\emph{Executability}}, since each instance requires a suitable and often distinct Docker environment; and \textbf{\emph{Verifiability}}, because heterogeneous task outputs preclude unified, standardized verification. To address these challenges, we propose \textbf{TerminalTraj}, a scalable pipeline that (i) filters high-quality repositories to construct Dockerized execution environments, (ii) generates Docker-aligned task instances, and (iii) synthesizes agent trajectories with executable validation code. Using TerminalTraj, we curate 32K Docker images and generate 50,733 verified terminal trajectories across eight domains. Models trained on this data with the Qwen2.5-Coder backbone achieve consistent performance improvements on TerminalBench (TB), with gains of up to 20\% on TB~1.0 and 10\% on TB~2.0 over their respective backbones. Notably, \textbf{TerminalTraj-32B} achieves strong performance among models with fewer than 100B parameters, reaching 35.30\% on TB~1.0 and 22.00\% on TB~2.0, and demonstrates improved test-time scaling behavior. All code and data are available at https://github.com/Wusiwei0410/TerminalTraj.

1. Introduction

TerminalTraj addresses the difficulty of scaling realistic terminal-agent training data by grounding task generation and verification in executable Docker environments. The pipeline produces diverse environments and verified trajectories, improving TerminalBench performance and test-time scaling.

  • TerminalBench evaluates agents in terminal environments that preserve execution constraints and tool-mediated interactions found in real-world workflows.
  • Existing trajectory-generation methods remain decoupled from actual execution, limiting their capture of environment-dependent terminal behaviors.
  • TerminalTraj curates Docker-executable instances, generates environment-aligned tasks, and verifies rollouts with instance-specific executable validation.
  • 32,325 Docker images and 50,733 verified trajectories are generated across eight programming languages and specialized domains.
  • 35.30% on TB 1.0 and 22.00% on TB 2.0 are achieved after fine-tuning Qwen2.5-Coder on TerminalTraj, with nearly 20% and 10% absolute gains.
  • TerminalTraj-7B shows a steeper pass@k scaling slope than Qwen2.5-Coder-7B, while TerminalTraj-32B reaches 63% at pass@16.

2. TerminalTraj Pipeline

TerminalTraj transforms repositories into reproducible Docker environments, Docker-aligned tasks, and executable validation workflows. Its curation and generation stages combine repository quality scoring, domain-specific context, and agent execution for verified trajectories.

  • TerminalTraj addresses the scarcity of training environments that are both scalable and verifiable, especially for reinforcement learning.
  • The pipeline collects repositories, curates Docker images for reproducibility, and generates tasks that are verified through agent execution loops.
  • The collection spans 899,741 repositories, 2,010,113 code files, and 20,000 auxiliary input links across programming and domain-specific signals.
  • Docker Image Curation: A ScoreModel assigns each code file a completeness-and-executability score, and repositories below τ = 0.2 are discarded.
  • Instance Generation: The pipeline uses documentation and execution logic to synthesize task queries and executable validation code from repository-specific materials.
  • Instance Generation: Pytest validation checks state-based side effects, such as file creation, configuration changes, or process output, rather than requiring one command sequence.

3. Empirical Analysis of TerminalTraj Dataset

TerminalTraj produces a substantial, consistently filtered dataset from selected repositories, while repository scores predict both Docker build success and generated-instance quality.

  • 32,325 Docker images were constructed from repositories spanning eight programming languages, with an approximately 17% build success rate.
  • 50,733 verified trajectories were synthesized despite a 4% verified-trajectory rate, reflecting high-precision executable filtering.
  • Stable retention rates across programming languages make final data yield predictable for a fixed repository budget.
  • Higher repository scores consistently corresponded to higher Docker build success rates; scores above 0.5 achieved 19.5% versus 9% for random selection.
  • Human and LLM evaluations consistently found that instance quality increases with repository scores.
  • Restricting generation to repositories scoring above 0.2 produced mostly instances with human scores above 5.5.

4. Experiment Setup

The experiments use Qwen2.5-Coder models, standardized terminal-agent evaluation with Terminus2, and comparisons against diverse closed- and open-source baselines.

  • Qwen2.5-Coder was selected as the backbone because it predates TerminalBench and avoids potential contamination from terminal-specific optimizations.
  • Models were trained with multi-turn supervised fine-tuning using the Megatron-LM framework.
  • Evaluation used TB 1.0 and TB 2.0 with the Terminal Bench protocol and Terminus2 as the primary agent scaffold.
  • The study compares leading closed-source and open-source agentic models to provide comprehensive baseline coverage.

5. Results

TerminalTraj improves TerminalBench performance across model sizes and strengthens test-time scaling through diverse, execution-grounded trajectories. Ablations show that domain coverage, executable verification, and environment diversity each contribute to performance.

  • Main Results: More than +20 points on TB 1.0 and +10 points on TB 2.0 are achieved across all evaluated model sizes after TerminalTraj training.
  • Main Results: 35.30 on TB 1.0 and 22.00 on TB 2.0 are achieved by TerminalTraj-32B, outperforming Qwen3-32B-Nex-N1 by 6.55 and 5.3 points, respectively.
  • Best-of-K Trajectory Sampling: 63% pass@16 is reached by TerminalTraj-32B, while TerminalTraj models gain 20%–30% as sampling increases from k = 1 to 16.
  • Domain-Specific Data Ablation: 8.9%/7.4% and 9.7%/8.7% drops occur on TB 1.0/TB 2.0 when WS and QEMU data are excluded, respectively.
  • Trajectory Verification: Code-verified trajectory selection consistently outperforms LLM-verified selection across training sets of 1K, 2K, 4K, and 8K trajectories.
  • Environment Diversity: 45.00% pass@8 is achieved with over 32K Dockerized environments, compared with 12.5% from SETA’s 400 curated environments.

6. Related Work

Related work moves from code completion toward repository-level, executable code-agent tasks, while terminal-specific trajectory synthesis remains comparatively underexplored. Existing efforts combine executable environments, synthetic tasks, and trajectories in code-agent settings.

  • Repository-level code-agent benchmarks evaluate issue localization, multi-file editing, and execution-based validation in realistic codebases.
  • SWE-Gym, SWE-rebench, and SWE-Factory build executable environments, while SWE-Smith, SWE-Mirror, and BugPilot synthesize tasks or trajectories for supervised fine-tuning.

7. Conclusion

TerminalTraj constructs scalable, executable, and verifiable terminal-task trajectories and improves Qwen2.5-Coder performance on TerminalBench. Its 32B model reaches strong performance among models below 100B parameters while showing test-time scaling.

  • TerminalTraj builds 32K Docker images and 50,733 trajectories from 899,741 GitHub repositories across eight programming languages.
  • Training TerminalTraj models improves their Qwen2.5-Coder backbones by 20% on TB 1.0 and 10% on TB 2.0.
  • TerminalTraj-32B achieves 35.30% on TB 1.0 and 22.00% on TB 2.0 among models with fewer than 100B parameters.
  • TerminalTraj-32B reaches 63.75% pass@16, demonstrating strong test-time scaling.

Impact Statement

TerminalTraj releases execution-grounded trajectories for terminal agents intended to support work across technical domains. Because terminal agents can execute destructive commands, deployment requires strong isolation and privilege controls.

  • The pipeline targets software engineering, system administration, and automated debugging with large-scale, execution-grounded trajectories.
  • The authors aim for verified trajectories to support more robust and reliable agents for error-prone technical tasks.
  • Terminal agents can execute destructive commands, including filesystem deletion and unintended network requests.
  • Deployment should use rigorous sandboxing and privilege limitations to prevent unintended host-system damage.

Ethics Statement

The study addresses licensing, privacy, security, and deployment risks through repository screening, trajectory filtering, and sandboxed experimentation. Automated filtering reduces but may not eliminate sensitive-information and broader safety risks.

  • The initial pool contains approximately 900K GitHub repositories, but 32,325 were retained after license, buildability, and security screening.
  • License filtering retained repositories with permissive MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause licenses.
  • The pipeline uses Dockerfiles and agent-environment trajectories rather than repository source code itself as training data, and removes 1,184 trajectories containing copied code.
  • Regex-based privacy and security filtering removes trajectories containing potentially sensitive information, including credentials and secret keys.
  • Automated filtering may not achieve perfect recall, leaving residual privacy and security risk.
  • Experiments use sandboxed environments, while deployment outside controlled settings should require explicit confirmation for potentially consequential operations.

A. Trajectory and Tool Use Analysis

Trajectory analysis shows that Linux-centric domains cluster around shared terminal capabilities, while domain composition and verified-data scale materially affect performance. QEMU-related data has a particularly strong contribution, and performance continues improving with more verified trajectories.

  • Trajectory and Tool Use Analysis: SentenceBERT and PCA are used to analyze eight domains’ trajectory distributions and domain-specific tool usage.
  • Trajectory and Tool Use Analysis: Web Service, QEMU-related, Security, and SQL trajectories form more closely clustered distributions because they operate primarily in Linux terminal environments.
  • Trajectory and Tool Use Analysis: Specialized tools such as nginx, savevm, select, and find distinguish several otherwise related domains.
  • Domain Ablation: Removing any domain lowers performance, with WS and QEMU producing the largest reported drops.
  • Domain Ablation: A size-matched full-domain setting still outperforms domain-removed variants, indicating degradation mainly reflects domain removal rather than sample-count differences.
  • Trajectory Scaling: TerminalBench 1.0 performance rises from 28 at 8k verified trajectories to 32 at 32k and 35 with the full dataset.

D. Comparing with Existing Dataset

SETA 2 improves Qwen3-8B’s test-time scaling but is constrained by 400 Dockerized environments, whereas TerminalTraj scales Docker-aligned executable instances to enhance backbone capability.

  • TerminalTraj automatically collects repositories, constructs Docker environments, and generates corresponding task instances for data-level scaling.
  • TerminalTraj yields substantial improvements over the original backbone and the compared dataset on TerminalBench pass@8.
  • The ScoreModel uses code-file content to produce normalized quality scores in [0, 1] through a regression head and sigmoid.
  • Approximately 900K repositories make direct large-teacher scoring prohibitively expensive because repositories may contain dozens of code files.
  • A lightweight Qwen2.5-Coder-0.5B ScoreModel approximates Qwen2.5-Coder-32B judgments for scalable repository screening at roughly two orders of magnitude lower inference cost.

G.3. Teacher–Student Consistency Analysis

Teacher–student analyses evaluate whether the lightweight ScoreModel preserves teacher behavior for repository filtering, using file-level rankings, repository-level curation quality, and high-scoring repository overlap.

  • The ScoreModel preserves teacher file-quality rankings with a Spearman correlation of 0.81 across 1,000 sampled code files.
  • Above repository quality scores of 0.2, teacher and student models produce highly similar downstream instance quality in the ranges used for curation.
  • The teacher is slightly better at separating low-quality repositories in the [0, 0.2] score range.
  • 85% overlap between teacher- and student-ranked top-50 repositories shows substantial preservation of high-quality repository selection.
  • The analyses conclude that the ScoreModel closely approximates the teacher for large-scale repository filtering while costing much less to run.
  • The benchmark evaluation reports mean accuracy with 95% confidence intervals and higher accuracy than the backbone across both TerminalBench benchmarks.

M. Instance Evaluation Criteria

TerminalTraj evaluates generated instances through task-design criteria and strict trajectory verification, combining human or LLM assessment with executable checks against concrete environment outcomes.

  • Task Evaluation Dimensions: Task soundness measures whether an instance is coherent, realistic, well-defined, and achievable in its environment.
  • Task Evaluation Dimensions: Evaluability measures whether task success can be objectively and reliably verified, prioritizing automatic and deterministic checks.
  • Task Evaluation Dimensions: Information sufficiency assesses whether required inputs, environment assumptions, and constraints are explicitly specified.
  • Task Evaluation Dimensions: Behavioral constrainedness rewards tasks requiring concrete actions and real execution rather than permitting fabricated final text.
  • Trajectory Verification: Trajectory evaluation rejects missing evidence, partial success, or ambiguity and outputs only a binary solution judgment.
  • Trajectory Verification: Executable validation checks final environment state, such as exact output files, expected contents, running services, and successful requests.

P.4.3. TRAJECTORY {

The trajectories show agents using terminal interactions to complete heterogeneous tasks, while success is expressed through concrete files, outputs, and task states. Examples span maze navigation, Sudoku solving, Git deployment, object detection, and database extraction.

  • Maze navigation: Maze navigation combines systematic exploration with assembling key fragments into a solution file.The trajectory reports DFS exploration, fragments 7A, 9B, and 4C, and writing 7A9B4C to /app/maze_key.txt.
  • Sudoku solving: Sudoku solving requires writing nine rows of nine digits to /app/solution.txt.The reported trajectory claims the output uses the required nine-line format, despite a filename mismatch involving the puzzle image.
  • Git deployment: A Git task configures SSH password authentication, branch-specific HTTPS endpoints, and deployment through a post-receive hook.The task distinguishes main and dev branch content at separate HTTPS paths.
  • Object detection: An object-detection trajectory produces a runnable tool, detections.json, and two detected persons with bounding boxes and confidence scores.The reported JSON uses integer bounding-box coordinates and floating-point confidence values.
  • Validation: The examples illustrate terminal task completion through concrete environment changes and generated artifacts rather than command-sequence matching.Executable validation can check resulting side effects such as files, system configuration, or process output.
  • Database extraction: A database trajectory inspects SQLite tables, extracts course objectives for program_year_id = 101, and saves five results as JSON.The output is reported at /results/course_objectives_py101.json with course_objective_id and title keys.
Loading 2602.01244v3…