Source-linked AI summary
TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agents
Kaijie Zhu, Yuzhou Nie, Yijiang Li, Yiming Huang, Jialian Wu, Jiang Liu, Ximeng Sun, Zhenfei Yin, Lun Wang, Zicheng Liu, Emad Barsoum, William Yang Wang, Wenbo Guo
TL;DR
Open-weight terminal agents face scarce, low-fidelity executable environments and training trajectories that underrepresent runtime mistakes. TermiGen synthesizes verifiable Docker environments and error-rich expert trajectories, and its 32B model reaches 31.3% on TerminalBench, a reported open-weights state-of-the-art. The paper also identifies synthetic isolated environments and exclusive SFT as limitations of its current setup.
Problem
Open-weight terminal agents lack scalable, high-fidelity executable environments and expert trajectories containing the failure states needed for runtime error recovery.
Method
TermiGen combines multi-agent synthesis and validation of Docker-based environments with Generator-Critic error injection that creates error-diagnosis-recovery trajectories.
Results
31.3% pass rate on TerminalBench makes TermiGen-Qwen2.5-Coder-32B a reported new state-of-the-art among open-weights models, surpassing existing fine-tuning baselines and o4-mini with Codex CLI.
Takeaways & Limitations
Grounded environments and error-correction trajectories improve terminal-agent training over simulation-based and standard-trajectory baselines.
Takeaways & Limitations
The environments are synthetic and isolated, so they cannot fully replicate the stochasticity and scale of real-world production systems.
Abstract
from arXiv · showhide
Executing complex terminal tasks remains a significant challenge for open-weight LLMs, constrained by two fundamental limitations. First, high-fidelity, executable training environments are scarce: environments synthesized from real-world repositories are not diverse and scalable, while trajectories synthesized by LLMs suffer from hallucinations. Second, standard instruction tuning uses expert trajectories that rarely exhibit simple mistakes common to smaller models. This creates a distributional mismatch, leaving student models ill-equipped to recover from their own runtime failures. To bridge these gaps, we introduce TermiGen, an end-to-end pipeline for synthesizing verifiable environments and resilient expert trajectories. Termi-Gen first generates functionally valid tasks and Docker containers via an iterative multi-agent refinement loop. Subsequently, we employ a Generator-Critic protocol that actively injects errors during trajectory collection, synthesizing data rich in error-correction cycles. Fine-tuned on this TermiGen-generated dataset, our TermiGen-Qwen2.5-Coder-32B achieves a 31.3% pass rate on TerminalBench. This establishes a new open-weights state-of-the-art, outperforming existing baselines and notably surpassing capable proprietary models such as o4-mini. Dataset is avaiable at https://github.com/ucsb-mlsec/terminal-bench-env.
1. Introduction
Terminal tasks are difficult because executable training environments are scarce and standard expert trajectories underrepresent the mistakes smaller models must recover from. TermiGen addresses these gaps with verifiable environment synthesis and controlled error-correction trajectories, achieving strong TerminalBench performance.
- Challenges: Terminal tasks require complete executable specifications of system architecture, dependencies, and runtime state, making scalable environment construction expensive.TerminalBench required extensive manual curation for 200 tasks.
- Challenges: Repository-based synthesis lacks diversity and requires manual success criteria, while purely LLM-based simulation can hallucinate execution logs.The two synthesis paradigms therefore have complementary limitations: grounding without scalability versus scalability without reliable runtime state.
- Challenges: Long-horizon terminal tasks are vulnerable to cascading failures, but expert trajectories rarely contain the simple mistakes and recovery phases smaller models encounter.This creates a distributional mismatch between standard training data and runtime failures.
- TermiGen: TermiGen synthesizes diverse verifiable tasks and validates solvability in real Docker containers before collecting trajectories.Its multi-agent environment pipeline is paired with interactive execution validation to eliminate simulation hallucinations.
- Results: 31.3% pass rate on TerminalBench makes TermiGen-Qwen2.5-Coder-32B a new open-weights state-of-the-art, outperforming Reptile by over 12%.The model also surpasses o4-mini with Codex CLI according to the reported evaluation.
2. Existing Works and Limitations
Existing environment-construction methods either depend on repositories that constrain task diversity or synthesize executions without reliable runtime grounding. Standard expert-trajectory distillation likewise omits failure states and recovery behavior, producing fragile agents.
- Environment Construction: Repository-based environment construction can be semi-automated or fully automated, but dependence on pre-existing projects constrains generalization to diverse unseen tasks.
- Trajectory Training: Standard supervised fine-tuning distills expert trajectories to clone optimal behavior.
- Trajectory Training: Because experts rarely make simple mistakes, expert-only datasets lack failure states and recovery phases.
- Trajectory Training: Agents trained solely on expert trajectories can fail catastrophically when runtime exceptions create out-of-distribution states.
3. Key Technique
TermiGen uses a two-phase pipeline: it synthesizes and validates executable terminal environments, then collects trajectories with controlled failures that require diagnosis and recovery. The design targets environment scarcity and exposure bias through multi-agent refinement and error injection.
- Challenges: TermiGen frames terminal-agent training around two challenges: scarce executable environments and the need for error-correction training.
- Challenges: Synthetic execution can hallucinate state because simulators may fail to preserve persistent file changes, motivating executable and scalable environments.
- Phase I: Task and Environment Synthesis: Phase I uses a multi-agent framework and an 11-category, three-tier taxonomy to generate diverse task seeds and concrete specifications.
- Phase II: Error-Correction Trajectory Collection: Phase II uses a Generator-Critic architecture to inject controlled failures and produce trajectories containing explicit error → diagnosis → recovery cycles.
- Phase I: Task and Environment Synthesis: The environment pipeline plans files, generates contents, builds Docker containers, and feeds build errors back for up to N = 5 refinement iterations.
- Phase I: Task and Environment Synthesis: Task proposals are refined until they score above 4 in every quality dimension, with up to three feedback-guided refinement rounds.
- Phase II: Error-Correction Trajectory Collection: At each step, an intent is sampled with error probability ϵ = 0.2, distinguishing deliberate error injection from attempted task advancement.
4. Experiments
TermiGen is evaluated against proprietary, open-weight, and specialized terminal agents, then examined through controlled ablations of environment fidelity, error correction, and trajectory filtering. Results favor physically verifiable execution, injected error-correction trajectories, and retaining imperfect trajectories.
- Main Comparisons: TermiGen is compared with proprietary frontier, general-purpose open-weight, and specialized terminal-task models on TerminalBench 1.0.The benchmark uses realistic terminal tasks in isolated Docker containers and pass rate as its core metric.
- Main Comparisons: 31.3% pass rate makes TermiGen-Qwen2.5-32B a new open-weights SOTA on TerminalBench, ahead of Reptile at 18.9% and TerminalAgent at 15.5%.It exceeds o4-mini at 20.0% by 11.3%, while Apex2 with Claude-4.5-Sonnet reaches 64.5%.
- RQ1: Verifiable Environments vs. Simulation: 25.0% Pass Rate for verifiable environments exceeds the simulation baseline by 2.5%, a relative improvement of ≈ 11%, with N = 800.Both conditions use identical task seeds, prompts, and 20% error injection; only the execution layer differs.
- RQ1: Verifiable Environments vs. Simulation: Simulation audits found observation errors in 26% of sampled trajectories, including Spurious Verbosity at 53%, Semantic Deviation at 35%, and State Inconsistency at 12%.These errors can produce incorrect shell-state reasoning and ineffective debugging loops.
- RQ2: Error-Correction vs. Standard Trajectory: 25.0% Pass@1 with error-correction trajectories outperforms the standard-trajectory baseline at 21.8% under the same data size, N = 800.Naturally occurring baseline errors are dominated by Verification Failures at ≈50%, whereas active injection exposes agents to a broader failure spectrum.
- RQ3: The Value of Negative Trajectories: Including trajectories with Test Pass Rate ≥ 0% performs best, surpassing the strict Test Pass Rate ≥ 100% filtering baseline.The analysis attributes this to coverage of difficult scenarios and local repairs within ultimately unsuccessful runs.
5. Conclusion and Limitations
TermiGen presents verifiable terminal environments and error-injection distillation, while identifying training, agent-memory, and environment-realism limitations for future work.
- TermiGen presents a framework for synthesizing verifiable terminal environments and an error-injection distillation strategy.
- Training currently relies exclusively on supervised fine-tuning, leaving reinforcement learning as a proposed next step.Deterministic automated tests could support exploration and learning from trial-and-error beyond the fixed training distribution.
- The evaluated agent lacks a memory component, limiting its use of interaction histories.The authors propose more sophisticated agents with memory as future work.
- Synthetic, isolated environments cannot fully replicate the stochasticity and scale of real-world production systems.The stated boundary includes distributed clusters and high-concurrency traffic, motivating transfer studies on large-scale infrastructure tasks.
Impact Statement
The paper frames TermiGen as improving terminal-agent reliability and accessibility while acknowledging safety and misuse risks from autonomous command execution.
- TermiGen improves autonomous-agent reliability in software engineering and system administration by enabling open-source models to use terminal tools.
- Autonomous terminal commands can accidentally cause data loss or system crashes, such as deleting the wrong files.The paper states that error correction helps reduce this risk by teaching agents to identify and fix mistakes.
- The techniques could theoretically be misused to automate cyber-attacks.The paper recommends isolated environments such as Docker containers and continuous human supervision.
A. Detailed Task Categories
The task taxonomy covers approximately 3,500 tasks across 11 categories, with t-SNE showing semantic clustering by task type.
- ≈3,500 tasks span 11 categories in the task taxonomy.
- The t-SNE visualization shows semantic clustering by task type.
B. Implementation Details
The training pipeline uses full-parameter supervised fine-tuning with a 20,000-token context and specified AdamW scheduling settings.
- Full-parameter supervised fine-tuning is performed on the base model using the synthetically generated dataset.The pipeline is built upon the LLaMA-Factory framework.
- The maximum sequence length is set to 20,000 tokens for multi-turn terminal interactions.
- AdamW optimization uses a 5.0 × 10−6 learning rate, cosine scheduling, 0.1 warmup ratio, and zero weight decay.
C. Task and Error-Correction Trajectory
The case study presents a synthetic macOS automation task requiring recursive application discovery, version extraction, and a precisely validated JSON output. Its pytest suite checks file location, schema, types, and exact application counts.
- Task Description: The task recursively locates .app bundles under /Applications and extracts version information from each Info.plist file.It must handle nested bundles, missing or malformed Info.plist files, and unreadable version information.
- Task Description: The required output is /tmp/app inventory.json with counts for total apps, apps with versions, and apps without versions.
- Directory Structure: The benchmark directory includes task metadata, Docker configuration, Compose configuration, test scripts, application fixtures, and automated tests.
- Evaluation Framework: The pytest suite validates output existence, JSON validity, exactly three integer fields, and a total application count of 4.
C.2. An Error-Correction Example in Training Trajectories
The example demonstrates a deliberately injected CVC5 command error and a subsequent diagnosis that connects the parser failure to an incompatible SMT-LIB2 logic declaration. The trajectory then considers overriding the declared logic before retrying execution.
- Task Setup: The task asks CVC5 to determine satisfiability and save either a password model or an unsatisfiable result in /workspace/result.txt.
- Initial State: The trajectory verifies the SMT file, CVC5 installation, and password constraints before running the solver.
- Injected Error: The intentionally incorrect CVC5 command produces a parse error because the symbol >= is not declared as a variable.
- Diagnosis: The diagnosis identifies a mismatch between QF S string-only logic and integer comparisons involving str.len.
- Correction: The proposed correction is to use QF SLIA or allow CVC5 to auto-detect the logic, followed by a retry using --force-logic=QF SLIA.
D. Case Study: Learned Error-Correction Capabilities in TermiGen
The Coq case study contrasts a baseline that confidently terminates without verifying its compiled proof against TermiGen’s iterative diagnosis and correction of compilation errors. The comparison frames execution feedback as a navigational signal for recovery.
- Case Study: The case study compares Qwen2.5-Coder-32B and TermiGen-Qwen2.5-Coder-32B on proving commutativity of addition in Coq.
- Baseline Failure: The baseline fails after assuming its generated proof is correct without verifying the compiled .vo object file.Its induction-and-reflexivity attempt contains a mathematically incomplete proof and ends in confident termination.
- TermiGen Success: TermiGen encounters a unification error, diagnoses the distinction between definitional and propositional equality, and applies rewriting-based correction.
- Interpretation: The comparison associates error-injection training with treating execution feedback as a navigational signal rather than a terminal state.