Source-linked AI summary
CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents
Zhanbo Hua, Yifan Yao, Weihao Xie, Yongchi Zhao, Minghao Liu, Ruizhi Qiu, Zhewei Huang, Zun Wang, Yiyan Ji, Yunhai Ye, Letian Zhu, Xinping Lei, Han Li, Zhiyuan Ma, Zili Wang, Zhaoxiang Zhang, Jiaheng Liu
TL;DR
High-quality, executable training data for terminal agents remains scarce, while existing synthesis pipelines often produce shallow or weakly supervised tasks. CLI-Universe combines structured capability specifications, evidence-guided research, and executable verification to construct tasks, and fine-tuning Qwen3-32B on 6,000 trajectories achieves 33.4% on Terminal-Bench 2.0.
Problem
High-quality training data for terminal agents remains scarce, especially tasks requiring realistic multi-step interaction and rigorous executable supervision.
Method
CLI-Universe specifies tasks across a capability taxonomy, grounds them in real technical materials, and verifies Dockerized environments through rubric-gated tests, hint filtering, and fail-to-pass checks.
Results
33.4% on Terminal-Bench 2.0 is achieved by Qwen3-32B fine-tuned on CLI-Universe-6K, surpassing all open-source-data models at or below 32B.
Takeaways & Limitations
CLI-Universe shows that carefully structured and independently verified tasks can provide strong supervision for terminal agents with limited training data.
Takeaways & Limitations
The synthesized data quality is bounded by the underlying LLM agents, and a performance gap to the strongest frontier models remains.
Abstract
from arXiv · showhide
While recent LLM-based terminal agents have demonstrated promising capabilities, the scarcity of high-quality, executable training data remains a critical bottleneck. Existing synthesis pipelines typically scale by retrofitting surface-level artifacts into tasks, frequently yielding ambiguous instructions, shallow execution paths, and brittle tests that provide weak learning signals. To overcome this, we introduce CLI-Universe, a principled synthesis engine that constructs terminal-agent tasks. CLI-Universe generates candidate tasks by sampling combinations across a multi-dimensional capability taxonomy (domain, skill type, capability, and engineering pillar), then grounds each candidate through evidence-guided deep research over real-world technical materials. To ensure rigorous supervision, validated blueprints are instantiated into Dockerized environments and subjected to a multi-stage executable verification pipeline featuring rubric-gated test construction, hint-conditional filtering, and strict fail-to-pass checking. Across the full pipeline, from candidate generation to verification, approximately two-thirds of candidates are discarded, retaining only those that are genuine, verifiable, and non-trivially challenging. To validate our framework, we instantiate a highly distilled dataset of 6,000 trajectories called CLI-Universe-6K. Remarkably, fine-tuning Qwen3-32B on CLI-Universe-6K achieves 33.4% on Terminal-Bench 2.0. This sets a new state-of-the-art for models trained on open-source data at or below 32B parameters, and outperforms several models an order of magnitude larger, demonstrating the profound data efficiency of structured, high-fidelity synthesis.
1 Introduction
CLI-Universe addresses scarce high-quality terminal-agent training data with an inside-out synthesis pipeline grounded in structured capability specifications and evidence-guided research. Its executable verification stages retain high-signal tasks, yielding CLI-Universe-6K and strong Terminal-Bench 2 performance from Qwen3-32B fine-tuning.
- Motivation: High-quality training data remains scarce because effective terminal-agent training requires genuinely difficult, multi-step tasks.Existing agents perform complex CLI tasks across software debugging, system administration, security analysis, and data engineering.
- Task synthesis: CLI-Universe defines candidates across domain, skill type, capability, and engineering pillar, then grounds them through evidence-guided research over real technical materials.Its inside-out design produces Dockerized environments grounded in realistic constraints and failure modes rather than retrofitting surface artifacts.
- Verification: Approximately two-thirds of candidates are rejected by rubric-gated tests, hint-conditional filtering, and fail-to-pass checks that enforce executable, high-fidelity supervision.The stages respectively construct tests and solutions independently, remove trivially solvable tasks, and verify a meaningful transition from an unsolved state to a solution.
- Empirical validation: 33.4% on Terminal-Bench 2 is achieved by Qwen3-32B fine-tuned on CLI-Universe-6K’s 6,000 trajectories, surpassing open-source-data models at ≤32B scale and several much larger open-weight models.Performance also scales monotonically with model size and generalizes to out-of-domain benchmarks.
- Verification: Removing any verification component costs 3–6 points on Terminal-Bench 2, indicating that each stage contributes to downstream performance.The ablation is reported as evidence that the verification pipeline concentrates useful training signal.
2 Related Work
Related work spans the rapid expansion of LLM-based agents from repository-level coding tasks to interactive terminal workflows, alongside synthetic-data methods that scale training tasks through taxonomies or existing infrastructure. These approaches broaden coverage and reuse executable environments, motivating more rigorous task construction and validation.
- Terminal Agents: LLM-based agents have progressed from repository-level issue resolution to interactive terminal environments supporting multi-step command-line workflows.Agent scaffolds increasingly enhance planning, execution, and tool use for this setting.
- Synthetic Data for Terminal Agents: Synthetic terminal-agent data is commonly scaled by generating tasks from LLM-produced skill or domain taxonomies.This strategy systematically enumerates capability dimensions to achieve broad topical coverage.
- Synthetic Data for Terminal Agents: A second strategy extracts tasks from existing infrastructure, including Docker environments harvested from open-source repositories.The passage identifies infrastructure reuse as a distinct alternative to taxonomy-based task generation.
3 Method
CLI-Universe synthesizes terminal-agent tasks through structured candidate specification, evidence-guided refinement, Dockerized environment realization, and executable verification. Its filtering conditions retain tasks whose tests and solution trajectories demonstrate meaningful, reproducible progress from failure to success.
- Task Candidate Specification: CLI-Universe organizes task candidates across domain, skill type, capability, and engineering pillar dimensions.Capability examples include exploration, error recovery, constraint satisfaction, and long-horizon planning; engineering pillars include feature creation, debugging, DevOps, and refactoring.
- Evidence-Guided Refinement: A research agent iteratively grounds candidates in repositories, documentation, issue discussions, tutorials, and usage examples.The resulting specifications incorporate realistic tools, constraints, failure modes, and concrete input/output contracts.
- Blueprint Formation and Validation: Validated blueprints record the user instruction, internal solution hint, and environment checklist before Dockerized realization.Only blueprints with sufficiently clear specifications and reliably verifiable setups proceed to environment construction.
- Environment Realization: Materialized assets are adapted, packaged with pinned dependencies and runtime configuration, and discarded when smoke tests fail.Smoke tests check installation, service startup, filesystem layout, and basic end-to-end reachability.
- Test Construction: 91% of official solutions passed synthesized tests on 89 Terminal-Bench2 tasks, while semantic match with official tests reached 88%.Semantic match was measured with Codex and a GPT-5.4 judge.
- Solution and Task Filtering: Hint-conditional and fail-to-pass filters retain tasks only when hint-free attempts fail, hinted solutions succeed, and tests transition from failing to passing.These checks remove trivially solvable tasks, vacuous tests, and unsupported solution trajectories.
4 Experiments
Experiments show that CLI-Universe training substantially improves Terminal-Bench performance, with gains increasing by model size and outperforming comparable open-source-data baselines. Ablations further show that task quality, verification components, successful-trajectory filtering, and scalable data contribute materially to performance and transfer.
- Main results: 33.4 on TB 2.0: CLI-Universe-32B outperforms SkillSynth-32B (29.6), Nemotron-Terminal-32B (27.4), and TerminalTraj-32B (22.0).The advantage also holds at 14B and 8B under the Terminus 2 scaffold.
- Scaling: +8.4 →+19.0 →+30.0: CLI-Universe raises Qwen3 baseline scores from 2.5 / 4.0 / 3.4 to 10.9 / 23.0 / 33.4 at 8B / 14B / 32B.The Qwen3 baselines remain essentially flat across sizes, while CLI-Universe gains increase monotonically.
- Pipeline ablations: −6.2: removing the asset strategy causes the largest ablation drop, followed by test-case rubrics (−3.9) and query rubrics (−3.4) relative to 26.7.These results come from a 1k-task subset using Qwen3-32B and indicate complementary contributions from environment diversity, query quality, and verification.
- Data ablations: 33.4 on TB 2.0: retaining 6k successful trajectories beats the unfiltered 10k set (28.2) by +5.2 points.Using the same CLI-Universe tasks, Kimi-K2.6 reaches 33.4 versus DeepSeek-V4-Pro at 31.2, showing robustness to teacher choice.
- Data efficiency: 33.4: CLI-Universe produces the highest Qwen3-32B TB 2.0 score versus Nemotron (28.9) and TerminalTraj (18.0), with a +30.0 gain over the 3.4 baseline.Qwen3-8B performance improves steadily as the number of CLI-Universe trajectories increases, without saturation at the largest budget.
5 Conclusion
CLI-Universe constructs terminal-agent training tasks through structured capability specification, evidence-guided deep research, and multi-stage executable verification. Fine-tuning Qwen3 on its trajectories yields consistent gains across 8B, 14B, and 32B students and transfers to BFCL-v4 and VitaBench.
- 5 Conclusion: CLI-Universe constructs terminal-agent training tasks using structured capability specification, evidence-guided deep research, and multi-stage executable verification.The pipeline combines task specification, research grounding, and executable validation.
- 5 Conclusion: Fine-tuning Qwen3 on CLI-Universe trajectories yields consistent gains across 8B, 14B, and 32B students.The reported gains span all three student model sizes.
- 5 Conclusion: The resulting models transfer to out-of-domain benchmarks including BFCL-v4 and VitaBench.The conclusion identifies transfer beyond the training-task setting.
6 Limitations
CLI-Universe relies on LLM-based agents throughout synthesis, so the quality ceiling of its data is bounded by the underlying models. Although it narrows the gap to proprietary systems at ≤32B scale, a performance gap to the strongest frontier models remains.
- Model Dependence: LLM-based agents drive ideation, environment construction, solution generation, and test construction.The pipeline depends on models across all four stages.
- Model Dependence: The quality ceiling of synthesized data is ultimately bounded by the capability of the underlying models.Rubric gating and executable verification do not remove this dependence.
- Performance Gap: Although CLI-Universe substantially narrows the gap to proprietary systems at ≤32B scale, a clear gap to the strongest frontier models remains.The passage identifies residual performance limitations despite the framework’s progress at this scale.
Contributions
The paper is authored by a multi-institutional team spanning five organizations, with several equal contributors and one corresponding author.
- Author contribution notes: Five authors are designated as equal contributors, and Jiaheng Liu is identified as the corresponding author.An asterisk marks equal contribution, while a dagger marks the corresponding author.
- Institutional affiliations: The authors are affiliated with Nanjing University, StepFun, ZODA, Huazhong University of Science and Technology, and Shanghai AI Lab.The affiliations correspond to indices 1–5 in the author list.
A Task Taxonomy … A.4 Engineering Pillars
CLI-Universe organizes terminal-agent tasks along four orthogonal dimensions—domain, skill type, capability, and engineering pillar—and uses this taxonomy to seed random candidate assignments. Each dimension constrains a distinct aspect of task realism, required knowledge, elicited reasoning, or engineering activity.
- A Task Taxonomy: The taxonomy spans Domain, Skill Types, Capabilities, and Engineering Pillars, distilled from patterns observed across TB-Agent tasks.These dimensions describe where a task lives, what knowledge it requires, which reasoning behaviors it elicits, and what engineering activity it entails.
- A Task Taxonomy: The taxonomy seeds random task-candidate assignment in the synthesis pipeline, linking structured task attributes to candidate generation.The four dimensions are used jointly to construct candidate combinations rather than merely to label completed tasks.
- A.1 Domain: Domains are human-curated entry points whose allowed values on the other dimensions keep random task combinations realistic within each application area.Each domain carries its own pool of permitted values for skill types, capabilities, and engineering pillars.
- A.2 Skill Types: Skill Types identify the single load-bearing specialized technical knowledge required by a task, excluding auxiliary skills such as light shell glue.A task may involve auxiliary skills, but they are not counted unless they constitute the specialized technical core.
- A.3 Capabilities: Capabilities represent the reasoning behaviors intended during environment interaction, with each capability evaluated by a strict TRIGGER / SIGNATURE / NOT rule.A task can carry multiple capabilities, allowing several reasoning behaviors to be assessed in one trajectory.
- A.4 Engineering Pillars: Engineering Pillars classify tasks by whether they build something new, fix something broken, or restructure something existing, with one dominant pillar per task.The pillar captures the primary engineering activity rather than every activity that may appear in the task.
B Training Details … C.4 Context Loss (Coherence)
The paper fine-tunes Qwen3 students with multi-turn SFT while analyzing executable-agent failures through turn-level timelines, revealing specification drift, repetition, termination unawareness, and context loss.
- B Training Details: Multi-turn SFT fine-tunes Qwen3 models, retaining failed and incomplete trajectories unless otherwise noted.Training hyperparameters and hardware setup are summarized in Table 7.
- C Failure Mode Examples: Failure modes are illustrated with CLI-Universe-32B trajectories on Terminal-Bench 2 through turn-level timelines.Each turn consists of reasoning, a command, and an observation.
- C.1 Disobey Task Specification (Execution): 21 turns show an agent replacing correct file-path handling with directory semantics and never testing the specified CSV path.The task required /app/test_output.csv, but later rewrites and all four test invocations used directory handling or /app/output.
- C.2 Step Repetition (Execution): 44–165 repeated curl, find, and grep commands loop without convergence across a 357-turn POV-Ray compilation task.Reasoning mentions strategy changes, but the actual commands remain unchanged.
- C.3 Unaware of Termination (Execution): 159 turns and 163 commands yield 0 successes because duration requests cannot overcome the hard 120s timeout ceiling.The agent escalates requested duration from 1s to 72,000s, and even echo hello fails.
- C.4 Context Loss (Coherence): Context loss also produces repeated diagnoses and patches, with the same evidence and parser reconsidered instead of serving as a stable base.The agent re-examines identical ELF facts and rewrites vm.js with the same parser.
- C.4 Context Loss (Coherence): The agent repeatedly re-derives established facts, including entry point 0x400110 at least 4 times, a write8 bug twice, and endianness 3 times.This persists despite hardcoding constants in vm.js at Turn 22.
C.5 Task Derailment (Coherence) · C.6 Reasoning–Action Mismatch (Coherence) · C.7 Premature Termination (Verification)
The three cases expose distinct agent failures: abandoning implementation for diagnostics, reasoning that contradicts executed code, and declaring success despite invalid verification. Together, they show how coherence and verification breakdowns leave core deliverables incomplete or incorrect.
- C.5 Task Derailment (Coherence): C.5: The agent’s actions remained diagnostic throughout the task, despite the requested deliverable being a regex-based /app/re.json.The task required regex pairs for chess move generation via re.sub.
- C.6 Reasoning–Action Mismatch (Coherence): C.6: The mismatch made the correct differential insight ineffective because the executed implementation never followed the backward reasoning.The agent did find the correct differential, but the code continued using the incompatible forward search.
- C.6 Reasoning–Action Mismatch (Coherence): C.6: The agent repeatedly rewrote attack.py, but each attempt remained forward brute-force rather than implementing the stated backward plan.The final code still looped over seed in range(65536) and used getright() with seed*1234567.
- C.6 Reasoning–Action Mismatch (Coherence): C.6: From Turn 2 across 35 turns and 3 retraction cycles, reasoning described backward differential propagation while embedded code used forward brute-force.The contradiction persisted through repeated explanations and rewrites.
- C.7 Premature Termination (Verification): C.7: Across all 5 code versions, the book-owner update used offset 0–3 instead of 24–27, corrupting the book-ID field.The intended owner field occupied offset+24:offset+28.
- C.7 Premature Termination (Verification): C.7: The verification script raised a NameError, the agent acknowledged it without rerunning corrected verification, and still declared task_complete: true.No valid comparison result was produced, so the persistent wrong-offset bug remained undetected.
C.8 No or Incorrect Verification (Verification) · C.9 Weak Verification (Verification)
The examples expose verification failures that declare success without exercising the intended failure path or rechecking all required properties. In one case, the wrong exception path masks a real SIGINT race; in another, only 4/11 properties are verified after invalidated and failed checks.
- C.8 No or Incorrect Verification (Verification): The tests were structurally wrong because they did not simulate SIGINT and raised TimeoutError rather than KeyboardInterrupt.Both failures prevented validation of the intended interrupt-cleanup behavior.
- C.8 No or Incorrect Verification (Verification): “Cleanup verified via timeout simulation” incorrectly marked the task complete even though the SIGINT path was unverified.The agent reaffirmed completion despite the verification mismatch.
- C.8 No or Incorrect Verification (Verification): The real bug was that sentinels were queued behind existing items, so the race condition was never exposed.This bug remained hidden after the agent declared completion.
- C.8 No or Incorrect Verification (Verification): 3 test iterations each exercised the wrong exception path, so the real SIGINT race condition was never triggered.asyncio.wait_for raises TimeoutError, not KeyboardInterrupt.
- C.9 Weak Verification (Verification): Malformed commands never executed, yet Turn 26 declared task_complete: true without rechecking numpy or verifying cinvariants.This combined execution failure with an unsupported completion claim.
- C.9 Weak Verification (Verification): 4/11 properties were verified for the Cython extension task.The build timed out, and later verification commands contained JSON escaping errors and never ran.
- C.9 Weak Verification (Verification): The --force-reinstall at Turn 13 invalidated the numpy check from Turn 1, while cinvariants timed out and was never re-verified.The initial numpy version check therefore could not establish the final environment state.