CLI-Universe: Towards Verifiable Task Synthesis Engine for Terminal Agents

1 Introduction

• Existing synthesis pipelines often retrofit surface materials into tasks, producing ambiguous queries, shallow execution paths, or brittle tests that provide weak learning signals.The paper argues these pipelines scale task sources rather than task quality.Source: §1 · p. 1
• CLI-Universe defines candidates across domain, skill type, capability, and engineering pillar, then researches real technical materials to ground blueprints in realistic tools, constraints, and failure modes.Validated blueprints are instantiated in Dockerized environments with materialized assets and runtime state.Source: §1 · pp. 1–2
• CLI-Universe combines rubric-gated tests, hint-conditional filtering, and fail-to-pass checks, rejecting approximately two-thirds of candidates; removing any stage costs 3–6 Terminal-Bench 2 points.Role-separated agents independently construct tests and reference solutions; filtering retains tasks only when the internal hint is genuinely necessary, and fail-to-pass checks verify a meaningful state transition.Source: §1 · pp. 1–2
• 33.4% on Terminal-Bench 2: Qwen3-32B fine-tuned on 6,000 CLI-Universe trajectories surpasses all open-source-data-trained models at ≤32B and several models an order of magnitude larger.Performance scales monotonically with model size and generalizes to out-of-domain benchmarks including BFCL v4 and VitaBench (He et al., 2025).Source: §1 · p. 2

Terminal Agents

• Terminal agents have expanded from repository-level issue resolution to multi-step CLI workflows, while open-source models lag proprietary systems and face a high-quality training-data bottleneck.Terminal-Bench evaluates agents on hand-crafted tasks across diverse domains in Docker environments.Source: §2 · p. 2

Synthetic Data for Terminal Agents

• Existing synthesis scales through LLM-generated skill or domain taxonomies and infrastructure-derived tasks, but offers limited per-task verification and training-signal guarantees.Infrastructure-derived tasks include harvested Docker environments and debugging scenarios created by transforming working configurations into faulty states.Source: §2 · p. 2
• CLI-Universe complements prior strategies by grounding structured capability candidates in technical research and retaining Dockerized tasks only after rubric-gated tests and fail-to-pass checks.A test agent and solution agent independently verify each task before retention.Source: §2 · p. 2, Figure 1 · p. 3

3.2 Task Blueprint Construction

• Task blueprints sample skill type, capability, and engineering-pillar combinations within a domain, then refine candidates through research grounded in real technical materials.Evidence-guided refinement requires 3.45× more solver turns and reduces pass rate by 13.3 points versus unrefined tasks; rubric-based review raises acceptance from 72% to 91% for human judges and 75% to 93% for LLM judges. Candidates that cannot be adequately grounded are discarded.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.1 Overview

• Only 33.6% of candidates survive CLI-Universe’s end-to-end synthesis and filtering pipeline.The pipeline builds and validates task blueprints, realizes executable environments, and applies executable filtering.Source: §3.1 · p. 3, Figure 2 · p. 4

3.4 Test Construction and Executable Filtering

• Synthesized tests agree with Terminal-Bench 2 ground truth on 91% of sampled tasks, with 88% semantic match to official tests.The comparison applies the test-construction pipeline to 89 Terminal-Bench2 tasks; official solutions pass the synthesized tests on 91% of sampled tasks.Source: Figure 2 · p. 4, §3.4 · p. 5

3.3 Environment Realization

• Validated blueprints become Dockerized environments built from acquired or synthesized assets, pinned dependencies, specified configurations, and smoke-tested services and filesystem layouts.Retrieved assets may be normalized, fault-injected, reconfigured, or scoped to the intended workflow; environments that fail installation, startup, layout, or reachability checks are discarded.Source: §3.3 · pp. 4–5

3.4 Test Construction and Executable Filtering

• Tasks are retained only when a hint-guided solution passes executable tests that initially fail, while a hint-free attempt fails.Rubric-checked tests assess correctness, determinism, and edge cases; successful solution trajectories are retained as training data only when consistent with the test signal.Source: §3.4 · p. 5

4.1 Main Results

• 33.4 on Terminal-Bench 2.0: CLI-Universe-32B surpasses open-source trajectory-trained models at ≤32B, including SkillSynth-32B (29.6), Nemotron-Terminal-32B (27.4), and TerminalTraj-32B (22.0).It also exceeds 480B Qwen3-Coder (23.9) and 1T Kimi-K2-Instruct (27.8), while remaining below Claude-Opus-4.5 (57.8).Source: §4.1 · p. 5, Table 1 · p. 6
• 33.4 after CLI-Universe training at 32B, versus 3.4 for the Qwen3 baseline; gains rise from +8.4 at 8B to +30.0 at 32B.Qwen3 baselines score 2.5 / 4.0 / 3.4 at 8B / 14B / 32B, while CLI-Universe models score 10.9 / 23.0 / 33.4, indicating continued gains with model size.Source: §4.1 · pp. 5–6, Figure 3 · p. 7

4.2.1 Effect of different components

• On a 1k-task subset, removing asset strategy, query rubrics, and test-case rubrics reduces TB 2.0 from 26.7 to 20.5, 23.3, and 22.8, respectively.Asset strategy removal is most damaging; the results indicate that environment diversity, query quality, and high-fidelity verification contribute complementarily.Source: §4.2.1 · pp. 6–7, Figure 3 · p. 7

4.2.2 Effect of different selection strategies

• 33.4 on TB 2.0: keeping 6k successful trajectories outperforms the unfiltered 10k set (28.2) by +5.2 points.The comparison suggests that failed or incomplete interactions add noise, making correctness filtering more important than raw trajectory volume at this model scale.Source: Table 2 · p. 6, §4.2.2 · p. 7

4.2.3 Effect of different teachers

• 33.4 on TB 2.0 with Kimi-K2.6 trajectories, compared with 31.2 using DeepSeek-V4-Pro trajectories on the same tasks and at the same 6k volume.The result suggests strong supervision does not require a specific frontier teacher.Source: Table 2 · p. 6, §4.2.3 · p. 7

4.3.1 Data scaling

• Qwen3-8B performance improves steadily as CLI-Universe training data grows, with no sign of saturation at the largest tested budget.The authors interpret this as evidence that the pipeline can continue generating useful supervision and that this data regime has not reached diminishing returns.Source: §4.3.1 · p. 7

4.3.2 Data efficiency

• 33.4 on TB 2.0 from CLI-Universe’s 6k trajectories, versus 28.9 for Nemotron and 18.0 for TerminalTraj at matched volume.CLI-Universe gains +30.0 over the 3.4 Qwen3-32B baseline, compared with +25.5 for Nemotron and +14.6 for TerminalTraj, supporting higher per-trajectory information density.Source: §4.3.2 · p. 7, Figure 3 · p. 7

4.4.1 Cross-benchmark transfer

• 58.0 on BFCL v4 and 27.0 on VitaBench for CLI-Universe-32B, exceeding Qwen3-32B by +11.3 and +11.6, respectively.The passage attributes this cross-benchmark transfer to broadly applicable tool orchestration, environment-state tracking, and multi-step planning skills.Source: §4.4.1 · p. 8, Figure 4 · p. 8

• CLI-Universe combines structured capability specification, evidence-guided deep research, and multi-stage executable verification to construct terminal-agent training tasks.Source: §5 · p. 9
• Fine-tuning Qwen3 on CLI-Universe trajectories yields consistent gains across 8B, 14B, and 32B students.Source: §5 · p. 9
• The resulting fine-tuned models transfer to the out-of-domain benchmarks BFCL-v4 and VitaBench.Source: §5 · p. 9
• The results suggest carefully constructed, independently verified tasks can provide strong supervision for terminal agents even at limited data scale.Source: §5 · p. 9

6 Limitations

• The synthesized data quality ceiling is bounded by the capabilities of LLM-based agents used for ideation, environment construction, solution generation, and test construction.Rubric gating and executable verification do not remove this dependence.Source: §6 · p. 9
• Although CLI-Universe narrows the gap to proprietary systems at ≤32B scale, a performance gap to the strongest frontier models remains.The work leaves open whether larger open-source base models or reinforcement-learning-based training can further close the gap.Source: §6 · p. 9
• The current dataset contains only 6k trajectories, and scaling to larger, more diverse task pools may unlock further gains.The authors leave this scaling exploration to future work.Source: §6 · p. 9

A Task Taxonomy

• The four-dimensional taxonomy seeds candidate assignment so tasks in the same domain can exercise different capability patterns.The dimensions are Domain, Skill Types, Capabilities, and Engineering Pillars.Source: A Task Taxonomy · p. 13

A.1 Domain

• Domains are human-curated application areas that serve as the synthesis pipeline’s entry point.Each domain has its own allowed values for the other taxonomy dimensions.Source: §A.1 · p. 13
• Domain-specific value pools constrain random dimension combinations to keep candidate tasks realistic within each application area.Source: §A.1 · p. 13

A.2 Skill Types

• Each task receives one load-bearing skill type; auxiliary skills, such as light shell glue, do not count.Source: §A.2 · p. 14

A.3 Capabilities

• Capabilities describe intended reasoning behaviors, allow multiple tags per task, and are judged from trajectories using TRIGGER, SIGNATURE, and NOT rules.Source: §A.3 · p. 14

A.4 Engineering Pillars

• Each task receives one dominant engineering pillar: building something new, fixing something broken, or restructuring something existing.Source: §A.4 · p. 15

B Training Details

• Qwen3 models are fine-tuned with multi-turn SFT, retaining failed and incomplete trajectories unless otherwise noted; trajectory selection is studied in Section 4.2.2.Training hyperparameters and hardware setup are summarized in Table 7.Source: B Training Details · p. 16, Table 7 · p. 16

C Failure Mode Examples

• Failure modes are illustrated with CLI-Universe-32B trajectories on Terminal-Bench 2 as turn-level timelines, where each turn is reasoning, command, then observation.Source: C Failure Mode Examples · p. 16

C.1 Disobey Task Specification (Execution)

• In the sam-cell-seg task, the specification required convert_masks.py to treat --output_path /app/test_output.csv as a CSV file path.The trajectory lasted 21 turns.Source: §C.1 · p. 16
• The task-specification violation was never exposed, and the agent nevertheless marked the task fully complete.Source: §C.1 · p. 16
• Although the Turn 6 draft handled output_path as a file, from Turn 12 onward the agent used directory semantics with os.makedirs and os.path.join.Source: §C.1 · p. 16
• All four test invocations used --output_path /app/output rather than the specified /app/test_output.csv, so the required file path was never tested.Source: §C.1 · p. 16

C.2 Step Repetition (Execution)

• The agent repeated the same curl command after recognizing it should switch strategies, then continued searching the filesystem without changing state.The same filesystem state and file listings recurred.Source: §C.2 · p. 17
• Across 357 turns, the agent repeated curl, find, and grep commands 44–165 times each without changing its actions or converging.Its reasoning mentioned strategy changes, but the commands remained the same.Source: §C.2 · p. 17

C.3 Unaware of Termination (Execution)

• Despite repeated exit-124 timeouts, the agent increased requested durations from 60 to 1,200 seconds while the 120-second ceiling remained unchanged.It later attributed the failure to an extremely slow environment and requested 3,600 seconds.Source: §C.3 · p. 17
• 159 turns and 163 commands produced 0 successes as the agent escalated requested durations from 1 to 72,000 seconds against a hard 120-second ceiling.Even the trivial command “echo hello” timed out; condensing commands into one script did not help.Source: §C.3 · p. 17

C.4 Context Loss (Coherence)

• After committing constants to vm.js, the agent repeatedly reran diagnostics and rewrote the file with the same values.It re-read bytes, reran readelf, and rechecked the entry point and BSS layout.Source: §C.4 · p. 18
• Near the end of the 300-turn task, the agent rewrote vm.js with the same ELF parser instead of building on the parser it had already established.It summarized the ELF structure and entry point before repeating the rewrite.Source: §C.4 · p. 18
• The agent repeatedly lost established facts: it rechecked entry point 0x400110 at least four times, patched the same write8 omission twice, and re-derived endianness three times.These repetitions continued despite the relevant constants being hardcoded in vm.js at Turn 22.Source: §C.4 · p. 18

C.5 Task Derailment (Coherence)

• The chess agent’s manual FEN parsing led to rank-order confusion and further probes on castling and en passant instead of implementation.The task required regex pairs for chess move generation via re.sub.Source: §C.5 · p. 18
• On the regex-chess task, the agent spent 10 turns probing chess moves and FEN notation but never began writing the required /app/re.json deliverable.Its Python-chess diagnostics included move enumeration and FEN inspection.Source: §C.5 · p. 18

C.6 Reasoning–Action Mismatch (Coherence)

• The FEAL agent’s initial attack.py used brute-force over a large key space, too slowly; despite discussing a backward approach, its later attack.py search was still forward.Source: §C.6 · p. 19
• Across 35 turns and 3 retraction cycles, backward differential reasoning never corrected the embedded forward brute-force code.The final attack still used a forward brute-force search, despite repeated backward analyses and rewrites.Source: §C.6 · p. 19

C.7 Premature Termination (Verification)

• After its comparison script raised a NameError and produced no valid result, the agent acknowledged the error but declared completion without rerunning verification.The output still had a corrupted book-ID field.Source: §C.7 · p. 19
• The COBOL modernization agent used the book-ID byte offset instead of the owner-field offset in all five code versions, leaving the core bug uncaught.The correct owner field was at bytes 24–27, but the code repeatedly wrote to bytes 0–3.Source: §C.7 · p. 19

C.8 No or Incorrect Verification (Verification)

• The agent nevertheless declared cleanup verified and the task complete after confirming timeout-path cleanup, not SIGINT cleanup.The test was structurally wrong for simulating SIGINT, despite the claim that all five cleanups ran.Source: §C.8 · p. 20
• The cancellation task’s real race—sentinels queued behind pending items and never exposed—remained untested.The task required correct SIGINT/KeyboardInterrupt cleanup, but the implementation’s race condition was never triggered.Source: §C.8 · p. 20
• All three test iterations used asyncio.wait_for, which raises TimeoutError rather than KeyboardInterrupt and therefore never exercised the SIGINT path.The tests could confirm timeout-path cleanup, but not cleanup after SIGINT.Source: §C.8 · p. 20

C.9 Weak Verification (Verification)

• The --force-reinstall step could overwrite numpy and invalidated the earlier numpy 2.3.0 check, but the version was not rechecked.The agent checked the version before reinstalling, then marked the task complete without verifying it again.Source: §C.9 · p. 20
• Although 18 pytest tests passed, that result did not establish all evaluator properties or verify the timed-out cinvariants build.The cinvariants build timed out and was never re-verified.Source: §C.9 · p. 20
• Only 4 of 11 evaluator properties were verified for the Cython build task.The build of cinvariants timed out, and the final verification commands had JSON escaping errors and never ran.Source: §C.9 · p. 20