CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion

Motivation

• Environment-intensive tasks are neglected despite CLI agents’ weak performance: models with hundreds of billions of parameters resolve fewer than 40% of Terminal-Bench tasks, versus over 70% on SWE-bench.Unlike code histories, runtime environments vary across developers, Dockerfiles lack rich histories and context, and repository issues rarely identify environment-intensive tasks.Source: §1 · p. 2

Method

• CLI-Gym derives tasks by having an agent explore command-based environment histories from a healthy state, use execution feedback to reach unit-test failures, and package the resulting error messages.This models Dockerfile command sequences as agentic actions and differs from Dockerfile generation without feedback.Source: §1 · p. 2

Results

• +21.1% (to 46.1%) on Terminal-Bench 1.0 and +12.9% (to 31.0%) on Terminal-Bench 2.0 were achieved by fine-tuned Qwen3-235B-A22B-Instruct, outperforming larger open-source models.The pilot fine-tuning used 291 successful task trajectories.Source: §1 · p. 3

Contributions

• The pipeline produces 1,655 environment-intensive tasks from 29 open-source repositories, nearly 20× the dozens of manually labeled Terminal-Bench tasks.The authors also present the first publicly available pipeline for scalable derivation of these tasks.Source: §1 · p. 3

Agentic Coding via CLI

• Environment-intensive CLI tasks center on interacting with systems, but their construction remains reliant on human-written instances and a fragmented, largely closed ecosystem.Code-intensive tasks instead emphasize writing code and have benefited from large-scale PR-based task environments.Source: §2 · p. 3

Scaling Training Environment for Agentic Coding

• Code-intensive tasks have scalable executable environments built from pull requests or synthetic faults, whereas Terminal-Bench lacks open pipelines and closed-source models dominate its leaderboard.Executable, verifiable environments provide success signals for training and evaluation; the lack of public data pipelines limits open-source progress.Source: §2 · p. 3

3. Method

• CLI-Gym inverts conventional agentic coding by having an agent degrade a gold environment into a poor state, creating tasks that simulate environment histories.This inverse perspective enables scalable synthesis of CLI task instances for improving environment-intensive agentic coding.Source: §3 · p. 3, §3.2 · p. 5

3.1. Agentic Coding from an Environment Perspective

• Agentic coding is formalized as modifying an environment configuration comprising a base image, Dockerfile, and codebase, from a state failing at least one test toward one passing all tests.The gold and poor states distinguish environments where all unit tests pass from those where at least one fails.Source: §3.1 · p. 3

3.2. Instance Construction by Environment Inversion

• Starting from a gold environment, CLI-Gym uses test-derived prompts and prior-task summaries to guide Dockerfile edits that induce failures, then validates the degraded environment and generates an issue description from failing tests and error feedback.The resulting task comprises an executable environment, a natural-language issue description, and unit tests; the synthesized Dockerfile records the degradation trajectory for reproducibility.Source: §3.2 · pp. 4–5

3.3. CLI-Gym

• From 29 Python repositories, CLI-Gym generated 1,655 CLI tasks and collected 417 successful trajectories, retaining 291 after filtering for training.The tasks span application domains and include more unit tests than Terminal-Bench; the filtered trajectories cover dependency resolution, configuration debugging, and system-level troubleshooting.Source: §3.3 · p. 5

4.2. Main Results

• 46.1 on Terminal-Bench 1.0 made LiberCoder-235B-A22B the leading open-weight model; its 31.0 on 2.0 surpassed most open-weight baselines.LiberCoder-32B and LiberCoder-235B-A22B improved over their corresponding Qwen base models by +28.6% and +21.1% on Terminal-Bench 1.0, and +13.8% and +12.9% on 2.0, respectively.Source: Table 3 · p. 6, §4.2 · p. 7

4.1. Experimental Setups

• OpenHands was used for environment corruption and task completion, with evaluation on Terminal-Bench 1.0 (80 tasks) and 2.0 (89) using verification-based pass@1 and pass@3.Training first used 48K open-source SWE trajectories, disjoint from Terminal-Bench, then fine-tuned on Filtered-Success trajectories.Source: §4.1 · pp. 6–7

4.3. Ablation studies

• After agentic coding pretraining, filtered high-quality trajectories outperformed the larger unfiltered set; without pretraining, 417 unfiltered trajectories performed slightly better than 291 filtered ones.The results suggest trajectory quality matters more than quantity once agentic capabilities are established.Source: Table 5 · p. 7, §4.3 · p. 8

4.4. Results and Failure Case Visualization

• Training reduced editing- and localization-related errors, but increased extensive exploration and the risk of exceeding the 128k inference-context limit.The comparison concerns Qwen3-235B-A22B-Instruct before and after training on CLI-Gym data.Source: Figure 5 · p. 7, §4.4 · p. 8
• CLI-Gym training improved performance by more than 20 points across software engineering, system administration, security, and debugging, while gaming and scientific computing remained challenging.The authors identify those remaining categories as future research directions.Source: Figure 6 · p. 7, §4.4 · p. 8

4.3. Ablation studies

• Combining SWE-style pretraining with CLI-Gym trajectories achieved the best performance across model scales, while CLI-Gym-only training outperformed the corresponding base models.SWE-style agentic trajectories also improved performance over base models at both 32B and 235B scales.Source: §4.3 · pp. 7–8, Table 4 · p. 7
• With 100 training trajectories fixed, performance improved monotonically as the number of source repositories increased, supporting environment diversity as a factor in robust behavior.The ablation varied repository count while holding the total trajectory budget constant.Source: Figure 7 · p. 8, §4.3 · p. 8
• Performance rose as more filtered CLI-Gym trajectories were added, then plateaued beyond approximately 200 trajectories.The scaling experiment kept the open-source agentic trajectories fixed; the plateau suggests data quality and task diversity may matter more than quantity.Source: Figure 8 · p. 8, §4.3 · p. 8

5. Conclusion

• 1,655 task instances and 291 successful trajectories were curated, and fine-tuning on the data achieved top-tier Terminal-Bench performance among open-source models.The paper presents CLI-Gym as its first publicly available approach for scaling training environments of CLI agentic coding tasks.Source: §5 · p. 8

A. Detailed CLI-Gym Pipeline

• The pipeline constructs gold instances from 29 open-source GitHub repositories, each comprising an environment, codebase, and unit tests.Repository information, unit-test counts, and task-instance totals are summarized in Table 9.Source: A. Detailed CLI-Gym Pipeline · p. 11

A.1. Query Generation

• For initial query generation, the pipeline samples 1–3 intervention directions and 200 repository unit tests, prompting an LLM with those tests and previous task titles.The resulting initial task prompt is generated after the sampled material is included in the prompt.Source: §A.1 · p. 11
• A randomly applied second-stage refinement aligns generated task prompts more closely with selected unit tests, improving yield and diversity.Source: §A.1 · p. 11
• Each generated prompt is packaged as an agent task with Docker Compose trajectory and log mapping, a base-image Dockerfile, a test script, and task.yaml.Source: §A.1 · p. 11

A.2. Environment Inversion

• The modified Terminal-Bench harness runs each task for approximately 15 minutes, prompting the agent to induce failures and interact with the environment before terminating at its final thought.Docker Compose mounts trajectories and logs and invokes the Dockerfile, which launches the agent.Source: §A.2 · p. 11
• Test evaluation labels failed tests fail-to-pass and successful tests pass-to-pass; if the test command fails, all selected tests are fail-to-pass, while all-pass tasks are discarded.Source: §A.2 · p. 11
• The harness also prompts the agent to summarize a Dockerfile capturing the degraded environment so the constructed failure can be reproduced deterministically.Source: §A.2 · p. 11

A.3. Task Generation

• Three randomly selected prompt styles generate issue descriptions with stronger, weaker, or balanced guidance from the original task prompt and failed unit-test information.The styles correspond to prompts shown in Figures 14–16.Source: §A.3 · p. 11
• Each generated statement includes a hint that a rule-based filter can remove, yielding hinted and unhinted repair tasks from the same issue.Source: §A.3 · p. 11
• The problem template combines the generated statement with extracted unit tests and a Dockerfile; the pipeline uses OpenHands and supplies run-tests.sh templates.Source: §A.3 · p. 11

B. Detailed Experiments

• The section details the full experimental configurations and execution procedures.Source: B. Detailed Experiments · p. 12

• 4,066 prompts from 29 gold instances yielded 1,655 problem instances, including faulty images, failed unit tests, and failure-inducing Dockerfiles.Source: §B.1 · p. 12
• 291 of 417 successful trajectories were retained for training after filtering out 126.Source: §B.1 · p. 12
• Trajectories with fewer than 20 steps and those exploiting unintended shortcuts, such as cached Git information or Conda logs, were discarded.The filtering aimed to exclude trivial or low-difficulty tasks and solutions that bypassed intended problem-solving.Source: §B.1 · p. 12

B.2. Training Details

• Qwen3-32B training uses a 2 × 10−5 cosine-decayed learning rate, 5% linear warmup, batch size 16, and best-validation checkpoint selection across 10, 15, or 20 epochs.The qwen3-coder template is used, with a 100k-token maximum sequence length achieved by extending the native 40k context window using YaRN at expansion factor 2.5.Source: §B.2 · p. 12
• Qwen3-235B-A22B-Instruct training uses a 1 × 10−5 cosine-decayed learning rate, 5% linear warmup, and best-performing checkpoint selection across 10, 15, or 20 epochs.The qwen3-coder template and 100k-token maximum sequence length are used; warmup increases the learning rate from 1 × 10−6 to 1 × 10−5.Source: §B.2 · p. 12

B.3. Agent and Inference Details

• OpenHands provides the Agent–Computer Interface for single autonomous agents to interact with CLI environments through structured calls and iteratively act on environment feedback.Available interactions include shell execution, file editing, and environment inspection; an agent acts until solving the task or reaching a termination condition.Source: §B.3 · p. 12
• Agents invoke the tools in Figure 19, with all actions executed inside isolated Docker containers constructed for each task.Source: §B.3 · p. 12
• Episodes terminate when the agent calls the finish tool or exceeds Terminal-Bench’s global time limit.Source: §B.3 · p. 12
• Inference uses greedy decoding with temperature = 0 and a maximum context length of 128k tokens, applied uniformly to all models for fair comparison.Source: §B.3 · p. 12

B.4. Ablation Studies and Case Visualization

• SWE-style pretraining trajectories exclude all Terminal-Bench tasks and initialize general agentic coding abilities rather than environment-specific knowledge.The training uses one epoch of SWE-style pretraining before fine-tuning on Filtered-Success trajectories.Source: §B.4 · p. 12
• Training first uses one epoch of SWE-style trajectories, then fine-tunes on Filtered-Success trajectories for 15 epochs with other hyperparameters unchanged.The unchanged settings include learning rate, batch size, context length, optimizer, and agent interface.Source: §B.4 · p. 12
• Repository-scaling experiments uniformly sample trajectories across repositories to balance coverage and train each dataset at its best-performing epoch count.Other training settings are held constant across datasets.Source: §B.4 · p. 13
• Trajectory-quality comparisons use each dataset’s best-performing configuration, including 15 epochs for Filtered-Success and 10 for Raw-Success.Other factors are held constant to attribute differences to data quality rather than training instability.Source: §B.4 · p. 13
• Category-wise analysis compares Qwen3-32B with LiberCoder-32B on Terminal-Bench@1.0 using pass@3 across official task categories.The passage frames pass@3 as a measure of the models’ capacity.Source: §B.4 · p. 13
• Failure analysis compares Qwen3-235B-A22B-Instruct with LiberCoder-235B-A22B by categorizing unresolved tasks from full Terminal-Bench@1.0 evaluation runs.Source: §B.4 · p. 13

Impact of Different Agents

• Sampled officially validated Terminal-Bench@2.0 examples indicate that the OpenHands agent used by the authors did not perform very well.Table 6 presents the performance comparison; the supplied passage does not provide its scores.Source: Table 6 · p. 13, Impact of Different Agents · p. 13

Comparison with other datasets

• CLI-Gym generates many CLI-centric tasks from a small set of base images, resulting in a substantially lower storage footprint.Table 7 compares representative datasets by task instances, base images or environments, and storage size.Source: Impact of Different Agents · p. 13, Table 7 · p. 13

Gain with Hint

• Hints increased usable trajectories from 104 to 291 and improved downstream performance, while hint-augmented and unaugmented data performed comparably at 104 trajectories.The matched-size comparison subsampled the full dataset to 104 trajectories.Source: Impact of Different Agents · p. 14, Table 8 · p. 14

Performance Benefits Model Behavior

• As filtered-success trajectories increased, pass@1 performance steadily improved while looped behavior fell from 42.7% to 3.0%.The reported pattern suggests environment-repair supervision helps agents avoid repetitive action loops and maintain effective long-horizon control.Source: Figure 9 · p. 14, Impact of Different Agents · p. 14