PaperlayerView exact revision on arXiv

Source-linked AI summary

CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion

Yusong Lin, Haiyang Wang, Shuzhe Wu, Lue Fan, Feiyang Pan, Sanyuan Zhao, Dandan Tu

arXiv:2602.10999v1cs.AI

TL;DR

Environment-intensive CLI tasks are underexplored and difficult to obtain at scale for agentic coding. CLI-Gym inverts healthy environment histories to derive buggy task instances, and fine-tuning on curated trajectories improves Terminal-Bench performance by +21.1% to 46.1%.

  • Problem

    Environment-intensive CLI tasks are underexplored in agentic coding, despite requiring agents to resolve dependencies, repair configurations, and fix environment variables.

  • Method

    CLI-Gym uses agents to invert healthy Dockerfile-defined environment histories into buggy states with corresponding errors, enabling scalable task derivation.

  • Results

    +21.1% to 46.1%: fine-tuning on 291 successful trajectories improves Terminal-Bench 1.0 performance for LiberCoder-235B-A22B over its base model.

  • Takeaways & Limitations

    CLI-Gym provides a public pipeline and 1,655-task collection for training environment-centric coding agents, achieving top-tier open-weight performance on Terminal-Bench.

Abstract

from arXiv · show

Agentic coding requires agents to effectively interact with runtime environments, e.g., command line interfaces (CLI), so as to complete tasks like resolving dependency issues, fixing system problems, etc. But it remains underexplored how such environment-intensive tasks can be obtained at scale to enhance agents' capabilities. To address this, based on an analogy between the Dockerfile and the agentic task, we propose to employ agents to simulate and explore environment histories, guided by execution feedback. By tracing histories of a healthy environment, its state can be inverted to an earlier one with runtime failures, from which a task can be derived by packing the buggy state and the corresponding error messages. With our method, named CLI-Gym, a total of 1,655 environment-intensive tasks are derived, being the largest collection of its kind. Moreover, with curated successful trajectories, our fine-tuned model, named LiberCoder, achieves substantial absolute improvements of +21.1% (to 46.1%) on Terminal-Bench, outperforming various strong baselines. To our knowledge, this is the first public pipeline for scalable derivation of environment-intensive tasks.

1. Introduction

CLI-Gym addresses the lack of scalable environment-intensive coding tasks by simulating environment histories through agentic exploration and state inversion. It produces a large task collection and successful trajectories for improving CLI-capable coding agents.

  • Motivation: Environment-intensive tasks involve complex runtime interactions, but existing agentic-coding research largely neglects them and LLM agents achieve less than 40% resolution on Terminal-Bench.Examples include resolving dependency issues, repairing configurations, and fixing broken environment variables.
  • Motivation: Environment-intensive tasks cannot be reliably derived from repository histories because runtime environments vary across developers and lack comprehensive centralized histories.Dockerfiles share containerized environments but do not provide rich environment histories.
  • CLI-Gym: CLI-Gym simulates environment histories by having an agent execute commands from a healthy environment, use execution feedback, and invert resulting states into tasks.The method uses Dockerfiles as representations of environments and formulates task derivation as an agentic exploration problem.
  • Results: 1,655 environment-intensive task instances are derived from 29 popular open-source repositories, alongside 291 successful trajectories for fine-tuning studies.The collection is described as nearly 20× larger than the dozens of manually labeled Terminal-Bench tasks.

2. Related Work

Related work distinguishes code-intensive from environment-intensive agentic coding and highlights the importance of executable, verifiable environments for training and evaluation. Existing scalable environment construction is more mature for code-intensive tasks, while environment-intensive settings remain less scalable.

  • Agentic Coding via CLI: Agentic coding tasks divide into code-intensive tasks centered on writing code and environment-intensive tasks centered on interacting with runtime environments.
  • Scaling Training Environment for Agentic Coding: Executable and verifiable environments provide reliable success signals for agentic-coding training and evaluation.
  • Scaling Training Environment for Agentic Coding: SWE-gym, R2E-gym, and SWE-smith scale code-intensive tasks by crawling pull requests or injecting synthetic faults, whereas environment-intensive settings such as Terminal-Bench lack comparable scalability.

3. Method

CLI-Gym treats agentic coding as environment-state transformation and reverses this process to generate defective, reproducible CLI tasks from functional repository environments. Agents induce failures through Dockerfile and code perturbations, then package execution feedback into task descriptions and tests.

  • Environment-Centric Formalization: CLI-Gym models agentic coding as transforming a defective environment Spoor into a gold environment Sgold where all unit tests pass.The environment includes a base image, Dockerfile, and codebase; Spoor fails at least one unit test.
  • Environment Inversion: Data collection reverses this process by starting from Sgold and applying structured environment-level and code-level perturbations to induce failures and create CLI task instances.The method frames scalable training-data construction as agentic environment inversion.
  • Environment Inversion: An autonomous agent generates Dockerfile commands that manipulate filesystems, virtual environments, dependencies, and system state to violate selected unit tests.The resulting Dockerfile records the degradation trajectory for reproducible reconstruction and can require system-level diagnosis beyond code repair.
  • Problem Instance Construction: A perturbed environment becomes a valid task when execution produces failing tests, which are converted into fail-to-pass evaluations and error feedback for issue-description synthesis.Each task contains an executable environment, a natural-language issue description, and unit tests.
  • Pipeline Outcome: The inverse pipeline automatically generates realistic, diverse CLI tasks from oracle GitHub repositories, enabling scalable training of agentic coding systems.The generated collection is produced without requiring task-by-task manual construction.

4. Experiments

Experiments train models on CLI-Gym trajectories and evaluate them with OpenHands on Terminal-Bench 1.0 and 2.0. LiberCoder achieves strong gains from curated environment-repair trajectories, while ablations show benefits from specialized data, diversity, and quality filtering.

  • Evaluation setup: OpenHands evaluations use Terminal-Bench 1.0 and 2.0, comprising 80 and 89 Linux command-line environment-manipulation tasks, respectively.Success requires passing verification scripts; pass@k measures tasks solved by at least one successful run among k attempts.
  • Main results: +28.6% and +21.1% are LiberCoder-32B and LiberCoder-235B-A22B improvements over their respective base models on Terminal-Bench 1.0.On Terminal-Bench 2.0, the corresponding improvements are +13.8% and +12.9%, using OpenHands and 291 successful environment-repair trajectories.
  • Main results: 46.1 is LiberCoder-235B-A22B’s Terminal-Bench 1.0 score, while 31.0 is its Terminal-Bench 2.0 score.LiberCoder-32B scores 38.9 on Terminal-Bench 1.0 and outperforms substantially larger open-weight models.
  • Ablations: Combining open-source SWE trajectories with CLI-Gym data consistently achieves the best performance across model scales.CLI-Gym-only training yields larger gains than SWE-style pretraining alone, indicating complementary generic software-engineering and terminal-interaction skills.
  • Ablations: Performance improves monotonically as more repositories are included while total data volume remains constant, supporting environment diversity as a key factor.The results suggest diversity matters more than trajectory count alone for robust agentic coding behaviors.
  • Error analysis: CLI-Gym training improves environment-intensive categories by more than 20 points, but gaming and scientific computing remain challenging.Training also reduces editing and localization errors, while increasing exploration-related context-length failures beyond the 128k inference limit.

5. Conclusion … A.2. Environment Inversion

CLI-Gym scales CLI agentic coding environments by representing configurations with Dockerfiles and using agents to simulate environment histories. Its pipeline constructs repository-based gold instances, generates and packages tasks, executes them to invert environments, and curates 1,655 task instances with 291 successful trajectories.

  • 5. Conclusion: CLI-Gym is presented as the first publicly available approach for scaling training environments for CLI agentic coding tasks.It represents each environment with a Dockerfile for precise configuration and version control, while agents simulate environment histories.
  • 5. Conclusion: 1,655 task instances and 291 successful trajectories are curated with the CLI-Gym toolkit.The paper reports that fine-tuning on this data substantially enhances environment-centric agentic coding, though the supplied passage is truncated before further results.
  • A. Detailed CLI-Gym Pipeline: Gold instances are constructed from 29 open-source GitHub repositories, each containing an environment, codebase, and unit tests.Repository information, unit-test counts, and final task-instance totals are summarized in Table 9.
  • A.1. Query Generation: The pipeline samples 1–3 intervention directions and 200 unit tests, then prompts an LLM with these inputs and previous task titles to generate an initial task prompt.The specific prompt is shown in Figure 10.
  • A.1. Query Generation: A second-stage prompt refinement aligns generated tasks more closely with selected unit tests and improves yield and diversity.The refinement is applied randomly after generating a task prompt in a predefined format.
  • A.1. Query Generation: Each generated task is packaged with Docker Compose, a base-image Dockerfile, a test-execution script, and a task specification containing the concrete prompt.Docker Compose maps trajectories, logs, and other artifacts between the executing container and local path.
  • A.2. Environment Inversion: Tasks run through a modified Terminal-Bench harness for approximately 15 minutes, with the agent prompted to deliberately induce failures and complete the task.The harness mounts trajectories and logs, invokes the Dockerfile, launches the agent, and supplies the task prompt.
  • A.2. Environment Inversion: After execution, unit-test outcomes determine fail-to-pass and pass-to-pass records, while all-passing tasks are discarded and degraded environments are summarized as reproducible Dockerfiles.If the test command cannot execute, all selected tests are recorded as fail-to-pass tests.

A.3. Task Generation

CLI-Gym converts completed inverse tasks into repair tasks by generating fault-targeted problem statements from the original prompt and failed unit tests. It packages each statement with extracted tests and a Dockerfile, while optionally producing versions with or without hints.

  • Problem Statement Generation: The problem statement combines the original task prompt with failed unit-test information and is generated using one of three prompts with varying guidance.The prompts range from explicit, strongly guided issue descriptions to weaker, less directive formulations.
  • Hint Variants: Every generated statement includes an LLM-produced hint, which a rule-based filter can optionally remove to create paired repair tasks.The two variants differ only in whether the hint is retained.
  • Task Packaging: The statement is inserted into a task template and combined with extracted unit tests and a Dockerfile to form a complete repair task.This packaging step turns the generated issue description and environment artifacts into the final task format.

B. Detailed Experiments

This section presents the paper’s full experimental configurations and execution details.

  • The section presents the full experimental configurations.
  • The section describes execution details for the experiments.
  • Together, the configurations and execution details constitute the section’s experimental description.

B.1. Training Set Construction

CLI-Gym expands 29 gold instances into 1,655 problem instances and retains 291 filtered successful trajectories for training after removing short or cheating trajectories.

  • Dataset generation: 1,655 problem instances were generated from 29 gold instances through 4,066 task prompts, covering faulty images, failed unit tests, and failure-inducing Dockerfiles.The process also collected 417 successful trajectories using strong language models with OpenHands.
  • Trajectory curation: 291 Filtered-Success trajectories were retained for training after filtering 126 from the 417 successful trajectories.The trajectories were collected using strong language models with OpenHands.
  • Filtering criteria: Trajectories with fewer than 20 interaction steps were removed because they were treated as trivial or low-difficulty solutions.The method uses interaction-step count as a proxy for task difficulty.
  • Filtering criteria: Cheating trajectories were discarded when they exploited historical artifacts such as cached Git information or Conda logs to bypass intended task solving.The passage identifies these artifacts as unintended shortcuts.

B.2. Training Details · B.3. Agent and Inference Details

Training uses cosine-decayed learning rates with 5% linear warmup and selects the best checkpoint across 10, 15, and 20 epochs for both Qwen3 models. Inference uses OpenHands in isolated Docker containers, with iterative tool-based interaction, fixed termination rules, and shared greedy decoding settings.

  • B.2. Training Details: Qwen3-32B uses an initial learning rate of 2 × 10−5 with cosine decay and 5% linear warmup from 1 × 10−6.Training evaluates 10, 15, and 20 epochs, uses batch size 16, and reports the best validation checkpoint.
  • B.2. Training Details: Both training setups use qwen3-coder as the agent template.The supplied training passages also specify model-specific learning-rate schedules and checkpoint selection procedures.
  • B.2. Training Details: Qwen3-235B-A22B-Instruct uses an initial learning rate of 1 × 10−5 with cosine decay and 5% linear warmup from 1 × 10−6.The model is trained for 10, 15, and 20 epochs, with the best-performing checkpoint selected.
  • B.3. Agent and Inference Details: OpenHands provides the execution interface, exposing a unified Agent–Computer Interface for structured interaction with containerized CLI environments.Supported interactions include shell command execution, file editing, and environment inspection; each model operates as a single autonomous agent that iteratively observes feedback and acts.
  • B.3. Agent and Inference Details: The agent invokes the tools shown in Figure 19, with every action executed inside an isolated Docker container constructed for the task.The action space is therefore constrained to the available environment-interaction tools.
  • B.3. Agent and Inference Details: Episodes terminate when the agent calls the finish tool or when Terminal-Bench’s global time limit is exceeded.These are the two termination conditions specified for inference episodes.
  • B.3. Agent and Inference Details: Inference uses greedy decoding with temperature = 0 and a maximum context length of 128k tokens for all models.Applying the same decoding configuration supports fair comparison across models.

B.4. Ablation Studies and Case Visualization

The ablation studies isolate the effects of agentic coding pretraining, repository diversity, and trajectory quality, while additional analyses compare category-level gains and failure distributions on Terminal-Bench@1.0.

  • Agentic Coding Pretraining with SWE Tasks: SWE-style trajectories exclude all Terminal-Bench tasks, preventing benchmark contamination and evaluation leakage during agentic coding pretraining.They initialize general abilities including repository navigation, tool invocation, and multi-step program repair rather than environment-specific knowledge.
  • Environment Diversity via Scaling Repositories: Uniform trajectory sampling across repositories balances environment coverage and prevents any single codebase or configuration from dominating training.Each dataset uses its respective best-performing epochs while keeping other settings unchanged.
  • Filtering low-quality Trajectories: Best-performing configurations, including 15 epochs for Filtered-Success and 10 epochs for Raw-Success trajectories, control for underfitting, overfitting, and training instability.All other training factors remain the same, attributing observed differences to data quality.
  • Category-wise Improvement: Category-wise comparisons evaluate baseline Qwen3-32B and LiberCoder-32B on Terminal-Bench@1.0 using pass@3 scores.The study uses official Terminal-Bench registry categories to measure capability improvements induced by training.
  • Failure Type Distribution: Failure analysis compares unresolved-task failure modes between Qwen3-235B-A22B-Instruct and LiberCoder-235B-A22B on Terminal-Bench@1.0.Failure cases are collected from a full evaluation run and categorized according to their observed modes.

B.5. More Experiments Impact of Different Agents

CLI-Gym experiments show that agent choice affects Terminal-Bench performance, while hint-augmented generation increases usable trajectories and downstream performance. More filtered-success trajectories are associated with higher pass@1 and sharply fewer repetitive action loops.

  • Impact of Different Agents: OpenHands did not perform very well on officially validated Terminal-Bench scoring examples, demonstrating the impact of agent choice.The comparison is reported in Table 6.
  • Comparison with Other Datasets: CLI-Gym generates many CLI-centric problem instances from a small set of base images, substantially reducing the required storage footprint.Table 7 compares task instances, base images/environments, and storage size across representative datasets.
  • Gain with Hint: 291 trajectories were collected with hints, compared with 104 without them, producing substantial performance improvement.When both variants were subsampled to 104 trajectories, performance remained comparable, indicating that hints primarily increase data yield.
  • Performance Benefits Model Behavior: 42.7% to 3.0%: increasing filtered-success trajectories steadily improves pass@1 while sharply reducing repetitive action loops.Figure 9 evaluates Qwen3-32B models fine-tuned with increasing numbers of filtered-success trajectories on Terminal-Bench@1.0.
Loading 2602.10999v1…