Source-linked AI summary
Endless Terminals: Scaling RL Environments for Terminal Agents
Kanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos
TL;DR
Terminal-agent RL lacks scalable, diverse, automatically verifiable environments, limiting training beyond small curated benchmarks. Endless Terminals procedurally generates and validates terminal tasks, then trains agents with vanilla PPO and a minimal interaction loop. Training produces consistent gains that transfer to TerminalBench 2.0, while the authors note limits from task realism and frontier-model solvability filtering.
Problem
Terminal-agent RL lacks a scalable supply of diverse, automatically verifiable environments, while existing benchmarks contain at most hundreds of tasks and manual curation is expensive.
Method
Endless Terminals autonomously generates terminal tasks through task-description generation, container validation, completion-test generation, and solution-based filtering, then trains agents with PPO.
Results
Training on 3,255 generated tasks produces consistent improvements across model scales that transfer to the human-curated TerminalBench 2.0 benchmark.
Takeaways & Limitations
The results demonstrate that simple RL setups can succeed when terminal environments scale, including without complex agentic scaffolds.
Takeaways & Limitations
Generated tasks tend to resemble competitive-programming problems rather than ambiguous real-user terminal requests, and solvability filtering cannot generate tasks beyond the frontier validator’s capability.
Abstract
from arXiv · showhide
Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals, a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale.
1 Introduction
Endless Terminals addresses the lack of scalable, diverse, automatically verifiable terminal environments for reinforcement-learning training. Its autonomous pipeline generates tasks and enables simple RL training to improve performance on both held-out and human-curated benchmarks.
- Existing terminal benchmarks provide at most hundreds of tasks, while manual environment curation is expensive for robust RL training.
- Endless Terminals procedurally synthesizes terminal-use tasks without human annotation or distillation.
- The pipeline generates task descriptions, builds and validates environments, produces completion tests, and filters tasks for solvability.
- Llama-3.2-3B improves from 4.0% to 18.2%, Qwen-2.5-7B from 10.7% to 53.3%, and Qwen-3-8B-openthinker-sft from 42.6% to 59.0% on the held-out dev set.These gains use vanilla PPO with a minimal interaction loop and no retrieval, tool use, or multi-agent scaffolding.
- The improvements transfer to human-curated benchmarks, demonstrating that scalable environments can support effective simple RL training.
2 Related Work
Related work addresses terminal-agent capability through scaffolds, supervised distillation, curated benchmarks, and synthetic environments, but these approaches leave environment scalability unresolved. Endless Terminals combines procedural generation with vanilla PPO and reports transfer to held-out benchmarks.
- Agentic Scaffolds: Agentic scaffolds add specialized commands, tools, retrieval, context management, or multi-agent coordination, whereas Endless Terminals uses a simpler command-emitting scaffold.
- Supervised Finetuning and Distillation: Supervised finetuning and distillation can provide RL warm starts, but teacher-based approaches depend on strong-model traces and, in some cases, proprietary-model access.
- Benchmarks and Interactive Environments: Human-curated benchmarks support execution-based evaluation and multiturn interaction, but they were not designed to provide an unlimited training environment.
- Synthetic Environment Generation: SWEGym provides 2438 executable Python tasks but relies on existing GitHub issues, while prior terminal work includes human-generated queries and commands.
- Synthetic Environment Generation: Endless Terminals contributes procedurally generated environments at arbitrary scale, with vanilla PPO yielding improvements that transfer to held-out benchmarks.
3 Procedural Generation of Tasks
Endless Terminals constructs procedurally generated terminal tasks through four sequential, automatically verified stages. Generation varies task content and context, validates containers, tests completion states, and retains tasks with at least one successful solution.
- The pipeline generates task descriptions, validates environments, creates completion tests, and filters tasks using model-generated solutions.Automatic verification is applied throughout the stages.
- Task descriptions sample categories, complexity levels, and scenario contexts to produce diverse terminal-use instructions paired with privileged ground truth.Examples span file management, text processing, log analysis, git operations, databases, and security scanning.
- Container generation iteratively builds and tests Dockerfiles or Apptainer definitions for up to k = 3 rounds, discarding tasks whose prerequisites cannot be validated.
- Completion tests verify expected post-task states, including created files, modified configurations, and computed results derived from privileged ground truth.
- The pipeline samples n = 16 o3 solution attempts and retains tasks when at least one succeeds, using pass@16 > 0 as the solvability criterion.
- Failed tasks are discarded automatically while processing proceeds in parallel, producing a controlled and automatically verifiable generation process.
4 Interacting with the Terminal
The terminal agent uses a minimal multiturn loop: it reads accumulated reasoning and shell output, emits commands or completion, and observes the resulting state. Persistent containers preserve the environment across interactions, while tests determine episode success.
- At each turn, the model receives conversation history and shell outputs, then produces either a command or a completion signal.
- Previous reasoning remains visible in later turns, allowing the model to reference prior actions, correct mistakes, and build on partial progress.
- Apptainer episodes preserve filesystem state, environment variables, and running processes across commands through a persistent interactive shell session.
- The system prompt requires one command per turn, non-interactive flags, and verification before declaring completion, excluding tools such as vim and htop.
- Episodes terminate on completion, 16 training turns, or 16k training tokens, after which held-out final tests determine success.
- The resulting loop repeatedly cycles through reasoning, action, observation, and reasoning with the full interaction context.
5 Experiments
Endless Terminals provides a scalable source of procedurally generated, solvable terminal tasks for PPO training. Across models and evaluations, training produces consistent gains, transfers to human-curated benchmarks, and reveals failures concentrated in looping, turn exhaustion, and difficult domains.
- Task generation and dataset: 3255 tasks were generated, spanning file operations, log management, data processing, text processing, scripting, archiving, and database operations.Approximately half of generated candidates were discarded when all 16 o3 solution attempts failed.
- Training setup: PPO training used 16 rollouts per prompt, up to 16 turns, binary episode-level rewards, and no intermediate rewards or KL penalty.The agent generated up to 2048 tokens per turn within a 16k-token conversation context.
- Development-set results: Llama-3.2-3B improved from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0% on the held-out development set.Reward increased throughout training for all three models.
- Transfer evaluation: TerminalBench 2.0 performance improved from 0.0% to 2.2% for Llama-3.2-3B, 2.2% to 3.4% for Qwen2.5-7B, and 1.1% to 6.7% for Qwen3-8B-openthinker-sft.The benchmark was not seen during training, and the RL-trained models outperformed alternative versions of the same base architectures.
- Failure analysis: Pass@5 declined with task difficulty, from 25% for easy tasks to 14.5% for medium tasks and 10% for hard tasks.These rates corresponded to 1/4 easy, 8/55 medium, and 3/30 hard tasks on TerminalBench 2.0.
- Failure analysis: Loop failures accounted for 39% of failures and turn exhaustion for 26%, while remaining failures often occurred in specialized domains.The two primary categories overlapped in 11 tasks; the model performed best on software-engineering tasks and achieved zero success on mathematics, machine-learning, and model-training tasks.
6 Discussion
Endless Terminals improves terminal-agent training through scalable procedural environments, while exposing limits in task realism, validator capability, and domain coverage. The authors identify richer validation, scaffolds, rewards, and world models as directions for extending the approach.
- Discussion: 3,255 valid tasks spanning file operations, log management, data processing, and text processing support PPO training without human annotation or distillation.The pipeline uses task generation, container setup and validation, completion-test generation, and solution-based filtering.
- Discussion: 39% of TerminalBench 2.0 failures are loop failures, while 26% result from turn exhaustion.Successful tasks show higher command diversity after the first error than looping failures: 0.49 versus 0.18.
- Discussion: 23% success on software engineering tasks contrasts with zero success in mathematics, machine learning, and model training.The authors suggest this gap may reflect insufficient coverage of those domains in the procedural generation pipeline.
- Limitations: Procedurally generated tasks tend to resemble competitive programming problems rather than messy, underspecified user requests.Automatically capturing ambiguous goals, implicit context, and clarifying questions while preserving verifiability remains an open challenge.
- Limitations: Solvability filtering with o3 retains tasks where at least one of 16 sampled solutions succeeds, discarding roughly half of generated candidates.This ensures retained tasks are solvable but prevents the pipeline from generating tasks beyond the frontier validator’s capabilities.
- Future Directions: Human validation, richer agentic scaffolds, denser rewards, and learned world models could improve task quality, diversity, or agent performance but introduce trade-offs.Human involvement increases generation cost, while the paper frames terminal-agent capability as a multifaceted challenge.