PaperlayerView exact revision on arXiv

Source-linked AI summary

ECHO: Terminal Agents Learn World Models for Free

Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, Dimitris Papailiopoulos

arXiv:2605.24517v1cs.LGcs.CL

TL;DR

Terminal agents receive rich command consequences, but standard RL largely uses only sparse outcome rewards. ECHO trains on environment observations alongside action policy gradients, nearly doubling TerminalBench-2.0 pass@1 for tested Qwen3 models.

  • Problem

    Terminal agents receive dense environment feedback, but standard RL largely ignores these observations while optimizing sparse, delayed, binary rewards.

  • Method

    ECHO adds environment-observation cross-entropy to GRPO’s action-token policy-gradient loss using the same forward pass and rollout.

  • Results

    TerminalBench-2.0 pass@1 nearly doubles for Qwen3-8B, from 2.70% to 5.17%, and Qwen3-14B, from 5.17% to 10.79%.

  • Takeaways & Limitations

    Environment observations can provide dense on-policy supervision that improves terminal-dynamics prediction and learning from failed interactions.

  • Takeaways & Limitations

    Verifier-free environment-only adaptation works best when exploration produces clean, predictive feedback directly linked to actions.

Abstract

from arXiv · show

CLI agents are the closest thing language models have to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it: GRPO-style training updates action tokens with sparse outcome-level rewards while ignoring environment responses already in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduce ECHO (Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standard policy-gradient loss on action tokens with an auxiliary loss that trains the policy to predict environment observation tokens resulting from its own actions. ECHO reuses the same forward pass as GRPO, requires no additional rollouts, and turns terminal feedback into dense supervision for all rollouts. ECHO doubles GRPO pass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. ECHO also produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy while GRPO alone barely changes it. From base Qwen3-8B, ECHO matches expert-SFT-then-GRPO performance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, the environment prediction loss alone enables verifier-free self-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.

1 Introduction

Terminal agents receive rich outputs from every action, but standard GRPO largely ignores them because supervision is sparse, delayed, and binary. ECHO adds environment-token prediction to GRPO, turning terminal feedback into dense on-policy supervision without extra rollouts or forward passes and substantially improving benchmark performance.

  • Motivation: Terminal interactions produce detailed feedback, but sparse binary rewards leave most failed on-policy rollouts with little policy-gradient signal.In the Qwen3-8B setting, often fewer than 15% of on-policy rollouts solve the task.
  • Motivation: Failed trajectories contain outputs such as errors, logs, file contents, traces, and command results, yet these tokens enter computation only as context, not the loss.The paper identifies this unused terminal feedback as a dense supervision source already present in rollouts.
  • Method: ECHO adds auxiliary cross-entropy on environment-observation tokens to GRPO’s action-token loss, using the policy’s own rollouts as on-policy training targets.The objective requires no teacher model, extra rollouts, or additional forward pass, reusing logits from the policy update.
  • Results: On TerminalBench-2.0, ECHO nearly doubles GRPO’s pass@1 rate from 2.70% to 5.17% for Qwen3-8B and from 5.17% to 10.79% for Qwen3-14B.The improvement is reported across the tested Qwen3-8B and Qwen3-14B settings.
  • Results: The resulting policies become substantially better predictors of terminal behavior on held-out off-policy trajectories, while ECHO converts even failed interactions into learning signal.The introduction reports improvements on internal held-out evaluations and the public TerminalBench-2.0 benchmark.

2 Preliminaries

Terminal-agent training interleaves assistant actions with environment observations, with each action conditioned on the prior transcript. GRPO optimizes only action tokens using sparse rollout-level rewards, leaving observation feedback untargeted and unsuccessful trajectories weakly supervised.

  • Multi-Turn Rollout Structure: Training sequences interleave a system prompt, user task, and repeated assistant-action/environment-observation pairs.The transcript follows the form [action1] [obs1] [action2] [obs2] ··· [actionK] [obsK].
  • Multi-Turn Rollout Structure: At each turn, the policy conditions on the full prior transcript, while the harness executes parsed bash commands and appends terminal output as observations.Trainers compute log-probabilities over the full sequence, although policy-gradient loss is restricted to assistant-action positions.
  • Group-Relative Policy Optimization: GRPO applies group-normalized, clipped policy-gradient advantages uniformly to assistant action-token positions, without a learned value function.For each prompt, sampled rollouts receive scalar group-normalized advantages from binary rewards.
  • Group-Relative Policy Optimization: Observation tokens remain contextual but are not policy-gradient targets, so sparse binary rewards leave all-zero groups without contrast and unsuccessful mixed-group trajectories with only weak trajectory-level negative signals.Learning consequently concentrates on rare successful rollouts.

3 Method

ECHO augments GRPO with a length-normalized cross-entropy loss on environment observation tokens from the policy’s own rollouts. The losses share one forward and backward pass, while training targets terminal-output tokens rather than predictable warning messages.

  • Objective: ECHO combines GRPO’s action-token policy-gradient loss with supervised next-token prediction on selected observation tokens.The total objective is L_total = L_GRPO + λL_Env.
  • Objective: Observation loss is normalized by total observation length |O|, keeping runs with different target subsets comparable per observation.The targets are generated by the current policy’s own rollouts, making ECHO on-policy as visited terminal states evolve.
  • Efficient implementation: Both losses use the same actor forward pass and already-computed logits, so ECHO needs no second rollout, teacher model, or second forward pass.The added computation is a masked log-probability sum over observation positions in the same backward pass.
  • Target tokens: ECHO trains on terminal-output environment tokens while excluding harness warning tokens because warnings are quickly memorized and provide little useful gradient.Environment outputs instead carry task-specific feedback such as file names, test failures, byte counts, and error formats.
  • Loss weighting: The productive λ range is 0.01–0.05; λ = 0.1 can plateau or degrade policy quality, while λ = 0.2 can cause degenerate rollouts.The sweep tested λ ∈{0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2}.

4 Experimental Setup

The experiments train terminal agents on a curated and newly generated task corpus using a Docker- and Harbor-based multi-turn harness. They compare three starting policies under a common GRPO recipe and evaluate across held-out and benchmark task suites.

  • Training Task Corpus: The training corpus contains 8,870 curated and newly generated terminal tasks spanning task specification, Dockerfile validation, and Harbor-format export.It combines 2,700 curated tasks with 6,170 additional tasks generated through a modified Endless Terminals pipeline.
  • Harness and Runtime Environment: At each turn, the policy emits reasoning and Qwen XML-format bash commands or a task-done signal, which the harness executes and returns as the next observation.Episodes run for up to 16 turns in Docker under Harbor orchestration, with optional format warnings, stdout/stderr, and exit codes returned.
  • Models: Training compares Qwen3-8B, OpenThinker-Agent-v1-SFT, and Qwen3-14B starting policies.OpenThinker-Agent-v1-SFT is a Qwen3-8B model trained on approximately 15,000 expert terminal-agent demonstrations from GLM-4.6.
  • Evaluation: Evaluation covers val100, internal-dev, OpenThoughts-TBLite, and TerminalBench-2.0 task suites.val100 contains 100 held-out training-corpus tasks, while internal-dev contains 71 tasks focused on data processing, systems operations, and development/tooling.

5 Results

ECHO improves task success, terminal-dynamics prediction, training efficiency, and inference-budget use across the evaluated settings. It also recovers much of expert-SFT’s internal benefit and can enable verifier-free improvement, though gains depend on informative interaction feedback.

  • Task success: 2.70%→5.17% and 5.17%→10.79%: ECHO nearly doubles TerminalBench-2.0 pass@1 for Qwen3-8B and Qwen3-14B, respectively.ECHO improves every internal evaluation and consistently boosts TerminalBench-2.0 performance across the tested starting policies.
  • Terminal-dynamics prediction: ECHO sharply lowers environment-token cross-entropy on held-out, off-policy Qwen3-32B trajectories, while GRPO alone barely changes it.For Qwen3-14B, cross-entropy falls from 0.24→0.07 on val100, 0.39→0.31 on ITD, and 0.30→0.23 on TBLite; Qwen3-8B shows corresponding drops from 0.29→0.07, 0.46→0.32, and 0.35→0.25.
  • Expert-SFT gap: 101.6%, 103.9%, and 88.9%: ECHO recovers these shares of the OT-SFT advantage on Qwen3-8B internal val100, ITD, and TBLite evaluations.On TerminalBench-2.0, it closes 50.0% of the pass@1 gap, 48.6% of the pass@3 gap, and 50.0% of the pass@5 gap without expert behavior cloning.
  • Training and inference efficiency: 1.5–2.3× fewer training steps: at 8B, ECHO reaches the GRPO-only peak faster; at 14B, it reaches a higher plateau at the same peak step.Aggregate 8B speedups are 1.54–1.92×, while Qwen3-8B TBLite reaches a 2.27× speedup.
  • Training and inference efficiency: 19.8%→9.0%: ECHO cuts Qwen3-8B TerminalBench-2.0 timeouts and reduces completion tokens by 30%.For OT-SFT, timeouts fall from 45.2% to 24.7% and turns by 43%; Qwen3-14B uses 13% fewer tokens despite being the timeout-and-turns exception.
  • Verifier-free self-improvement: +3.8pp, +10.0pp, and +5.2pp: verifier-free environment-only adaptation improves val100, PyTerm, and ITD, respectively, without reward signals.The gains require useful prediction targets: unfiltered adaptation can model failure modes on harder OOD tasks, and the same recipe decreases TBLite by −3.9pp.

6 Related Work

ECHO builds on prior work treating environment interaction as supervision for world modeling, planning, and control. For LM agents, it differs by injecting observation prediction directly into on-policy GRPO rather than relying on separate training stages or corpora.

  • World models and embodied agents: Classical world-model methods learn environment dynamics for planning, imagination, or search, while recent embodied-agent work uses world or action models for planning and control.The paper positions ECHO within research that extracts supervision from environment interaction beyond sparse rewards.
  • LM-agent comparisons: CWM trains a large model on observation–action trajectories from Python and Docker environments, alongside work using agent experience or rich textual feedback to densify learning.ECHO’s closest LM-agent comparison is CWM; related methods also seek learning signals beyond scalar rewards.
  • LM-agent comparisons: ECHO injects observation prediction directly into on-policy GRPO without a separate corpus, world-modeling stage, or feedback generator.This distinguishes ECHO’s training setup from the cited LM-agent approaches.

7 Conclusion · Appendix

ECHO turns terminal environment responses already present in agent rollouts into dense supervision by adding cross-entropy on terminal-output tokens to GRPO’s logits. Across policies and model sizes, it improves RL-only training, learns faster, captures much of expert-demonstration benefits, and roughly doubles TerminalBench-2.0 pass@1 without extra rollouts or architectural changes.

  • 7 Conclusion: ECHO adds cross-entropy on terminal-output tokens to the same logits used for GRPO.The method converts environment responses already present in agent rollouts into supervision.
  • 7 Conclusion: ECHO requires no extra rollouts, forward passes, data, or architectural changes.Its auxiliary objective reuses the existing GRPO computation and training setup.
  • 7 Conclusion: Across different starting policies and model sizes, ECHO improves over RL-only training.The conclusion reports this improvement across both initialization conditions and model scales.
  • 7 Conclusion: ECHO learns faster than RL-only training across the reported settings.The conclusion identifies faster learning as a recurring benefit across starting policies and model sizes.
  • 7 Conclusion: ECHO matches much of the benefit provided by expert demonstrations.This result is reported alongside improvements over RL-only training across different starting policies and model sizes.
  • 7 Conclusion: ECHO roughly doubles TerminalBench-2.0 pass@1.The conclusion summarizes this benchmark improvement as a central empirical result.

A Environment-Token Cross-Entropy Trajectories

Environment-token cross-entropy separates into rapidly memorized warning structure and persistently informative terminal-output prediction. Warning CE reaches near-zero quickly, whereas env CE plateaus above zero and sustains gradients throughout training.

  • Warning versus terminal-output prediction: Warning CE drops from ∼5.6 nats to <0.05 nats by step 60, indicating rapid memorization of warning structure.Figure 6 reports the same near-zero convergence within ∼60 steps.
  • Warning versus terminal-output prediction: Env CE plateaus at 0.05–0.10 nats, reflecting irreducible entropy in variable filenames, byte counts, and error formats.The reported variability characterizes real terminal output.
  • Training implications: Env CE provides sustained gradient throughout training, unlike the warning gradient that auto-anneals under a constant λ schedule.This motivates retaining the environment-on-policy objective component.

B Hyperparameters and Reproducibility

The experiments use fixed AdamW optimization, constant learning rate, specified GRPO and environment-prediction settings, and standardized sampling and run budgets. Reproducibility protocols define evaluation configurations and uncertainty estimates for internal evaluation and TerminalBench-2.0.

  • Optimization: AdamW uses β1 = 0.9, β2 = 0.95, and weight decay 0.01.
  • Optimization: The learning rate is 1 × 10−6 constant with no warmup or decay, and gradient clipping is 0.2.
  • GRPO and environment prediction: GRPO uses n = 16 rollouts per prompt, batch size 16, no KL penalty, prompt-level advantage normalization, and ϵlo = 0.2, ϵhi = 0.28.
  • GRPO and environment prediction: Environment prediction uses λ ∈{0.02, 0.05} for SFT versus base, terminal-output tokens, and per-sequence normalization by total observation length.
  • Reproducibility: Internal evaluation uses 8 rollouts per task at temperature 0.6 with a 16-turn budget; pass-rate variance is ±0.05 per task and approximately ±0.025 across 100 tasks.
  • Reproducibility: TerminalBench-2.0 uses temperature 0.6, nattempts = 5, 1200 s agent and verifier timeouts, seed 42, and approximately 1.5 pp standard error on pass@1.

C Expert-SFT Gap

Table 4 evaluates how much of the expert-SFT performance gap ECHO closes on Qwen3-8B base without expert demonstrations. It compares matched GRPO, SFT+GRPO, and ECHO configurations across absolute pass-rates, SFT gaps, ECHO lifts, and recovered-gap fractions.

  • C Expert-SFT Gap: The table reports the SFT gap as SFT+GRPO minus GRPO and ECHO’s lift as ECHO minus GRPO.These quantities are reported in pass-rate units (%).
  • C Expert-SFT Gap: ECHO’s recovered-gap fraction measures the portion of the SFT initialization gap closed without using expert demonstrations.Internal columns are mean pass-rate, while TerminalBench-2 columns are pass@k.
  • C Expert-SFT Gap: Table 4 compares absolute pass-rates for matched GRPO, SFT+GRPO, and ECHO configurations on Qwen3-8B base.The comparison covers three configurations and quantifies the expert-SFT gap closed by ECHO without expert demonstrations.

D OT-SFT Training Curves

On OpenThinker-Agent-v1-SFT Qwen3-8B, ECHO quickly surpasses GRPO and remains consistently ahead throughout training. Figure 7 reports these OT-SFT-init Qwen3-8B training curves.

  • D OT-SFT Training Curves: ECHO quickly surpasses GRPO and remains consistently ahead during training for OpenThinker-Agent-v1-SFT Qwen3-8B.The curves follow a similar trend as those for Qwen3-8B and Qwen3-14B.
  • D OT-SFT Training Curves: Figure 7 presents the OT-SFT-init Qwen3-8B training curves.
Loading 2605.24517v1…