PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Generalizable Behaviors for Terminal Agents

Yihang Yao, Bo Pang, Xuan Phi Nguyen, Ding Zhao, Shafiq Joty, Semih Yavuz

arXiv:2608.22631v1cs.LG

TL;DR

Terminal-agent RL faces scarce real-user data, synthetic-environment gaps, and limited understanding of how generalization arises. The paper proposes Agentic Compositional Generalization and River, which improve reward quality by focusing RL on reusable behaviors and filtering environments. River achieves strong cross-benchmark performance and larger RL gains with fewer training environments, though its filtering can produce false positives and negatives.

  • Problem

    Terminal-agent RL lacks abundant real-user interaction data, while synthetic environments can have domain gaps and limited fidelity and existing work under-explores reward-signal quality and generalization mechanisms.

  • Method

    The paper proposes Agentic Compositional Generalization and River, which filters low-quality environments and improves verifier quality through behavior-level regularization.

  • Results

    River generalizes across model families, scales, agent harnesses, and RL objectives, achieving 106% and 30% larger RL gains on Terminal-Bench-Lite and Terminal-Bench-v2.1 using fewer than 30% of TMax environments.

  • Takeaways & Limitations

    Verifier quality is a key factor shaping the high-level behaviors learned by RL, beyond simply increasing environment quantity or diversity.

  • Takeaways & Limitations

    The environment-filtering process may produce both false positives and false negatives, and one environment may contain multiple issues.

Abstract

from arXiv · show

Terminal agents are a compelling application of large language models (LLMs), with the potential to integrate deeply into users' daily workflows. Reinforcement learning (RL) is a key technique for improving their capabilities, making scalable training environments a central challenge. Since public real-user interaction data are scarce, synthetic environments provide a practical alternative, but often suffer from domain gaps and limited fidelity, leading to poor generalization. Existing work mainly scales the quantity and diversity of synthetic environments, while reward-signal quality and the mechanisms governing generalization remain under-explored. We study how RL improves terminal agents and propose the Agentic Compositional Generalization hypothesis: rather than teaching new domain-specific skills from scratch, RL primarily shapes high-level decision-making behaviors that compose and route low-level skills acquired during pre-training and supervised fine-tuning (SFT). This account is consistent with our empirical results and suggests that verifier quality, which determines which behaviors are reinforced, is more important than simply increasing environment quantity or diversity. Motivated by this insight, we propose River, a simple training recipe that improves reward quality by filtering low-quality environments and augmenting outcome rewards with process-level behavior regularization. Using this recipe, our RL-trained agent achieves the best performance among evaluated open-source RL-trained 8B models across four terminal-agent benchmarks. River also generalizes across model families, scales, agent harnesses, and RL objectives. Using fewer than 30% of the TMax training environments, River improves RL gains by 106% and 30% on average for models ranging from 2B to 27B on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively.

1 INTRODUCTION

The paper argues that terminal-agent RL generalizes through high-level behaviors that compose pre-existing skills, making verifier quality more important than simply scaling synthetic-environment coverage.

  • Terminal-agent post-training generalization remains unclear beyond increasing task-category and skill coverage.
  • Agentic compositional generalization assigns low-level atomic skills to pre-training and SFT, while RL shapes behaviors that compose, route, and coordinate them across domains.
  • RIVER filters low-quality environments and improves verifier quality through behavior-level regularization.
  • 106% and 30% larger RL gains are achieved on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively, using fewer than 30% of TMax’s environments across 2B–27B models.
  • The paper provides extensive experiments testing its hypothesis and examining RL learning dynamics for terminal agents.

2 RELATED WORK

Related work develops terminal agents and synthetic interactive environments while investigating how supervised learning and RL support generalization beyond training distributions.

  • Terminal-agent research targets software-engineering and coding workflows evaluated on long-horizon benchmarks.
  • Synthetic-data research has progressed from static SFT demonstrations to interactive environments that provide executable feedback for agents.
  • Compositional-generalization work studies complementary roles for SFT and RL in explaining how models generalize beyond their training distributions.

3 MECHANISM INTERPRETATION AND METHOD

The paper explains terminal-agent generalization as RL shaping high-level behaviors that compose pretraining- and SFT-acquired skills, then proposes RIVER to improve the reward signals that shape those behaviors. RIVER filters low-quality environments and adds behavior-level verifier feedback to make RL training more reliable.

  • Agentic Compositional Generalization: Behavior-oriented trajectory features predict task success better than individual low-level skills, with AUC-ROC 0.74 versus about 0.55.The behavior-success link strengthens from SFT to RL, from 0.64 to 0.74.
  • Agentic Compositional Generalization: RL primarily shapes high-level behaviors that compose, route, and coordinate low-level skills acquired during pretraining and SFT.Behaviors span multiple turns, including exploration, verification, planning, and abandoning failing approaches.
  • Roles of SFT and RL: SFT equips weaker models with domain and interaction-harness familiarity, enabling trajectories correct enough to provide meaningful RL learning signals.The supplied passages describe complementary roles for SFT and RL rather than RL learning atomic skills efficiently.
  • RIVER: RIVER filters synthetic environments because weak or mismatched verifiers can provide noisy supervision that encourages shortcut solutions, reward hacking, or over-fitting.Rubric-based filtering identifies eight quality-issue categories without executing the underlying sandbox.
  • RIVER: More than 60% of TMax environments exhibit at least one quality issue, leaving fewer than 40% suitable for RL training.The filtering process can produce both false positives and false negatives, and one environment may contain multiple issues.
  • RIVER: RIVER optionally augments sparse trajectory-level rewards with turn-level behavior-verifier signals that reinforce desirable patterns and discourage undesirable ones.A rule-based verifier can penalize repetitive interaction loops with little information gain after reward normalization.

4 EXPERIMENT RESULTS

RIVER achieves strong performance across terminal-agent benchmarks and model scales by improving environment and verifier quality rather than relying only on larger environment collections.

  • RIVER-8B achieves the best overall performance among evaluated open-source 8B agents across four terminal benchmarks.
  • Qwen3-8B model family: Behavior verifiers reshape targeted behaviors by reducing repetitive interaction and increasing verification before task completion.
  • Qwen3.5 and Qwen3.6 families: 106% and 30% larger RL gains are achieved on Terminal-Bench-Lite and Terminal-Bench-v2.1, respectively, using approximately 30% of the TMax environments.
  • Qwen3.5 and Qwen3.6 families: RIVER outperforms the original TMax across model families, model sizes, agent harnesses, and RL algorithms.

5 ANALYSIS

The analyses support a division of labor in which SFT supplies broad low-level skills while RL reorganizes their use through high-level behaviors that generalize across domains.

  • Skill preserving and behavior shaping in RL: Skill exhibition ratios remain strongly correlated before and after RL, indicating that RL does not substantially expand the observed skill repertoire beyond the SFT checkpoint.
  • Skill preserving and behavior shaping in RL: ρ = 0.83 for skill-skill structure versus ρ = 0.27 for task-task structure, showing preserved skill co-occurrence but substantially reorganized task-specific skill combinations.
  • Skill preserving and behavior shaping in RL: Behavior-oriented trajectory features predict success more strongly than skill-presence features, with AUC = 0.64 to 0.74 versus 0.48 to 0.56.
  • SFT analysis: Without SFT, direct RL on the base model performs worst, while broader atomic-skill coverage creates a stronger warm start for RL generalizability.
  • SFT analysis: Diverse-SFT improves post-RL evaluation under both binary and partial rewards and can mitigate reward-hacking risks.
  • Behavioral generalization: RL-induced behavioral changes extend beyond training domains and contribute to performance improvements in untrained domains.

6 CONCLUSION

The paper concludes that terminal-agent RL primarily shapes reusable behaviors around skills supplied by pre-training and SFT, motivating RIVER’s focus on reward-signal integrity.

  • Agentic compositional generalization posits that RL shapes high-level decision-making behaviors that compose, route, and coordinate pre-existing atomic skills across domains.
  • RIVER filters low-quality environments and optionally adds behavior-level regularization to improve reward-signal integrity.
  • RIVER achieves the best performance among evaluated RL-trained 8B models across four terminal-agent benchmarks and generalizes across model families, scales, harnesses, and RL objectives.
  • The Qwen3-8B implementation uses full-parameter SFT followed by GRPO with sparse binary verifier rewards.

A.3 EVALUATION

Evaluation uses fixed serving and decoding settings across four terminal-agent benchmarks, with benchmark-specific continuous or binary scoring.

  • All checkpoints are evaluated with identical serving settings, including SGLang, tensor parallelism TP=8, and the model’s full 40,960-token context window.
  • Rollouts use temperature T=0.6, top-p=1.0, up to 2048 tokens per turn, and caps of 64 action turns or 600 seconds.
  • Four terminal-agent benchmarks are evaluated using their default scoring criteria: continuous scoring for OpenThoughts-TBLite and binary pass/fail scoring for the others.

B.1 SFT DATASET COVERAGE

The SFT datasets are compared by coverage of 60 command-level atomic skills, while trajectory conversion and filtering align source data with the training harness. Environment figures then characterize the TMax pool and rubric evaluations across environment sources.

  • Figure 9 measures the fraction of trajectories exhibiting each of 60 command-level skills across three SFT datasets.
  • Source SFT trajectories are deterministically translated from the Terminus-2 format into the EndlessAgent harness format.
  • Trajectories are dropped when required task markers or intermediate JSON parsing conditions fail, while a final non-JSON confirmation is converted to done.
  • Diverse-DS-SFT serves as a size control for Diverse-SFT because its atomic-skill coverage is indistinguishable across all 60 skills.
  • Figure 10 breaks down 14,399 TMax environments by nine task categories and eight skill groups.
  • Figure 11 compares GPT-5.4 rubric evaluations for environments from TMax, TerminalTraj, and TermiGen.

C ENVIRONMENT EVALUATION

The environment evaluation framework checks whether instructions, verifiers, and environments are mutually consistent and distinguishes clean tasks from several defect categories. Across pools, weak verification is the dominant problem, while judge agreement is stronger for defect labels than for clean labels.

  • TMax has 35.8% CLEAN and 40.4% VERIFIER-TOO-WEAK environments, while TerminalTraj-5k has 3.3% CLEAN and 67.2% VERIFIER-TOO-WEAK.
  • The evaluation agent checks mutual consistency among the instruction, reward verifier, and environment, assigning each task one of eight verdicts.
  • CLEAN requires clear instructions, solvability, and a verifier that genuinely validates correctness rather than merely checking output form.
  • VERIFIER-TOO-WEAK environments can pass with fabricated outputs because their verifiers check superficial properties or hardcodable cases.
  • The two models agree on only 22.5% of eight-way verdicts across 120 examples, partly because one environment can exhibit multiple failure modes.
  • 100/120 examples yield binary clean-versus-defective agreement, but the stratified sample does not estimate natural full-pool agreement.
  • Defect labels have 89.5% conditional agreement, whereas CLEAN labels have 40.0%, suggesting reported clean rates may be optimistic.
  • Examples show weak verifiers, instruction-verifier mismatches, missing runtime artifacts, answer leaks, and unspecified tie-breaking conventions.

TASK-TRIVIAL

The task is labeled TASK-TRIVIAL because an accessible oracle reveals the transformation that the agent was supposed to recover from audio. This permits hard-coding without using the audio input.

  • The accessible reference implementation reveals the projection matrix, making the audio-recovery component unnecessary.
  • The transformation can therefore be hard-coded without using the audio.

C.4 EXAMPLES OF VERIFIER FAILURES ON THE UNFILTERED ENVIRONMENT SET

The unfiltered environments contain verifier defects that reward shortcuts or penalize correct solutions, making reward an unreliable proxy for intended task success. These failures include leaked answers, broken or mismatched oracles, incorrect references, and brittle implementation checks.

  • False-positive rewards: Leaked reference outputs let agents replace failed task-specific reconstruction with exact answers and still receive full reward.This occurs for both ticker recovery and binary inspection tasks.
  • False-positive rewards: Degenerate oracles make trivial behavior optimal when the verifier rewards agreement with the oracle rather than the intended specification.A constant exit-0 replacement receives full reward for a broken semantic-version resolver.
  • False-negative rewards: Correct solutions receive zero reward when verifiers use erroneous hard-coded references or incorrect domain assumptions.Examples include a wrong posterior constant and an FFT verifier that expects a presumed fundamental-frequency index instead of the actual maximum.
  • False-negative rewards: Oracle equivalence can reject correct implementations when the reference violates algorithmic assumptions, as with binary search over unsorted PTS values.The agent returns the correct latest-frame result, but the verifier treats the buggy oracle output as ground truth.
  • False-negative rewards: Brittle source-pattern checks can reject functionally correct implementations solely because they use a different valid API.The Rust pipeline passes functional output tests but fails a literal substring check.

D BEHAVIOR ANALYSIS DETAILS

The behavior analysis extracts trajectory features deterministically and evaluates their predictive power for RL-checkpoint success. Verification before completion is the strongest positive signal, while command repetition is the strongest negative signal.

  • Feature extraction: Behavior features are extracted deterministically from trajectories using rule-based parsing without a model or LLM in the loop.Commands and resulting observations are parsed turn by turn, with inspections identified from a predefined read-only tool set.
  • Evaluation: AUC measures each feature’s cross-validated AUC-ROC for predicting per-task success, averaged over 15 seeds of 5-fold cross-validation.The table also reports correlation direction and sorts rows by AUC.
  • Key signals: Verifying before finishing is the strongest positive individual signal, whereas repeated similar commands are the strongest negative signal.Repetition rate serves as a broader proxy for harmful repetition.
Loading 2608.22631v1…