PaperlayerView exact revision on arXiv

Source-linked AI summary

T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks

Junyao Yang, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Zongxia Li, Haitao Mi, Leowei Liang

arXiv:2609.11042v1cs.LGcs.AI

TL;DR

Long-horizon terminal agents must execute persistent actions and withstand verification in stateful environments. T1 applies reinforcement learning to real terminal tasks with executed verifier rewards and a stabilization stack, reaching 64.0% resolved on Terminal-Bench 2.1 from a 49.4% supervised checkpoint.

  • Problem

    Long-horizon terminal tasks require environment comprehension, decomposition, recovery from partial failure, and execution-verified outcomes beyond ordinary reasoning benchmarks.

  • Method

    T1 is a 122B mixture-of-experts terminal agent trained by reinforcement learning on executed outcomes, using self-contained verifier-based tasks and a stabilization stack for large-scale sparse agentic RL.

  • Results

    64.0% resolved on Terminal-Bench 2.1, up from 49.4% for the supervised checkpoint and above GPT-5.4 at 54.8% and DeepSeek-V4-Flash at 56.9%.

  • Takeaways & Limitations

    The results support reinforcement learning on executed outcomes as a route toward terminal agents capable of handling demanding, verifier-checked software tasks.

  • Takeaways & Limitations

    TITO is exact only on the loss region, while re-tokenized history affects 2.6% of tokens in the audited run; isolating dense-reward effects also remains unresolved.

Abstract

from arXiv · show

Agent usage is shifting toward long-horizon tasks such as coding and scientific discovery, among which terminal tasks are especially important. We introduce T1, a Mixture-of-Experts model of 122B total trained with reinforcement learning, operating a real shell in a cloud sandbox for up to 300+ tool-call turns per task, rewarded by executing each task's own verifier. We provide a comprehensive recipe: First, an aggressively warm-started to stabilize actor-critic training, with a dense process reward scoring trajectories by the absolute number of passing verifiers. Second, stable optimization through TITO construction, training on the exact sampled token identifiers with drift repair at turn boundaries, and rollout routing replay, recording the sampler's per-token expert choices at every MoE layer and replaying them during training. Third, fully out-of-distribution training corpus: isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1 ensures gains reflect genuine capability transfer over benchmark overfitting. Together, TITO and R3 cut the training-to-inference log-probability difference from 0.021 to 0.013, with exactly aligned zero token drift in the loss region. On Terminal-Bench 2.1, our post-train pipeline raises initial base model from 43.8% to T1 with 64.0% resolved. On Long-Horizon Terminal Bench, T1 reaches 27.9% and surpasses GPT-5.4 and GLM-5.1.

1 Introduction

T1 targets autonomous long-horizon terminal execution with reinforcement learning grounded in executed verification. Its stabilization methods and out-of-distribution training yield substantial gains on held-out terminal benchmarks.

  • Stabilization: TITO and R3 address token and expert-routing inconsistency, reducing the measured training-inference log-probability gap from 0.021 to 0.013 with zero token drift in training.The methods target exact sampled-token conditioning and replay of sparse MoE routing decisions.
  • Reward design: Dense rewards count passing assertions on a fixed global scale, replacing binary trajectory outcomes that never exceeded the supervised baseline.A warm-started critic is trained at 30× the actor learning rate.
  • Main results: 64.0% resolved on Terminal-Bench 2.1 places T1 above GPT-5.4 and DeepSeek-V4-Flash, approaching Claude Opus 4.7 at 66.1%.The result comes from three PPO epochs starting from a 49.4% supervised checkpoint and represents a 28.5% relative gain from RL alone.
  • Evaluation setting: The training corpus uses isolated seeds and synthesized tasks disjoint from Terminal-Bench 2.1, supporting capability transfer rather than benchmark fitting.The evaluation is therefore held out from the training corpus by construction.
  • Approach: T1 is a 122B mixture-of-experts terminal agent trained by reinforcement learning on executed outcomes for tasks requiring up to 300+ tool-call turns.The system operates in real terminal environments and uses task-specific verification rather than modeled preference rewards.

2 Training framework

The training framework couples executable task verification, asynchronous sandbox rollouts, and stable MoE reinforcement-learning updates. Across held-out terminal benchmarks, the resulting pipeline reaches 64.0% on Terminal-Bench 2.1, 27.9% on Long-Horizon Terminal Bench, and 38.0% on Terminal-Bench Hard.

  • Pipeline: Inference replicas generate oversampled trajectories while an assembler converts interaction logs into TITO-stitched samples carrying routing records and dense rewards.The trainer updates the critic and actor before synchronizing weights back, while sandboxes are verified and reclaimed after execution.
  • Tasks: Each task supplies a long-horizon instruction, environment specification, held-out verifier, and reference solution, while per-assertion verification produces executable reward.The reference solution is hidden from the agent, and per-epoch task shuffling prevents reward distributions from drifting with task quality.
  • MoE consistency: The pipeline exchanges token identifiers and per-layer expert-routing choices, preserving alignment from sampled behavior through training gradients across multi-turn interactions.Tool outputs are masked context, sampled tokens receive loss, and the same offsets transform the per-token streams.
  • Reward: Absolute assertion counts provide dense process reward so harder tasks can contribute proportionally more verified progress than a capped pass ratio.The reward is fixed on a global scale and uses the verifier’s per-assertion outcomes.
  • Evaluation: 64.0% resolved on Terminal-Bench 2.1, 27.9% on Long-Horizon Terminal Bench, and 38.0% on Terminal-Bench Hard are reported for T1 across the main evaluations.The results place T1 above GPT-5.4 and DeepSeek-V4-Flash on Terminal-Bench 2.1, match Gemini-3.1-Pro on Long-Horizon Terminal Bench, and exceed DeepSeek-V4-Pro on Terminal-Bench Hard.

3 Terminal Dataset

T1’s terminal dataset combines recursively synthesized, sandbox-validated tasks with a quality audit and category-balanced reporting. The selected pool emphasizes command-line engineering while screening for verifier, instruction, solution, and task-value quality.

  • 3.1 Overview: The dataset pools include TMax-15k with binary verifiers, RST-38k with 37,484 synthesized tasks, and T1-15k with 15,000 audited tasks.TMax-15k contains 14,601 converted tasks, while RST recursively extends seed solutions and realigns environments, verifiers, and instructions.
  • 3.1 Overview: 15,000 T1-15k tasks are distributed across 17 merged categories, with the top five categories comprising 67.9% of the pool.The category visualization encodes share by angle, while radius uses a rank-based power scale and is not proportional to share.
  • 3.2 Dataset Construction: The audit weights verifier quality at 45%, solution quality at 25%, instruction quality at 20%, and task value at 10%.Instruction-verifier alignment carries the largest individual weight at 20%, while verifier dimensions receive 15% for fairness and 10% for coverage.
  • 3.2 Dataset Construction: T1-15k is selected from recursively synthesized tasks after executable validation and an LLM audit that rejects hidden requirements, test leakage, shortcuts, and weak verifiers.The audit proceeds through aggregation, static and executable checks, semantic review, instruction-only repair, and re-audit.

4 Stabilizing MoE RL training

T1 stabilizes long-horizon MoE reinforcement learning by aligning the sampled token stream and expert routing between inference and training, while warm-starting the critic. These mechanisms reduce train–inference mismatch and maintain stable critic estimates.

  • 4.2 TITO: Token-In-Token-Out: TITO enforces token fidelity by preserving sampled prefixes, stitching turns under loss masking, and repairing boundaries through progressively weaker prefix relations.The trainer consumes the identifiers emitted by the sampler rather than relying on re-rendered histories that may be retokenized.
  • 4.3 R3: Rollout Routing Replay: R3 enforces routing fidelity by recording the sampler’s per-position MoE expert choices and replaying those masks in the training forward pass.This avoids selecting potentially different experts when training and inference stacks use different kernels and reduction orders.
  • 4.1 Training-Inference Mismatch: The two fidelity conditions are independent: enforcing only token or routing fidelity leaves the other mismatch mode intact across long tool-calling trajectories.The problem is amplified by roughly 10^4 loss-bearing positions, where discrepancies accumulate along the trajectory.
  • 4.4 Measuring Training Stability of T1: 0.021 to 0.013: TITO and R3 reduce the train–inference log-probability gap while achieving zero token drift in the loss region.TITO reuses sampled token identifiers with boundary repair, while R3 replays rollout-time expert selections during training.

5 Dense Verification Reward Design

T1 replaces sparse binary task rewards with verifier-based dense feedback that counts satisfied assertions, allowing learning from partial progress. The resulting reward rises early and then stabilizes without the observed runaway turn growth in the 122B run.

  • 5.1 Motivation and verifier signals: Counting passed assertions converts unsuccessful trajectories into graded supervision, exposing partial progress that binary task rewards treat as zero.The verifier emits structured per-assertion outcomes, and the reward stage aggregates the number of passing assertions.
  • 5.2 Per-assertion verification and the test-count reward: Absolute passing counts preserve cross-task reward differences better than pass ratios when tasks vary in difficulty and assertion count.A ratio would assign the same reward to 10/20 and 2/4, despite the former potentially requiring substantially more work.
  • 5.3 Credit assignment: The scalar reward is placed on the final response token, while GAE with γ = λ = 1 propagates credit backward across the interaction horizon.The reward remains unnormalized because one sample per task provides no group statistic for a relative baseline.
  • 5.4 Reward hacking and safeguards: The 122B run uses the plain test-count reward without the runaway turn growth observed in 27B experiments, while monitoring turn counts and sequence lengths.Task filtering rejects hidden requirements, test leakage, shortcuts, and weak verifiers before training.

6 Results

T1’s reinforcement-learning post-training improves terminal-agent performance across standard, longer-horizon, and harder evaluations, while increasing interaction length and retaining domain-specific weaknesses. The strongest gains come from RL beyond supervised initialization, but case studies show that longer searches can become inefficient.

  • 6.2 Main results: 64.0% resolved on Terminal-Bench 2.1 places T1 above GPT-5.4 and DeepSeek-V4-Flash, within two points of Claude Opus 4.6, under the same harness.T1 uses 10B active parameters; the comparison is subject to harness choices and training-harness asymmetry.
  • 6.2 Main results: 27.9 average reward on Long-Horizon Terminal Bench exceeds T1’s earlier checkpoints and shows improvement on longer-horizon tasks beyond Terminal-Bench 2.1.RL adds 4.3 average-reward points over SFT and 9.0 over the base model.
  • 6.2 Main results: 38.0% resolved on Terminal-Bench Hard exceeds the SFT checkpoint by 9.7 points, the base model by 18.0 points, and DeepSeek-V4-Pro by 2.0 points.The comparison evaluates the complete training recipe and does not isolate dense reward from data or stabilization mechanisms.
  • 6.2 Main results: RL adds 14.6 percentage points after SFT’s 5.6-point gain, accounting for 72.3% of the 20.2-point improvement from base to final on Terminal-Bench 2.1.The three stages are base 43.8%, SFT 49.4%, and T1 64.0%.
  • 6.4 Case study: detailed evaluation across benchmarks: T1’s gains are uneven: it reaches 100.0% on debugging and 88.9% on system administration, but GPT-5.6 Sol remains ahead on data science, scientific computing, and mathematics.File operations also remain difficult for T1.
  • 6.4 Case study: detailed evaluation across benchmarks: T1 uses 94.4 evaluation turns on average versus 31.5 for SFT and 41.1 for base, while additional turns do not establish productive progress on individual tasks.The large turn increase accompanies only a modest improvement on the Hard group.
  • 6.4 Case study: detailed evaluation across benchmarks: On six representative unsolved tasks, T1 times out after 164–473 turns versus 6–40 for GPT-5.6 Sol, while resource differences limit attribution to model capability alone.Three further tasks fail on sandbox errors, and the cases suggest that diagnosis, recovery, and stopping decisions matter beyond turn budget.

7 Infrastructure

The infrastructure co-locates large actor–critic models across asynchronous training and inference while analytically controlling memory, throughput, synchronization, and straggler costs. Sequence-sharded recurrent computation and admission-control mechanisms make long-context training feasible, but oversampling introduces selection bias against slow tasks.

  • 7 Infrastructure: Analytical memory planning is required because expert optimizer state is invariant to expert redistribution, so feasibility depends on device budget rather than tuning search.Reducing expert sharding increases resident weights and gradients without reducing optimizer state; the plan is admitted before initialization.
  • 7 Infrastructure: Native sequence sharding reduces recurrent activation peaks to roughly one half, one quarter, and one eighth at κ ∈{2, 4, 8}, while preserving exact forward computation.The sharded forward pass is bit-exact against a single-rank reference, with gradients agreeing to 1.2 × 10−7 at tested sequence lengths.
  • 7 Infrastructure: Oversampling bounds step time by accepting the first 512 of 560 trials, but cancelled tails concentrate the hardest task families and create selection bias.The bias arises because completion time depends on task family, difficulty, and agent behavior; the trade-off is quantified as an open coverage issue.
  • 7 Infrastructure: 2.6× faster steps resulted from rebalancing inference replicas from one to three, reducing generation from 58.9 to 19.6 minutes without additional hardware.The production plan operates near the balance point, hiding approximately 16 minutes of generation under approximately 41 minutes of training.
  • 7 Infrastructure: Asynchronous execution overlaps generation and training, with a publication barrier enforcing one-step policy staleness and coordinating replica updates.The outer iteration admits trials, updates the critic and actor, quiesces generation, publishes parameters, and reclaims dead replicas.

8 Related Work

Related work frames terminal-agent training around executable task environments and the mismatch between inference and training systems. Existing approaches motivate both large-scale task synthesis and explicit correction of token and routing inconsistencies in sparse models.

  • 8 Related Work: Terminal-task synthesis uses repository recovery, injected perturbations, and task-conditioned generation to scale executable environments beyond manual authoring.These routes pair tasks with workspaces and tests or construct environments conditioned on task requirements.
  • 8 Related Work: Inference–training mismatch can corrupt importance ratios even without policy staleness, and sparse routing amplifies the discrepancy by changing activated experts.The paper positions TITO and R3 as responses to differing kernels, numerics, parallelism, and discrete expert selection.

9 Lessons learned and what did not work

The paper’s lessons emphasize that long-horizon RL depends on reward observability, retained history, task selection, and estimator design—not only on PPO optimization. Experiments show why dense verification rewards and a learned critic were preferred over binary rewards and GRPO in this setting.

  • 9 Lessons learned and what did not work: Improved critic explained variance did not improve rollout reward, showing that value prediction cannot expose progress absent informative verifier feedback.The dense reward makes partial completion visible, while critic warm-starting improves value modeling of the resulting returns.
  • 9 Lessons learned and what did not work: Context-compaction failures increased average turns from 22 to 30 because agents repeated completed work after losing history, making turn growth an ambiguous diagnostic.A mismatch between context budget and requested summary length caused 98.3% of full-summary attempts to fail.
  • 9 Lessons learned and what did not work: Completion-time selection can favor faster tasks, so cancelled straggler trajectories may remove precisely the hard-task partial progress that dense rewards would otherwise supervise.The paper retains oversampling for wall-clock benefits but treats long-task coverage as an open trade-off.
  • 9 Lessons learned and what did not work: 51.7% held-out resolution at both GRPO checkpoints showed no improvement, while rollout reward fluctuated without a trend under the same harness and reward.The result corresponds to 46 of 89 tasks resolved at both step 10 and step 20.
  • 9 Lessons learned and what did not work: GRPO sacrifices task diversity and temporal credit assignment because each task requires repeated rollouts for a group-relative advantage, while slowest group members delay completion.PPO spends the same generation budget across distinct tasks because a learned value function supplies the baseline.

10 Limitations and future work

The paper identifies limitations in trajectory fidelity, verifier integrity, rollout selection, data coverage, asynchrony, reporting, and evaluation scope. Future work targets better retention and coverage, while the transferable claims are limited to mechanisms and failure modes rather than the reported benchmark numbers.

  • 10 Limitations and future work: 2.6% of audited tokens remain affected by re-tokenized history, because eliminating the residual would require carrying token identifiers as the history source of truth.TITO is exact on the loss region, but later turns can be trained on sampled identifiers while inference used re-tokenized history.
  • 10 Limitations and future work: Binary-versus-dense reward evidence is campaign-level rather than a single-axis ablation, because reward, pool, initialization, and routing replay changed together.A matched binary run would be needed to isolate the reward contribution.
  • 10 Limitations and future work: Verifier integrity is filtered rather than enforced because verifiers execute inside agent-controlled sandboxes without runtime tamper detection.Proposed safeguards include read-only test mounts and checksummed verifiers.
  • 10 Limitations and future work: Oversampling drops the hardest trials from training, while remaining evaluation failures also concentrate on long, timeout-prone tasks.The paper proposes partial-rollout continuation, per-task budget respect, larger evaluation budgets, and difficulty-aware scheduling.
  • 10 Limitations and future work: The evidence is limited to one model family, agent harness, and benchmark, so transferable claims concern mechanisms and failure modes rather than specific numbers.The stated scope boundary covers token-faithful construction, routing replay, critic scheduling, and operational guardrails.

A Notation

The notation fixes separate policy versions, execution engines, token-indexed sequences, and routing collections. It also formalizes one-step asynchronous training, where batch Bt is generated under πt−1 while πt performs the update.

  • A Notation: Policy versions use training-step indices: πt has weights θt, while Bt is generated under the immediately preceding policy πt−1.This distinction is essential in the asynchronous training loop.
  • A Notation: Rollout and training superscripts identify engines, so πr and πt can evaluate the same policy weights through different implementations.The implementation distinction motivates the discrepancy analyzed later.
  • A Notation: Bold lowercase denotes token-indexed sequences, while calligraphic uppercase denotes sets or index collections such as the expert indices Iℓj.The conventions prevent token positions and routing collections from being conflated.
  • A Notation: The notation distinguishes the PPO clip ε from the router bound ϵ and the discount γ from the top-k margin γℓj.These similarly shaped symbols have different roles and are explicitly separated in the paper’s tables.
  • A Notation: Training and rollout run on disjoint devices, with generation for step t+1 overlapping training at step t and every Bt sample drawn under exactly πt−1.The synchronized publication of θt makes staleness σ fixed rather than random in the reported 122B setup.

B.2 Three distinct gaps, one measured quantity

The measured log-probability discrepancy separates legitimate version skew from implementation error and routing effects. TITO and R3 target only the implementation and routing mismatch while preserving genuine policy movement.

  • B.2 Three distinct gaps, one measured quantity: The discrepancy Δj decomposes into version skew and implementation gap, with TITO and R3 assigned to eliminate the latter under fixed routing.The decomposition distinguishes causes that require different remedies.
  • B.2 Three distinct gaps, one measured quantity: Only implementation mismatch is a defect: version skew reflects legitimate learning and is what PPO’s clip is designed to bound.Therefore, increasing discrepancy from policy evolution is not automatically a regression.
  • B.2 Three distinct gaps, one measured quantity: R3 cancels the implementation term by holding routing Ij fixed across numerator and denominator, while gradient-free and gradient-carrying replays supply the two evaluations.Sampler log-probabilities q are retained only for diagnostics.
  • B.2 Three distinct gaps, one measured quantity: Recorded routing R belongs to generating policy πr_t−1 and is replayed during gradient computation for θt, freezing expert selection while current gating remains trainable.This asymmetry preserves the sampled sub-network while allowing gradients through the current router.
  • B.2 Three distinct gaps, one measured quantity: At σ = 8, baseline combinations collapse, whereas combinations using R3 survive; at σ = 1, frozen routing is one update stale and matches PPO’s corrected staleness.The result identifies large asynchronous delay as a stress test for routing alignment.

B.3 Routing alignment across a version boundary

Routing alignment connects per-turn captures under the generating policy to stitched multi-turn training under the current policy. The procedure preserves token and routing indices while leaving kernel numerical differences unresolved.

  • B.3 Routing alignment across a version boundary: Algorithm 4 preserves routing indices from per-turn capture under πt−1 through stitched replay under θt, asserting each invariant rather than assuming alignment.A violation can silently train the wrong MoE sub-network.
  • B.3 Routing alignment across a version boundary: Per-turn routing records are concatenated with the same masking and indexing decisions that construct x, m, and q, with gap repair restricted to masked positions.The resulting R[j] remains aligned to the corresponding token position.
  • B.3 Routing alignment across a version boundary: The forward and backward passes use separate routing cursors because activation recomputation would otherwise consume one cursor twice per layer.Reference and critic passes select routing freely because they do not require behavioral fidelity to the rollout policy.
  • B.3 Routing alignment across a version boundary: Routing alignment does not make πt equal πt−1: it aligns which sub-network is differentiated, while kernel numerical error remains.The remaining term is especially relevant for 48-layer top-8-of-256 routing.
  • B.3 Routing alignment across a version boundary: The critic updates first and supplies pre-update values to the actor, anchoring GAE and the ±0.2 value clip to Vϕt−1 rather than the in-flight critic.The critic does not replay rollout routing because it performs supervised regression on returns.
  • B.3 Routing alignment across a version boundary: The frozen reference policy has no version index and uses its own routing, so R3 prevents claims that initial divergence must vanish.The policies differ in routing source even when their weights are unchanged.

C Case Study

Two matched software-engineering case studies show T1 succeeding where its RST-SFT initialization fails, despite both models reaching relevant intermediate conclusions. The differences concern abandoning exhausted strategies and satisfying exact final-state requirements.

  • C Case Study: T1 resolves both medium-difficulty tasks while the RST-SFT model does not, under the same task specifications and verifiers.The comparison attributes reward differences to agent behavior, while the harness limits differ and are explicitly noted as a caveat.
  • C Case Study: In build-pov-ray, T1 abandons the dead download host for a live FTP mirror, whereas RST-SFT repeats retries, causing 51 context overruns and 50 placeholder turns.T1 passes all three assertions after switching strategies.
  • C Case Study: The cases illustrate RL changing behavior after a conclusion is available: discarding an exhausted hypothesis and treating the final machine state as part of the deliverable.Neither recorded failure cause would have been repaired by additional context, according to the authors.
  • C Case Study: In the polluted-workspace task, both models produce a working Python/C polyglot and matching Fibonacci values, but only T1 removes the extra cmain binary and scores 1.0.RST-SFT leaves the binary in place and scores 0.0 because the verifier requires exactly main.py.c.
Loading 2609.11042v1…