PaperlayerView exact revision on arXiv

Source-linked AI summary

Terminal-RSI: Recursive Self-Improving Terminal Agents via Recursive Self-Rewrite and Policy-Refine

Junyao Yang, Zongxia Li, Yucheng Shi, Zhongzhi Li, Ruhan Wang, Ruiwen Zhou, Haitao Mi, Shuicheng Yan, Leowei Liang

arXiv:2610.22192v1cs.AI

TL;DR

Terminal agents need to turn local progress into persistent task completion despite long interaction histories and shifting critic targets. Terminal-RSI alternates verified trajectory rewriting with critic-inheriting policy refinement, turn-aware credit, and evolving rewards. Terminal-RSI-27B reports 50.58% macro-average success across four terminal benchmark suites, alongside improved LHTB outcomes.

  • Problem

    Terminal agents must produce executable environment changes that pass protected verifiers, while long interaction histories and shifting policy and reward targets complicate learning from delayed outcomes.

    Source: §1 · p. 2

  • Method

    Terminal-RSI alternates verified actor-only Recursive Self-Rewrite with critic-inheriting Recursive Policy-Refine, using turn-aware credit assignment and evolving rewards.

    Source: §1 · p. 2

  • Results

    50.58% macro-average success; Terminal-RSI-27B also reports 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard.

    Source: §5 · p. 13

  • Takeaways & Limitations

    The reported results span terminal benchmarks and LHTB, while controlled studies are still needed to distinguish how the components interact.

    Source: §5 · p. 13

    Three evaluations do not establish statistical superiority or replace independent training seeds, and component effects remain unattributed without matched controls.

    Source: §5 · p. 13

Abstract

from arXiv · show

Converting verifier outcomes into useful policy updates and partial progress into complete task resolution is crucial for terminal agents. We present Terminal-RSI, which alternates Recursive Self-Rewrite (RSR) of verified trajectories with critic-inheriting Recursive Policy-Refine (RPR). We formulate Turn-Boundary Skip-Observation GAE (TB-GAE) to retain token-specific baselines while applying decay across interaction turns. Family Dense Reward (FDR) calibrates semantic progress across task families; Success-Prioritized Reward (SPR) separates strict success from bounded partial credit. A Warm-up Critic Model, recursive critic handoff, and stage-specific learning rates coordinate the two refinement phases. The five-stage sequence from Qwen3.8-27B base model through RSR-1, RPR-1, RSR-2, and RPR-2 yields Terminal-RSI-27B: 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard. On LHTB, process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46; SWR100 success doubles from 3.00% to 6.00%. By recursively coupling verified trajectory rewriting with critic-inheriting policy refinement, each retained stage consolidates discovered behaviors and progressively advances the agent from partial local repairs to strict terminal verification.

1 Introduction

Terminal agents must turn exploration into persistent changes that pass protected verifiers, despite long interaction histories and shifting value targets. Terminal-RSI alternates verified actor rewriting with critic-inheriting policy refinement, alongside turn-aware credit assignment and evolving rewards.

Motivation

  • Terminal agents must change executable environments that pass protected verifiers, since fluent plans and locally passed checks alone do not complete the task.Long-Horizon-Terminal-Bench emphasizes that substantial local progress may still fall short of a complete solution.Source: §1 · p. 2
  • Token-level GAE can attenuate distant credit according to textual length, so equivalent decisions and outcomes may receive different credit when logs differ in verbosity.Masking observation tokens from policy loss does not remove them from the recursion; summarization also changes the conditioning history.Source: §1 · p. 2
  • Recursive rewriting shifts actor histories, while reinforcement and reward recalibration further change the critic’s prediction targets, making transferred values potentially miscalibrated.Resetting discards learned outcome structure, whereas uncalibrated transfer can affect advantages and value bootstraps.Source: §1 · p. 2

Method

  • Terminal-RSI alternates actor-only RSR with critic-inheriting RPR, preserving critic weights across supervised transitions and adapting inherited values before joint optimization.The recursive loop returns the retained policy to the next cycle; RPR begins from the RSR-1 actor and a Warm-up Critic Model.Source: §1 · p. 2, Figure 2 · p. 3
  • TB-GAE keeps token-specific baselines while decaying across turns; FDR calibrates family progress, and SPR separates strict success from bounded partial credit.Independent rescoring reduces failed reward share from 50.09% to 28.72%.Source: §1 · p. 2

Main results

  • Four-suite mean success rises 29.90 and 16.33 percentage points over Base and RSR-1, respectively; LHTB process reward reaches 0.37 and strict completion reaches 5/46.SWR100 success doubles to 6.00%; task completion, shaped reward, and checker coverage are reported as distinct signals.Source: §1 · p. 2

2 Terminal-RSI Method

Terminal-RSI combines verified trajectory rewriting, critic handoff, reward evolution, and turn-aware credit assignment. Its refinement phases recalibrate inherited values before joint policy optimization.

2.1 Interaction, supervision, and parameter handoff

  • RSR retains and verifies actor demonstrations using public-task runbooks and fresh execution; critic weights stay fixed during rewriting and are recalibrated before RPR joint updates.Strict success requires passing the complete protected verifier, not an agent declaration or a subset of checks.Source: §2.1 · p. 3

2.2 Reward evolution: family calibration and success prioritization

  • FDR calibrates partial progress by task family, while SPR gives bounded partial credit subordinate to strict success; fixed-trajectory rescoring reduces failed reward share from 50.09% to 28.72%.The SPR bound does not imply lower gradient variance or convergence.Source: §2.2 · pp. 4–5

2.3 TB-GAE: turn-boundary credit assignment

  • TB-GAE skips observation nodes in its recursion and applies decay across turn boundaries, avoiding observation-length-driven attenuation while retaining token-specific values.Exact cross-partition continuation requires both the successor value and trace; summarization changes conditioning and does not establish fixed-input equivalence.Source: §2.3 · p. 5

2.4 RPR objectives and two-level critic warmup

  • RPR uses a clipped policy surrogate with detached advantages and value targets; the documented first phase instead uses standard GAE with γ = λ = 1.The paper states that neither TB-GAE deployment nor an isolated online gain is established by unit-clock equivalence.Source: §2.4 · p. 6

3 Implementation and Empirical Results

Terminal-RSI combines phase-adaptive refinement with verified rewriting, then evaluates retained checkpoints across terminal benchmarks and task-level execution contracts. The results show gains alongside benchmark-specific tradeoffs, reward-allocation differences, and gaps between local checks and complete verification.

3.1 Infrastructure and phase-adaptive training configuration

  • RPR-1 and RPR-2 adapt actor learning rates while retaining the critic learning scale, and the evaluation reports three-run Mean@3 for retained checkpoints.The five-stage sequence uses Qwen3.8-27B; the evaluation also reports LHTB and SWR100, which test long-horizon completion and scientific reconstruction, respectively.Source: §3.1 · pp. 6–7, Table 1 · p. 6

3.2 Retained stages and initialization order

  • 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard for the final RPR-2 checkpoint.The four-suite equal-weight success mean rises from 20.68% to 50.58%; the reference-model comparison is benchmark-specific, with different models scoring higher on individual suites.Source: Table 2 · p. 7, §3.2 · p. 8, Figure 5 · p. 8

3.3 Training dynamics and checkpoint comparison

  • Retained gains coexist with suite-specific regressions: the SPR Long-Reward RPR-2 checkpoint reaches 19.84% on Terminal-Bench 4.0 and 86.52% on Terminal-Bench 2.1, while later continuation loses ground on three suites.RPR-1 reward and turn means rise together, but differing clocks and task mixtures prevent attributing those trends to rollout-level behavior; training–inference mismatch remains nonzero.Source: §3.3 · pp. 8–9, Figure 6 · p. 9

3.4 Difficulty-stratified outcomes and reward sensitivity

  • Medium and Hard strata improve across stages, while Extreme falls and recovers; fixed-trajectory rescoring reduces failures’ reward-mass share from 50.09% to 28.72%.The stratified analysis fixes difficulty groups before observing outcomes, and the rescoring comparison holds outcome labels constant; family calibration also raises reward for selected long, low-return families.Source: §3.4 · p. 11

3.5 From local repairs to complete verification

  • Both final checkpoints reach 100.00% checker completion on the two tested task contracts, but equal scores can reflect different procedures and do not establish broader checker accuracy.For the Rust–C evaluator, the final model’s first trial delegates expression evaluation to Python, so fuzz-test success does not demonstrate a native Rust–C AST execution path.Source: §3.5 · pp. 12–13, Figure 12 · p. 12

4 Related Work

Terminal-RSI is situated among work on recursive supervision, terminal-agent optimization, asynchronous training, and task construction. Its distinction is coupling these components through critic-preserving transitions and reward calibration.

4 Related Work

  • Terminal-RSI couples recursive supervision components through critic-preserving transitions, unlike prior approaches that use generate–filter–train, verbal feedback, or terminal PPO with warm-started values.RSR rewrites cross-harness discoveries, and its runbook reviewer is not a learned value critic [23].Source: §4 · p. 13
  • Terminal-RSI uses fixed asymmetric clipping among policy-optimization approaches that also include critic-free advantage estimation and learned value baselines.Other variants use length-normalized sequence ratios, token masking, or mismatch-adaptive clipping [47] [53] [54].Source: §4 · p. 13
  • Terminal-RSI’s critic handoff and reward calibration complement curricula that recursively expand solutions, environments, verifiers, scenarios, or skills.SWR and LHTB distinguish progress resolution from feedback frequency [20] [21] [22].Source: §4 · p. 13

5 Limitations and Conclusion

Terminal-RSI coordinates verified rewriting, critic-inheriting refinement, turn-aware credit, and evolving rewards, with reported gains across terminal benchmarks and additional evaluations. The paper notes limits on causal attribution, evaluation breadth, verification coverage, overlap auditing, and generalization beyond one backbone.

5 Limitations and Conclusion

  • The study does not isolate component effects because stage-specific learning rates, rewards, and initialization change together, and compute-matched controls remain necessary.Offline rescoring and estimator identities do not isolate training effects.Source: §5 · p. 13
  • Three evaluations do not establish statistical superiority or variation across independent training seeds, and checkpoint selection requires protected validation.The paper also notes that verification certifies enforced requirements only, SWR100 is training-disjoint while broader overlap audits remain incomplete, and one 27B backbone establishes neither scaling laws nor universal monotonicity.Source: §5 · p. 13
  • Terminal-RSI-27B reaches 19.84% on Terminal-Bench 4.0, 86.52% on Terminal-Bench 2.1, 16.95% on Terminal-Bench 3.0, and 79.00% on Terminal-Bench Hard.LHTB process reward rises from 0.21 to 0.37 with 5/46 strict completions; SWR100 reaches 6.00%.Source: §5 · p. 13
  • Terminal-RSI combines verified rewriting, critic-inheriting refinement, turn-aware credit, and evolving rewards.Source: §5 · p. 13

A Training Configuration and Reproducibility

The appendix documents phase-specific optimization and sampling settings, separates training monitoring from final benchmark evaluation, and records initialization and reproducibility caveats.

A Training Configuration and Reproducibility

  • RPR-2 reduces the actor learning rate to 1 × 10−7 while retaining the critic rate, and both actor schedules are constant within phase.RPR-1 uses actor and critic rates of 5 × 10−7 and 1.5 × 10−5; each phase reserves four of 36 rollout steps for critic-only updates.Source: A Training Configuration and… · p. 18
  • Final benchmark evaluation uses three independent runs per stage checkpoint and suite, reporting their arithmetic mean as Mean@3.These repeats characterize inference variability rather than variation across independently trained models.Source: A Training Configuration and… · p. 18
  • The final benchmark protocol distinguishes context and interaction budgets from benchmark-specific termination and time limits.The table describes the final protocol, not training monitoring.Source: Table 3 · p. 18
  • RPR starts from the RSR-1 actor, freezes it for four initial critic-only updates, then begins joint training; the saved RPR-2 launch requests weights-only transfer.Available records do not resolve every actor–critic pairing or isolate warmup’s contribution to calibration and sample efficiency.Source: A Training Configuration and… · p. 18

B Reward Calibration and Offline Evidence

The appendix specifies reward-calibration assumptions and lookup-table construction, then reports offline rescoring and trajectory-distribution evidence with limits on interpretation.

B Reward Calibration and Offline Evidence

  • The recalibrated 46-family table contains 2,304 samples, while the historical report records 2,087 valid trials and its 45-family table accounts for 1,834.Unrepresented families use the global fallback; the structured records preserve old and new divisors and source provenance.Source: B Reward Calibration and Offline… · p. 19
  • Eight family divisors are doubled, nine quartered, one newly calibrated, and 28 unchanged; the frozen lookup table defines the offline comparison.The saved Long-Reward RPR-2 configuration selects SPR, while matched online comparisons are required to isolate family-calibration effects.Source: B Reward Calibration and Offline… · p. 19
  • Halving rather than quartering the selected long-family divisors produces a failed reward share of 26.97%, versus 28.72% for the chosen configuration, but provides less partial credit.Source: B Reward Calibration and Offline… · p. 19
  • Across 2,802 shared tasks, mean interaction length differs by −23.63 turns, with a task-bootstrap interval of [−25.09, −22.18].Task resampling does not control initialization, execution budgets, or temporal dependence, so this is a distributional difference rather than a treatment effect.Source: B Reward Calibration and Offline… · p. 19
  • Mean generated length decreases from 49,391 to 33,110 tokens, interaction length from 69.04 to 44.28 turns, and success rises from 21.35% to 28.68%.In the later population, 3,104 failed trajectories still receive 50.09% of total reward mass.Source: B Reward Calibration and Offline… · p. 19
  • The earlier per-attempt evaluation success is 82.02%, versus approximately 67.00–68.00% under earlier training conditions with different initialization.This is neither a causal estimate of SPR’s effect nor a pass@3 result; the later eight steps were already available when the reward was designed.Source: B Reward Calibration and Offline… · p. 19

C Additional TB-GAE Derivations and Checks

The appendix checks how TB-GAE handles skipped observations, reward timing, and value-baseline error. It states conditions under which the estimator recovers standard GAE or remains unchanged.

C Additional TB-GAE Derivations and Checks

  • TB-GAE’s correction separates discount changes from trace-decay changes without adding a reward bonus or subtracting independently computed advantages.Preserving the original token-time objective instead requires elapsed-position discounts and appropriately aggregated rewards.Source: C Additional TB-GAE Derivations and… · p. 19
  • Return, value, and return–value covariance variances jointly determine residual variance, so explained variance is not squared correlation.An action-independent baseline preserves the expected unclipped on-policy Monte Carlo policy gradient only under the stated sampling-policy and interchange conditions; this does not extend unchanged to clipped asynchronous PPO.Source: C Additional TB-GAE Derivations and… · p. 20
  • TB-GAE is unchanged by inserting zero-reward observation nodes when retained prefix inputs and value predictions are fixed.Computational partitioning also preserves the result when both successor value and successor trace are transported without changing retained inputs.Source: C Additional TB-GAE Derivations and… · p. 20
  • Moving turn rewards to termination can change returns, so computational boundaries must preserve episode identity, turn structure, and causal reward placement.For a complete terminal chain, the appendix derives the residual error difference; when all trace coefficients equal one, only the initial error term remains.Source: C Additional TB-GAE Derivations and… · p. 20

D Conditional KL Interpretation and Selection

This appendix gives a conditional KL interpretation for reward-tilted trajectory distributions and masked replay likelihood. It also states limits on what those identities and the reported checkpoint sequence establish.

D Conditional KL Interpretation and Selection

  • Under fixed-task and integrability assumptions, the reward-tilted trajectory law characterizes a KL-regularized objective, not the zero-KL clipped PPO objective used here.The tilted law may be unrealizable under a restricted policy family or fixed environment dynamics.Source: D Conditional KL Interpretation and… · p. 20
  • For fixed accepted native trajectories and parameter-independent environment factors, masked likelihood equals forward KL up to a parameter-independent constant.Length-normalized losses reweight the replay distribution; the represented-family bound does not guarantee discovery of absent families or restoration of entropy.Source: D Conditional KL Interpretation and… · p. 20
  • The reported five-stage sequence records retained checkpoints, not a prospective checkpoint-selection rule; repeated selection on final benchmark feedback would confound learning with test-set adaptation.The passage requires predefined validation objectives, patience, tolerances, evaluation frequency, and retention criteria for prospective selection.Source: D Conditional KL Interpretation and… · p. 20
  • Five stages of one 27B backbone establish neither a model-size scaling law nor an equal-compute benefit of recursion.The represented-family bound penalizes loss of represented families as policy mass approaches zero, but does not guarantee discovering absent families.Source: D Conditional KL Interpretation and… · p. 20

E Supplementary Evaluation and Provenance

The supplementary evaluation distinguishes strict completion and repeated-run metrics from shaped reward and historical aggregates. It also documents coverage differences and the cost mismatch in the separate base-initialized control.

E Supplementary Evaluation and Provenance

  • The final benchmark protocol averages three independent evaluation runs, while historical summaries have heterogeneous repeat counts that cannot be reconstructed from that protocol.Mean@R and pass@R are distinct metrics, and repeated evaluation does not replicate training.Source: E Supplementary Evaluation and… · p. 21
  • LHTB stage-ledger completion entries progress from 0/46 to 5/46, but the compact 5/46 entry does not identify per-run counts or imply that each run solved the same tasks.Strict completion is reported separately from mean process reward.Source: E Supplementary Evaluation and… · p. 21
  • Earlier records cover 74 Terminal-Bench 3.0 tasks and 66 Terminal-Bench 4.0 tasks rather than the nominal historical denominators, requiring reconciliation before assuming identical coverage.The nominal denominators are 70 and 63 tasks for Terminal-Bench 3.0 and 4.0, respectively.Source: E Supplementary Evaluation and… · p. 21

E.1 Separate base-initialized comparison

  • Direct PPO primarily improves Terminal-Bench Hard within the observed budget, but its step-zero control is its own initialization and it does not match RSR’s discovery and supervision costs.Table 4 retains historical aggregates without smoothing or imputation.Source: Table 4 · p. 21

F Data Coverage and Prospective Controlled Comparisons

This section describes the training and evaluation separation, the structural proxies used for difficulty strata, and qualifications on comparisons across stages and runs.

F Data Coverage and Prospective Controlled Comparisons

  • SWR100 evaluation tasks do not overlap with the nominal training pool of 2,400 semantic-verifier tasks and 600 supplementary tasks.Reproduction requires category frequencies, task and repository manifests, workflow mappings, and sampling weights; controlled tests should preserve protected verifiers and measure strict success separately from shaped reward.Source: F Data Coverage and Prospective… · p. 21
  • Figure 10’s difficulty strata use 468 benchmark–task pairs across six suites, with structural proxies rather than a universal scale of cognitive difficulty.Missing structural fields are assigned zero and may underestimate incompletely documented tasks.Source: F Data Coverage and Prospective… · p. 21
  • Figure 13 compares measured success at aligned local update counts, not total compute, because RSR-first refinement also incurs discovery and supervised-rewriting costs.Historical repeat protocols require reconciliation before treating the comparison as uniformly matched.Source: Figure 13 · p. 22
  • The four-suite stratified analysis covers 322 pairs, but its RSR-1 and RPR-2 evaluation provenance has qualifications that distinguish it from the primary suite-level comparison.The Terminal-Bench 2.1 RSR-1 analysis substitutes an alternative retained report; the RPR-2 report’s checkpoint identity has not been independently reconciled with the retained step-18 checkpoint.Source: F Data Coverage and Prospective… · p. 22

G Supplementary Training Diagnostics

The diagnostics report continued LHTB gains alongside an SWR100 plateau, while documenting how training-log displays and reward metrics were constructed. They also caution that aggregate curves do not isolate causal effects without matched task exposure and repeated seeds.

G Supplementary Training Diagnostics

  • RPR-1 full-export raw binary-reward endpoints increase from 0.27 to 0.41 and dense-reward endpoints from 0.10 to 0.20, unlike the displayed-prefix endpoints.The displayed prefix instead changes from 0.27 to 0.42 for raw binary reward and from 0.10 to 0.18 for dense reward; its turn range is 25–105 versus 25–182 in the full exports.Source: G Supplementary Training Diagnostics · p. 22
  • Table 5 proposes prospective component controls, not measured effect estimates, with comparisons matched on task streams, initialization, budgets, and repeated seeds.The proposed records include terminal reasons and strict success.Source: Table 5 · p. 23
  • RPR-1 display coordinates use t = 20i/30 without an origin sample and do not recover a common physical clock across logs or checkpoint evaluations.The mismatch log includes both step-20 entries; its causal trailing mean uses the current observation and up to four preceding observations without a restart reset.Source: G Supplementary Training Diagnostics · p. 23
  • RPR-2 reward and turn-count series use separate clocks and causal means, and their aggregate curves cannot isolate causal effects without matched task exposure and repeated seeds.Reward functions differ, so their absolute values cannot rank strict success; the historical FDR export labels interchange metric names.Source: G Supplementary Training Diagnostics · p. 23
  • LHTB process reward rises from 0.21 to 0.37 and strict completion from 0/46 to 5/46, while SWR100 success rises from 3.00% to 6.00%.The five checkpoints are Base, RSR-1, RPR-1 at step 20, RSR-2, and RPR-2 at step 18; SWR100 remains at 6.00% for the last three stages.Source: Figure 14 · p. 24

H Qualitative Case Studies and Trajectory Evolution

The case studies track verifier outcomes across stages on embedding-drift and Rust–C tasks. Later-stage trials pass the reported checks, while the output-contract results do not establish native AST evaluation or broader correctness.

H Qualitative Case Studies and Trajectory Evolution

  • Drift-monitor checker completion rises from 81.82% at Base to 100.00% at RSR-2 and RPR-2; Rust–C checker means reach 100.00% at both final stages.The Rust–C 75.00% RPR-1 mean combines one half-passing and one fully passing attempt; incomplete verifier scores are excluded, not imputed as zero.Source: H Qualitative Case Studies and… · p. 25
  • A representative RSR-1 drift repair clips signed MMD estimates, changing the estimator contract, but the failed gate does not identify clipping as its unique cause.Two other RSR-1 repeats pass; RPR-1 retains the signed estimator, while RSR-2 probes return approximately −0.02 for self-comparison.Source: H Qualitative Case Studies and… · p. 25
  • Both RPR-2 drift trials pass all 11 assertions, including held-out calibration, reference invariance, hysteresis, and zero-norm handling.The normalization safeguard replaces zero norms with one, and incoming samples update only the current buffer, leaving the reference unchanged.Source: H Qualitative Case Studies and… · p. 25
  • Both RSR-2 Rust–C repeats pass the protected wrapper and fuzz-equivalence gates after establishing a static executable and automatic musl-target rebuilding.The task requires an automatically built executable wrapper and exact output agreement with a hidden reference under fuzz testing.Source: H Qualitative Case Studies and… · p. 25
  • RSR-2 and RPR-2 each pass the 11 drift assertions in two trials, and each passes both Rust–C wrapper and fuzz-equivalence gates in two trials.The figures report separate stage evaluations; passing evidence does not imply an error-free history or a wholly native AST path.Source: H Qualitative Case Studies and… · p. 25, Figure 15 · p. 27, Figure 16 · p. 28
  • RPR-2 passes the tested output gates, but its first trial delegates expression evaluation to Python rather than the requested native AST path.The Rust binary’s static-pie linkage retains a Python runtime dependency; the results do not prove memory safety or complete specification fidelity.Source: H Qualitative Case Studies and… · pp. 25–26, Figure 16 · p. 28

Related papers

Loading 2610.22192v1…