PaperlayerView exact revision on arXiv

Source-linked AI summary

ReTerm: Recovery Training for Self-Improving Terminal Agents

Wei Xie, Kaituo Feng, Shiming Fu, Yong Liu, Xiangyu Yue, Shuangjia Zheng

arXiv:2610.18484v1cs.AIcs.LGcs.SE

TL;DR

Terminal agents can struggle to recover from errors during long-running execution, and existing hand-crafted recovery errors may differ from agents’ own failures. ReTerm builds recovery demonstrations from self-generated failures and trains recovery through online reinforcement learning; its ReTerm-9B improves over its base model on Terminal-Bench 2.0.

  • Problem

    Terminal agents frequently encounter execution errors and struggle to recover, while hand-crafted errors may mismatch their naturally generated failures and prior recovery training leaves RL unexplored.

    Source: §1 · p. 1

  • Method

    ReTerm constructs recovery demonstrations from self-generated failure states and uses ReTerm-GRPO to fork independent recovery attempts whose verifier outcomes guide online learning.

    Source: §1 · pp. 1–2

  • Results

    34.8% Pass@1 on Terminal-Bench 2.0, a 14.6-point gain over the Qwen3.5-9B base model.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    ReTerm’s recovery capability can co-evolve with task-solving capability through recovery-oriented post-training.

    Source: §1 · p. 2

Abstract

from arXiv · show

Terminal agents have shown strong abilities in complex coding tasks. However, they often struggle to recover from errors during long-running execution (e.g., command errors, faulty implementations), eventually leading to task failures. To this end, we present ReTerm, a self-improving terminal agent trained by learning how to recover from its own failed terminal tasks. To support its training, we construct a customized data pipeline and produce ReTerm-10K, a high-quality dataset that provides effective recovery demonstrations for SFT cold start. Furthermore, we propose ReTerm-GRPO, an online RL algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts. By learning from whether these attempts successfully complete the task, the agent self-improves through its own recovery experience. Built on Qwen3.5-9B, our ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0, improving over the base model by 14.6 points. Further analysis shows that our method achieves consistently higher recovery rates across diverse error categories compared with standard GRPO training. All code, data and model are released at https://github.com/twothsnd/ReTerm.

1 INTRODUCTION

Terminal agents can struggle to recover from errors during extended execution, and hand-crafted recovery errors may differ from agents’ naturally generated failures. ReTerm trains recovery from self-generated failure states using curated demonstrations and online reinforcement learning.

1 INTRODUCTION

  • Prior terminal agents struggle to recover from execution errors, while hand-crafted error injection may mismatch natural failures and leaves RL’s recovery potential unexplored.TermiGen is described as using actively injected errors and SFT alone.Source: §1 · p. 1
  • ReTerm is a recovery-oriented self-improving terminal agent trained on failures produced during its own execution.Its recovery training targets continuation from faulty environment states rather than solving tasks from a clean environment.Source: §1 · pp. 1–2
  • ReTerm-GRPO forks multiple independent recovery attempts from each sampled failure state and uses verifier outcomes for group-relative learning.A recovery task queue stores failures across training steps, prioritizing newly constructed tasks and removing the oldest when capacity is exceeded.Source: §1 · p. 2
  • 34.8% Pass@1 on Terminal-Bench 2.0 and 58.4% Pass@1 on SWE-bench Verified; ReTerm also improves over the base model on out-of-domain benchmarks.The reported out-of-domain gains are 20.0 points on AIME 2025 and 11.0 points on SpreadsheetBench Verified; recovery rates are consistently higher than standard GRPO across five error categories.Source: §1 · p. 2
  • ReTerm-10K is built from base-model failures, stronger-model recovery continuations, and four-step quality filtering.The filters remove unsuccessful, malformed, non-compliant, or inefficient trajectories, retaining recovery demonstrations for SFT cold start.Source: §1 · p. 2, Figure 2 · p. 3

2 RELATED WORK

Prior work advances terminal-agent task execution through verifiable environments and reinforcement learning, while self-correction research has expanded from static reasoning to interactive terminal settings. ReTerm’s related-work framing emphasizes recovery from self-generated failure states.

Terminal Agent

  • Terminal-agent research has advanced task execution through scalable verifiable environments and reinforcement learning, while ReTerm complements this work by studying recovery ability.The passage cites work on TMax’s synthetic tasks and update filtering, and ECHO’s auxiliary observation-prediction objective.Source: §2 · p. 3

Self-Correction

  • Self-correction methods address static reasoning and interactive settings, where erroneous actions can alter the environment and affect subsequent execution.TermiGen uses actively injected errors for SFT; ReTerm instead trains recovery from self-generated failure states.Source: §2 · p. 3

3 METHOD

ReTerm builds recovery demonstrations from failed terminal attempts and trains the policy to recover from evolving failures during online RL. Its pipeline preserves failure states, generates and filters continuations, then combines original-task and recovery-task learning.

3.1 RECOVERY DATASET FORMULATION

  • The recovery-data pipeline samples base-model failures, restores their saved states, generates teacher continuations, and filters them for quality before SFT.Filtering checks verifier success, format, compliance, and efficiency; the final dataset contains roughly 10K trajectories.Source: §3.1 · p. 4
  • SFT conditions on the task instruction, failed interaction history, and a minimal recovery prompt, while computing loss only on teacher-generated continuation tokens.The structured workflow used to generate demonstrations is removed from the training examples.Source: §3.1 · p. 5

3.2 RETERM-GRPO

  • Because policy failures may shift beyond the fixed SFT dataset, ReTerm-GRPO adapts recovery training alongside task-solving as the policy evolves.The method is intended to let self-recovery capability co-evolve with task-solving ability.Source: §3.2 · pp. 5–6
  • For online recovery training, ReTerm-GRPO turns failed original-task rollouts into recovery tasks and generates multiple independent attempts from each resulting failure state.Each recovery task retains the original instruction and failed interaction history, and is sampled from a queue in later iterations.Source: §3.2 · p. 5
  • A bounded recovery-task queue carries failures across training steps, prioritizes newer tasks, and limits mismatch with the evolving policy’s failure distribution.It supplements batches with few failures without requiring extra original-task rollouts solely for failure collection.Source: §3.2 · p. 6
  • ReTerm-GRPO jointly optimizes original-task and recovery-task groups using separately computed group-relative advantages and combined objectives.Verifier rewards are normalized within each group before joint optimization.Source: §3.2 · p. 6

4 EXPERIMENT

Experiments compare ReTerm with its base model and training variants across agentic benchmarks, harnesses, and task domains. Ablations, error analysis, and reward curves examine recovery training’s contributions and behavior.

4.1 EXPERIMENTAL SETUP

  • The evaluation spans seven benchmarks, using a shared ReAct-style Bash harness for Qwen3.5-9B and ReTerm-9B comparisons.Benchmarks include long-horizon terminal tasks, repository software issues, math reasoning, multilingual coding, and spreadsheet manipulation; inference uses a fixed task seed of 0.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • ReTerm-9B gains 16.4, 14.6, and 13.2 points over Qwen3.5-9B on OpenThoughts-TBLite, Terminal-Bench 2.0, and SWE-bench Verified, respectively.It also outperforms TMax-9B across all three benchmarks and several 32B agents on selected benchmarks.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 GENERALIZATION

  • ReTerm-9B consistently improves over the base Qwen3.5-9B model across all four agent harnesses, gaining 14.6 percentage points when training and evaluation harnesses match and 6.7–12.4 points under three alternatives.Source: §4.3 · p. 8
  • ReTerm-9B performs better than the base model on all four out-of-domain benchmarks, with gains of +20.0 points on AIME 2025 and +11.0 on SpreadsheetBench Verified.Positive changes are also reported on AIME 2024 and Aider Polyglot.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 ABLATION STUDY

  • SFT, standard GRPO, and ReTerm-GRPO each improve Terminal-Bench 2.0 performance over the base model, with ReTerm-GRPO strongest among the four settings.The comparison includes Base, SFT, standard GRPO, and ReTerm-GRPO; the authors report an additional benefit from explicitly optimizing recovery behavior beyond standard RL.Source: Table 4 · p. 8, §4.4 · pp. 8–9

4.5 ERROR ANALYSIS

  • ReTerm-GRPO has higher recovery rates than both the base model and standard GRPO across all five error categories, but rates remain at or below 32% outside tool use.The categories are planning, tool use, implementation, verification, and resource management; outcomes were judged by an LLM and manually verified.Source: Figure 5 · p. 9, §4.5 · p. 9

4.6 TRAINING CURVES

  • Reward improves rapidly early in training for both original and recovery tasks, then stabilizes; the recovery-task group improves more from its initial performance.Reward on original tasks also improves and remains stable during training.Source: Figure 6 · p. 9, §4.6 · p. 9

5 CONCLUSION

ReTerm combines recovery demonstrations for SFT cold start with online RL from self-generated failures. The data-curation protocol emphasizes reconstructing the task, diagnosing failure, testing assumptions, and handling inherited state.

5 CONCLUSION

  • ReTerm uses ReTerm-10K recovery demonstrations for SFT cold start and ReTerm-GRPO to train recovery from self-generated execution errors.The conclusion describes recovery capability as co-evolving with task-solving capability and reports effectiveness across diverse benchmarks.Source: §5 · p. 10

A.2 DATA CURATION

  • The curation protocol audits inherited state, reconstructs the original task, diagnoses failure, tests assumptions independently, and keeps, repairs, isolates, or discards state as appropriate.It aims to preserve useful progress while preventing state from a wrong hypothesis from misleading later recovery.Source: §A.2 · pp. 13–14

A.3 ERROR ANALYSIS

The annotation protocol audits complete first-attempt trajectories and verifier results, assigning category-specific error states based on concrete evidence. It distinguishes error types and requires evidence that a correction worked before marking an error recovered.

A.3 ERROR ANALYSIS

  • Auditors examine task requirements, agent actions, tool outputs, final artifacts, and official verifier results to assess the complete first-attempt trajectory.Unresolved requires a demonstrated error that contributed to final failure and was not effectively corrected before termination.Source: §A.3 · pp. 15–16
  • The audit distinguishes planning, tool-use, implementation, verification, and resource-management errors, with category-specific boundaries for each.For example, defective submitted files count as implementation errors, while a single timeout alone is insufficient for resource-management error.Source: §A.3 · p. 15
  • Each category receives one state, and categories are assessed independently; an error is Recovered only when the trajectory shows the error, corrective action, and confirming evidence.An attempted fix or an agent’s success claim alone is insufficient, and overall task success does not by itself establish recovery.Source: §A.3 · p. 15

B CASE STUDY

The case study resumes the GRPO baseline and ReTerm-GRPO from identical failure states and interaction histories to compare their recovery behavior.

B CASE STUDY

  • Five representative Terminal-Bench 2.0 failure states, one for each error type, are used to compare how GRPO and ReTerm-GRPO diagnose failures, adjust actions, and pursue completion.The cases are distribution-search, qemu-alpine-ssh, tune-mjcf, break-filter-js-from-html, and git-multibranch.Source: B CASE STUDY · p. 16

B.1 PLANNING ERROR

The agent cycles through grouped distributions and power laws but repeatedly fails to meet both KL targets. A two-segment power-law search eventually produces a distribution that passes both tolerance checks.

B.1 PLANNING ERROR

  • Two-group and modified grouped searches fail to satisfy both KL constraints, and some optimization scripts encounter syntax errors.The agent eventually returns to the power-law family after these unsuccessful searches.Source: §B.1 · pp. 16–18
  • KL(P||U)=10.207118 while KL(U||P)=9.999934 in the final check of the saved simple power law, so it remains outside tolerance.The closest reported simple power law has α ≈1.956, but does not meet tolerance.Source: §B.1 · pp. 17–18
  • The agent identifies a limitation in two-group and power-law models: they cannot independently control the two KL values because of their mathematical relationship.Source: §B.1 · p. 18
  • KL(P||U)=9.999864636366118 and KL(U||P)=9.99999297833879; the saved distribution passes both 0.001 checks.The verified distribution has shape (150000,) and sums to 0.9999999999999994.Source: §B.1 · p. 19

B.2 TOOL USE ERROR

The agent repeatedly encounters QEMU launch errors, especially a conflict on host port 2222, before getting QEMU running with that port listening. The supplied passages do not establish that SSH inside the guest was configured.

B.2 TOOL USE ERROR

  • Attempts to launch QEMU repeatedly fail because it cannot set up the host-forwarding rule for port 2222.The agent discovers an existing QEMU process using that forwarding rule and kills it before retrying.Source: §B.2 · pp. 20–21
  • The first launch also uses an unavailable CPU model, prompting a switch to qemu64 while preserving the host-forwarding setup.Source: §B.2 · p. 21
  • QEMU is eventually reported alive with port 2222 listening after a launch using the generic qemu64 CPU model.The Alpine ISO and QEMU executable are present.Source: §B.2 · pp. 19, 21

B.3 IMPLEMENTATION ERROR

The agent finds a MuJoCo configuration that matches the reference final state while running substantially faster. Earlier attempts illustrate the trade-off: faster settings can violate the state tolerance, while matching settings may miss the runtime target.

B.3 IMPLEMENTATION ERROR

  • Faster timestep changes fail the final-state tolerance, while a reference-like configuration matches state but runs at 100.33% of reference time.Some tested candidates also fail XML parsing, and comparison scripts contain errors.Source: §B.3 · pp. 22–23
  • The accepted solver names map Newton to 2, CG to 1, and PGS to 0; PGS with fewer iterations is tested for speed while checking state difference against 10^-5.Source: §B.3 · p. 23
  • A reported run took 0.5993 seconds, 0.49 of its 1.2331-second reference time, with final-state difference 0.0000, satisfying the at-most-60%-runtime and 10−5 state-match checks.Separately, PGS with 10 iterations took 0.708s versus 1.476s for Newton/100 and matched within 1e-5.Source: §B.3 · pp. 21, 23

B.4 VERIFICATION ERROR

The agent initially treats a silent test-script exit as evidence that its HTML payload works, despite noticing that the link requires a click. It later attempts direct browser verification by waiting for an alert.

B.4 VERIFICATION ERROR

  • The initial javascript: link requires a click, so it does not meet the task’s requirement for an automatically triggered alert.The agent nevertheless treats silent test output as evidence that the payload works.Source: §B.4 · pp. 23–24
  • The agent interprets a test run reported as passed as proof of automatic alert execution and submits the attempt, but the official evaluation is still pending.Source: §B.4 · p. 24
  • The agent eventually loads the HTML in headless Chrome and waits for an alert, rather than relying only on the test script’s exit code.Source: §B.4 · p. 24

B.5 RESOURCE MANAGEMENT ERROR

The task was to configure SSH-accessible Git deployment for separate main and dev HTTPS pages. The inherited attempt left deployment unverified, while the later ReTerm-GRPO attempt is labeled a successful recovery.

B.5 RESOURCE MANAGEMENT ERROR

  • The task required SSH Git access and a post-receive hook deploying main and dev branches to their corresponding HTTPS pages.The setup also called for Nginx and a self-signed certificate.Source: §B.5 · p. 25
  • The inherited setup had useful components, but the agent stopped before a Git push or deployed-page check, leaving SSH and deployment unresolved.The Nginx syntax check passed, but that did not verify the full workflow.Source: §B.5 · p. 25
  • The ReTerm-GRPO attempt is labeled a successful recovery.The supplied passage does not describe the final verification or deployment outcome.Source: §B.5 · p. 26
  • The inherited hook deployed to both pages for every branch push, and SSHD and Nginx were not running.The recovery agent audited the repository, hook, directories, configuration, and services before revising the setup without deleting the repository.Source: §B.5 · pp. 26–27
  • The recovery work included starting SSH and Nginx, rebuilding deployment components, and resetting the Git user's password after SSH access was denied.The supplied passages do not establish whether the subsequent access test or end-to-end deployment succeeded.Source: §B.5 · p. 27

Related papers

Loading 2610.18484v1…