PaperlayerView exact revision on arXiv

Source-linked AI summary

RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding

Jingxuan Xu, Zihang Xu, Yuechen Luo, Ziqi Miao, Wenqiang Zhu, Weihao Li, Junhao Fan, Xinshun Feng, Zhiyi Lai, Yanan Wu, Haoyang Huang, Ken Deng

arXiv:2610.31276v1cs.AI

TL;DR

Long-horizon coding agents may critique failures without changing their behavior. RIPPLE trains a single policy with prefix-only recovery guidance and stage-specific credit; across four benchmarks, it improves the base policy by an arithmetic mean of 4.8 score points.

  • Problem

    Coding agents may produce an evidence-grounded critique yet take a follow-up action inconsistent with their diagnosis, while task outcomes alone do not isolate recovery-stage quality.

    Source: §1 · p. 2

  • Method

    RIPPLE distills prefix-only recovery guidance and uses stage-specific signals with local validity gates to train reflection, diagnosis, and corrective action while retaining terminal task supervision.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    4.8 score points is RIPPLE’s arithmetic mean improvement over the base policy across four software-engineering benchmarks, without an online critic.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    Recovery gains concentrate where diagnostic evidence is sufficient and repository states remain reversible.

    Source: §5 · p. 8

    Evaluation uses Qwen3.5-27B in one coding scaffold, so transfer across model families and domains remains untested.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Long-horizon coding agents can observe a failure and generate a plausible critique yet continue along the same incorrect trajectory. We call this the Critique-to-Action Handoff Gap. We introduce RIPPLE, a recovery-oriented training framework that internalizes process-level guidance into a single self-recovering policy. A prefix-only teacher provides evidence-grounded recovery supervision without access to future actions or terminal outcomes. After lightweight distillation, reinforcement learning assigns stage-specific credit to reflection triggering, diagnosis, and corrective action. Local validity signals gate trajectory advantages so that intermediate recovery decisions do not indiscriminately inherit terminal outcomes. Across four repository-level software-engineering benchmarks, RIPPLE improves the base policy by an arithmetic mean of 4.8 score points, with no auxiliary critic at deployment. Controlled ablations isolate the benefits of local, stage-aligned credit, and matched-prefix interventions show that critique semantics affect subsequent correction. Recovery gains concentrate where the prefix contains sufficient diagnostic evidence and the repository state remains reversible within the remaining budget.

1 Introduction

The introduction identifies a gap between evidence-grounded critiques and subsequent behavior, then presents RIPPLE’s staged recovery training and benchmark improvement.

1 Introduction

  • The Critique-to-Action Handoff Gap describes an evidence-grounded critique that fails to induce effective correction, a mismatch not resolved by critique imitation or terminal rewards alone.Source: §1 · p. 2
  • RIPPLE uses a prefix-conditioned teacher and lightweight distillation, then trains one self-recovering policy to reflect, diagnose, and correct.Source: §1 · p. 2
  • 4.8 points is RIPPLE’s arithmetic mean improvement across four software-engineering benchmarks, without an online critic.Source: §1 · p. 2
  • Stage-specific process signals and local validity gates assign recovery-span credit while retaining terminal supervision for task completion.Source: §1 · p. 2
  • End-to-end comparisons, credit-assignment ablations, and matched-prefix interventions examine recovery’s functional contribution and the conditions under which it remains effective.Source: §1 · p. 2

2 Related Work

Related work spans repository-level coding agents, critique-based self-correction, and process-level credit assignment; RIPPLE links recovery stages to subsequent behavior.

2 Related Work

3 RIPPLE

RIPPLE trains a single policy to interleave repository work with recovery, using prefix-only supervision to initialize recovery behavior and reinforcement learning to refine it. Its process signals localize credit across recovery decisions while terminal reward continues to supervise task completion.

3.1 Overall Framework

  • The framework interleaves task actions and explicit self-critiques, whose three recovery spans determine when to reflect, why the trajectory deviated, and how to revise behavior.Only the learned policy is used at inference.Source: §3.1 · p. 3

3.2 Recovery Supervision and Policy Initialization

  • A lightweight teacher generates recovery boundaries and evidence-grounded critiques from trajectory prefixes, without conditioning on future actions, final patches, or terminal verifier outcomes.Its recovery trajectories insert critiques at recoverable positions and preserve corrective continuations to initialize interleaved recovery behavior.Source: §3.2 · p. 3

3.3 Recovery-Oriented Reinforcement Learning

  • Trigger credit is positive only when reflection occurs within a tolerance window of the earliest reliably diagnosable deviation; unmatched reflections receive zero signal.Unmatched boundaries have no local trigger span and remain governed by trajectory-level advantage.Source: §3.3 · p. 4
  • Diagnosis and corrective-action signals require prefix-supported diagnoses and test-grounded attributable improvement, while local validity gates how recovery stages inherit positive or negative trajectory advantage.The corrective-action signal is zero when there is no post-critique write or attributable improvement.Source: §3.3 · p. 5
  • Recovery-oriented reinforcement learning separates recovery into triggering, diagnosis, and corrective action, while retaining terminal supervision for task completion.The executable verifier supplies task reward; local process signals determine which recovery decisions inherit trajectory-level advantage.Source: §3.3 · pp. 4–5
  • When every rollout in a group has the same terminal reward, the trajectory-level advantage is zero, so sign gating cannot create a policy-gradient learning signal.Snapshot execution and frozen evaluators add training costs; deployment uses the self-recovering policy alone.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate RIPPLE’s benchmark performance, stage-aligned credit, and critique-guided recovery. Results show gains across benchmarks and indicate that recovery depends on both effective credit assignment and recoverable repository states.

4.1 Experimental Setup

  • The setup compares shared edit–execute methods under a 150-tool-turn cap and evaluates end-to-end resolution plus fixed-state recovery measures.Trained methods share a Recovery-SFT starting point and controlled training conditions; the Recovery Challenge Set contains 800 recoverable and 400 non-deviating prefixes.Source: §4.1 · pp. 5–6, §4 · p. 5

4.2 RQ1: End-to-End Effectiveness

  • RIPPLE improves the base policy by an arithmetic mean of 4.8 points across four benchmarks: 1.3 on Verified, 6.5 on Pro, 4.8 on Multilingual, and 6.4 on SWE-Compass.The repository-clustered 95% interval crosses zero on Verified but excludes zero on the other three benchmarks.Source: §4.2 · p. 6, Table 1 · p. 7
  • Recovery opportunities vary across benchmarks, and deviation density is associated with benchmark gains (r = 0.68).Recoverable deviations range from 17.8% on Verified to 43.6% on Pro; on Pro, gains rise from 1.8 points in the Easy quartile to 11.9 in Extreme.Source: §4.2 · p. 6
  • At approximately 1.17× actor tokens, RIPPLE scores 57.3 on Pro, versus 51.7 for longer execution and 53.2 for split-budget restarts.RIPPLE uses 1.23× base-policy p95 latency, remains within 0.6 points of the online same-teacher reference on each benchmark, and eliminates auxiliary deployment calls.Source: §4.2 · p. 6

4.3 RQ2: Locality and Stage Alignment

  • Event-local generic credit raises Full-Loop Recovery from 45.1% to 50.4%, while stage-aligned local credit reaches 59.6% and Pro 57.3.Stage-aware event-wide credit reaches 53.1% Full-Loop Recovery and 56.1% Pro; random-block and shuffled controls reach only 45.8% and 48.4% Full-Loop Recovery. Positive-credit contamination falls from 38.4% for outcome-only GRPO to 24.6% for event-local credit and 11.2% for RIPPLE.Source: §4.3 · pp. 6–7
  • Removing diagnosis credit lowers Diagnosis Accuracy from 81.5% to 69.8% and Action Alignment from 74.8% to 64.3%; removing action credit leaves diagnosis near 80.9% but lowers alignment to 56.8%.Removing action credit also lowers Pro from 57.3 to 54.0 and Full-Loop Recovery from 59.6% to 43.5%.Source: §4.3 · p. 7

4.4 RQ3: Functional Recovery Mechanisms

  • On fixed shared states, RIPPLE reaches 74.8% Action Alignment, 70.2% Functional Recovery, and 59.6% Full-Loop Recovery; its false-trigger rate is 8.9% versus 71.5% for periodic reflection.Grounded Evidence is 84.0% versus 81.5% Diagnosis Accuracy, and RIPPLE’s training trigger rate converges to 0.142, near the 0.15 reference deviation rate.Source: §4.4 · pp. 7–8, Table 2 · p. 7
  • With policy, prefix, repository snapshot, budget, and seed fixed, full critiques outperform removal and generic critiques; wrong-direction critiques perform worse than removal.Removal changes K-step recovery from 62.9% to 39.8%; generic critique reaches 42.0%, while wrong-direction critique reaches 25.6%. The recovery gains are conditional on frozen states and intervention design.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

RIPPLE addresses the Critique-to-Action Handoff Gap by internalizing prefix-only recovery guidance and assigning distinct credit to recovery stages. Its gains are strongest where diagnostic evidence is sufficient and repository states remain reversible.

5 Conclusion

  • RIPPLE improves the base policy by an arithmetic mean of 4.8 score points across four software-engineering benchmarks, without an online critic.Credit-assignment controls and matched-prefix interventions support the functional contribution of learned critiques.Source: §5 · p. 8
  • The framework internalizes prefix-only recovery guidance and distinguishes reflection triggering, diagnosis, and corrective action while retaining terminal task supervision.Local validity gates prevent ineffective recovery decisions from inheriting positive trajectory credit and protect useful corrections from unrelated negative credit.Source: §5 · p. 8
  • Recovery gains concentrate where diagnostic evidence is sufficient and repository states remain reversible within the remaining budget.The conclusion points to improved exploration in uniform-reward groups and direct estimation of local recovery effects across evolving repositories, programming languages, and deployment environments.Source: §5 · p. 9

Limitations

The limitations cover evaluation scope, supervision, recovery boundaries, and attribution. The appendix describes prefix-regenerated teacher labels and illustrates actionable guidance and abstention.

Limitations

  • Evaluation uses Qwen3.5-27B in one coding scaffold; transfer across model families and domains remains untested.Repository-disjoint post-training does not exclude pretraining exposure, and approximate process labels and offline candidate nomination from complete trajectories constrain supervision.Source: Limitations · p. 9
  • Sign gating provides no learning signal in uniform-reward groups and cannot overcome missing evidence or irreversible states within a fixed budget.Auxiliary supervision and snapshot tests add training cost without a complete amortized analysis; deployment uses 1.17× actor tokens and 1.23× p95 latency relative to the base policy.Source: Limitations · p. 9
  • Fixed-prefix metrics and interventions condition on selected states and do not characterize each policy’s on-policy failure distribution.The Verified gain’s confidence interval crosses zero; the mean averages benchmark-specific score differences rather than pooling issue-resolution rates, and F2P/P2F analyses cover only executable repair-like tasks.Source: Limitations · p. 9

A Recovery Supervision and Prefix-Only GRM Outputs

  • Training-time recovery guidance is regenerated from prefix records that exclude future actions, final patches, and terminal verifier outcomes.Complete offline trajectories nominate bounded candidate regions, but labels used for policy training are regenerated without those future outcomes.Source: A Recovery Supervision and Prefix-Only… · p. 12, §A.1 · p. 12

A.1 Candidate nomination and prefix regeneration

  • Candidate anchors include prefix-visible failures and inconsistencies, while infrastructure failures, flaky transitions, and future-only deviations are excluded.The GRM abstains when no prefix supports a concrete recovery direction, and field-level checks reject suffix actions, final patches, or terminal rewards.Source: §A.1 · p. 12

A.2 Illustrative GRM outputs

  • The illustrative actionable output cites prefix-present evidence and does not expose a hidden patch.The examples are excluded from quantitative evaluation; one identifies a configuration-precedence error after a failing test.Source: Figure A.1 · p. 12, §A.2 · p. 12
  • A transient tool failure alone does not establish a repository-level deviation, so the illustrative teacher abstains.The examples are schema-valid illustrations, not evaluation instances.Source: Figure A.2 · p. 12, §A.2 · p. 12

A.3 Recovery-trajectory construction and leakage audit

  • Recovery trajectories insert the GRM critique and retain a state-consistent corrective continuation.Candidate filtering excludes infrastructure failures, flaky transitions, and deviations observable only from the future suffix; the prefix-only audit compares prospective boundary recovery with a full-trajectory oracle.Source: §A.3 · p. 13, Table A.1 · p. 13

B Recovery Episodes, Challenge Set, and Metrics

The section defines recovery episodes, local attribution, and a fixed challenge set for measuring recovery independently of each policy’s own deviation distribution. Its statistical protocol uses repository-clustered inference and matched intervention conditions.

B Recovery Episodes, Challenge Set, and Metrics

  • Recovery is measured with trigger, diagnosis, and action metrics, with grounded evidence treated as auxiliary because cited evidence can still support an incorrect diagnosis.The metrics section fixes how recovery events are segmented, matched, and scored independently of the training-time GRM.Source: Table B.1 · p. 13, B Recovery Episodes, Challenge Set,… · p. 13

B.1 Episode segmentation and chronological matching

  • An explicit <recover> delimiter separates trigger, critique, and corrective-action spans, while malformed or consecutive delimiters receive no positive local signal.Critiques are matched one-to-one to teacher boundaries within a window, maximizing matched pairs before minimizing temporal distance.Source: §B.1 · p. 13

B.2 Fixed Recovery Challenge Set

  • All policies receive the same 800 recoverable and 400 non-deviating prefixes, so fixed-set metrics measure local recovery ability rather than endogenous deviation distributions.A separate evaluator labels boundaries and root causes, execution supplies test transitions, and a human-audited subset reports Cohen’s κ of 0.81, 0.76, and 0.73 for boundary, diagnosis, and action attribution.Source: §B.2 · p. 13

B.3 Statistical protocol

  • End-to-end uncertainty uses repository-clustered paired bootstrap, with Holm correction for four benchmark-level primary contrasts and McNemar tests for paired binary changes.Interventions hold the prefix, repository snapshot, remaining budget, policy checkpoint, and decoding seed fixed.Source: §B.3 · p. 14

C Training and Stage-Specific Credit Details

RIPPLE’s reported benchmark gains average 4.75 points, rounded to +4.8, with uncertainty differing across benchmarks. The section also describes stage-local credit gating and reports larger score gaps at greater static difficulty.

C Training and Stage-Specific Credit Details

  • Main comparisons use Qwen3.5-27B in a shared SWE-agent-style scaffold, and frozen recovery-support models are removed at deployment.The table caption identifies implementation settings relevant to comparison validity and deployment cost.Source: C Training and Stage-Specific Credit… · p. 14, Table C.1 · p. 14
  • Separate binary validity labels gate trigger, diagnosis, and corrective-action advantages, blocking positive credit to locally invalid stages and protecting valid stages from unrelated suffix failures.Ordinary action tokens retain the original advantage; the gate adds no process reward, so identical terminal rewards still yield zero policy-gradient advantage.Source: C Training and Stage-Specific Credit… · p. 14

D.1 Benchmark gains and uncertainty

  • 4.75 points is the arithmetic mean benchmark gain, reported as +4.8 after rounding; the Verified interval crosses zero while the other three exclude zero.The result is reported with repository-clustered 95% confidence intervals.Source: Table D.1 · p. 14

D.2 Difficulty and recoverable-deviation prevalence

  • Base/RIPPLE scores are 71.2/73.0 on Easy, 58.4/62.8 on Medium, 44.8/52.7 on Hard, and 28.8/40.7 on Extreme.The equal-weight averages reproduce Pro’s 50.8/57.3 scores; recoverable-deviation prevalence ranges from 17.8% on Verified to 43.6% on Pro.Source: §D.2 · p. 14, Figure D.1 · p. 14

D.3 Task, language, and distribution-shift slices

Task, language, and metadata-shift slices locate performance differences without assigning them a causal role. On Pro, RIPPLE has the highest repair rate and lowest damage rate among trained comparisons; the locality diagnostic distinguishes sign gating from validity-assessment disagreement.

D.3 Task, language, and distribution-shift slices

  • Table D.2’s grouped slices reproduce benchmark aggregates up to rounding and locate performance differences without attributing them causally to language or feedback type.Pro’s metadata-shift score is fixed before inference and split at its median.Source: §D.3 · pp. 14–15, Table D.2 · p. 15

D.4 Repair-versus-damage decomposition

  • Repair minus Damage equals the absolute Pro gain; RIPPLE has the highest repair rate and lowest damage rate among trained comparisons.Both rates use all Pro instances as the denominator.Source: Table D.3 · p. 15, §D.4 · p. 15

E.1 Complete locality and alignment ladder

  • Under Equation 11, blocks labeled hm k = 0 cannot receive positive shaped advantage; contamination signals disagreement in validity assessment, not failure of sign gating.The contamination diagnostic must be interpreted separately from the algebraic sign-gating guarantee.Source: §E.1 · p. 15

E.2 Leave-one-stage-out analysis

The leave-one-stage-out results distinguish the effects of trigger, diagnosis, and action credit. Removing corrective-action credit produces the largest declines in Pro and Full-Loop Recovery.

E.2 Leave-one-stage-out analysis

  • Removing diagnosis credit lowers diagnosis to 69.8% and alignment to 64.3%; its Pro score is 54.8 and Full-Loop Recovery is 47.6%.Source: §E.2 · p. 15
  • Removing trigger credit lowers trigger F1 from 0.79 to 0.67 while preserving diagnosis at 80.8%; Pro is 55.6 and Full-Loop Recovery is 50.8%.Source: §E.2 · p. 15, Figure E.2 · p. 16
  • Removing action credit preserves diagnosis at 80.9% but lowers alignment to 56.8%, Pro to 54.0 from 57.3, and Full-Loop Recovery to 43.5% from 59.6%.The figure identifies corrective-action credit removal as producing the largest end-to-end drop in Pro and Full-Loop Recovery.Source: §E.2 · p. 15, Figure E.1 · p. 16

F.1 Extended matched-prefix intervention

Matched-prefix interventions test critique content and timing while holding the policy, prefix, repository snapshot, budget, and decoding seed constant. Full critique benefits are greatest when diagnostic evidence is sufficient and the repository state remains reversible within budget.

F.1 Extended matched-prefix intervention

  • The intervention changes only the critique at a matched boundary, fixing the policy checkpoint, prefix, repository snapshot, remaining tool-turn budget, and paired decoding seed.Conditions include critique removal, generic and wrong-direction critiques, and extensions varying evidence, direction, timing, or teacher output; conclusions remain conditional on selected prefixes.Source: §F.1 · p. 16
  • The full critique adds 23.1 recovery points and reduces P2F by 4.4 points relative to removal; direction alone retains 15.6 benefit points, versus 4.9 for evidence alone.The internal critique is 3.6 recovery points below the teacher reference, with a 0.3-point P2F gap.Source: §F.1 · p. 16, Figure F.1 · p. 16, Table F.1 · p. 17

F.2 Label noise and selective triggering

  • On 400 non-deviating prefixes, RIPPLE’s false-trigger rate is 8.9%, versus 71.5% for prompted periodic reflection; flipping 10–20% of process labels costs fewer than 1.5 Pro points.Source: §F.2 · p. 17, Table F.2 · p. 17

F.3 Evidence sufficiency and state reversibility

  • When both sufficient diagnostic evidence and a reversible repository state are present (n = 248), RIPPLE gains 20.4 points over the base policy; gains are single-digit otherwise.Explicit and implicit evidence support similar improvements in the 248-prefix decomposition.Source: §F.3 · p. 17, Table F.3 · p. 17, Table F.4 · p. 17

Related papers

Loading 2610.31276v1…