PaperlayerView exact revision on arXiv

Source-linked AI summary

OPRD: On-Policy Reminder Distillation for Long-Horizon Coding Agents

Yuechen Luo, Zhiyi Lai, Miao Ziqi, Zihang Xu, Jingxuan Xu, Songwei Yu, Wenqiang Zhu, Weihao Li, Junhao Fan, Haoyang Huang, Yanan Wu, Ken Deng

arXiv:2610.21496v1cs.CL

TL;DR

Long-horizon coding agents need to preserve user constraints and use tool feedback, but final verification outcomes provide limited process-level supervision. OPRD trains reminder-free student rollouts using scoped Reminders and routes successful and failed trajectories to different objectives. It improves benchmark-averaged task resolution and reduces diagnostic error events.

  • Problem

    Final verification outcomes provide limited process-level supervision for constraint adherence and feedback use at different scopes.

    Source: §1 · pp. 1–2

  • Method

    OPRD uses Global and observation-anchored Turn-local Reminders with outcome-based routing: successful rollouts receive self-imitation, while failed rollouts receive Reminder-conditioned distillation.

    Source: §1 · p. 2

  • Results

    OPRD improves benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model and reduces aggregate error-event count from 446 to 222 (50.2%).

    Source: §1 · p. 2

  • Takeaways & Limitations

    The student operates without Reminders at inference after internalizing scope-specific corrections during training.

    Source: §1 · p. 2

Abstract

from arXiv · show

Reliable repository-level coding requires agents not only to generate correct code but also to preserve constraints and respond to tool feedback across long interactions. However, our diagnostic analysis shows that agents often fail to act on relevant information already available in context, while final Pass/Fail outcomes provide only sparse process supervision. We introduce On-Policy Reminder Distillation (OPRD), which combines Multi-Scope Reminder supervision with outcome-based routing on reminder-free student rollouts. Successful rollouts provide autoregressive self-imitation targets, whereas failed rollouts receive Reminder-conditioned reverse-KL supervision from a teacher that re-evaluates the same student-generated prefixes. A Global Reminder converts final-test diagnostics into trajectory-level guidance, while Turn-local Reminders apply only to the next decision following the relevant environment observation. The teacher and student share a two-stage initialization from Coding SFT and Instruction-Following (IF) SFT. At inference, the student operates without Reminders. On SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual, OPRD improves the benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model. Under the same diagnostic protocol, it reduces the aggregate error-event count from 446 to 222, a 50.2% reduction.

1 Introduction

The introduction frames reliable long-horizon coding as preserving task constraints and responding to tool feedback, while noting that outcome-based training offers limited process supervision. OPRD trains on reminder-free student rollouts with scoped reminders and outcome-based routing, reporting gains on software-engineering benchmarks and fewer diagnostic error events.

1 Introduction

  • On 200 SWE-rebench V2 validation tasks, the same annotation protocol found fewer error events across four categories with Reminders than without them.Post-hoc annotation audits completed trajectories, whereas runtime Reminder selection uses only the currently visible interaction prefix.Source: §1 · p. 1
  • Existing imitation and executable-outcome training do not specify which constraints persist or which observation should guide the next decision; runtime Reminders also consume context and depend on an agent scaffold.The introduction identifies risks that local corrections may be diluted or propagated too broadly, and that models may ignore constraints and feedback.Source: §1 · pp. 1–2
  • OPRD improves benchmark-averaged task resolution rate by 11.3 points over the open-source Qwen3.5-4B base model and reduces aggregate error-event count from 446 to 222 (50.2%).The evaluations cover SWE-bench Verified, SWE-bench Pro, and SWE-bench Multilingual; the error-event reduction uses the same diagnostic protocol.Source: §1 · p. 2
  • OPRD routes successful reminder-free student rollouts to autoregressive self-imitation and failed rollouts to reminder-conditioned reverse-KL distillation.Global Reminders provide trajectory-level guidance, while Turn-local Reminders apply only to the next decision after a relevant observation.Source: §1 · p. 2
  • Coding SFT followed by Instruction-Following SFT provides a shared teacher–student initialization intended to improve responsiveness to constraints, execution feedback, and Reminders.The introduction presents this cold start as a stronger starting point for subsequent reminder-conditioned distillation.Source: §1 · p. 2

2 Related Work

Related work covers repository-level coding agents and training resources, feedback-guided reasoning and reflection, and on-policy or privileged-context distillation. OPRD combines explicitly scoped Reminders with outcome-based routing for long-horizon coding.

Repository-Level Software Engineering Agents

  • Repository-level agent work spans tool interfaces, program-aware search, structured repair, trajectory and verifier resources, task generation, executable-outcome training, and capability evaluation.Examples named include SWE-agent, AutoCodeRover, Agentless, SWE-Compass, and KAT-Coder.Source: §2 · p. 2

Feedback-Guided Reasoning and Reflection

  • Feedback-guided methods use observations, reflection, critique, search, and execution or runtime verification to guide correction, motivating learning corrective-signal use without runtime Reminder injection.The passage names ReAct, Reflexion, self-refinement, FTTT, and FLARE among related approaches.Source: §2 · p. 2

On-Policy and Privileged-Context Distillation

  • OPRD combines explicitly scoped Reminders and outcome-based routing with on-policy and privileged-context distillation approaches for long-horizon coding.Prior work described includes reverse- or skew-KL objectives, context distillation, and context-conditioned teacher matching through reverse KL.Source: §2 · p. 2

3 Method

OPRD frames repository coding as multi-turn interaction where task constraints and environment feedback have different temporal scopes. It trains a reminder-free student from successful and failed on-policy rollouts, using a shared two-stage supervised initialization and reminder-conditioned teacher guidance.

3.1 Problem Formulation

  • A coding task unfolds through assistant turns and environment observations, while an executable verifier reduces the final repository state to a binary pass or failure.Source: §3.1 · p. 3

3.2 Multi-Scope Reminders

  • Task-level constraints must persist across turns, but test errors and tool responses usually concern specific decisions, so OPRD separates Global from Turn-local Reminders.Global Reminders provide persistent task-level guidance; Turn-local Reminders target a decision point and apply only to the next assistant turn.Source: §3.2 · pp. 3–4

3.3 Two-Stage Supervised Cold Start

  • The two-stage cold start first uses Coding SFT for repository understanding, tool use, and editing, then IF SFT to improve responsiveness to constraints and feedback.Source: §3.3 · p. 4

3.4 Multi-Scope Reminder-Conditioned Distillation

  • The student generates reminder-free rollouts from a shared teacher–student initialization, and only the student is deployed after training.The teacher, reminder-construction module, and reminders are removed at inference.Source: §3.4 · pp. 4–5
  • Successful rollouts are routed to autoregressive self-imitation, while failed rollouts receive reminder-conditioned distillation rather than imitation of erroneous decisions.The teacher evaluates the same student-generated prefixes, with reminders supplied only to the teacher; reverse KL weights discrepancies according to the student distribution.Source: §3.4 · pp. 4–5

4 Experiments

Experiments evaluate OPRD across three software-engineering benchmarks using shared supervised preparation and reminder-free inference. Results show gains across both model sizes, with ablations examining reminder scope, distillation design, and process errors.

4.1 Experimental Setup

  • Training uses 8,000 Coding SFT trajectories, 2,000 IF SFT reminder-following examples, and 4,000 tasks for online OPRD rollouts; main results use reminder-free inference.Evaluation covers SWE-bench Verified, Pro, and Multilingual under a shared agent harness and inference configuration.Source: §4.1 · p. 5

4.2 Main Results

  • 47.9 average resolution rate for Qwen3.5-4B improves 11.3 points over the base model, while Qwen3.5-9B reaches 55.9, 11.5 points above its open-source checkpoint.The 4B result is also 6.3 points above IF SFT; the 9B result is 5.4 points above IF SFT. Gains hold across all three benchmarks for both backbones.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Ablation Study

  • At T = 0.6, Top-100 and full-vocabulary KL perform similarly, while Top-100 with T = 1.0 reaches the best average score of 47.9.Both lower and higher Top-100 temperatures perform worse.Source: §4.3 · p. 7, Table 3 · p. 7
  • Reverse KL outperforms forward KL under both training strategies, and outcome-routed OPRD gains 1.9 points over OPSD at fixed KL direction.OPRD assigns self-imitation to verified successes and Reminder-conditioned KL to failures; reverse KL focuses updates on student-favored candidates the teacher considers inappropriate.Source: §4.3 · pp. 6–7, Table 3 · p. 7
  • The balanced 1:1 weighting of successful-trajectory self-imitation and failed-trajectory Reminder distillation performs best on all three benchmarks, with a 47.9 average.SWE-bench Pro and Multilingual are more sensitive to imbalance because they include more tasks requiring correction from tool feedback and cross-file context.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • Four evenly spaced teacher refreshes perform best on every benchmark, reaching a 47.9 average; shared teacher–student parameters perform worst.The four-refresh schedule copies student parameters to the teacher at 20%, 40%, 60%, and 80% of training.Source: §4.3 · pp. 6, 8
  • Combining Global and Turn-local Reminders performs best across preparation stages, with a 4.9-point average benefit after IF SFT.Turn-local Reminders generally provide larger gains, while Global Reminders add complementary cross-turn guidance.Source: Figure 5 · p. 7, §4.3 · p. 8
  • After IF SFT, aggregate Reminder-following reaches 77.1; individually, all four functional categories outperform reminder-free evaluation, and enabling all raises average resolution from 36.6 to 39.2.On 200 validation tasks, OPRD also reduces diagnostic triggers from 446 to 222, a 50.2% reduction; the largest category decrease is in Post-Edit Control and Validation.Source: §4.3 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5 Conclusion

OPRD combines multi-scope reminders, outcome-based routing, and a two-stage supervised cold start to improve long-horizon coding agents’ task resolution and reduce adherence failures. The appendix describes supplementary implementation, reminder-construction, and diagnostic-annotation materials.

5 Conclusion

  • OPRD routes successful rollouts to autoregressive self-imitation and failed rollouts to Reminder-conditioned reverse-KL distillation, using Global Reminders for trajectory guidance and Turn-local Reminders for the next decision.Coding SFT followed by Instruction-Following SFT improves responsiveness to corrective signals; the student operates without Reminders at inference.Source: §5 · p. 9

A Appendix

  • The appendix provides training implementation and hyperparameters, Reminder construction and diagnostic annotation procedures, evaluation results, and ablations.Sections B–D cover implementation, Reminder construction and taxonomy, and the deterministic and model-based annotation protocol.Source: A Appendix · p. 12

B Training Implementation and Hyperparameters

The Qwen3.5-4B OPRD training setup specifies distributed implementation, model configuration, periodic teacher refreshes, and reverse-KL support selection.

B Training Implementation and Hyperparameters

  • The main configuration uses sequence length 131,072, global batch size 32, three epochs, and peak learning rate 1 × 10−5 with cosine decay.The implementation uses VeOmni, FlashAttention 2, fused MoE operators, FSDP2, Ulysses sequence parallelism, and DCP checkpoints.Source: B Training Implementation and… · p. 12
  • The teacher is refreshed from the current student at 20%, 40%, 60%, and 80% of training, then held fixed until the next refresh.The teacher is not optimized by gradient descent.Source: B Training Implementation and… · p. 12
  • Reverse KL is evaluated on a shared Top-100 support selected by the largest joint importance scores from the student and teacher distributions.Both distributions are renormalized on the same selected tokens before reverse KL is computed.Source: B Training Implementation and… · p. 12
  • The main loss weights successful-trajectory self-imitation and failed-trajectory Reminder-conditioned distillation equally, with λ+ = 1 and λ− = 1.Source: B Training Implementation and… · p. 12

C.1 Construction Procedure and Functional Taxonomy

The construction procedure separates how Reminders are obtained, filtered, inserted, and verified from the functional taxonomy of the failures they address.

C.1 Construction Procedure and Functional Taxonomy

  • Reminder construction and functional categorization are independent of temporal scope: Global Reminders provide persistent final-test guidance, while runtime construction targets observation-anchored Turn-local Reminders.Source: §C.1 · p. 12
  • Candidates are retained only when their triggers are observable without hidden correctness labels, their desired next actions are local and verifiable, and they neither reveal reference patches nor invent task requirements.Executable ReBench evaluation determines correctness; trajectories serve only as behavioral evidence.Source: §C.1 · p. 12
  • Passive Reminders follow tool results that expose concrete issues and apply once; active Reminders test a trigger at a pre-action boundary, discard the probe, and reroll after insertion.The active verifier checks the rerolled tool choice, arguments, target path, edit scope, and immediate follow-up behavior.Source: §C.1 · p. 12
  • Insertion requires deterministic hard-rule matching or a contextual judge meeting confidence 0.90 and phase gates, followed by caps, deduplication, and evidence checks.If candidates fail the gates, no Reminder is inserted; inserted text follows, rather than sits inside, the tool response.Source: §C.1 · p. 12
  • The procedure separately records trajectory capture, Reminder insertion, required local behavior, and executable verification of the final source-only patch.These quantities have different denominators and are not substituted for one another.Source: §C.1 · p. 12

C.2 Global Reminder Construction and Evidence Judge

Global Reminders are built from eligible failed attempts and use verifier evidence to guide a full retry without exposing a reference solution.

C.2 Global Reminder Construction and Evidence Judge

  • A Global Reminder is constructed only after executable verification rejects a complete attempt and is placed at the input of the next full retry.Eligible attempts must contain a non-empty patch, receive zero benchmark reward, be retryable, apply successfully, and remain unaccepted; empty or unapplied patches, successful attempts, and infrastructure-only records are excluded.Source: §C.2 · pp. 12–13
  • The evidence judge extracts pass, pending, and fail counts, touched files, and up to five deduplicated signals, prioritizing root-cause-bearing failures over generic ones.Its feedback reports what failed and the evidence source without revealing a reference patch or prescribing an unsupported implementation.Source: §C.2 · p. 13

C.3 Four-Class Functional Taxonomy

The finalized rubric maps 32 fine-grained Reminder categories into four mutually exclusive reporting classes. It assigns each error event one most-specific category, while allowing multiple Reminders only for distinct supported risks and next actions.

C.3 Four-Class Functional Taxonomy

  • 32 fine-grained categories—10 deterministic hard rules and 22 model-judged contextual strategies—map into four reporting classes without adding fine-grained categories.The four classes are Tool-Feedback Recovery, Scope and Source Grounding, Scenario-Condition Reasoning, and Post-Edit Control and Validation.Source: §C.3 · p. 13
  • Each error event receives exactly one most-specific category; multiple Reminders require distinct risks and next actions supported by visible evidence.A trajectory may contain multiple independent events.Source: §C.3 · p. 13

C.4 Complete Runtime Judge Prompt

  • The deployed API judge receives the task, recent visible trajectory, latest tool result, and previously inserted Reminders.Figures 9–13 present the complete runtime prompt, including categories, gates, caps, and the JSON output contract.Source: §C.4 · p. 13

D Diagnostic Error Annotation

Diagnostic annotation combines deterministic detection of explicit tool and execution failures with GPT-5.4 classification of remaining context-dependent failures. It audits completed trajectories, unlike the online judge, which decides whether to intervene from the current visible prefix.

D Diagnostic Error Annotation

  • Deterministic rules identify failures exposed by explicit tool and execution signals; GPT-5.4 classifies remaining context-dependent failures using the problem and complete trajectory.The annotations use four classes, and each fine-grained category is counted at most once per trajectory.Source: D Diagnostic Error Annotation · p. 14
  • Post-hoc diagnostic annotation audits completed trajectories for aggregate error statistics, while the online judge acts on the current visible prefix to decide whether to intervene.Figure 9–13 captions identify the complete runtime judge prompt across five parts.Source: D Diagnostic Error Annotation · p. 14, Figure 9 · p. 15, Figure 10 · p. 16

Related papers

Loading 2610.21496v1…