PaperlayerView exact revision on arXiv

Source-linked AI summary

Shockingly Simple Self-retrospection Improves Agentic Models Without RL

Jonathan Light, Christopher Zhang Cui, Jeonghye Kim, Roger Creus Castanyer, Emiliano Penaloza, Zhengyan Shi, Alessandro Sordoni, Marc-Alexandre Côté, Xingdi Yuan, Minseon Kim

arXiv:2609.35741v1cs.AIcs.CL

TL;DR

Outcome rewards can show whether an attempt succeeded without identifying which decisions mattered, leaving open whether agents can learn from their own explanations. ROFT trains on self-generated retrospections of attempts and feedback, then tests later behavior. It achieves higher held-out solve rates than GRPO at the reported checkpoints on both benchmarks, with faster early progress.

  • Problem

    Outcome-only success or failure rewards do not identify which assumptions or decisions mattered or when corrections apply, particularly in sparse-reward all-failure groups.

    Source: §1 · p. 2

  • Method

    ROFT trains the model with next-token prediction on self-generated retrospection tokens from task attempts and feedback, without external teachers or reward-based policy updates.

    Source: §1 · p. 2

  • Results

    ROFT reached 49.2% on SWE-bench Verified and 29.0% on SWE-bench Pro after 20 updates, versus GRPO’s 48.0% and 25.3% after 40 updates, and progressed faster early.

    Source: §1 · p. 2, §3.1 · p. 5

  • Takeaways & Limitations

    Self-generated retrospections can be useful training targets, and their content may steer subsequent behavior, including when binary outcomes provide no contrast.

    Source: §1 · p. 3, §6 · p. 10

    The experiments focus on software engineering with Qwen3.5-4B, and the authors call for larger, controlled studies across models and domains.

    Source: §5 · p. 10

Abstract

from arXiv · show

People learn not only by repeating successful actions, but also by recounting and explaining their experiences, revising their understanding to guide future behavior. Can a language-model agent improve its future actions by training only on explanations of its own experience? We investigate this question by studying Retrospection-Only Fine-Tuning (ROFT), a minimal online procedure designed to isolate the effect of explanation-only training on subsequent behavior. The agent attempts a task, observes available feedback, generates a retrospective explanation, and is fine-tuned with a next-token prediction loss on the explanation tokens alone. The procedure uses neither an external teacher nor a reward-based policy update. In software-engineering experiments with Qwen3.5-4B, ROFT is trained on problems with mixed successful and unsuccessful base-model attempts. On held-out SWE-bench Verified and Pro, it reaches 49.2% and 26.8% solve rates after 20 updates without using a verifier, compared with GRPO's 48.0% and 25.3% after 40 updates in the evaluated runs, and makes faster early progress in training time and sampled attempts. It also learns to solve individual tasks on which all 64 sampled base-model attempts failed, showing that learning can begin without any initially successful trajectories. Behavioral analyses find that ROFT indirectly assigns credit to actions, encouraging good actions and discouraging incorrect ones. Moreover, prompting retrospections to emphasize more direct solutions yields shorter subsequent attempts even without an explicit length penalty. Together, these findings show that learning to explain can also improve learning to do, establishing self-generated retrospections as useful training targets and motivating further study of explanation-to-action transfer.

1 INTRODUCTION

The paper asks whether agents can improve future actions by training on explanations of their own experience, addressing what outcome-only rewards leave unspecified about decisions and corrections. It introduces explanation-only training and reports that it can improve subsequent software-engineering behavior.

1 INTRODUCTION

  • Recounting an experience can change an agent’s interpretation of what happened and its understanding of how to act next time.The paper uses this idea to motivate retrospective explanations as a training target alongside actions.Source: §1 · p. 2
  • Outcome-only success or failure rewards do not identify which assumption or decision mattered or when a correction applies, especially for sparse-reward all-failure groups.The authors note that such groups lack within-group binary-reward contrast despite potentially useful evidence in their trajectories.Source: §1 · p. 2
  • Prior reflection methods use retrospections as later-attempt context or to obtain training solutions, while critique fine-tuning transfers teacher critiques offline; self-generated reflection-only training remained unclear.The paper frames its central question as whether training only on an agent’s own retrospective explanations can improve future actions.Source: §1 · p. 2
  • ROFT trains on self-generated retrospections of attempts, rather than directly supervising task-solving actions, using neither an external teacher nor a reward-based policy update.Both successful and unsuccessful attempts are eligible, and benefits must transfer through updated weights.Source: §1 · p. 2
  • In software-engineering experiments, ROFT outperformed GRPO at reported held-out checkpoints, progressed faster early, and learned on tasks where all 64 sampled base-model attempts failed.Behavioral analyses found changes consistent with indirect credit assignment; emphasizing direct solutions in retrospection also produced shorter later attempts without an explicit length penalty.Source: §1 · p. 2
  • The findings suggest retrospection content is a complementary, potentially steerable source of training supervision, including when binary outcomes provide no contrast or verdict.Changing what an agent emphasizes in retrospection can change how it subsequently acts.Source: §1 · p. 3

2 RETROSPECTION-ONLY TRAINING

The paper distinguishes learning to act from learning to explain a completed attempt, then tests whether training on explanations alone transfers to later behavior. ROFT gathers attempts and feedback, fits self-generated retrospections, and evaluates subsequent attempts without retained lessons or an added reflection step.

2 RETROSPECTION-ONLY TRAINING

  • Unsuccessful attempts may still provide experience to interpret, so the method evaluates how training on retrospections affects subsequent task performance.An unsuccessful attempt may be a poor example to imitate while still containing useful experience.Source: §2 · p. 3

2.1 LEARNING TO EXPLAIN, EVALUATED BY DOING

  • Learning to do directly optimizes task-solving outputs, whereas learning to explain trains retrospective accounts; Figure 2 contrasts the two training targets.Retrospections can summarize trajectories, track beliefs, explain decisions or outcomes, offer corrections, or state reusable lessons; the attempt supplies context rather than prediction targets.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 ROFT: A MINIMAL TEST OF EXPLANATION-TO-ACTION TRANSFER

  • ROFT collects task attempts and available feedback, then has the same model generate retrospections that interpret evidence and may state corrections or triggers for future use.Successful and unsuccessful attempts are eligible; available verdicts condition the model’s interpretations but do not become reward weights.Source: §2.2 · pp. 3–4
  • The model is fine-tuned on retained retrospection tokens alone, with attempt records and feedback masked as prediction targets, and later benefits must transfer through updated weights.There is no action-target loss, reward-based policy update, independent semantic-quality filter, stored retrospection at evaluation, or added reflection step.Source: §2.2 · pp. 3–4

3 RESULTS

The experiments test whether retrospection-only training improves held-out performance, can learn from all-failure starts, and changes subsequent behavior. Results span mixed-success training, single-task learning, behavioral analyses, and ablations.

3.1 LEARNING ZONE

  • 49.2% on SWE-bench Verified and 29.0% on SWE-bench Pro for ROFT at 20 updates, versus 48.0% and 25.3% for GRPO at 40 updates.ROFT’s training reward levels off around 20 updates, while GRPO’s Verified solve rate falls from 48.0% at update 40 to 46.0% at update 90.Source: §3.1 · p. 5, Figure 4 · p. 6
  • ROFT completed 40 updates in 4.32 hours with 6,035 solution attempts, versus GRPO’s 8.30 hours and 11,576 attempts.After ten updates and 1.29 hours, ROFT reached 49.0% on SWE-bench Verified, above GRPO’s 48.0% after 40 updates and 8.30 hours.Source: §3.1 · p. 5, Figure 5 · p. 6

3.2 LEARNING BEYOND THE MODEL’S FRONTIER

  • After 40 updates from 0/64 successful base attempts, ROFT reached 1.75% on SymPy and 3.29% on SymbiFlow.The runs trained on individual tasks using only self-generated retrospections; single-problem training did not significantly degrade test-set performance.Source: §3.2 · p. 5, Figure 6 · p. 6

3.3 BEHAVIORAL ANALYSIS

  • Correct turns became more likely under RR than incorrect turns (32.41% versus 21.21%), while GRPO showed similar rates (44.14% versus 45.30%).The separation is consistent with indirect credit assignment without direct action supervision.Source: §3.3 · pp. 6–7, Figure 7 · p. 7
  • Prompting retrospections to seek more direct solutions yielded 11.7% fewer assistant tokens and 13.1% fewer turns; evidence-, correction-, and attention-weighting variants also had the highest observed solve rates.Their solve rates were 54.69%, 54.38%, and 54.22%, respectively, versus 50.63% for Uniform.Source: §3.3 · p. 7, Figure 8 · p. 8
  • After one update, reasoning showed the largest mean forward KL, 3.45× tool calls; read/search contexts shifted 1.35–1.89 times more than write/edit contexts.Prediction changes were also largest in the first normalized assistant-turn decile.Source: §3.3 · p. 7, Figure 8 · p. 8

3.4 ABLATING THE RETROSPECTION LEARNING SIGNAL

  • Online RR scored 49.2% on SWE-bench Verified at 20 updates, versus 47.4% for off-policy reflections and 48.0% for offline RR; four retrospections per rollout led the tested sweep.The advantage over GRPO also held for Qwen3.5-9B: 58.8% versus 55.8% after 20 updates, using 1.93 versus 5.33 training hours. Providing a final correctness verdict did not change the observed score: both variants reached 49.2%.Source: §3.4 · pp. 8–9, Figure 9 · p. 9

4 RELATED WORK

The paper distinguishes retrospection reinforcement from approaches that train task-solving outputs, train on feedback alongside actions, or keep reflections available at inference. Its defining distinction is that retrospections themselves are the training target, with later behavior expected to change through updated weights.

Context internalization

  • Context-internalization methods train actions, often transferring privileged information or teacher guidance into model weights; this procedure instead targets retrospections.The comparison includes STaR, OPCD, RLTF-SD, OPSD, ERL, and SSD; Table 1 summarizes training and conditioning distinctions.Source: §4 · p. 9, Table 1 · p. 10

Learning from textual feedback

  • Feedback-training methods may retain action supervision or use teacher critiques, whereas this work uses the actor’s own outcome-conditioned retrospections without action-target losses.The related-work comparison includes RLTF-FM, Early Experience, Critique Fine-Tuning, and Self-Critique Fine-Tuning.Source: §4 · p. 9

Inference-time reflection

  • Inference-time reflection methods pass feedback into later generations; this procedure carries no reflection forward, so any benefit must transfer through updated actor weights.The paper contrasts this route with Reflexion, Self-Refine, search methods, Retroformer, and RetroAct.Source: §4 · p. 10

5 LIMITATIONS

The experiments are limited in breadth by the cost of training and evaluating long-horizon agents.

5 LIMITATIONS

  • Experiments focus on software engineering with Qwen3.5-4B; a 40-update GRPO run costs $500, and one SWE-bench Pro evaluation costs $200 in GPU costs alone.The authors call for larger, controlled studies across models and domains.Source: §5 · p. 10

6 CONCLUSION

The paper frames retrospection-only training as a distinct route for transferring experience into later behavior and concludes that self-generated explanations can improve action. It distinguishes this route from action-target training and inference-time reflection, while noting limits on validation and deployment.

6 CONCLUSION

  • ROFT isolates online learning from self-authored retrospections, with later behavior changing through updated weights rather than retained text.The conclusion motivates language-based credit assignment for long-horizon agents, where sparse outcomes offer little guidance about intermediate decisions.Source: §6 · p. 10

ETHICS STATEMENT

  • The study evaluates software-engineering agents on existing benchmarks rather than deploying them to end users.Benchmark success does not establish safety, security, or reliability outside evaluated tasks; deployment would require validation, restricted permissions, isolation, and human oversight.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

  • The authors report AI assistance and describe reproducibility materials, while warning that exact reruns may differ under stochastic generation and asynchronous execution.AI-assisted text and claims were checked by the authors; model-generated behavioral labels are not independently audited ground truth. Selected code components are slated for release upon acceptance.Source: AI USE STATEMENT · pp. 10–11, REPRODUCIBILITY STATEMENT · p. 11

A RELATED WORK (EXTENDED)

  • Prior work routes reflection into action targets, explanation prediction, or inference-time context; RR instead predicts the retrospection itself.Figure 10 presents these as combinable routes and depicts action-target distillation as one context-internalization example.Source: A RELATED WORK (EXTENDED) · p. 17, Figure 10 · p. 17

A.1 CONTEXT INTERNALIZATION: LEARNING BEHAVIOR INFORMED BY FEEDBACK

  • Context-internalization methods train task-response behavior, whereas RR trains an account of the completed attempt without requiring a successful replacement trajectory.The comparison also distinguishes RR from approaches combining reflection with reward-based updates or revised-response imitation.Source: §A.1 · p. 18

A.2 LEARNING FROM TEXTUAL FEEDBACK: PREDICTING THE INTERPRETATION

  • Textual-feedback precedents include externally supplied critiques and critique-to-answer transfer, while RR uses online retrospections authored by the evolving actor.The paper says RR does not independently verify explanation correctness: a reliable test verdict can ground a retrospection without validating its causal claims.Source: §A.2 · pp. 18–19

A.3 INFERENCE-TIME REFLECTION: RETAINING INTERPRETATIONS AS CONTEXT

  • Unlike methods that retain reflections or use them to guide later search, RR tests whether their effects transfer through updated parameters without carried-over text.The comparison is framed as a distinction in learning pathway, not a general claim of superiority over other reflection routes.Source: §A.3 · pp. 19–20

B EXPERIMENTAL SETUP

  • The appendix introduces the Qwen3.5-4B online training procedure and directs readers to later appendices for budgets and experiment-specific changes.Source: B EXPERIMENTAL SETUP · p. 20

B.1 MODEL, TASKS, AND AGENT ENVIRONMENT

The setup applies online ROFT to repository tasks with Qwen3.5-4B, generating retrospections from attempts and feedback and training only on retained retrospective targets. It specifies the optimization, compute, checkpointing, and baseline comparisons.

B.1 MODEL, TASKS, AND AGENT ENVIRONMENT

  • Qwen3.5-4B trains on 767 software-engineering tasks in repository containers, with tools for editing files, searching, and running shell commands.Tasks are shuffled and traversed cyclically; asynchronous completion can affect batch composition.Source: §B.1 · p. 20

B.2 RETROSPECTION GENERATION AND SUPERVISION

  • After each attempt, the policy generates four retrospections from the task, trajectory summary, patch, and test feedback, with both successful and failed attempts eligible as targets.The prompt requests evidence-linked decisions, corrections, and application triggers; failed generations and overlength targets are discarded, while accepted target tokens are supervised and prompt tokens masked.Source: §B.2 · pp. 20–21

B.3 ONLINE OPTIMIZATION

  • Each nominal update uses 64 source attempts and up to 256 retrospection targets, with one pass over retained targets and no replay of earlier batches.All model parameters are optimized with AdamW at a constant learning rate of 10^-6; no reference-policy KL penalty or entropy bonus is used.Source: §B.3 · p. 21

B.4 COMPUTE AND CHECKPOINTING

  • Training uses eight NVIDIA B200 GPUs, and checkpoints are saved every five successful updates and at the final update.Held-out evaluations run separately, without stored retrospections or an added reflection step in the solver context.Source: §B.4 · p. 22

C BASELINES

  • The baselines span reinforcement learning, teacher-generated critique distillation, context internalization, and self-critique-based revision.The GRPO baseline is an enhanced variant previously shown to improve performance.Source: C BASELINES · p. 22

C.1 HELD-OUT PERFORMANCE

On SWE-bench Verified, ROFT records the highest reported solve rate among the listed methods. The section compares it with prior-method baselines and describes how their training targets differ from ROFT’s retrospection-only objective.

C.1 HELD-OUT PERFORMANCE

  • The authors hypothesize that off-policy supervision contributes to ERL and CFT’s lower solve rates relative to ROFT.Their interpretation points to privileged-context mismatch in ERL and a separate critique teacher in CFT; frozen-base retrospection generation also yields lower held-out performance than using the evolving learner.Source: §C.1 · p. 22
  • 49.2%: ROFT’s SWE-bench Verified solve rate exceeds GRPO’s 48.0% and the other listed methods’ reported rates.The figure reports Verified solve rates for Qwen3.5-4B; SCFT’s successful-revision supervision does not necessarily improve first-attempt performance as much as retrospections of the initial attempt.Source: §C.1 · pp. 22–23, Figure 11 · p. 23

C.2 GRPO

  • GRPO directly updates task-solving tokens using group-relative rewards, unlike ROFT’s subsequent retrospection targets.Source: §C.2 · p. 23

C.3 ERL

  • ERL rewards reflection-guided retries and revised attempts when the retry succeeds, directly supervising revised task behavior as well as reflections.Unlike ROFT, it requires a successful revision for training loss.Source: §C.3 · p. 24

C.4 CFT

  • CFT trains a student to generate critiques supplied by a separate teacher, with conditioning inputs masked from the loss.Source: §C.4 · p. 24

C.5 SCFT

  • SCFT retains critique-and-revision continuations only when the final solution passes the task’s correctness criterion.Training supervises the critique and revision, not just a post-attempt account.Source: §C.5 · pp. 24–25

C.6 SSD

  • SSD imitates its own sampled task-solving trajectories, including unsuccessful behavior, rather than learning to explain them.Source: §C.6 · p. 25

D RETROSPECTION PROMPTS AND EXAMPLES

This appendix illustrates how retrospections turn failed and successful coding attempts into concise, evidence-based lessons. Its examples show diagnosing incomplete fixes, tracing relevant code paths, and identifying when a lesson should apply.

D RETROSPECTION PROMPTS AND EXAMPLES

D.1 FAILED ATTEMPT: DATABASE-SESSION PARAMETER COLLISION

  • The retrospection identifies that the fix covered _save but missed related entry points, recommending checks of get and create when internal parameters can collide with model fields.It treats the failing get_or_create test as evidence that the fix was incomplete.Source: §D.1 · pp. 25, 33
  • The system prompt asks for a short reflection naming a consequential decision, evaluating evidence, stating a correction if needed, and identifying triggers for applying the lesson.It also cautions that a passing verdict does not prove every decision correct, nor a failing verdict every decision wrong.Source: §D.1 · p. 25
  • The collision arose because a model’s session field arrived in kwargs alongside the internal session argument passed to _save.The agent renamed the internal parameter to _session, but the resulting patch did not cover every affected path.Source: §D.1 · pp. 28, 33
  • After renaming the parameter, 1 test failed and 22 passed, including the get_or_create name-conflict test.The failed test indicates that the attempted fix did not resolve the collision across all paths.Source: §D.1 · pp. 32–33
  • The database-session example identifies an incomplete fix: renaming `_save`’s parameter missed the same collision in `_get_or_create`, as the failing test indicated.The reflection recommends renaming both parameters and flags similar collisions with common model-column names as a trigger for the lesson.Source: §D.1 · p. 33

D.2 SUCCESSFUL ATTEMPT: CONTEXT-MANAGER CLEANUP

  • The successful-attempt example’s stated lesson is that cleanup must also run after an exception, with failures in subsequent tests serving as a signal of leaked state.The reported bug is that `verify()` is skipped when the first test raises, leaving interactions uncleared for the next test.Source: §D.2 · pp. 33–34
  • The reflection prompt asks for a short paragraph naming a consequential decision, weighing evidence, correcting a mistaken assumption, and identifying triggers for applying the lesson.It also says not to restate the task or label the response as a reflection or review.Source: §D.2 · p. 34
  • The prompt cautions that a passing verdict does not validate every decision and a failing verdict does not invalidate every decision.When evidence is insufficient, it instructs the agent to state uncertainty and say what to try next time to learn more.Source: §D.2 · p. 34
  • The context-manager example concerns tests that share Pact interactions: when one test raises an exception, interactions persist and can make a subsequent test fail.The reported behavior is a test-order dependency: the next test sees interactions from the earlier test.Source: §D.2 · p. 34
  • The diagnosis attributes the leak to exceptional exits bypassing `verify()`, the usual cleanup path that clears the interactions.The example identifies cleanup on the exception path as necessary to prevent state leaking into later tests.Source: §D.2 · pp. 37, 39
  • The issue was scoped to the implementation with an `_interactions` list; the V3 module uses a different FFI-based approach and was not affected in the same way.The attempt notes that V3 does not use the same interaction list.Source: §D.2 · p. 44
  • When an exception made `__exit__` return without calling `verify()`, stale interactions remained and could affect later tests using the same Pact object.The explanation traces how interactions from one test could persist into the next and cause verification to fail.Source: §D.2 · pp. 41, 45
  • The reproduction case passed, and the existing Pact and consumer tests passed after the change.The attempt reports all 70 tests passing.Source: §D.2 · pp. 44–45
  • The fix clears `self._interactions` in `__exit__` when an exception occurs, before returning early.The retrospective describes this as a minimal one-line change that avoids relying on `verify()` for cleanup.Source: §D.2 · pp. 46, 47, 49
  • The retrospective connects the skipped `verify()` call to the missing cleanup, using `verify()`'s list-clearing behavior as evidence for the fix.It identifies the applicable trigger as an exception in a Pact context manager followed by failures from residual state in later tests.Source: §D.2 · p. 49
  • The reported lesson applies when an exception in a Pact context manager leaves residual interaction state that causes later tests using the same object to fail.The retrospective frames clearing state in `__exit__` as necessary even when `verify()` is skipped.Source: §D.2 · p. 49

E.1 EXPERIMENTAL PROTOCOL

The protocol compares ROFT and GRPO from Qwen3.5-4B and reports their training costs and curve-measurement conventions. ROFT uses fewer attempts and less elapsed time at 40 updates.

E.1 EXPERIMENTAL PROTOCOL

  • ROFT uses 64 solution attempts per update and up to four retrospections per attempt; GRPO uses 32 accepted groups of eight attempts.GRPO rejects groups with identical rewards because they provide no relative advantage signal.Source: §E.1 · p. 50
  • At 40 updates, ROFT takes 4.32 hours and 6,035 attempts, versus GRPO’s 8.30 hours and 11,576 attempts.Attempt counts include completed solution attempts rejected before optimization; retrospection generations count toward elapsed time, not attempts.Source: §E.1 · p. 50

E.2 TRAINING CURVES

  • Fig. 5 curves report training reward before group rejection, not held-out task success, and are smoothed over a preceding hour, five updates, or 1,024 attempts.Time averages weight observations by duration, while update and sample averages weight them by their numbers of attempts.Source: §E.2 · p. 50
  • Fig. 4a extends GRPO to 100 updates while ROFT ends at 40, with one-hour smoothing throughout.Each panel ends at the smaller of the two methods’ resource budgets.Source: §E.2 · p. 50

E.3 HELD-OUT PERFORMANCE

ROFT outperforms GRPO on the reported held-out benchmarks at the compared checkpoints, while training reward and held-out performance do not move together throughout training.

E.3 HELD-OUT PERFORMANCE

  • ROFT reaches 49.0% Verified solve rate after ten updates and 1.29 hours, versus GRPO’s 48.0% after 40 updates and 8.30 hours.Source: §E.3 · p. 50
  • At update 20, ROFT solves 246/500 Verified and 212/731 Pro tasks, versus GRPO’s 240/500 and 185/731 (29.0% versus 25.3% on Pro).The checkpoints compared are ROFT update 20 and GRPO update 40.Source: §E.3 · p. 50
  • ROFT’s training reward peaks at 0.563 near update 18 and falls to 0.483 by update 40, while GRPO’s Verified solve rate falls from 48.0% at update 40 to 46.0% at update 90.During ROFT training, SFT loss and retrospection entropy also decrease.Source: §E.3 · p. 50

F.1 SINGLE-TASK TRAINING AND MEASUREMENT

Single-task runs start from Qwen3.5-4B and train on self-generated retrospections; their reported learning curves count observed solution successes and use attempt-based smoothing.

F.1 SINGLE-TASK TRAINING AND MEASUREMENT

  • Each single-task run starts from Qwen3.5-4B and uses 64 solution attempts and up to four self-generated retrospections per update, training only retrospection tokens at 10−6 learning rate.Tasks include SymPy 20916, Django 15554, and Pylint 6386 from SWE-bench Verified, plus SymbiFlow 17 from SWE-rebench.Source: §F.1 · p. 50
  • Initial batches contain 0/64 successes for SymPy, SymbiFlow, and Django, and 2/64 for Pylint.Source: §F.1 · p. 50
  • Learning curves count successful attempts before reward shaping or batch filtering, retaining execution failures assigned zero reward; smoothing pools recent intervals containing at least 1,024 attempts when available.Each rate is total successes divided by total attempts in the window; full windows contain 1,024–1,237 attempts.Source: §F.1 · pp. 50–51

F.2 ADDITIONAL SINGLE-TASK LEARNING RESULTS

Additional single-task runs show different solve-rate trajectories across Django, Pylint, and SymbiFlow, with each reported rate drawn from an extended run.

F.2 ADDITIONAL SINGLE-TASK LEARNING RESULTS

  • Django peaks at 1.15% and ends at 0.73% after 40 updates; resumed Pylint peaks at 17.86% and ends at 15.86% after 100 updates.Figure 12 covers Django through 40 updates and Pylint through 100; the latter resumes from its update-40 checkpoint.Source: Figure 12 · p. 51, §F.2 · p. 51
  • SymbiFlow’s smoothed solve rate reaches 4.49% at update 60, its highest value within that interval.Figure 13 extends the same run’s trajectory to 60 updates.Source: Figure 13 · p. 51, §F.2 · p. 51

F.3 TEST-SET PERFORMANCE AFTER SINGLE-TASK TRAINING

The analyses show that retrospection-only training shifts likelihood toward correct turns more selectively than GRPO, while efficiency-focused retrospections produce shorter coding rollouts. Single-task evaluation uses all 500 SWE-bench Verified tasks, including training instances for three runs.

F.3 TEST-SET PERFORMANCE AFTER SINGLE-TASK TRAINING

  • The four single-task runs are evaluated on all 500 SWE-bench Verified tasks after 40 updates, with execution failures counted as unresolved.The SymPy, Django, and Pylint training instances remain in the evaluation set.Source: §F.3 · pp. 51–52, Figure 14 · p. 52

G.1 LIKELIHOOD CHANGES BY TURN CORRECTNESS

  • At update 10, RR’s correct-minus-incorrect increased-likelihood gap is 11.20 percentage points versus GRPO’s −1.17.At update 40, the gaps are 10.08 percentage points for RR and 1.83 for GRPO; most turns in each correctness category decrease in likelihood under both methods at that checkpoint.Source: Figure 15 · p. 53, §G.1 · p. 53

G.2 ROLLOUT LENGTHS WITH EFFICIENCY-FOCUSED RETROSPECTIONS

  • 11.7% fewer tokens and 13.1% fewer turns: the faster-retrospection variant shortens rollouts across all attempts versus the baseline.The reductions also hold among solved attempts (12.3% fewer tokens; 12.9% fewer turns) and unsolved attempts (10.3% fewer tokens; 13.0% fewer turns); solve rates in this window are 61.6% for baseline and 58.4% for the faster variant.Source: Table 4 · p. 54, §G.2 · p. 54

G.3 FINE-GRAINED TRAJECTORY AND RETROSPECTION-TOKEN ANALYSES

The analyses measure how one retrospection-only update changes action-token distributions and test whether emphasizing different retrospective content changes solving performance. Token-level examples locate especially large distributional changes in particular coding-rollout tokens.

G.3 FINE-GRAINED TRAJECTORY AND RETROSPECTION-TOKEN ANALYSES

  • A fixed batch of 64 original trajectories supplies the retrospection-only update, and the resulting checkpoints are compared on the same 64 problems with ten repetitions.The evaluation comprises 640 attempts per model; saved retrospections are not included in solver context.Source: §G.3 · pp. 55, 57
  • The task-balanced mean forward KL is 3.27893 × 10−4 nats; reasoning has the largest reported role mean, 5.71843 × 10−4 nats.The first progress decile averages 4.12157 × 10−4 nats, compared with 2.99964 × 10−4 in the last decile.Source: §G.3 · p. 56, Figure 16 · p. 56
  • The one-update comparisons reweight retrospection targets by semantic category or by base-model attention to the recorded trajectory.Evidence, correction, and lesson tokens are separately identified for category-based weighting; attention-up assigns continuous weights to nonstructural target tokens.Source: §G.3 · pp. 56–57
  • Attention-up solves 350/640 problems (54.69%), versus 324/640 (50.63%) for Uniform, a +4.06 percentage-point change.Correction-up and evidence-up yield +3.75 and +3.59 percentage points relative to Uniform.Source: Table 5 · p. 57, §G.3 · p. 57

G.4 TOKEN-LEVEL EXAMPLES

  • The three highest token-level peaks occur in a Sphinx diagnostic print statement, malformed Black tool markup, and a MechanicalSoup Bash argument.The Sphinx peak is 0.77878670 nats; the selected Black and MechanicalSoup peaks are 0.19109935 and 0.10787710 nats, respectively.Source: §G.4 · pp. 57–58, Figure 17 · p. 58

H ABLATIONS

The ablations compare verdict-conditioned and verdict-free retrospection, alternative reflection-generation arrangements, different numbers of retrospections per rollout, and scaling to a larger model. Across these settings, the reported outcomes vary by condition, while the 9B ROFT run exceeds GRPO’s solve rate at the reported checkpoint.

H ABLATIONS

  • The Qwen3.5-4B ablations use the same base checkpoint and training dataset, with loss applied only to retrospection tokens.Each update uses 256 nominal retrospection targets and a constant learning rate of 10−6.Source: H ABLATIONS · p. 59

H.1 RETROSPECTION WITH AND WITHOUT A VERDICT

  • Both verdict-conditioned and no-verdict retrospection reach 49.2% on SWE-bench Verified after 20 updates, resolving 246/500 tasks each.The no-verdict retrospections use the final patch and chronological trajectory summary without the separate task description or final grader feedback.Source: Figure 18 · p. 59, §H.1 · p. 59

H.2 ONLINE, OFF-POLICY REFLECTIONS, AND OFFLINE RR

  • Online RR solves 246/500 tasks (49.2%), compared with 237/500 (47.4%) for off-policy reflections and 240/500 (48.0%) for offline RR.Online RR uses the evolving learner to generate attempts and retrospections; off-policy reflections use a frozen base-model reflector, while offline RR trains on a fixed corpus.Source: §H.2 · pp. 59–60

H.3 NUMBER OF RETROSPECTIONS PER ROLLOUT

  • After ten updates, the 2-, 4-, and 8-retrospection settings solve 239, 245, and 231 problems, respectively.The four-retrospection control has the highest observed score, and the three scores span 2.8 percentage points.Source: §H.3 · p. 60

H.4 SCALING TO QWEN3.5-9B

  • At 20 updates, ROFT solves 294/500 SWE-bench Verified tasks (58.8%), versus GRPO’s 279/500 (55.8%).ROFT uses 1.93 hours and 1,727 completed solution attempts, compared with GRPO’s 5.33 hours and 7,152.Source: §H.4 · p. 60

Related papers

Loading 2609.35741v1…