PaperlayerView exact revision on arXiv

Source-linked AI summary

RLTL;DR: Self-improvement by Internalizing Self-generated Feedback

Michael Kirchhof, Eleonora Gualdoni, Andrew Szot, Khashayar Gatmiry, Aryo Lotfi, Abbas Kazerouni, Omar Attia, Sanjoy Chowdhury, Alexander Toshev

arXiv:2609.37633v1cs.LGcs.AIcs.CLstat.ML

TL;DR

On very difficult self-improvement tasks, RLVR may yield no successful rollouts, and stronger teachers or example solutions may be unavailable. RLTL;DR uses sequential self-generated insights and trains the policy to predict them from the task. It reaches 12–13% Pass@1 without insights or sequential attempts at evaluation, where GRPO faces the learning barrier.

  • Problem

    On difficult self-improvement tasks, policies may have no successful rollouts, while stronger teacher models and example solutions are unavailable.

    Source: §1 · p. 1

  • Method

    RLTL;DR conditions sequential retries on insights generated from failed attempts and trains the policy to predict those insights from the task alone.

    Source: §3.2 · p. 4, §3.3 · p. 4

  • Results

    12–13% Pass@1 without insights or sequential attempts at evaluation breaks the learning barrier GRPO faces on Pass@128=0 tasks.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The paper demonstrates that compacted insights can be backpropagated into a policy as an auxiliary self-improvement objective or as a training signal without full rollouts.

    Source: §8 · p. 11

    The authors expect task-to-insight training to be less useful for highly specific tasks that share few common rules, such as proofs requiring a specific trick.

    Source: §7.2 · p. 10

Abstract

from arXiv · show

The common paradigm of reinforcement learning with verifiable rewards (RLVR) is to let agents make multiple attempts at a task, and optimize towards the successful ones. This becomes problematic in the realms of self-improvement, where tasks are so difficult that the agent has a low or even no chance of success, and where there are no teacher models or example solutions to distill from. In this paper, we introduce RLTL;DR. After each failed attempt, we show the policy the verifier outputs and let it write its own feedback, in the form of a single TL;DR insight. The next rollout is conditioned on all previous insights, and we sequentially sample rollouts until a solution is found. Moreover, we enable backpropagation on the in-context insights to internalize a direct task to insight mapping. On challenging tool-calling and coding datasets (filtered to Pass@128=0), standard GRPO training of a Qwen 3.5 9B Thinking policy stays flat at a Pass@1 of 0% to 1%. RLTL;DR breaks through this learning barrier, achieving a Pass@1 of 14-31% with insights in context during training and, crucially, 12-13% when no insight is in context at eval time. We identify that the key is the task to insight internalization. To study this further, we reduce our approach to SFTL;DR, training only on (task, insight) tuples, without showing or backpropagating on any rollouts. Training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts. This demonstrates a promising compacted training paradigm of the form "on this sort of task, keep this sort of thing in mind", which we hope to inspire future research on.

1 Introduction

On tasks with no successful Qwen 3.5 9B Thinking attempts in 128 tries, RLTL;DR combines sequential self-generated insights with training that internalizes them. It reaches 11–14% Pass@1 without insights or sequential sampling at evaluation, while GRPO remains at 0% to 1%.

1 Introduction

  • Standard RLVR rewards successful rollouts and penalizes unsuccessful ones, but difficult tasks with no successful attempts provide no useful learning signal.In self-improvement settings, the agent may also lack a stronger teacher or example solution.Source: §1 · p. 1
  • Sequential rollouts let the policy use verifier outcomes from failed attempts to generate TL;DR insights that guide later attempts, finding solutions for 14–59% of tasks at Pass@k.Without insight internalization, this exploration gain stays close to the policy’s original performance on insight-free first attempts.Source: §1 · p. 1
  • Backpropagating on the in-context insight tokens trains a task-to-insight mapping, so the policy can internalize learnings without predicting insights at test time.The policy receives an SFT loss on the insights alongside the GRPO loss on rollouts.Source: §1 · p. 2, Figure 1 · p. 2
  • 11–14% Pass@1 without insights or sequential sampling at evaluation, versus GRPO’s 0% to 1%, breaks the learning barrier on tasks with no success in 128 attempts.The authors identify backpropagation on the task-to-insight mapping as the key ingredient.Source: §1 · p. 2

2 Related Works

Prior work addresses RLVR’s learning barrier and explores ways to guide attempts with privileged information or transfer context into model weights. RLTL;DR instead uses short self-generated insights and trains the policy to predict them from the task.

2 Related Works

  • When all rollouts receive the same reward, group-relative advantages can vanish; methods for high-success settings include entropy control, Pass@k objectives, and difficulty-matched curricula.The paper targets frontier splits where success rates are close to zero.Source: §2 · p. 2
  • Guided-exploration methods vary in guidance source, explicitness, and transfer mechanism, while RLTL;DR uses failed unit tests as its only guidance and backpropagates short self-generated insights.Prior sources include gold solutions, teacher models, and failed attempts; transfer ranges from none to mixing guided and unguided rollouts or importance-sampling corrections.Source: §2 · pp. 2–3
  • Unlike feedback-distillation methods that train on a full rollout, RLTL;DR predicts about 17 insight tokens, relying on short insights to transfer to full-rollout behavior.The paper relates this to context distillation and findings that models can apply knowledge in a different format from the one used in training.Source: §2 · p. 3

3 Methods

The method first uses GRPO for agent trajectories, then modifies sampling to retry hard tasks with insights generated from failures. It also trains the policy to predict those insights from the task alone, internalizing them for insight-free evaluation.

3.1 RL and GRPO preliminaries

  • In the agent setup, a task goal and chat history condition actions; environment observations and a verifier’s final binary reward complete each trajectory used to optimize expected reward.Episodes end when the agent signals completion or reaches a 50-action horizon.Source: §3.1 · p. 3

3.2 Sequential self-generated insights

  • After failed attempts, the policy generates a short natural-language insight from its trajectory and environment feedback, then retries the same task conditioned on accumulated insights.Insights are inserted when at most 50% of prior attempts succeeded, and the successive attempts form one GRPO group.Source: §3.2 · p. 4
  • At K = 8, asynchronous sequential sampling makes the sampling phase 4.5× slower and overall walltime 1.5× higher than parallel sampling.The paper keeps data per update equal between GRPO and RLTL;DR for benchmarking fairness.Source: §3.2 · p. 4

3.3 Insight internalization

  • Sequential sampling improves exploration, but gains with insights in context do not directly transfer to insight-free evaluation, motivating a separate internalization objective.At evaluation, the agent must solve the task in a single attempt.Source: §3.3 · p. 4
  • The SFT objective predicts previously generated insights from the task description alone and is added to the GRPO loss, training a direct task-to-insight mapping.The insight tokens already appear in the rollout context, so the loss is implemented by changing the backpropagation mask without increasing runtime.Source: §3.3 · pp. 4–5
  • The internalization loss targets task learnings, not self-critique from a rollout; the method does not generate insights from this task-only prediction at training or evaluation.Each inserted insight averages 17 tokens.Source: §3.3 · p. 5

4 RLTL;DR breaks through the learning barrier

On tasks where Qwen 3.5 9B had no successes in 128 attempts, the paper tests whether sequential feedback and internalized insights can restore learning. RLTL;DR improves Pass@1 without insights at evaluation, though the reported generalization does not extend to Leetcode.

4.1 Experiment setup

  • The evaluation filters tasks to Pass@128=0 for Qwen 3.5 9B; the held-out sets are unfiltered and include Appworld test-challenge, unseen SAPI apps, and unseen Leetcode problems.SAPI, Appworld, and Leetcode contribute 458, 34, and 123 filtered tasks, respectively.Source: §4.1 · pp. 5–6

4.2 Results

  • 12–13% Pass@1: RLTL;DR improves on very hard tasks without insights or sequential sampling at evaluation, while Leetcode shows no generalization.With insights in context, train-time Pass@1 is 14–31%; RLTL;DR and RLTF-SD both learn during training, but the paper does not claim either outperforms the other.Source: Figure 2 · p. 6, §4.2 · pp. 6–7
  • GRPO remains at 0%–1% Pass@1 on these difficult tasks, while SGE behaves similarly; RLTF-SD also learns, but neither it nor RLTL;DR is claimed to outperform the other.The paper attributes GRPO's flat performance to a lack of successful rollouts and learning signal; SGE does not reflect on failed unit tests.Source: §4.2 · p. 6
  • On unfiltered datasets, where at most 3–5% of tasks are frontier-difficult, GRPO learns and RLTL;DR neither helps nor hurts performance except on Leetcode.The authors position RLTL;DR as a method for training on challenging tasks.Source: §4.2 · pp. 6–7

5 Reducing to the secret sauce: From RLTL;DR to SFTL;DR

The section isolates insight-token supervision as the key component of RLTL;DR and tests whether rollouts are needed for training. On the SAPI frontier split, training on task–insight pairs alone approaches the performance of RLTL;DR and full-rollout SFT, with much lower update compute.

5.1 Experiment setup

  • The SAPI frontier split uses 458 frontier-difficult and 184 very difficult tasks; it is divided into 256 evaluation and 386 training tasks.Qwen 3.5 9B achieves 4% Pass@1 on this split; GRPO fails to learn, while RLTL;DR reaches 21.5% Pass@1.Source: §5.1 · p. 7
  • SFTL;DR backpropagates only on insight tokens, skipping rollout forward computation and gradient updates; its standard setup can train repeatedly on insights across rollouts.The deduplicated variant instead trains one-by-one on unique task–insight tuples.Source: §5.1 · p. 7

5.2 Reducing RLTL;DR to just SFTL;DR

  • Reducing the insight-loss weight from λ = 0.5 to 0.01 or 0 severely hurts train and evaluation performance; insight-token SFT alone recovers almost all RLTL;DR performance.The additional GRPO loss helps runs with non-zero GRPO loss converge faster, so the authors recommend leaving it active.Source: §5.2 · p. 7
  • Classical SFT on successful RLTL;DR rollouts is slightly above RLTL;DR performance, within standard deviation, and reducing its training rollouts from 3.5k to 100 loses little performance.The table reports results with ±0.6% standard deviation.Source: §5.2 · pp. 7–8, Table 2 · p. 8
  • Training on unique task–insight pairs alone, without rollouts, almost reaches RLTL;DR and full-rollout SFT performance.SFTL;DR-deduplicated trains on 4k unique tuples, each pairing a task with an insight.Source: §5.1 · p. 7, §5.2 · p. 8
  • The 4k-tuple setup uses 4.6M forward and 68k backward update tokens, versus 9.5M and 217k for SFT on 100 rollouts and 400M and 6.6M for SFT on 3.5k rollouts.These counts cover policy updates only; all approaches still require about 1B tokens to collect the rollouts used to generate insights.Source: §5.2 · p. 8

6 What makes for good insights?

The paper finds that short, broad TL;DR insights are more effective for unaided performance than more detailed diagnostics, summaries, or code corrections. Insight accuracy depends strongly on access to failed unit tests, while the choice of generator matters less than the information it receives.

Insight content

  • The default insight is a short procedural sentence averaging 17 words; the design tests fuller diagnostics, attempt summaries, code corrections, and different context limits.The maximum number of insights in context varies from 1, 2, 4, and 8, instead of the default 16.Source: §6 · pp. 8–9
  • 21.4 to 20.3 train Pass@1: replacing TL;DR with a diagnostic paragraph lowers performance, with summary and corrected-code variants reaching 13.8 and 15.6.With insights in context, summary plus diagnostic paragraph gives +41.1% over unaided rollouts, versus +36.0% for TL;DR; these results suggest detailed hints help in-context solving more than internalization or transfer.Source: §6 · p. 9, Table 3 · p. 9

Insight generator

  • With 16 insights, GLM 5.2 with thinking reaches 24.5 versus 21.4 for the student; with one insight, the gap is about five points, 26.2 versus 21.4.The passage says thinking makes little difference, but its comparison is incomplete.Source: §6 · p. 9
  • Removing access to failed unit tests reduces Pass@1 to 3.4, while changing the insight generator has smaller, sometimes noisy effects.The authors conclude that correctly identifying the failure matters most; generators can be on-policy, off-policy, or potentially human-provided, if they have useful information.Source: §6 · p. 9
  • Insights should use privileged information that makes them accurate while remaining broad enough to generalize across tasks.The paper presents this as the design takeaway from its insight-format and generator comparisons.Source: §6 · p. 9

7 Discussion

The discussion argues that task-conditioned insight training can internalize feedback and transfer it beyond the insight format, while the generalization mechanism and limits remain uncertain. It considers insight quality and contrasts training insights into a model with retrieval-based insight databases.

7.1 The surprising learning only from (task, insight) tuples

  • Task-conditioned insight training may transfer feedback into rollout generation; the authors attribute this to backpropagation smoothness, but say the generalization dynamics remain unknown.They hypothesize that semantic similarity in pretrained models updates parameters for similar tasks and output formats.Source: §7.1 · p. 10
  • Related work reports analogous internalization or cross-format transfer from backpropagating terminal outputs, documentation, or think traces.The paper connects these findings to its task-to-insight training setup.Source: §7.1 · p. 10

7.2 Limits of (task, insight) learning

  • Compacted task-insight training may not generalize when tasks require highly specific tricks or share few rules; the authors also hypothesize it requires sufficient pretraining.They suspect the specific-solution issue may apply to their Leetcode results.Source: §7.2 · p. 10

7.3 Source and quality of insight

  • Insight quality matters: the student’s verifier-informed insights help, while removing verifier outcomes makes them too low-quality; larger teacher models can improve performance further.The authors consider the insight source a pragmatic choice if insights are useful for the next rollout and generic enough to transfer.Source: §7.3 · pp. 10–11

7.4 Beyond insight databases

  • For untrainable frontier models, the authors favor retrieval-based insight databases; when training is possible, they argue insight training may avoid a dedicated retrieval system, pending benchmarking.They attribute this potential simplification to backpropagation generalizing insights to similar tasks.Source: §7.4 · p. 11

8 Conclusion and outlook

The conclusion presents task-conditioned backpropagation on compact insights as a way to internalize and generalize feedback, including as a training signal without full rollout backpropagation. It outlines open questions and implementation, evaluation, training, and task-specific details.

8 Conclusion and outlook

  • The paper presents compact, task-conditioned insight backpropagation as a way to internalize and generalize feedback, including as a training signal without full rollout backpropagation.The authors identify knowledge generalization and the limits of insight-only learning as open questions; tool-use experiments used research-only simulated environments, not deployed products or production systems.Source: §8 · p. 11

A.1 Insight generation

  • After failed rollouts, a separate judge prompt receives the task, actions, observations, and verifier outputs; only the final one-sentence insight is used for the next rollout.The longer summary and feedback did not outperform the one-sentence insight and reduced performance, according to the authors.Source: §A.1 · p. 15

A.2 Insight conditioning

  • Insights from prior failures in the current batch are inserted as user messages after the task when the batch success rate is ≤50%; earlier-update insights and full prior chats are excluded.The paper says including the full chat introduced contextual drag.Source: §A.2 · p. 16

A.3 Loss details

  • The authors accumulated changes to the loss and architecture to improve the GRPO baseline on Appworld-train, then applied them across methods for fairness.They note this may give the baseline a slight advantage because the changes were tuned toward it.Source: §A.3 · pp. 16–17

A.3.1 Changes taken over from DAPO

  • The training setup omits KL regularization, averages policy losses at token level, and removes standard-deviation scaling from the GRPO advantage.The authors report no degeneration without KL divergence and say removing standard-deviation scaling improved results; they link the latter to avoiding destabilizing gradients on very hard tasks.Source: §A.3.1 · p. 17

A.3.2 Positive-ratio filtering

  • Positive-ratio filtering retains rollouts so 75% have positive rewards, addressing entropy explosion and policy collapse in the GRPO baseline.The authors say it is less critical for RLTL;DR and SGE because insight conditioning and SFT loss already produce many positive-advantage rollouts.Source: §A.3.2 · p. 17

A.4 Train hyperparameters

  • Leetcode and Appworld use different hyperparameters; Leetcode deactivates thinking because hard problems exceeded its token budget, and asynchronous collection raises average attempts above the minimum.Experiments ran on 8×B200 GPU nodes and took 1–4 days each.Source: §A.4 · p. 17, Table 4 · p. 18

A.5 Fixing Qwen’s tendency to overthink on hard Leetcode problems

  • On very hard Leetcode tasks, Qwen 3.5 9B produced overly long solutions without code blocks; after 100k GRPO steps, the checkpoint rarely exhausted 16k tokens and was used to start Leetcode experiments.The paper’s 123 frontier-difficult tasks still had Pass@128=0 for that checkpoint.Source: §A.5 · pp. 17–18

B Reproducing the RLTL;DR Implementation

Reproducing RLTL;DR requires generating insights in a separate judge context, conditioning later rollouts on them, and backpropagating on insight tokens. The authors recommend checking insight effectiveness and deconfounded success rates during implementation and evaluation.

B Reproducing the RLTL;DR Implementation

  • After each rollout, call the agent again in a new chat with a judge prompt, then extract the TL;DR insight from its JSON output.The implementation uses up to 4096 think and 4096 action tokens for this generation.Source: B Reproducing the RLTL;DR Implementation · p. 18
  • Insert insights as user messages after the task when the current batch’s average success rate is ≤50%.The authors recommend logging the literal context during debugging because context construction was a frequent error source.Source: B Reproducing the RLTL;DR Implementation · p. 18
  • Flip the updater’s masks on insight tokens to backpropagate on them; when multiple insights occur, flip all their masks simultaneously.The authors say this simplifies implementation by avoiding new contexts and forward computations.Source: B Reproducing the RLTL;DR Implementation · p. 18
  • Before and after training, check that insights help in the domain, insight advantage is > 0 during training, and deconfounded success without insights is reported after training.The authors suggest plotting insight effectiveness before training.Source: B Reproducing the RLTL;DR Implementation · p. 18

C.1 Example TL;DR insights

The section presents recurring insights as task-vocabulary corrective actions and contrasts them with more detailed feedback that may contain episode-specific information. It also flags evaluation confounders as a concern for insight-conditioned RL metrics.

C.1 Example TL;DR insights

  • Recurring TL;DR insights each name a single corrective action in the task’s own vocabulary.Table 5 groups insights pointing to the same correction by lexical overlap.Source: §C.1 · p. 19, Table 5 · p. 19

C.2 Examples including previous autoregressive fragments

  • The feedback variants escalate from a transferable hint to verifier diagnostics and corrected code, whose identifiers or credentials may be episode-bound and unusable in the next episode.Table 6 presents these as building blocks: the diagnostic paragraph replaces the insight, while corrected code is appended to the paragraph.Source: §C.2 · pp. 19–20, Table 6 · p. 20

D How to evaluate without confounders

D.1 Deconfounded RL train and eval metrics

The appendix deconfounds training and evaluation metrics by accounting for rollout throughput and insight conditioning, and defines measures for insight quality. It also reports checkpoint-update comparisons and finds that revisited tasks improve without insights in context.

D.1 Deconfounded RL train and eval metrics

  • Training metrics use macro-averaged per-task success, report both constant Pass@8 and variable Pass@k, and compare runs against environment steps rather than updates alone.The authors track tokens and walltime as secondary metrics and keep group-size distributions comparable; evaluation also separates rollouts with and without insights.Source: §D.1 · p. 21

D.2 Gauging insight strength

  • Insight quality is gauged by with-versus-without insight advantage, insight Pass@k, too-easy and too-hard ratios, and generalization across rolling held-out tasks.The too-easy ratio matters less when training with insight SFT, while the rolling train curve is an indirect measure of generalization.Source: §D.2 · p. 22

E.1 Distance to original policy

  • SFT updates are two orders of magnitude larger and more spread out than RL updates, likely because RL's bf16 precision drops small updates; compare only within RL or within SFT runs.The authors report that LSFT-only has update magnitude similar to RLTL;DR despite backpropagating on 839k tokens versus 12M, and SFTL;DR has smaller, more concentrated updates than full-rollout SFT.Source: §E.1 · pp. 22–23, Table 7 · p. 23

E.2 Performance on revisited tasks

  • 0.138 → 0.253 without insights (n = 85, p = 0.001): revisited tasks are solved nearly twice as often, with gains comparable to the +12.4pp on tasks not revisited.The passage concludes that the gain does not appear confined to insight-conditioned rollouts.Source: §E.2 · p. 23, Figure 3 · p. 24

E.3 GRPO loss is not necessary but speeds up training

The appendix finds that hint-SFT alone can eventually approach RLTL;DR, while a GRPO component accelerates learning. It also examines how insights change over training, how much successful trajectories rely on them, and performance on a less difficult task split.

E.3 GRPO loss is not necessary but speeds up training

  • Any nonzero GRPO component outperforms hint-SFT alone in mid-early training, but hint-SFT alone can match full RLTL;DR given more time.The no-GRPO comparison is conditioned on training to convergence and required approximately 25% more environment interactions than the standard 170k-step budget.Source: §E.3 · p. 24, Figure 4 · p. 25

E.4 Insight evolution over time

  • 0.19 within-visit versus 0.82 between visits: insights vary more across a task's first and last visit, consistent with the policy adopting a different strategy.The comparison covers 92 revisited tasks; within-visit dissimilarity measures variation in verifier wording at one moment.Source: §E.4 · p. 24

E.5 Insight reliance

  • Insight reliance rises from +0.009 in the first half of training to +0.034 in the second, but remains mild; successful trajectories remain plausible with insights removed.The authors relate this to the gains on revisited tasks without insight context.Source: §E.5 · pp. 24–25

E.6 Less difficult Synthetic API split

  • On the 642-task Synthetic API split, GRPO and SGE remain at the base model's 4% Pass@1, while RLTL;DR and RLTF-SD break through the learning barrier.The split includes 184 tasks that became solvable in at least one of 128 attempts under later optimized sampling hyperparameters.Source: §E.6 · p. 25

E.7 Normal-difficulty dataset splits

On normal-difficulty splits, GRPO can match RLTL;DR with sufficient training, while insight-based methods show stronger reward-learning dynamics in some comparisons. The appendix also examines sequential insight exploration and adding insight internalization to RLTF-SD.

E.7 Normal-difficulty dataset splits

  • GRPO matches RLTL;DR when sufficiently trained on normal-difficulty splits where at least 95–97% of tasks are solvable in fewer than 128 attempts; the authors do not claim outperformance there.On the no-insight Pass@1 measure, RLTL;DR and RLTF-SD are similar to GRPO on Synthetic API, while RLTF-SD falls behind on Appworld and both insight methods stagnate on Leetcode.Source: §E.7 · pp. 25, 27, Figure 10 · p. 28
  • As training progresses, RLTL;DR and RLTF-SD move more tasks to higher Avg@8 solve rates, while GRPO fails to show the same improvement; RLTL;DR maintains more fully solved tasks than RLTF-SD.Because these plots include differing amounts of insight conditioning, they compare learning dynamics rather than performance.Source: §E.7 · p. 27
  • After 91K environment interactions, RLTF-SD and RLTL;DR collect the reward GRPO reaches in 200k interactions, yielding 54% efficiency even after correcting for approximately 1.5× higher walltime.The methods perform similarly on Synthetic API, and both achieve higher rewards than GRPO during training.Source: §E.7 · p. 27, Figure 9 · p. 28

E.8 Exploration effectiveness of sequential insight conditioning

  • Before training, sequential insights find solutions for approximately 57% of Pass@128∼0 tasks and 66% of Pass@32∼0 tasks, versus 38% and 54% with one insight, respectively.The corresponding i.i.d. sampling results are approximately 6% and 13%; sequential conditioning rises faster and higher than repeatedly using only the latest insight.Source: §E.8 · pp. 27–28, Figure 11 · p. 29, Figure 12 · p. 29

E.9 Adding Internalization to RLTF-SD

  • Adding LSFT to RLTF-SD by flipping the insight-token backpropagation masks might further help performance, a direction the authors describe as promising.The authors note this change is simple to implement.Source: §E.9 · p. 29, Figure 13 · p. 30

F.1 Training the insight generator versus internalizing insight directly

The appendix compares training the insight generator with internalizing insights directly in the actor, then tests loss-function and insight-insertion ablations. Generator training alone does not recover actor-side LSFT's signal, while some variants improve when added to RLTL;DR.

F.1 Training the insight generator versus internalizing insight directly

  • RLTF-FM trains on every generated insight regardless of rollout success, while Self-play insight SFT trains only on insights followed by a successful task solution.Both approaches disable the actor-side LSFT loss to isolate the insight generator's training signal.Source: §F.1 · p. 30
  • Neither insight-generator SFT variant recovers RLTL;DR's actor-side LSFT signal; added to RLTL;DR, RLTF-FM adds little, while Self-play insight SFT gives a more noticeable gain.Table 8 reports training success without insight context, averaged over the first 80 updates on the SAPI-challenging split.Source: §F.1 · p. 30, Table 8 · p. 31

F.2 Loss functions and GRPO advantage groups

  • A 0.5 success-rate threshold for inserting insights improves over 0.33, while 0.1 slightly reduces performance; using the first 10 rollouts to decide also improves over the other λ = 0.1 runs.The authors attribute the latter result to inserting more insights, which are generally helpful on very hard tasks.Source: §F.2 · p. 31, Table 9 · p. 31
  • Against the 19.6% run, splitting GRPO by insight conditioning leaves performance unchanged, importance-sampling correction for training on insight-conditioned rollouts reduces it, and standard-deviation scaling slightly reduces it.The authors recommend a proper self-distillation loss if the aim is to train on insight-conditioned rollouts after removing insights at update time.Source: §F.2 · p. 31, Table 9 · p. 31

Related papers

Loading 2609.37633v1…