Measuring and Mitigating Solution Mode Collapse in RLVR

1 INTRODUCTION

• RLVR gives all verified-correct solutions the same task reward, regardless of their solution mode.This leaves the weighting of alternative correct solutions undetermined by correctness alone.Source: §1 · p. 2
• The paper introduces MODEBENCH, which records correctness and solution-mode identity, and uses it to study mode collapse under RLVR post-training.Its benchmark spans five executable domains and five difficulty levels.Source: §1 · p. 2
• Across open-weight models and three scales, RLVR substantially reduces solution-mode diversity under both Dr.GRPO and GRPO.The paper also reports fewer than 1.5 effective modes for all measured frontier models, out of up to 355 unique solutions.Source: §1 · p. 2
• Mode replay added to Dr.GRPO raises accuracy and pairwise correct-mode diversity across three model scales and all five MODEBENCH domains.Across the five-domain aggregate, replay methods outperform eight evaluated RLVR comparators on correctness and conditional solution diversity.Source: §1 · p. 2

2 MODEBENCH: MEASURING SUCCESSFUL ALTERNATIVES

• MODEBENCH requires multiple valid discrete solutions and a way to identify each response’s mode; its five domains include graph coloring, computation trees, code, algebra, and constraint optimization.What counts as a mode is domain-specific: it can mean a different route to the same answer or a different valid answer.Source: §2 · p. 2, Figure 2 · p. 3

2.1 SUCCESS AND VERIFIED-MODE DIVERSITY

• PCMD measures the probability that two correct responses use different solution modes, separating conditional diversity from correctness.PCMD is zero when all correct answers share one mode; the section calculates it only when a model produces a minimum number of correct solutions.Source: §2.1 · pp. 3–4
• GPT-5.6 Sol has near-perfect pass@8 on all five tasks considered, yet weaker Qwen2.5-Instruct models can have higher PCMD.The section defines solution-mode collapse as a decline in PCMD during RLVR.Source: Figure 3 · p. 3, §2.1 · p. 4

2.2 IMPACT OF TASK DIFFICULTY ON MODE COLLAPSE

• .991 of frontier-model questions from Levels 1–3 are correct under normalized grading, while mean PCMD is .232, below .570 for the most diverse sub-2B model.The sub-2B model answers .273 of its prompts correctly; across increasingly difficult levels, pass@8 generally declines while PCMD patterns differ by model.Source: §2.2 · p. 4, Figure 4 · p. 4

2.3 CONCENTRATION IN FRONTIER MODELS

• 1.16 distinct colorings emerge from eight draws on Level-3 Graph problems, although all 1,024 draws verify; 95.1% of correct pairs repeat one coloring.In Python, 500 draws surface only one to four solutions, under 2% of possible solutions and under 1% at Levels 2 and 3.Source: Figure 5 · p. 4, §2.3 · p. 5
• Larger sampling budgets remain far below the enumerated or certified solution support in several domains, with recovery curves flattening below the available modes.Figure 5 tracks distinct verified modes recovered as a share of enumerated or certified modes, out to 512 responses per problem.Source: Figure 5 · p. 4, §2.3 · p. 5
• Mixing responses from two independent models covers more verified modes than either alone, while raising inference temperature can expose more modes, sometimes at an accuracy cost.For local models, quadrupling control sampling to 32 draws changes PCMD by at most .010, and MathIR at 64 draws is as concentrated as at 8.Source: §2.3 · p. 5

3.1 THEORETICAL INTUITION FOR MODE COLLAPSE

• As a mode becomes rare, its probability of appearing in a rollout and its expected gradient contribution approach zero, making collapse difficult to reverse under on-policy training.The idealized solution-mode analysis finds winner-takes-all dynamics, while binary rewards alone do not prefer concentration.Source: §3.1 · p. 5

3.2 REPLAYING DISTINCT VERIFIED SUCCESSES

• A fixed-buffer analysis shows that replay keeps a stored mode’s probability bounded away from zero throughout subsequent training under the theorem’s assumptions.The replay intervention is motivated by avoiding dependence on the current policy rediscovering a correct mode.Source: §3.2 · p. 6
• For each training prompt, the buffer stores one exemplar the first time each verified mode appears and revisits nonempty buffers on a deterministic schedule.Subsequent rediscoveries do not modify the buffer.Source: §3.2 · p. 6
• Uniform replay rehearses each retained mode equally, breaking the link between discovery frequency and replay frequency.Frequently sampled modes receive no additional replay.Source: §3.2 · p. 6
• Adding replay to MaxRL yields Re:Max, and applying the modular mechanism to Dr.GRPO yields Re:Dr, with modes contributing through a separate replay loss rather than the on-policy reward group.Replay buffers contain only training-prompt responses, so held-out evaluation measures policy changes rather than buffer retrieval.Source: §3.2 · p. 6
• Re:Max combines fresh-sample learning with equal-weight revisiting of stored verified exemplars, preserving discovered modes while continuing to explore for new ones.The figure contrasts this with MaxRL’s reliance on the policy’s current samples.Source: Figure 6 · p. 5, §3.2 · p. 6

• The main post-training experiment compares replay with Dr.GRPO and MaxRL on five Level 1 domains using five seeds and three model scales.The models are Qwen2.5-0.5B, Falcon3-1B, and Qwen2.5-3B.Source: §4 · p. 6
• Further experiments evaluate Levels 2 and 3 with Qwen2.5-0.5B and compare replay with frequency-weighted replay and other approaches on Level 1.The comparison includes UCPO, RLEP, GAPO, SetPO, and fixed Semantic-MaxEnt under a specified setting.Source: §4 · p. 7
• Inference-only evaluations test seven hosted models on MODEBENCH Levels 1–3 and four open-weight families across available scales and Levels 1–5.Eight responses per prompt are graded with a fixed formatting normalizer; no replay or training is applied.Source: §4 · p. 7

5 RESULTS

• Replay reverses RLVR’s mode collapse without sacrificing correctness and outperforms the eight evaluated RLVR comparators on the five-domain aggregate.Replay gains also persist across difficulty levels, KL anchoring, and domain changes.Source: §5 · p. 7

5.1 RLVR CONCENTRATES SOLUTION MODES

• In 14 of 15 domains, Qwen2.5-3B has fewer solution modes after RLVR; in Python, the number of prompts solved in multiple ways falls from 73 of 128 to as few as 22.The pattern also appears across model scales and difficulty levels; Re:Max increases the percentage of rollout groups with a learning signal.Source: §5.1 · p. 7, Figure 8 · p. 7

5.2 REPLAY IMPROVES BOTH ACCURACY AND DIVERSITY

• Replay shifts outcomes toward both higher pass@8 and greater solution-mode diversity across five MODEBENCH domains and three model scales.This comparison adds replay to Dr.GRPO and MaxRL against their matched no-replay controls.Source: §5.2 · p. 7, Figure 9 · p. 8
• Uniform replay recovers roughly 0.6 additional modes per prompt in Graph and raises pass@8 by 6.6 percentage points, without an aggregate correctness difference.The comparison is against frequency-weighted replay, with buffer construction, advantage scheduling, and loss update held unchanged.Source: §5.2 · p. 8

5.3 REPLAY REMAINS EFFECTIVE BEYOND THE BASE TRAINING SETTING

• At every tested level, replay improves average pass@8 and PCMD; at Level 2, Graph rises from 1.04 to 1.55 effective verified modes under Dr.GRPO and from 1.13 to 1.48 under MaxRL.Gains vary by domain but remain positive in aggregate at Level 3.Source: §5.3 · p. 8, Figure 10 · p. 9
• When an option is removed from a task, Re:Max and Re:Dr are more likely to produce a valid alternative across all five domains and find a new valid answer in fewer attempts when needed.The tests remove an ingredient in PantryPlan or a color in Graph from a portfolio of eight generated answers.Source: §5.3 · pp. 8–9
• At matched diversity, KL anchoring costs model correctness on average across three reportable domains, although the reference is broad enough for the anchor to win in Python and Pantry.Increasing the KL coefficient preserves more diversity but lowers pass@8.Source: §5.3 · p. 9

• MODEBENCH defines solution identity through the executable check used to assign reward, offering a narrower but cheaper diversity measure than semantic clustering, which depends on a judge.Preserving alternatives matters because sampling-based selection relies on generating multiple viable proposals.Source: §6 · p. 9
• KL anchoring traces a diversity–accuracy curve: increasing the KL coefficient preserves more diversity but lowers pass@8.The figure compares the anchor with replay over three domains with reportable initial PCMD.Source: Figure 11 · p. 9
• Many diversity-expanding approaches still operate on current policy samples, while standard replay revisits past successes in proportion to their frequency; this work instead balances replay across modes.The passage relates mode-balanced replay to quality-diversity methods.Source: §6 · p. 9

7 CONCLUSION

• Across objectives, model scales, and task difficulties, replay can retain solution diversity while improving correctness.The authors suggest solution diversity may itself be a target for optimization.Source: §7 · pp. 9–10
• Replay retains only modes found through sampling, and the experiments do not establish its benefit or feasibility in more open-ended solution spaces.Mode count also does not measure the worth of additional solutions.Source: §7 · p. 10

A.1 MEASURING SUCCESS AND DIVERSITY

• PCMD estimates the probability that two independently sampled correct responses represent different modes, separating conditional diversity from correctness measured by pass@K.PCMD requires at least two verified responses per prompt; estimates with fewer eligible prompts have limited support or are omitted.Source: §A.1 · p. 17, §A.2 · p. 18

A.3 WHY BALANCE SOLUTION MODES IN REPLAY

• Replay gives each discovered, stored mode equal weight, favoring both probability mass on buffered modes and a more balanced distribution among them.At fixed buffered probability mass, a balanced allocation increases the expected number of distinct buffered modes in repeated samples.Source: §A.3 · p. 18

B.1 SOLUTION-MODE DEFINITIONS

• MODEBENCH uses executable verifiers to assign correctness and mode identity across five domains, where modes may be different routes or different valid answers.For example, Countdown and MathIR define modes by routes to an answer, while Graph and PantryPlan use distinct valid answers.Source: B BENCHMARK AND EVALUATION PROTOCOL · p. 19, §B.1 · p. 19, Table 2 · p. 19

B.4 MATCHED LEVEL CONSTRUCTIONS

• The benchmark increases task complexity across levels while preserving each domain’s verifier and solution-mode definition; untrained-model diversity does not track correctness or scale in lockstep.Cross-level evaluation populations are not identically distributed in every domain, so those differences do not isolate the causal effect of difficulty.Source: Table 3 · p. 20, §B.4 · p. 20, §B.5 · p. 21

D REPLAY ALGORITHM

• Verified replay stores one exemplar per discovered mode and trains on stored exemplars with equal weight, using a deterministic round-robin schedule across nonempty prompt buffers.Each buffer holds at most 16 modes without eviction; modes found after it fills receive no replay support.Source: D REPLAY ALGORITHM · p. 24, §D.3 · p. 25

D.4 MATCHED CONTROL

• Replay and control share prompts, fresh-rollout budgets, validation, scheduling, and optimizer updates, but total computation and wall-clock time are not matched.The control runs the same replay machinery with the replay gradient set to zero.Source: §B.6 · p. 21, §D.4 · p. 25

D.5 VERIFIER REQUIREMENTS AND APPLICABILITY

• Replay requires training-time verification that assigns correct responses to task-relevant modes; final-answer-only graders merge distinct derivations unless a finer identity rule is used.The reported scale extension uses one matched seed per domain, so it establishes a domain contrast rather than a model-size trend.Source: §D.5 · p. 26, §E.2 · p. 30

E.1 TERMINAL COMPARISONS ACROSS SCALES

• Re:Dr raises mean PCMD in all 14 measurable model–domain comparisons, while Re:Max does so in 14 of 15 comparisons.Falcon3-1B Graph is the lone near-zero Re:Max diversity comparison; Qwen2.5-3B has positive Re:Max effects in all five domains.Source: §E.1 · p. 28
• Replay improves mean correctness and conditional diversity across model scales under both fresh objectives, with domain- and objective-dependent effect sizes.Table 5 reports replay-minus-control effects at the terminal checkpoint; Figure 13 shows effects across three scales and five domains.Source: Table 5 · p. 28, §E.1 · pp. 28–29, Figure 13 · p. 28

E.2 LEVEL 1 AT QWEN2.5-7B

• At Qwen2.5-7B, mean terminal pass@8 rises from .447 for MaxRL to .787 for Re:Max across four domains, but the aggregate masks domain variation.Correctness gains range from +.019 on PantryPlan to +.744 on MathIR; corresponding PCMD changes are +.137 on Countdown, +.298 on Graph, and −.036 on MathIR.Source: §E.2 · p. 29
• At terminal evaluation, Re:Max pass@8 is .805 versus MaxRL’s .631 on Countdown, .818 versus about .40 on Graph, and .744 versus zero on MathIR.On PantryPlan, terminal pass@8 is .779 for Re:Max and .760 for MaxRL, with PCMD .348 and .384, respectively.Source: §E.2 · p. 30

E.3 RESULTS ON HARDER BENCHMARK LEVELS

• On Level-2 Graph, Re:Dr and Re:Max reach pass@8 of .763 and .739 versus .202 and .434 for their controls, with PCMD gains of +.314 and +.212.Replay also improves PantryPlan correctness under both objectives and yields positive Python diversity estimates, while MathIR remains concentrated; cross-level contrasts do not isolate difficulty alone.Source: §E.3 · pp. 30–31, Figure 15 · p. 31

E.4 DIVERSITY THROUGHOUT TRAINING

• Replay's terminal diversity advantage reflects sustained training differences, with the clearest separation on Graph and near-zero terminal diversity under no replay on smaller-scale PantryPlan runs.MathIR remains concentrated under all four methods on the harder Level-2 tasks.Source: §E.4 · pp. 31–32, Figure 16 · p. 31, Figure 17 · p. 32

E.5 SUPPORTING COMPARATORS

• An online RLEP-Dr pool raises PCMD by +.129 and pass@8 from .444 to .583, recovering about 35% of Re:Dr's aggregate diversity gain; much of the gap remains.RLEP-Dr pools contain few modes, and replay is active on only .20 of Python updates from pass 2 onward; Re:Dr buffers average 1.15–6.88 modes at Qwen2.5-0.5B.Source: §E.5 · pp. 32–33, Table 7 · p. 32

E.6 DIVERSITY-PRESERVING COMPARATORS

• Against the shared Dr.GRPO control, five-domain mean PCMD effects are +.225 for GAPO, +.066 for SetPO, and +.320 for Re:Max.GAPO is strongest on PantryPlan, while SetPO raises Python pass@8 from .338 to .834.Source: §E.6 · pp. 33–34

E.7 REFERENCE KL TRADEOFFS

• Reference KL's tradeoff depends on domain: at β = .3 it reaches PantryPlan PCMD .931 and pass@8 .825, but MathIR PCMD .099 and pass@8 .207.Replay has the stronger reported correctness–diversity balance in Graph and MathIR under the tested settings; KL can preserve high diversity when the reference itself is broad.Source: §E.7 · p. 34

E.8 UNIFORM VERSUS FRESH-FREQUENCY REPLAY

• Uniform replay adds +.318 extra verified modes per prompt over the five-domain aggregate, while its pass@8 effect is +.010 and not detectably different from zero.The diversity gain is driven primarily by Graph, Countdown, and PantryPlan; several larger-model comparisons show the same direction.Source: §E.8 · p. 34, Figure 19 · p. 35

F.1 FRESH 64-RESPONSE EVALUATION

• Replay's PCMD advantage persists with 64 fresh responses per prompt: Dr.GRPO reduces PCMD in all six model–domain comparisons, while replay improves it in all twelve comparisons.The paired analysis uses jointly eligible prompts and five training seeds.Source: §F.1 · p. 35

F.2 CONDITIONAL-CONCENTRATION ROBUSTNESS

• Under both disjoint sampling assignments, seven of ten Level-2 replay–control comparisons favor replay, one is unchanged, and two PantryPlan comparisons favor control.Several cells have small eligible populations.Source: §F.2 · p. 35

F.3 DIVERSITY BEFORE AND AFTER RLVR-ONLY TRAINING

• Across Dr.GRPO and GRPO, additional verified modes decline at every tested scale, by approximately 98–99% at Qwen2.5-0.5B and 42–72% at larger scales.Domain-average PCMD declines approximately 99% at Qwen2.5-0.5B and 51–65% at larger scales, although individual domain–model combinations need not decline.Source: §F.3 · pp. 35–36, Figure 20 · p. 36
• This breadth loss can coexist with stable or improved correctness: pass@8 does not consistently decrease, and distinct@8 can rise when correctness gains outweigh lost additional modes.Conditional diversity also declines, so the reduction in observed breadth is not explained solely by changes in correct-response probability.Source: §F.3 · pp. 35–36

F.4 SPARSE TASK GRADIENTS AND OUTPUT REPETITION

• Only 1.5% of Python control groups contain both correct and incorrect responses, leaving 98.5% with zero fresh task advantage; replay can still supply a likelihood gradient.For Dr.GRPO, identical within-group rewards make all within-group advantages vanish.Source: §F.4 · p. 36, Table 9 · p. 37
• At terminal evaluation, Python groups contain 1.00 distinct completion per prompt under Dr.GRPO versus 1.97 under Re:Dr.All five control seeds produce a single repeated completion per prompt through terminal evaluation.Source: §F.4 · pp. 36–37

F.5 REPLAY GAINS WITH ACTIVE AND TUNED CONTROLS

• Replay remains beneficial among active controls, and all six comparisons with nonzero initial pass@8 and at least 10% mixed rollout groups favor replay on both pass@8 and PCMD.The subset analysis tests whether gains persist when controls show task learning and a nontrivial fresh reward signal.Source: §F.5 · p. 37, Table 10 · p. 38
• Correctness-based tuning strengthens no-replay controls in four domains, yet every paired seed favors Re:Dr on conditional diversity in both reported tuned-control domains.In these cases, replay-trained policies remain less concentrated and more successful at the terminal checkpoint.Source: §F.5 · pp. 37–38, Table 11 · p. 38, Table 12 · p. 38

G HUMAN-AUTHORED PROGRAMMING EVALUATION

• The evaluation uses 21 Codeforces problems with released checkers and frozen hidden-input suites; eight train and thirteen serve development.Modes are canonical witness signatures across the finite suite, so they characterize behavior on that suite rather than semantic program equivalence.Source: G HUMAN-AUTHORED PROGRAMMING EVALUATION · p. 38
• Replay uses one verified exemplar per discovered mode, uniform rehearsal, and capacity 16 per prompt, with five paired seeds per arm and 128 updates.The controls build buffers and score exemplars identically but set the replay gradient to zero.Source: G HUMAN-AUTHORED PROGRAMMING EVALUATION · pp. 38–39
• Pooling ten matched replay–control seed pairs, replay raises the verified fraction of fresh rollouts by +.014, with interval [+.007, +.021] and positive effects in nine pairs.The replay mechanism transfers unchanged to these tasks and produces a small, consistent training-side correctness improvement.Source: G HUMAN-AUTHORED PROGRAMMING EVALUATION · p. 39

H STANDARD MATHEMATICAL REASONING EVALUATION

• Across all six benchmarks, Re:Dr has higher pass@32 than matched Dr.GRPO; on AMC23 it reaches .850 versus .767 after Dr.GRPO and .875 for the base model.On OlympiadBench, pass@32 is .620 for Re:Dr, .546 for Dr.GRPO, and .628 for the base model.Source: H STANDARD MATHEMATICAL REASONING… · p. 39, Table 13 · p. 39
• Re:Dr increases avg@32 on every benchmark, including MATH-500 from .470 to .537, while greedy correctness remains essentially unchanged.Majority-vote accuracy changes substantially less than pass@32, consistent with increased sampled access to correct alternatives.Source: H STANDARD MATHEMATICAL REASONING… · p. 39, Table 13 · p. 39
• MaxRL reaches MATH-500 pass@32 .902 and per-response accuracy .498, compared with .889 and .470 for Dr.GRPO; it also recovers most of Dr.GRPO's AMC23 coverage reduction.MaxRL's pattern is consistent with its stronger weighting of successes when they are rare.Source: H STANDARD MATHEMATICAL REASONING… · p. 39, Table 14 · p. 40
• Re:Max stays within .033 pass@32 of MaxRL on each paired benchmark, because final-answer verifiers treat all correct derivations as one mode and cannot balance them.On MODEBENCH, where verifiers distinguish correct solutions, Re:Max can rehearse multiple modes and gains substantially over MaxRL.Source: H STANDARD MATHEMATICAL REASONING… · p. 40

I PROMPT-GUIDANCE ABLATION

• The comparison uses 32 matched Python, MathIR, and PantryPlan problems at Levels 2–3; Python’s interface change also removes an executable example and divisor guidance.The Python results therefore reflect a combined prompt-interface change, not an isolated hint removal.Source: I PROMPT-GUIDANCE ABLATION · p. 41
• Every eligible trained MathIR comparison remains at PCMD = .000 under benchmark and neutral wording, while neutral Level-2 Re:Dr pass@8 is .975 versus .994.The Python interface intervention sharply reduces correctness, especially for Dr.GRPO; Re:Dr neutral pass@8 is .556 at Level 2 and .494 at Level 3.Source: I PROMPT-GUIDANCE ABLATION · p. 41, Table 15 · p. 41
• Neutral wording increases eight-correct-draw breadth for GPT-5.4 and GPT-5.6 Sol, but the corresponding Grok 4.3 comparisons are inconclusive.The hosted-model sensitivity indicates that some measured concentration depends on prompt guidance, unlike the local trained MathIR checkpoints.Source: I PROMPT-GUIDANCE ABLATION · p. 41

J INFERENCE-TIME DECODING CONTROLS

• Across six temperatures, control PCMD stays at or below .021 in Graph, .016 in Countdown, .002 in MathIR, and .004 in PantryPlan, below Re:Dr at T = 1.Python controls lack enough verified responses for conditional-diversity estimates.Source: J INFERENCE-TIME DECODING CONTROLS · pp. 41–42
• Increasing control group size from K = 8 to K = 32 changes PCMD by at most .010, and no K = 32 control mean exceeds .054.These fixed-checkpoint decoding interventions do not test rollout-sampling changes during training.Source: J INFERENCE-TIME DECODING CONTROLS · pp. 41–42
• Larger sampling groups leave control diversity low across eligible domains, so tested decoding changes do not eliminate the gap to replay-trained policies.Missing PCMD estimates indicate insufficient verified responses, not proof that alternative correct modes do not exist.Source: J INFERENCE-TIME DECODING CONTROLS · p. 42

K.1 SAMPLING-BUDGET DESIGN

• The sampling-budget analysis collects up to 64 local responses and 512 hosted responses per prompt, retaining incorrect, empty, refused, and truncated outputs in the sampling denominator.Breadth is defined as modes observed within the stated finite response pool and sampling budget.Source: §K.1 · p. 42

K.2 DISCOVERY AT LARGER SAMPLING BUDGETS

• At 64 draws, trained MathIR checkpoints still produce at most one verified mode per problem at Level 2, and nearly all retain near-single-mode behavior at Level 3.Python and PantryPlan reveal additional modes with larger budgets, but the overall strongest concentration patterns remain, so the result is not an eight-draw artifact.Source: §K.2 · p. 42

K.3 DISCOVERY AT 512 DRAWS

• At 512 draws, GPT-5.6 Sol’s mean distinct@512 remains far below available support, including MathIR’s 1.00 and .97 modes against at least five certified modes per problem at Levels 2–3.Python’s Level-1–3 means are 3.75, 1.44, and 1.34 against enumerated averages near 222, 336, and 355 modes; late discovery varies by domain.Source: §K.3 · pp. 42–43

L.1 COMPLEMENTARITY ACROSS HOSTED MODELS

• Every tested hosted-model pair gains expected distinct modes over the average of its constituents, by .115–.266 modes, and gains persist with coarser mode definitions and normalized grading.At four verified responses, each pair’s gain over the average constituent is .073–.177 modes.Source: §L.1 · p. 43

L.2 SENSITIVITY TO SOLUTION-MODE IDENTITY

• Coarsening solution identity lowers observed counts and can shrink replay’s Python distinct@8 gain from .431 to .075 at Level 2 and .419 to .000 at Level 3.The measured diversity gain therefore depends partly on which distinctions count as meaningful; neutral-wording coarse-mode gains remain .556 and .500.Source: §L.2 · p. 43

M PORTFOLIO SURVIVAL UNDER WITHDRAWN OPTIONS

• At four verified draws after one binding option withdrawal, replay improves survival most strongly in Graph, Countdown, Python, and PantryPlan, while MathIR changes little.The test removes an option that eliminates at least one certified mode while leaving another feasible.Source: M PORTFOLIO SURVIVAL UNDER WITHDRAWN… · p. 43

M.1 TWO WITHDRAWALS AT ONCE

• Under two withdrawals, survival gains remain strongest in Graph, Python, and PantryPlan, but the Re:Max Countdown point estimate reverses.Survival gains are therefore not uniform across domains and objectives.Source: §M.1 · pp. 43–44, Table 16 · p. 44

M.2 DECODING SETTINGS AS A SUBSTITUTE

• Replay exceeds the best of eleven tested decoding controls in all five domains for four-draw survival, with paired gains of 1.37–36.97 percentage points.Ordinary-sampling recovery intervals favor replay, while diversity prompting is less consistent and reverses the comparison in PantryPlan; replay’s gain is a property of the trained portfolio, not a diversity-only mediation estimate.Source: §M.2 · p. 44, §M.3 · p. 44, Table 17 · p. 44

N INFERENCE-ONLY CONCENTRATION IN GPT-5.6 SOL

• On Level-3 Graph, all 1,024 hosted responses verify, yet mean distinct@8 is 1.16 and 95.1% of correct-response pairs share a mode; concentration also appears across hosted deployments and benchmark regimes.Prompt-interface effects differ across deployments, and disabling explicit reasoning lowers overall pass@8 and distinct@8 in all five tested deployments, while conditional-diversity changes are not uniform.Source: N INFERENCE-ONLY CONCENTRATION IN… · p. 45, §O.1 · p. 45, §O.2 · p. 46

O.4 TEMPERATURE SENSITIVITY ACROSS DEPLOYMENTS

• From T = 0 to T = 2, GPT-5.6 Sol pass@8 rises 13.8 percentage points and distinct@8 rises .473 modes, while per-response accuracy falls from 62.1% to 55.9%.For Grok 4.3 and Kimi K3, temperature changes are deployment-dependent, including truncation-related drops for Kimi K3.Source: §O.4 · p. 47, Table 24 · p. 47, Table 23 · p. 47

O.5 NATIVE PROVIDER OUTCOMES

• Claude Opus 5 is the only hosted deployment with provider-declared refusals in the cohort: 2,522 of 15,360 responses, mostly in Python.These refusals explain why several of its conditional-diversity cells are undefined.Source: §O.5 · p. 47

P.1 A MODE-LEVEL MODEL OF RLVR

• In the mode-level model, Dr.GRPO and practical MaxRL expected updates are positive scalar multiples of ∇zP, so their mean trajectories share a direction but can move at different speeds.The analysis intentionally abstracts from neural-training details including reference-KL, entropy regularization, finite-step effects, optimizer dynamics, and parameter sharing.Source: §P.1 · pp. 48–49

P MODE COLLAPSE AND VERIFIED REPLAY

• When one correct mode initially has strictly greater probability, the idealized flow can converge to winner-take-all concentration; binary reward itself does not prefer that collapsed solution.The concentration arises from optimization dynamics, not a distinct reward for the winning mode.Source: P MODE COLLAPSE AND VERIFIED REPLAY · p. 48, §P.1 · p. 49

P.2 RARE MODES BECOME DIFFICULT TO RECOVER

• As a mode’s probability shrinks, both its chance of appearing in an on-policy rollout group and its score-gradient signal shrink toward zero.This dependence on rediscovery is the mechanism replay removes for stored modes.Source: §P.2 · pp. 49–50

P.3 REPLAY PRESERVES STORED MODES

• With a fixed nonempty replay buffer and positive replay weight, every stored mode remains at positive probability, but the theorem does not guarantee discovery of unseen modes.The analysis identifies a need for a discovery-oriented objective to find new modes.Source: §P.3 · p. 50, §P.4 · p. 51

Q.3 LONGER-HORIZON CONTROL

• A 12-pass no-replay control remains far below eight-pass Re:Dr: mean pass@8 is .434 versus .748 and mean PCMD is .013 versus .376.The longer control uses 50% more fresh rollouts; Python control pass@8 declines from .338 to .182, and wider groups likewise fail to close the aggregate gap.Source: §Q.3 · p. 51, §Q.4 · p. 52

Q.5 SINGLE-SLOT REPLAY

• Single-slot replay recovers 58% of aggregate Re:Dr–control diversity improvement, while balancing multiple stored modes adds a further .155 PCMD across 25 matched cells.Replay’s distinguishing organization is one exemplar per discovered mode with equal rehearsal weight; replay retains past successes and balances rehearsal rather than merely adding rollout compute.Source: §Q.5 · p. 52, R COMPARISON WITH DIVERSITY-PRESERVING… · p. 53