PaperlayerView exact revision on arXiv

Source-linked AI summary

All Work And No Play Makes Jack a Dull Boy: Understanding and Preventing Catastrophic Strategy Collapse in RLVR

Qiyuan Huang, Tianshi Xu, Meng Li

arXiv:2610.02835v1cs.LG

TL;DR

RLVR can undergo abrupt late-stage collapse, while its underlying mechanism remains poorly understood. This paper explains collapse through strategy concentration versus the capacity needed for successful reasoning, derives MEI, and proposes Mesh Learning, which outperforms baselines across benchmarks and model families.

  • Problem

    RLVR can exhibit severe late-stage instability across model families and training recipes, but the mechanism underlying this collapse remains poorly understood.

    Source: §1 · p. 2

  • Method

    The paper models strategies through policy-update interactions, derives a theory of strategy concentration and capacity, develops MEI as an online proxy, and proposes Mesh Learning with Coach Prompting and balancing regularization.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across AIME26, AIME25, MATH-500, GPQA, and LiveCodeBench, Mesh Learning consistently outperforms strong baselines across Qwen and Phi, with gains up to 13.4 and 11.5 pp, respectively.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results identify strategy preservation as a key principle for stable and effective RLVR optimization.

    Source: §1 · p. 2

    The collapse theorem is scoped to GRPO, GSPO, or DAPO with step-wise old-policy refresh under its stated learning-rate, clipping, trajectory-length, and accuracy conditions.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

During post-training of large language models (LLMs) with Reinforcement Learning with Verifiable Rewards (RLVR), GRPO-style algorithms can exhibit severe late-stage collapse. Prompt-based probing reveals that this is not benign strategic pruning, but a harmful contraction of effective strategy capacity that makes distinct reasoning strategies increasingly inaccessible. To characterize this phenomenon, we define strategies through trajectory-level policy-update interactions and develop a unified theoretical framework combining optimization dynamics and information theory. We prove that major RLVR objectives progressively concentrate probability mass onto a single strategy, while sustaining nontrivial task accuracy requires a minimum strategy capacity. The conflict between these two results provides a mechanistic explanation for catastrophic collapse. We further derive the {Mirrored Entanglement Index (MEI)} as a lightweight online warning signal. To prevent collapse, we propose \textbf{Mesh Learning}, which exposes multiple reasoning strategies and prevents any single strategy from dominating optimization. Across AIME26, AIME25, MATH-500, GPQA, and LiveCodeBench, Mesh Learning consistently outperforms strong baselines across Qwen and Phi model families, with gains of up to 13.4 pp and 11.5 pp, respectively. These results establish strategy preservation as a key principle for stable RLVR. Code is available at https://github.com/Ayanami-0123/Open-Mesh-Learning.

1 INTRODUCTION

RLVR can exhibit severe late-stage collapse, which the paper explains as a conflict between optimization-driven strategy concentration and the strategy capacity required for successful reasoning. It derives MEI as an online warning signal and proposes Mesh Learning, which combines strategy exposure with balancing regularization.

1 INTRODUCTION

  • RLVR shows abrupt late-stage accuracy degradation after apparently successful optimization across Qwen and Phi models and five training recipes.The observed recipes include GRPO, DAPO, GSPO, GRPO+KL, and GRPO+JS.Source: §1 · p. 2
  • The theory shows that GRPO, DAPO, and GSPO drive strategy concentration, while nontrivial task accuracy requires a minimum strategy capacity, creating a conflict that explains collapse.The paper describes this conflict as a mechanistic explanation for catastrophic collapse.Source: §1 · p. 2
  • MEI is a lightweight online proxy that rises above a calibrated 3σ threshold before the accuracy cliff across model families, RLVR algorithms, and regularization schemes.Independent strategy-level analysis links the transition to loss of distinct reasoning strategies.Source: §1 · p. 2
  • Mesh Learning consistently outperforms strong RLVR and divergence-regularized baselines across five benchmarks and Qwen/Phi, with gains up to 13.4 and 11.5 pp, respectively.The benchmarks are AIME26, AIME25, MATH-500, GPQA, and LiveCodeBench.Source: §1 · p. 2
  • Mesh Learning combines Coach Prompting, which elicits distinct approaches, with Strategy-Balancing Regularization, which prevents any strategy from dominating optimization.Together, these components expose and preserve multiple viable reasoning strategies throughout training.Source: §1 · p. 2

2 FROM TRAJECTORY TO STRATEGY

The paper groups trajectories by how they respond together to policy updates, defining strategies through their update interactions rather than exact recurrence. It finds that these dynamically defined groups align with distinct solution methods on MATH-500.

2 FROM TRAJECTORY TO STRATEGY

  • Strategies group trajectories that respond coherently to local policy updates, since exact recurrence of individual trajectories is unlikely and offers limited insight into training dynamics.Trajectories in a group tend to be reinforced or suppressed together.Source: §2 · p. 3

2.1 THE CONCEPT OF STRATEGY

  • The paper defines a strategy through neighborhoods of trajectories that are positively coupled and have similar coupling profiles under policy updates.The resulting dynamic-equivalence relation induces strategy equivalence classes.Source: §2.1 · p. 3

2.2 THE SEMANTIC PROXY OF STRATEGY

  • On MATH-500, dynamically defined strategies align empirically with distinct solution methods: within-method trajectory pairs are more positively coupled than cross-method pairs.This correspondence motivates solution-method prefixes as practical proxies for latent strategies.Source: §2.2 · p. 3, Figure 2 · p. 3

3 THEORETICAL ANALYSIS

The analysis explains collapse as a conflict between RLVR’s tendency to concentrate strategy mass and the capacity needed to sustain nontrivial accuracy. It also derives MEI as an early warning signal and shows why fixed divergence penalties cannot universally prevent concentration.

3.1 THE MAIN THEOREM OF STRATEGY COLLAPSE

  • Under non-collapse conditions, GRPO, GSPO, and DAPO concentrate at least 1 − ζ of a query’s strategy mass on one strategy after burn-in, reducing effective strategy capacity.Theorem 1 gives this result with probability at least 1 − δ under its stated scale and training assumptions.Source: §3.1 · p. 4

3.2 POWER LAW FOR STRATEGY CAPACITY LOWER BOUND

  • Higher accuracy requires greater usable strategy capacity, so RLVR’s concentration can violate the lower bound and make performance unsustainable.The entropy loss from collapsing mϵ strategies to one is approximately log mϵ, exceeding the available o(log mϵ) budget; all tested models satisfy the predicted lower bound.Source: §3.2 · pp. 4–5, Figure 3 · p. 5

3.3 HOW TO MEASURE AND FORECAST COLLAPSE

  • MEI is calibrated to 1.013 at a 3σ threshold and crosses it before its final surge and the subsequent accuracy collapse.This provides an online signal without explicitly tracking parameter-space gradients during optimization.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4 WHY DIVERGENCES FAIL

  • When the reward gap relative to regularization exceeds a finite threshold, KL- or JS-regularized optima can place at least 1 − ϵ mass on one strategy.Thus, fixed divergence penalties cannot universally prevent concentration when reward dominates regularization.Source: §3.4 · p. 5

4 METHODOLOGY: MESH LEARNING

Mesh Learning combines multi-strategy exploration with regularization that balances strategy growth. Coach Prompting supplies strategy prefixes during training, while the policy proposes its own strategy at inference.

4 METHODOLOGY: MESH LEARNING

  • Mesh Learning replaces homogeneous GRPO rollouts with multi-strategy exploration and jointly optimizes GRPO with Strategy-Balancing Regularization.Its two components elicit multiple strategies and prevent any one strategy from dominating; under the analyzed conditions, the dynamics converge to an equilibrium retaining nonzero probability for every strategy.Source: §4 · p. 6

4.1 STRATEGY ELICITATION: COACH PROMPTING

  • An offline Coach LLM generates fixed strategy prefixes for training, but the policy proposes its own strategy at inference and uses the same verification procedure.Prefixes are treated as candidates during training, and Coach-provided tokens are masked from the training loss; no Coach LLM is needed at test time.Source: Figure 6 · p. 6, §4.1 · p. 6

4.2 STRATEGY-BALANCING REGULARIZATION

  • Strategy-Balancing Regularization compares strategy growth against a fixed pre-RLVR reference to avoid length bias and padding shortcuts.Its gradient suppresses relatively fast-growing strategies and reinforces lagging ones, while allowing their common improvement and preserving within-strategy trajectory competition to first order.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

Mesh Learning improves results across the reported models and benchmarks, while its training dynamics and ablations support preserving multiple strategies during optimization. The experiments compare it with established RLVR baselines across math, science, and coding tasks.

5.1 EXPERIMENTAL SETUP

  • Experiments use Qwen3-4B and Qwen2.5-7B-Instruct, with additional Phi-4-mini-reasoning tests, and compare Mesh Learning with GRPO, DAPO, GSPO, and KL- or JS-regularized GRPO.Evaluations cover AIME25/26, MATH-500, GPQA-Diamond, and LiveCodeBench v6.Source: §5.1 · p. 7

5.2 MAIN RESULTS

  • Mesh Learning improves Qwen2.5-7B-Instruct by 1.6–3.4 pp and Qwen3-4B by 2.1–13.4 pp over the strongest baseline across eight model–benchmark settings.On Qwen3-4B AIME26, accuracy rises from 43.3% to 56.7%.Source: §5.2 · p. 7, Table 2 · p. 8
  • Baselines can undergo abrupt accuracy collapse as MEI rises beyond the 3σ threshold; Mesh Learning keeps MEI low and accuracy higher, except on GPQA, where the theorem's condition is not met.The paper attributes the GPQA exception to its four answer choices implying γ ≈1.Source: §5.2 · p. 7, Figure 7 · p. 8
  • At Pass@128, Mesh Learning's gains are 6.9 and 7.3 pp for Qwen2.5-7B on AIME26 and AIME25, and 32.6 and 28.8 pp for Qwen3-4B, respectively.Its advantage remains substantial as the inference budget increases.Source: §5.2 · p. 8, Figure 8 · p. 8
  • On Phi-4-mini-reasoning, Mesh Learning beats the stronger baseline by 8.3, 6.7, 3.3, and 11.5 pp on AIME26, AIME25, MATH-500, and GPQA-Diamond, respectively.The reported results extend beyond the Qwen model family.Source: §5.2 · pp. 8–9, Table 3 · p. 8
  • On LiveCodeBench v6, Mesh Learning reaches 56.4% on Qwen3-4B and 36.3% on Phi-4-mini-reasoning, beating DAPO by 4.2 pp and 3.6 pp, respectively.This evaluation tests transfer beyond math reasoning.Source: §5.2 · p. 9, Table 4 · p. 9

5.3 ABLATION STUDIES

  • Coach Prompting alone brings little or no improvement, whereas full Mesh Learning performs consistently better, indicating that strategy priors alone do not account for the gains.The comparison removes Strategy-Balancing Regularization from the Coach Prompting variant.Source: §5.3 · p. 9
  • Increasing the strategy count from 3 to 4 makes only small overall differences, while an overly small count limits exposed strategies and the capacity preserved during training.The default strategy count is m = 4.Source: §5.3 · p. 9

6 RELATED WORK

Related work improves RLVR optimization and documents collapse through several diversity- and performance-related accounts. This paper instead develops a strategy-level theory connecting strategy concentration with the capacity needed for nontrivial performance.

RLVR optimization

  • Prior RLVR optimization work targets sampling, clipping, importance weighting, and objectives, while this paper focuses on the mechanism underlying late-stage collapse.Examples include DAPO, CISPO, GSPO, SAPO, Dr. GRPO, VAPO, REINFORCE++, GMPO, and FAPO.Source: §6 · p. 9

Understanding and Preventing Collapse

  • Earlier studies connect collapse to reduced generation diversity, negative interference, winner-take-all dynamics, saturated-problem optimization, or long-horizon training.The passage also describes fixed-reference KL and alternative divergences as approaches to preserving diversity.Source: §6 · p. 9
  • The paper derives both a tendency toward strategy concentration and a lower bound on strategy capacity for nontrivial task performance, using their tension to explain collapse and motivate Mesh Learning.Source: §6 · p. 9

7 CONCLUSION

The paper characterizes RLVR collapse as a conflict between strategy concentration and the capacity needed for successful reasoning, and derives MEI as an online warning signal. It proposes Mesh Learning to preserve diverse strategies and reports effectiveness across Qwen and Phi models and several benchmarks.

7 CONCLUSION

  • The conclusion presents strategy preservation as a principle for stable and effective RLVR, because strategy concentration conflicts with the capacity needed for successful reasoning.The paper also describes MEI as a lightweight online warning signal and Mesh Learning as a way to expose and preserve diverse strategies during optimization.Source: §7 · p. 9
  • The experiments report Mesh Learning's effectiveness across AIME26, AIME25, MATH-500, GPQA, and LiveCodeBench, spanning Qwen and Phi model families.Source: §7 · p. 9
  • The paper says its code and scripts are available in an anonymous GitHub repository and training datasets are in supplementary materials, except the coding dataset, which is too large to include.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1.1 NOTATIONS

  • The notation appendix identifies π as the model policy, θ as model parameters, D as the test dataset, Q as prompts, and τ as a generated trajectory.Source: §A.1.1 · p. 13

A.1.2 MATHEMATICAL EXPRESSIONS OF THREE RLVR OBJECTIVES

  • The appendix presents policy-gradient objectives for GRPO, DAPO, and GSPO, with KL regularization analyzed separately.Source: §A.1.2 · p. 13

A.1.3 MEASURES ON STRATEGY SPACE

  • The strategy-space appendix defines the conditional probability mass of a strategy under prompt Q at optimization step k.Source: §A.1.3 · p. 13

A.2 PROOF FOR THEOREM 1

The proof lifts policy updates to strategy-distribution dynamics and analyzes their evolution, including within-strategy competition and the timescale separation from correction terms and within-strategy distributions. It also uses hitting-time and persistence arguments to examine how strategy concentration can arise and persist.

A.2.1 STEP 1: DERIVING THE DYNAMICS OF TRAJECTORY PROBABILITIES:

  • Clipping is absorbed into effective advantages, yielding shared first-order trajectory-probability dynamics for GRPO, GSPO, and DAPO.For GRPO and GSPO the clipping mask is response-level, while DAPO applies it token-wise.Source: §A.2 · p. 14

A.2 PROOF FOR THEOREM 1

  • A potential with positive drift and an Azuma–Hoeffding bound establishes that a dominant strategy is reached within the adiabatic zone and remains dominant afterward.The proof focuses on group-relative rewards; divergence regularization is analyzed separately in Theorem 3.Source: §A.2 · pp. 14–15

A.2.2 STEP 2: DERIVING THE DYNAMICS OF TRAJECTORY DOMINANCE WITHIN A STRATEGY:

  • Within a fixed strategy, normalized trajectory shares follow stochastic replicator-like dynamics: above-average-fitness trajectories gain probability, while lower-fitness expressions are pruned.Because fitness is computed over the dataset, even correct but less general or field-specific expressions can lose probability to more general alternatives.Source: §A.2.2 · p. 19

A.2.3 STEP 3: ADIABATIC APPROXIMATION

  • Under the stated training conditions, strategy-level mass evolves faster than the kernel, advantages, noise, and within-strategy distribution, enabling an adiabatic approximation.Within a finite training window, the score-function magnitude scales as Θ(|τ|); the passage explicitly limits this to local-in-time scaling, not monotonic growth over training.Source: §A.2.3 · pp. 20–21
  • The kernel evolves more slowly than trajectory probabilities on long reasoning trajectories, and the correction factor likewise changes on a slower scale than the policy.The correction-factor argument relies on its variation being a second difference of the log-policy, after first-order increments cancel.Source: §A.2.3 · pp. 22–23
  • For DAPO, the effective correction entering population dynamics evolves more slowly than trajectory probability, even though its raw correction factor need not be adiabatic.After sequence-level importance-ratio cancellation, the remaining token-level ratios and mixed kernels evolve adiabatically relative to trajectory probability.Source: §A.2.3 · pp. 24–25
  • Only an O(η^2|τ|) proportion of trajectories undergoes a clipping transition during an update step.The passage uses this bound in analyzing the relative change rate of the clipped reward signal.Source: §A.2.3 · p. 26
  • Coarse-graining trajectories into strategies preserves the anti-symmetric flow structure, while within-strategy fitness variation is uniformly O(ϵB).This separates strategy selection by mean fitness from slower redistribution among trajectories within a strategy.Source: §A.2.3 · pp. 28, 30
  • Within-strategy distributions and the quantities q, A, K, and r remain effectively frozen over the local time scale of strategy-level evolution.The argument contrasts strategy selection driven by mean fitness with within-strategy redistribution driven by smaller fitness fluctuations.Source: §A.2.3 · p. 30

A.2.4 STEP 4: HITTING TIME ANALYSIS

The hitting-time analysis derives strategy concentration from a positive-drift potential, then shows that concentration persists beyond the adiabatic window. Across GRPO, GSPO, and DAPO, faster strategy-level competition amplifies existing advantage and drives probability mass toward a dominant strategy.

A.2.4 STEP 4: HITTING TIME ANALYSIS

  • The strategy potential has positive drift outside the collapse region, yielding a near-collapsed distribution during the adiabatic period under the stated scale conditions.The analysis bounds the drift below by a positive constant and shows the burn-in time is much shorter than the focused validity horizon.Source: §A.2.4 · pp. 30, 35, 36, 37

A.2.5 STEP 5: PRESERVATION AFTER THE ADIABATIC WINDOW.

  • After burn-in, the dominant strategy retains its mass through the fixed validity horizon, so concentration persists beyond the adiabatic window.The proof concludes that most strategy probability mass lies on one dominant strategy.Source: §A.2.4 · p. 37, §A.2.5 · p. 39

A.2.6 THE CONSEQUENCES OF STRATEGY DYNAMICS

  • For GRPO, GSPO, and DAPO, strategy-level redistribution is faster than within-strategy rearrangement, and advantage differences are progressively amplified.A strategy with relative advantage gains probability while suppressing competitors, providing the proposed dynamical mechanism for strategy collapse.Source: §A.2.6 · p. 39

A.3 PROOF FOR THEOREM 2

The proof combines an adiabatic bound on strategy-distribution change with an information-theoretic lower bound on strategy capacity at nontrivial accuracy. It also develops a lightweight gradient-based monitor whose alignment increases as trajectories approach collapse.

A.3.3 BOUNDED CROSS ENTROPY

  • The proof controls conditional strategy entropy using the pre-RLVR distribution and bounds cross-entropy at the chosen resolution.Source: §A.3 · p. 40, §A.3.3 · p. 42

A.3 PROOF FOR THEOREM 2

  • At a fixed, pre-specified strategy resolution, the proof bounds the usable strategy capacity needed to sustain nontrivial accuracy.The final bound scales as Ωρ,γ(mϵN Pacc,k−γ).Source: §A.3 · p. 40, §A.3.1 · p. 40

A.3.1 ρ-ADMISSIBLE STRATEGY RESOLUTION

  • Theorem 2 uses a ρ-admissible partition chosen after fixing ρ, so its capacity claim is not about arbitrarily fine partitions.The resolution remains fixed during optimization.Source: §A.3.1 · p. 40

A.3.2 AN ADIABATIC KL BOUND AT THE CHOSEN RESOLUTION

  • Within the adiabatic window, the strategy-level divergence from its initial distribution is O(η²|τ|²), a finite-time consequence of scale separation.The strategy-level and trajectory-level divergences are exactly equal.Source: §A.3.2 · pp. 41–42

A.3.4 COLLISION-AWARE FANO STEP

  • Under γ-separation, matching a strategy’s terminal answers to known ground-truth answers gives a query decoder with error at most 1−Pacc,k+γ.The resulting mutual-information bound uses Fano’s inequality.Source: §A.3 · p. 40, §A.3.4 · p. 43

A.3.5 COMBINING THE TWO INFORMATION REQUIREMENTS

  • Combining the entropy and information requirements yields a nonvacuous strategy-capacity lower bound in the adiabatic regime; concentration removes about log mϵ entropy while only O(η²|τ|²) can be recovered.The paper links this mismatch to the non-collapse regime becoming unsustainable and accuracy declining.Source: §A.3.5 · p. 43

B.1 REQUIREMENTS FOR A HIGH QUALITY METRICS

  • A proposed high-quality monitor should track the theory’s evolutionary tendency while remaining easy to compute and acquire without GPU-memory pressure.Source: §B.1 · p. 44

B.2 THE CONVERGENCE OF GRADIENTS; B.3 HEURISTIC INSIGHT; B.4 THE FORMAL DEFINITION OF MEI AND ITS CONVERGENCE

  • As strategy mass concentrates, sampled parameter-space gradients also concentrate, and the heuristic alignment index approaches the rollout count G as trajectories align.MEI formalizes a low-dimensional logit-gradient monitor that is easy to acquire and compute; its convergence to the parameter-space signal is supported under a stable mapping.Source: §B.2 · p. 44, §B.3 · p. 45, §B.4 · pp. 45–46

B.4.1 PROOF OF SUPPORTING LEMMAS 1 & 2

The supporting lemmas bound the Jacobians of Transformer components and combine those bounds through the model’s layer composition. This supplies a capped Jacobian condition for the argument relating parameter-space and logit-space gradients.

B.4.1 PROOF OF SUPPORTING LEMMAS 1 & 2

  • The proof treats Transformer layers as continuous mappings and bounds the Jacobian contributions of residual connections, attention, normalization, and feed-forward layers.Source: §B.4.1 · pp. 46–47
  • Bounded attention kernels and inputs are required for the attention-head bound, while the layer-component bounds together cap a Transformer layer’s Jacobian.Source: §B.4.1 · p. 47
  • The proof composes the layer Jacobians by the chain rule to obtain an upper limit on the model Jacobian.Source: §B.4.1 · pp. 46–47

C.1 PROOF FOR THEOREM 3

This section analyzes strategy-level RLVR dynamics, including how reverse KL, forward KL, and JS regularization can fail to prevent strategy concentration and the full-support equilibrium condition for strategy preservation. It also defines strategy-balancing dynamics and describes two coach-prompting harnesses tailored to different model and task settings.

C.1.2 STEP 2: WHY FORWARD KL DIVERGENCES FAILS

  • The proof identifies forward KL as vulnerable to excessive rewards.Source: §C.1 · p. 48

C.1.1 STEP 1: WHY DOES REVERSE KL DIVERGENCE FAIL

  • The analysis finds that reverse KL yields a Gibbs-form optimum that can become near one-hot.The Gibbs distribution follows from the strategy-level objective; the resulting concentration is described as near-one-hot.Source: §C.1.1 · p. 49

C.1.3 STEP 3: WHY JS DIVERGENCE FAILS

  • JS regularization entangles global normalization with the denominator, and a fixed JS regularizer cannot prevent near-one-hot concentration when reward gaps are sufficiently large.Source: §C.1.3 · pp. 50–51

C.2.1 LOSS SHAPING

  • Strategy-balancing loss uses a softmax distribution over reference-centered scores, analyzed under a deterministic continuous-time approximation with fixed reference, score Jacobians, and mean GRPO drift.The approximation excludes sampling noise and feature drift.Source: §C.2.1 · p. 52

C.2.2 CROSS-PROMPT DYNAMICS AND THE GLOBAL ATTRACTOR

  • When a full-support equilibrium exists, every trajectory initialized from finite logits is attracted to full-support equilibria, preserving represented strategies; existence depends on the dataset and regularization strength, with no universal guarantee.Source: §C.2.2 · pp. 52, 55

D.1 COACH PROMPTING HARNESS

  • The prompting harness instructs competition coaches to give exactly four distinct strategic approaches, without calculations or step-by-step solutions, for mathematics and GPQA tasks.Source: §D.1 · p. 56
  • For non-reasoning models or hard tasks, the harness uses detailed instructions because the task may be too demanding for precise adherence to coach guidance.The prescribed format separates deep thinking, including method identification and step-by-step derivation, from a brief final answer.Source: §D.1 · pp. 57–58
  • The anti-hacking harness targets reasoning models or easy tasks, because detailed coach instructions can discourage independent reasoning and cause severe hacking.It injects strategy information while preserving the model’s reasoning agency.Source: §D.1 · p. 58
  • The anti-hacking prompt labels the suggested method untrusted and requires checks of applicability, predicted mechanism, and the strongest competing answer before committing.The strategy suggestions include mechanism-based analysis, evaluation against experimental evidence, causal pathway construction, and elimination of alternative hypotheses.Source: §D.1 · p. 58

D.2 EXPERIMENT CONFIGURATIONS

The appendix specifies Mesh Learning’s training setup, benchmark-dependent generation lengths, and the sequence-length conditions used to align experiments with Theorem 1.

D.2 EXPERIMENT CONFIGURATIONS

  • Unless otherwise specified, all experiments use a training batch size of 32, learning rate 10^-6, sampling temperature of 1.0, and 16 rollouts per query.Incorrect responses that satisfy the required ANSWER format receive a 0.1 format reward.Source: §D.2 · pp. 58–59
  • Mesh Learning uses clip ratio 0.2; DAPO uses maximum clipping ratio 0.28, while GSPO uses bounds 0.0003 and 0.0004.The appendix says DAPO and GSPO clipping configurations follow their original papers; rollout generation uses SGLang.Source: §D.2 · p. 59
  • Maximum generation lengths vary by model and benchmark, with experiments testing 1024, 4096, and 8192 tokens.The authors state that their chosen scales satisfy 1 ≪ |τ| ≪ η^-1, aligning the experiments with Theorem 1’s assumptions.Source: §D.2 · p. 59, Table 5 · p. 59

E APPENDIX E: CASE STUDY

The appendix uses representative reasoning trajectories to compare collapsed and non-collapsed behavior, focusing on two recurring microscopic effects of Mesh Learning.

E APPENDIX E: CASE STUDY

  • The case studies examine two effects: preserving strategy capacity across distinct approaches and pruning within each strategy to improve reasoning efficiency.They are representative examples rather than an exhaustive analysis of trajectories.Source: Table 6 · p. 60, E APPENDIX E: CASE STUDY · p. 60
  • The comparisons cover collapsed versus non-collapsed trajectories, baseline versus Mesh Learning, and behavior before versus after training.The examples focus on two recurring phenomena summarized in Table 6.Source: E APPENDIX E: CASE STUDY · p. 60
  • The case study uses one problem from MATH and one from GPQA to illustrate the trajectory comparisons.The MATH question asks for the eighth term of a geometric sequence; the GPQA question concerns product-particle kinetic energy.Source: E APPENDIX E: CASE STUDY · p. 60

E.1 MATH CASE STUDY

The MATH case study contrasts baseline reasoning with Mesh Learning on a geometric-sequence problem, showing distinct solution approaches and a converged distribution that retains both strategies.

E.1 MATH CASE STUDY

  • The base model uses many explanatory tokens and mixes two valid solutions, producing a wrong result.The authors characterize its trajectory entropy as relatively low because generated word tokens contribute little entropy.Source: §E.1 · p. 60
  • During warm-up, Mesh Learning elicits explicit step-by-step calculation, raises trajectory entropy, and no longer mixes the two strategies.The case study presents this as a contrast with the base model’s longer explanation and mixed solutions.Source: §E.1 · pp. 60–61
  • The geometric-sequence example illustrates two approaches: using the power-law formula and brute calculation.The strategy descriptions both work from the common ratio to obtain the eighth term, although the supplied passages do not consistently render the final fraction.Source: §E.1 · pp. 60–61
  • After convergence, the 10 evaluation rollouts show a 30% Strategy I and 70% Strategy II distribution, with within-strategy token-level semantic similarity exceeding 95%.The authors describe the rationales as decoupled while each sub-policy has crystallized into a deterministic response template.Source: §E.1 · p. 63

E.1.1 SCIENCE CASE STUDY

This science case study contrasts Mesh Learning and Normal RLVR reasoning trajectories in terms of coherence, focus, and termination, while examining a pion-decay problem whose correct answer is option B. It also states the neutrino and muon energies and explains their difference through equal momentum and the neutrino’s lower mass.

Mesh Learning

  • Mesh Learning produces organized reasoning that combines conservation laws and the relativistic energy–momentum relation, with productive local self-correction and natural termination at the correct answer.Source: §E.1.1 · p. 65

Normal RLVR

  • Normal RLVR reasoning meanders through unnecessary branches, partially infers the answer from the options, and introduces erroneous formulas and numerical inaccuracies before recovering the correct answer through repeated correction.The trajectory is less efficient and less reliable than the coherent solution.Source: §E.1.1 · pp. 68–69

Collapsed

  • The collapsed trajectory repeatedly reaches the correct solution but fails to terminate, cycling through redundant derivations and peripheral considerations without adding information.It retains local problem-solving ability but cannot maintain a stable global reasoning trajectory.Source: §E.1.1 · p. 72

E.1.1 SCIENCE CASE STUDY

  • Option B gives the decay-product kinetic energies as about 4.12 MeV for the muon and 29.8 MeV for the neutrino.The neutrino’s energy is about 29.78 MeV, and the relativistic muon kinetic energy is about 4.12 MeV.Source: §E.1.1 · pp. 78, 81
  • Because the products have equal momentum, the lighter neutrino receives more kinetic energy than the muon.The passage notes that the 4.12 MeV muon value reflects a relativistic calculation rather than the nonrelativistic approximation.Source: §E.1.1 · pp. 78, 81

Related papers

Loading 2610.02835v1…