Mitigating Capability Degradation in LLM Self-Improvement via Representation Geometry Regularization

1 Introduction

• OPSD uses the same LLM as teacher and student under different conditioning contexts, with privileged information supplied to the teacher.The student responds to the original query; the teacher uses additional information such as verified reasoning traces.Source: §1 · p. 1
• Existing OPSD methods can progressively turn previously correct responses incorrect even while overall performance improves.The degradation occurs across representative methods, though to varying degrees, and may undermine self-improvement stability.Source: §1 · p. 2
• The authors associate capability degradation with shrinking inter-cluster distances and less distinct boundaries in hidden representations.When responses remain correct, representations show more structured clusters, larger distances, and clearer boundaries.Source: §1 · p. 2
• RGR adds intra-cluster compactness and inter-cluster separation constraints to OPSD to mitigate degradation while retaining targeted capability improvements.The authors report experiments across four existing self-distillation models and three benchmarks.Source: §1 · p. 2

2 Related works

• OPSD methods divide into privileged-information-driven and external-feedback-driven approaches, based on self-teacher signal construction and external feedback.Privileged information is unavailable to the student policy at inference time; external-feedback methods incorporate environmental feedback, verification, or outcome rewards.Source: §2 · p. 2
• OPSD uses verified reasoning traces to condition the same model as a self-teacher, while the student is conditioned only on the question.This is the paper’s representative privileged-information-driven method.Source: §2 · p. 2
• SDPO, SRPO, and RLSD use feedback differently, including distillation targets, trajectory-specific updates, and reward-guided token-level policy updates.SDPO uses textual or reward-derived feedback; SRPO routes successful and failed trajectories to different updates; RLSD uses verifiable outcomes and teacher–student discrepancies.Source: §2 · p. 2

3 Preliminaries

• The student generates an on-policy response from the original query, while the teacher conditions on additional privileged information such as a verified reasoning trace.The two roles share model parameters.Source: §3 · p. 3
• OPSD minimizes divergence between teacher and student token distributions along the student’s generated trajectory.The divergence may use forward KL, reverse KL, Jensen–Shannon divergence, or another suitable distribution-matching objective.Source: §3 · p. 3
• Because teacher and student share parameters, optimizing the OPSD objective updates the LLM itself and distills teacher-only information into ordinary query-conditioned behavior.Responses are generated token by token conditioned on the query and previously generated tokens.Source: §3 · pp. 2–3

4.1 Progressive Capability Degradation

• A fixed query’s answer changes from correct at epochs 25 and 50 to incorrect at epoch 75 as representation clusters become less distinct.Across OPSD, SDPO, RLSD, and SRPO, geometry changes consistently coincide with correctness changes; the authors hypothesize that less coherent reasoning is reflected in more entangled representations.Source: §4.1 · p. 3, Figure 2 · p. 3

4.2 Theoretical Analysis for Capability Degradation

• The analysis defines excess distillation risk as the current student’s distillation risk minus the risk achieved by the optimal student.A smaller gap indicates a closer approximation to the optimal student.Source: §4.2 · pp. 3–4
• Theorem 4.1’s excess-risk upper bound is affected by intra-cluster variance and representation deviation from teacher and optimal-teacher representations.Reducing the intra-cluster variance term can tighten the bound.Source: §4.2 · p. 4
• The analysis concludes that preserving capabilities requires teacher alignment as well as compact clusters and clear separation between clusters.The theoretical setup considers an LLM policy trained using OPSD.Source: §4.2 · pp. 3–4

5 Methodology

• RGR targets two complementary properties: compact representations within clusters and sufficient separation between clusters.The method is motivated by blurred cluster boundaries that coincide with capability degradation during OPSD.Source: §5 · p. 4

5.1 Intra-cluster compactness

• The intra-cluster target keeps student cluster compactness close to the teacher’s, avoiding the excessive compactness that direct WCSS minimization may cause.The paper notes that overly similar representations within a cluster can reduce discriminative information.Source: §5.1 · pp. 4–5

5.2 Inter-cluster separation

• Student inter-cluster separation is matched to the teacher’s using low-end Laplacian eigenvalues, avoiding excessive expansion and geometric distortion.The Laplacian is constructed from pairwise relationships among hidden representations.Source: §5.2 · p. 5

5.3 Optimization

• A student–reference Gram-matrix discrepancy provides a unified objective for controlling both geometry properties, alongside the original OPSD and entropy-related objectives.The coefficients α and β weight the geometry and entropy terms, respectively.Source: §5.3 · p. 6

6.1 Experiment setup

• Across Qwen3-1.7B and Qwen3-4B, four self-distillation methods were evaluated on up to 30K OpenThoughts math pairs and AIME24, AIME25, and HMMT25 using Avg@12.The experiments also evaluate performance across 12 generated responses per problem.Source: §6.1 · pp. 6–7, §6.2 · p. 7

6.2 Main results (RQ1)

• RGR generally improves OPSD, SDPO, and SRPO across both Qwen3 models and diverse math benchmarks, though gains vary and RLSD gains are limited.The authors attribute larger gains for OPSD and SDPO to their stronger reliance on dense self-distillation signals.Source: Table 1 · p. 7, §6.2 · p. 7

6.3 Ablation Study (RQ2)

• Full RGR performs best across all three benchmarks; removing geometry consistently hurts more than removing entropy, which drops performance on one benchmark.The study identifies geometry preservation as the primary contributor to improvements, with entropy regularization providing complementary stabilization.Source: Table 2 · p. 7, §6.3 · pp. 7–8

6.4 Training Dynamics Visualization (RQ3)

• Without RGR, degraded queries show shrinking inter-cluster center separation and less distinct boundaries; RGR maintains more compact clusters and clearer boundaries in most cases.The visualizations compare representations across models and training steps with response correctness indicated.Source: Figure 4 · p. 8, §6.4 · pp. 8–9
• RGR eliminates OPSD degradation and reduces degradation ratios for SDPO, RLSD, and SRPO by 14.28%, 7.15%, and 21.43%, respectively.The comparison measures correct-to-incorrect response transitions on AIME24.Source: Figure 3 · p. 8, §6.4 · p. 8

6.5 Cross-Task Forgetting Study (RQ4)

• 5.28→1.94 (SDPO) and 3.06→1.11 (SRPO): RGR reduces AIME24 forgetting gaps after tool-use training, while RLSD rises from 3.05 to 3.39.All four methods show AIME24 performance degradation after sequential tool-use training; the reported forgetting-gap reduction applies to three methods.Source: Table 3 · p. 9, §6.5 · p. 9

7 Conclusion

• RGR reduces correct-to-incorrect transitions while maintaining OPSD’s self-improvement capability across multiple variants and mathematical reasoning benchmarks.The paper proposes future work on more adaptive geometry-preservation strategies and extending the framework beyond self-distillation.Source: §7 · p. 9

A Sensitivity Analysis

• Performance follows a smooth increasing-then-decreasing trend as either α or β varies, suggesting stability across a relatively broad range of settings.β is varied with α fixed at 2.5 × 10−2; α is varied with β fixed at 5 × 10−2.Source: A Sensitivity Analysis · p. 11
• The selected settings are α = 2.5 × 10−2 and β = 5 × 10−2.Figure 5 shows performance under different β values on the left and α values on the right, with the selected setting marked by stars.Source: Figure 5 · p. 11, A Sensitivity Analysis · p. 11

B Implementation Details

• All experiments use LoRA with rank r=64 and α=128, a learning rate of 5 × 10−6, and maximum gradient norm 0.1.The shared hyperparameters are listed in Table 4.Source: B Implementation Details · p. 11, Table 4 · p. 11
• The maximum sequence length is 20,000 tokens, and completions are capped at 1,024 tokens.Source: B Implementation Details · p. 11
• Rollout sampling uses temperature 1.1, top-p 0.95, and top-k 20, with training in BF16 precision.Source: B Implementation Details · p. 11

D Proof of Theorem 4.1

• The proof introduces centered logits and uses Taylor expansion when their L2 norm approaches zero.It uses softmax’s invariance to constant shifts to work with centered logits.Source: D Proof of Theorem 4.1 · pp. 12–13
• For Qwen3, the original logits are Wh, and the centered logits z = AWh induce the same next-token distribution because softmax is invariant to constant shifts.The proof assumes hidden representations are normalized with RMSNorm without learnable scaling parameters.Source: D Proof of Theorem 4.1 · pp. 12–13

E Proof of Theorem 5.1

• The proof’s notation defines |Cref| as the number of clusters in the reference model.Source: E Proof of Theorem 5.1 · p. 14
• Theorem 5.1’s proof treats the all-cluster intra-cluster objective through pairwise-disjoint principal submatrices and Cauchy–Schwarz.Source: E Proof of Theorem 5.1 · p. 14

• The proof invokes the Laplacian matrix definition and the Cauchy–Schwarz inequality.Source: F Proof of Proposition 5.2 · p. 14
• The proof explicitly uses the Cauchy–Schwarz inequality in a subsequent step.Source: F Proof of Proposition 5.2 · p. 14
• Table 5 summarizes the paper’s main notation.Source: Table 5 · p. 15