Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

1 Introduction

• Across all 7 studied RL algorithms and 10 experimental models, RL-induced updates form subnetworks as little as 5% of model parameters in some cases, without explicit sparsity-promoting regularization.The remaining parameters are effectively unchanged.Source: §1 · pp. 1–2
• Subnetworks show consistency across random seeds, training data, and RL algorithms, and isolated subnetwork finetuning converges to an effectively identical model.The authors suggest this may support more efficient RL training methods that leverage update sparsity.Source: §1 · pp. 2–3
• Finetuning only the identified subnetwork produces a model nearly identical to full finetuning in test accuracy and parameter values.The comparison uses the same data, hyperparameters, and number of gradient updates while masking gradients outside the subnetwork.Source: §1 · p. 2
• In PRIME, 20% of parameters are consistently updated, 8% have gradients that cancel, and 72% are never updated.The consistently updated parameters constitute the subnetwork.Source: Figure 2 · p. 2, §1 · p. 2
• Updates are not concentrated in particular transformer layers; nearly all parameter matrices receive similarly sparse updates, while layer normalization receives little to no updates.Despite being sparse, updates to almost all parameter matrices are nearly full-rank and span almost their full representable subspaces.Source: §1 · pp. 2–3
• Training on data near the policy distribution is a primary factor associated with update sparsity; KL regularization and gradient clipping have limited impact.The paper contrasts near-policy RL data with distribution shifts that often occur in SFT.Source: §1 · p. 3

2.1 Related Work

• Unlike the Lottery Ticket Hypothesis, this work studies naturally emerging subnetworks during pretrained-LLM finetuning and reports recovery of both performance and nearly the same model.The Lottery Ticket Hypothesis studies sparse subnetworks identified by pruning and focuses on models trained from scratch.Source: §2.1 · p. 3
• Sparse-training and LoRA approaches explicitly constrain updates, whereas naturally emerging subnetworks can match or surpass full-model finetuning despite nearly full-rank updates.LoRA constrains updates to a small number of parameters.Source: §2.1 · p. 3

2.2 Background

• Update sparsity measures the fraction of parameter changes that are zero; a sparse update does not imply a sparse finetuned model.The background notes that the initial model may be dense.Source: §2.2 · p. 3
• Table 1 covers update sparsity across open models and RL algorithms, with at least 68.5% of parameters unchanged after RL.Source: Table 1 · p. 4
• In-distribution training means training on data closely matching the current policy; on-policy RL and SFT on subsequent RL data are examples.The paper identifies in-distribution training as a primary reason for RL update sparsity.Source: §2.2 · p. 4
• KL regularization constrains policy shifts and gradient clipping bounds update norms, but both have limited impact on update sparsity.Source: §2.2 · p. 4

Setup

• The study measures checkpoint-to-checkpoint update sparsity across publicly released models, including both RL and SFT stages.The analyzed models include Tulu, Eurus, DeepSeek Math, and KTO/SimPO models.Source: §3 · p. 4

Results

• 68.5%–96.0% of parameters remain unchanged after RL across the studied models, while SFT produces dense updates with only 6%-15% sparsity.The RL sparsity emerges without sparsity-promoting regularization or architectural constraints.Source: §3 · pp. 4–5

Layerwise and per-matrix sparsity

• Updates are distributed with similar sparsity across layers and parameter matrices; layer normalization is the exception, receiving little to no updates.Figure 3 compares layerwise and per-matrix sparsity for DPO and PRIME.Source: Figure 3 · p. 5, §3 · p. 5

Update rank

• RL update matrices are nearly full-rank despite sparse updates, indicating selected parameters span almost the full subspaces the matrices can represent.Table 2 reports mean update-matrix ranks as percentages of maximum possible rank.Source: §3 · p. 5, Table 2 · p. 5

Training loss

• For DPO, training the subnetwork in isolation consistently produces lower training loss than full finetuning.Source: Figure 4 · p. 6

Results

• Subnetwork-only finetuning can achieve better test performance than full finetuning for DPO and PRIME.Table 3 reports test-set performance for θfull and θsub; MATH500 results are divided by difficulty level.Source: Table 3 · p. 6

Setup

• The comparison evaluates full and subnetwork-only finetuning with DPO and PRIME.Table 3 reports test-set performance for both models.Source: Table 3 · p. 6

• The study compares one-sided overlap against a uniform random-selection baseline, varying the seed, dataset, or seed, data, and algorithm together.Source: §5 · p. 7
• Changing the random seed produced 60.5% and 60.6% one-sided overlap; changing seed, data, and RL algorithm together still yielded 59.1% and 33.2%.Source: §5 · p. 7
• Across seeds, datasets, and algorithms, observed subnetwork overlap was substantially higher than random guessing, indicating partial transferability across settings.Source: §5 · p. 7
• Because overlap falls short of 100%, partial reuse may reduce redundant computation in repeated RL runs such as hyperparameter sweeps or ablations.Source: §5 · p. 7
• Part of a cheaper DPO subnetwork could be reused in costlier PPO training without sacrificing performance.Source: §5 · p. 8

• Training-data proximity to the policy distribution is the dominant factor associated with update sparsity, while total gradient updates also matter.Source: §6 · p. 8
• GRPO with KL regularization reached 69.8% sparsity versus 68.8% without it, and sparse SimPO updates further indicate KL is not a primary driver.Source: §6 · p. 8
• DeepSeek-R1-Zero exhibits high update sparsity despite forgoing prior SFT, suggesting prior SFT is not required for the phenomenon.Source: §6 · p. 8
• PRIME update sparsity slowly decays toward 80%; DeepSeek-R1-Zero remains 86% sparse after 8K steps, over 20× PRIME’s training steps.Source: Figure 5 · p. 8, §6 · pp. 8–9
• About 8.5% of parameters ever updated fall outside the final subnetwork; their share rises early, then declines as some updates cancel out.Source: Figure 6 · p. 8, §6 · p. 9
• The authors note the possibility that the studied models are undertrained and that update sparsity could diminish with substantially more training.Source: §6 · p. 9
• In-distribution SFT yielded around 90.0% update sparsity, RAFT++ yielded 69.4%, and out-of-distribution DPO yielded 6.8% update sparsity.Source: §6 · p. 9, Table 5 · p. 9

• Because RL is computationally demanding, experiments vary one factor at a time, so interactions among factors may contribute to the observed sparsity.Source: §7 · p. 9
• Fully controlled experiments are computationally prohibitive, leading the authors sometimes to use public checkpoints.Source: §7 · p. 9
• The experiments focus on language models; whether the same questions apply to multimodal and diffusion models remains open.Source: §7 · p. 9
• Confounders remain, and their causes are nontrivial and warrant further investigation; the authors also call for theoretical explanations of update sparsity.Source: §7 · pp. 9–10

• RL finetuning updates approximately 5%-30% of LLM parameters, and finetuning that subnetwork alone reproduces full-model performance while closely matching its parameter values.Source: §8 · p. 10

NeurIPS Paper Checklist

• The checklist says the paper presents empirical results rather than theoretical proofs and assumptions.It identifies the results as mostly empirical and marks the theory-assumptions-and-proofs item not applicable.Source: NeurIPS Paper Checklist · p. 16
• The checklist reports that datasets, models, hyperparameters, training and test details, and compute-resource information are provided for reproducibility.It also marks the main experimental results reproducible and says most experiments use publicly released models, datasets, and codebases.Source: NeurIPS Paper Checklist · pp. 16–17
• The checklist marks statistical-significance reporting as absent, explaining that most accuracies are measured on deterministic setups.The checklist answers [No] to reporting suitably defined error bars or other statistical-significance information.Source: NeurIPS Paper Checklist · p. 17
• The checklist marks societal-impact and high-risk-release safeguards questions not applicable, citing no immediate societal impacts and no such risks.It also says the work does not involve crowdsourcing or human-subject research.Source: NeurIPS Paper Checklist · pp. 17–18

A Appendix

• Table 6 is captioned as reporting parameter-update sparsity percentages across thresholds, RL algorithms, and RL checkpoints.The caption identifies the table's scope but does not provide any cell values.Source: Table 6 · p. 19

B Hyperparameter choices for Gradient Masking experiments

• DPO finetunes LLaMA-3.1-Tulu-3-8B on the preference-mixture dataset for one epoch, using a peak learning rate of 5 × 10−7 and effective batch size 128.Training uses bfloat16, sequence length 2048, and DeepSpeed Stage 3 across 8 processes.Source: B Hyperparameter choices for Gradient… · p. 19
• PRIME finetunes Qwen2.5-Math-7B on GSM8K and MATH for 15 epochs, with batch size 64 and four rollouts per sample.The actor learning rate is 5 × 10−7, the reward-model learning rate is 1 × 10−6, gradient clipping is 10.0, and temperature β is 0.05.Source: B Hyperparameter choices for Gradient… · p. 19

C Model Checkpoints: SFT vs RL sparsity comparison

• The SFT-versus-base comparisons pair Llama-3.1 8B and 70B models, and Qwen2.5-Math-7B with Eurus-2-7B-SFT.The listed pairs are meta-llama/Llama-3.1-8B versus allenai/Llama-3.1-Tulu-3-8B-SFT; the corresponding 70B pair; and Qwen/Qwen2.5-Math-7B versus PRIME-RL/Eurus-2-7B-SFT.Source: C Model Checkpoints: SFT vs RL… · p. 19
• The RL-checkpoint comparisons pair Tulu-3 8B and 70B SFT checkpoints with their DPO checkpoints, and Eurus-2-7B-SFT with Eurus-2-7B-PRIME.These are the listed SFT-to-RL checkpoint pairs for the sparsity comparison.Source: C Model Checkpoints: SFT vs RL… · p. 19

D Training Dynamics

• The section asks whether sparsity decays with training or converges to a non-zero level.It frames this as examining how sparsity evolves with training time.Source: D Training Dynamics · pp. 19–20
• The analysis tracks sparsity from the base model to each of 21 checkpoints and between pairs of checkpoints.It denotes checkpoint parameters θ1 through θ21, with θinit as the corresponding base-model parameters.Source: D Training Dynamics · p. 20
• On average, only 7% of weights receive a non-zero gradient update between successive PRIME checkpoints, while checkpoint-to-base sparsity converges to a numerically significant asymptote.The analysis covers intermediate checkpoints and their MATH500 and AIME2024 accuracy; all checkpoints show non-trivial sparsity.Source: D Training Dynamics · p. 20, Figure 7 · p. 20

E Random Guessing baseline

• For two models with sparsities s1 and s2, the random-guessing expected overlap is (1−s1)·(1−s2), the product of their update densities.The passage explains that this equals the density of model 2 multiplied by model 1's density.Source: E Random Guessing baseline · p. 20