PaperlayerView exact revision on arXiv

Source-linked AI summary

Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

Zijian Zhang, Rizhen Hu, Athanasios Glentis, Dawei Li, Chung-Yiu Yau, Hongzhou Lin, Mingyi Hong

arXiv:2607.01232v2cs.LGcs.CL

TL;DR

It remains unclear where RL gains emerge within transformer networks because standard post-training updates all layers jointly. This paper systematically trains layers in isolation and finds that gains are concentrated in middle layers, with layer-aware strategies outperforming uniform RL training.

  • Problem

    Existing RL post-training updates all transformer layers jointly, leaving limited evidence about where within the network RL gains emerge.

  • Method

    The paper conducts systematic single-layer RL training across models, algorithms, and tasks, introducing layer contribution to quantify each layer’s recovered RL improvement.

  • Results

    Up to 114% of full-parameter RL gains are recovered by individual layers, with high-contribution layers consistently concentrated in the transformer stack’s middle.

  • Takeaways & Limitations

    Layer-aware training that prioritizes high-contribution or middle layers consistently outperforms standard uniform RL training within the evaluated scope.

  • Takeaways & Limitations

    Guided layer-aware training strategies were validated only on mathematical reasoning, while their extension to coding and agentic tasks remains future work.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a central component of post-training large language models (LLMs), yet little is understood about how RL adaptation is distributed across transformer layers. Existing approaches typically update all model parameters uniformly, implicitly assuming that every layer contributes similarly to the gains obtained during RL post-training. In this work, we challenge this assumption through a systematic layer-wise study of RL training. Surprisingly, we find that training a single transformer layer can recover most of the gains achieved by full-parameter RL training, and in some cases even surpass it. To quantify this phenomenon, we introduce the quantity layer contribution, which measures the fraction of full RL improvement recovered by training a layer in isolation. Across seven models spanning two model families (Qwen3, Qwen2.5), three RL algorithms (GRPO, GiGPO, Dr. GRPO), and multiple task domains including mathematical reasoning, code generation, and agentic decision-making, we observe a remarkably stable pattern: RL gains are highly concentrated in a small subset of, and in many cases even a single, transformer layers. More strikingly, the same structural pattern consistently emerges: high-contribution layers concentrate in the middle of the transformer stack, while layers near the input and output ends contribute substantially less. The resulting layer rankings remain strongly correlated across datasets, tasks, model families, and RL algorithms.

1 Introduction

This study shows that RL post-training gains are concentrated in a small, stable subset of transformer layers rather than distributed uniformly across the network. Training one layer can recover most full-parameter RL gains, with layer contributions varying systematically across models and tasks.

  • Findings: Single-layer RL can often recover most full-parameter RL gains and sometimes outperform full-parameter training.This challenges the intuition that RL improvements require coordinated adaptation across the entire network.
  • Method: The study independently trains each transformer layer with RL while freezing all others, then compares its improvement with standard full-parameter RL training.It introduces layer contribution as the fraction of full RL improvement recovered by training a layer in isolation.
  • Findings: 114% is the maximum gain recovered by the best individual layers, while the weakest layers recover less than 30% of full-parameter RL gains.These results show that layer contributions vary dramatically across the network.
  • Findings: High-contribution layers consistently concentrate in the middle of transformer networks across seven models, two model families, three RL algorithms, and multiple task domains.The studied families are Qwen3 and Qwen2.5, while the algorithms include GRPO, GiGPO, and Dr. GRPO.
  • Implications: The concentration of RL gains reveals a structural property that can be exploited algorithmically through layer-aware training.The authors develop simple layer-aware training methods guided by layer contribution.

2 Preliminaries

RLVR optimizes language-model policies using binary correctness rewards, while GRPO estimates advantages from sampled response groups without a learned value network. The paper isolates individual transformer-layer updates and defines layer contribution to measure recovered RL improvement relative to full-parameter training.

  • RLVR trains a policy πθ by maximizing expected reward from objectively verifiable answers, using a binary correctness signal for each generated response.
  • GRPO samples G responses per prompt and computes group-normalized advantages without using a learned value network.
  • GRPO updates the policy with a clipped surrogate objective using an importance-sampling ratio, KL penalty coefficient β, and fixed initial reference policy πref.
  • The layerwise framework updates one target transformer layer at a time while gradients are backpropagated through the full network and all other parameters remain frozen.
  • Layer contribution compares a layer-trained model’s in-domain score with the pretrained baseline and full-parameter GRPO score to quantify recovered RL improvement.
  • A contribution of 1.0 matches full-parameter gains, values above 1.0 surpass them, and values near 0 indicate negligible RL improvement.

3 Measuring Layer Contribution in RLVR

Across seven models, RLVR gains are unevenly distributed across layers: middle layers consistently contribute most, and the best single layer matches or surpasses full-parameter training. Layer rankings remain consistent across datasets, tasks, model families, and RL algorithms, indicating a stable contribution pattern.

  • Qwen3 results: On Qwen3-1.7B-Base, contribution ranges from 0.28 (Layer 24) to 1.14 (Layer 10), with 5 out of 28 layers exceeding 1.0.Seven layers fall below 0.5, and the results suggest a single layer can capture the entirety of full-training gains.
  • Qwen3 results: On Qwen3-8B-Base, Layer 16 reaches C = 1.07, whereas Layer 0 has a negative contribution of C = −0.51.The negative value indicates that training Layer 0 in isolation degrades math performance below the base model.
  • Layer-wise contribution pattern: Middle layers consistently exhibit higher contribution, while layers near the input and output ends contribute substantially less.This structure appears across Qwen3 and Qwen2.5 experiments, including distilled models.
  • Cross-dataset and cross-task consistency: Layer rankings remain correlated across math and code, with Spearman ρ = 0.59, p < 0.001, indicating that the same layers tend to contribute most.The findings support layer contribution as an intrinsic property of pretrained weights rather than a property determined by specific training data or tasks.
  • Summary across all seven models: Across all seven models, the best single layer matches or surpasses full-parameter training, with C ≥1.0.The models span two families, three RL algorithms, multiple datasets, three task domains, and scales from 1.5B to 8B parameters.

4 Guiding Full-Parameter RLVR by Layer Contribution

Layer contribution can guide full-parameter RLVR by prioritizing high-contribution layers, yielding better performance than uniform or low-contribution emphasis. Selective training and a profiling-free middle-layer heuristic can also surpass the full-parameter baseline while retaining substantial RL gains.

  • Contribution-guided learning rates: 66.43 ± 0.40 (+0.99) demonstrates that boosting high-contribution layers improves math performance, whereas boosting the lowest-contribution layers consistently decreases performance.High-contribution layers receive 1 × 10−5 learning rates; other layers use 5 × 10−6, with k ∈ {5, 10}.
  • Selective layer training: Only B5 reaches 65.87 ± 0.70 (+2.90 over baseline) on Qwen3-4B-Base, representing 27% of the total RL gain while training only five layers.On Qwen3-1.7B-Base, Only B5 and Only B10 also exceed the full-parameter baseline: 51.53 ± 0.24 and 51.39 ± 0.46 versus 50.82 ± 0.40.
  • Selective layer training: Only B10 reaches 69.11 ± 0.10 (+2.68) on Qwen3-8B-Base, representing 32% of the total RL gain and surpassing full-parameter training.The experiments train only the best k layers and freeze all remaining layers, with k ∈ {5, 10}.
  • Profiling-free middle-layer heuristic: 51.35 ± 0.28 from the middle-five-layer heuristic on Qwen3-1.7B-Base surpasses the full-parameter baseline of 50.82 ± 0.40 without per-layer profiling.The heuristic selects layers by position, based on the observed concentration of contribution in the transformer’s middle layers.
  • Profiling-free middle-layer heuristic: The middle-layer heuristic achieves a +1.76 improvement, representing 21% of the total RL gain, and captures a meaningful portion of contribution-guided selection’s benefit.Across all three model scales, heuristic selection surpasses the full-parameter baseline without per-layer profiling.

5 Discussion

High-contribution layers learn complementary problem-solving abilities rather than identical solutions, enabling their ensemble to outperform individual layers and full-parameter training. Weight-change analyses further show that layer contribution is not explained by parameter-update magnitude, while the framework is primarily an analysis tool.

  • Complementarity: High-contribution layers solve different subsets of OlympiadBench problems, indicating complementary abilities among similarly performing layer-trained models.For Qwen3-1.7B-Base, the top-7 layers each achieve 23%–28% accuracy, motivating comparison of their newly solved problem sets.
  • Complementarity: 33.6 ± 0.91% ensemble accuracy surpasses the best individual layer-trained model at 28.3 ± 0.25% and the full-parameter baseline at 26.9 ± 0.40%.The top-7 layer-trained models are aggregated by majority voting on OlympiadBench.
  • Scope and implications: The layer-wise training and majority-voting framework is intended as an analysis tool rather than a practical training strategy because it requires independently training multiple layer-specific models.The analysis nevertheless provides insight into representation diversity and layer contributions during RL post-training.
  • Weight changes: Under full-parameter training, per-layer weight changes range approximately from 0.5 to 0.8 despite the highly non-uniform layer-contribution profile.Middle layers do not change more than other layers during full training, yet contribute far more when trained in isolation.
  • Weight changes: Under single-layer training, weight changes are approximately 0.8–1.0 versus 0.5–0.7 during full training, but remain similar across layers with different contribution values.This suggests that a single layer compensates by moving farther in parameter space when absorbing all RL-induced improvement.

6 Related Work

Related work has examined layer importance through pruning and identified layers whose removal severely harms performance. This work complements those studies by framing single-layer training as a structured way to explore diverse regions of pretrained weight space.

  • Layer Importance in LLMs: Prior studies assess LLM layer importance by pruning, finding that some layers are critical while others can be removed with little effect.Zhang et al. identify “cornerstone layers” whose removal reduces performance to near-random guessing.
  • Layer Importance in LLMs: Nepal et al. show that critical layers for mathematical reasoning are determined during pretraining.
  • Diverse Solutions in Weight Space: Single-layer training offers a structured way to explore pretrained weight space, with different layers accessing different solution-space regions.Layer-trained models exhibit low pairwise agreement, providing evidence of diversity among the resulting models.

7 Conclusion · Appendix

The conclusion finds that RL adaptation is concentrated in middle transformer layers and that prioritizing these layers can outperform uniform full-parameter training. It also identifies task-coverage and metric-definition limits, while leaving the mechanism behind middle-layer importance unresolved.

  • 7 Conclusion: Single-layer experiments across seven models, two model families, three RL algorithms, and two task domains reveal large differences in layers’ capacity to capture RL improvement.The passage identifies Qwen3 and Qwen2.5, plus mathematical reasoning and agentic decision-making.
  • 7 Conclusion: High-contribution layers consistently concentrate in the middle of the network.
  • 7 Conclusion: Prioritizing high-contribution layers with adaptive learning rates or selective training consistently outperforms standard uniform RL.
  • 7 Conclusion: Training only middle layers surpasses full-parameter training even without per-layer profiling.This is presented as a simple heuristic for practical training.
  • 7 Conclusion: The guided training strategies have been validated only on mathematical reasoning, leaving coding and agentic-task extensions for future work.
  • 7 Conclusion: The layer contribution metric depends on a specific training configuration, and the reason middle layers matter disproportionately remains theoretically unresolved.

A Training Details and Hyperparameters … A.5 DeepSeek-Distilled-Qwen-7B (GRPO, Skywork-OR1)

The appendix details training configurations across the paper’s model families, RL algorithms, and task settings. Single-layer training generally freezes all parameters except one decoder layer and follows the corresponding full-parameter baseline hyperparameters unless otherwise noted.

  • A.1 Overview: Single-layer training freezes every parameter except the target decoder layer, including the embedding layer and language model head.It uses the same hyperparameters as the corresponding full-parameter baseline unless otherwise noted.
  • A.2 Qwen3 Models (GRPO, NuminaMath-CoT): Qwen3 experiments use veRL with GRPO and AdamW on Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen3-8B-Base.The models have 28, 36, and 36 layers, respectively.
  • A.2 Qwen3 Models (GRPO, NuminaMath-CoT): Qwen3 single-layer experiments independently train each layer on 50K decontaminated NuminaMath-CoT problems, while the full-parameter baseline unfreezes all layers.Full-parameter learning rates are tuned over {1 × 10−6, 3 × 10−6, 5 × 10−6, 1 × 10−5}, with the best result reported at 5 × 10−6; single-layer training also uses 5 × 10−6.
  • A.3 Qwen2.5-Math-1.5B (Dr. GRPO): Qwen2.5-Math-1.5B is trained with unbiased Dr. GRPO on MATH using binary answer-matching rewards verified by Math-Verify.The objective removes response-length and question-level normalization terms from GRPO, and one decoder layer is trained at a time.
  • A.3 Qwen2.5-Math-1.5B (Dr. GRPO): Layer contribution C(k) is evaluated using the unweighted Avg6 across AIME 2024, AIME 2025, AMC, MATH500, Minerva Math, and OlympiadBench.This benchmark suite follows the Dr. GRPO setup and differs from the Qwen3 suite.
  • A.4 Qwen2.5-Instruct Models (GiGPO, ALFWorld): Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct are trained with GiGPO on ALFWorld, which contains 2,435 household tasks across six categories.The reward is binary: 1 for successful task completion and 0 otherwise.
  • A.5 DeepSeek-Distilled-Qwen-7B (GRPO, Skywork-OR1): DeepSeek-R1-Distill-Qwen-7B, a 28-layer model distilled from DeepSeek-R1 into the Qwen architecture, is trained with GRPO on approximately 48K Skywork-OR1 mathematics problems.The reward is binary answer-matching; full-parameter learning rates are tuned over {1 × 10−6, 3 × 10−6, 5 × 10−6, 1 × 10−5}, with the best result at 5 × 10−6, while single-layer training uses 5 × 10−6.

A.6 Training Datasets

The study uses a downsampled and decontaminated NuminaMath-CoT dataset as its primary training source, alongside curated datasets for cross-dataset, cross-task, model-specific, and agentic validation. These datasets cover mathematical reasoning, code generation, and interactive household tasks with binary rewards.

  • NuminaMath-CoT: 860K competition-level math problems are downsampled to 50K for primary training from NuminaMath-CoT.The dataset includes chain-of-thought solutions and undergoes strict decontamination filtering to prevent test-set leakage.
  • DeepScaleR: 40K reasoning-intensive mathematics problems from AIME, AMC, and other archives support cross-dataset validation.DeepScaleR uses the same binary answer-matching reward.
  • DeepCoder: 24K programming problems from LiveCodeBench and Codeforces support cross-task validation.DeepCoder rewards generated code with 1 when it passes all test cases and 0 otherwise.
  • Skywork-OR1: 48K mathematics problems from Skywork-OR1 are used to train DeepSeek-Distilled-Qwen-7B.The dataset uses the same binary answer-matching reward.
  • ALFWorld: 2,435 household tasks across six categories form the ALFWorld training benchmark for Qwen2.5-Instruct models.Agents interact with a simulated environment through text commands, receiving 1 for successful completion and 0 otherwise.

A.7 Learning Rate Ablation

A learning-rate ablation shows that layer contribution rankings are robust rather than artifacts of the default learning rate. Under a 3× higher learning rate, low-contribution layers remain low-contribution and high-contribution layers retain their advantage.

  • A.7 Learning Rate Ablation: The ablation tests whether contribution rankings change when individually retraining selected layers with a higher learning rate.The study evaluates the top-5 and bottom-5 layers on Qwen3-1.7B-Base while keeping other hyperparameters unchanged.
  • A.7 Learning Rate Ablation: 3× higher learning rate (1.5 × 10−5 vs. the default 5 × 10−6) leaves the bottom-5 layers low-contribution, with their C values changing by at most 0.02.None of the bottom-5 layers approaches the high-contribution group.
  • A.7 Learning Rate Ablation: The contribution profile is robust to learning-rate choice, supporting the conclusion that layer contribution is an intrinsic property of the model.The result indicates that the observed rankings are not caused by suboptimal default hyperparameters.
  • A.7 Learning Rate Ablation: The top-5 layers retain their high-contribution status under the boosted learning rate, while the bottom-5 layers remain low-contribution.The table compares contribution C under the default and 3× boosted learning rates.

B Benchmark Selection Criteria

The evaluation covers 12 benchmarks across four categories, including in-distribution mathematics and out-of-distribution code, reasoning, and language tasks. Selection spans mathematical reasoning, code generation, scientific and general knowledge, instruction following, and multilingual evaluation.

  • Benchmark Selection Criteria: 12 benchmarks are grouped into four categories for evaluation.The categories include mathematics, out-of-distribution code, out-of-distribution reasoning, and out-of-distribution language.
  • Benchmark Selection Criteria: The mathematics category includes MATH500, GSM8K, OlympiadBench, and AMC.These benchmarks cover competition-level, grade-school, Olympiad-level, and American Mathematics Competition problems; AMC reports Average@32 because its dataset is very small.
  • Benchmark Selection Criteria: The out-of-distribution code category includes HumanEval+, MBPP, and LiveCodeBench.They assess function-level code generation, basic Python programming, and recent competitive programming problems collected after model training cutoff dates.
  • Benchmark Selection Criteria: The out-of-distribution reasoning category includes GPQA-Diamond and MMLU-Pro.GPQA-Diamond contains graduate-level science questions curated by domain experts, while MMLU-Pro provides harder, more discriminative questions than MMLU.
  • Benchmark Selection Criteria: The out-of-distribution language category includes C-Eval, IFEval, and MGSM.These benchmarks cover Chinese subjects, instruction following under formatting and content constraints, and multilingual mathematical reasoning.

C Full Per-Layer Results

This section reports complete per-layer evaluation results across Qwen3 models and Qwen2.5-Math-1.5B. The tables cover 12 Qwen3 benchmarks and 6 Qwen2.5-Math benchmarks, with layer contribution defined on math or average performance.

  • Qwen3 Results: The Qwen3 tables cover all 12 benchmarks at each model scale.The complete results are organized across the three Qwen3 model sizes.
  • Qwen3 Results: Tables 13–15 report full per-layer results for Qwen3-1.7B-Base, Qwen3-4B-Base, and Qwen3-8B-Base.For these tables, C denotes layer contribution on math.
  • Qwen2.5 Results: Table 16 reports full per-layer results for Qwen2.5-Math-1.5B using Dr. GRPO.Here, C denotes layer contribution on Avg.
Loading 2607.01232v2…