Source-linked AI summary
Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies
Yuqiao Tan, Minzheng Wang, Shizhu He, Huanxuan Liao, Chengfeng Zhao, Qiunan Lu, Tian Liang, Jun Zhao, Kang Liu
TL;DR
Existing RL methods optimize LLMs as unified policies while overlooking their internal mechanisms. The paper decomposes these policies, analyzes layerwise entropy and internal alignment, and introduces BuPO, whose experiments demonstrate effectiveness on complex reasoning benchmarks.
Problem
Existing RL approaches optimize only the final output distribution of LLMs, overlooking their evolving internal information flow.
Method
The paper decomposes LLM policies into internal layer and modular policies, analyzes their entropy, and optimizes internal layer policies early through BuPO.
Results
Internal policies shift from high-entropy exploration to deterministic refinement; Qwen is progressive while Llama converges abruptly, and BuPO demonstrates effectiveness on complex reasoning benchmarks.
Takeaways & Limitations
Internal policy alignment refines foundational reasoning features in lower layers and guides the overall policy toward more effective reasoning.
Takeaways & Limitations
BuPO requires substantial computational resources and is studied in reasoning-oriented settings with long generation trajectories.
Abstract
from arXiv · showhide
Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a unified policy, overlooking their internal mechanisms. In this paper, we decompose the LLM-based policy into Internal Layer Policies and Internal Modular Policies via the Transformer's residual stream. Our entropy analysis of internal policy reveals distinct patterns: (1) universally, internal policies evolve from high-entropy exploration in early layers to deterministic refinement in the top layers; and (2) Qwen exhibits an explicit progressive reasoning structure, contrasting with the abrupt convergence in Llama. Furthermore, we discover that optimizing internal layers induces feature refinement, forcing lower layers to capture high-level reasoning representations early. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that reconstructs the LLM's reasoning foundation from the bottom up by optimizing internal layers in early stages. Extensive experiments on complex reasoning benchmarks demonstrate the effectiveness of BuPO.
1 Introduction
Existing RL methods optimize only an LLM’s final output distribution, overlooking evolving internal information flow. This paper decomposes internal policies, analyzes their entropy dynamics, and proposes bottom-up optimization to improve reasoning.
- Existing RL methods treat the LLM as a unified policy and optimize only its final output distribution, overlooking evolving internal information flow.
- Internal policies universally shift from high-entropy exploration in early layers to near-deterministic refinement in top layers.
- Qwen progressively reduces uncertainty across layers, whereas Llama converges abruptly near its final layers.
- Internal policy alignment refines lower-layer features, enabling them to capture high-level reasoning capabilities early.
- BuPO optimizes fine-grained internal layer policies early in training to reconstruct foundational reasoning abilities and improve complex-reasoning performance.
- The paper decomposes LLM policies into internal layer and modular policies, revealing distinct roles in reasoning.
2 Preliminary
The preliminary framework models Transformer computation as residual-stream updates through attention and FFN modules, with hidden states projected into vocabulary distributions. Language-model generation is formulated as a token-level MDP optimized with GRPO under sparse rewards.
- 2.1 The Residual Stream in Transformer: A decoder-only Transformer stacks layers containing self-attention and FFN modules connected by residual updates.
- 2.1 The Residual Stream in Transformer: Hidden states are projected with the unembedding matrix into token distributions, enabling internal layer and module policies to be analyzed.
- 2.2 Reinforcement Learning for Language Model Policy: Language-model generation is a token-level MDP in which each state contains the prompt and previously generated tokens.
- 2.2 Reinforcement Learning for Language Model Policy: GRPO samples response groups and uses importance ratios for policy optimization under sparse rewards.
3 Language Model Policy Secretly Contains Internal Policies
The paper treats intermediate hidden states as internal policies whose entropy reveals how reasoning evolves across layers and modules. Across models, reasoning generally moves from early exploration toward top-layer convergence, while Qwen and Llama differ in transition dynamics.
- 3.1 Definition of Internal Policy: Internal policies represent hidden states at individual layers or modules as token distributions through the unembedding matrix, exposing components of the final policy.The residual stream makes the final hidden state a combination of earlier contributions, allowing each component to be analyzed as part of policy formation.
- 3.2 Internal Policy Entropy Dynamics: Early layers maintain high entropy for exploration, while top layers converge toward near-zero entropy for final decision-making across models.Entropy Change measures incremental information gain between a module’s input and output, separating exploration, exploitation, and convergence dynamics.
- 3.2 Internal Policy Entropy Dynamics: Qwen3 attention sustains positive entropy change, whereas Qwen2.5-Math-7B contracts earlier and Llama shows weaker positive exploration.These patterns indicate model-dependent differences in how attention changes the hidden exploration space.
- 3.2 Internal Policy Entropy Dynamics: Llama FFN entropy remains weakly positive across nearly all layers and converges only at the final layer.This contrasts with the hierarchical progression observed in Qwen models.
- 3.2 Internal Policy Entropy Dynamics: Qwen3-4B FFNs progress through exploration in layers 1–6, stable integration in layers 7–26, and convergence in layers 27–36.The three stages correspond to positive, approximately zero, and negative entropy change, respectively.
4 Internal Policy Alignment
Internal Policy Alignment transfers reward feedback to selected intermediate policies through clipped updates. The analysis shows that limited alignment can refine lower-layer representations, whereas prolonged optimization destabilizes policy behavior and degrades performance.
- 4 Internal Policy Alignment: Internal Policy Alignment transfers reward feedback to intermediate representations using an advantage-weighted clipped update ratio.The update limits changes to the selected layer-induced distribution on reward-driven tokens while retaining clipped-policy stability.
- 4 Internal Policy Alignment: The penultimate-layer policy has minor entropy fluctuations but causes repetition and excessively long responses, while the last integration-region policy shows unstable increased entropy.These distinct dynamics depend on which internal layer policy is optimized.
- 4 Internal Policy Alignment: Optimizing layer 6 makes its hidden states increasingly similar to higher-layer representations, indicating early acquisition of higher-level reasoning features.The corresponding internal policy also shows progressive convergence during optimization.
- 4 Internal Policy Alignment: Layer-6 alignment remains close to later-layer policies early in training but deteriorates with prolonged optimization alongside worsening perplexity.The observed trade-off makes limited internal alignment preferable to continued optimization.
5 Bottom-up Policy Optimization
BuPO decomposes and aligns internal layer policies before optimizing the overall language-model policy, using entropy dynamics to select target layers. Across reasoning benchmarks, it improves over baselines, while ablations show that moderate alignment and entropy-indicated boundary layers are important for performance.
- 5 Bottom-up Policy Optimization: BuPO sequentially optimizes an internal layer policy before the language-model policy, using the current step and internal-policy training steps to control this schedule.The method is motivated by evidence that aligning internal policies refines lower-layer reasoning features.
- 5.1 Main Results: BuPO consistently improves over baselines across models and benchmarks, including gains of 4.69 and 2.30 points on Qwen3-4B and 4.58 and 0.76 on Qwen3-8B for AIME24 and AIME25.Llama models improve by 1.01 points on Llama-OctoThinker-3B-Base and 3.68 points on Llama-OctoThinker-8B-Base.
- 5.1 Main Results: BuPO achieves the best Pass@K results for all K on Qwen3-8B and all Llama models, with Pass@256 gains of 7.48 and 7.93 points on the two Llama backbones.Qwen3-4B is the only exception at K = 256.
- 5.3 Ablation Study: Early internal-policy alignment increases entropy exploration during training, with Llama lower-layer alignment providing a larger exploration space for the overall policy.For Qwen models, optimizing layer 6 maintains stable exploration.
- 5.3 Ablation Study: Moderate bottom optimization improves Qwen3-4B average score from 55.08 to 58.51 at 30 internal steps, whereas 50 or 70 steps reduce performance.The authors attribute excessive alignment to destabilization of the language-model policy.
- 5.3 Ablation Study: The best target layer varies by backbone, but across all four backbones it is the Boundary Layer: the last layer with positive FFN entropy change before integration or convergence.The results suggest entropy-change patterns can guide target-policy selection.
6 Related Work
The paper connects BuPO to RLVR and LLM interpretability while shifting RL optimization toward internal policies. It defines the Boundary Layer as the target layer selected by positive FFN entropy change.
- 6 Related Work: BuPO shifts RLVR research from holistic output-policy optimization toward direct optimization of internal layer policies during early training.The approach is presented as refining internal reasoning representations and improving performance.
- 6 Related Work: The Boundary Layer is the target layer selected for BuPO, corresponding to the last layer with positive FFN entropy change.This definition links the optimization target to the transition from exploration toward integration or convergence.
- 6 Related Work: The paper analyzes hidden states from a policy-centric perspective to reveal structured internal reasoning patterns that motivate BuPO.This work builds on interpretability studies of attention, FFN, and internal information routing.
7 Conclusion
The paper decomposes language-model policies to expose layerwise reasoning dynamics and introduces BuPO to align internal layer policies early in training. Experiments report effectiveness, while internal alignment refines foundational reasoning features and improves the overall policy’s reasoning capacity.
- 7 Conclusion: Internal policies transition from high-entropy exploration in early layers to deterministic convergence in higher layers, with progressive Qwen reasoning contrasting with abrupt Llama convergence.These patterns are identified through entropy analysis of internal layer and modular policies.
- 7 Conclusion: BuPO aligns internal layer policies during early training and improves reasoning performance on complex benchmarks.The method reconstructs the reasoning foundation from the bottom up.
- 7 Conclusion: Internal policy alignment refines foundational reasoning features, improving the reasoning capacity of the overall policy.The conclusion connects lower-level feature refinement with the final policy’s reasoning ability.
Limitations
BuPO has substantial computational costs, especially for rollout-based RL and large-sample reasoning evaluation.
- BuPO requires substantial resources for rollout generation, reward evaluation, and policy optimization, limiting accessibility for constrained researchers.The experiments used a single node with 8 NVIDIA A100 GPUs.
- Reasoning-focused settings increase training and evaluation costs because models generate long trajectories before final answers.Large-sample metrics such as Pass@K with 300 sampled responses further increase evaluation costs.
- Improving RL training efficiency remains an important direction for future research.
Ethical considerations
The paper reports no ethical concerns because it uses public datasets and identifies no privacy issues.
- The authors report no ethical concerns arising from the research.
- The datasets used in the study are public.
- The authors report no privacy issues associated with the datasets.
A.6 The Cost of Internal Policy Analysis
Internal policy entropy analysis is a lightweight, one-time forward-pass diagnostic that adds little cost relative to RL training and supports layer selection.
- The analysis requires only a standard forward pass with hooks, hidden-state projection, and token-level entropy computation.It introduces no backward pass, policy update, or reference-model pass.
- 209.25 seconds versus 13.22 hours: internal policy entropy analysis takes approximately 0.44% of Qwen3-4B RL training time.The analysis-to-training ratio is approximately 1 : 227.4.
- BuPO's out-of-domain evaluation covers advanced mathematics, general reasoning, and coding benchmarks, where it consistently outperforms vanilla and GRPO baselines.
- BuPO also outperforms conventional layer-aware GRPO variants that freeze lower layers or reduce upper-layer learning rates.
B.3 Multi-seed Robustness
Across three random seeds, BuPO generally outperforms GRPO, with gains that are reported as stable relative to their standard deviations.
- +3.77, +2.53, +1.95, and +3.62 points: BuPO's average gains over GRPO across four benchmarks span all evaluated model variants.The gains correspond to Qwen3-4B, Qwen3-8B, OctoThinker-3B, and OctoThinker-8B, respectively.
- BuPO outperforms GRPO on most benchmarks when rerun with three random seeds under the same training setup.Table 10 reports means and standard deviations across seeds.
- The generally small standard deviations relative to performance gaps indicate that BuPO's improvements are stable across random seeds.