Source-linked AI summary
Surfing the Activation Wave: Analyzing the Information and Geometry of LLM Hidden States for Speculative Decoding
Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani, Risheek Garrepalli, Mingu Lee
TL;DR
Speculative decoding commonly selects target-model layers with fixed positional rules, although choosing an optimal multi-layer set is combinatorial. This paper uses entropy and CKA-transition signals to select hidden-state sets, and reports practical gains over fixed baselines in EAGLE3-style conditioning.
Problem
Speculative decoding commonly uses fixed positional heuristics to select target-model layers, while choosing informative, non-redundant multi-layer sets is combinatorial.
Source: §1 · p. 1Method
The framework uses matrix entropy and linear CKA transitions for layer-set selection, with beam-search ELSI, CKA-guided ELSI-C, and calibration token-entropy filtering.
Source: §1 · p. 2Results
ELSI/ELSI-C-selected sets provide practical gains over the default [N−1] and [2, N//2, N−3] rules in EAGLE3-style conditioning.
Source: §1 · p. 2Takeaways & Limitations
Across two models, non-uniform entropy profiles, entropy gains under concatenation, and CKA transition boundaries support selecting layer sets by information and diversity rather than fixed indices.
Source: §1 · p. 2, §6 · p. 7Throughput effects from the smaller down-projection may not be visible on non-memory-bound GPUs, and edge-device evaluation remains future work.
Source: §5.6.2 · p. 7
Abstract
from arXiv · showhide
Decoder hidden states from different layers of large language models (LLMs) often encode complementary information. However, existing speculative decoding methods typically rely on a small set of target-model layers chosen using fixed, position-based heuristics. Selecting an optimal subset of layers is inherently a combinatorial problem, making exhaustive evaluation infeasible even for moderately deep models. In this work, we present a principled framework for multi-layer hidden-state selection for speculative decoding based on two complementary criteria: informativeness, measured through matrix-entropy analysis, and diversity, quantified using centered kernel alignment (CKA). We investigate layer-wise Shannon entropy, aggregate entropy under hidden-state concatenation, and pairwise CKA dynamics across depth, revealing distinct geometric and informational compartments within decoder representations. Motivated by these findings, we propose entropy and CKA-based layer selection algorithms that favor hidden-state sets that are both highly informative and minimally redundant. As part of this framework, we introduce token-entropy filtering, which removes hidden states associated with low predictive-entropy tokens, thereby improving the quality of layer-selection statistics. In EAGLE3-style speculative decoding, our selected sets outperform fixed baseline choices, including the default $[2, N//2, N-3]$ rule, while providing a clearer explanation of why some defaults can remain competitive. Our results provide a practical recipe for multi-layer hidden-state selection for speculative decoding, insights into the geometry and information in LLM decoder states, and a foundation for extending similar analysis to related applications.
1 Introduction
The introduction frames multi-layer hidden-state selection as a combinatorial problem requiring a balance between informativeness and redundancy. It presents entropy- and CKA-guided selection, token-entropy filtering, and practical gains in EAGLE3-style conditioning over fixed layer rules.
1 Introduction
- Selecting multiple layers is combinatorial, while fixed positional heuristics lack a principled explanation of when chosen layers are informative or redundant.For a 36-layer model, choosing three layers already yields thousands of combinations.Source: §1 · p. 1
- Across Qwen3-4B-Instruct-2507 and Llama-3.1-8B-Instruct, entropy profiles and concatenation gains vary with depth, while CKA transitions mark representational compartments.These patterns motivate ELSI and its diversity-aware extension, ELSI-C.Source: §1 · p. 2
- The framework measures layer-set informativeness with matrix entropy and redundancy with linear CKA, jointly targeting informative, less-redundant hidden-state sets.The stated objective is multi-layer target-model selection for speculative decoding.Source: §1 · p. 2
- ELSI uses beam search over concatenated hidden states to find high-entropy layer sets at realistic model depths.ELSI-C extends entropy-initialized selection through CKA-transition-guided expansion.Source: §1 · p. 2
- Calibration token-entropy filtering removes hidden states of low predictive-entropy tokens to sharpen layer-discriminative selection statistics.The framework focuses those statistics on significant and informative positions.Source: §1 · p. 2
- ELSI/ELSI-C-selected sets provide practical gains over the default [N−1] and [2, N//2, N−3] rules in EAGLE3-style conditioning.The introduction also presents the selections as improving interpretability.Source: §1 · p. 2
2 Related Work
Related work studies layer specialization, redundancy, spectral structure, and hidden-state conditioning, but does not jointly select activation sets for draft conditioning. This work combines entropy and CKA-transition signals and evaluates selected sets in EAGLE3-style speculative decoding.
Layer specialization and redundancy in transformers
- Layer studies find middle layers can be partially interchangeable while early and late layers remain more distinct; pruning work reports limited quality loss after removing middle-layer computation [2].Representation-geometry analyses also describe structured hidden-state evolution across depth.Source: §2 · p. 2
Spectral and rank-based analyses
Speculative decoding and hidden-state conditioning
- EAGLE and DFlash [12] condition draft models on selected target hidden states, usually chosen by fixed positional rules that may yield redundant representations.ELSI uses matrix entropy and CKA-transition structure, with beam-overlap diagnostics explaining when positional baselines remain competitive.Source: §2 · pp. 2–3
Positioning of this work
- The paper jointly optimizes hidden-layer sets for draft conditioning by measuring fused-representation entropy, guiding expansion with CKA transitions, and evaluating EAGLE3-style decoding.Source: §2 · p. 3
3 Preliminaries
The preliminaries define selected hidden states as a concatenated representation that constrains the information available to the draft model. Matrix entropy measures information in layer representations, while CKA transitions capture representational changes across depth.
3 Preliminaries
- Selected hidden states are concatenated and down-projected, making the layer set a bottleneck for information seen by the draft model.The representation concatenates chosen layers along the feature dimension.Source: §3 · p. 3
3.1 Matrix Entropy
- Matrix entropy is computed from the normalized spectrum of a representation and applied to concatenated layer states to assess their joint feature information.The paper uses S1 to study per-layer trends and select high-entropy layers.Source: §3.1 · p. 3
3.2 Centered-KernelAlignment(CKA)
- CKA compares representations through their Gram matrices, and changes in CKA across adjacent layers yield transition scores whose large peaks indicate compartment boundaries.The transition score can also be extended to concatenated layers.Source: §3.2 · p. 3
4 Method
The method filters token-level hidden states, builds layer representatives, and selects layer sets through entropy-driven search with a CKA-aware expansion option.
4 Method
- The pipeline combines token-entropy filtering, hidden-state aggregation, entropy-driven beam search (ELSI), and transition-aware CKA expansion (ELSI-C).ELSI targets informative sets, while ELSI-C adds diversity-oriented refinement.Source: §4 · p. 3
4.1 Token-Entropy Filtering
- Low-predictive-entropy token positions are removed from sequence hidden states before layer-selection statistics are computed.The experiments use a filtering fraction of ρ = 0.3.Source: §4.1 · p. 3
4.2 Hidden State Collection and Block Averaging
- Filtered states are partitioned into non-overlapping 32-token blocks and averaged across blocks and samples to form layer representatives.The analyses use 200 calibration samples from ShareGPT.Source: §4.2 · p. 4
4.3 Matrix Entropy-based Expansion (ELSI)
- ELSI seeds a beam with high-entropy singleton layers, expands sets using concatenated-state entropy, and deduplicates and prunes candidates by entropy.It selects the highest-entropy set at the target size.Source: §4.3 · p. 4
4.4 ELSI-C: Entropy-Initialized CKA Expansion
- ELSI-C initializes and prunes by entropy but restricts each expansion to candidates with high CKA transition scores.Transition-aware candidates are aggregated and deduplicated before entropy-based beam pruning and final selection.Source: §4.4 · p. 4
5 Experiments
Experiments on Qwen3-4B-Instruct-2507 and Llama-3.1-8B-Instruct examine layer-wise entropy, pairwise concatenation entropy, and the effect of token filtering.
5 Experiments
- Experiments use open-source models from two different families, Qwen3-4B-Instruct-2507 and Llama3.1-8B-Instruct, to analyze per-layer entropy, concatenated entropy, CKA transitions across layers, and ELSI- and ELSI-C-selected layer candidates.Source: §5 · p. 4
5.1 Per-layer matrix entropy
- Llama3.1-8B-IT entropy drops after a high first layer and rises to a peak around layer 27, while Qwen3-4B-IT shows a high-low-high profile peaking at layer 28.The authors consequently require a high-entropy singleton layer to occur after the midpoint.Source: §5.1 · p. 5, Figure 2 · p. 5
5.2 Entropy of layer pairs
- Concatenated layer pairs show non-uniform entropy gains: some pairs add complementary directions, while others are largely redundant.Different high-entropy seed layers benefit from different companion layers, motivating joint optimization of layer sets.Source: §5.2 · p. 5
5.3 Token-Entropy Filtering
- Token filtering preserves the same depth-wise activation-wave profile seen before filtering.The provided passage states this profile is preserved in Figure 4.Source: §5.3 · p. 5
5.4 CKA compartments and transitions
CKA transitions reveal depth-wise representational compartments, with different transition patterns across Qwen3-4B-IT and Llama3.1-8B-IT.
5.4 CKA compartments and transitions
- Pairwise CKA transition analysis reveals distinct depth-wise representational compartments.The analysis overlays transition boundaries on pairwise CKA.Source: §5.4 · p. 5
- Qwen3-4B-IT has more transitions, including two large middle compartments; Llama3.1-8B-IT has similar CKA across depth except at the first and last layers.Source: §5.4 · p. 5
- EAGLE2-like last-layer-only conditioning may miss other high-information layers, while EAGLE3’s [2, N//2, N −3] layers span different compartments but may retain overlapping information.Source: §5.4 · p. 5
5.5 Selection tables and baseline intuition
ELSI and ELSI-C select layer sets with higher entropy than the EAGLE2 and EAGLE3 choices across both evaluated models.
5.5 Selection tables and baseline intuition
- Across Qwen3-4B-IT and Llama3.1-8B-IT, ELSI and ELSI-C layer sets have higher entropy than EAGLE2 and EAGLE3 selections.The comparison covers single layers, pairs, and triples.Source: §5.5 · p. 5
- Figure 3 pairs top-k layers ranked by entropy with remaining layers to show entropy for layer pairs.The caption describes the pairing, but does not report individual pair values.Source: Figure 3 · p. 6
- At ρ = 0.30, token-entropy filtering lowers entropy variance across prompts while preserving the activation-wave shape.Source: Figure 4 · p. 6
5.6 Impact on Speculative Decoding
On Qwen3-4B-Instruct, ELSI with three layers leads on average across the reported settings, while one selected layer can match or exceed EAGLE3; Llama results show smaller gains from adding layers.
5.6.1 Baselines and Evaluation.; 5.6.2 Training and Evaluation Setup.
- Draft models use ShareGPT and Ultrachat distillation data and are evaluated on SpecBench at 30 or 60 tree nodes and T=0.0 or T=1.0; throughput gains may not appear on GPUs that are not memory-bound.The reported evaluation compares EAGLE2, EAGLE3, ELSI/ELSI-C, and low-entropy negative sets.Source: §5.6.2 · pp. 6–7, §5.6.1 · p. 6
5.6.2 Training and Evaluation Setup.
- ELSI (m=3), matching the number of EAGLE3 layers, is the best-performing method on average across all settings, improving over EAGLE3 by 1.6–2.2% (+0.085 τ on average) and over EAGLE2 by 6.1–7.3%.Source: §5.6.2 · p. 7
- Across all four settings, ELSI (m=1) matches or exceeds EAGLE3 on average, with gains of +0.01 to +0.04 τ; a single well-chosen layer matches EAGLE3’s default 3-layer heuristic.Source: §5.6.2 · p. 7
- ELSI tends to provide the most benefit where EAGLE3 performs weakest, improving acceptance rate in Translation by 2.6–5.6% and Summarization by 2.1–4.2%, while also improving STEM by 2.4–3.9%.Source: §5.6.2 · p. 7
- On Llama3.1-8B-IT, ELSI (m=1) exceeds EAGLE2 by 2.8–4.9% and attains ∼99.0% of EAGLE3’s block efficiency, while ELSI (m=3) trails EAGLE3 by 1.0% on average.Average τ varies by only 0.017 across m = 1, 2, 3; at T=1.0, ELSI (m=3) improves over EAGLE3 on Roleplay and Coding, while Writing losses leave the average unchanged.Source: §5.6.2 · p. 7
- Llama’s broad regions of high pairwise CKA correspond to more interchangeable layers, smaller gains from layer choice, and good performance from uniformly spaced selections.The authors attribute Llama’s reduced headroom to this layer interchangeability.Source: §5.6.2 · p. 7
6 Conclusion
The conclusion presents ELSI and ELSI-C as practical selectors grounded in entropy and CKA structure, with empirical gains over speculative-decoding baselines and a potential broader research scope.
6 Conclusion
- Across Qwen3-4B-Instruct and Llama-3.1-8B-Instruct, decoder states show non-uniform entropy, non-uniform concatenation gains, and CKA transition boundaries.Source: §6 · p. 7
- ELSI and ELSI-C select layer combinations to maximize information and minimize redundancy, and token-entropy filtering reduces cross-prompt entropy variance.The conclusion describes the algorithms as practical for typical model sizes.Source: §6 · p. 7
- The experiments indicate that layer choice matters more than layer count and that selected sets can outperform EAGLE2 and EAGLE3.Source: §6 · p. 7
- The framework is proposed for future study with broader target models and draft-conditioning strategies, as well as embedding and interpretability tasks.Source: §6 · p. 7
A Transition boundary comparison with and without token filtering
The appendix compares default and token-filtered CKA transition plots for both models to isolate the effect of filtering on representational compartment boundaries.
A Transition boundary comparison with and without token filtering
- Default and token-filtered CKA transition plots are compared side by side for both models to examine filtering’s effect on compartment boundaries.Source: A Transition boundary comparison with… · p. 10
B ELSI Beam Analysis
The ELSI beam analysis shows the layer sets selected at successive beam depths and compares CKA transition detection with and without token filtering.
B ELSI Beam Analysis
- The ELSI beam tree for Qwen3-4B-IT expands using high joint entropy of token-filtered hidden states.Figure 7 shows the top-ranked layer at each depth for the best single-, pair-, and triple-layer sets.Source: B ELSI Beam Analysis · p. 10, Figure 7 · p. 12
- Figure 7’s top-ranked layers at each beam depth represent the best single-, pair-, and triple-layer sets, respectively.Source: B ELSI Beam Analysis · p. 10, Figure 7 · p. 12
- Figure 6 compares CKA transitions with and without token filtering; yellow dashed lines mark transition boundaries under matched detector settings.Source: Figure 6 · p. 11