Source-linked AI summary
A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
Raghavv Goel, Risheek Garrepalli, Sudhanshu Agrawal, Chris Lott, Mingu Lee, Fatih Porikli
TL;DR
The paper asks whether diffusion objectives fundamentally reshape language-model representations, and answers by comparing layer- and token-wise dynamics across native and AR-initialized models. Using cosine similarity and static layer skipping, it finds that native diffusion models develop global, redundant representations that tolerate compression, while AR-like structure persists under AR initialization.
Problem
Whether diffusion training materially changes how language models organize representations across depth and tokens remains insufficiently understood.
Method
The study compares native dLLMs, AR models, and AR-initialized dLLMs using cosine similarity across layers and tokens, with static inference-time layer skipping as a redundancy probe.
Results
Native dLLMs develop global representations with early-layer redundancy and reduced recency bias, while AR models remain locally structured; LLaDA retains 91.8% average performance at 18.75% FLOPs reduction.
Takeaways & Limitations
Diffusion objectives are associated with depth redundancy that supports inference-time compression, whereas AR initialization preserves AR-like representational dynamics despite diffusion training.
Takeaways & Limitations
The analysis covers only one native dLLM and one AR-initialized dLLM, leaving generalization across diffusion architectures and training objectives open.
Abstract
from arXiv · showhide
Autoregressive (AR) language models build representations incrementally via left-to-right prediction, while diffusion language models (dLLMs) are trained through full-sequence denoising. Although recent dLLMs match AR performance, whether diffusion objectives fundamentally reshape internal representations remains unclear. We perform the first layer- and token-wise representational analysis comparing native dLLMs (LLaDA), native AR models (Qwen2.5), and AR-initialized dLLMs (Dream-7B), using cosine similarity across layers and tokens alongside static inference-time layer-skipping as an analytical probe of redundancy. We find that diffusion objectives produce more global representations with substantial early-layer redundancy and reduced recency bias, while AR objectives yield tightly coupled, locally structured representations. AR-initialized dLLMs retain AR-like dynamics despite diffusion training, revealing persistent initialization bias. Leveraging this redundancy, native dLLMs absorb up to 18.75% FLOPs reduction while retaining over 90% performance on math-reasoning and coding benchmarks, whereas AR models collapse under identical skipping, revealing that diffusion objectives, rather than architecture alone, induce depth redundancy that enables principled compression.
Introduction
The paper asks whether diffusion training changes how language models organize representations, then analyzes this question across layers and tokens while probing redundancy through inference-time layer skipping.
- Motivation: Diffusion language models replace left-to-right next-token prediction with full-sequence denoising, yet their internal representational organization remains insufficiently understood.Recent diffusion models perform comparably to strong autoregressive baselines, motivating a direct representational comparison.
- Efficiency Probe: A static, task-agnostic layer-skip policy uses this redundancy without KV-cache sharing or architectural modifications.The policy tests how much computation can be removed before representations and task performance degrade.
- Approach: The study systematically compares native diffusion, autoregressive, and AR-initialized diffusion models using layer-wise and token-wise similarity analysis.The comparison includes LLaDA, Qwen2.5, and Dream-7B.
- Findings: Diffusion objectives produce more global representations with early-layer redundancy and reduced recency bias, whereas AR objectives preserve incremental, locally structured refinement.AR-initialized diffusion models retain AR-like representational patterns, indicating persistent initialization bias.
Layer-wise and Token-wise Similarity Analysis
The analysis uses cosine similarity across layers and tokens to characterize how representations evolve, finding global, redundant early processing in native dLLMs but locally coupled, recency-biased dynamics in AR-like models.
- Method: The study tracks cosine similarity between consecutive layer representations for each token and aggregates these values across tokens and prompts.Cosine similarity isolates directional representational change while avoiding confounding from hidden-state magnitude variation.
- Recency Bias & Representational Abstraction: LLaDA exhibits minimal recency bias and global representations across tokens, while Qwen2.5 and Dream-7B show strong recency bias throughout depth.In LLaDA, recency bias emerges mainly in later layers; in the AR-like models, it remains prominent across layers and tokens.
- Initialization Bias: Dream-7B retains Qwen2.5-like similarity patterns and recency bias despite diffusion training, demonstrating persistent AR initialization bias.Its high-similarity regions and recency patterns occur in nearly identical layer ranges to the AR initialization.
- Layer-wise Similarity: Native dLLMs show high early-layer similarity plateaus above 0.95, followed by lower-similarity refinement in later layers.The layer-wise pattern remains largely consistent across denoising steps.
- Magnitude Evolution: The magnitude analysis finds small hidden-state changes across the first 60–70% of layers followed by a steep rise, with unusually high-magnitude sink tokens.The authors note that cosine similarity is invariant to magnitude, motivating this complementary check.
- Interpretation: The authors treat cosine similarity as a meaningful first-order proxy for potential representational redundancy, while acknowledging that it does not measure intrinsic information content.The analysis complements output-level ARness measures by probing internal hidden-state dynamics.
Layer-skip at Inference
The inference-time probe selects highly similar nonconsecutive layers for static skipping, testing whether representational redundancy can reduce computation without modifying model architectures.
- Design Properties: The method is static, task-agnostic, and architecture-agnostic, requiring neither per-task tuning nor architectural modification.It is complementary to KV-caching because skipping reduces depth and FLOPs, whereas caching targets memory and cross-token computation.
- Skip Policy: The skip policy selects N layers with the highest adjacency similarity while preventing any two selected layers from being consecutive.Layers are ranked by similarity and selected greedily under the nonconsecutiveness constraint.
- Skip Policy: During inference, a selected transformer block is bypassed by passing its input directly to the next layer through the residual stream.This discards only the block’s small additive update rather than truncating the residual stream.
- Analytical Probe: The hypothesis is that high similarity indicates redundancy and therefore supports minimal quality loss under skipping, while incremental AR refinement should make skipping more disruptive.The intervention is used as an analytical stress test rather than as a claim about a new architecture.
Experimental Setup
The study compares native diffusion, native autoregressive, and AR-initialized diffusion models across standard reasoning and coding benchmarks using representational analysis and inference-time layer skipping.
- Models: The evaluation includes LLaDA, Qwen2.5, and Dream-7B to disentangle training objective from initialization.LLaDA represents native diffusion, Qwen2.5 native autoregressive training, and Dream-7B AR-initialized diffusion.
- Scope: The comparison uses 7–8B models because few open-source diffusion models are trained from scratch at scale.The authors identify LLaDA-8B as the only comparable from-scratch checkpoint and leave fully controlled scaling comparisons for future work.
- Benchmarks: Benchmarks cover GSM8K, HumanEval, and MATH-500 for mathematical reasoning and code synthesis.GSM8K and MATH-500 use exact-match accuracy, while HumanEval uses pass@k with the official harness.
- Representational Analysis: The analysis computes layer-wise and token-wise cosine similarity across diffusion denoising steps and autoregressive forward passes.Statistics are aggregated across diverse prompts from the evaluation benchmarks.
- Layer-Skipping Evaluation: The study applies static, task-agnostic top-k layer skipping at inference time without KV sharing or architectural changes.Evaluated skip counts are k ∈ {0, 2, 4, 6}.
Results
Native diffusion models show early-layer redundancy and stronger tolerance to layer skipping, while autoregressive models degrade sharply under deeper pruning and AR-initialized diffusion models remain intermediate.
- Native Diffusion Robustness: 18.75% FLOPs reduction preserves 93–97% of LLaDA’s baseline performance across GSM8K, MATH500, and MBPP.HumanEval retention is slightly lower at 83%.
- Representational Dynamics: LLaDA’s early layers show stabilization, while token-wise analysis highlights global context integration across layers and models.Figure 4 compares layers 1, 8, 16, and 24 under limited and full-sequence decoding.
- Magnitude Evolution: Hidden-state norms remain stable through roughly 60–70% of depth before rising sharply near the network top.Maximum norms are dominated by rare sink-token spikes, often ≥10^3, rather than typical token magnitudes.
- AR-Initialized Diffusion: Dream-7B retains 50–80% at 4 skipped layers and 8–25% at 6, exceeding Qwen2.5 at matched deeper skips but remaining below LLaDA.Its intermediate robustness is consistent with partial restructuring from diffusion fine-tuning without matching native diffusion behavior.
- Skip Strategy: Consecutive layer skipping reduces LLaDA’s 8-layer GSM8K retention from 94% to 40% and HumanEval retention from 60% to 8%.The selected strategy maintains representational continuity, with skipped layers concentrated in the first 40–60% of depth.
Related Work
Diffusion language models have achieved competitive performance, but their internal representational structure remains insufficiently understood relative to AR and AR-initialized models.
- Diffusion language models replace autoregressive decoding with bidirectional denoising, enabling parallel decoding and global context modeling.
- Recent models including SEDD and LLaDA achieve competitive language-modeling performance.
- Figure 6 examines the distribution of skipped layer indices for a six-layer skip on LLaDA using HumanEval.
- Dream-7B adapts pretrained AR models to diffusion training, while MDLM simplifies diffusion objectives.
- How diffusion objectives shape internal representations relative to AR and AR-initialized models remains insufficiently understood.
Effects.
Systematic comparative evidence shows that AR-adapted diffusion models retain representational patterns associated with their AR initialization.
- The study provides direct layer- and token-level evidence of initialization bias in AR-adapted diffusion language models.
- This analysis contrasts AR-initialized diffusion models with the representational dynamics of AR models.
- Earlier layers generally capture coarse features, while deeper layers refine task-specific abstractions in hierarchical language-model representations.
Summary and Future Work
The paper finds distinct representational organizations for diffusion and AR objectives, while identifying limits in the similarity probe and model coverage.
- Diffusion objectives produce global representations with substantial early-layer redundancy and reduced recency bias.
- AR objectives yield tightly coupled, locally structured representations, while AR initialization persistently imprints this structure after diffusion training.
- Cosine similarity captures directional stability across layers but does not characterize the intrinsic information content of representations.
- The analysis covers one native dLLM and one AR-initialized dLLM, leaving generalization across diffusion models and training objectives open.
Additional Results
Additional results report accuracy retention and wall-time savings from layer skipping, including an integrated dual-cache configuration.
- Accuracy retention is reported for different numbers of skipped layers across all models evaluated in Table 3.
- 12.5% FLOPs saving accompanies 88% accuracy retention when four layers are skipped with dual-cache integration.
- Wall-time measurements for LLaDA-8B-Instruct show faster inference with minimal accuracy drop.
Additional Analysis and Visualizations
Token-wise cosine-similarity visualizations show stable, globally structured representations in LLaDA’s early layers but persistent recency-biased updates across Dream-7B. The additional figures organize these patterns by network depth and complement the layer-wise analysis.
- Analysis scope: Token-wise similarity analysis complements the preceding layer-wise analysis by exposing how hidden representations evolve across tokens within individual layers.The analysis targets recency bias and global-versus-local representation patterns.
- Token-wise similarity: LLaDA’s early layers show consecutive-token similarity above 0.9, indicating smooth transitions and minimal recency bias.Later layers become more variable and less similar as representations undergo task-specific refinement.
- Token-wise similarity: Dream-7B exhibits substantial consecutive-token representation changes throughout the network, mirroring autoregressive token-by-token refinement.This pattern persists despite diffusion training and is presented as evidence of persistent AR-initialization structure.
- Efficiency evaluations: Additional tables report performance retention under layer skipping, computation savings, and a combined layer-skipping and dual-cache evaluation.These tables cover model retention rates, downstream-task wall-time reduction, and GSM8K evaluation.
- Visualization organization: The visualizations compare token-wise cosine similarity across layers and use network-depth groupings spanning early, early-middle, late-middle, and late layers.Figures 7 and 8 divide the layers into ranges 0–7, 8–15, 16–23, and 24–31.