PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unseen Bias: How Norm Discrepancy in Pre-Norm MLLMs Leads to Visual Information Loss

Bozhou Li, Xinda Xue, Sihan Yang, Yang Shi, Xinlong Chen, Yushuo Guan, Yuanxing Zhang, Wentao Zhang

arXiv:2512.08374v1cs.CV

TL;DR

The paper examines norm disparity between visual and text tokens in Pre-Norm MLLMs and analyzes its effects on representation updates. It validates the dynamic across mainstream models and proposes gradient-aware norm alignment; reported experiments show gains on multimodal and text-only benchmarks.

  • Problem

    Pre-Norm MLLMs can have severe norm disparity between high-norm visual tokens and low-norm text tokens, a problem the paper examines for its effect on cross-modal fusion.

    Source: §1 · p. 1, §2.3 · p. 4

  • Method

    The paper theoretically analyzes asymmetric updates, empirically checks norm disparities across mainstream models, and proposes LayerNorm-based norm alignment with Global Weight Compensation.

    Source: §1 · p. 2

  • Results

    The paper reports that Gradient-Aware Norm Alignment yields significant gains on multimodal and text-only benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that norm alignment with global Weight Compensation addresses the optimization dilemma of extreme norm compression while improving reported multimodal and text-only performance.

    Source: §1 · p. 2

    The theoretical analysis relies on simplifying assumptions, including uniform update magnitude, consistent update geometry, and symmetric rotational directions.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs), which couple pre-trained vision encoders and language models, have shown remarkable capabilities. However, their reliance on the ubiquitous Pre-Norm architecture introduces a subtle yet critical flaw: a severe norm disparity between the high-norm visual tokens and the low-norm text tokens. In this work, we present a formal theoretical analysis demonstrating that this imbalance is not a static issue. Instead, it induces an ``asymmetric update dynamic,'' where high-norm visual tokens exhibit a ``representational inertia,'' causing them to transform semantically much slower than their textual counterparts. This fundamentally impairs effective cross-modal feature fusion. Our empirical validation across a range of mainstream MLLMs confirms that this theoretical dynamic -- the persistence of norm disparity and the resulting asymmetric update rates -- is a prevalent phenomenon. Based on this insight, we propose a remarkably simple yet effective solution: inserting a single, carefully initialized LayerNorm layer after the visual projector to enforce norm alignment. Experiments conducted on the LLaVA-1.5 architecture show that this intervention yields significant performance gains not only on a wide suite of multimodal benchmarks but also, notably, on text-only evaluations such as MMLU, suggesting that resolving the architectural imbalance leads to a more holistically capable model.

1 INTRODUCTION

The introduction identifies Pre-Norm norm disparity as an architectural problem in MLLMs and links it to slower semantic updates for visual tokens. It presents empirical validation and a gradient-aware norm-alignment approach.

1 INTRODUCTION

  • MLLMs couple a pre-trained vision encoder to a language model, using an adapter to project visual tokens into the LLM’s word-embedding space.The vision encoder typically divides images into patches and encodes them as visual tokens.Source: §1 · p. 1
  • Pre-Norm leaves the residual path unaltered, but hidden-state variance and L2 norms accumulate, producing high-norm visual and lower-norm text tokens in a shared backbone.The introduction notes that visual tokens come from a deep Pre-Norm vision encoder.Source: §1 · p. 2
  • High-norm visual tokens undergo slower semantic transformation than lower-norm text tokens, creating asymmetric updates and impairing multimodal fusion.The paper attributes this representational inertia to the interplay between Pre-Norm and the MLLM paradigm, rather than to an intrinsic modality difference.Source: §1 · p. 2
  • Across mainstream open-source vision-language models, norm disparities and asymmetric update rates are prevalent; the proposed response is norm alignment with Global Weight Compensation.The compensation mechanism separates forward norm compression from backward gradient magnitude to support learning under extreme alignment constraints.Source: §1 · p. 2
  • The paper reports significant gains on multimodal and text-only benchmarks from Gradient-Aware Norm Alignment, which uses a Weight Compensation mechanism to address extreme norm compression.The introduction presents this as a simple, robust solution to the optimization dilemma.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries review Transformer self-attention and normalization, then contrast Pre-Norm and Post-Norm residual architectures. They explain why Pre-Norm’s training-friendly identity-like path can leave multimodal token norms imbalanced.

2.1 SELF-ATTENTION

  • Self-attention projects hidden states into queries, keys, and values, then computes unnormalized dot-product scores QK^T.The section frames self-attention as the Transformer’s computational core.Source: §2.1 · p. 2

2.2 NORMALIZATION LAYERS IN TRANSFORMERS

  • LayerNorm normalizes each token across its feature dimension, with learnable gain and bias parameters restoring expressive power.The preliminaries also identify RMSNorm as a simplified variant that omits re-centering.Source: §2.2 · p. 3

2.3 THE PRE-NORM VS. POST-NORM RESIDUAL ARCHITECTURE

  • Pre-Norm places normalization before the sublayer within the residual branch, preserving an uninterrupted identity-like skip path that eases deep-model training.Unlike Post-Norm, it does not re-normalize the residual output, so hidden-state variance and L2 norms tend to accumulate across layers.Source: §2.3 · p. 4
  • In MLLMs, high-norm visual tokens from deep Pre-Norm vision encoders enter the LLM’s lower-norm embedding space, establishing norm disparity at the modality interface.The section identifies this mismatch as the starting point for the dynamic imbalance that can impair cross-modal fusion.Source: §2.3 · p. 4

3 THEORETICAL ANALYSIS OF NORM-INDUCED DECOUPLING EFFECT

The analysis models how norm imbalance in Pre-Norm MLLMs affects token updates and cross-modal attention. It argues that visual-token inertia limits semantic alignment and weakens attention’s relevant signal.

3 THEORETICAL ANALYSIS OF NORM-INDUCED DECOUPLING EFFECT

  • The theoretical analysis argues that initial norm imbalance accelerates geometric divergence between modalities and suppresses the cross-modal attention signal.The paper supplies the full mathematical derivation in the appendix.Source: §3 · p. 4

3.2 ASYMMETRIC ANGULAR VELOCITY AND GEOMETRIC DIVERGENCE

  • For high-norm visual tokens, a uniform-magnitude update produces lower effective angular velocity than for lower-norm text tokens, yielding greater representational inertia.The proof framework treats update magnitude as decoupled from input norm and assumes consistent update geometry and symmetric rotational directions.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 SIGNAL DEGRADATION AND ATTENTION COLLAPSE

  • Visual-token inertia slows reciprocal orientation changes, leaving semantically related text-query and visual-key pairs angularly misaligned and capping their relevant dot-product signal.The analysis describes this as a barrier to alignment in the shared metric space.Source: §3.3 · p. 5
  • Because relevant-pair scores are suppressed while background noise remains approximately static, attention’s signal-to-noise ratio falls and Softmax becomes less able to distinguish target visual regions.The paper connects this mechanism to diffuse attention maps and effective visual-information loss.Source: §3.3 · p. 5

4 EMPIRICAL VALIDATION: PROBING THE DYNAMICS OF NORM IMBALANCE

Because the theoretical analysis relies on simplifying assumptions, the authors test whether norm imbalance and asymmetric updates appear in mainstream MLLMs. Across continuous-projection models, update-rate divergence tracks the initial norm gap; Ovis 2.5 retains a large norm gap but shows synchronized updates.

Empirical validation

  • The experiments test whether norm imbalance and its predicted update dynamics persist in real-world Pre-Norm MLLMs, beyond the theory’s simplifying assumptions.The empirical investigation covers multiple models and includes Ovis 2.5, which uses a discrete visual vocabulary rather than continuous projected features.Source: §4 · pp. 5, 7

RQ1: Existence of Initial Norm Disparity

  • Vision-encoder outputs have substantially larger L2 norms than text embeddings, and the gap persists because contrastive pretraining does not align their scales.The study measured four vision encoders and three LLM embedding layers on 1,000 samples from MMBench, POPE, and MM-Star.Source: §4 · p. 6, Table 1 · p. 6

RQ2: The Efficacy of the Adapter

  • Projectors vary in norm compression, but visual and text token norms remain substantially different across all analyzed models.Internal normalization layers in all analyzed models except LLaVA-v1.5 do not guarantee aligned output norms.Source: §4 · p. 6, Table 2 · p. 6

RQ3: Asymmetry in Update Dynamics

  • Across analyzed models, visual and text update rates diverge, with larger differences in LLaVA-1.5 and Qwen-2.5-VL than in Kimi-VL and GLM-4.5V.The measured asymmetry appears correlated with initial norm disparity; models with smaller gaps show less pronounced update differences.Source: §4 · p. 7

RQ4: Norm Discrepancy in Models with Visual Embedding Tables

  • Ovis 2.5 has an average visual-token norm of 64.00 versus 1.38 for text, yet its visual and text update rates remain synchronized.The study attributes this contrast in dynamics to the discrete visual-tokenization paradigm as a possible mitigation of representational inertia.Source: §4 · p. 7, Figure 2 · p. 7, Table 3 · p. 7

5 EXPERIMENTS

The experiments test whether norm alignment addresses cross-modal dynamics and downstream performance. The proposed LayerNorm uses text-token norms as its target, with Global Weight Compensation to counter the gradient bottleneck of a minute gain initialization.

5.1 METHOD: GRADIENT-AWARE NORM ALIGNMENT

  • An added LayerNorm after the visual projector aligns visual-token norms to text-token norms by initializing its learnable gain to match their average L2 norm.The method addresses a tiny-gain initialization that can dampen gradients to the vision encoder; Global Weight Compensation rescales backward gradients while retaining forward norm alignment.Source: §5.1 · pp. 8–9

5.2 EXPERIMENTAL SETUP

  • Experiments use the LLaVA-1.5 framework with Llama-3.2-3B-Instruct or Qwen2.5-7B-Instruct, SigLIP-SO400M-Patch14-384, and greedy decoding.The benchmarks are listed in Appendix E.Source: §5.2 · p. 9

5.3.1 MAIN PERFORMANCE GAINS

  • Llama-3.2 improves with simple norm alignment, while Qwen2.5 needs Global Weight Compensation to overcome vanishing gradients and gain across multimodal and text domains.The reported response depends on the backbone; the table caption identifies comparisons across benchmarks and backbones but supplies no cell values.Source: §5.3.1 · p. 9, Table 4 · p. 9
  • Norm-aligned attention focuses text queries on semantically relevant image regions, unlike the baseline’s broad concentration on bottom image regions.The authors attribute the baseline pattern to positional proximity bias from RoPE’s distance-decay property.Source: §5.3.1 · p. 9

5.3.2 ABLATION STUDY: THE CRITICAL ROLE OF INITIALIZATION

  • After Stage 1 pre-training, default-initialized LayerNorm parameters remain largely unchanged, whereas targeted initialization yields meaningful updates.The authors conclude that adding a norm layer alone is insufficient; targeted initialization is needed for effective learning.Source: §5.3.2 · p. 10, Table 5 · p. 10

5.3.3 DIAGNOSTIC ANALYSIS: VERIFYING THE MECHANISM OF IMPROVEMENT

  • Norm alignment harmonizes visual and text token norms across model depth, while the baseline retains norm divergence.The accompanying cosine-similarity analysis shows more synchronized update rates with the intervention, mitigating the baseline’s visual-token representational inertia.Source: §5.3.3 · p. 10
  • The diagnostic findings link alleviating static norm disparity with reduced update asymmetry and improved downstream performance.The authors phrase this relationship as appearing to facilitate the observed gains.Source: §5.3.3 · p. 10

6 CONCLUSION

The paper identifies asymmetric token updates in Pre-Norm MLLMs and reports that norm alignment with Global Weight Compensation yields gains on multimodal and text-only tasks. It situates this analysis within LLaVA-style fusion and contrasts it with work seeking alternative normalization placements.

6 CONCLUSION

  • The paper formalizes asymmetric updates in Pre-Norm MLLMs and proposes norm alignment via a single LayerNorm with Global Weight Compensation.It reports significant gains on multimodal and text-only tasks as validation of its analysis.Source: §6 · p. 10

A.1 MULTIMODAL LARGE LANGUAGE MODELS

  • LLaVA-style MLLMs project visual features into the LLM’s word-embedding space, treating images as visual words processed alongside text.Other approaches explore projector designs, visual vocabularies, deeper fusion, architecture adaptation, and data-centric methods.Source: §A.1 · p. 16

A.2 NORMALIZATION

  • Pre-Norm places normalization within the residual branch for gradient flow and stability, but hidden-state L2 norms can grow with depth because the main path is not re-normalized.The paper argues that prior explanations attributing norm-discrepancy effects to RoPE overlook that Pre-Norm normalizes vectors before query and key projections.Source: §A.2 · p. 16
  • Unlike Peri-Norm and HybridNorm’s search for static designs for unimodal models, this work diagnoses imbalance induced by standard Pre-Norm in multimodal settings.The related-work discussion frames the paper as an analysis of the existing architecture rather than a new general architecture.Source: §A.2 · p. 17

B APPENDIX: DETAILED DERIVATION AND PROOFS

The appendix derives how norm imbalance slows visual-token rotation, reduces cross-modal similarity retention, and limits attention signal strength; the embedding analysis also reports a multimodal norm shift in Qwen2.5-VL.

B.1 STEP 1: KINEMATICS — NORM IMBALANCE INDUCES VELOCITY ASYMMETRY

  • Higher-norm tokens have lower angular velocity under uniform update magnitude and consistent update geometry, so visual tokens rotate more slowly than text tokens.The derivation decomposes updates into magnitude-scaling and direction-rotation components.Source: §B.1 · p. 17

B.2 STEP 2: DYNAMICS — RECURSIVE DECAY OF SEMANTIC SIMILARITY

  • Under symmetric orthogonal rotational updates, expected cross-modal cosine similarity evolves recursively with γ_eff, the product of the individual angular-velocity cosines.Alignment retention depends on cos(θ_vis) cos(θ_txt).Source: §B.2 · p. 17

B.3 STEP 3: OPTIMIZATION — ASYMMETRY ACCELERATES GEOMETRIC DIVERGENCE

  • For a fixed total update potential, symmetric angular velocities maximize similarity retention, while asymmetry strictly decreases it and increases the accumulated Lag Angle.The paper defines the Lag Angle as the final angle in the imbalanced case minus that in the balanced case.Source: §B.3 · p. 18

B.4 STEP 4: FROM GEOMETRIC MISALIGNMENT TO SIGNAL-TO-NOISE RATIO COLLAPSE

  • With projection-matrix capacity bounded by R, the norm-induced geometric lag strictly reduces the maximum extractable attention score.The derivation states that recovering the lost signal energy would require unboundedly increasing projection weights.Source: §B.4 · p. 18
  • The lag caps relevant-token signal by cos(∆ϕ), while the noise floor remains constant, reducing the signal-to-noise gap and producing higher-entropy, more uniform attention.The paper links this reduced gap to diffuse attention maps observed in experiments.Source: §B.4 · p. 18

C EMPIRICAL ANALYSIS OF EMBEDDING NORM SHIFTS

C.1 GLOBAL STATISTICS

  • 0.80 → 0.90: Qwen2.5-VL’s mean embedding norm rises relative to its base model, whereas Qwen2-VL maintains a similar mean norm.These are the reported mean-norm comparisons.Source: §C.1 · p. 19, Table 6 · p. 19

C.2 DETAILED ANALYSIS OF TOP-NORM TOKENS

The top-norm token comparison shows that multimodal training changes which tokens dominate the embedding norms.

C.2 DETAILED ANALYSIS OF TOP-NORM TOKENS

  • ≈2.4: Qwen2-VL visual boundary tokens reach norms far above the ≈1.0–1.1 maximums of pure-text base-model outliers.The reported base-model high-norm tokens are rare subwords or syntactic markers; the visual boundary tokens become extreme outliers in the VL model.Source: §C.2 · p. 19, Table 7 · p. 19

D DETAILED IMPLEMENTATION OF NORM ALIGNMENT

The implementation compresses projected visual-token norms in the forward pass while compensating backward gradients to preserve supervision to the vision encoder.

D DETAILED IMPLEMENTATION OF NORM ALIGNMENT

  • GlobalWeightCompensatedLayerNorm applies a small gain initialized from the target text norm and zero bias, then rescales backward gradients to offset the small gain.It clamps the global mean absolute gain at δ = 10^-3 for numerical stability; the stated aim is to preserve effective vision-encoder supervision from the first training iteration.Source: D DETAILED IMPLEMENTATION OF NORM… · pp. 19–20

E.1 EXPERIMENTAL SETUP

The experiments use two language-model backbones with a shared vision encoder and two-stage training, then evaluate multimodal and text-only capabilities.

E.1 EXPERIMENTAL SETUP

  • Experiments couple Llama-3.2-3B-Instruct or Qwen2.5-7B-Instruct with SigLIP-SO400M-Patch14-384 and use one epoch each of feature alignment and instruction tuning.Images are resized to 384 × 384, and the setup does not use dynamic high-resolution strategies.Source: §E.1 · p. 21
  • Evaluation covers seven multimodal benchmarks plus HellaSwag and MMLU for text-only language understanding and commonsense reasoning.The multimodal suite includes MMBench-EN, MM-Star, OCRBench, SEED-Bench-2-Plus, ScienceQA, AI2D, and POPE.Source: §E.1 · p. 21

F TRAINING DYNAMICS

The section tracks layer-wise cosine similarity across pre-training checkpoints to compare visual-token updates with and without norm alignment. Without normalization, visual similarity remains high and nearly static; with the method, it starts lower and rises as the LayerNorm gain is optimized.

F TRAINING DYNAMICS

  • Without normalization, visual tokens retain persistently high inter-layer similarity throughout pre-training, consistent with representational inertia.The passage describes this pattern as locking visual features against semantic transformation from the beginning of training.Source: F TRAINING DYNAMICS · p. 21
  • With norm alignment, visual similarity starts significantly lower, indicating active feature updates, then rises as the LayerNorm gain is optimized.The upward trend converges during training.Source: F TRAINING DYNAMICS · p. 21
  • The converged similarity trend suggests perfectly matched image- and text-token update rates may not be optimal.The passage distinguishes this from the baseline’s substantial update inconsistency associated with initial norm disparity.Source: F TRAINING DYNAMICS · p. 21

G APPENDIX: ATTENTION VISUALIZATION

The appendix presents attention heatmap pairs for text queries, with the baseline above the norm-applied model. Figure 5 describes layer-wise cosine similarity across checkpoints during multimodal pre-training.

G APPENDIX: ATTENTION VISUALIZATION

  • In each attention heatmap pair, the baseline is shown above the norm-applied model, and each pair is tied to its text query.The listed queries ask about image mood, subjects, counts, people, color, and focus.Source: G APPENDIX: ATTENTION VISUALIZATION · pp. 21, 23
  • Figure 5 tracks layer-wise cosine similarity across checkpoints during multimodal pre-training with norm applied.The passage identifies the figure’s scope but does not provide its plotted values.Source: Figure 5 · p. 22

Related papers

Loading 2512.08374v1…