PaperlayerView exact revision on arXiv

Source-linked AI summary

Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

Xiaomin Yu, Yi Xin, Yuhui Zhang, Wenjie Zhang, Chonghan Liu, Hanzhen Zhao, Chen Liu, Xiaoxing Hu, Ziyue Qiao, Hao Tang, Xiaobin Hu, Chengwei Qin, Hui Xiong, Yu Qiao, Shuicheng Yan

arXiv:2602.07026v3cs.CVcs.AIcs.MM

TL;DR

The paper addresses the persistent Modality Gap and the limitations of isotropic assumptions by modeling its geometry in a fixed reference frame. It proposes ReAlign for training-free statistical alignment and ReVision for scalable MLLM training with unpaired text, reporting stronger performance than baselines and a cost-effective scaling pathway.

  • Problem

    The Modality Gap persists despite multimodal contrastive learning, while simplified isotropic assumptions limit precise geometric modeling and large-scale application.

  • Method

    The paper decomposes the gap into stable biases and anisotropic residuals in a fixed frame, then uses ReAlign and ReVision to align and train with unpaired text data.

  • Results

    ReVision significantly mitigates the modality gap and outperforms baselines, with ReVision-2M scoring 49.75 versus Unicorn’s 43.94 on the reported comparison.

  • Takeaways & Limitations

    Statistically aligned unpaired data can substitute for expensive image-text pairs and provide a cost-effective pathway for scaling MLLMs.

Abstract

from arXiv · show

Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models~(MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.

1 Introduction

The paper frames the Modality Gap as a structured geometric mismatch that limits cross-modal interchangeability and motivates scalable learning from unpaired data. It introduces Fixed-frame Modality Gap Theory, ReAlign, and ReVision to model and exploit this structure.

  • Motivation: The Modality Gap describes systematically offset representation regions for modalities expressing identical semantics.
  • Limitations of Prior Work: Existing geometric corrections often assume isotropic residual noise and remain limited to small-scale tasks, while text-only methods degrade on fine-grained visual tasks.
  • Research Questions: The paper asks whether the modality gap can be precisely characterized beyond mean shifts and whether its shape can support scalable training with unpaired data.
  • Contributions: Fixed-frame Modality Gap Theory separates stable bias components from anisotropic residuals within dominant and orthogonal subspaces.
  • Contributions: ReAlign uses Anchor, Trace, and Centroid Alignment to map text representations into the image distribution using unpaired-data statistics, linear transformations, and normalization without training.
  • Contributions: ReVision uses ReAlign during modality-substitution pretraining on unpaired text, followed by visual instruction tuning with real images.
  • Overall Contribution: The framework presents statistically aligned unpaired data as a cost-efficient route for scaling multimodal large language models.

2 The Isotropic Fallacy

Prior modality-gap methods commonly model the residual as isotropic noise after correcting a centroid shift. The paper argues that this simplification is geometrically flawed for anisotropic multimodal representations.

  • C3 and subsequent state-of-the-art strategies characterize the gap as a centroid shift plus random alignment noise.
  • The paper identifies isotropic residual modeling as a critical simplification because multimodal contrastive representation distributions are inherently anisotropic.

3 Modality Gap

The paper analyzes the modality gap in a fixed, data-driven subspace frame and decomposes it into first-order biases, anisotropic residuals, and normalization-induced centroid drift. These components reveal structure beyond random noise or simple mean shifts.

  • Framework: A dual-encoder study tracks representation geometry during training and separates task-active and orthogonal directions using a data-driven dominant subspace.
  • Framework: The fixed frame U ⊕ V is constructed from probe covariance at a reference time and held constant while later modality-gap components are compared.
  • First-order Bias: The first-order gap is an anchor displacement decomposed into principal mean bias β(t) in U and passive orthogonal bias γ(t) in V.
  • First-order Bias: The passive orthogonal bias evolves slowly because late-stage gradients along the orthogonal complement remain weak.
  • Residual Shape: Mean alignment leaves a residual with stable anisotropic second-order structure rather than isotropic white noise.
  • Residual Shape: Within U, residual energy concentrates along a small number of principal directions, producing a stable spectral hierarchy.
  • Residual Shape: Within V, the lower-energy residual forms an elongated ellipsoid that is geometrically decoupled from the passive-bias direction.
  • Phantom Drift: Spherical normalization couples the aligned anchor with residual covariance, creating Phantom Drift and preventing Euclidean mean alignment from guaranteeing spherical centroid alignment.

4 ReAlign: Training-Free Modality Alignment

ReAlign is a training-free alignment operator that corrects the modality gap through sequential anchor, residual-scale, and spherical-centroid calibration. Its design preserves anisotropic structure while addressing normalization-induced drift.

  • Pipeline: ReAlign performs Anchor, Trace, and Centroid Alignment to correct three geometric components of the modality gap.
  • Anchor Alignment: Anchor Alignment translates the source modality to the target mean, removing first-order mismatch without destroying second-order spectral structure.
  • Trace Alignment: Trace Alignment matches global residual energy while preserving the source covariance’s eigenvectors and trace-normalized spectrum.
  • Trace Alignment: The trace-matching factor is s = Tx/(Ty + ε), scaling residual energy from the source toward the target without reshaping its spectral directions.
  • Centroid Alignment: Centroid Alignment corrects the Phantom Drift introduced when spherical projection shifts the centroid after Euclidean calibration.
  • Centroid Alignment: A second normalization returns representations to the unit hypersphere while leaving only a small residual centroid error.

5 ReVision: A Scalable MLLM Training Paradigm

ReVision is a two-stage MLLM training paradigm that uses ReAlign to turn unpaired text into pseudo-visual representations before supervised visual instruction tuning. This enables pretraining on abundant text while reserving real images for fine-grained visual capabilities.

  • ReVision combines modality substitution pretraining with subsequent visual instruction tuning.The first stage uses ReAlign-derived pseudo-visual features; the second introduces real images for supervised learning.
  • Stage 1: Modality Substitution Pretraining: During modality substitution pretraining, ReAlign converts large-scale text into pseudo-visual representations while the LLM backbone remains frozen.An adapter is trained on these representations to absorb world knowledge and visual semantics without costly image-text pairs.
  • Stage 1: Modality Substitution Pretraining: ReAlign preserves text semantics while matching the anisotropic geometric statistics of real image representations.The training-free substitution operator maps text embeddings into the image-space distribution.
  • Stage 2: Visual Instruction-Tuning: Visual instruction tuning introduces real image embeddings to recover fine-grained visual details and support complex instructions and reasoning tasks.The stage uses standard supervised instruction tuning on image-instruction pairs.
  • Inference: At inference, ReVision directly accepts real images without multi-image calibration or additional computational overhead.This follows from aligning text representations to the image representation distribution during pretraining.

6 Experiments

Experiments evaluate ReAlign’s geometric alignment and ReVision’s scalable text-only pretraining. ReAlign models anisotropic structure more precisely, while ReVision-2M substantially outperforms Unicorn among text-only methods.

  • RQ1. Geometric Alignment: ReAlign reduces the modality gap from approximately 0.4 in the original representations, while C3 reduces it substantially but remains limited by isotropic noise assumptions.The geometric evaluation uses Euclidean centroid distance on 100k Bunny-pretrain and DenseFusion samples.
  • RQ1. Geometric Alignment: ReAlign reduces the aligned-centroid gap to the 10−4 scale, whereas C3 stagnates near ≈0.0023 because of isotropic assumptions.The result is attributed to ReAlign’s modeling of anisotropic covariance.
  • RQ1. Geometric Alignment: ReAlign adaptively matches each dataset’s covariance trace, improving flexibility across representation variations and diverse data sources.Additional spectral, angular, and visualization diagnostics are reported in Appendix D.
  • RQ2. Large-scale MLLM Training: The experiments compare ReVision with Blind, No Align, and C3 Align under a common two-stage training setup.Stage 1 uses 1M Bunny-pretrain text samples, followed by InternVL-Chat-V1-2-SFT for Stage 2.
  • RQ3. Text-only Scaling: ReVision-2M scores 49.75 versus Unicorn’s 43.94 among text-only methods in the cost-performance comparison.The comparison uses 2M versus 1M unpaired text samples; the paper attributes the difference to ReVision’s statistically shaped pseudo-features.
  • RQ3. Text-only Scaling: The text-only scaling study compares paired image-text pretraining, Unicorn, ReVision-1M, and ReVision-2M.The paired setting uses 1M ground-truth image-text pairs, while the ReVision settings use unpaired text samples.

7 Conclusion

The paper frames the modality gap as a structured geometric consequence of dual encoders, spherical normalization, and dot-product contrastive training. Fixed-frame analysis motivates ReAlign and ReVision, which explicitly correct geometric misalignment and use unpaired text for scalable MLLM training.

  • Fixed-frame Theory decomposes the modality gap into stable biases and anisotropic residuals within a frozen reference frame.The decomposition separates the effective task subspace U from its orthogonal complement V.
  • ReVision uses statistically aligned unpaired text as a substitute for expensive image-text pairs, supporting scalable MLLM training.The reported framework significantly mitigates the modality gap and outperforms baselines.
  • Dual-encoder isolation creates independent modality anchors, producing a nonzero mean gap before contrastive alignment acts on final embeddings.Separate encoders process modalities independently, allowing their representation distributions to develop different centroids.
  • Spherical normalization couples anchor direction with residual covariance, so anisotropic residuals can create Phantom Drift after Euclidean mean alignment.Normalization is nonlinear, making Euclidean mean alignment insufficient for spherical centroid alignment.
  • Dot-product InfoNCE concentrates embedding-level gradients in the current contrastive-set span, leaving the orthogonal complement weakly corrected.This mechanism helps explain why the passive orthogonal bias evolves slowly rather than being rapidly optimized away.
  • ReAlign performs Anchor, Trace, and Centroid Alignment to remove mean mismatch, match global residual energy, and correct spherical centroid drift.Trace Alignment preserves the trace-normalized spectral shape while changing only the global residual energy scale.

C U--V Weak Coupling Analysis

The weak-coupling analysis refines the fixed-frame picture by modeling predictable residual leakage from U to V and separating it from subspace rotation. Its diagnostics distinguish geometric leakage, weak coupling, and unexplained remainder contributions.

  • Residual-level weak coupling: The residual model writes ζ(t) as a weak linear function of δ(t) plus an uncorrelated zero-mean background residual.The map L: U → V has bounded spectral norm and quantifies weak coupling between residual components.
  • Residual-level weak coupling: The cross-moment identity E[δ(t)ζ(t)⊤] = Σ_U L⊤ makes the coupling map identifiable from second moments when Σ_U is nonsingular on its effective support.In practice, the map is estimated in reference bases using a ridge estimator.
  • Residual-level weak coupling: A zero coupling map means V-side residuals contain no linearly predictable component from U-side residuals, whereas small nonzero coupling indicates weak dependence.The explained-variance diagnostic distinguishes largely independent residuals from predictable leakage.
  • Connection to gradient leakage: Gradient leakage combines geometric leakage from rotation between U_t and U, weak coupling through L, and an unexplained remainder r_g.When ρ = 0, leakage is fully explained by subspace rotation and the coupling map.
  • Connection to gradient leakage: In the idealized ρ = 0 case, leakage is bounded by the sum of the principal-angle baseline sin θ(U_t, U) and the weak coupling strength.The principal-angle term captures projection caused by subspace rotation.
  • Interpretation: Small estimated coupling and explained variance support interpreting the orthogonal bias as passively evolving rather than aggressively corrected by direct V-direction optimization.This diagnostic remains a refinement of the main geometric explanation, whose dominant component is instantaneous-to-fixed subspace rotation.

D.1 Spectral Analysis: Preserving Semantic Hierarchy

ReAlign preserves the source text’s spectral hierarchy and angular topology, whereas C3’s noise injection distorts both; PCA further shows ReAlign moving text representations into the visual support region.

  • Spectral hierarchy: C3 flattens the eigenspectrum to α ≈1.06, while ReAlign retains the source-text decay rate of α ≈1.33.The flatter C3 spectrum indicates reduced spectral anisotropy and diluted fine-grained semantic structure.
  • Angular topology: C3 produces a severe angular distribution shift, while ReAlign restores overlap with the target prior and preserves sample relationships.The angular comparison uses KDEs of pairwise cosine similarities and Jensen-Shannon divergence.
  • Angular topology: ReAlign achieves JS Divergence = 0.0066 versus 0.1924 for C3, indicating substantially closer angular alignment.The figure reports these divergences for ReAlign and C3, respectively.
  • Global manifold alignment: ReAlign increases k-NN mixing to 4.35%, compared with 1.31% for C3 and 0.32% for unaligned text.This corresponds to over 3× improvement over C3 and over 10× over the unaligned state.

E.1 Data Efficiency & Stability

ReAlign estimates alignment statistics efficiently from unpaired data, converges with moderate sample sizes, and scales linearly in time with constant peak memory; calibration remains domain-specific.

  • Sample efficiency: The modality gap decreases rapidly and stabilizes beyond approximately 10,000 unpaired samples.The reported convergence is based on moment estimation over repeated subset trials.
  • Numerical stability: Float32 accumulation reaches an alignment error of approximately 9.56 × 10^-8 at N = 500,000, motivating higher numerical precision.The passage contrasts single-precision accumulation with double precision for high-dimensional statistics.
  • Computational efficiency: Processing time scales as O(N), while peak memory remains approximately 48.95 MB, or O(1), as dataset size grows.Processing time increases from 0.05s for 100k samples to 0.57s for 1M samples.
  • Domain sensitivity: Cross-domain statistics transfer substantially increases the modality gap, so precise alignment requires domain-specific calibration.Both General → Medical and Medical → General transfer degrade performance relative to in-domain alignment.
  • Alignment procedure: ReAlign uses fixed-frame, closed-form alignment based on first- and second-order statistics from large unpaired datasets.Its stages anchor means, align geometry on the dominant and orthogonal subspaces, and recenter after spherical projection.

F.4 Failure Analysis: Why Fine-Grained Alignment Fails?

Fine-grained alignment fails when full covariance matching becomes numerically unstable or distorts local semantics, while long captions introduce truncation, diffuse geometry, and linguistic noise.

  • Covariance alignment failure: Blockwise Covariance Alignment underperforms ReAlign across benchmarks because numerical instability and semantic-manifold distortion degrade performance.The paper attributes the failure to covariance inversion instability and aggressive whitening.
  • Numerical instability: A covariance condition number of κ ≈1.10 × 10^3 causes whitening to amplify tail noise and destabilize minor feature directions.ReAlign’s isotropic scaling remains stable regardless of the spectral shape.
  • Semantic preservation: Blockwise whitening retains only 10.1% of local semantic neighborhoods, whereas ReAlign preserves 87.3%.The comparison uses k-Nearest Neighbor overlap with k = 10.
  • Truncation-induced mismatch: Dense captions can exceed the text encoder’s context window, leaving textual embeddings aligned to truncated prefixes rather than complete visual descriptions.The visual embedding retains global image information while the textual embedding omits content from the truncated tail.
  • Long-caption paradox: Long captions exhibit diffuse, high-rank geometry, whereas short captions provide a more compact approximation of visual content and greater statistical stability.The reported effective ranks are approximately 52.9 for long captions, 57.5 for visual embeddings, and 41.0 for short captions.
  • Long-caption paradox: Non-visual linguistic content acts as noise vectors that pull the semantic centroid away from the visual anchor and increase the modality gap by approximately 30%.The paper concludes that visual grounding and geometric compactness matter more than raw caption length for statistical alignment.

H.4 Cost Analysis

The cost analysis uses token-based input and output pricing to compare methods, identifying Unicorn as the most expensive and standard ReVision (1M) as the least expensive among the listed methods.

  • Cost formula: Total cost is computed as Ctotal = Pin · Tin + Pout · Tout using token counts and prices per million tokens.The standardized rates are Pin = $1.25 and Pout = $10.00 per million tokens.
  • Method comparison: Unicorn costs $1893.27, while standard ReVision (1M) costs $176.10 under the stated pricing assumptions.Unicorn uses 17.50M input and 187.14M output tokens; ReVision (1M) uses 11.64M input and 16.15M output tokens.

I More MLLM Experiments

Additional experiments support ReAlign’s three-stage design, the necessity of Modality Substitution Pretraining, and ReVision’s scalability with more instruction data and larger backbones. The analysis also examines qualitative visual capabilities and positions the approach as a data-efficient alternative to massive paired datasets.

  • ReAlign component ablation: Anchor, Trace, and Centroid Alignment each contribute gains, supporting ReAlign as a three-stage calibration procedure.Anchor corrects mean mismatch, Trace matches global residual energy while preserving spectral structure, and Centroid corrects drift from spherical normalization.
  • Modality Substitution Pretraining: Removing Modality Substitution Pretraining decreases the average score from 50.16 to 43.58 across all benchmarks.The ablation compares the complete pipeline with a setting that directly performs visual instruction tuning without pseudo-visual pretraining.
  • Stage-2 data scaling: Increasing Stage-2 SFT data from 30% to 60% and 100% consistently improves performance.Stage 1 remains fixed while only the amount of visual instruction-tuning data changes.
  • LLM backbone scaling: Scaling the LLM backbone from Llama-3-8B-Instruct to Llama-3-70B-Instruct raises the average score from 50.16 to 52.33.All other encoders, operators, training data, and evaluation settings remain unchanged; gains are especially visible on reasoning-intensive benchmarks.
  • Qualitative analysis: Qualitative evaluation examines ReVision across General Visual Perception, Domain-Specific Knowledge, and Logical Reasoning, including complex abstract and spatial reasoning tasks.Figure 12 presents qualitative examples of visual perception across complex tasks, including matrix pattern completion.
  • Data-efficient scaling: ReVision decouples visual-semantic knowledge injection from massive paired datasets by using unpaired text for pretraining and limited real images for SFT.The paper frames this as a data-efficient training paradigm that can lower barriers to MLLM development and support low-resource domains where paired data is scarce.
Loading 2602.07026v3…