PaperlayerView exact revision on arXiv

Source-linked AI summary

Anisotropic Modality Align

Xiaomin Yu, Yijiang Li, Yuhui Zhang, Hanzhen Zhao, Yue Yang, Hao Tang, Yue Song, Xiaobin Hu, Chengwei Qin, Shuicheng Yan, Hui Xiong

arXiv:2605.07825v1cs.MMcs.CV

TL;DR

Unpaired multimodal alignment lacks a systematic account of whether modality representations share usable geometry and how their remaining gap should be corrected. The paper characterizes the gap as anisotropic residual structure and proposes AnisoAlign, which preserves source semantics while matching the target distribution; experiments show benefits in geometric diagnostics and text-only MLLM training.

  • Problem

    Existing unpaired alignment methods lack a systematic characterization of whether modality discrepancies reflect incompatible geometry, global shifts, or structured directional residuals.

  • Method

    AnisoAlign performs structured geometric correction that preserves source semantic geometry while correcting dominant anisotropic residuals toward the target-modality distribution.

  • Results

    AnisoAlign achieves the highest average score of 47.49 in fully text-only MLLM training, while diagnostics identify compatible dominant geometry and low-dimensional anisotropic residuals between modalities.

  • Takeaways & Limitations

    The modality gap is a structured, correctable geometric phenomenon, enabling substitute target-modality representations without reliance on paired image-text data.

Abstract

from arXiv · show

Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data. Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data. However, the key premise of this paradigm remains insufficiently understood: can representations from different modalities be reliably interchanged? The core obstacle lies in the persistent Modality Gap in the shared space. In this work, we revisit the geometric nature of the modality gap. We find that modality representations already share compatible dominant semantic geometry. What truly hinders modality interchangeability is not a simple global shift, but an anisotropic residual structure concentrated along a small number of dominant directions. Based on this finding, we further propose the principle of anisotropic modality gap alignment: effective modality alignment should align with the target-modality distribution while preserving the semantic structure of the source modality. Guided by this principle, we propose an anisotropic geometric correction framework, AnisoAlign, for unpaired modality alignment. This framework leverages the internal geometric prior of the target modality and performs bounded correction on source-modality representations, thereby constructing substitute representations in the target modality. Experiments confirm its benefits in both geometric diagnostics and text-only MLLM training. Overall, this work recasts the modality gap from an empirical observation into a correctable, structured geometric phenomenon and provides a new representation alignment perspective for training multimodal models with unimodal data.

1 Introduction

The paper reframes the modality gap as a structured geometric phenomenon: image and text representations share dominant semantic geometry, while their residual discrepancy is anisotropic and concentrated along a few directions. It introduces AnisoAlign to correct those directions while preserving source semantics and matching the target-modality distribution for unpaired multimodal training.

  • Geometric diagnosis: Image and text representations share compatible dominant semantic geometry, with similar covariance-spectrum long-tail decay and principal-subspace overlap above a random baseline.These diagnostics indicate that the modalities are not arbitrary or unrelated distributions in the shared space.
  • Geometric diagnosis: Global centroid shifting does not remove most cross-modal discrepancy; mean-corrected residuals form anisotropic structure concentrated along a few dominant directions.The residual is therefore low-effective-dimensional rather than isotropic noise.
  • Alignment principle: Effective alignment must preserve source-modality semantic geometry while correcting dominant residual directions so representations enter the target-modality distribution.The paper argues that minimizing global distributional discrepancy alone is insufficient.
  • AnisoAlign: AnisoAlign performs unpaired alignment through a fixed dominant-subspace decomposition and blockwise polar parameterization of radius and phase within that subspace.The framework explicitly models anisotropic geometric variations along dominant directions.
  • Empirical support: Experiments show that AnisoAlign improves target-geometry matching while preserving source semantics, and its substitute representations strengthen fully text-only MLLM training and text-only pretraining before visual instruction tuning.The results support using large-scale text-only data as a substitute for paired image-text supervision.

2 Preliminaries

The preliminaries define the modality gap as geometric discrepancy between semantically corresponding representations and formulate modality alignment as unpaired mapping into the target modality while preserving semantics.

  • Definition 2.1: Modality Gap: The modality gap occurs when cross-modal representations share latent semantics but need not coincide geometrically in the normalized shared space.For x ∈ X and y ∈ Y, σ(x) = σ(y) even when x and y differ geometrically.
  • Definition 2.2: Modality Align: Modality alignment seeks an unpaired mapping T that preserves semantic labels and makes transformed source representations match the target modality’s conditional distribution.The transformed T(y) is called a substitute representation in the target modality.

3 Modality Gap

The modality gap is not caused by unrelated global geometry or a simple centroid shift: image and text representations share dominant semantic structure, while their mismatch is a structured anisotropic residual concentrated in a low-effective-dimensional subspace. Effective alignment should therefore preserve source semantic geometry while correcting dominant residual directions for target-distribution compatibility.

  • Compatible Dominant Geometry: Image and text representations share compatible dominant geometry, including similar variance spectra and non-random principal directions.The spectral correlation is Cλ = 0.845; at q = 128, overlap is O128 = 0.441 versus random baseline q/d = 0.100.
  • Anisotropic Residual: The residual lies in a low-effective-dimensional subspace, with anisotropy ratio Ar = 28.6 and effective-dimension ratio deff/d = 0.284.These measurements indicate concentrated residual energy rather than isotropic noise.
  • Centroid Bias Is Insufficient: Mean correction is insufficient because covariance-shape mismatch remains and the corrected paired distance stays high.Centroid displacement is Gµ = 0.392, covariance-shape discrepancy is GΣ = 0.066, corrected paired distance is eD = 1.264, and residual ratio is eD/D = 0.89.
  • Anisotropic Residual: The remaining gap is anisotropic rather than isotropic, with residual energy concentrated along a small number of dominant directions.The residual spectrum shows dominant eigen-directions above the isotropic baseline and a long-tail decay.
  • Anisotropic Modality Alignment: Effective modality alignment should preserve source semantic geometry while correcting dominant anisotropic residual directions for target-distribution compatibility.The diagnostic transformations include identity, centroid, moment, random target replacement, and controlled interpolation; random replacement matches the target distribution but destroys semantic correspondence.

4 AnisoAlign

AnisoAlign establishes a fixed dominant-subspace geometry, models anisotropic residuals with blockwise polar coordinates, and learns a frozen target-modality phase prior from images. It then transforms text embeddings through global initialization, bounded instance-conditioned refinement, prior matching, and final centroid calibration.

  • Shared geometric frame: AnisoAlign decomposes embeddings into a shared dominant subspace U and orthogonal complement V using the top-r eigenvectors of Σ = Σt + Σi + λI.The decomposition provides a fixed geometric frame, with U capturing dominant statistical directions and V containing the remaining component.
  • Anisotropic polar representation: A continuous orthogonal mixing matrix R removes arbitrary dependence on eigenvector ordering before U is represented as m = r/2 two-dimensional blocks.Each block is converted into radius and phase coordinates, yielding blockwise radii ρ and phases θ.
  • Stage I: target prior: Stage I estimates image phase marginals and cross-block phase dependencies, then trains a phase-aware score network using image-derived perturbed phase samples.The resulting phase score prior is determined by the target image distribution and remains frozen for Stage II.
  • Stage II: bounded refinement: Stage II initializes each text embedding by mean recentering, radial distribution transfer, and V-subspace initialization before predicting residual corrections for phase, radius, and V.The refinement retains only the V-component of the residual correction and bounds updates with wrapping, positivity, and tanh parameterizations.
  • Target-compatible reconstruction: The refined phase is constrained by the frozen target prior and sparse relative-phase graph, while the reconstructed representation undergoes normalization and final global centroid calibration.The calibrated embedding is used as the final substitute representation in the target modality.

5 Experiments

Experiments evaluate AnisoAlign through geometric diagnostics, source-semantic preservation, text-only MLLM training, scalability, and ablations. Across these settings, AnisoAlign improves target-geometry compatibility, preserves source semantics, and achieves strong downstream performance.

  • Target-geometry compatibility: AnisoAlign achieves near-zero centroid discrepancy, balanced local support matching, and a weaker anisotropic residual than Text and C3.Text has ∆µ = 0.393, C3 has ∆µ = 0.276, while ReAlign and AnisoAlign reduce it to about 0.012; ReAlign obtains M Z_k = 0.337 versus C3’s 0.075.
  • Source-semantic preservation: AnisoAlign achieves the best performance across instance consistency, relative geometry consistency, and neighborhood consistency.C3 scores approximately 0.899, 0.925, and 0.840 on Φ, Ψ, and Ωk, respectively, while introducing noticeable disruption to local neighborhood structures.
  • Fully text-only MLLM training: AnisoAlign provides substitute visual representations for fully text-only MLLM training without using image-text pairs throughout training.All methods use pretraining on Unicorn-1.2M followed by instruction-tuning on Unicorn-Instruction-417K.
  • Text-only pretraining before visual instruction tuning: AnisoAlign achieves the highest average score of 51.59 in text-only pretraining followed by visual instruction tuning.It outperforms ReAlign (50.16), C3 Align (48.06), and W/o. Align (47.50), using Bunny-pretrain text samples and InternVL-Chat-V1.2-SFT for visual instruction tuning.
  • Ablation studies: The full AnisoAlign model obtains the best ablation average score of 47.49, outperforming all ablated variants.Global initialization scores 43.59, instance-conditioned refinement raises it to 44.93, target-side geometric guidance reaches 46.56, and adding LΦ reaches 46.45.

6 Conclusion

The paper reframes the modality gap as a structured anisotropic residual built on compatible semantic geometry. It introduces anisotropic modality alignment and an unpaired method that generates target-modality substitute representations, reducing reliance on paired image-text data.

  • 6 Conclusion: The modality gap is a structured anisotropic residual built upon compatible semantic geometry.The paper revisits the gap from a geometric perspective.
  • 6 Conclusion: The paper proposes the principle of anisotropic modality alignment.
  • 6 Conclusion: The proposed unpaired modality alignment method generates target-modality substitute representations.
  • 6 Conclusion: Experiments show that the method can help eliminate reliance on paired image-text data.

A Theoretical Derivation of the Anisotropic Modality Gap … A.5 Efficiency of Dominant Residual-Direction Correction

The appendix formalizes the modality gap as a centroid displacement plus a structured centered residual, showing that modalities share a dominant geometric backbone while residual energy is anisotropic and concentrated in a few directions. These results motivate bounded corrections that preserve source semantic geometry and optimally target dominant residual directions.

  • A.1 Overview and Notation: Paired representations are used for geometric diagnostics, while the proposed alignment method does not require paired supervision.The framework defines modality means, centered variables, covariance matrices, cross-modal moments, eigenvalues, principal eigenvectors, and projectors in the shared space.
  • A.2 Formalizing Dominant Geometric Compatibility: Principal-subspace overlap Oq ≫ q/d shows that the modalities share non-random dominant geometric directions, supporting alignment that preserves source semantic geometry.Spectral compatibility alone is insufficient because similar eigenvalue spectra can correspond to nearly orthogonal eigenspaces; principal subspaces must also be compared.
  • A.3 Mean--Residual Decomposition: The expected squared cross-modal discrepancy decomposes into centroid displacement and centered residual discrepancy, so centering isolates structure unexplained by a global mean shift.The residual depends on both marginal covariance structures and cross-modal correspondence structure Σxy.
  • A.3.2 Residual after Centroid Correction: A large residual ratio rejects the explanation that the modality gap is primarily a centroid bias.If centroid correction were sufficient, tr(Σr) would become small; a large remaining residual indicates otherwise.
  • A.4.1 Null Hypothesis: Under the isotropic residual null hypothesis, all residual covariance eigenvalues are equal and residual energy is uniformly distributed across directions.The null hypothesis provides the reference for testing whether centered residuals contain dominant geometric structure.
  • A.4.2 Residual Anisotropy Ratio: An empirical anisotropy ratio Ar ≫ 1 indicates that some directions carry residual energy far above the average, contradicting isotropic noise.By definition, Ar ≥ 1, while the isotropic null gives Ar = 1.
  • A.4.3 Cumulative Spectral Energy: If E(K) ≫ K/d for small K or deff(Σr)/d ≪ 1, residual energy is concentrated in a small number of dominant directions rather than spread isotropically.Under isotropy, the effective dimension equals d and top-K cumulative energy follows K/d; the reported residual spectrum satisfies these concentration conditions simultaneously.
  • A.5 Efficiency of Dominant Residual-Direction Correction: When correction is restricted to a K-dimensional subspace, selecting the top K residual eigen-directions minimizes the remaining squared residual energy.This establishes the efficiency principle behind dominant residual-direction correction.

A.5.1 Optimal Projection Result

The optimal rank-constrained correction removes residual components in the subspace spanned by the top K eigenvectors of the residual covariance. This choice minimizes expected remaining residual energy by maximizing the projected residual variance.

  • Optimization criterion: Minimizing the remaining residual energy is equivalent to maximizing tr(PΣr).The projector P is constrained to have rank K and be orthogonal.
  • Optimal rank-constrained correction: The rank-K orthogonal projector onto the top K eigenvectors of Σr minimizes expected remaining residual energy.This is the stated optimal rank-constrained residual correction.
  • Optimality proof: The optimum follows from the Ky Fan maximum principle and is attained by the projector onto the top-K eigenspace of Σr.Substitution into J(P) = tr(Σr) − tr(PΣr) gives the corresponding minimum objective value.

A.5.2 Comparison with Random Correction … A.7 Semantic Preservation under Bounded Correction

The paper shows that correcting dominant residual directions is more efficient than random correction, while an unpaired joint-covariance surrogate offers a computable alternative. It further establishes that target-distribution matching alone is non-identifying, motivating bounded correction to preserve source semantics.

  • A.5.2 Comparison with Random Correction: Dominant residual directions remove more residual energy than random rank-K subspaces, with the advantage increasing as residual anisotropy strengthens.When E(K) ≫ K/d, dominant-direction correction is substantially more efficient than random directions or isotropic perturbations; for K = 1, the gain is exactly specified in the analysis.
  • A.5.3 From the Residual Principal Subspace to the Joint Covariance Subspace: In unpaired alignment, residual covariance cannot be directly used because its cross-modal term Σxy is not reliably estimable from unpaired samples.The oracle correction target is the principal subspace of Σr, but constructing it requires cross-modal correspondence.
  • A.5.3 From the Residual Principal Subspace to the Joint Covariance Subspace: The method instead uses the top r eigenvectors of the joint marginal covariance as a computable surrogate correction subspace based only on marginal statistics.This surrogate is not claimed to be strictly identical to the principal subspace of Σr.
  • A.5.3 From the Residual Principal Subspace to the Joint Covariance Subspace: The joint surrogate is motivated by shared dominant geometric structure and residual-gap concentration, and its residual-energy coverage ratio ηU assesses whether it captures residual energy.A high ηU supports using the joint dominant subspace as an unpaired surrogate correction subspace.
  • A.5.4 Implication: The analysis concludes that dominant residual correction is rank-constrained optimal in the oracle setting, whereas joint marginal covariance supplies the practical unpaired surrogate.Its use is motivated by observed dominant geometric compatibility and the structured residual gap.
  • A.6 Non-identifiability of Distribution Matching Alone: Matching the target marginal distribution alone does not identify a semantics-preserving alignment map.If T0 matches the target distribution, composing it with any target-distribution-preserving transformation S yields another equally valid marginal match.
  • A.6 Non-identifiability of Distribution Matching Alone: Such target-preserving transformations can arbitrarily permute or distort instance-level semantic correspondence, so effective alignment must also preserve source semantic geometry.Random target replacement can match the target distribution while destroying correspondence with the original source sample.
  • A.7 Semantic Preservation under Bounded Correction: The next subsection addresses this limitation by showing that bounded correction controls distortion of the source-modality semantic structure.This follows the result that target-distribution matching alone does not guarantee semantic preservation.

A.7.1 Similarity Preservation under Additive Perturbation … B.1 Setting

The paper shows that bounded corrections preserve source similarity while enabling controlled movement toward the target distribution. It further derives a periodic phase prior from target-modality geometry and evaluates aligned text representations as substitute visual tokens in a two-stage MLLM pipeline.

  • A.7.1 Similarity Preservation under Additive Perturbation: With unit-norm source representations and correction norm at most ε, similarity changes remain bounded: representation deviation is at most 2ε/(1−ε), while inner-product deviation is at most 4ε/(1−ε).Thus, controlled correction cannot arbitrarily distort the source modality’s pairwise inner-product structure.
  • A.7.2 Connection to Stage-II Bounded Residual Refinement: Stage II applies bounded corrections to phase, radius, and V-subspace components; when ε_eff < 1, Lemma A.4 directly guarantees similarity preservation.Without explicit V-side clipping, the guarantee is conditional on monitoring and controlling the realized correction norm through α_v, regularization, or early stopping.
  • A.7.3 Implication: Larger α_θ, α_ρ, and α_v permit stronger geometric correction but risk source-semantic distortion, whereas smaller scales preserve source geometry but may not reach target-modality support.The bounded parameterization therefore tunes the trade-off between semantic preservation and target-distribution compatibility.
  • A.8 Geometric Motivation for the Periodic Phase Prior; A.8.1 Two-Dimensional Blockwise Polar Decomposition: The method uses two-dimensional polar blocks in the dominant subspace because radial coordinates encode block energy, while periodic phase coordinates encode directional structure.Phase lies on [−π, π), motivating wrapped rather than ordinary Euclidean noise models.
  • A.8.2 Phase Marginals and Phase Couplings: The target phase prior captures marginal phase anchors and pairwise phase-difference dependencies, summarized by ψ̄_k, α_k, A_kℓ, and η_kℓ.These statistics represent low-order internal geometry of the target image modality in phase space.
  • A.8.3 Periodic Potential and Drift Field: Its periodic potential attracts each phase toward target marginal anchors and encourages inter-block phase differences to match target dependencies, yielding a local geometric constraint field.The resulting drift field is constructed from internal phase statistics of the target image modality.
  • A.8.4 Wrapped Gaussian Score Prior: The wrapped-Gaussian score prior models periodic phase structure directly, learning the target modality’s internal phase geometry rather than a text-to-image mapping.Perturbed phases are wrapped back to [−π, π), and the score network predicts the wrapped Gaussian score in phase space.
  • A.8.5 Implication; B Experiment Details; B.1 Setting: Stage I therefore supplies a target-modality geometric constraint for Stage-II bounded correction, while experiments use marginal-only statistics to prevent pairwise correspondence leakage and feed aligned text as substitute visual tokens.The MLLM setup uses Llama-3-8B-Instruct with a two-layer GELU MLP projector in a two-stage training pipeline.

B.2 Metrics · B.3 Baselines

B.2 evaluates alignment through source-semantic preservation, target-modality mixing, and residual anisotropy rather than downstream labels. B.3 compares against Unicorn, C3 Align, and ReAlign, which provide synthesis, mean-based correction, and statistical alignment baselines.

  • B.2 Metrics: Source semantic consistency Φ(T) measures cosine similarity between each original source representation and its transformed substitute, with larger values indicating closer semantic positions.The original source representations Y serve as the semantic reference, while transformed representations are Z = T(Y).
  • B.2 Metrics: Relative-geometry consistency Ψ(T) measures correlation between original and transformed pairwise similarities, so larger values indicate more stable source semantic relations.The metric is computed over a randomly sampled pair set P.
  • B.2 Metrics: Local neighborhood consistency Ωk(T) compares k-nearest-neighbor sets before and after transformation, with larger values indicating better preservation of local semantic neighborhoods.The comparison uses the same samples in the original source space Y and transformed space Z.
  • B.2 Metrics: Directional mixing scores MZk(T) and MXk(T) assess whether transformed sources enter target support and whether transformed sources cover target-modality support, respectively.They use normalized binary entropy of local modality-origin proportions in the joint set Q = X ∪ Z.
  • B.2 Metrics: Residual anisotropy ratio Ar(T) evaluates concentration along dominant residual directions, with smaller values indicating better suppression of anisotropic residual structure.The ratio uses the largest residual covariance eigenvalue relative to the residual dimension.
  • B.3 Baselines: Unicorn is a text-only VLM data-synthesis baseline that expands captions, generates instruction-tuning data, and transfers LLM2CLIP text representations into visual representations.Its three-stage pipeline constructs multimodal training data without real images.
  • B.3 Baselines: C3 Align is a training-free baseline that subtracts modality-specific means and injects Gaussian noise under the Connect-Collapse-Corrupt principle.It assumes related concepts already occupy a shared representation space after multimodal contrastive learning.
  • B.3 Baselines: ReAlign is a training-free statistical baseline that maps source representations to the target distribution through closed-form anchor, trace, and centroid alignment steps.It estimates low-order statistics from unpaired data, removing mean bias and rescaling centered source residuals.

B.4 Evaluation Setting · C Applicability

The evaluation spans general perception, complex reasoning, and hallucination benchmarks, using accuracy as a unified metric. AnisoAlign assumes source and target modalities occupy a shared normalized space with compatible dominant geometry and structured anisotropic residual differences.

  • B.4 Evaluation Setting: The evaluation covers three visual-understanding aspects: general perception, complex reasoning, and hallucination assessment.These categories organize the benchmark suite.
  • B.4 Evaluation Setting: General perception is evaluated with MME, MMStar, ScienceQA-image, and RealWorldQA.ScienceQA uses its image dev&test split.
  • B.4 Evaluation Setting: Complex reasoning is evaluated with single-image MMMU validation, single-image MMMU-Pro, VisuLogic train, and LogicVista.The listed benchmarks target complex reasoning under the specified splits.
  • B.4 Evaluation Setting: Hallucination assessment uses CRPE, POPE, and HallusionBench.These benchmarks constitute the hallucination-focused evaluation group.
  • B.4 Evaluation Setting: Accuracy (acc) is reported across all benchmarks as the unified evaluation metric for consistent method comparison.The same metric is used throughout the evaluation suite.
  • C Applicability: The analysis and method assume source and target modalities are embedded in a shared normalized representation space from a pretrained multimodal contrastive encoder.This shared space is the foundation of the modality-alignment setting.
  • C Applicability: Within this space, the modalities are assumed to share dominant geometric structure, while their remaining discrepancy forms a structured anisotropic residual.The applicability premise treats the modality gap as occurring inside an already semantically compatible space.
Loading 2605.07825v1…