PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMs

Yihang Du, Juhao Liang, Zhengzhao Lai, Siyu Li, Yan Hu

arXiv:2608.15085v1cs.CL

TL;DR

Non-English visual reasoning in MLLMs remains weaker despite multilingual text backbones, and the multimodal mechanisms behind this gap are insufficiently understood. The paper identifies Ghost Anchor modality asynchrony and proposes ANCHOR, which improves cross-lingual visual reasoning across fine-tuned and zero-shot languages.

  • Problem

    The multimodal mechanisms underlying MLLMs’ substantial non-English visual-reasoning degradation and English-dependent alignment remain insufficiently understood.

  • Method

    The paper uses layer-wise mechanistic analysis and controlled interventions to identify Ghost Anchor, then trains ANCHOR with Proactive Visual Anchoring to accelerate early visual semantic emergence.

  • Results

    ANCHOR consistently improves cross-lingual visual reasoning across xMMMU, MaXM, and CVQA, including zero-shot Japanese xMMMU accuracy rising from 32.0% to 34.4% on 7B.

  • Takeaways & Limitations

    Early visual anchoring can restore visual influence during linguistic translation and support more robust cross-lingual visual reasoning in the evaluated MLLMs.

  • Takeaways & Limitations

    The English Semantic Manifold is an empirical property of the studied English-dominant MLLMs, not a universal semantic space, and the mechanistic metrics measure decodability rather than necessarily usage.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) exhibit substantial performance degradation in non-English visual reasoning, despite the strong multilingual competence of their text-only backbones. While mechanistic evidence from text-only models suggests that non-English inputs are routed through an English-centric latent space, the multimodal implications of this phenomenon remain unexplored. Through rigorous mechanistic analysis, we identify the \textbf{Ghost Anchor} phenomenon: a temporal modality asynchrony where linguistic translation to the English semantic manifold completes in early layers, while visual semanticization remains immature. Consequently, visual signals are physically present yet functionally invisible during the early alignment window. To rectify this, we propose \textbf{ANCHOR}, a training framework employing Proactive Visual Anchoring (PVA) to accelerate early visual semantic emergence, ensuring visual representations proactively guide linguistic translation. Mechanistic interventions confirm that ANCHOR successfully restores the causal influence of visual signals during early translation. Furthermore, extensive experiments on XMMMU, MaXM, and CVQA demonstrate that ANCHOR consistently outperforms standard baselines, achieving robust visual reasoning across both fine-tuned and zero-shot languages.

1 Introduction

MLLMs perform substantially worse on non-English visual reasoning because linguistic translation reaches an English-centric semantic space before visual semantics mature. The paper identifies this Ghost Anchor failure and introduces ANCHOR with Proactive Visual Anchoring to restore early visual influence.

  • Motivation: MLLMs excel in English but exhibit substantial performance degradation in non-English visual understanding.
  • Motivation: Visual signals provide limited assistance for cross-lingual alignment despite the expectation that vision should serve as a universal semantic bridge.
  • Ghost Anchor: Linguistic representations converge toward the English semantic manifold in early layers, while visual grounding remains semantically immature until much deeper in the network.
  • Ghost Anchor: The Ghost Anchor phenomenon makes visual signals physically present but semantically invisible during early alignment, causing a visual-agnostic translation path that degrades non-English visual reasoning.
  • ANCHOR: ANCHOR accelerates visual semantic emergence through Proactive Visual Anchoring, encouraging visual signals to become grounded early and influence linguistic translation.The framework explicitly supervises early-layer visual representations using an external visual foundation model and improves cross-lingual visual performance on xMMMU, MaXM, and CVQA.

2 Related Work

Prior work explains cross-lingual alignment mechanistically in text-only multilingual models and traces non-English processing through English-centric internal representations. Multilingual MLLMs have instead progressed from explicitly aligned pretraining objectives toward scale-first pipelines that expect translated or synthetic instructions to induce visual competence implicitly.

  • Mechanistic Perspectives on Cross-lingual Alignment: Text-only studies show semantically equivalent expressions converge in hidden space, supporting cross-lingual transfer.Mechanistic analyses use logit-lens and causal-tracing probes to investigate this alignment.
  • Mechanistic Perspectives on Cross-lingual Alignment: Many multilingual LLMs process non-English inputs through an English-centric internal representation space.This finding comes from mechanistic analyses of multilingual language models.
  • Multilingual Large Vision Language Models: Early multilingual vision-language models explicitly optimized cross-lingual and cross-modal objectives during pretraining.Examples include M3P, UC2, and related BERT-era frameworks.
  • Multilingual Large Vision Language Models: Mainstream multilingual MLLMs increasingly use scale-first pipelines that expect translated or synthetic instruction mixtures to induce multilingual visual competence implicitly.This shift characterizes the LLM era of multilingual MLLMs.

3 Unveiling Modality Asynchrony: A Mechanistic Analysis

This section shows that multilingual text translates toward the English semantic manifold in layers 1–5, while visual semanticization remains delayed until beyond layer 20. Their temporal mismatch produces the Ghost Anchor phenomenon, in which visual tokens are present but exert limited influence during early linguistic alignment.

  • Mechanistic Framework: The analysis tracks the independent trajectories and interaction of multilingual text and visual signals within MLLM representations.Both streams progressively converge toward a shared English semantic manifold but follow distinct paths.
  • Early Linguistic Translation: Source-language representations rotate toward English and decode into English vocabulary as early as layers 1–5.English Similarity (Simen) and English Translation Ratio (RET) reveal rapid convergence in the initial layers.
  • Delayed Visual Semanticization: Visual Grounding Score (SVG) remains near-zero through early-to-middle layers and rises only beyond layer 20.Visual semanticization from Svis toward Sen is therefore slower than intra-lingual translation.
  • Ghost Anchor Phenomenon: The Ghost Anchor phenomenon occurs when linguistic representations settle into Sen while visual tokens remain near Svis and primarily encode low-level sensory features.Visual tokens are physically present but provide limited assistance during early alignment.
  • Visual Causal Influence: The study tests whether early linguistic translation depends on visual context by replacing visual inputs with matched Gaussian noise.The intervention preserves input dimensions, patch count, and token positions while disrupting recognizable objects and scene structure.

4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring

ANCHOR addresses modality asynchrony by supervising early visual representations so they become semantically ready before cross-modal alignment closes. Its PVA component uses a visual foundation model as a semantic target and shifts visual semantic emergence earlier.

  • 4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring: ANCHOR synchronizes text and vision by supervising the early-layer evolution of visual representations rather than relying only on external data-centric alignment.PVA is designed to ensure visual features actively guide linguistic translation before the critical alignment window closes.
  • 4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring: PVA supervises visual semantics at L_early because standard training may lose fine-grained visual information before semantic verbalization completes and the alignment window closes.SigLIP-SO400M/14 provides visually grounded semantic features as a shared reference across target languages.
  • 4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring: PVA compares projected MLLM visual-token states at layer L_early with fine-grained semantic features from SigLIP-SO400M/14 using negative mean cosine similarity.The alignment network is a two-layer GELU perceptron mapping MLLM hidden states into the VFM feature space.
  • 4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring: The combined objective adds PVA anchoring to multilingual visual instruction tuning, with full-parameter updates limited to the multimodal projector and first L_early transformer layers.The vision encoder and remaining transformer layers are frozen, while the alignment network is trained; an ablation finds this early-window restriction better preserves downstream performance.
  • 4 ANCHOR: Rectifying Modality Asynchrony through Proactive Visual Anchoring: PVA shifts visual semantic emergence earlier in model depth, making grounded visual evidence available to influence linguistic alignment.This directly addresses the timing mismatch between visual semanticization and linguistic translation.

5 Empirical Results and Analysis

ANCHOR is evaluated against visual-intervention baselines across multilingual benchmarks, covering both SFT and unseen languages. Results indicate preserved English ability, stronger target-language and zero-shot reasoning, broader MLLM generalization, and restored early visual causal influence.

  • Evaluation Setup: ANCHOR is compared with Std. LoRA and Early-Layer FFT, isolating the contribution of PVA to early-layer visual grounding.Std. LoRA omits visual-side intervention, while Early-Layer FFT fine-tunes the first Learly layers without the PVA objective.
  • Evaluation Setup: Evaluation spans xMMMU, MaXM, and CVQA across languages used in SFT and languages unseen during training.The benchmarks target expert-domain semantic reasoning, low-resource cross-lingual transfer, and culturally nuanced visual mappings, respectively.
  • Macroscopic Results: ANCHOR generally preserves English proficiency, substantially improves target SFT languages, and strengthens zero-shot cross-lingual performance.On 7B xMMMU, zero-shot Japanese improves from 32.0% to 34.4%; on 13B CVQA, Arabic rises from 38.4% under standard early-layer adaptation to 40.1% with ANCHOR.
  • Macroscopic Results: ANCHOR consistently improves over Std. LoRA on LLaVA-NeXT and InternVL3 across English, SFT-language, and zero-shot groups.The evaluations use corresponding training and evaluation settings, with detailed results reported for xMMMU, MaXM, and CVQA.
  • Mechanistic Validation: ANCHOR restores early visual causal influence, whereas baseline visual influence remains limited and near zero in early layers.Baseline ΔRET averages −0.016% for 7B and 1.18% for 13B; ANCHOR shifts both ΔRET and ΔSimen positively across model scales.

6 Limitations

The analysis treats the English Semantic Manifold as an empirical property of the studied English-dominant MLLMs, while noting that semantic geometries and metric interpretations may vary across models.

  • Model and metric scope: The English Semantic Manifold is empirical to the studied English-dominant MLLMs, not a universal or normative semantic space.Models with different language distributions or architectures may exhibit different intermediate semantic geometries.
  • Model and metric scope: The logit lens, English Translation Ratio, and Visual Grounding Score measure semantic-information decodability rather than directly establishing a universal semantic geometry.

7 Conclusion · A Experimental Setup for Mechanistic Analysis

The paper explains limited cross-lingual visual assistance through Ghost Anchor modality asynchrony and introduces ANCHOR to address it. The provided passage does not report the experimental setup details.

  • 7 Conclusion: The work investigates why visual signals provide limited assistance as cross-lingual bridges for multilingual MLLMs.
  • 7 Conclusion: Mechanistic analysis identifies Ghost Anchor as a phenomenon associated with Modality Asynchrony.
  • 7 Conclusion: Linguistic streams quickly converge to an English-centric semantic manifold in early layers.
  • 7 Conclusion: Visual semantics emerge later than linguistic streams during cross-modal processing.
  • 7 Conclusion: This temporal mismatch limits the influence of visual inputs during early cross-modal alignment.
  • 7 Conclusion: The paper proposes ANCHOR to address the identified modality-asynchrony problem.
  • A Experimental Setup for Mechanistic Analysis: The provided passage contains no substantive description of the experimental setup for mechanistic analysis.

A.1 Input Data Construction … B Datasets

The study constructs strictly monolingual multilingual visual inputs from COCO images and analyzes layer-wise text–vision dynamics using complementary representation and intervention-based metrics. It separately tracks English alignment, lexical translation, visual grounding, and the effect of removing recognizable visual semantics.

  • A.1 Input Data Construction: 100 COCO 2017 validation images are paired with English captions translated into seven target languages using GPT-5.1.The parallel corpus is designed to preserve semantic consistency across languages.
  • A.1 Input Data Construction: Each input contains only visual tokens, a target-language instruction, and a target-language caption to prevent English-prior leakage.The instruction asks the model to verify whether the description matches the image.
  • A.2 Hidden State Extraction: Layer-wise hidden states are split into multilingual text tokens and foreground-object visual tokens identified from COCO segmentation masks.Ambiguous background patches are excluded from the visual subset.
  • A.3 Metrics Overview: The analysis uses English Similarity (Simen), English Translation Ratio (RET), and Visual Grounding Score (SVG), with logit lens projections for vocabulary-level analysis.Simen and RET characterize the text stream, while SVG characterizes the vision stream.
  • A.3.1 English Similarity (Simen): Simen measures cosine alignment between source-language and structurally matched English sequence representations, with low initial values indicating geometric separation from English.The compared sequences contain corresponding instructions and captions, although token counts may differ.
  • A.3.2 English Translation Ratio (RET): RET measures the proportion of linguistically meaningful source tokens whose top-5 logit-lens candidates include a word classified as English.GPT-4o excludes language-agnostic symbols and determines whether candidates belong to a natural language.
  • A.3.3 Visual Grounding Score (SVG): SVG measures the successful semantic hit rate between top-5 decoded candidates from labeled visual tokens and ground-truth object labels.A candidate counts when it matches or forms a valid substring of an object label, directly measuring visual semantic verbalization.
  • A.4 Noise Intervention: Implementation Details: The Gaussian-noise intervention preserves sequence length, token positions, dimensions, patch count, and per-channel first- and second-order statistics while reducing recognizable visual semantics.It removes recognizable contours, textures, and scene structure without assuming that every decodable visual signal disappears.

B.1 Training Datasets … C.2 Full Results on the CVQA Benchmark

The paper builds a filtered multilingual visual instruction-tuning dataset, evaluates ANCHOR across three multilingual multimodal benchmarks, and reports mechanistic and benchmark results demonstrating cross-architecture generalization. These evaluations cover xMMMU, MaXM, and culturally diverse CVQA settings, including language–region pairs and unseen languages.

  • B.1 Training Datasets: The training dataset samples and filters image–text pairs from ShareGPT4V to reduce machine-translation noise and annotation artifacts.The filtering mechanism prioritizes text length, annotation quality, and structurally straightforward sentences.
  • B.2 Details of Evaluated Multilingual Multimodal Benchmarks: The evaluation spans xMMMU, MaXM, and CVQA to assess multilingual visual understanding and reasoning across linguistic and cultural contexts.These are three distinct multilingual multimodal benchmarks used to comprehensively evaluate the proposed method.
  • B.2 Details of Evaluated Multilingual Multimodal Benchmarks: xMMMU tests complex multimodal reasoning across academic subjects, languages, and modalities using 183 subfields and 30 image types.Its questions include specialized visual content such as charts, diagrams, and chemical structures.
  • B.2 Details of Evaluated Multilingual Multimodal Benchmarks: MaXM is a test-only open-ended visual question-answering benchmark with 2K culturally contextualized questions across seven languages.Its images come from regions where the corresponding languages are spoken, and answers match the question language.
  • B.2 Details of Evaluated Multilingual Multimodal Benchmarks: CVQA evaluates culturally diverse multimodal reasoning with over 21K multiple-choice questions spanning 39 language–region pairs, 31 languages, and 30 countries.The benchmark is designed to reflect global cultural nuances through broad linguistic and regional coverage.
  • C.1 Mechanistic Results on More MLLMs: Across Qwen2.5-VL, Gemma 3, and LLaVA-NeXT, visual intervention produces small early-stage changes in ∆RET and ∆Simen, supporting Ghost Anchor’s architectural generality.Table 4 averages these effects over the early translation stage, L ∈[1, 10].
  • C.2 Full Results on the CVQA Benchmark: The additional benchmark tables report ANCHOR accuracy differences against Std. LoRA on xMMMU, MaXM, and CVQA, including macro-averages across language groups or language–region pairs.The CVQA results additionally distinguish language–region pairs whose languages were included in SFT from languages unseen during training, with Avg. computed across all 39 pairs.

C.3 Ablation Study

The ablation study shows that ANCHOR’s target layer is critical for cross-modal alignment, with Learly = 10 producing the strongest downstream performance. Targeted early-layer intervention also outperforms full-parameter fine-tuning with the same PVA constraint.

  • Target Layer (Learly): Learly defines the early translation window during which cross-modal alignment must occur, making its selection critical to ANCHOR.The study ablates different target-layer values to assess their impact on downstream reasoning.
  • Target Layer (Learly): The ablation varies Learly from 6 to 16 layers and reports macro-averaged accuracy on xMMMU and MaXM across evaluated languages.All experiments use LLaVA-1.5-7B.
  • Target Layer (Learly): Learly produces an inverted U-shaped trend: 6 layers may give visual semantics insufficient time to emerge, whereas 16 layers may permit excessive English dominance before visual guidance.The best evaluated setting is Learly = 10, consistent with the mechanistic analysis of visual semantic emergence.
  • Intervention Scope: Early-Layer vs. Full SFT: Full SFT with PVA applied at layer 10 degrades overall accuracy on both benchmarks relative to the targeted early-layer intervention.The controlled comparison keeps the PVA constraint at layer 10 in both settings.

D Complete Implementation Details

The implementation uses a lightweight two-layer GELU alignment network and a one-epoch full-parameter training setup with specified optimization, data, hardware, and input constraints.

  • Model Architecture: MLPalign is a two-layer perceptron with GELU activation.
  • Training Configuration: Training runs for 1 epoch on 135K image-text pairs using AdamW, a 2 × 10−5 peak learning rate, cosine decay, and 3% warmup.The PVA loss weight is α = 0.4.
  • Training Configuration: Experiments use NVIDIA A100 (80GB) GPUs with a global batch size of 128, 336 × 336 images, and a 2048-token maximum sequence length.The passage also specifies full-parameter fine-tuning of the multimodal projector.
Loading 2608.15085v1…