PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

Donghoon Han, SungHyun Moon, Aidyn Zhakatayev, Junghun Cha, SeungJae Lee

arXiv:2608.30725v1cs.CL

TL;DR

Multilingual vision-language encoders show a large HRL–LRL retrieval gap, but it is unclear where trained representations diverge. The paper tests linear concept erasure, EOS activation patching, and front-layer calibration, finding that retrieval is tied to a depth-dependent EOS trajectory rather than output-level linear language bias. Patching and training-time calibration recover LRL performance across encoders and benchmarks while preserving HRL performance.

  • Problem

    Multilingual vision-language retrieval trails by 30+ pp on LRLs versus HRLs, motivating localization of the gap within trained encoders.

  • Method

    The paper combines LEACE and INLP with EOS activation patching and front-layer trunk calibration to test linear-bias and forward-trajectory explanations.

  • Results

    EOS trajectory interventions recover LRL retrieval near English reference levels, while linear erasure changes retrieval by at most ±1.5 pp; calibration gains LRL retrieval across four benchmarks.

  • Takeaways & Limitations

    The alignment-relevant failure is localized to the EOS hidden state’s depth-dependent forward-path trajectory, not the linearly extractable output language direction.

  • Takeaways & Limitations

    The study localizes the failure within fixed trained encoders but does not explain why trajectory divergence arises during pretraining, and it lacks calibration seed variance.

Abstract

from arXiv · show

Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by $30^+$\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from $>99\%$ to near chance and iterated INLP to $37$--$50\%$ while LRL retrieval moves within $\pm 1.5$\,pp and all tier means within $2.2$\,pp, tracking random controls. The linear bias is a \emph{symptom}, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from $22.1\%$ to $69.1\%$ on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering $+9.6$ / $+17.1$\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

1 Introduction

The paper localizes the low-resource retrieval gap within multilingual vision-language encoders, rejecting output-level linear language bias as its cause and implicating the EOS hidden-state trajectory instead.

  • Motivation: 30+ pp separates HRL and LRL retrieval across MetaCLIP-2, SigLIP-2, and four multilingual benchmarks.The study examines where representations diverge along the encoder forward path.
  • Rejecting linear bias: LEACE and INLP remove or attenuate linear language information, yet LRL retrieval changes within ±1.5 pp and tier means within 2.2 pp.These results track random controls, making linear bias a symptom rather than the retrieval cause.
  • Mechanistic localization: EOS activation patching three blocks before the projector rescues LRL retrieval near the single-source English reference on both encoders.Mid-token, random-English-EOS, and source-language controls rule out pooled-position tautology and English specificity.
  • Mechanistic localization: The alignment-causal factor is the EOS hidden state’s per-language forward-path trajectory, which diverges from English early and increasingly with depth.Trajectory divergence, rather than output bias direction, controls retrieval.
  • Training-time corroboration: A front-layer trunk corroborates the diagnosis, improving LRL retrieval across four benchmarks while preserving HRL performance.The intervention reshapes per-layer geometry and targets the same trajectory-level object as patching.
  • Scope: The causal claim is limited to interventional localization within fixed trained encoders, not a complete account of why divergence arises during pretraining.Data prevalence, tokenization, attention regime, and optimization dynamics remain candidate upstream causes.

2 Related Work

Related work motivates concept erasure, activation patching, and partial-encoder calibration as tools for analyzing multilingual representation geometry and causal mechanisms.

  • Representation geometry and concept erasure: Prior work uses linear probing, CKA, and concept-erasure methods to study cross-lingual subspaces and linearly encoded attributes.LEACE and INLP provide closed-form linear erasure, while prior debiasing results show linear extractability need not imply causality.
  • Activation patching: Activation patching replaces intermediate activations to test mechanisms, while multilingual language surgery provides a related intermediate-layer intervention.This paper adapts patching to dual-encoder retrieval with parallel-content and random-content controls.
  • Partial-encoder distillation: The front-layer trunk adapts multilingual sentence-embedding distillation to a partial encoder and targets the trajectory locus identified by patching.Its training objective uses InfoNCE plus centroid-anchored cosine rather than MSE.
  • Data sources: XM3600 supplies headline retrieval and probes, while Flickr30k-200, XTD-200, and Babel-ImageNet provide cross-benchmark replication.FLORES-200 and COMET-22 are used only for translator-quality checking and selecting the training pool.

3 Setup and Notation

The study evaluates frozen causal and bidirectional text encoders, formalizes pooled EOS states and their trajectories, and separates high-resource from low-resource language groups.

  • Models and states: MetaCLIP-2 uses a 24-block causal text tower, whereas SigLIP-2 uses a 27-block bidirectional text tower.Both are evaluated with released frozen weights and with front-layer trunk calibration.
  • Notation: The encoder is decomposed into transformer blocks followed by a projection head, with h_k denoting the pooled EOS row after k blocks.The projector input is emitted at block N−1, and three blocks before it corresponds to ℓ=N−4.
  • Trajectory: A caption’s forward-path trajectory is the sequence of pooled hidden states h_1(x^L), …, h_N(x^L).The study measures each language’s distance from its parallel English trajectory across depth.
  • Languages and data: The language pool contains HRL-6—English, French, German, Spanish, Chinese, and Korean—and LRL-5—Bengali, Filipino, Hindi, Swahili, and Telugu.The LRL designation is operationally based on low prevalence in web-scale vision-language pretraining data.

4 The Linear Language Direction

Language identity is strongly linearly encoded and multilingual averaging improves retrieval, but erasing the linear direction leaves retrieval largely unchanged, separating language identity from retrieval-causal structure.

  • Language identifiability: 99.8% and 99.7% classifier accuracy identify source language from MetaCLIP-2 and SigLIP-2 output embeddings, respectively.Chance for the 13-way probe is approximately 7.7%, and alternative classifiers also detect the signal.
  • HRL averaging: +17.4 pp raises MetaCLIP-2 XM3600 HRL-6 retrieval from 78.1 to 95.5 through averaging six HRL embeddings.Across five encoders and two benchmarks, HRL-pool averaging yields +5.6 to +38.5 pp, while LRL-pool averaging is positive in every cell.
  • LEACE and INLP: LEACE reduces classifier accuracy to chance, but retrieval changes by ≤1.5 pp and matches same-rank random orthogonal ablation.INLP reproduces the pattern at ranks 12, 64, and 128.
  • LEACE and INLP: Rank-128 INLP lowers classifier accuracy to 37–50%, yet no tier mean changes by more than 2.2 pp.Removing the language direction therefore does not appear retrieval-causal up to rank 128.
  • Interpretation: Averaging’s content redundancy and LEACE’s erased language-identity direction appear to occupy different subspaces.The nonlinear regime remains inconclusive under the adversarial-MLP eraser.

5 Pinpointing the Locus on the Forward-Path Trajectory

Activation patching localizes the retrieval failure to the EOS hidden state’s forward-path trajectory rather than an output-layer linear language direction. Parallel-content EOS substitution rescues low-resource retrieval at a non-tautological intermediate layer, with controls excluding norm, pooling, and English-specific explanations.

  • Intervention design: The study probes the forward path by replacing a target caption’s EOS hidden state with its parallel English value while leaving later blocks unchanged.At each layer, only the pooled EOS row is substituted and retrieval is scored against a fixed image gallery.
  • Rescue curves: 44.1% to 73.4% (+29.3 pp) is the MetaCLIP-2 LRL-5 rescue at ℓ=N−4, with Swahili rising from 22.1% to 69.1%.This checkpoint passes through three unmodified back-half blocks; Filipino rises from 39.9% to 73.5%.
  • Controls: Norm-matched random EOS substitution collapses retrieval, showing that the parallel-English rescue is signal-specific rather than norm-driven.The control falls from approximately 80% to 8.5% for HRL and from LRL performance to 1.6%.
  • Controls: At ℓ=N−4, mid-sentence patching leaves retrieval unchanged, whereas EOS patching rescues Swahili and Filipino through three subsequent blocks.This separates substantive rescue from the near-tautological ℓ=N−1 pooling endpoint.
  • Generalization beyond English: Parallel French and six-HRL EOS states lift Swahili to 78.7% and 88.0%, respectively, so the English reference is not a hard upper bound.English EOS reaches 69.1% versus the 69.4% single-source English reference at ℓ=N−4.

6 Trunk Calibration: A Controlled Intervention

Trunk calibration tests the trajectory diagnosis with a general-input intervention: train only front text-encoder blocks to move multilingual projections toward a shared parallel-content anchor while freezing the back-half and vision tower. The method operationalizes a depth-M intervention without using oracle parallel substitution at inference.

  • Controlled intervention: The first M text-encoder blocks are re-fitted toward a shared parallel-content target, while the back-half, projection head, and vision tower remain frozen.The depth is M=4 for MetaCLIP-2 and M=3 for SigLIP-2.
  • Anchor objective: The trunk objective moves each language’s projection toward a parallel-content centroid formed from normalized frozen embeddings over the 11-language anchor set.The anchor is selected using paired cosine, centroid alignment, and held-out low-resource retrieval criteria.
  • Training setup: Parallel-translation groups from an 11-language pool train the trunk for 20,000 AdamW steps using translated CC12M captions.Training uses text-only parallel groups; downstream evaluation uses multilingual benchmarks.
  • Depth-M interpretation: The projected-output loss acts through the fixed back-half map on the depth-M pooled row, making calibration an operational depth-M intervention.The paper chooses M as the earliest tractable point on the trajectory identified by patching.
  • Theoretical scope: The Lipschitz analysis is a sanity envelope for the pooled-row component, not an identifiability proof, because non-pooled rows also differ and the converse requires an unestablished inverse-Lipschitz assumption.Sampled sensitivity ratios average 0.27–0.46, with moderate residual-to-retrieval correlations of 0.55–0.69.

7 Retrieval as Evidence for the Diagnosis

Trunk calibration supports the forward-path diagnosis beyond a single patching example: it improves low-resource retrieval across languages, benchmarks, and additional encoders while preserving or slightly improving high-resource performance. The gains therefore transfer from the controlled intervention to general-input evaluation.

  • Evaluation format: Per-language XM3600 R@1 is compared between frozen and trunk-calibrated states, with bold indicating lifts of at least +5 pp.The table uses the 1,000-image XM3600 subset.
  • XM3600 retrieval: Double-digit LRL gains occur in 6 of 10 language–model pairs, including Swahili gains of +20.0 pp on MetaCLIP-2 and +23.3 pp on SigLIP-2.Other reported gains include Filipino +12.8/+18.1 pp, Bengali +12.5 pp, and Telugu +26.7 pp.
  • Cross-benchmark replication: +7.8 pp and +14.9 pp are the four-benchmark LRL-5 tier-mean lifts for MetaCLIP-2 and SigLIP-2, while HRL-6 rises +1.7 and +3.0 pp.Positive changes appear on Flickr30k-200, XTD-200, XM3600, and Babel-ImageNet.
  • Additional encoders: The intervention transfers to AltCLIP, NLLB-CLIP-L, and mSigLIP, recovering +10 to +37 pp on LRL XM3600 and +9 to +51 pp on LRL Flickr30k-200.These encoders differ in pooling, text-tower scale, and pretraining objective.
  • Evaluation format: The cross-benchmark table reports trunk-calibration changes by benchmark and encoder, using i2t R@1 for retrieval suites and top-1 accuracy for Babel-ImageNet.Its XM3600 row is the full 3,600-image evaluation suite rather than the 1,000-image subset.

8 Geometric Analysis Through Trunk Calibration

Trunk calibration improves cross-lingual geometry after the trained front blocks: HRL–LRL similarity rises toward HRL–HRL similarity at the projector input, while the frozen model preserves a tier gap.

  • Measurement: CKA tracks HRL↔HRL and HRL↔LRL similarities across layers, with their difference serving as the geometric counterpart of the retrieval tier gap.The analysis uses 1,000 image-aligned XM3600 captions and averages over relevant unordered language pairs.
  • Frozen geometry: 0.575/0.458 and 0.388/0.302: at the last text block, within-HRL/HRL–LRL CKA remains separated in the frozen MetaCLIP-2 and SigLIP-2 encoders.Near the projector input, the within-HRL curve is consistently approximately 0.10–0.12 above the HRL–LRL curve.
  • What trunk calibration changes: 0.458 → 0.556 on MetaCLIP-2 and 0.302 → 0.451 on SigLIP-2: trunk calibration raises HRL–LRL CKA at the projector input.These correspond to gains of +0.10 and +0.15, respectively.
  • How calibration reaches the later blocks: Four measurements converge: projector-input CKA rises, depth-M residual structure partly explains it, back-half pooled-row sensitivity becomes more uniform, and divergence falls at every depth.The measurements connect trunk calibration to changes in both front-half representations and the frozen back-half response.
  • Divergence and pretraining prevalence: Pretraining prevalence correlates negatively with frozen depth-M residual divergence, but the sample size is n=10 and prevalence covaries with corpus quality and tokeniser coverage.The authors treat this as corroborating context rather than a separately identified finding.
  • Scope and caveats: CKA measures cross-lingual textual similarity rather than image–text alignment directly, so it is used as a geometric proxy for the trajectory account.Top-K Jaccard and modality-gap corroborations are reported separately on the same XM3600 captions.

9 Discussion

The discussion separates the clean but oracle-dependent EOS patch from general-input trunk calibration and argues that the alignment-relevant failure is a depth-dependent trajectory, not a removable output bias. It limits the causal claim to interventional localisation and leaves upstream causes unresolved.

  • Why two interventions, not one: Patching reaches 73.4% MetaCLIP-2 LRL retrieval at ℓ=N−4, whereas the trunk reaches 53.7%, because patching receives parallel content at inference.The trunk must generalise from translation supervision, while Equation 4 captures only its pooled-row component.
  • What the linear bias is: LEACE and INLP erase or attenuate linearly extractable language identity, yet LRL retrieval changes little, dissociating output bias from retrieval-relevant structure.The discussion characterises the causal factor as a nonlinear, depth-dependent EOS trajectory.
  • Trajectory divergence as the back-half’s distribution boundary: The trajectory marks a boundary of back-half competence: patching restores an HRL-distribution hidden state, while the trunk pulls front-half states toward that region.Both interventions therefore target where the forward path leaves the hidden-state region handled well by the frozen back-half.
  • Alternative upstream causes: Tokenisation, attention regime, optimisation dynamics, and data volume remain candidate upstream causes, but separating them requires controlled pretraining runs outside this paper’s scope.The paper claims interventional localisation rather than an explanation of why divergence arises during pretraining.
  • Diagnostic protocol: The proposed retraining-free checklist combines per-layer CKA, output-embedding LEACE, and EOS-patch sweeps with parallel-English, random-English, and mid-token controls.These measurements are intended to diagnose trajectory divergence and distinguish substantive rescue from pooled-position effects.

10 Conclusion

The paper concludes that linearly visible language identity is not necessarily the alignment-causal feature: the relevant factor is the pooled hidden state’s forward-path trajectory. Front-layer intervention can therefore repair the failure while leaving the back-half frozen, although the origins of divergence remain open.

  • Conclusion: A parallel-content EOS substitution three blocks before the projector recovers LRL retrieval on both causal and bidirectional encoders, while front-layer trunk calibration corroborates the diagnosis.The conclusion contrasts the causal intervention with the training-time general-input test.
  • Implication: Per-language drift at the depth-M EOS state identifies the front blocks as a correction site while the back-half remains frozen across tested pooling and architectural variants.The account is reported across causal, bidirectional, CLS-pooled, and language-code-pooled encoders.
  • Open questions: The paper leaves open when trajectory divergence arises during pretraining, which optimisation signals shape it, and whether the structure appears in text-only multilingual encoders.These are explicitly identified as remaining steps rather than resolved mechanisms.

Limitations

The study localizes a correctable failure within the text encoder but leaves several causal, architectural, data, and evaluation boundaries unresolved.

  • Scope: Generalization beyond the tested models remains open, despite appendix extensions spanning four architectures and three text-encoder families.The main paper evaluates two pretrained text encoders; BLIP-2 and EVA-CLIP-multilingual are not covered.
  • Causal scope: The intervention identifies a repairable EOS forward-path locus, not the upstream causes of its divergence during pretraining.Data prevalence, tokenisation, attention regime, and optimisation dynamics remain candidate contributors.
  • Residual gap: Trunk calibration narrows but does not close the HRL–LRL gap or reach HRL parity.The intervention improves HRL–LRL CKA and shrinks the tier gap, but residual differences remain.
  • Unaddressed components: The frozen projection head and vision tower may also contribute to the observed gap, because the intervention targets only the text-encoder path.The vision-side bias contribution to the modality-gap denominator is not addressed.
  • Data and supervision: Trunk calibration requires adequate parallel-text supervision and is demonstrated only on five languages passing the translation-quality screen.Languages below the quality threshold may exhibit additional failure modes, and generalization beyond the trained LRL-5 remains open.
  • Evaluation uncertainty: Reported calibration uncertainty reflects evaluation-row resampling conditional on a trained checkpoint and selected layer, not full training-and-selection variability.Seed variance for calibration runs is not reported, and the bootstrap intervals use each cell’s best patch layer.

Appendix D: Multi-Benchmark Calibration Gains

Across multilingual benchmarks, front-layer trunk calibration improves low-resource performance while generally preserving high-resource retrieval, with a notable CVQA exception for SigLIP-2.

  • Aggregate gains: +7.8 pp on MetaCLIP-2 and +14.9 pp on SigLIP-2 average LRL gains across the five-language pool and four non-CVQA benchmarks.The larger SigLIP-2 gains reflect its lower frozen LRL baseline and greater room toward the single-source English reference.
  • Cross-benchmark pattern: LRL performance improves on all four non-CVQA benchmarks for both encoders.The evaluated suites include Flickr30k-200, XTD-200, XM3600, and Babel-ImageNet.
  • High-resource behavior: HRL R@1 is preserved or slightly improved on retrieval benchmarks after calibration.The reported exception is CVQA, rather than the retrieval suites.
  • CVQA exception: −3.4 pp on SigLIP-2 HRL CVQA, while SigLIP-2 LRL CVQA remains flat.The authors associate this exception with CVQA’s higher noise floor on SO400M.
  • Anchor robustness: ±1–2 pp across HRL-6-only and EN-only anchor ablations on trained-LRL means indicates the replication conclusions are not anchor-specific.The canonical anchor uses the 11-language centroid.

E.1 Per-direction results (i2t and t2i)

Per-direction evaluation confirms that trunk calibration improves low-resource retrieval across benchmarks while largely preserving high-resource performance. Cross-lingual averaging and trajectory measurements further support shared alignment-relevant content beyond a linear output bias.

  • Tier behavior: HRL performance changes modestly while LRL retrieval shows consistent double-digit gains, with CVQA remaining noisier because it uses relative similarity among four candidate texts.The reported pattern is consistent with stronger HRL baselines and substantial LRL gains across retrieval and Babel-ImageNet.
  • Cross-lingual averaging: ≥+10 pp: averaging five individually weak LRL embeddings improves LRL retrieval in every encoder-benchmark cell.On XM3600, four of five encoders exceed 55% LRL retrieval after LRL-pool averaging, while AltCLIP remains at 18.5%.
  • Cross-lingual averaging: +0.063 and +0.059: frozen MetaCLIP-2 centroid cosine exceeds mean per-language cosine for HRL and LRL, respectively.The corresponding frozen SigLIP-2 advantages are +0.027 for HRL and +0.010 for LRL.
  • Trajectory measurements: 0.119 → 0.022 and 0.143 → 0.045: calibration reduces the last-block HRL–LRL trajectory gap on MetaCLIP-2 and SigLIP-2.Cross-lingual CKA also rises at the projector input, while the back-half responds more uniformly to calibrated states.
  • Intervention comparison: +9.6 / +17.1 pp: calibration improves LRL retrieval, whereas deep single-row patching reaches 73.4% from 44.1% on MetaCLIP-2 LRL.Same-depth patching is much weaker than calibration, showing that the interventions are distinct despite acting on the same trajectory-level object.

Appendix H: Sampled Sensitivity of the Frozen Back-Half

The appendix evaluates how changes at the trunk output propagate through the frozen back-half. Sampled sensitivity is informative for observed states but is not a certified Lipschitz bound, while calibration places states in a more stable response region.

  • What sensitivity measures establish: Only the spectral product is a genuine upper bound; sampled sensitivity probes are diagnostics evaluated on the trunk-produced state manifold.The spectral bound is architecture-only and too loose to use, whereas sampled probes measure in-distribution back-half responses.
  • Sensitivity under calibration: 39% and 32%: calibration reduces the mean observed back-half sensitivity ratio on MetaCLIP-2 and SigLIP-2, respectively.The maximum observed ratio also drops by 6.0× and 2.6×, respectively.
  • Sensitivity under calibration: [0.27, 0.33]: every calibrated language falls within this observed sensitivity range, compared with approximately 0.8 for some frozen LRL cells.Calibration therefore tightens the observed response region across all 11 trained languages.
  • Sensitivity and retrieval lift: ρ=0.55 and ρ=0.69: Pearson correlations between observed sensitivity-weighted residuals and calibration lift are moderate for MetaCLIP-2 and SigLIP-2.The pooled Spearman correlation is 0.75, but these associations remain diagnostic rather than causal evidence.
  • Patch control: −0.1 to +2.5 pp: same-depth single-row patching near the trunk depth is far weaker than calibration’s +9.6–17.1 pp gains.The strongest nearby layer reaches only +3.5 pp, indicating multi-layer front-half restructuring rather than a single-row effect.
  • EOS patch controls: Parallel-English EOS replacement rescues retrieval through three remaining back-half blocks, while random-English EOS collapses it below baseline.The comparison supports content-specific rescue rather than merely moving states toward an English-distributed vector.
Loading 2608.30725v1…