Source-linked AI summary
Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA
Jingbo Wang, Sendong Zhao, Haochun Wang, Bing Qin, Ting Liu
TL;DR
Medical VQA evaluation is predominantly English-centric, and existing multilingual benchmarks lack fine-grained analysis of how language affects distinct medical reasoning capabilities. The paper constructs an eight-language, four-scenario benchmark and introduces MedVL-XLRepE, a training-free inference-time representation intervention; across three LVLMs and eight languages, it consistently mitigates degradation with gains up to 6.33%.
Problem
Existing multilingual medical VQA benchmarks lack fine-grained analysis of distinct capabilities and broader language coverage.
Method
The paper constructs an eight-language benchmark organized into four capability-isolating scenarios and uses MedVL-XLRepE to steer non-English representations toward English counterparts at inference time.
Results
Cross-lingual degradation is highly scenario-dependent, and MedVL-XLRepE consistently mitigates it across three LVLMs and eight languages, with gains up to 6.33%.
Takeaways & Limitations
Medical VQA requires scenario-specific cross-lingual analysis and correction rather than a single global intervention.
Takeaways & Limitations
The method requires white-box access, paired English and target-language inputs, and held-out calibration data, limiting its use for closed-source models and uncharacterized low-resource settings.
Abstract
from arXiv · showhide
Medical visual question answering (VQA) is a crucial task in clinical AI, yet its evaluation has so far centered almost exclusively on English, limiting its relevance to linguistically diverse patients and clinicians. Recent multilingual medical VQA benchmarks show that large vision-language models (LVLMs) degrade in non-English languages, but lack a fine-grained analysis of how cross-lingual variation affects the distinct capabilities that medical VQA requires. To this end, we construct a multilingual medical VQA benchmark over eight languages, organized into four representative scenarios that isolate the core capabilities medical VQA requires. Evaluating five open- and closed-source LVLMs, we find that cross-lingual degradation is not uniform but highly scenario-dependent. We therefore propose MedVL-XLRepE, a training-free scenario-aware representation engineering method, leveraging LVLMs' superior English medical VQA capability to steer non-English representations toward their English counterparts at inference time. Across three LVLMs and eight languages, MedVL-XLRepE consistently mitigates cross-lingual degradation, with gains of up to 6.33\%.
1 Introduction
Medical VQA evaluation remains overwhelmingly English-centric, while existing multilingual benchmarks provide limited fine-grained and language coverage. This paper introduces an eight-language, four-scenario benchmark and MedVL-XLRepE, which consistently reduces scenario-dependent degradation.
- English-centric evaluation limits medical VQA’s applicability to linguistically diverse patients and clinicians.
- Existing multilingual benchmarks measure overall success but lack fine-grained analysis of distinct medical VQA capabilities and broader language coverage.
- Across three LVLMs and eight languages, MedVL-XLRepE yields consistent improvements of up to 6.33%.
- The benchmark covers eight languages and four scenarios designed to isolate core medical VQA capabilities.
- Evaluations across five open- and closed-source LVLMs show that cross-lingual degradation is strongly scenario-dependent.
- MedVL-XLRepE is a training-free, scenario-aware method that steers non-English representations toward English counterparts at inference time.
2 Related Work
Prior multilingual medical VQA benchmarks broaden evaluation beyond English, while representation-engineering research motivates inference-time steering of language-dependent hidden states. Together, these lines of work support studying and correcting cross-lingual medical VQA gaps.
- Recent multilingual benchmarks use medical images and questions from national examinations and clinical consultations to evaluate LVLMs across languages.
- Prior studies report that medical VQA performance degrades in non-English languages, while existing analyses remain limited in scope.
- Semantically aligned multilingual inputs occupy divergent latent-space regions, a divergence associated with cross-lingual performance gaps.
- Representation engineering shows that hidden-state interventions can steer model outputs at inference time.
3 Multilingual Medical VQA Benchmark
The benchmark reorganizes medically aligned multilingual data into four capability-focused scenarios and controls visual evidence and clinical intent across languages. Results reveal measurable overall language gaps and substantial scenario-dependent variation that does not simply track task difficulty or language-resource level.
- Medical VQA Scenarios: The benchmark targets four recurring capabilities: finding recognition, attribute characterization, temporal comparison, and image-text interpretation.
- Medical VQA Scenarios: Perceptual Recognition asks whether a visible finding, structure, or abnormal pattern is present in a single image.
- Medical VQA Scenarios: Attribute-Aware Recognition requires identifying clinically relevant attributes such as laterality, location, size, severity, morphology, or spatial relations.
- Medical VQA Scenarios: Sequential Images Understanding compares temporally ordered studies to determine progression, improvement, stability, or other changes.
- Medical VQA Scenarios: Vision-Text Integrated Reasoning combines medical images with complementary clinical text when visual content alone is insufficient.
- Benchmark Construction: Benchmark construction controls visual evidence and clinical intent across languages, then reorganizes samples by task mechanism using closed-form questions.
- Benchmark Construction: The pipeline translates English-source samples across eight languages and refines medical terminology while preserving clinical intent.
- Cross-Lingual Evaluation: English reaches 67.73% mean accuracy, while Arabic reaches 62.23%, producing a 5.50% overall gap.
4 MedVL-XLRepE
MedVL-XLRepE constructs scenario-aware cross-lingual representation shifts from paired English and target-language inputs, then applies inference-time interventions to steer target-language hidden states toward English representations. Its vectors combine language-level and medically structured corrections while preserving activation magnitude.
- 4.1 Preliminaries: MedVL-XLRepE uses the final input token’s hidden state as the anchor for comparing paired English and target-language multimodal inputs.This choice follows from the autoregressive LVLM architecture, where the final token summarizes the encoded multimodal context.
- 4.1 Preliminaries: The sample-level cross-lingual shift is the hidden-state difference between English and target-language inputs at the same layer.This difference is the basic quantity used to construct the intervention vectors.
- 4.2 Scenario-Aware Intervention Vector: The method separates each scenario’s shift into a language-level component and finer medical shifts shaped by scenario-specific clinical factors.The language vector captures systematic English-target variation, while medical vectors capture variation tied to medically meaningful structure.
- 4.2 Scenario-Aware Intervention Vector: Medical vectors partition samples within each scenario by anatomy and a scenario-specific factor, then average shifts within selected groups.The second factor is pathology for Perceptual Recognition and Sequential Images Understanding, while other scenarios use their corresponding clinical organization.
- 4.3 Cross-lingual Representation Engineering: At inference, the intervention vector combines the language vector with the two factor-specific medical vectors to balance general and scenario-specific corrections.For each target-language input, factor groups are selected using similarity to target-language hidden-state centroids.
- 4.3 Cross-lingual Representation Engineering: The corrected hidden state is added during the forward pass and renormalized to the original ℓ2 norm, leaving other activations unchanged.This training-free activation intervention is intended to steer behavior without updating model parameters or distorting activation magnitude.
5 Experiments
Experiments show that MedVL-XLRepE improves performance across clinical scenarios, model backbones, and target-language resource groups, while representation shifts and hyperparameter sensitivity support its intended intervention mechanism.
- Main Results: 2.91% is the largest mean gain for AAR, while SIU, the most language-stable scenario, still improves by 2.51%.MedVL-XLRepE yields positive mean gains in all four clinical scenarios.
- Main Results: Figure 4 compares PCA projections of multilingual hidden states before and after MedVL-XLRepE.The supplied caption identifies the figure as a PCA visualization of multilingual hidden states.
- Main Results: 2.38% is the overall mean accuracy gain across scenarios and target languages.Per-model mean gains are 2.08% for Gemma3-12B-IT, 3.25% for Qwen3.5-9B, and 1.81% for InternVL3.5-14B-Instruct.
- Main Results: 2.68% is the mean gain for lower-resource languages, exceeding the 2.16% gain for mid-to-high-resource languages.Bulgarian, Thai, and Arabic comprise the lower-resource group; Chinese, Spanish, French, and Japanese comprise the mid-to-high-resource group.
- Hyperparameter Sensitivity Analysis: Figure 5 evaluates performance as the intervention layer l and strength λ vary on Qwen3.5-9B/PR.The layer analysis covers middle layers 12–22, while the strength analysis evaluates λ from 0.06 to 0.14.
- Hyperparameter Sensitivity Analysis: λ=0.10 produces the accuracy peak simultaneously for Bulgarian, Japanese, and Chinese, so it is adopted as the default.The three languages represent relatively low-, mid-, and high-resource target languages in the sensitivity analysis.
6 Conclusion
The paper studies cross-lingual degradation in medical VQA with an eight-language, four-scenario benchmark and proposes MedVL-XLRepE to steer non-English representations toward English at inference time. Across three LVLMs and eight languages, the method consistently mitigates degradation, with gains of up to 6.33%.
- The conclusion reports that MedVL-XLRepE consistently mitigates cross-lingual degradation, with gains of up to 6.33%.
Limitations
The study is limited by its language and task coverage, its need for white-box access, and its dependence on paired inputs and calibration data.
- The benchmark covers only a small portion of the world’s languages and does not capture the full diversity of clinical tasks.
- MedVL-XLRepE requires white-box access to internal activations, restricting its use to open-source LVLMs.
- The method depends on paired English-target inputs and a held-out calibration split, with performance under target-only or severely sparse calibration settings uncharacterized.
Ethical Considerations
The benchmark and MedVL-XLRepE are research artifacts for studying and mitigating cross-lingual degradation in medical VQA, not validated patient-facing systems.
- Patient-facing use would require additional clinical validation and human oversight.
- Multilingual items are machine-translated and then reviewed by medical and linguistic experts to preserve clinical intent.
- The work targets equitable medical LVLM performance beyond English-speaking patient and clinician populations.
A.1 Scenario-wise Data Statistics
The benchmark contains 2,071 English-source items and 16,568 multilingual item-language pairs across eight languages and four capability-isolating scenarios. Its construction uses translation, terminology refinement, expert review, scenario-specific calibration, and evaluation analyses.
- Benchmark scale: 2,071 English-source items expand to 16,568 multilingual item-language pairs under the eight-language setting.The scenario counts are PR 731, AAR 236, SIU 536, and VTI 568 original items.
- Data split: Calibration and test halves are disjoint and stratified by language and medical factor, preserving distributions while preventing calibration items from entering evaluation.MedVL-XLRepE estimates language-level and medical vectors on the calibration half.
- Translation refinement: GPT-5.4 terminology refinement substantially modifies Google Translate drafts across all seven target languages in the PR and AAR subsets.The revision rates support a dedicated terminology-refinement stage after initial machine translation.
- Quality control: Expert review checks medical and linguistic criteria, correcting or discarding items that fail either review before final benchmark inclusion.The review process is conducted by researchers with medical and linguistics backgrounds.
- Scenario-wise analysis: Cross-lingual performance varies by scenario: AAR and VTI have the largest gaps, while PR and SIU are more stable, and language rankings change by scenario.Aggregated results also indicate that closed-source models are stronger overall and more stable across languages than open-source models.
C.2 Component Ablation: Language Vector vs. Medical Vector
The ablation separates MedVL-XLRepE's language-level and medical corrections, showing that their joint use removes isolated regressions and can produce complementary gains. The evaluation protocol tests whether improvements are systematic across models, scenarios, and languages.
- Component design: MedVL-XLRepE combines a language vector steering target-language representations toward English with a medical vector re-centering them within scenario-specific factor subspaces.The component ablation is performed on Qwen3.5-9B in PR and AAR.
- Individual components: Language-only and medical-only corrections provide limited average gains and can reduce performance for individual languages.Examples include −1.69% for Japanese with the medical vector in AAR and −0.28% for Arabic with the language vector in PR.
- Joint correction: +2.34% on PR and +3.99% on AAR are achieved by the combined method, eliminating the isolated regressions observed for individual components.The AAR gain exceeds the isolated contributions’ sum of +1.09%, indicating reinforcement when both corrections are applied jointly.
- Evaluation protocol: The protocol treats 84 model–scenario–language configurations as paired observations and compares baseline accuracy with MedVL-XLRepE accuracy.It applies paired t, Wilcoxon signed-rank, and sign tests, with separate model-level analyses to avoid pooling artefacts.
D.2 Results
MedVL-XLRepE improves performance consistently across the evaluated model–scenario–language configurations. The results support directionally consistent gains rather than improvements attributable only to pooled or stochastic evaluation effects.
- Overall results: +2.38% is the mean gain across 84 configurations, with a 95% confidence interval of [+2.05, +2.71] above zero.The paired t-test rejects the no-effect null with t(83) = 14.37 and p < 10^-15; Wilcoxon gives z = 7.77 and p = 7.8 × 10^-15.
- Directionality: 80 configurations improve, 4 remain unchanged, and none decreases under MedVL-XLRepE.The sign test also rejects the no-effect null at p < 10^-15.
- Model-level results: +2.08%, +3.25%, and +1.81% are the per-model mean gains for Gemma3-12B-IT, Qwen3.5-9B, and InternVL3.5-14B-Instruct, respectively.Each model has a 95% confidence interval strictly above zero and is individually significant under all three tests.