Source-linked AI summary
Said Aloud, Read Different: Cross-Modal Instability in Multimodal Models
Basel Mousi, Fahim Dalvi, Shammur Chowdhury, Firoj Alam, Nadir Durrani
TL;DR
The paper investigates whether semantically equivalent visually grounded queries receive consistent decisions across text and speech and across English and Arabic. It constructs a culturally grounded contrastive benchmark and introduces contrastive instability, finding that modality and language shifts produce triplet-level inconsistencies that aggregate accuracy can miss, especially under speech.
Problem
It is unclear whether semantically equivalent visually grounded queries yield consistent decisions across text versus speech and English versus Arabic.
Method
The paper builds a 10,150-triplet culturally grounded benchmark across 18 MENA countries and defines contrastive instability over triplets with one supported and two unsupported statements.
Results
Speech increases triplet-level instability, particularly in Arabic, while model scaling and joint speech–text input improve robustness without eliminating cross-modal or cross-lingual inconsistency.
Takeaways & Limitations
Contrastive evaluation reveals grounded-reasoning instability beyond standard aggregate accuracy metrics.
Abstract
from arXiv · showhide
Multimodal foundation models are increasingly used in speech-first assistants that must interpret spoken queries and produce visually grounded decisions. Yet it remains unclear whether semantically equivalent queries yield consistent judgments across modality (text vs. speech) and language (English vs. Arabic). We introduce a speech-augmented visually grounded contrastive triplet benchmark spanning 10,150 culturally grounded images from 18 MENA countries, where each image is paired with one supported statement and two plausible but unsupported alternatives. We define contrastive instability as the conditional rate at which a model fails to resolve all statements within a triplet, isolating fragmented reasoning from complete failure. Evaluating recent multimodal models under text and speech in English and Arabic, we find that modality and language shifts introduce substantial triplet-level inconsistencies that are not fully captured by aggregate accuracy, with speech amplifying partial failures. We make the benchmark publicly available to the community.
1. Introduction
The paper asks whether visually grounded decisions remain consistent when semantically equivalent inputs are delivered through text or speech and across English and Arabic. It introduces a culturally grounded contrastive benchmark and contrastive instability to evaluate this consistency beyond aggregate accuracy.
- Benchmark and evaluation: Contrastive triplets require coherent distinctions among one supported statement and two plausible unsupported alternatives.This evaluates matched sets rather than isolated statements.
- Metric: Contrastive instability measures failures to resolve all statements within partially successful triplets, exposing fragmented reasoning beyond aggregate accuracy.The conditional metric separates internal inconsistency from complete failure.
- Evaluation: The evaluation covers Qwen2.5, Qwen3-30B-Omni, and Phi-4 multimodal models with text and speech inputs, including acoustic perturbations.English and Arabic speech queries enable controlled comparisons across modality, language, and speaker variation.
- Findings: Speech increases triplet-level instability, Arabic shows higher instability than English, and speech amplifies the cross-lingual gap.Scaling improves stability and acoustic-noise robustness but does not eliminate cross-modal or cross-lingual inconsistency; joint speech–text input mitigates instability without fully restoring text-only coherence.
- Benchmark and evaluation: 10,150 culturally grounded triplets spanning 18 MENA countries support consistency evaluation across modality and language.Each image is paired with one supported statement and two plausible but unsupported alternatives.
2. Dataset
The dataset combines culturally grounded images with contrastive, image-dependent statements spanning MENA countries, then evaluates annotation reliability and creates matched Arabic and speech variants. Human verification indicates reliable grounding labels, while acoustic perturbations test stability beyond clean audio.
- Dataset construction: The curated repository spans 18 Arab countries and culturally distinctive themes including architecture, clothing, cuisine, religious settings, and public spaces.It builds on the OASIS image collection and M2CQA.
- Contrastive construction: Each image becomes a contrastive triplet with one supported answer and two culturally plausible distractors after image-blind filtering removes linguistically solvable items.The retained options are rewritten as standalone statements.
- Human verification: 15% of the dataset was human-verified with stratified country and category coverage using Need-Image and Q-Correctness labels.Disagreements were adjudicated by a third annotator.
- Human verification: AC1 agreement was approximately 0.996 for Need-Image and 0.741 for Q-Correctness, with raw agreement approximately 0.996 and 0.775, respectively.Q-Correctness agreement was slightly higher for supported statements than unsupported alternatives.
- Reliability findings: Most annotation disagreements concerned subtle counterfactual or fine-grained cultural distinctions, while image insufficiency was rare.These findings support the dataset as a foundation for triplet-level contrastive consistency evaluation.
- Multilingual and speech variants: English statements were translated into Arabic and synthesized into matched male and female speech variants, with noise and reverberation added for acoustic robustness testing.Lexical content was kept identical to the original text during speech synthesis.
3. Contrastive Instability
The paper introduces contrastive instability to distinguish coherent triplet resolution from partial success or complete failure. The metric evaluates whether models accept the supported statement while rejecting both plausible unsupported alternatives.
- Motivation: Independent statement accuracy can miss models that accept both a supported statement and a plausible unsupported alternative.Such behavior reflects inconsistent discrimination rather than coherent grounding.
- Triplet design: Each contrastive triplet contains one supported statement, Q+, and two plausible unsupported alternatives, Q−1 and Q−2.The design introduces minimal semantic competition among closely competing interpretations.
- Triplet consistency: A triplet is fully consistent only when the model accepts Q+ and rejects both Q− alternatives.Triplet accuracy counts the proportion of such fully correct triplets.
- Metric definition: CI = 1 − Nconsistent/Npartial, where Npartial counts triplets with at least one correct statement and Nconsistent counts fully consistent triplets.The conditional denominator focuses on partially successful triplets.
- Interpretation: CI separates decision-boundary instability from overall performance degradation and is not a monotonic transformation of triplet accuracy.Models with identical triplet accuracy can differ in CI when their failures are global versus internally inconsistent.
4. Results
Across language, modality, model scale, and acoustic conditions, contrastive instability exposes grounded inconsistencies that aggregate accuracy can miss. Speech produces the strongest instability in Arabic, while scaling and joint speech–text input mitigate but do not eliminate it.
- Evaluation: CI measures conditional incoherence within triplets, complementing aggregate Q+ and Q− accuracy and F1.A triplet is fully consistent only when all three statements are answered correctly; CI focuses on partially correct triplets that fail full consistency.
- Modality and language: Q2.5-3B Arabic CI rises from 0.43 under text to 0.71 under speech.This increase shows that speech can make partially correct triplets internally inconsistent, while English remains largely stable across modalities.
- Modality and language: Q2.5-3B Arabic CI is 0.20 versus 0.43 in English and Arabic text, widening to 0.22 versus 0.71 under speech.Across models, Arabic instability is higher than English instability, with the gap widening under speech.
- Model scaling: Arabic CI decreases from 0.43 to 0.35 to 0.28 under text and from 0.71 to 0.63 to 0.47 under speech as model scale increases.The values correspond to Q2.5-3B, Q2.5-7B, and Q3-30B; English instability remains comparatively low and stable.
- Acoustic robustness: CI increases monotonically as SNR decreases, with sharper degradation in Arabic than English.Q3-30B maintains lower CI across SNR levels, but the language gap persists even at higher SNRs.
- Joint signaling: Audio-only input has the highest Arabic CI, text-only the lowest, and joint speech–text input reduces CI without fully restoring text-only stability.The stabilizing effect of joint input is weaker for smaller models; English differences across conditions are minimal.
5. Related Work
Prior multimodal benchmarks and evaluation studies show that aggregate accuracy is insufficient for characterizing reliability and decision coherence. This motivates behavioral and contrastive testing under controlled variation, including multilingual settings.
- Evaluation gaps: Aggregate accuracy alone is insufficient to characterize multimodal model reliability.Existing benchmarks primarily report independent-sample performance rather than coherence across related decisions.
- Evaluation gaps: Behavioral testing and contrastive probing assess robustness under controlled variation beyond isolated correctness.This framing motivates evaluating whether models maintain coherent decisions across semantically related inputs.
6. Conclusion
The paper tests whether semantically equivalent text and speech inputs yield consistent visually grounded decisions. It finds that speech, especially in Arabic and under acoustic degradation, increases triplet-level instability even when aggregate accuracy is preserved.
- Contribution: The benchmark and CI measure internal decision incoherence across semantically matched triplets.The evaluation focuses on grounded consistency across speech and text rather than isolated statement correctness.
- Findings: Speech can preserve aggregate Q+ and Q− accuracy while increasing triplet-level instability, particularly in Arabic and under acoustic degradation.Model scaling and joint speech–text signaling improve robustness but do not fully eliminate cross-modal or cross-lingual instability.
- Implication: Contrastive evaluation diagnoses grounded reasoning stability beyond standard accuracy metrics.The conclusion emphasizes CI as a complementary way to reveal inconsistencies that aggregate measures may not capture.
7. Use of Generative AI Disclosure
Generative AI tools supported data construction and writing assistance, while the authors retained responsibility for research design, analysis, and conclusions.
- Disclosure: Generative AI tools were used for question generation, translation, speech synthesis, and writing assistance.The disclosure assigns research design, analysis, and conclusions to the authors.