Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs

1. Introduction

• Cross-modal inconsistency means different answers to semantically identical information across modalities, raising the question of whether models reason as effectively from image-rendered text as from native text.The concern is that answers may depend on prompt modality rather than the information itself.Source: §1 · p. 1
• Prior inconsistency benchmarks either cover one model or do not control readability, potentially conflating OCR failures with reasoning differences.Text and image embeddings also occupy different regions of joint embedding spaces, and smaller gaps are associated with better downstream performance.Source: §1 · p. 1
• REST and REST+ present the same semantic content as text, image, or mixed input and evaluate reasoning conditional on correct OCR.The benchmarks use MMLU, ARC, GSM-Symbolic, and newly generated SOEBENCH; REST+ also varies visual properties.Source: §1 · pp. 1–2
• Across 15 frontier MLLMs, the authors find at least ∼10% cross-modal inconsistency even when controlling for OCR.Matching samples have higher cosine similarity than non-matching pairs, and the similarity difference correlates with the benchmark consistency score.Source: §1 · p. 2, Figure 1 · p. 2

2. Our Benchmarks: REST and REST+

• REST evaluates OCR, text, image, and mixed tasks, filtering out questions over 800 characters or containing LaTeX and rendering text on white backgrounds at 200 DPI.Mixed inputs place context in the image and questions in text, with a reversal for multiple-choice questions; the paper reports no configuration is inherently easier across MLLMs.Source: §2 · p. 2
• REST+ creates 10 legible visual permutations per MMLU question by varying font, DPI, and text colour while preserving semantic content.The study samples 10% of each subject class, yielding 1,085 questions, and evaluates text and image tasks.Source: §2 · pp. 2–3
• SOEBENCH tests basic algebra by asking models to solve generated systems of linear equations with a single integer solution.Its restricted symbols reduce OCR complexity, and its newly generated instances ensure no prior exposure by existing MLLMs.Source: §2 · p. 3

3. Benchmarking Methodology

• For each REST question, models first transcribe the image for OCR evaluation, then answer text, mixed, and image tasks using chain-of-thought prompting.Temperature is zero; outputs are parsed with regular expressions, and format-invalid responses count as incorrect.Source: §3 · p. 3
• RER measures the fraction of questions receiving the same answer in every modality, where 1 indicates perfect consistency.The inputs span text, image, and mixed formats.Source: §3 · p. 3
• Cross-modality inconsistency is evaluated among questions correct in at least one modality but not all, excluding questions failed in every modality.The score is zero for a perfectly consistent model.Source: §3 · p. 3
• Max Modal Coverage measures the fraction of questions solved through at least one modality, with 1 indicating potentially perfect performance when each question uses a suitable modality.Questions failed in every modality represent inability rather than inconsistency in the cross-modality measure.Source: §3 · p. 3
• Consistency analysis includes only questions whose letters and digits are perfectly recognised, with OCR quality also measured by character error rate.Character error rate counts insertions, deletions, and substitutions normalized by reference length.Source: §3 · p. 3

4.1. RQ1: Cross-modal inconsistency in MLLMs

• 6.6%–90.7%: RER consistency spans 15 MLLMs; with OCR✓, GPT-5-mini and Claude Haiku 4.5 lead at 90.7% and 90.3%, while Qwen-2.5 32B leads open-source at 84.7%.Source: Table 1 · p. 3, §4.1 · p. 4

4. Benchmarking Results: REST

• Adding mathematical symbols produces negligible differences across datasets, while consistency scores remain unchanged.Source: §4 · p. 3

4.1. RQ1: Cross-modal inconsistency in MLLMs

• CFR reaches 82.3% for Phi-4, versus 8.7% for GPT-5-mini, despite both solving affected questions in at least one modality.Source: §4.1 · p. 4
• For Phi-4, max modal coverage is 85.9%, but only 36.7% is solvable in all modalities, leaving 49.2% potentially dependent on the optimal modality.Source: §4.1 · p. 4, Figure 2 · p. 5
• t = 17.7 (p < 0.05): pooled tests favor text over image, while image is more difficult than mixed (t = −7.2, p < 0.05).Source: §4.1 · p. 4, Table 2 · p. 4

4.2. RQ2: Controlling for OCR

• On novel SOEBENCH, OCR is near-perfect for most models, yet Gemma and GPT still outperform through text; Phi-4 instead performs better on images.Source: §4.2 · p. 4, Table 3 · p. 5
• OCR-first prompting helps some models on many benchmarks but can substantially hurt others, including InternVL3 2B on SOEBENCH.The approach therefore does not consistently remove OCR as a source of performance differences.Source: §4.2 · pp. 4–5

5. Benchmarking Results: REST+

• REST+ has lower overall RER and CFR than REST; its best CFR is 30.2%, and changing font, colour, or resolution can determine correctness.Source: §5 · p. 5, Table 5 · p. 6
• OCR-first prompting does not improve accuracy consistently across models and benchmarks, and can substantially hurt some cases.Source: Table 4 · p. 5
• InternVL3-14B has the highest consistency on both question sets, but Claude Haiku and Mistral decrease significantly on the full set.Source: §5 · p. 5

5.1. RQ3a: Impact of resolution on inconsistency

• At DPI@50, some models, including Claude and Gemma, show lower OCR and RER, while InternVL3-14B and Gemini remain consistent across resolutions.Source: §5.1 · p. 5, Table 6 · p. 6
• Qwen2.5-32B gets 1.8% higher accuracy with 104 image tokens on average at 50 DPI than with 142 text tokens; InternVL3-14B uses about 1600 visual tokens versus 160 text tokens.Source: §5.1 · p. 6, Figure 3 · p. 6
• Models generally reason better with text and need fewer text tokens for comparable performance, except Qwen2.5-VL-32B, which can achieve higher accuracy with fewer vision tokens.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2. RQ3b: Impact of font, colour on inconsistency

• Most models’ image accuracy stays within 2% across fonts, while coloured text often performs better than black; DeepSeek-Small and Qwen-32B exceed 5% relative improvement.Source: §5.2 · p. 6, Figure 4 · p. 7

• The analysis uses 1,000 ImageNet samples, adds images of written-down labels, excludes REST, and separately checks that inconsistency extends to chess positions.Source: §6 · p. 7
• Modality-specific prompts aim to elicit the same response while focusing attention on the relevant input; token sequences are represented by their mean embeddings.Source: §6 · p. 7
• The authors calculate cosine similarity between image and text representations, then measure how often each sample retrieves its matching cross-modal pair; scores are computed across layers and both retrieval directions.Source: §6 · p. 7
• Image–word and written-down-label–word similarity correlate with REST’s RER; written-down labels are more similar to matching words than images are.Source: §6 · p. 7, Figure 6 · p. 8

• Omni-R pairs questions across modalities but evaluates only closed-source MLLMs and does not control OCR; other benchmarks also differ in OCR setting, model coverage, or task framing [8] [43].Source: §7 · p. 8
• Related studies address semantic mismatches, matching entities with complementary rather than identical information [5], privacy concepts [30], and modality collapse, where text often dominates.Source: §7 · p. 8
• Prior work finds text and image embeddings occupy different regions of joint space [18] [28] [31] [32], with cosine similarity linked to task performance [32] and parameter sharing reported to improve downstream performance in CLIP.Source: §7 · p. 8

8. Discussion and Conclusion

• Cross-modal inconsistency scores range from 10% to 90% across MLLMs, and almost all perform best in text.GPT-5-mini and Claude Haiku 4.5 achieve the highest consistency scores.Source: §8 · p. 8
• Correct OCR does not eliminate inconsistency, and neither rendering text as an image nor rendering images as text resolves it.Higher DPI correlates with more consistent performance; font has no clear effect, while text colour does.Source: §8 · p. 8
• Cross-modal similarity of internal representations is positively correlated with the RER consistency score.The authors identify whether this relationship is causal as a question for future work.Source: §8 · p. 8

9. Benchmark Implementation Details

• The authors provide benchmark specifications and plan to release code, data, and model outputs publicly upon acceptance.The materials are intended to facilitate future cross-modal consistency research.Source: §9 · p. 11

9.1. REST

• REST renders MMLU, ARC, and GSM8K-Symbolic content as images, with mixed inputs splitting text and image content according to task format.Questions over 800 characters and those with LaTeX are excluded to reduce OCR complexity; images use black DejaVu Sans at 200 DPI.Source: §9.1 · p. 11, Figure 7 · p. 12
• REST prompts use consistent Chain-of-Thought instructions across modalities, with a separate transcription-only prompt for OCR assessment.Prompt templates adapt the text/image split to multiple-choice or open-ended tasks.Source: §9.1 · p. 11, Figure 8 · p. 13

9.2. REST+

• REST+ creates 10 visual permutations per question across three fonts and three resolutions, plus a 200-DPI colour variant.The study samples 1,085 MMLU questions and omits mixed modality for computational feasibility.Source: §9.2 · p. 11, Figure 9 · p. 14

9.3. SOEBENCH

• SOEBENCH adds 150 novel equation puzzles with 3–5 variables, unique solutions, and values from 1 to 9.Its restricted symbol set and readable rendering are designed to keep OCR burden low while testing reasoning across formats.Source: §9.3 · p. 11, Figure 10 · p. 14
• SOEBENCH presents clue equations as images and the final equation as text in mixed mode, while image mode renders all equations together.OCR instructions require sequential equation numbering to support checking semantic structure and validating outputs.Source: §9.3 · pp. 11–12, Figure 11 · p. 15

9.4. Model Configuration and Hardware

• The experiments use deterministic outputs where configurable and consistent Chain-of-Thought instructions across models.They run with vLLM on single-GPU RTX 6000 Ada or H100 systems, depending on model memory needs.Source: §9.4 · p. 12

10.1. REST

• Several models achieve near-perfect solvability in their best-performing modality; GPT-5-mini reaches 100% MMC on SOEBENCH, 96.0% on ARC, and 97.3% on GSM8k-Symbolic, but this coverage does not ensure consistent performance across modalities.Source: §10.1 · p. 16
• Models’ solvable-question distributions reflect differing consistency, which decreases only marginally with larger models.Flipping the mixed-modality layout changes scores by 1–5%, with no consistently easier design across MLLMs.Source: §10.1 · p. 16, Figure 12 · p. 20, Figure 13 · p. 21
• OCR-correct and full-set consistency scores differ minimally, indicating that REST inconsistency persists when text recognition is correct.Tables 7 and 8 report RER, CFR, and OCR rates for the OCR-correct subset and full dataset, respectively.Source: §10.1 · p. 16, Table 7 · p. 17, Table 8 · p. 17

10.2. REST+

• In REST+, text systematically outperforms image input, and lower resolution substantially affects OCR and downstream reasoning.Resolution robustness varies: DeepSeek-Small and Mistral-Small degrade at lower DPI, while Qwen2.5-32B and InternVL3-14B remain stable.Source: §10.2 · p. 23, Table 13 · p. 23, Table 15 · p. 23
• Most models perform consistently across fonts, while every model achieves higher accuracy with at least one colour than with black at the same resolution.Claude Haiku-4.5's Courier New drop appears in the complete dataset but not the OCR-correct subset.Source: §10.2 · p. 23, Table 19 · p. 24, Table 20 · p. 25
• Only Qwen2.5-32B is reported to achieve higher accuracy with fewer vision tokens than text tokens.The paper reports token consumption by modality and resolution in Table 21.Source: §10.2 · pp. 23–24, Table 21 · p. 25
• REST and REST+ scores generally correlate with MMMU scores, but the correlation is low for high-MMMU models on REST+.Figure 14 presents the comparison and highlights the REST+ pattern among high-scoring models.Source: §10.2 · p. 24, Figure 14 · p. 25

10.3. Natural chess images instead of rendered text

• In chess questions, accuracy differs by up to 35% across modalities for GPT-5-mini, and RER ranges from 58.5 to 77.5 across models.The study verifies semantic understanding by querying the board state; models strong on REST also perform well on chess.Source: §10.3 · pp. 25–26, Figure 15 · p. 25, Table 22 · p. 26