PaperlayerView exact revision on arXiv

Source-linked AI summary

When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context

Yuxing Cheng, Yuan Wu, Yi Chang

arXiv:2609.34781v1cs.CVcs.AI

TL;DR

VLMs can use semantic context to interpret scene text, but that context may override unusual printed characters, raising a question of faithful transcription when text and scene conflict. SceneFaith evaluates the conflict with generated images and controlled scene comparisons; its results show rewriting across models and sensitivity to surrounding context and visual degradation.

  • Problem

    VLMs interpret scene text using linguistic and visual semantics that can override unusual printed strings, so faithful transcription when scene context conflicts with the text remains an open question.

    Source: §1 · p. 1

  • Method

    SceneFaith uses generated scenes with perturbed printed words, Literal/Canonical/Other outcomes, and controlled comparisons that vary context around a fixed target patch.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    All 15 evaluated model variants showed clear-image canonical rewriting, at rates of 8.45%–58.51%; removing surrounding scene information reduced rewriting and improved literal accuracy.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Reliable recognition in the evaluated conditions requires preserving clear characters while using context mainly when visual evidence is uncertain.

    Source: §5 · p. 9, §7 · p. 10

    SceneFaith uses generated scenes with deliberately perturbed text, so generalization to natural images and broader OCR tasks remains untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45\% to 58.51\%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.

1 INTRODUCTION

VLMs’ semantic priors can help when visual evidence is incomplete, but may override unusual printed text, raising the question of whether they faithfully transcribe text under misleading scene context. SceneFaith evaluates that conflict and finds canonical rewriting across models, with context, text degradation, and lexical preference shaping the observed behavior.

1 INTRODUCTION

  • VLMs interpret text with linguistic and visual semantics, which can override visible characters when strings such as misspellings, names, or identifiers are atypical.This contrasts with conventional OCR’s primary aim of recovering the visible character sequence.Source: §1 · p. 1
  • 8.45%–58.51% of clear-image outputs were canonical rewrites across 15 model variants from seven families, and every evaluated model showed the effect.Removing peripheral scene information reduced rewriting and improved literal accuracy; changing the scene around the same target patch could also change predictions.Source: §1 · p. 2
  • SceneFaith’s 781 generated images across 17 categories use Literal/Canonical/Other outcomes to separate faithful transcription, canonical rewriting, and other recognition errors.Each sample embeds a perturbed conventional word in a semantically matching scene, with quality control and blind crop reading before acceptance.Source: §1 · p. 2
  • The benchmark also examines factors affecting rewriting: blur increases it, stronger lexical preference is associated with higher rates, and input presentation matters.The study compares full images and target crops as well as target-text degradation and lexical preference.Source: §1 · p. 2

2 RELATED WORK

Prior work studies linguistic information in recognition and faithfulness under visual–semantic conflict, while mitigation methods adjust attention or decoding. The reviewed evaluations often do not distinguish plausible substitutions from other errors or measure the surrounding scene’s contribution.

2.1 TEXT RECOGNITION SYSTEMS AND BENCHMARKS

  • Recognition systems incorporate linguistic dependencies, while OCR benchmarks assess transcription, localization, parsing, and text-centric reasoning without separating plausible substitutions from ordinary errors.The review describes sequence-level linguistic modeling, whole-image conditioning, and document-VLM approaches that use post-training, unified OCR formats, token reordering, or global-layout/local-recognition separation.Source: §2.1 · p. 2

2.2 FAITHFULNESS UNDER VISUAL–SEMANTIC CONFLICT

  • Reviewed conflict studies span scene text, altered documents, meaningless words, handwriting, and historical scripts, but fixed-image gold-string scoring leaves substitutions indistinguishable from other errors and the scene’s contribution unmeasured.The review also notes an association between stronger visual-token compression and greater reliance on language priors in DeepSeek-OCR.Source: §2.2 · pp. 2–3

2.3 IMPROVING OCR FAITHFULNESS

  • Mitigation work targets vocabulary reliance through character-representation contrasts or attention and decoding adjustments; this paper complements it by testing scene removal and target blur.The authors frame these experiments as identifying what mitigation methods need to address.Source: §2.3 · p. 3

3 BENCHMARK DESIGN AND DIAGNOSTIC FRAMEWORK

SceneFaith tests whether models faithfully transcribe deliberately perturbed text when the surrounding scene supports a conventional spelling, under clear and blurred target conditions. Its construction combines generated scene-text conflicts with checks that the unusual string is legible and not leaked elsewhere in the image.

3.1 BENCHMARK OVERVIEW

  • The benchmark tests clear images and target-local blur, with context defined as information outside the target region and lexical preference analyzed separately.The printed string differs from the conventional spelling supported by both scene and lexical preference.Source: §3.1 · p. 3
  • SceneFaith is a stress test of 781 generated images across 17 categories, designed to measure behavior when unusual printed text conflicts with familiar scene content rather than estimate general OCR accuracy.Each image has a marked target, observed printed text as gold, and a different conventional alternative stored for analysis.Source: §3.1 · p. 3
  • A fixed-target-patch set holds the target pixels and their local properties identical while changing only the surrounding scene between conventional-word-supporting and observed-text-supporting contexts.The paired analysis is reported separately from the main benchmark and uses the observed string as the gold answer.Source: §3.1 · p. 3

3.2 CONSTRUCTION PIPELINE

  • Generation perturbs a conventional word through character or spacing changes, then places the unusual string into a semantically relevant scene with plausible typography and materials.The target is marked, and the surrounding scene suggests the conventional word.Source: §3.2 · pp. 3–4
  • Quality control checks spelling, target marking, scene integration, clarity, and answer leakage, then uses blind crop transcription to verify the unusual string can be recovered without the scene.Accepted records store observed and canonical strings, image and target metadata, hashes, and review evidence; gold fields agree with observed text.Source: §3.2 · p. 4, Figure 1 · p. 4

4 EVALUATION AND DISCUSSION

The evaluation separates faithful transcription, canonical rewriting, and other errors, then tests how scene context, target clarity, and lexical properties relate to rewriting. Across the benchmark and controlled experiments, rewriting appears on clear images and increases when target text is blurred.

4.1 EVALUATION PROTOCOL

  • Literal, Canonical, and Other classify outputs as the printed gold, conventional spelling, or neither; evaluation uses zero-shot prompts and reports both literal accuracy and rewriting rate.The full benchmark uses 781 images; “rewriting” describes the final output category, not an inferred correction process.Source: §4.1 · pp. 4–5

4.2 SCENEFAITH EVALUATION RESULTS ANALYSIS

  • 8.45%–58.51% rewriting rates across 15 aliases show canonical rewriting on clear images, but these constructed stress-set rates are not general OCR accuracy.The 15-alias mean is 26.35%.Source: §4.2 · p. 5, Table 1 · p. 6
  • Closed-source models have lower mean rewriting than open-source models on clear and moderately blurred images, but the group differences are descriptive.The groups differ in model family, scale, and training setup.Source: §4.2 · p. 5
  • Rewriting rate alone can misrank models for literal accuracy because equally frequent rewriting can accompany different rates of correct and Other outputs.For example, GPT-5.5 and Qwen3-VL-8B Instruct both have 19.46% rewriting but literal accuracies of 78.10% and 62.23%.Source: §4.2 · p. 6

4.3 CONTEXT EFFECT ANALYSIS

  • With target pixels fixed, gray masking lowers rewriting by 6.53–20.10 points and raises literal accuracy by 7.17–20.36 points across four models.The controlled fixed-target set also finds mean rewriting rises from 6.85% in identifier scenes to 11.15% in semantic scenes, while mean Other stays 3.45%; semantic and visual effects remain entangled.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 TARGET BLUR INCREASES REWRITING

  • Moderate blur increases rewriting for all 15 aliases, while stronger blur further increases it in a four-model experiment.Under strong blur, rewriting reaches 43.02–60.69% for those four models.Source: §4.4 · p. 8

4.5 ANALYZING LEXICAL PRIORS IN REWRITING

  • Rewriting is associated with stronger lexical preference and longer conventional strings, while lowercase l↔i substitutions have higher rewriting than other letter-shape substitutions.The length association remains after controlling for lexical scores or perturbation type; the lexical measure uses external proxies, not VLM internal priors.Source: §4.5 · p. 8

4.6 POST-TRAINING MITIGATION PILOT

  • SFT+GRPO modestly improves exact match, pair accuracy, and character error rate over SFT in the post-training pilot.Exact match improves by 0.53 percentage points and pair accuracy by 1.07 percentage points; character error rate falls from 5.80% to 5.64%.Source: §4.6 · pp. 8–9, Table 4 · p. 9

5 DISCUSSION

The discussion finds that transcription fidelity depends on surrounding context and image presentation, not only target characters. It argues for preserving clear text while using context when visual evidence is ambiguous.

  • Gray masking improves literal accuracy, while changing the scene around an unchanged target patch can change model outputs.Under blur, removing the scene still increases rewriting in three of four models, suggesting lexical completion and visual ambiguity may both contribute.Source: §5 · p. 9
  • Crop recovery shows that some clear-image errors remain recoverable under a different view.The discussion recommends evaluating helpful, neutral, and misleading contexts around the same degraded target.Source: §5 · p. 9
  • A lower rewriting rate is not necessarily better: canonical outputs can become either literal matches or other errors.Reporting Literal, Canonical, and Other together keeps literal accuracy as the main measure of recovery.Source: §5 · p. 9

6 LIMITATIONS

The study’s conclusions are bounded by its generated, deliberately perturbed scenes and by limitations in several analyses and experiments.

  • SceneFaith uses generated scenes with deliberately perturbed text, so generalization to natural images and broader OCR tasks remains untested.Source: §6 · p. 9
  • Scene interventions combine semantic and visual changes, while external lexical scores provide indirect evidence about internal mechanisms.Source: §6 · p. 9
  • Blurred targets lack complete readability validation, making information loss difficult to separate from lexical completion.Source: §6 · p. 9
  • The post-training pilot uses one model; further evaluation requires matched compute, independent test data, and additional models.Separate API runs may also introduce provider variation.Source: §6 · p. 9

7 CONCLUSION

SceneFaith makes canonical substitution auditable and finds that context, image views, degradation, and lexical preference relate to transcription behavior. The conclusion frames peripheral-input sensitivity as an OCR reliability concern in the evaluated conditions.

7 CONCLUSION

  • Fixed-target scene removal reduces rewriting and improves literal accuracy in four models; complementary diagnostics characterize sensitivity to scene substitutions, image views, degradation, and lexical preference.The authors identify peripheral-input sensitivity as a practical OCR reliability concern in the evaluated conditions.Source: §7 · pp. 9–10

AI USE STATEMENT

  • The authors report that generative models assisted with benchmark construction, experiment design, code, and paper polishing, and that they checked the work.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses generated scenes and model outputs, with no personal data or human-subject annotations.Retained raw responses make exact-match errors from familiar-spelling substitutions traceable.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Retained manifests, image hashes, prompts, parser, raw responses, and deterministic audits support the reported results.Appendices document external-scorer, paired-image, fixed-patch, consensus, and blur/mask audits.Source: REPRODUCIBILITY STATEMENT · p. 10

A CONSTRUCTION INVENTORY AND AUDIT

B LEXICAL PRIOR ANALYSIS

The lexical-prior analysis finds an association between the external preference gap and rewriting, while string length and blur analyses provide additional patterns. These results are exploratory and use external scores rather than identifying a VLM-internal lexical mechanism.

B LEXICAL PRIOR ANALYSIS

B.1 SCORING AND ESTIMATION

  • The analysis uses DistilGPT2 scores after four fixed neutral prefixes; D measures the conventional string’s advantage over the printed string, while A measures its likelihood.Regressions adjust for model, domain, string properties, and target-box geometry; intervals account for repeated outputs from an image.Source: §B.1 · p. 13

B.3 STRING LENGTH

  • Rewriting rises from 14.27% for conventional strings of 4–5 characters to 45.02% for strings of at least 10 characters.The adjusted odds ratio is 1.230 per additional character ([1.157, 1.309]), or 1.127 ([1.039, 1.223]) after adding A and D.Source: §B.3 · p. 13

B.2 ASSOCIATIONS AND SENSITIVITY

  • D correlates with rewriting (ρ = 0.255, 95% CI [0.189, 0.321]); RR rises from 20.47% in the lowest to 38.03% in the highest quintile, a 17.57-point gap.In the joint adjusted model, D remains associated with rewriting, whereas A does not.Source: §B.2 · p. 13, Table 7 · p. 14, Table 6 · p. 14

C SUPPLEMENTARY STATISTICS FOR THE CLEAR-IMAGE EVALUATION

D SENSITIVITY TO MODEL-FAMILY COMPOSITION

E QWEN INSTRUCT–THINKING COMPARISONS ANALYSIS

Thinking reduced literal accuracy at 8B and 32B compared with Instruct, with no clear change at 235B-A22B; the section also examines perturbation-linked rewriting and blur.

E QWEN INSTRUCT–THINKING COMPARISONS ANALYSIS

  • Thinking lowered literal accuracy by 13.70 points at 8B and 13.44 points at 32B, with no clear change at 235B-A22B (+1.15 points).These comparisons are between distinct served aliases, not a thinking toggle within one model.Source: E QWEN INSTRUCT–THINKING COMPARISONS… · p. 14
  • Longer conventional strings accompany more canonical substitutions.Source: Table 8 · p. 14
  • All 2,343 Thinking outputs retain reasoning text, unlike Instruct outputs; exact searches find both printed and canonical strings in seven canonical-answer traces (1, 1, and 5 by size), where recognition may still yield incorrect final answers.Source: E QWEN INSTRUCT–THINKING COMPARISONS… · p. 16
  • Blur amplification remained unresolved, and external lexical gap D showed no positive association at 8B or 32B.At 235B, the slope was −3.56 points per SD ([−6.29, −0.94]); exploratory intervals were unadjusted for multiple checks, and associations do not identify a lexical mechanism.Source: E QWEN INSTRUCT–THINKING COMPARISONS… · p. 16

F CHARACTER PERTURBATIONS AND CANONICAL REWRITING

  • The eight perturbation groups include letter-shape, digit/symbol, and split/merge edits, as well as duplication, deletion, insertion, transposition, and other substitution.Inserting an adjacent identical character counts as duplication.Source: F CHARACTER PERTURBATIONS AND… · p. 16
  • For 31 lowercase l↔i images, RR exceeded that of the other 120 letter-shape images by 22.62 points ([12.59, 32.90]).The adjusted image-clustered estimate was OR 3.39 ([2.16, 5.30]).Source: F CHARACTER PERTURBATIONS AND… · p. 16, Table 13 · p. 17

G BLUR STATISTICS

H FIXED-TARGET-PATCH DIAGNOSTIC ON 150 PAIRS

With target pixels fixed, changing the surrounding scene changed rewriting rates; a gray-background control also showed that removing scene information can raise rewriting.

I CONSENSUS FILTERING AND RECOVERABLE SHARED ERRORS

Consensus filtering recovered shared nonliteral errors, while crossed blur and scene-removal results suggest surrounding cues can reinforce canonical rewriting when target evidence weakens.

I CONSENSUS FILTERING AND RECOVERABLE SHARED ERRORS

J CROSSING TARGET BLUR WITH PERIPHERAL SCENE REMOVAL

Loading 2609.34781v1…