When MLLMs miss what humans read: diagnosing and learning to read squint text

1 INTRODUCTION

• Frozen-weight diagnostics indicate that hidden text leaves a strong coarse signal but weak glyph identity, which becomes more accessible when correct spatial regions are supplied.Suppressing high-frequency content also improves reading at matched visual-token budgets, while token count may contribute secondarily.Source: §1 · pp. 1–2
• The failure matters for moderation because policy-violating semantics can reside in embedded text even when models report that no text is present.Production examples include domains, contact strings, and slogans embedded in coarse image structure.Source: §1 · p. 1
• SQUINT OPSD reaches 88.61 ANLS on SQUINTBENCH by training with a downscaled view and ground-truth transcription while retaining original-only inference.The frozen teacher supplies token-level distributional supervision to the student.Source: §1 · p. 2
• SQUINTBENCH contains 1076 production moderation images, while SQUINTCTRL provides controlled synthetic cases for mechanism diagnosis.The benchmarks support realistic evaluation and controlled study of hidden-text representation, localization, and recoverability.Source: §1 · p. 2

2.1 PROBLEM SETTING

• Transcription requires reporting every string a human reader would, including text carried in an image’s coarse structure.The study compares a single-image input with a two-view input containing the original and a bicubically downscaled view, usually at one-eighth linear scale.Source: §2.1 · p. 2
• SQUINTBENCH uses set-matched ANLS so correct transcriptions in a different order are not penalized.Predicted and reference regions are matched as sets before scoring.Source: §2.1 · p. 3

2.2 DATASETS

• SQUINTBENCH comprises 1076 production-derived moderation images with human transcriptions of visible and hidden text for realistic end-to-end evaluation.The images are typically profile photos containing embedded text that human readers recognize readily.Source: §2.2 · p. 3
• SQUINTCTRL uses controlled synthetic scenes with known glyph masks, enabling comparisons that hold non-target generation variables fixed.Its four subsets test text presence, glyph identity and localization, recoverability across conditioning strengths, and open-vocabulary read-out.Source: §2.2 · p. 3

3.1 HIDDEN-TEXT REPRESENTATION

• Text presence reaches 0.956–0.981 AUC in frozen vision features, while Chinese glyph identity peaks at 0.718 and Latin at 0.612.Shuffled-label controls are near chance, and glyph identity remains above chance but weaker than presence.Source: §3.1 · p. 3

3.2 SPATIAL LOCALIZATION

• Oracle pooling at glyph-bearing locations improves glyph-identity prediction by about 0.10 AUC over global and random controls, implicating spatial selection as a major bottleneck.The supplied Figure 2 caption likewise reports that restricting read-out to glyph-bearing tokens improves prediction.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 FREQUENCY AND TOKEN COUNT

• At a fixed 64-token budget, stronger low-pass filtering raises ANLS from 29.2 to 67.0, supporting high-frequency interference as a major contributor.Token-count effects remain possible but are not fully disentangled from frequency; contrast-only transforms yield smaller gains.Source: §3.3 · p. 4, Figure 2 · p. 4

3.4 CORRECTIVE VIEWING

• Adding a one-eighth-scale view raises SQUINTBENCH ANLS from 26.50 to 58.96 with frozen weights, motivating transfer of this reading behavior to original-only inference.The accompanying candidate-solutions table identifies SQUINT OPSD as the only method satisfying original-only inference, localization-bearing supervision, and distributional targets.Source: §3.4 · p. 4, Table 1 · p. 5
• Distributional targets retain probabilities over plausible next tokens, unlike a one-hot ground-truth label.Those relative probabilities can express which alternative glyphs remain visually plausible.Source: §3.4 · p. 5

4 METHOD

• The teacher receives the downscaled view and ground-truth output as privileged context, while the student receives only the original image during training and inference.The downscaled view changes the viewing condition; the transcription supplies privileged label information.Source: §4 · p. 5, §4.1 · p. 5

4.1 SQUINT OPSD

• SQUINT OPSD samples trajectories from the current student policy and minimizes token-level Jensen–Shannon divergence between teacher and student distributions along them.Gradients flow only through student logits; teacher top-K tokens define explicit support, with remaining probability mass grouped into a tail bucket.Source: §4.1 · p. 5

4.2 PRIVILEGED VIEW

• The downscaled view is a deterministic, lossy transform of the original and adds no conditional Shannon information, so its value is computational rather than informational.The method does not assume the student can realize the teacher’s complete distribution from the original image and task instruction alone.Source: §4.2 · p. 5

5.1 EXPERIMENTAL SETUP

• Training uses 5,905 prompts, and evaluation centers on 1,076 deployment-derived SQUINTBENCH images, with transfer and general-capability benchmarks also included.The teacher receives a downscaled view and ground-truth output; inference uses the original image alone.Source: §5.1 · pp. 5–6, Figure 3 · p. 6

5.2 MAIN RESULTS

• 88.61 ANLS on SQUINTBENCH exceeds BASE (26.50), BASE +1/8 view (58.96), single-view SFT (72.47), and DAPO (73.55).SQUINT OPSD uses the original image alone at inference.Source: §5.2 · p. 6, §5.1 · p. 6, Table 2 · p. 6

5.3 ABLATIONS

• Alternative teacher views, entropy-based teacher selection, EMA updates, and SFT warm starts do not improve on the fixed-teacher recipe.SFT warm starts underperform direct OPSD, consistent with the reported advantage of distributional supervision from the outset.Source: §5.2 · p. 7, §5.3 · p. 8

5.2 MAIN RESULTS

• On AdvOCR, SQUINT OPSD reaches 66.7 versus BASE at 3.7, and on HC-Bench it reaches 44.6 versus 1.8; Illusory VQA WER falls from 178.10 to 67.00.Qwen3-VL-235B scores 24.44 on original SQUINTBENCH images.Source: §5.2 · p. 7, Table 3 · p. 7
• The hidden-text gains accompany task-dependent declines: AI2D changes from 86.66 to 85.33, DocVQA from 95.93 to 92.98, and OCRBench from 879 to 833.The paper characterizes this as a measurable specialization trade-off while retaining substantial general vision-language capability.Source: §5.2 · p. 7

5.3 ABLATIONS

• Combining the aligned downscaled view and transcription hint reaches 88.61, versus 54.68 for view-only and 41.79 for hint-only teacher input.Replacing the matched view with another sample's view reduces the paired score from 84.60 to 78.60.Source: Table 4 · p. 7, §5.3 · p. 8
• Replacing teacher distributions with one-hot predictions lowers performance from 88.61 to 68.31, below ground-truth SFT at 72.47.The authors attribute the gain to information in the distribution rather than better hard labels.Source: Table 4 · p. 7, §5.3 · p. 8
• Updating only the vision tower reaches 71.12, only the language side reaches 29.25, and joint adaptation reaches 88.61.The authors conclude that visual adaptation accounts for a large part of the gain, while the strongest read-out requires both sides to adapt.Source: Table 4 · p. 7, §5.3 · p. 8

6 ANALYSIS

• Glyph-identity AUC rises from 0.718 to 0.975 on Chinese and from 0.612 to 0.779 on Latin, while near-ceiling presence detection changes little.The authors interpret this as training making glyph identity more accessible in the visual representation.Source: §6 · p. 8
• Replacing glyph-bearing embeddings reduces accuracy from 90.5% to 71.5%, while replacing an equal number of background embeddings leaves it at 91.0%.At dilation d = 2, this intervention supports a causal dependence of learned reading behavior on glyph-bearing visual regions.Source: §6 · p. 8
• Visual organization of candidate probabilities rises from ρvis = 0.126 for BASE to 0.286 for SQUINT OPSD, while ρlang remains close to BASE (0.544 vs. 0.555).SFT reaches ρvis = 0.242 and ρlang = 0.334.Source: §6 · p. 8

• Figure 4 groups post-training changes into next-token distributions, glyph-region attention and embedding intervention, and candidate distributions at 120 matched glyph positions.Source: Figure 4 · p. 9
• SQUINT OPSD is presented as the first post-training approach for squint-visible hidden text that retains original-only inference.Earlier approaches change the viewing condition or direct search at inference.Source: §7 · p. 9
• Prior work spans reading-oriented data and architectures, text-rich benchmarks, modality-gap studies, added visual views, and privileged or on-policy distillation.These works largely target fine-grained visual detail retrieval.Source: §7 · p. 9
• Representation probes test intermediate features, while VLM studies localize object information to visual tokens; this work probes coarse-text identity at the correct locations (Alain & Bengio, 2018).The passage also cites outlier artifact-token work.Source: §7 · p. 9

• 88.61 ANLS on SQUINTBENCH compares with 26.50 for the base model.Source: §8 · p. 9
• SQUINT OPSD transfers to external hidden-text benchmarks with measurable general-capability trade-offs.Source: §8 · p. 9
• After training, glyph identity is more discriminative, reading depends on glyph-bearing regions under the embedding intervention, and alternatives become more visually structured while retaining language-side structure.Source: §8 · p. 9
• The method relies on recoverable visual evidence and fixed privileged supervision.Source: §8 · p. 9

• SQUINTBENCH contains domains, contact strings, and policy-violating promotional material, raising privacy and dual-use concerns.Source: ETHICS STATEMENT · p. 10
• The production-derived portion is access-gated and available on approved request because its images embed real websites.Source: ETHICS STATEMENT · p. 10
• The paper states that these patterns are studied solely to improve visual-moderation robustness.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

• Generative AI assisted with code implementation, Chinese prompt-template translation, and manuscript language editing; authors reviewed and verified the output and take responsibility for the paper.Source: AI USE STATEMENT · p. 10

A.1 PER-LAYER LOCALIZATION

• Glyph-bearing-token pooling consistently improves glyph-identity separability over GLOBAL, while RANDOM tracks GLOBAL across depth and script.Both scripts peak in the middle encoder layers, and the localization gap persists across depth.Source: §A.1 · p. 16, Figure 5 · p. 16

A.2 PATCH-MERGER CONTROL

• Global glyph separability changes by at most 0.018 AUC across the 2×2 patch merger, so the merger does not explain the localization gap.Table 5 reports the before-and-after separability comparison.Source: §A.2 · p. 16, Table 5 · p. 16

A.3 PRESENCE-PROBE CONTROL

• Bypassing the ControlNet branch for text-free samples inflates text-presence AUC by roughly 0.03–0.05, so results use matched generation paths; shuffled-label controls remain at 0.47–0.52 AUC.Source: §A.3 · p. 16

A.4 GLYPH-IDENTITY PROBE AFTER TRAINING

• Chinese macro-AUC rises from 0.718 to 0.975 and Latin macro-AUC from 0.612 to 0.779 after training; presence detection was already near ceiling.The comparison is at L18, the strongest base layer shared by both scripts.Source: §A.4 · p. 16, Table 7 · p. 17, Table 6 · p. 17

B.1 EXTERNAL-BENCHMARK AND PREPROCESSING NOTES

• The 1/8 and 40×40 reduced-view rows both reach about 2562 pixels and 64 visual tokens after preprocessing, unlike direct pooling to 64 tokens, which scores 38.22 versus 29.2 for the base model.Table 3 labels describe images before preprocessing; the two 64-token conditions differ in spectrum and score.Source: §B.1 · p. 17

B.2 SPECTRAL AUDIT OF THE REDUCED VIEWS

• At matched token counts, the 40×40 view retains more low-band energy and less high-band energy than the 1/8 view, so nominal scale alone does not characterize the corrective view.The spectral comparison is reported for three SQUINTBENCH viewing conditions.Source: §B.2 · p. 17, Table 8 · p. 17

B.3 FREQUENCY AT CONSTANT TOKEN COUNT

• Every tested Gaussian low-pass setting improves on the unfiltered 26.44 ANLS baseline by 5.2–14.4 points, despite unchanged visual-token count.The gain is non-monotonic in σ; Figure 6 reports hidden-text ANLS across all 1076 SQUINTBENCH images.Source: §B.3 · p. 17, Figure 6 · p. 18

B.4 CONTRAST CONTROLS

• CLAHE yields essentially no gain, while bilateral filtering improves more; contrast-only manipulation therefore does not reproduce the frequency effect.The controls are compared against the unfiltered 26.44 ANLS baseline.Source: §B.4 · p. 18, Table 9 · p. 18

C TEACHER AND DISTILLATION CONTROLS

• With a fixed teacher, replacing its top-K distribution with a one-hot prediction lowers performance from 88.61 to 68.31, below ground-truth SFT at 72.47.The teacher-view ablation also finds that view and hint together reach 88.61, above hint-only at 41.79 and view-only at 54.68; mismatching the view reduces the paired score from 84.60 to 78.60.Source: C TEACHER AND DISTILLATION CONTROLS · p. 18

C.1 CORRECTIVE-VIEW AND TRAINING VARIANTS

• The default 1/8-view teacher reaches 88.61, outperforming probability averaging across 1/8, 1/4, and 40×40 teacher distributions, which reaches 83.05; student inference uses only the original image and task instruction.The 40×40 view retains nearly identical performance to the default 1/8 view (88.09 vs. 88.61), while the 1/4 view reaches 78.51.Source: §C.1 · pp. 18–19, Table 10 · p. 19

D RECOVERABILITY BOUNDARY

• At strength 0.5, BASE and SQUINT OPSD are statistically indistinguishable, while at 0.9 SQUINT OPSD improves from 0.886 to 0.944; at 1.2 both are near ceiling.Figure 7 shows the clearest benefit in the partially recoverable regime; training does not reconstruct glyph structure when strokes are largely destroyed.Source: Figure 7 · p. 19, D RECOVERABILITY BOUNDARY · p. 19

E.1 CANDIDATE-DISTRIBUTION METRICS

• The candidate analysis uses top-20 alternatives at aligned glyph positions; the matched-position comparison preserves its ordering, but at K=50 SFT exceeds SQUINT OPSD in ρvis.The matched intersection has n=120; for K=50, SFT is 0.170 versus 0.144 for SQUINT OPSD (BASE=0.098), so the claim is scoped to high-probability alternatives rather than the full vocabulary tail.Source: §E.1 · p. 20, §E.2 · p. 20, §E.3 · p. 20

E.4 ATTENTION-MASS DEFINITION

• AMRnorm compares glyph-region attention mass with chance-level allocation; replacing glyph-region embeddings at d=2 lowers SQUINT OPSD accuracy from 90.5% to 71.5%, while random replacement changes it from 90.5% to 91.0%.The intervention keeps token count and positions unchanged; the matched random control replaces the same number of background-token embeddings.Source: §E.4 · p. 20, §E.5 · p. 21, Table 12 · p. 21

F.1 TRAINING MIXTURE

• The training mixture contains 5905 prompts, and the 1045-example held-out split is drawn from the same source mixture for model selection and validation.The default run updates all student parameters, uses a fixed teacher copied from the initial backbone, and trains for up to three epochs; student rollouts are sampled on-policy, with gradients only through student logits.Source: Table 13 · p. 21, §F.2 · p. 21, §F.3 · p. 22

F.4 VISUAL ADAPTATION UNDER SUPERVISED FINE-TUNING

• Under supervised fine-tuning, updating the vision tower and language model reaches 72.47 on SQUINTBENCH, versus 27.29 with the vision tower frozen.This visual-adaptation dependence also appears with a standard supervised objective.Source: §F.4 · p. 22, Table 15 · p. 22

F.5 TRAINING PROMPTS

• During training, the student receives the original image and task instruction, while the teacher also receives a 1/8 view and the ground-truth output; SQUINTCTRL uses a synthetic branch with known glyph masks.OCR-style samples use a JSON list format, and the main synthetic branch uses Stable Diffusion 1.5 with QR Code Monster-style conditioning.Source: §F.5 · pp. 22–23, §F.6 · p. 23

G QUALITATIVE DEPLOYMENT EXAMPLES

• In four production-derived examples, two frontier commercial MLLMs report no text on the original image, while SQUINT OPSD reads the embedded strings from that image alone.The strings become legible to a human at reduced scale; the examples contain no personal data.Source: G QUALITATIVE DEPLOYMENT EXAMPLES · p. 23, Figure 9 · p. 25