Source-linked AI summary
Sorry Robot, Happy Human: Vision-Language Models Read Only One of Two Legible Typographic Layers
Mert İncidelen, Yamen Kashkash, Asya Berker, Murat Aydoğan
TL;DR
Prior studies generally examine typographic attacks from added image text, whereas this study asks how well models read two superimposed layers that humans can distinguish. The study evaluates recent closed-source VLMs on DecoyBench with naive and guided prompts to measure extraction of both text layers. At 512 × 512, contour accuracy was close to human performance, but no model exceeded 1% EM on shading; when sharp contour lines disappeared, contour text was unreadable and shading EM was 93.7%–100.0%.
Problem
Prior studies generally examine typographic attacks from added image text, whereas this study asks how well models read two superimposed layers that humans can distinguish.
Source: §1 · p. 1Method
The study evaluates recent closed-source VLMs on DecoyBench with naive and guided prompts to measure extraction of both text layers.
Source: §1 · p. 2, §3.2 · p. 3Results
At 512 × 512, contour accuracy was close to human performance, but no model exceeded 1% EM on shading; when sharp contour lines disappeared, contour text was unreadable and shading EM was 93.7%–100.0%.
Source: §4 · pp. 3–4Takeaways & Limitations
Shading-layer reading failure persisted across all three evaluated model families and may matter for automated processing of watermarks, stamps, and archival documents.
Source: §5 · p. 4The benchmark uses 300 examples of one Decoy Font style, English text pairs, and only two resolutions; human validation involved 10 participants.
Source: Limitations · p. 4
Abstract
from arXiv · showhide
Vision-language models (VLMs), despite their success in optical character recognition (OCR) tasks, are vulnerable to typographic attacks and have a fragile structure for images with multiple text layers. In this study, the DecoyBench dataset was created using the Decoy Font method. The dataset consists of 300 images, each containing text with sharp contour lines superimposed on another text with soft shading. Six recent closed-source models from three different model families were evaluated using this dataset under two different prompting conditions (naive and guided) and at two different resolutions ($512\times512$ and $64\times64$). A validation study showed that human participants could read both text layers with high accuracy. In contrast, the models, with most variants and both prompting methods, read the contour text with near-human accuracy at high resolution, but almost never fully extracted the shading text. At low resolution, the contour text could not be read by either the models or humans, while the shading text could be extracted with high accuracy. The findings indicate that the evaluated VLMs exhibit a consistent behavioral limitation when processing typographic structures containing multiple spatial frequency layers.
1 Introduction
Although VLMs succeed at OCR, prior work documents their vulnerability to text added to images. This study asks whether they can recover both superimposed layers that people can distinguish, evaluating recent closed-source VLMs with DecoyBench.
1 Introduction
- Prior work shows that typographic attacks can reduce model performance, with text properties and embedded prompts also affecting model behavior.Reported effects include changes in font size, color, and opacity, as well as handwritten text and prompts embedded in images.Source: §1 · p. 1
- Unlike studies focused on text added to images as an attack, this study examines whether models can read two superimposed layers distinguishable by humans.The motivation draws on differences between model and human visual processing, including human perception of hybrid images at different viewing distances.Source: §1 · p. 1
- The study evaluates recent closed-source VLMs on DecoyBench to test their ability to extract both texts in images containing human-distinguishable superimposed text.The Decoy Font method places sharply outlined decoy text alongside a shaded hidden letter form.Source: §1 · pp. 1–2
2 The DecoyBench Dataset
DecoyBench contains 300 images of paired, superimposed contour and shading text, with human legibility checked at two resolutions. Participants read both layers well at high resolution, while low-resolution readability differed by layer.
2.1 Construction
- DecoyBench pairs contour text made of thin, sharp high-frequency lines with softly graded shading text in a shared visual plane.The paired texts match in word count and character length; Figure 1 shows “SORRY ROBOT” overlaid with “HAPPY HUMAN”.Source: §2.1 · p. 2, Figure 1 · p. 2
- The benchmark comprises 300 matched text pairs rendered as images at 512 × 512 and 64 × 64 pixels, with contour and shading labels.The generated images use the Decoy Font method.Source: §2.1 · p. 2
2.2 Human Legibility Verification
- Ten independent participants evaluated the 300 images, split into five groups of two, at both resolutions without viewing-time or distance restrictions.Results combine the performance averages of the five participant groups.Source: §2.2 · p. 2
- At high resolution, participants read 99.7% of contour text and 96.3% of shading text; at low resolution, contour text was unreadable and shading text reached 98.7%.Participants read both layers completely in 96.0% of images at high resolution; these results are reported as Exact Match (EM).Source: §2.2 · p. 2, Table 2 · p. 2
3 Experimental Setup
The experimental setup evaluates six models from three families using naive and guided prompts. Model outputs are compared with validated layer text using exact-match and normalized similarity measures.
3.1 Evaluated Models
- Six models from the GPT, Gemini, and Claude families were evaluated; they were the most recent versions of their model lines available via API at experiment time.The evaluated models were GPT-5.6 Luna and Terra, Gemini 3.6 Flash and 3.5 Flash-Lite, and Claude Sonnet 5 and Haiku 4.5.Source: §3.1 · pp. 2–3
3.2 Prompting Strategies
- Naive prompts measure direct transcription without layer information, while guided prompts support layer distinction and incremental inference.Both prompt strategies used a strict JSON output schema and formatting rules to standardize model outputs.Source: §3.2 · p. 3
3.3 Evaluation Strategies
- Outputs were compared with validated contour and shading text using Exact Match (EM) and normalized Levenshtein Similarity (LS).EM tests for an identical output; LS scores approximation to the target from zero to one.Source: §3.3 · p. 3
4 Results and Discussion
At high resolution, models read contour text far better than shading text, despite human legibility of both. At low resolution, shading text becomes readable while contour text does not.
4 Results and Discussion
- Shading-text failure was consistent across model variants within each family, while contour performance was comparable across Gemini and Claude variants.Under naive prompting, the GPT variants showed a larger contour-performance gap.Source: §4 · p. 3
- At 512 × 512, guided contour accuracy reached 94% or higher, while no model exceeded 1% shading-text EM, even with guided prompting.GPT-5.6 Luna reached 99.7% contour EM in the guided condition; human shading-text EM was 96.3%.Source: Table 3 · p. 3, §4 · p. 3
- Guided prompting generally increased multiple-text outputs, especially for Gemini, but the increase did not correspond to true text detection.Table 4 reports the average number of text outputs; Table 3 reports transcription accuracy.Source: Table 4 · p. 4, §4 · p. 4
- Some outputs captured a few shading-text letters but completed the sequence with possible letters; most alternative expressions were unrelated to the actual shading text.The authors suggest models may produce a second text because the instruction requests one, rather than extracting it from the image.Source: §4 · p. 4
- When sharp contour lines disappeared, both humans and all models were unable to read contour text, while shading-text EM was 98.7% for humans and 93.7%–100.0% across models.The authors suggest that models may prioritize high-frequency contour information when high- and low-spatial-frequency cues coexist.Source: §4 · p. 4
5 Conclusion
DecoyBench tests whether VLMs can parse superimposed typographic layers. Models largely failed on shading text at high resolution but read it at low resolution, where contour text was unreadable; the pattern appeared across all three evaluated model families.
5 Conclusion
- DecoyBench was created to evaluate VLMs’ ability to parse superimposed typographic layers.Source: §5 · p. 4
- At high resolution, humans read both layers with high accuracy, but models failed to read the shading text; at low resolution, contour text was unreadable while shading text was read with high accuracy.Source: §5 · p. 4
- The high-resolution shading-text failure appeared across all three evaluated model families, and the authors identify automated processing of real-world superimposed-text documents as a possible implication.Watermarks, stamps, and archival documents are examples; future studies could test similar methods on those document types.Source: §5 · p. 4
Limitations
The study’s findings are bounded by its dataset design, evaluation settings, language scope, and small human-validation sample.
Limitations
- DecoyBench contains 300 examples in a single style from the Decoy Font method.Source: Limitations · p. 4
- The study did not test text layers in isolation and used only two resolutions; effects of font, contrast ratio, and shading density were outside its scope.Source: Limitations · p. 4
- Evaluation used English text pairs, and human legibility validation involved 10 participants, limiting generalizability to other languages and larger populations.Source: Limitations · p. 4
A Human Legibility Verification Instructions
Participants were asked to transcribe the text they could read in each image, keeping separate texts distinct and avoiding guesses.
A Human Legibility Verification Instructions
- Participants were instructed to write down the texts they could read in the provided image.Source: A Human Legibility Verification… · p. 5
- They were told to write words from the same text together and put each readable text on a separate line.Source: A Human Legibility Verification… · p. 5
- Participants were instructed to report only what they actually read, avoid uncertain guesses, and leave the response blank if nothing was readable.Source: A Human Legibility Verification… · p. 5
B Prompting Templates
Models shared one system prompt and received either a naive or guided user prompt. The guided prompt explicitly separates reading the contour and shading layers, while the system prompt constrains output to complete, non-guessed text items in JSON.
B Prompting Templates
- Both prompting conditions use a shared system prompt and differ in the user prompt.Source: B Prompting Templates · p. 5
B.1 Naive Prompt
- The naive user prompt simply asks models to transcribe all text phrases in the image.Source: §B.1 · p. 5
B.2 Guided Prompt
- The guided prompt describes two superimposed texts, with sharp contour lines and diffuse shading overlapping at each letter position.Source: §B.2 · p. 5
- The guided prompt first asks models to read the thin contour text, then to ignore it and read the broad shading pattern.The shading instruction frames the image as blurred, with sharp lines gone.Source: §B.2 · p. 5
- The response lists contour text first and shading text second, with each layer's words kept together in one item.If only one text is read, the response contains only that item.Source: §B.2 · p. 5
B.3 System Prompt
- The system prompt requests an OCR-expert response as a valid JSON object without markdown or commentary.Its schema uses a texts array, which may contain any number of items, including none.Source: §B.3 · p. 5
- Each texts item must contain one complete text read as a unit, not fragments, and models are told to report only what they actually read.The example treats “THE TREES HAVE BLOSSOMED” as one item rather than three.Source: §B.3 · p. 5