SCAM: A Real-World Typographic Robustness Evaluation for Multimodal Foundation Models

• Typographic attacks insert misleading readable text into images, prompting models to focus on text rather than visual content and make incorrect predictions.Source: §1 · p. 2
• SCAM contains real-world images with unrelated handwritten attack text, alongside clean NoSCAM and digitally reintroduced SynthSCAM variants.Source: §1 · p. 2
• Both VLMs and LVLMs lose accuracy on real-world typographic attacks, and synthetic attacks produce comparable performance drops.Source: §1 · p. 3
• LVLMs including LLaVA remain vulnerable, reflecting weaknesses in their vision encoders.Source: §1 · p. 3
• Larger LLM backbones reduce LVLM susceptibility while improving typographic understanding.Source: §1 · p. 3

• Multimodal pretraining aligns image and text in a shared latent space, while LVLMs combine language models with vision encoders.Source: §2 · p. 3
• Prior work studies typographic attacks through synthetic additions, visual prompt injection, self-generated attacks, and scene-coherent attacks.Source: §2 · p. 3
• Existing real-world typographic attack datasets have limited scale and diversity, while SCAM expands object–word variability and aligns real, clean, and synthetic versions.Source: §2 · pp. 4–5
• SCAM examples reflect variation across nine contributors’ smartphones, handwriting, and recording environments.Source: Figure 2 · p. 5
• SCAM’s three aligned components pair attacked scenes with clean counterparts and synthetic versions of the same object–word pairs for controlled comparisons.Source: §2 · p. 5

3 SCAM Dataset

• SCAM pairs each object with an unrelated handwritten English attack word on a nearby post-it note, chosen as a realistic, reproducible attack surface.Contributors collected images across indoor and outdoor environments, with varied lighting and viewing angles.Source: §3 · p. 6
• NoSCAM obscures attack words while preserving the object images, and SynthSCAM digitally reintroduces the words in Roboto to create aligned clean, real, and synthetic versions.The synthetic variant uses a uniform font style as a deliberate simplification.Source: §3 · pp. 5–6
• SCAM contains 1,162 images, 660 object labels, and 206 attack words, exceeding RTA-100’s 100 objects and 100 attack words.The dataset also includes 1,147 unique object-word combinations.Source: §3 · p. 6, Table 2 · p. 7
• The dataset depicts everyday objects and contains no personal, biometric, or otherwise sensitive data; it is released for research use under its dataset-card license.Users are asked not to deploy models trained on SCAM to harm or deceive people.Source: §3 · p. 7

4 Evaluation

• The evaluation tests object prediction on SCAM, NoSCAM, SynthSCAM, PAINT, and RTA-100 to assess the effects of real-world and synthetic typographic attacks.The target is the depicted object rather than the attack word.Source: §4 · p. 7
• For VLMs, zero-shot predictions are selected by comparing image embeddings with text embeddings for the object and attack-word labels.The text embeddings use templates proposed by Radford et al. (2021).Source: §4 · p. 8
• For LVLMs, the evaluation checks whether the model changes its answer from the object label to the attack-word label when the image is modified.The prompt asks the model to choose between the two labels; their order is randomized to reduce positional bias.Source: §4 · p. 8
• Table 3’s average confusion matrices indicate that synthetic SynthSCAM attacks closely replicate the effects of real SCAM attacks.Values are mean ± standard deviation, rounded to integers.Source: Table 3 · p. 8

Zero-Shot Results for VLMs

• On average, VLM accuracy drops approximately 26 percentage points on SCAM and 36 percentage points on SynthSCAM, while synthetic attacks closely mirror real-world effects.The accuracy distributions across NoSCAM, SCAM, and SynthSCAM are shown for 99 VLMs.Source: Figure 4 · p. 9, §5 · p. 9

Impact of Training Dataset and Model Architecture

• Training data and architecture influence VLM susceptibility: LAION-trained ViT-B models are highly susceptible, while CLIP-ViT is more resilient than CLIP-RN and SigLIP-ViT more resilient than CLIP-ViT on average.Some CommonPool variants show lower vulnerability; the commonpool_l_clip filtering criterion appears to propagate susceptibility.Source: §5 · pp. 9–10, Table 5 · p. 10

Impact of Patch Size and Attack Size

• Smaller patch sizes generally increase susceptibility, and accuracy decreases as the post-it attack area grows.The post-it-area trend is reported across the 99 evaluated VLMs.Source: §5 · p. 10, Figure 6 · p. 11

Impact of Model Size

• There is no noticeable correlation between VLM size and the accuracy drop caused by typographic attacks, though original OpenAI CLIP models perform worse across sizes.Figure 5 summarizes susceptibility against model size.Source: §5 · p. 10, Figure 5 · p. 11

Prompt-Based Classification Results for LVLMs

• Larger LLM backbones can reduce LVLM susceptibility and improve typographic understanding, although vision-encoder weaknesses remain and model size alone does not determine robustness.Some smaller models show drops above 40–60 percentage points, while Claude Sonnet 4 and GPT-4o show drops of around 8 and 3 points, respectively.Source: §5 · pp. 10–12

Impact of VLM and LLM Size on LVLM Performance

• A preliminary vision-encoder substitution experiment found no clear robustness gain from choosing an encoder ranked safer by SCAM, and overall performance decreased.Scaling the LLM backbone also improves performance on SCAM and general typographic understanding on textual and OCR-based benchmarks.Source: §5 · pp. 11–12

Findings

• Synthetic attacks are just as effective as real-world attacks, producing comparable performance drops across the evaluated models.The discussion situates this finding alongside prior work on typographic-attack vulnerability.Source: §6 · p. 12
• Training regime, training data, and architecture influence VLM susceptibility, whereas model size alone does not correlate with robustness; larger LLM backbones can mitigate LVLM vulnerability and improve typographic understanding.LVLM vulnerability appears to carry over from the vision encoder.Source: §6 · p. 12

Limitations

• The evaluation relies on naive prompting and covers English-language attacks only, leaving multi-prompt evaluation and language-specific susceptibility for future investigation.The authors note that more advanced multi-prompt strategies may yield a more reliable robustness measure.Source: §6 · p. 12

Future Work

• Aligned NoSCAM/SCAM pairs support controlled causal and mechanistic analyses, while simply swapping LLaVA’s encoder for a more robust one did not reliably transfer robustness when integrated naively.Future work proposes systematic retraining with varied vision backbones to assess encoder strength and adaptation needs.Source: §6 · p. 12

Conclusion

• SCAM, NoSCAM, and SynthSCAM form the paper’s largest and most diverse real-world image collection for evaluating multimodal models’ vulnerability to typographic attacks.The authors expect their analysis of VLMs and LVLMs to help researchers choose typographically robust models for safety-critical downstream applications.Source: §7 · p. 13

Risks

• Typographic cues could let malicious actors probe deployed vision-language systems and mislead them in safety-critical settings such as medical decision support, navigation, or content moderation.The released dataset could also be used to study loopholes with the aim of designing more effective attacks.Source: Broader Impact Statement · p. 13

Benefits

• SCAM provides a standardized benchmark for comparing defenses and studying how training data and model design affect susceptibility.Because synthetic attacks reliably mirror real-world ones, they can reduce the need for ongoing collection of handwritten examples.Source: Broader Impact Statement · p. 13

• Figure 7 shows examples from SCAM and the generated NoSCAM and SynthSCAM datasets.Source: §A.1 · p. 20, Figure 7 · p. 20

• SCAM comprises 1,162 images with 660 objects and 206 attack words, collected by nine contributors across diverse indoor and outdoor environments.Source: §A.2 · p. 21
• PAINT and RTA-100 cover comparatively restricted sets of objects, attack words, and scene conditions relative to SCAM.Source: §A.2 · p. 21
• The figures show 20 random examples each from PAINT and RTA-100, but these samples cannot capture either dataset’s full variability.Source: §A.2 · p. 21, Figure 8 · p. 21, Figure 9 · p. 21

A.3 Zero-Shot Evaluation of LVMs: Text Templates

• Zero-shot evaluation averages embeddings from 32 text templates, each placing an object label or attack word in a different context.Source: §A.3 · p. 21

A.4 Model Accuracies on Typographic Attack Datasets

• The model-accuracy table covers VLMs and LVLMs on SCAM, PAINT, and RTA-100, while the appendix provides its model/training-data headings.Source: Table 6 · p. 22, §A.4 · p. 23

A.5 Architecture Comparison

• The architecture comparison reports averaged accuracies for CLIP-RN versus CLIP-ViT and CLIP-ViT versus SigLIP-ViT, with models grouped by architecture and fixed training data.Source: §A.5 · p. 24, Table 7 · p. 24

A.6 Prompt Variation

• Appending “Do not get fooled by typographic attacks” to the original prompt leaves susceptibility largely unchanged.Source: §A.6 · p. 24, Figure 10 · p. 25

A.7 Replacing VLM in LLaVA

• The CLIPA-based LLaVA shows a 25% robustness drop from NoSCAM to SCAM, versus 65% for the OpenAI-encoder model.It also outperforms the OpenAI encoder by 3% on average across 38 OpenCLIP-evaluated datasets.Source: §A.7 · p. 25
• The CLIPA-based LLaVA variant does not improve SCAM performance over the original model, despite using a more robust vision encoder in isolation.Source: §A.7 · p. 26, Table 8 · p. 26

A.8 Typographic Robustness vs. Typographic Understanding

• Larger LLM backbones are associated with improved typographic understanding and robustness, although stronger reading ability alone does not guarantee robustness.Source: §A.8 · p. 26, Table 9 · p. 27

A.9 SCAM Labels

• SCAM’s label frequencies have long tails: 437 object labels and 56 attack words appear exactly once.The tables summarize the most and least frequent labels for each category.Source: §A.9 · pp. 29–30, Table 10 · p. 29, Table 11 · p. 30