VCR: A Task for Pixel-Level Complex Reasoning in Vision Language Models via Restoring Occluded Text

1 INTRODUCTION

• VCR challenges models to restore occluded image text, which requires aligning visual cues, language, and embedded text.Source: §1 · pp. 1–2
• Traditional VQA overlooks the relationship between embedded text and image context, while OCR-derived context can fall short when text is woven into an image's visual narrative.Source: §1 · p. 2
• The synthetic-image pipeline adjusts text visibility; VCR-WIKI contains 2.11M English and 346K Chinese Wikipedia caption examples across easy and hard levels.Source: §1 · p. 2
• The authors also construct VCR-HIDDEN to avoid potential overfitting on VCR-WIKI and release the dataset construction code.Source: §1 · p. 2
• Evaluations show significant gaps between current models and human performance on VCR.Source: §1 · p. 2

• VCR reconstructs unique text while considering visual context, unlike VQA’s non-unique responses or OCR’s transcription of complete characters without image context.In hard mode, humans can fill blanks even when pixel-level character hints do not yield unique solutions for OCR-only models.Source: §2 · p. 3
• VCR probes the distinction between low-level recognition and high-level reasoning by testing recognition of partially occluded information.The paper connects this distinction to human cognitive abilities involving low-level visual processing and high-level cognitive functions.Source: §2 · p. 3
• Varying text occlusion creates a calibrated measure of vision-language integration, while targeting partially visible, damaged, or obscured text and providing unique ground-truth answers.The paper presents VCR as a way to measure incremental progress in model capabilities and text-visual alignment.Source: §2 · p. 3

3 DATASET CREATION

• VCR-WIKI uses correlated Wikipedia image-caption pairs, after filtering instances with sensitive content such as NSFW and crime-related terms.The pipeline illustration distinguishes the visual image and embedded text in the image from separately provided string text.Source: §3 · pp. 3–4, Figure 3 · p. 4
• Each example stacks an image with its caption, resizes the result to 300 pixels wide, truncates captions to five lines, and excludes images taller than 900 pixels.The stacked image combines the visual image (VI) and text embedded in the image (TEI).Source: §3 · p. 4, Figure 3 · p. 4
• The pipeline masks selected caption 5-grams with a white rectangle, exposing upper and lower text portions while varying coverage to control difficulty.Masked spans exclude specified entity types, numbers, dates, and times; total masked tokens do not exceed 50% of the caption.Source: §3 · p. 4
• English and Chinese versions each have easy and hard variants, with hard masks leaving only one to two pixels visible for most letters or characters.The easy variant is manageable for native speakers but almost always defeats open-source OCR models; both variants remain feasible for native speakers.Source: §3 · p. 4
• To avoid test-data leakage, VCR-HIDDEN is a hidden VCR test set with non-Wikipedia image-text pairs, randomized TEI placement, easy and hard mask selection, randomly used fonts, and image widths varying uniformly from 600 to 1000 pixels.Source: §3 · p. 4

3.1 DATASET FORMAT AND STATISTICS

• Training sets contain 2,095,733 English and 336,448 Chinese instances; validation and test splits contain 5,000 entities each per configuration.VCR-WIKI has English Easy, English Hard, Chinese Easy, and Chinese Hard configurations.Source: §3.1 · p. 5

4.1 MODELS; 4.2 METRICS

• The experiments evaluate open- and closed-source VLMs, fine-tune three selected open models, and score five-word restorations using Exact Match and bag-of-words similarity.Fine-tuning uses LoRA on 16,000 examples for one epoch; generated n-grams are matched to the closest evaluation-vocabulary n-gram by edit distance.Source: §4.1 · p. 5, §4.2 · p. 5

4.3 RELATIONSHIP TO OTHER BENCHMARKS.

• VCR-EN Easy is tentatively similar to ChartQA and TextVQA, but VCR-EN Hard stands apart, indicating VCR tests aspects absent from the other evaluated benchmarks.The Chinese VCR settings were excluded from the correlation analysis because few VLMs support Chinese.Source: §4.3 · p. 6, Figure 4 · p. 6

4.4 EXPERIMENTAL RESULTS

• Current VLMs struggle with VCR, especially harder settings, and most recent models perform worse on the hidden test than on VCR-WIKI.The authors attribute weak VCR-WIKI performance to reasoning or text-image alignment gaps rather than unfamiliarity with the underlying text.Source: §4.4 · p. 6
• Strong OCR or document-understanding ability does not ensure VCR success, which the authors distinguish as requiring reasoning beyond text recognition.InternLM-XComposer2-VL, DocOwl 1.5, and Monkey perform subpar across most VCR settings.Source: §4.4 · p. 6
• Models show significant performance degradation on Chinese configurations, motivating targeted improvements to multilingual support.The authors note this decline despite models’ claimed basic English-Chinese bilingual capabilities.Source: §4.4 · pp. 6–7
• Larger models do not necessarily perform better, higher image resolution is not directly correlated with gains, and adding V_I images generally reduces performance.The authors suggest image partitioning or compression may compromise spatial relationships or local details.Source: §4.4 · p. 7
• Fine-tuning responses vary by model: CogVLM2 improves across four settings, MiniCPM-V2.5 gains marginally, and Qwen2-VL-7B remains relatively strong before and after tuning.Figure 5 compares Exact Match before and after LoRA fine-tuning across selected models and settings.Source: §4.4 · p. 7, Figure 5 · p. 9

4.5 HUMAN EVALUATION

• Fluent speakers achieve over 90% accuracy across difficulties, while current state-of-the-art models struggle with the task.Every human-evaluation sample was solved by at least three volunteers; English scores are also reported with all errors and with date and person-name errors filtered.Source: §4.5 · p. 9, Table 3 · p. 9

Visual Question Answering

• VQA datasets address knowledge-based questions, synthetic visual reasoning, and reading or reasoning about image text through Text-VQA.Examples include FVQA and OK-VQA for external knowledge, CLEVR for synthetic reasoning, and TextVQA and ST-VQA for text in images.Source: §5 · p. 9

Vision Language Model

• Vision-language models combine multimodal encoders with language models, while specialized and general-purpose systems target diverse image-and-text tasks.The discussion includes Qwen-VL, InternVL-Chat, GPT-4, Claude, Gemini, Reka, and document-focused DocLLM.Source: §5 · p. 10

Optical Character Recognition

• OCR is well studied in constrained settings, and OCR modules or OCR-generated text have been incorporated into text-aware VQA methods.The passage also notes that classical OCR often performs poorly on unconstrained images captured in the wild.Source: §5 · p. 10

6 CONCLUSION

• VCR requires integrating visual cues with partially obscured text embedded in images, beyond simply processing natural-language tokens.The authors built a dataset pipeline from correlated image-text pairs for this task.Source: §6 · p. 10
• Evaluations in English and Chinese show substantial room for improvement, while fine-tuning does not consistently produce significant gains.The authors selected models from both high- and average-performing tiers for additional fine-tuning.Source: §6 · p. 10
• VCR-WIKI is intended as a controllable testbed for analyzing model behavior across languages, difficulty levels, image inclusion, and fine-tuning stages.The authors invite further work on strategies to improve VLM performance.Source: §6 · p. 10

A STATISTICS OF VCR-WIKI AND VCR-HIDDEN

• VCR-WIKI statistics cover split sizes, image dimensions, and obscured-text patterns, while Easy and Hard variants share splits and covered texts but differ in covered area.Table 4 reports summary statistics for stacked image height and obscured spans, including mean, standard deviation, and the 5th and 95th percentiles.Source: A STATISTICS OF VCR-WIKI AND VCR-HIDDEN · p. 19, Table 4 · p. 19

B DETAILS ABOUT THE EVALUATED MODELS

• The evaluation spans diverse VLM parameter sizes, architectures, and training paradigms, with model coverage dated October 2024 for VCR-WIKI and February 2025 for VCR-HIDDEN.The models are grouped as closed-source or open-source, and their specifications are listed in Table 5.Source: B DETAILS ABOUT THE EVALUATED MODELS · p. 19, Table 5 · p. 20
• Closed-source models were evaluated through official APIs or OpenRouter APIs.The evaluated proprietary systems include models from OpenAI, Anthropic, Google, Reka, and Qwen.Source: B DETAILS ABOUT THE EVALUATED MODELS · p. 19
• Open-source selections include leading OpenVLM Leaderboard families as of May 2024 and state-of-the-art Chinese VLMs, with three listed models identified as English-only.Cambrian-1, Idefics3, and Llama-3.2 are identified as English-only; CogVLM2-Llama3-19B-Chat has a Chinese variant.Source: B DETAILS ABOUT THE EVALUATED MODELS · p. 19

C HIDDEN TEST RESULTS ON VCR-HIDDEN

• Table 6 reports VLM performance on the English VCR-HIDDEN test.It marks the best result for each setting and metric; an asterisk identifies models released after June 10, 2024, and subscripts give bootstrapped standard deviations.Source: Table 6 · p. 21
• Both tables flag post-release models with an asterisk and use subscripts for bootstrapped standard deviations.Best results are marked separately for each setting and metric in the English and Chinese tables.Source: Table 6 · p. 21, Table 7 · p. 22
• Table 7 reports VLM performance on the Chinese VCR-HIDDEN test.It marks the best result for each setting and metric; an asterisk identifies models released after June 10, 2024, and subscripts give bootstrapped standard deviations.Source: Table 7 · p. 22

D ADDITIONAL EVALUATION RESULTS ON FIRST 100 AND 500 TEST CASES

• Table 8 reports open- and closed-source vision-language model results on the VCR task using the first 100 test cases; fine-tuned models use 16,000 VCR-WIKI training samples.The best result for each setting and metric is bolded, the best open-source model is underlined, and subscripts are bootstrap standard deviations.Source: Table 8 · p. 23
• Both tables define FT as fine-tuning on 16,000 VCR-WIKI training samples and show bootstrap standard deviations as subscripts.They distinguish best open-source results with underlining and best results overall for each setting and metric with boldface.Source: Table 8 · p. 23, Table 9 · p. 24
• Table 9 reports results for open- and closed-source vision-language models on the VCR task using the first 500 test cases; fine-tuned models use 16,000 VCR-WIKI training samples.The table marks the best result for each setting and metric in bold, underlines the best open-source model, and gives bootstrap standard deviations as subscripts.Source: Table 9 · p. 24

E MORE RELATIONSHIP BETWEEN VCR AND OTHER BENCHMARKS

• Figure 6 presents pairwise correlations among 23 benchmarks, computed from 38 VLMs' benchmark scores, with colors and values spanning -1 to 1.Warmer colors indicate higher positive correlations; cooler colors indicate weaker or negative correlations.Source: E MORE RELATIONSHIP BETWEEN VCR AND… · p. 25, Figure 6 · p. 25
• VCREN, HARD has minimal or sometimes negative correlation with other benchmarks, consistent with its focus on contextual, commonsense, and visual-narrative reasoning rather than straightforward image-text association or OCR.The passage characterizes this skill set as largely unrelated to those tested by popular tasks emphasizing those capabilities.Source: E MORE RELATIONSHIP BETWEEN VCR AND… · p. 25
• VCREN, EASY shows moderate overlap with a few benchmarks such as TextVQA but remains moderately independent of most others.Its easier connections and additional cues accompany visual commonsense reasoning, while retaining more interpretative reasoning than standard text-based vision benchmarks.Source: E MORE RELATIONSHIP BETWEEN VCR AND… · p. 25

F POTENTIAL QA

• The proposed inference pipeline divides images into segments, encodes them as visual tokens, filters those tokens, then combines them with language tokens for the LLM.The analysis says that disregarding pixel-level hints during filtering can prevent correct answers, particularly on hard partitions.Source: F POTENTIAL QA · p. 26
• The authors suggest including VCR in VLM pretraining and exploring multimodal chain-of-thought methods to refine attention to visual clues.They describe pretraining inclusion as analogous to including OCR parsing tasks and propose iterative focus refinement for chain-of-thought methods.Source: F POTENTIAL QA · p. 26
• CogVLM2 is reported as the best model on average across four partitions, which the authors largely attribute to its vision expert architecture.CogVLM2 and Cambrian-1 omit the token-filtering step and use mid-to-high-resolution encoders; the authors note possible challenges for extremely shaped images, while VCR-WIKI images are mostly near-square.Source: F POTENTIAL QA · p. 26
• Fine-tuning CogVLM2 on VCREN, HARD raised VCRZH, EASY from 9.15 to 42.55, while Chinese CogVLM2 fine-tuned on VCRZH, HARD improved VCREN, EASY and HARD scores.The reported VCREN scores rose from 79.9 to 87.57 on EASY and from 25.13 to 44.97 on HARD.Source: F POTENTIAL QA · pp. 26–27
• VCR-WIKI fine-tuning is reported to improve performance across benchmarks and language settings, with models reaching their highest scores within each model’s fine-tuning variations.The authors describe this as evidence of transferability across evaluation metrics, tasks, and linguistic contexts.Source: F POTENTIAL QA · p. 27

G REAL-WORLD CASE STUDY OF MODELS FINE-TUNED ON VCR-WIKI

• The real-world case study evaluates MiniCPM-V2.5 8B, CogVLM2 19B, and Qwen2-VL 7B on 11 photographs, using a curated set as a proxy because no standard benchmark exists.The examples include street maps and collected images; the evaluation marks text as fully recovered or partially correct or incorrect.Source: G REAL-WORLD CASE STUDY OF MODELS… · p. 28

DO NOT SIGN... VISIBLE DAMAGE?

• The “DO NOT SIGN... VISIBLE DAMAGE?” example lists “AMOUNT TIP TOTAL APPROVED” as its ground-truth text.Source: Figure 17 · p. 29

AMERICAN EXPRESS AID THANK YOU / MERCI CUSTOMER COPY

• The result marks both Qwen2-VL-7B and Qwen2-VL-7B-ft correct; the supplied figure-caption excerpt labels “Ground-truth:” but does not include the ground-truth text.Source: AMERICAN EXPRESS AID THANK YOU / MERCI… · p. 29, Figure 16 · p. 29