PaperlayerView exact revision on arXiv

Source-linked AI summary

ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction

Bojun Zhang, Junhong Liang, Feifei Zhai, Fengxian Ji, Yu Zhou

arXiv:2608.27154v1cs.CV

TL;DR

VLMs perform strongly on general visual tasks, but their deep understanding of text embedded in images remains insufficiently evaluated. ReViCo addresses this gap with a real-world visual text error-correction benchmark and evaluations using prompting and targeted training. Results show a substantial gap between current VLMs and human-level evaluation despite improvements from both strategies.

  • Problem

    VLMs excel at general visual processing, but the depth of their understanding and interaction with text embedded in images remains underexplored.

  • Method

    ReViCo evaluates VLMs on visual text error correction in real-world Chinese and English images using prompt-based strategies and targeted model training.

  • Results

    Despite improvements from prompt enhancement and targeted training, experiments reveal a significant performance gap between current VLMs and human-level evaluation.

  • Takeaways & Limitations

    ReViCo provides a benchmark foundation for developing more robust and text-aware VLMs by exposing limitations in fine-grained visual-textual perception.

  • Takeaways & Limitations

    Validation is confined to English and Chinese datasets, so generalizability to other languages, particularly low-resource ones, remains unexplored.

Abstract

from arXiv · show

Vision Language Models (VLMs) have shown great success in general visual tasks, yet they still struggle to deeply understand text within images. In this paper, we introduce ReViCo (Real Visual Correction), a benchmark designed to evaluate VLM text understanding through a novel task of visual text error correction. ReViCo challenges models to identify and fix text errors in real-world images, which requires a profound understanding of the interplay between visual text and its surrounding visual context. We benchmark various VLMs using two distinct paradigms: prompt-based strategy and targeted model training, both aimed at pushing the limits of current models. Our experiments reveal a striking performance gap between even the best VLMs and human, and further analysis also shows that most models struggle to accurately perceive the visual text, resulting in frequent correction errors. By highlighting these gaps, ReViCo provides a new benchmark foundation for developing more robust and text-aware VLMs.

1 Introduction

ReViCo introduces visual text error correction as a benchmark for whether VLMs understand embedded text together with its visual context. Evaluations using prompting and targeted training still reveal a substantial gap between VLMs and humans.

  • VLMs perform well on general multimodal tasks, but the depth of their understanding of text embedded in images remains underexplored.
  • Visual text error correction requires models to integrate fine-grained visual features with linguistic context rather than merely identify characters.Examples include resolving scene-text semantic mismatches and phonetic or typographical similarities.
  • ReViCo is a benchmark of real-world Chinese and English images with natural text errors, evaluating error detection and error correction.Detection identifies incorrect tokens or images, while correction restores their correct forms.
  • The benchmark evaluates VLMs through prompt-based strategies and targeted model training designed to improve fine-grained perception of visual text.The prompting strategies include Direct Correction and Background Information Enhancement.
  • Despite improvements from prompt enhancement and targeted training, experiments reveal a significant performance gap between current VLMs and human-level evaluation.The findings identify fine-grained visual text understanding as a continuing bottleneck.

2 Related Works

Related work situates VLMs at the intersection of visual-language alignment, text error correction, and reinforcement learning for visual reasoning. Existing approaches reuse strong unimodal foundations but must preserve fine-grained visual information.

  • Architecture of VLMs: Modern VLMs commonly project features from pretrained vision encoders into the embedding space of a pretrained language model.This reuses unimodal foundations while requiring precise alignment to retain fine-grained perceptual information.
  • Text Error Correction: Chinese spelling errors often involve phonetic or graphical similarities, whereas English errors commonly involve character-level substitutions or deletions.
  • Reinforcement Learning for Visual Reasoning: Reinforcement learning methods such as RLHF, PPO, DPO, and KTO have been used to improve reasoning or optimize preferences in language models.
  • Reinforcement Learning for Visual Reasoning: Recent visual reasoning work explores latent reasoning in visual embedding space and tool-augmented reinforcement learning for acquiring and verifying task-relevant information.

3 ReViCo Benchmark

The ReViCo benchmark combines manually curated real-world test data with automatically synthesized training data and evaluates visual text errors at image and token levels. Its protocol separates detection from correction and supports systematic analysis of Chinese and English error types.

  • Data Construction: ReViCo uses manual annotation for precise test data and automated synthesis for scalable training data construction.
  • Test Data Construction: The manually curated test set selects images whose text errors can be resolved only through visual cues and contains 1229 high-quality real-world samples.
  • Automated Training Data Construction: The training set is generated from EST-VQA and ST-VQA through a three-stage Generate-Edit-Examination pipeline using GPT-4o to introduce textual distortions.
  • Error Type Statistics: Chinese errors are categorized into phonetic, graphic, phonetic-graphic, and other types, while English errors are classified at letter and word levels.English letter-level categories include omission, insertion, substitution, and transposition; word-level categories include split-word errors.
  • Evaluation Protocol: The evaluation defines Detection and Correction tasks at Image and Token levels for both Chinese characters and Latin letters.Image Detection is binary, Token Detection pinpoints erroneous units, Token Correction transforms them, and Image Correction requires resolving all errors without unnecessary changes.
  • Evaluation Protocol: The benchmark reports precision, recall, and F1 scores for every task and evaluation level.

4 Evaluation Stratetgies

ReViCo evaluates visual text understanding through OCR+LLM baselines, prompting strategies, and targeted GRPO training. Its training pipeline uses hierarchical rewards to enforce valid formats, image-level error discrimination, and token-level correction quality.

  • Evaluation Paradigms: The evaluation compares an OCR+LLM baseline, direct correction prompting, background-information enhancement, and targeted model training.Oracle OCR text is used for the OCR+LLM baseline, while background enhancement asks models to describe visual context before correction.
  • Targeted Model Training: GRPO training uses automatically constructed datasets to refine fine-grained visual text understanding.The paper selects reinforcement learning because it reports greater robustness against forgetting than supervised fine-tuning.
  • Reward Design: Multi-Level Verification Reward decomposes feedback into format, image-level discrimination, and token-level correction stages.The hierarchy ensures foundational response constraints are satisfied before optimizing fine-grained correction.
  • Reward Design: The reward penalizes invalid formats and balances detection, correction, and redundant corrections.The correctness reward includes a length penalty for over-generated corrections, with α set to 0.1.

5 Experiments

Experiments compare open- and closed-source VLMs across image- and token-level detection and correction. Results show that correction is harder than detection, prompting and GRPO help unevenly, scaling is not uniformly beneficial, and humans remain substantially better.

  • Experimental Setup: The benchmark evaluates open-source Qwen and InternVL models alongside closed-source GPT-4o, GPT-5.4, Gemini2.5-Pro, and Seed-1.8 systems.Table 2 reports direct and background-enhanced prompts plus GRPO training across image- and token-level detection and correction.
  • Direct Prompting Results: Seed-1.8 and Gemini2.5-Pro reach 86.1 and 87.3 F1 in image detection but 67.0 and 75.9 in token correction.Qwen3-VL-235B also drops from 75.9 in image detection to 35.4 in image correction, illustrating the correction challenge.
  • Prompt Enhancement: Background information enhancement can hurt small models while substantially improving larger models.The reported explanation is that weaker visual processing in small models limits their ability to use additional contextual information.
  • Targeted Training: GRPO with MLVR improves most metrics across trained models, including +9.7 F1 in image detection and +7.9 in token correction for Qwen2.5VL-3B.Qwen2.5VL-7B gains +31.6 F1 in image detection but loses 0.4 in token detection.
  • OCR Comparison: Deep visual integration surpasses the best Oracle OCR token-correction score, with Qwen2.5VL-32B and Qwen3VL-32B reaching 36.5 and 32.7 versus 31.7.The comparison indicates that larger models can exceed the performance ceiling observed with text-only Oracle OCR inputs.
  • Attention Analysis: Attention case studies show that Qwen2.5-VL-7B may miss erroneous characters, hallucinate absent characters, and overlook multiple errors.Figure 4 compares original images, attention maps from layers 22 and 25, golden labels, and predictions.
  • Scaling Analysis: Increasing model size does not guarantee gains across metrics: Qwen2.5-VL-72B has 33.1 T-C versus 36.5 for Qwen2.5-VL-32B.Smaller models can achieve high recall through over-correction, whereas larger models can achieve high precision by assuming images are error-free.
  • Human Comparison: Human evaluation on 200 images shows a substantial advantage over VLMs in image-level detection and correction.Humans achieve 94.5% detection success, while Gemini manages 57 correctly and Qwen3-VL-32B achieves 26 in the reported counts.

6 Conclusion

ReViCo evaluates VLMs through Visual Text Error Correction and finds that prompting and targeted training still leave a substantial gap from human-level reasoning. The findings point to persistent difficulty integrating visual cues with textual semantics.

  • ReViCo evaluates VLMs through Visual Text Error Correction, requiring synthesis of textual functions within complex visual contexts.
  • Prompting and targeted training improve performance, but a substantial gap remains between state-of-the-art VLMs and human-level reasoning.
  • The results expose persistent difficulty bridging visual cues with textual semantics.
  • ReViCo motivates future research on fine-grained integration of visual perception and linguistic reasoning for text-aware multimodal systems.

7 Limitations

The benchmark and training methods are intended to be language-agnostic, but validation currently covers only English and Chinese. Additional boundaries include the use of pretrained VLMs and a focus limited to visual text errors.

  • Validation is confined to English and Chinese because accessible data resources for other languages are limited.
  • Generalizability to other languages, particularly low-resource ones, remains to be explored.
  • The evaluated models are pretrained VLMs, so performance could vary with task-specific fine-tuning.
  • The benchmark focuses on visual text errors and excludes grammatical or contextual mistakes.
  • Despite substantial improvements, a performance gap remains relative to human-level precision.

8 Ethical Considerations

The study describes safeguards for lawful data use, privacy protection, and annotation reliability. The test set uses real-world images, with independent annotation, quality checks, and adjudication of disagreements.

  • Benchmark data were collected under source terms of use and copyright policies for academic research, with distribution compliant with original licenses.
  • Personally identifiable information and sensitive metadata were manually screened and removed to protect privacy.
  • The test set contains 730 Chinese and 499 English instances from signs, menus, product labels, and other everyday scenes.
  • Each instance was independently annotated by two corresponding-language annotators after receiving detailed task guidelines.
  • 92.6% initial image-level agreement was followed by third-annotator adjudication of 91 disagreements, yielding Cohen’s κ=0.852.

A Error Type and Examples in ReViCo

This section points to the benchmark’s error-type tables and examples while documenting prompt templates and reinforcement-learning training-data construction. The available passages provide limited detail beyond these references and implementation snippets.

  • The section directs readers to Table 3 for error types and Figure 6 for ReViCo error examples.
  • Reinforcement-learning training data are constructed through a three-stage pipeline, with error statistics reported in Tables 4 and 5.
  • Chinese Visual Text Correction: Chinese visual text correction prompts require identifying incorrect characters and returning their corrections in a prescribed format.
  • Chinese Visual Text Correction: The Chinese prompt restricts checking to text errors, including spelling and misuse, while excluding grammar and punctuation errors.
  • English Visual Text Correction: English visual text correction prompts require checking every word and returning error-correction pairs in a prescribed format.
  • Table 6 contains the prompts used for Background Information Enhancement.

E Calculation of Metrics

ReViCo evaluates visual text error understanding at image and token levels, separating detection from correction and penalizing both missed and excessive corrections. Its metrics combine error awareness, localization, correction accuracy, and holistic exactness.

  • ReViCo represents each image with a possibly empty set of gold error–correction pairs and maps no-error predictions to an empty set.
  • Detection: Image Detection (I-D) is a binary decision about whether an image contains any visual text error.
  • Detection: Token Detection (T-D) evaluates localization of erroneous tokens while ignoring the proposed corrections.
  • Correction: Image Correction (I-C) requires every error in an image to be fixed exactly, making it the strictest correction setting.
  • Correction: Token Correction (T-C) evaluates exact error–correction pairs independently of whether all errors are corrected.
  • The four metrics jointly assess error recognition, token localization, individual correction quality, and complete image-level reliability while penalizing missed and over-corrections.

F OCR tools + LLM Experiments Results

The experiments compare OCR-plus-LLM and image-plus-VLM approaches, including OCR tools, prompting, and GRPO-based targeted training. Oracle OCR performs best for OCR pipelines, while GRPO benefits image-based correction more substantially but can also introduce failures.

  • The benchmark evaluates open-source and closed-source VLMs using prompt strategies, OCR systems, and targeted GRPO training.
  • OCR+LLM: Oracle OCR Text consistently yields the best performance across the evaluated LLMs, whereas imperfect OCR introduces cascading recognition errors that degrade correction.
  • Prompting: Background information enables GPT-4o to use surrounding spice-container context to correct 处 to 醋, while QwenVL-32B misses the correction.
  • Targeted Model Training: GRPO successfully corrects the graphic error 花哈 to 花蛤 in one Qwen2.5-VL-3B example by capturing fine-grained visual details.
  • Case Studies: After GRPO training, Qwen3-VL-8B fails on an image the base model corrected successfully, despite identifying more accurate visual details.
  • Performance Comparison: GRPO produces significant gains in Image+VLM but only marginal improvements in OCR+LLM, indicating stronger benefits for visual grounding than pure-text correction.
Loading 2608.27154v1…