Source-linked AI summary
OCR-Agent: Agentic OCR with Capability and Memory Reflection
Shimin Wen, Zeyu Zhang, Xingdou Bian, Hongjie Zhu, Lulu He, Layi Shama, Daji Ergu, Ying Cai
TL;DR
VLM self-correction can become unstable when models propose infeasible actions or repeat ineffective revisions. OCR-Agent addresses this with Capability Reflection and Memory Reflection, and its OCRBench v2 evaluation reports improvements over InternVL3-8B and other refinement strategies without additional fine-tuning.
Problem
VLM self-correction is limited by capability hallucination and refinement stagnation, with models proposing infeasible actions or repeating ineffective correction loops.
Method
OCR-Agent uses Capability Reflection to diagnose errors and filter infeasible corrective actions, and Memory Reflection to retain reasoning history and avoid redundant attempts.
Results
+2.0 points on English and +1.2 points on Chinese OCRBench v2 subsets over InternVL3-8B.
Takeaways & Limitations
Structured self-reflection enables more consistent and sustainable VLM self-correction without additional fine-tuning.
Takeaways & Limitations
Iterative reflection increases inference time and computational cost, which may hinder real-time deployment.
Abstract
from arXiv · showhide
Large Vision-Language Models (VLMs) have demonstrated significant potential on complex visual understanding tasks through iterative optimization methods.However, these models generally lack effective self-correction mechanisms, making it difficult for them to independently rectify cognitive biases. Consequently, during multi-turn revisions, they often fall into repetitive and ineffective attempts, failing to achieve stable improvements in answer quality.To address this issue, we propose a novel iterative self-correction framework that endows models with two key capabilities: Capability Reflection and Memory Reflection. This framework guides the model to first diagnose errors and generate a correction plan via Capability Reflection, then leverage Memory Reflection to review past attempts to avoid repetition and explore new solutions, and finally, optimize the answer through rigorous re-reasoning. Experiments on the challenging OCRBench v2 benchmark show that OCR-Agent outperforms the current open-source SOTA model InternVL3-8B by +2.0 on English and +1.2 on Chinese subsets, while achieving state-of-the-art results in Visual Understanding (79.9) and Reasoning (66.5) - surpassing even larger fine-tuned models. Our method demonstrates that structured, self-aware reflection can significantly enhance VLMs' reasoning robustness without additional training. Code: https://github.com/AIGeeksGroup/OCR-Agent.
1. Introduction
OCR-Agent addresses unstable and ineffective VLM self-correction with constrained, memory-aware reflection. On OCRBench v2, it improves over InternVL3-8B and standard refinement approaches without additional training.
- VLM prompting can suffer from capability hallucination and refinement stagnation, motivating reasoning-focused self-correction within executable capabilities.Models may propose infeasible actions such as image enhancement or human proofreading, then repeat ineffective correction loops.
- Carefully constrained self-reflection enables sustained and stable performance improvement in vision-language tasks.The approach targets limitations of standard chain-of-thought prompting through structured iterative correction.
- OCR-Agent combines Capability Reflection for diagnosing errors and planning feasible corrections with Memory Reflection for avoiding redundant attempts and exploring new paths.The framework is designed to stabilize iterative self-correction without additional training.
- OCR-Agent improves over naive chain-of-thought prompting and Self-Refine, especially on fine-grained visual-textual alignment and multi-step reasoning.The reported comparison is on OCRBench v2.
- +2.0 points on English and +1.2 points on Chinese OCRBench v2 subsets over InternVL3-8B.These experiments demonstrate gains against the cited open-source state-of-the-art baseline.
2. Related Work
Related work progresses from traditional OCR pipelines to deep-learning and large-VLM approaches, while self-reflection methods extend chain-of-thought reasoning through iterative refinement and memory.
- Optical Character Recognition (OCR): Traditional OCR methods use multi-stage pipelines with pattern matching and contextual rules for character recognition.Earlier Tesseract versions exemplify this approach.
- Optical Character Recognition (OCR): Deep-learning OCR combines computer vision and natural language processing, with CRNN using CNN feature extraction and RNN sequence transcription.These approaches improve adaptability to text detection, instance segmentation, and spatial structure modeling.
- Optical Character Recognition (OCR): Large VLMs provide strong zero-shot OCR capabilities, while fine-tuning can further improve performance on OCR tasks.OlmOCR is cited as an example based on a fine-tuned Qwen-2.5-7B-Instruct model.
- Self Reflection: Chain-of-thought prompting enables multi-step reasoning by eliciting intermediate reasoning steps before the final answer.It can be prompted zero-shot or supported through fine-tuning.
- Self Reflection: Self-Refine introduces iterative improvement, while Reflexion structures models as agents that use task feedback and maintain memory.These methods extend self-reflection beyond a single reasoning pass.
3. Method
OCR-Agent uses capability-aware and memory-based reflection to make iterative VLM self-correction more feasible, non-redundant, and effective. The framework filters infeasible actions, records prior reflections, and uses these signals to guide improved answers.
- Framework Overview: The framework extends self-reflection for language models to multimodal errors by tracing output problems to visual features within the image.Its reflection-refinement loop is designed specifically for VLMs.
- Capability Reflection: Capability Reflection diagnoses errors and adaptively plans corrective actions while excluding actions beyond the model’s capabilities.The feasibility indicator marks executable actions and filters the proposed plan before refinement.
- Capability Reflection: The filtered plan is passed with the input and previous answer to the refinement mechanism, which generates an improved answer using only feasible actions.The update is expressed as ynew = R(x, yprev, Pfeas).
- Memory Reflection: Memory Reflection stores historical reflections so each new refinement can use prior experiences instead of repeatedly exploring flawed strategies.The mechanism targets refinement stagnation and ineffective looping by retaining the full reflection history.
- Memory Reflection: At each iteration, the model generates a reflection from the image, question, previous answer, and memory store, then adds it to the record before refinement.The updated memory store conditions generation of the next improved answer.
4. Experiments
Experiments evaluate OCR-Agent on OCRBench v2 across multilingual OCR and multimodal understanding tasks, showing strong performance and sustained gains through iterative refinement.
- Dataset and Metrics: OCRBench v2 contains over 10,000 manually verified Chinese and English question-answer pairs spanning eight core tasks.The benchmark includes Recognition, Referring, Spotting, Extraction, Parsing, Calculation, Understanding, and Reasoning.
- Main Results: 51.01 average English score surpasses all open-source models and approaches Gemini-Pro’s 51.9.OCR-Agent also reaches 79.9 on Visual Understanding and 66.5 on Visual Reasoning.
- Main Results: 54.72 average Chinese score ranks second to Qwen2.5-VL-7B’s 55.6, while setting open-source records on Recognition, Information Extraction, and Visual Understanding.The corresponding task scores are 77.0, 68.8, and 65.1.
- Ablation Study: After baseline methods plateau or fluctuate, OCR-Agent improves steadily across three rounds, especially on English Reasoning and Chinese Understanding.The ablation study reports consistent gains across English and Chinese tasks, with particularly pronounced improvements in Understanding and Reasoning.
- Ablation Study: Combining Capability Reflection and Memory Reflection yields peak average scores of 51.0 in English and 54.7 in Chinese.The Chinese Recognition score rises from 37.7 to 77.0 in the reported comparison.
5. Limitation and Future Work
The framework’s main limitations are inference overhead and dependence on the base VLM, while future work targets efficiency, external tools, memory, and broader applications.
- Limitations: Iterative reflection requires multiple VLM calls, increasing inference time and computational cost and potentially hindering real-time deployment.The fixed three-round scheme may be inefficient for simple problems and insufficient for extremely complex ones.
- Limitations: OCR-Agent’s effectiveness remains bounded by the inherent capabilities of its base VLM.If the base model misperceives a critical visual element or lacks knowledge, refinement may continue within a wrong context.
- Future Work: Future directions include dynamic iteration control, model distillation, external image tools, broader vision-language tasks, structured memory, and human-in-the-loop refinement.These directions are proposed to improve efficiency, overcome model limitations, and extend application scope.
6. Conclusion
OCR-Agent addresses unstable and ineffective self-correction in VLMs with Capability Reflection and Memory Reflection. On OCRBench v2, it outperforms direct answering, standard CoT, and simple self-refine strategies, supporting more robust multimodal reasoning.
- Unconstrained self-reflection can produce unstable reasoning and ineffective corrections that impair VLM performance.
- OCR-Agent combines Capability Reflection and Memory Reflection to structure and sustain iterative self-correction.Capability Reflection diagnoses errors and plans corrective actions, while Memory Reflection uses historical reasoning traces to avoid redundant attempts.
- The framework improves VLM robustness in text-rich visual understanding tasks without additional fine-tuning.
- OCR-Agent significantly outperforms direct answering, standard CoT prompting, and simple self-refine strategies on OCRBench v2.
- The findings support carefully constrained self-reflection as a route toward more robust and sustainable reasoning in VLMs.The paper identifies computational efficiency and broader vision-language tasks as future work directions.