PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSeek-OCR 2: Visual Causal Flow

Haoran Wei, Yaofeng Sun, Yukun Li

arXiv:2601.20552v1cs.CV

TL;DR

VLMs commonly impose rigid raster ordering on visual tokens, unlike semantically guided human visual processing, raising a question about more causally coherent 2D understanding. The paper introduces DeepEncoder V2, an LM-style encoder with causal flow tokens and specialized attention, and reports improved document reading while preserving compression and decoding efficiency. It also identifies remaining weaknesses for text-rich newspapers and frames broader multimodal extension as future work.

  • Problem

    Existing VLM encoders rigidly flatten 2D images into raster-ordered 1D token sequences, motivating investigation of semantically and causally ordered visual encoding.

  • Method

    DeepEncoder V2 replaces CLIP with a compact LLM, uses causal flow tokens and customized attention masks, and feeds their outputs to the LLM decoder.

  • Results

    DeepSeek-OCR 2 improves OmniDocBench v1.5 performance by 3.73% over DeepSeek-OCR while preserving high visual-token compression and decoding efficiency.

  • Takeaways & Limitations

    The results provide preliminary validation for language-model architectures as VLM encoders with causal visual reasoning and potential extension across image, audio, and text modalities.

  • Takeaways & Limitations

    The current evaluation focuses on optical text reading and document parsing, while newspaper recognition remains weak because of limited visual-token capacity and only 250k relevant training samples.

Abstract

from arXiv · show

We present DeepSeek-OCR 2 to investigate the feasibility of a novel encoder-DeepEncoder V2-capable of dynamically reordering visual tokens upon image semantics. Conventional vision-language models (VLMs) invariably process visual tokens in a rigid raster-scan order (top-left to bottom-right) with fixed positional encoding when fed into LLMs. However, this contradicts human visual perception, which follows flexible yet semantically coherent scanning patterns driven by inherent logical structures. Particularly for images with complex layouts, human vision exhibits causally-informed sequential processing. Inspired by this cognitive mechanism, DeepEncoder V2 is designed to endow the encoder with causal reasoning capabilities, enabling it to intelligently reorder visual tokens prior to LLM-based content interpretation. This work explores a novel paradigm: whether 2D image understanding can be effectively achieved through two-cascaded 1D causal reasoning structures, thereby offering a new architectural approach with the potential to achieve genuine 2D reasoning. Codes and model weights are publicly accessible at http://github.com/deepseek-ai/DeepSeek-OCR-2.

1. Introduction

DeepSeek-OCR 2 rethinks VLM encoders around semantically ordered, causally driven visual processing rather than rigid raster scans. DeepEncoder V2 implements this approach with causal flow tokens and reports improved document-reading performance while preserving compression and decoding efficiency.

  • Motivation: Human vision follows semantically guided, causally dependent fixation sequences, motivating visual-token ordering based on image semantics rather than spatial coordinates.The paper contrasts this with rigid top-left-to-bottom-right processing in existing encoders.
  • Motivation: Flattening 2D patches into raster-ordered 1D sequences imposes a text-oriented positional bias that ignores semantic relationships in complex documents.Documents with layouts, formulas, and tables provide the primary testbed because their structures carry causal visual logic.
  • DeepEncoder V2: DeepEncoder V2 replaces DeepEncoder’s CLIP component with a compact LLM and introduces causal flow tokens that reorder visual tokens through customized attention masks.Visual tokens retain global receptive fields, while only the latter half of encoder outputs is passed to the LLM decoder.
  • DeepSeek-OCR 2: DeepSeek-OCR 2 preserves the image compression ratio and decoding efficiency of DeepSeek-OCR while improving performance with visual-token budgets constrained between 256 and 1120.The design targets both architectural research and high-quality training-data generation for LLM pretraining.
  • Broader direction: Language-model architectures are presented as VLM encoders that can support feature extraction and token compression across images, audio, and text through modality-specific learnable queries.The paper also connects this direction to LLM infrastructure such as MoE and efficient attention.
  • Results: 3.73% performance gains on OmniDocBench v1.5 are reported relative to the DeepSeek-OCR baseline, alongside advances in visual reading logic.The paper presents these results as validation of specialized attention mechanisms for causal visual flow.

2. Related Works

Prior work uses learnable queries and transformer components for parallel object detection, visual token compression, and multimodal initialization. DeepSeek-OCR 2 is situated within this encoder-projector-LLM lineage while using an LM-style encoder for causal visual modeling.

  • Parallelized queries: DETR introduced preset parallelized object queries that interact with feature maps through cross-attention and exchange information bidirectionally through self-attention.Its 100 learnable queries encode object priors such as shape and position through training.
  • Visual token compression: BLIP-2’s Q-former uses 32 learnable queries in a BERT-like architecture to compress visual tokens within the encoder-projector-LLM paradigm.It draws inspiration from DETR’s object queries while bridging visual representations to LLMs.
  • Multimodal initialization: Prior multimodal work shows that pretrained LLM transformer layers and lightweight or encoder-free models can provide useful initialization for multimodal systems.The cited examples span visual discriminative tasks, vision, and speech.
  • Positioning: DeepSeek-OCR 2’s DeepEncoder V2 replaces DeepEncoder’s CLIP module with a compact language-model architecture and uses customized masks to begin causal visual-sequence modeling.The figure description presents this as retaining knowledge compression while adding causal modeling.

3. Methodology

DeepSeek-OCR 2 replaces the conventional encoder with DeepEncoder V2, which uses causal flow queries and specialized attention to semantically reorder visual tokens before LLM decoding. A multi-crop tokenizer controls the visual-token budget while preserving efficient image compression.

  • 3.2. Language model as vision encoder: DeepEncoder V2 replaces the predecessor’s encoder component with an LLM-style architecture that introduces causal reasoning for visual-token processing.The overall pipeline retains an encoder-decoder structure, but upgrades DeepEncoder to DeepEncoder V2.
  • 3.2.1. Vision tokenizer: The vision tokenizer combines an 80M-parameter SAM-base with convolutional layers and achieves 16× token compression through window attention.Its output dimension is reduced to 896 for the subsequent pipeline, and the tokenizer can alternatively be replaced with patch embedding.
  • 3.2.1. Vision tokenizer: The multi-crop strategy produces 256–1120 visual tokens, using 256 global-view tokens and up to six local views with 144 queries each.The total is k × 144 + 256, with local crops using fixed query configurations.
  • 3.2.2. Language model as vision encoder: Visual tokens retain bidirectional global attention, while causal flow queries attend to all visual tokens and preceding queries to learn semantic ordering.The two streams are concatenated with visual tokens as a prefix and causal queries as a suffix.
  • 3.2.2. Language model as vision encoder: DeepEncoder V2 establishes two-stage cascade causal reasoning: the encoder reorders visual tokens and the LLM decoder autoregressively reasons over the resulting sequence.This design adapts token order to visual semantics rather than imposing a fixed spatial order.
  • 3.2.4. Attention mask: The attention mask concatenates a bidirectional region for visual tokens with a lower-triangular causal region for flow tokens.The lower-triangular component lets each causal flow token attend only to previous tokens, including itself.

4. Experimental Settings

DeepSeek-OCR 2 uses the prior data sources with targeted sampling and label refinements, then trains the system through staged encoder, query, and decoder optimization. The experimental setup includes continued LLM training and reports benchmark metadata for document reading.

  • 4.1. Training data: DeepSeek-OCR 2 retains OCR 1.0, OCR 2.0, and general vision data, with OCR data comprising 80% of the training mixture.OCR 1.0 sampling is balanced across text, formulas, and tables, and layout labels are refined by merging similar categories.
  • 4.2. Training strategy: Training proceeds in three stages: encoder pretraining, query enhancement, and decoder specialization.The stages progressively develop feature extraction, compression, and reordering before freezing the encoder and optimizing the decoder.
  • 4.2.1. Encoder pretraining: DeepEncoder V2 is pretrained with a language-modeling objective using 768×768 and 1024×1024 dataloaders, initialized from DeepEncoder and Qwen2-0.5B.After pretraining, only encoder parameters are retained for later stages.
  • 4.2.2. Query enhancement: The final pipeline integrates DeepEncoder V2 with DeepSeek-3B-A500M and uses four-stage pipeline parallelism across the vision tokenizer, encoder, and decoder layers.The described training setup uses 160 GPUs.
  • 4.2.3. Continue-training LLM: Freezing DeepEncoder V2 parameters during decoder specialization more than doubles training speed under the same global batch size.This stage updates only DeepSeek-LLM parameters and continues training for 20k iterations with a learning-rate decay from 1e-6 to 5e-8.

5. Evaluation

DeepSeek-OCR 2 improves document understanding over DeepSeek-OCR while retaining a compact visual-token budget. Gains include lower reading-order and document-parsing edit distances, though performance remains uneven across document types.

  • 5.1. Main Results: 91.09% overall performance is achieved with the smallest upper limit of visual tokens, V-token_max.The model improves 3.73% over DeepSeek-OCR under similar training data sources.
  • 5.1. Main Results: Reading-order Edit Distance decreases from 0.085 to 0.057 versus DeepSeek-OCR.The authors associate this reduction with selecting and arranging visual tokens from image information.
  • 5.2. Improvement Headroom: Across nine document types, DeepSeek-OCR 2 outperforms DeepSeek-OCR on reading-order Edit Distance in every case.Text-recognition improvements occur in most cases, but newspapers remain a notable weakness with Edit Distance above 0.13.
  • 5.2. Improvement Headroom: Newspaper recognition remains limited by the lower visual-token upper bound and only 250k relevant training samples.The authors suggest increasing local crops and adding newspaper data as future remedies.
  • 5.3. Practical Readiness: Production repetition rates fall from 6.25% to 4.17% for online user-log images and from 3.69% to 2.88% for PDF data production.Because production ground truth is unavailable, repetition rate is used as the primary observable quality metric.

6. Discussion and Future Works

The paper frames DeepSeek-OCR 2 as two cascaded 1D causal reasoners: an encoder reorders visual information and a decoder performs visual-task reasoning. It also outlines a possible shared encoder for multiple modalities, while noting that broader reasoning remains future work.

  • 6.1. Towards Genuine 2D Reasoning: The LLM-style encoder and decoder form two cascaded 1D causal reasoning stages aimed at genuine 2D reasoning.The encoder handles causal visual reordering, while the decoder reasons over the reordered representations.
  • 6.1. Towards Genuine 2D Reasoning: Multiple reexaminations and multi-hop visual reordering may require causal flow tokens substantially longer than the original visual-token sequence.The authors identify this as an unresolved requirement for advancing toward genuine 2D reasoning.
  • 6.2. Towards Native Multimodality: DeepEncoder V2 initially validates an LLM-style encoder for visual tasks.The paper presents this as an initial validation rather than a completed solution to general visual reasoning.
  • 6.2. Towards Native Multimodality: A shared encoder could process text, speech, and visual content using modality-specific learnable query embeddings.The proposed shared parameter space includes common projections, attention mechanisms, and feed-forward networks.

7. Conclusion

DeepSeek-OCR 2 introduces DeepEncoder V2, combining bidirectional and causal attention to support causal visual encoding and improved visual reading logic. The paper limits its current evidence primarily to document parsing, leaving broader visual understanding for future work.

  • 7. Conclusion: DeepEncoder V2 combines bidirectional and causal attention mechanisms to give the vision encoder causal reasoning capabilities.The reported consequence is improved visual reading logic in DeepSeek-OCR 2.
  • 7. Conclusion: DeepSeek-OCR 2 maintains high visual-token compression while achieving meaningful performance improvements over DeepSeek-OCR.The conclusion attributes this advancement to DeepEncoder V2.
  • 7. Conclusion: Document parsing represents only a small part of the broader visual-understanding landscape covered by the current work.The authors plan to adapt and refine the architecture for more diverse scenarios.
Loading 2601.20552v1…