PaperlayerView exact revision on arXiv

Source-linked AI summary

HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

Gengluo Li, Xingyu Wan, Shangpin Peng, Weinong Wang, Hao Feng, Yongkun Du, Binghong Wu, Zheng Ruan, Zhiqiong Lu, Liang Wu, Pengyuan Lyu, Huawen Shen, Zibin Lin, Shijing Hu, Jieneng Yang, Hongbing Wen, Guanghua Yu, Hong Liu, Bochao Wang, Can Ma, Han Hu, Chengquan Zhang, Yu Zhou

arXiv:2607.04884v2cs.CV

TL;DR

Existing OCR-specialized VLMs often focus on single-page document parsing despite broader text-centric visual tasks. HunyuanOCR-1.5 combines faster long-output decoding with agent-driven data construction and upgraded training, achieving broader OCR coverage and top-tier document parsing.

  • Problem

    Existing OCR-specialized VLMs largely focus on single-page document parsing, limiting coverage of diverse text-centric visual tasks such as spotting, extraction, QA, translation, and multi-image understanding.

  • Method

    HunyuanOCR-1.5 combines DFlash decoding with Agentic Data Flow and upgraded pretraining and post-training recipes to improve efficiency and diverse OCR capabilities.

  • Results

    HunyuanOCR-1.5 achieves faster inference and broader OCR capabilities, reaching an Overall score of 94.74 and SOTA performance among end-to-end OCR expert models on OmniDocBench v1.6.

  • Takeaways & Limitations

    The lightweight end-to-end model supports broader OCR scenarios and both server-grade and local deployment without redesigning its backbone.

  • Takeaways & Limitations

    OmniDocBench’s matching protocol splits multi-line formulas into single-line units, potentially underestimating the model’s formula-parsing capability.

Abstract

from arXiv · show

We present HunyuanOCR-1.5, a lightweight end-to-end OCR-specialized vision-language model. HunyuanOCR unifies document parsing, text spotting, information extraction, text-image translation, and multi-image document understanding within a single end-to-end VLM. Building upon the lightweight architecture of HunyuanOCR-1.0, HunyuanOCR-1.5 does not redesign the backbone, but systematically improves both efficiency and capability. For efficiency, we adapt DFlash to OCR decoding, significantly reducing the latency of long structured outputs such as dense documents, tables, and formulas while preserving output distribution. Powered by DFlash, HunyuanOCR-1.5 achieves a 6.37x Transformer inference speedup and a 2.14x speedup under vLLM, delivering the fastest inference among lightweight OCR VLMs. For capability, we propose Agentic Data Flow, an agent-driven data construction system that transforms model weaknesses into executable data requirements and autonomously performs material search, quality verification, and pipeline development. It substantially improves long-tail capabilities in ancient-script OCR, fine-grained chart and table parsing, multi-image text-centric QA, low-resource multilingual parsing, and document hallucination evaluation. HunyuanOCR-1.5 ranks among the top-tier end-to-end OCR solutions on OmniDocBench v1.6 while achieving new performance milestones across these long-tail tasks. Combined with an upgraded pretraining and post-training recipe, HunyuanOCR-1.5 further extends its capability in high-resolution, long-context, and multi-task scenarios. Experiments demonstrate faster inference, broader OCR capability coverage, and the deployment advantages of a lightweight end-to-end model. We will release the model weights and training code to support future research and real-world OCR applications.

1 Introduction

HunyuanOCR-1.5 extends lightweight end-to-end OCR beyond single-page document parsing to diverse text-centric visual tasks. It combines DFlash-based decoding acceleration, Agentic Data Flow, and upgraded training to broaden capability coverage while supporting efficient deployment.

  • Capability scope: HunyuanOCR-1.5 unifies document parsing, text spotting, information extraction, visual question answering, text-image translation, and other OCR capabilities in one lightweight end-to-end VLM.The model builds on HunyuanOCR-1.0's compact architecture and extends OCR beyond document parsing.
  • Faster: DFlash-based inference acceleration: 6.37x Transformer inference speedup and 2.14x vLLM speedup are achieved by adapting DFlash speculative decoding to long structured OCR outputs.The approach uses a lightweight block-diffusion draft model to reduce decoding overhead for dense documents, tables, and formulas.
  • Better: Agentic data flow and refined training recipes: Agentic Data Flow converts model weaknesses into executable data requirements and autonomously supports material search, quality verification, and pipeline development.It produces data for long-tail capabilities including low-resource OCR, ancient-script OCR, and multi-image question answering.
  • Evaluation: HunyuanOCR-1.5 is evaluated across document parsing, text spotting, multilingual and ancient-script OCR, translation, multi-image QA, information extraction, and hallucination-related reliability.The evaluation is capability-oriented rather than confined to a single benchmark, with the model established as an SOTA end-to-end OCR solution on OmniDocBench v1.6.
  • Better: Agentic data flow and refined training recipes: 4K image resolution and a 128K context window are introduced through an upgraded pretraining recipe incorporating capability, multi-image, and historical OCR data.Post-training further uses high-quality SFT data and task-specific reinforcement-learning exploration to extend the capability ceiling.
  • Deployment and release: The project plans to release model weights and training code and supports both server-grade and PC-side local OCR deployment through llama.cpp.This is intended to support community reproduction, fine-tuning, capability extension, and practical deployment.

2 Related Work

Related work spans general VLMs with broad OCR-related perception and reasoning, OCR-specific VLMs focused primarily on structured document parsing, and decoding methods targeting latency in long-output OCR scenarios.

  • General vision-language models: General VLMs recognize text across natural images, documents, charts, and screenshots, and support text-centric question answering or visual reasoning.Representative models include GPT-4o, Gemini, Qwen-VL, and InternVL.
  • OCR-specific vision-language models: Most OCR-specific VLMs target document parsing, converting page-level document images into structured Markdown, HTML, or LaTeX outputs.Recent work also explores lightweight OCR expert designs with promising results.
  • Multi-token prediction: Autoregressive decoding is a key latency bottleneck for long-output OCR tasks including document parsing, table reconstruction, and formula transcription.Speculative decoding uses lightweight draft models whose candidate tokens are verified by the target model while preserving the original output distribution.

3 Model Design

HunyuanOCR-1.5 retains HunyuanOCR-1.0’s compact end-to-end architecture while extending visual input resolution from 2K to 4K for finer structural detail. It accelerates long structured OCR decoding with DFlash, which uses lightweight parallel drafting and target-model verification.

  • Architecture: The model retains a native-resolution visual encoder, adaptive MLP connector, and lightweight language model in a compact end-to-end architecture.The connector compresses high-resolution features into layout-sensitive tokens, while the Hunyuan-0.5B language model with XD-RoPE autoregressively generates structured OCR outputs.
  • Architecture: 4K maximum input resolution, up from 2K, preserves native aspect ratios and spatial layouts while capturing finer structural detail.The visual encoder is built on Hunyuan-ViT.
  • DFlash decoding: DFlash targets autoregressive decoding bottlenecks in OCR tasks requiring long structured outputs, including dense tables, multi-column documents, and long formulas.It produces draft-token blocks through a lightweight parallel forward pass, allowing each target-model verification step to advance multiple tokens.
  • DFlash training: During training, the target model remains frozen while K sampled anchor blocks are concatenated and trained in one FlexAttention block-diagonal forward pass.Each block attends to the target hidden states and its own tokens, while different blocks remain isolated.
  • DFlash training: 90.7M parameters, block size B = 16, K = 16 anchors, γ = 7.0, and a 5-layer Transformer define the DFlash draft model configuration.The draft model is initialized from the target model’s last 5 decoder layers, and its position-weighted loss excludes anchor and invalid positions while downweighting farther positions.

4 Agentic Data Flow

Agentic Data Flow is a reusable, agent-driven data construction system that targets model weaknesses rather than simply scaling data volume. It closes the loop from capability-gap identification through material preparation, pipeline development, and training-data injection, instantiated for low-resource OCR, ancient-script OCR, and multi-image QA.

  • System overview: Agentic Data Flow converts concrete capability gaps into executable data requirements and systematically constructs data for low-resource OCR, ancient-script OCR, and multi-image QA.The system is explicitly oriented toward weaknesses including low-resource languages, ancient scripts, multi-image understanding, and difficult complex scenarios.
  • System overview: The agent uses tool-calling structures to access web search, OCR and vision-language services, file-processing scripts, image-cleaning tools, and data-generation tools.Algorithm engineers specify target capability requirements in natural language, such as synthetic low-resource OCR, ancient-script samples, failure-case mining, or multi-image QA.
  • Data construction workflow: The agent autonomously searches for required materials and verifies their quality, including multilingual corpora, TTF fonts, historical-script fonts, backgrounds, and multi-page PDFs.Tool-assisted inspection filters backgrounds containing interfering text, overly complex foreground objects, or unstable visual quality.
  • Data construction workflow: After material preparation, the agent develops and iterates weakness-oriented data pipelines by writing rendering or QA scripts, organizing resources, defining task formats, and supporting varied layouts, backgrounds, augmentations, and output schemas.The workflow includes progressive pipeline development and human-agent iteration before injecting the resulting data into subsequent training stages.
  • System overview: The closed loop connects model-weakness identification, material construction, data cleaning, pipeline development, and training-data injection.The resulting paradigm supports reusable capability expansion rather than being limited to a specific data type.

5 Training Recipe

HunyuanOCR-1.5 uses a three-phase training recipe—pretraining, supervised fine-tuning, and reinforcement learning—to expand capability boundaries, improve task ceilings, stabilize outputs, and mitigate document hallucinations. The recipe combines capability-focused data and larger input specifications with structured SFT preparation and fine-grained, extensible RL rewards.

  • Training overview: The training pipeline comprises pretraining, supervised fine-tuning, and reinforcement learning, shifting from general OCR construction toward capability-boundary expansion and higher task ceilings.SFT and RL jointly target stronger task performance, output stability, and reduced document hallucinations.
  • Pretraining: Stage3 pretraining injects capability-expansion, multi-image, and historical OCR data while extending inputs to 4K resolution and a 128K context window.The new data target low-resource, ancient-script, multi-image, hard-case, and long-tail OCR scenarios, while historical data preserve existing OCR and parsing strengths.
  • Supervised fine-tuning: SFT cleans and enriches training data, reserves mainly high-difficulty samples for RL, and unifies task-specific prompts to establish broad competence and clear interfaces.The process removes annotation, formatting, image-text, objective, and duplication problems, while reducing instruction ambiguity before RL.
  • Reinforcement learning: RL uses three complementary rewards: structure-aware factuality rewards for document parsing, consistency-based judging rewards for general QA, and degeneration-suppression rewards for stable generation.Together, these rewards provide fine-grained, discriminative, and extensible signals for more faithful, stronger, and comprehensive behavior.
  • Reinforcement learning: Translation judging uses source text and target-language metadata, assigning a [0, 5] soft score that is normalized to [0, 1] with a debiased mapping sensitive to mid-range quality differences.The auxiliary annotation fields enable more precise consistency judgments against reference translations.

6 Evaluation Tree

HunyuanOCR-1.5 is evaluated with a capability-oriented tree that tests both strengthened foundational OCR abilities and newly added long-tail, multimodal, structured-parsing, and reliability capabilities. The evaluation maps each capability group to dedicated benchmarks before reporting boundary and inherited-dimension results.

  • Evaluation purposes: The evaluation tree tests preserved and strengthened core OCR capabilities alongside newly extended capabilities and output reliability.Its three purposes are foundational OCR, boundary expansion, and output reliability.
  • Basic OCR and document understanding: Basic OCR and document understanding use OmniDocBench for structured document parsing and OCRBench for OCR-aware question answering.OmniDocBench covers printed and scanned documents, including body text, tables, formulas, and reading order.
  • Long-tail capability expansion: Long-tail capability expansion evaluates low-resource multilingual parsing with MORE across 149 languages and ancient-script recognition with Chronicles-OCR.These benchmarks target low-resource languages, rare writing systems, and seven historical forms of Chinese characters.
  • Structured visual element parsing: Structured visual element parsing evaluates table reconstruction with TableVerse-5K and chart text, structure, and semantics with ChartArena.These evaluations extend beyond plain text recognition to structured visual content.
  • Cross-page and cross-lingual understanding: Cross-page and cross-lingual understanding uses DUDE for multi-image document understanding and DoTA plus MMTIT for text-image translation.DUDE measures retrieval, comparison, and evidence aggregation; DoTA translates English documents into Chinese, while MMTIT covers 14 non-Chinese languages.
  • Application-oriented and reliability evaluation: Application-oriented evaluation measures structured information extraction and video subtitle extraction, while reliability evaluation assesses output faithfulness to visual evidence.The information-extraction benchmark targets cards, receipts, and forms; the faithfulness metric tests conflicts between visual evidence and language priors.

7 Experimental Results

HunyuanOCR-1.5 achieves strong end-to-end OCR performance while substantially accelerating decoding with DFlash. Evaluations also show expanded long-tail capabilities across documents, charts, tables, languages, multi-image QA, and output faithfulness, alongside some remaining limitations.

  • Inference Efficiency: 2.14× speedup in vLLM reduces average latency from 3.032s to 1.408s and increases throughput from 466.9 token/s to 1002.3 token/s.Under Transformers, the speedup is larger because autoregressive decoding is closer to naive token-by-token generation.
  • Inference Efficiency: 1.408s per page and 0.706 page/s make HunyuanOCR-1.5 with DFlash the fastest evaluated end-to-end system, about 1.17× faster than GLM-OCR and 1.24× faster than PaddleOCR-VL-1.6.The model retains a unified end-to-end OCR VLM formulation without explicit layout decomposition or region-wise cascaded inference.
  • Inference Efficiency: 6.67× speedup for 2048+ token outputs in Transformers, compared with 4.56× for shorter-output decoding.In vLLM, speedup rises from 1.31× on 0–256 token outputs to 2.30× on 2048+ token outputs as speculative iterations amortize more forward passes.
  • Inference Efficiency: Table pages obtain the largest DFlash speedup, followed by formula and text pages, because regular HTML-like table structures yield longer accepted prefixes.Across vLLM concurrency levels 1 to 32, DFlash maintains more than 1.8× speedup and reaches a highest speedup of 2.26× at concurrency 4.
  • Core OCR and Document Parsing: 94.74 Overall on OmniDocBench v1.6 establishes SOTA among end-to-end OCR expert models, with strong text, table, and reading-order parsing.The result improves full-page document parsing while preserving the lightweight end-to-end architecture.

8 Conclusion and Future Work

HunyuanOCR-1.5 advances HunyuanOCR-1.0 toward faster inference and broader OCR capabilities without redesigning its validated backbone. It integrates DFlash speculative decoding for long structured OCR generation, supports PC-side deployment via llama.cpp, and combines Agentic Data Flow with upgraded training methods.

  • Conclusion and Future Work: HunyuanOCR-1.5 targets faster inference and broader OCR capabilities while retaining the validated HunyuanOCR-1.0 backbone.The model advances HunyuanOCR-1.0 toward these two goals without redesigning the backbone.
  • Conclusion and Future Work: DFlash speculative decoding accelerates long structured OCR generation under both Transformers and vLLM.The passage describes DFlash as integrated specifically for long structured OCR generation.
  • Conclusion and Future Work: HunyuanOCR-1.5 supports PC-side deployment through llama.cpp.
  • Conclusion and Future Work: Agentic Data Flow is combined with upgraded pretraining and post-training methods to broaden the model’s OCR capabilities.

Supplementary Material · Overview

The supplementary material provides additional details to the main paper, covering model architecture, recommended instructions, reinforcement learning, task-wise performance improvements, and qualitative examples.

  • Overview: The supplementary material provides supplementary details to the main paper.
  • Overview: It includes detailed model architecture information.
  • Overview: It provides recommended instructions for using the model.
  • Overview: It presents reinforcement learning details.
  • Overview: It reports task-wise performance improvements within the reinforcement learning details.
  • Overview: It includes qualitative examples.

A Detailed Model Architecture … D.2 RL Reward Design

HunyuanOCR-1.5 uses a native-resolution, fully end-to-end architecture supporting diverse OCR outputs, with task-specific instructions and benchmarks covering seen-text faithfulness. Its RL system computes task-specific rewards online, combining structured accuracy measures, LLM judging, and degeneration suppression.

  • A Detailed Model Architecture: The native-resolution Hunyuan-ViT preserves input aspect ratios and spatial layouts for receipts, documents, tables, charts, and scene-text images.This design targets OCR inputs with diverse shapes while retaining native visual organization.
  • A Detailed Model Architecture: An adaptive MLP connector pools and projects high-resolution features into compact visual tokens while preserving text-dense and layout-sensitive regions.
  • A Detailed Model Architecture: The lightweight Hunyuan-0.5B language model autoregressively generates OCR outputs, using XD-RoPE across text, height, width, and time dimensions.This supports unified reasoning over text sequences, document layouts, and multi-image or temporal inputs.
  • A Detailed Model Architecture: HunyuanOCR-1.5 remains fully end-to-end, directly generating recognized text, spotting results, Markdown, HTML tables, LaTeX formulas, chart descriptions, and document-grounded answers.
  • B Recommended Instruction: Chinese instruction prompts are recommended for stable and reproducible benchmarking, with task-specific prompts covering JSON spotting, ancient-text recognition, chart deplotting, OmniDocBench, and DoTA.The newly supported spotting output uses JSON format.
  • C CHAOS-Bench Annotation Details: CHAOS-Bench edits characters in rendered document images and records original and perturbed words to test whether models follow visual text over language priors.Faithful models should reproduce the visually observed perturbed words in their parsed outputs.
  • D.1 RL Setup: RL rollout generation samples n = 16 responses per prompt with temperature 1.0, top-p = 1.0, and top-k = −1 for diverse advantage estimation and policy updates.The setup uses Adam, a constant learning-rate schedule, large global batches, and long-context settings; training uses vLLM with FSDP.
  • D.2 RL Reward Design: Rewards are computed immediately after each response, filtered for overlong or repetitive outputs, and routed by task to spotting, parsing, visual question answering, or translation metrics.Tail repetition detection flags a short consecutively repeated unit and assigns reward 0.

D.3 RL Dynamics

The RL experiments progressed from an ineffective baseline through higher-reward but unstable optimization to a final configuration that improved both training effectiveness and stability. The final setup used filtered training data and a batch size of 512, with smoother reward, KL-loss, and train–inference mismatch dynamics.

  • Exp A: Exp A produced nearly flat kl_loss and rewards/mean, while marginal weight changes indicated that the model was effectively stuck.This motivated testing a larger learning rate.
  • Exp B: 1 × 10−5 learning rate made rewards/mean consistently higher than Exp A but caused substantial kl_loss growth and rising probs_diff/mean, indicating degraded stability.Other settings were kept unchanged from Exp A.
  • Exp C: 4 × 10−6 learning rate, added KL loss, and larger sample count n kept probs_diff/mean stable while entropy decreased steadily and kl_loss rose smoothly to a reasonable level.Because the training data also changed, rewards/mean was not directly comparable with earlier experiments.
  • Exp D: 512 batch size and sampling-and-filtering data substantially improved rewards/mean while kl_loss and probs_diff/mean remained stable and varied more smoothly than in Exp C.This configuration was adopted as the final RL setup.

D.4 Task-wise Performance Improvements · E Qualitative Examples

HunyuanOCR-1.5 improves over its SFT checkpoint across core and boundary OCR tasks, while qualitative examples demonstrate its handling of historical documents, charts, and tables across diverse settings.

  • D.4 Task-wise Performance Improvements: RL training improves HunyuanOCR-1.5 over the SFT checkpoint across document parsing, text spotting, ancient-script recognition, chart parsing, and video subtitle extraction.These gains span both core OCR abilities and boundary capabilities.
  • E Qualitative Examples: The qualitative examples illustrate HunyuanOCR-1.5’s newly supported capabilities and characteristic scenarios.
  • E Qualitative Examples: On Chronicles-OCR, HunyuanOCR-1.5 handles historical Chinese character forms, historical documents, and ancient-script images.
  • E Qualitative Examples: The examples collectively cover ancient-script images, multilingual charts, and structurally varied tables.
  • E Qualitative Examples: On ChartArena, HunyuanOCR-1.5 parses charts across diverse chart families, visual scenarios, and languages.
  • E Qualitative Examples: On TableVerse-5K, HunyuanOCR-1.5 parses tables with diverse structures, styles, and content types.
Loading 2607.04884v2…