Source-linked AI summary
GLM-OCR Technical Report
Shuaiqi Duan, Yadong Xue, Weihan Wang, Zhe Su, Huan Liu, Sheng Yang, Guobing Gan, Guo Wang, Zihan Wang, Shengdong Yan, Dexin Jin, Yuxuan Zhang, Guohong Wen, Yanfeng Wang, Yutao Zhang, Xiaohan Zhang, Wenyi Hong, Yukuo Cen, Da Yin, Bin Chen, Wenmeng Yu, Xiaotao Gu, Jie Tang
TL;DR
GLM-OCR addresses the high computational cost, slow inference, and memory demands of large multimodal models for document understanding. It combines a compact vision-language architecture with layout-aware processing and shared-parameter multi-token decoding, achieving strong benchmark and real-world results while supporting broad deployment workflows. The report also notes that performance can degrade when training-data coverage and diversity are insufficient.
Problem
Large multimodal models improve document understanding but impose computational, latency, and memory costs that challenge high-concurrency and edge deployment.
Method
GLM-OCR combines a 0.4B CogViT encoder and 0.5B GLM decoder with layout analysis, parallel recognition, and shared-parameter Multi-Token Prediction.
Results
GLM-OCR achieves competitive results across document parsing, text and formula recognition, table parsing, key information extraction, and six practical scenarios.
Takeaways & Limitations
The compact architecture and structured generation support document-understanding workflows ranging from rapid prototyping to enterprise-scale deployment.
Takeaways & Limitations
Performance may degrade when scenarios fall outside the distribution and diversity of the training data.
Abstract
from arXiv · showhide
GLM-OCR is an efficient 0.9B-parameter compact multimodal model designed for real-world document understanding. It combines a 0.4B-parameter CogViT visual encoder with a 0.5B-parameter GLM language decoder, achieving a strong balance between computational efficiency and recognition performance. To address the inefficiency of standard autoregressive decoding in deterministic OCR tasks, GLM-OCR introduces a Multi-Token Prediction (MTP) mechanism that predicts multiple tokens per step, significantly improving decoding throughput while keeping memory overhead low through shared parameters. At the system level, a two-stage pipeline is adopted: PP-DocLayout-V3 first performs layout analysis, followed by parallel region-level recognition. Extensive evaluations on public benchmarks and industrial scenarios show that GLM-OCR achieves competitive or state-of-the-art performance in document parsing, text and formula transcription, table structure recovery, and key information extraction. Its compact architecture and structured generation make it suitable for both resource-constrained edge deployment and large-scale production systems.
1 INTRODUCTION
GLM-OCR targets document understanding under competing demands for recognition quality, throughput, latency, memory efficiency, and deployment flexibility. Its compact multimodal architecture, multi-token decoding, and layout-aware pipeline support strong results across benchmarks and practical scenarios.
- Traditional OCR systems struggle with complex layouts, diverse document formats, and production requirements despite handling plain-text transcription effectively.
- GLM-OCR combines a 0.4B-scale CogViT encoder with a 0.5B-scale GLM decoder in a 0.9B-parameter multimodal model.The compact design targets high-throughput and low-latency inference while maintaining strong recognition performance.
- MTP predicts multiple tokens per step, generating 5.2 tokens per decoding step on average and delivering approximately 50% throughput improvement.Shared parameters across draft models reduce the additional GPU memory overhead introduced by MTP.
- The system uses layout analysis followed by parallel content recognition to improve robustness and processing efficiency for complex real-world documents.PP-DocLayout-V3 detects structured regions before parallel recognition across document areas.
- GLM-OCR reaches 94.6 on OmniDocBench v1.5 and reports strong performance across text, formula, table, and information-extraction benchmarks.Reported scores include 94.0 on OCRBench (Text), 96.5 on UniMERNet, 85.2 on PubTabNet, and 86.0 on TEDS.
- Across six practical scenarios, GLM-OCR achieves 91.5 on real-world table recognition, 90.5 on seal recognition, and 94.5 on receipt KIE.The evaluated scenarios also include code parsing, handwritten text, and multilingual OCR.
2 METHODOLOGY
GLM-OCR formulates document parsing and key information extraction as conditional structured generation, combining a vision-language core with layout-aware preprocessing and multi-token decoding. Its staged training aligns visual and language representations and refines structured output reliability.
- Multi-Token Prediction: MTP addresses slow one-token-at-a-time decoding by predicting k tokens simultaneously, with structural benefits for locally dependent table tags and Markdown syntax.The method is intended to reduce broken tags and produce more robust structured outputs.
- Architecture: The framework centers on a vision-language core containing a 400M-parameter CogViT encoder and a 500M-parameter GLM decoder.Visual features are projected into the language embedding space and supplied to the decoder as prefix tokens.
- Multi-Token Prediction: MTP adds k shared-parameter auxiliary heads that predict future tokens simultaneously, reducing decoding latency and encouraging local structural coherence.The auxiliary heads model different future offsets while sharing parameters.
- Document Parsing: Document parsing applies PPDocLayoutV3 to divide pages into semantic regions, processes regions independently, and merges outputs into Markdown and JSON.The modular pipeline restores reading order and enables parallel region processing.
- Key Information Extraction: Key information extraction feeds the full document image and a task-specific prompt directly into the core to generate structured fields.Unlike document parsing, KIE does not use explicit layout cropping and instead relies on implicit visual attention under prompt guidance.
- Unified Formulation: Document parsing and KIE share a conditional structured-generation formulation while differing in preprocessing strategy and prompt specification.The unified formulation supports shared document-level representations and task-specific output formats.
- Training Recipe: Training progresses from vision-encoder training and vision-language pretraining through supervised fine-tuning and reinforcement learning.The stages cover multimodal alignment, structured-generation adaptation, OCR specialization, and task-aware reliability optimization.
3 EVALUATION
GLM-OCR is evaluated on public document-parsing and KIE benchmarks, then tested on custom real-world scenarios. It achieves leading or competitive results across diverse recognition, parsing, and extraction tasks, including challenging industrial conditions.
- Public Benchmarks: 94.6 on OmniDocBench v1.5 ranks GLM-OCR first among evaluated models despite its compact 0.9B size.
- Public Benchmarks: GLM-OCR achieves the highest reported scores on OmniDocBench v1.5, OCRBench Text, UniMERNet, and TEDS_TEST, while remaining competitive on PubTabNet.The reported scores are 94.6, 94.0, 96.5, 86.0, and 85.2, respectively.
- Public Benchmarks: GLM-OCR establishes a clear SOTA in KIE, scoring 93.7 on Nanonets-KIE and 86.1 on Handwritten-KIE among open-source competitors.
- Public Benchmarks: On OmniDocBench v1.5, GLM-OCR records the highest Overall score at 94.62, ahead of specialized and much larger general VLMs.Its table-recovery scores are 93.96 on TableT EDS and 96.39 on TableT EDS−S.
- In-House Benchmarks: Across six custom real-world categories, GLM-OCR achieves the highest score in five among compared open-weight models.It leads in seal recognition, multilingual text, code-document parsing, and real-world table extraction, while remaining effective in handwritten text and receipt KIE.
- In-House Benchmarks: GLM-OCR scores 90.5 on seal recognition, 69.3 on multilingual text, 84.7 on code parsing, 91.5 on real-world tables, and 94.5 on receipt KIE.
4 INFERENCE AND DEPLOYMENT
GLM-OCR supports localized and cloud deployment through mainstream inference frameworks, an SDK, throughput-oriented processing, and a MaaS API. It also provides direct fine-tuning for domain adaptation or task-specific performance.
- Local Deployment and SDK Integration: 0.9B parameters make GLM-OCR suitable for localized inference and resource-constrained environments.
- Local Deployment and SDK Integration: GLM-OCR supports vLLM, SGLang, and Ollama, with an SDK for end-to-end document parsing workflows.
- Throughput: 1.86 pages/second for PDFs and 0.67 images/second for standalone image files are reported under single-replica, single-concurrency testing.
- Cloud Deployment: The MaaS API charges 0.2 RMB per million input or output tokens, with 1 RMB processing approximately 2,000 scanned A4 images or 200 simple-layout PDFs.
- Fine-Tuning: Direct fine-tuning through LLaMA-Factory is supported for domain adaptation or enhanced task performance.
5.1 OVERVIEW
GLM-OCR offers two usage paradigms: an SDK for complex document-parsing workflows and direct base-model invocation for focused recognition and structured extraction.
- The GLM-OCR SDK targets complex document parsing, while the base model supports focused recognition and structured information extraction.
5.2 DOCUMENT PARSING WITH GLM-OCR SDK
The GLM-OCR SDK provides production-oriented, layout-aware parsing for heterogeneous documents and generates structured Markdown that preserves document organization and relationships.
- The SDK combines layout-aware parsing, multimodal recognition, and structured output generation for enterprise-grade document workflows.
- The SDK is intended for documents containing paragraphs, tables, mathematical expressions, and key-value pairs.
- Structured Markdown output preserves logical document hierarchy and structural relationships across heterogeneous content.
5.3 LIGHTWEIGHT OCR AND INFORMATION EXTRACTION WITH THE BASE MODEL
The base model supports lightweight OCR and targeted extraction through explicit prompts, covering text, tables, formulas, and structured information. Examples show preservation of visual structure and schema-conformant outputs across diverse document content.
- Task coverage: Explicit prompts control standalone OCR tasks, including text transcription, table reconstruction, formula conversion, and structured information extraction.The model can be used independently for lightweight OCR and targeted extraction with lower integration overhead.
- Text recognition: The text-recognition mode transcribes printed or handwritten content while preserving line structure, capitalization, punctuation, numerical values, and currency symbols.A multilingual restaurant-menu example includes handwritten-style fonts, perspective distortion, and background clutter.
- Table recognition: The table-recognition mode reconstructs hierarchical headers, merged-cell relationships, row ordering, column correspondence, and percentage–value pairings.The output is intended to preserve tabular structure in Markdown or other machine-readable formats.
- Formula recognition: The formula-recognition mode converts equations into syntactically valid LaTeX while preserving matrices, determinants, superscripts, subscripts, and other spatial structures.This supports direct reuse of reconstructed mathematical expressions in academic or technical documentation.
- Information extraction: Prompt-specified JSON schemas enable extraction of nested fields from dense forms while preventing hallucinated keys and supporting automated processing.A customs declaration example includes shipper details, social credit codes, and itemized goods information.
5.4 SUMMARY OF USAGE PARADIGMS
GLM-OCR offers two complementary usage paradigms: an SDK for comprehensive production parsing and base-model prompting for lightweight, flexible tasks. Together they span rapid prototyping through enterprise-scale document workflows.
- SDK workflow: The GLM-OCR SDK targets comprehensive, layout-aware, multi-element document parsing in production environments.It supports heterogeneous content such as paragraphs, tables, mathematical expressions, and key-value pairs.
- Base-model workflow: Base-model prompting supports lightweight OCR and targeted information extraction for modular or task-specific applications.This mode offers flexible prompting, lower integration overhead, and rapid prototyping.
- Combined scope: Together, the two paradigms support document-understanding workflows ranging from rapid prototyping to enterprise-scale deployment.The distinction separates comprehensive production parsing from modular task-specific use.
6 LIMITATIONS
GLM-OCR remains constrained by layout errors, training-data coverage, generative formatting variability, and prompt-dependent extraction behavior. These limitations define important boundaries for complex documents and strict structured-output workflows.
- Pipeline limitations: The two-stage pipeline can propagate inaccurate layout detection into downstream recognition and may reconstruct reading order imperfectly for cross-page or irregular multi-column layouts.The limitation follows from layout analysis preceding region-level recognition.
- Data coverage limitations: Performance may degrade on extremely low-resolution or heavily distorted documents.This is identified as a training-data and coverage-related limitation.
- Data coverage limitations: Performance may degrade for highly complex mathematical expressions and dense or irregular tabular structures.These cases are listed among scenarios where data distribution and diversity can limit robustness.
- Data coverage limitations: Performance may degrade for languages underrepresented in the training corpus.Language coverage is explicitly tied to the distribution and diversity of training data.
- Structured output variability: Generative formatting can vary in line breaks and whitespace, so strict formatting guarantees cannot be fully ensured.Reinforcement learning and structural supervision mitigate but do not eliminate this variation.
- Structured extraction: Prompt-based key information extraction depends on prompt specification and schema clarity, with ambiguous field boundaries causing incomplete or redundant outputs.The limitation is most relevant to complex forms with implicit or unclear field boundaries.
7 CONCLUSION
GLM-OCR targets structured document understanding under real-world system constraints through compact, layout-aware, and efficient generation design. The report presents modular deployment-oriented optimization as a route to competitive recognition with efficiency gains without increasing parameter scale.
- Core conclusion: GLM-OCR combines layout-aware preprocessing and multi-token decoding to improve throughput and stability while maintaining competitive recognition accuracy.The conclusion frames this combination as a practical solution for structured document understanding under real-world constraints.
- Efficiency design: The system emphasizes controllable latency, memory efficiency, and structured output reliability instead of relying on large model scaling.The design aligns architecture, decoding strategy, and task structure for efficiency gains without increasing parameter scale.
- Engineering implications: Modular pipelines, efficient generation mechanisms, and deployment-oriented optimization support local inference, cloud serving, and domain-specific fine-tuning.These capabilities enable integration into heterogeneous production environments.