Source-linked AI summary
Efficient Document Parsing via Parallel Token Prediction
Lei Li, Ze Zhao, Meng Li, Zhongwang Lun, Yi Yuan, Xingjing Lu, Zheng Wei, Jiang Bian, Zang Li
TL;DR
VLM-based document parsing is slowed by autoregressive decoding, motivating faster generation for large-scale use. The paper proposes PTP, which adds learnable registers and training objectives for parallel future-token prediction, supported by a scalable data-generation pipeline. Experiments report 1.6×-2.2× decoding speedups while preserving parsing accuracy, with reduced hallucinations and broader generalization.
Problem
Autoregressive next-token decoding in VLMs creates a significant speed bottleneck for document parsing, despite improved parsing quality.
Method
PTP inserts learnable register tokens and trains them to predict future tokens in parallel, supported by a large-scale document-parsing data-generation pipeline.
Results
1.6×-2.2× decoding speedups are achieved while fully preserving parsing accuracy.
Takeaways & Limitations
PTP provides a model-agnostic, plugable acceleration framework that improves document-parsing throughput and reduces hallucinations.
Takeaways & Limitations
Single-forward-pass latency may differ slightly from the assumed Lθ because of hardware effects, although the difference is negligible with sufficient computational resources.
Abstract
from arXiv · showhide
Document parsing, as a fundamental yet crucial vision task, is being revolutionized by vision-language models (VLMs). However, the autoregressive (AR) decoding inherent to VLMs creates a significant bottleneck, severely limiting parsing speed. In this paper, we propose Parallel-Token Prediction (PTP), a plugable, model-agnostic and simple-yet-effective method that enables VLMs to generate multiple future tokens in parallel with improved sample efficiency. Specifically, we insert some learnable tokens into the input sequence and design corresponding training objectives to equip the model with parallel decoding capabilities for document parsing. Furthermore, to support effective training, we develop a comprehensive data generation pipeline that efficiently produces large-scale, high-quality document parsing training data for VLMs. Extensive experiments on OmniDocBench and olmOCR-bench demonstrate that our method not only significantly improves decoding speed (1.6x-2.2x) but also reduces model hallucinations and exhibits strong generalization abilities.
1. Introduction
Document parsing transforms complex documents into structured outputs, but VLMs’ autoregressive decoding limits processing speed. PTP addresses this bottleneck while preserving accuracy and improving throughput.
- Document parsing converts unstructured or semi-structured documents into structured, machine-readable outputs while preserving content relationships.It must reconstruct text, images, formulas, and tables in their logical and hierarchical order.
- VLMs improve parsing quality through end-to-end generation or integration into pipeline stages, but next-token autoregressive decoding creates a major efficiency bottleneck.This is especially consequential for large-scale deployment, where processing speed matters alongside accuracy.
- 1.6×-2.2× throughput improvements are reported for PTP without compromising accuracy.On OmniDocBench, PTP-1 reaches 1.6× throughput over NTP, while PTP-2 achieves 2.2× acceleration.
- PTP provides a model-agnostic, plugable acceleration method that improves throughput without compromising accuracy.The paper presents PTP as a contribution alongside a high-quality document-parsing data generation framework.
2. Related Work
Document parsing has progressed from modular pipelines to VLM-based approaches, while efficiency research targets multi-token prediction and sequence reduction. PTP is positioned as parallel decoding that addresses the remaining autoregressive bottleneck.
- Document Parsing Approaches: Pipeline-based parsing decomposes recognition and ordering into sequential specialized modules, but errors accumulate across stages.These methods offer fine-grained optimization and interpretability but can degrade on challenging or domain-specific documents.
- Document Parsing Approaches: VLM-based methods increasingly support end-to-end document parsing or selected pipeline stages for multi-element recognition.
- Efficient Document Parsing: Earlier multi-token prediction methods use CTC-trained non-autoregressive models, but remain limited to span-level OCR and require architectural changes.Auxiliary multi-token-prediction heads have also been explored in language models, but their use in document parsing remains unexplored.
- Efficient Document Parsing: Sequence compression methods reduce input, output, or vocabulary tokens, yet do not fundamentally address the autoregressive decoding bottleneck.PTP instead enables parallel decoding in VLMs, while remaining model-agnostic and orthogonal to existing approaches.
3. Dataset Engine
Existing parsing datasets lack sufficient layout-level coverage, document diversity, and difficulty variation. The Dataset Engine addresses these gaps through scalable resource collection, layout partitioning, collaborative annotation, and filtering.
- Current datasets mainly provide span-level or file-level annotations and lack diverse document types, difficulty levels, and layout-level data.These limitations are identified as barriers to generalization in real-world scenarios.
- The resource pool contains 200k pages from open-source datasets, in-house data, and synthetic generated data.A document classification and difficulty assessment model identifies document categories and difficulty levels to maintain diversity and prevent imbalance.
- Layout analysis partitions pages into sub-regions such as text paragraphs, tables, and figures to construct layout-level data.Sub-images that are too small, too large, or incomplete because of boundary truncation are filtered out.
- Three model types provide annotations that are aggregated by majority voting, then refined with LLM post-processing and selective manual review.The models include a frontier VLM, an open-source VLM, and a specialized model.
- Multi-stage filtering removes corrupted or abnormal-aspect-ratio samples and applies embedding-based similarity deduplication to reduce redundancy.
4. Method
The method extends autoregressive vision-language models with learnable register tokens that predict future answer tokens in parallel. It combines specialized training objectives and causal inference-time cache handling to accelerate document parsing while preserving the original architecture.
- Preliminaries: NTP predicts one next token, whereas MTP generalizes this objective to predict multiple future tokens simultaneously.The paper uses NTP as the autoregressive baseline and MTP as the multi-token prediction formulation.
- Parallel-Token Prediction: PTP inserts continuous register tokens after answer tokens so each register predicts a future token at a position-dependent offset.For n = 2, registers after x_i predict x_i+2 and x_i+3, while sharing one learnable embedding and differing in positional encodings.
- Parallel-Token Prediction: Register tokens share a token ID and embedding but make region-specific future-token predictions through contextual conditioning and positional offsets.Their predictions depend on the surrounding context even though the register parameters are shared.
- Parallel-Token Prediction: PTP training isolates regular-token computation from register tokens with a causal attention mask and combines NTP and register losses using weight α.Regular tokens attend only to preceding regular tokens, while register-token visibility is restricted within their causal groups.
- Inference and Analysis: At inference, appending n register tokens enables n + 1 new predictions per decoding step, unlike approaches that discard registers.The method estimates acceleration by comparing single-token decode-step latency with a forward pass processing multiple tokens simultaneously.
- Inference and Analysis: PTP preserves the causal language-model architecture and requires removing register-token KV-cache entries after each decoding step.The cache is replaced using tokens predicted by registers, yielding more accurate cache states; the added computation is negligible when resources are sufficient.
5. Experiments
Experiments evaluate PTP on document parsing benchmarks, comparing recognition quality, throughput, training and inference efficiency, hallucination rates, and generalization across models, token counts, and tasks.
- Main Results: PTP-1 preserves or improves text recognition while achieving formula recognition performance comparable to NTP.Evaluations cover text and formula recognition on OmniDocBench and document-type results on olmOCR-bench.
- Throughput: 1.6× speedup is achieved by PTP-1 over NTP, while PTP-2 attains 2.2× speedup and reduces TPOT and average latency.The throughput evaluation uses an H20 GPU and integrates PTP into KsanaLLM.
- Efficiency Analysis: PTP shows more efficient and stable training than MTP, whose MTP-2 configuration converges significantly more slowly.The comparison uses identical datasets and training settings; the paper attributes the difference to MTP’s additional head and block parameters.
- Hallucination Mitigation: PTP exhibits substantially lower hallucination rates than NTP, particularly on hallucination-prone data, while improving use of global visual information.The paper links this behavior to predictions relying more on direct visual evidence than corrupted contextual cues.
- Generalizability Study: PTP generalizes across tested model architectures and remains effective when register-token counts differ between training and inference, despite modest degradation.The evaluated extrapolation settings are PTP-2 → PTP-3 and PTP-3 → PTP-4.
- Generalizability Study: PTP-1 achieves comparable performance to NTP on ScienceQA with substantial latency reductions, while self-speculative decoding reaches an 82% acceptance rate.The self-speculative configuration achieves performance identical to NTP with minimal latency overhead.
6. Conclusion
The paper introduces PTP as a plugable, architecture-agnostic framework for parallel token prediction in document parsing, alongside a layout-level data generation framework. It reports 1.6×–2.2× decoding speedups while preserving parsing accuracy.
- PTP enables VLMs to accelerate document parsing through parallel token prediction with an architecture-agnostic, plugable framework.The framework implements parallel prediction through register injection.
- The accompanying data generation framework constructs high-quality layout-level document parsing data for training.
- 1.6×–2.2× decoding speedup is achieved while fully preserving parsing accuracy.
7. Details of Dataset
The dataset details describe raw document and layout-level OCR data collection followed by annotation, cleaning, and augmentation. The supplied figure and table references identify supporting visual materials but provide no additional findings.
- Raw document and layout-level OCR data are collected from three channels.
- Open-source datasets include layout analysis, handwriting, and mathematical-formula data, yielding nearly 200K image-text pairs after standardization and normalization.
- Layout-based segmentation is followed by comprehensive data annotation, cleaning, and augmentation.
- The annotation pipeline combines multiple VLM annotators, confidence-based refinement, rule-based standardization, and LLM-assisted and manual verification.
8. Combining PTP with Speculative Decoding
PTP can be combined with speculative decoding through self-verification of register-token predictions. The supplied figure caption identifies sampled rendered images and ground-truth texts from normal and hallucinated datasets.
- PTP integrates with speculative decoding through self-verification of register-token predictions.The mechanism uses the model’s inherent predictions to validate register tokens from the previous decoding step.
- Self-verification is designed to ensure output consistency with standard NTP.
- Figure 6 presents sampled rendered images and ground-truth texts from normal and hallucinated datasets.
9. Details of Experiments
The experiments evaluate model accuracy and inference efficiency, including edit distance, CDM, latency, TPOT, and TPS. They also examine register-token counts and hallucination robustness using controlled perturbations.
- The evaluation metrics are divided into model-accuracy metrics and inference-efficiency metrics.
- Edit distance measures the minimum single-character operations needed to transform a prediction into the ground truth.
- Character Detection Matching evaluates formula recognition by visually matching rendered predicted and ground-truth formulas with spatial information.
- Latency, TPOT, and TPS measure request responsiveness, average decoding time per output token, and output tokens processed per second, respectively.The experiments focus on TPS as a fine-grained throughput measure.
- Figure 7 analyzes the number of register tokens, while Figure 8 shows rendered markdown outputs for document types in olmOCR-bench.
- Increasing register-token counts expands training sequence length and can cause computational overhead and error propagation in distant predictions.
- PTP consistently outperforms NTP on normal and hallucination data and shows robustness to hallucination-inducing perturbations.The test set randomly deletes or substitutes characters while keeping background, font, and rendering style consistent.
10. Qualitative Examples
This section presents several qualitative examples from Section 8.
- Several qualitative examples are presented.
- The examples are located in Section 8.
- The section focuses on qualitative rather than quantitative examples.