PaperlayerView exact revision on arXiv

Source-linked AI summary

PaDoc: Layout-Grounded Parallel Decoding for Document Parsing

Hao Yu, Jiabo Zhan, Kang Liu, Linnan Zhao, Dongxu Yue, Rui Chen, Jinglin Wang, Chong Sun, Chen Li, Jing Lyu, Chun Yuan

arXiv:2608.06146v1cs.AI

TL;DR

End-to-end document parsers serialize independent regions, limiting decoding parallelism, while crop-based alternatives repeat visual prefills and fragment page context. PaDoc uses layout-grounded branching over a shared page representation, and on one A800 GPU improves throughput by 67.4–118% while reducing P95 latency by 39.2–54.9% versus a same-backbone Sequential SFT baseline.

  • Problem

    End-to-end parsers serialize independent regions, whereas crop-based parsers repeat visual prefills and restrict access to page-level context.

  • Method

    PaDoc derives layout-conditioned regional branches over a shared page representation, enabling concurrent content decoding without separately materialized crops.

  • Results

    67.4–118% higher throughput and 39.2–54.9% lower P95 latency than the same-backbone Sequential SFT baseline make PaDoc the fastest end-to-end parser across tested concurrency levels.

  • Takeaways & Limitations

    Layout-grounded parallelism preserves competitive parsing quality while substantially improving end-to-end serving efficiency.

  • Takeaways & Limitations

    PaDoc relies on a region-specific conditional-independence assumption that content depends on its visual region and shared page context.

Abstract

from arXiv · show

End-to-end document parsers provide a unified interface, but serialize page layouts and regional contents into one autoregressive sequence. This formulation forces independent regions onto a decoding path whose length grows with the total content, whereas crop-based two-stage parsers expose region-level parallelism at the cost of repeated visual prefills and fragmented page context. To retain full-page context while removing dependencies, we propose PaDoc, a layout-grounded parser that treats the predicted layout as a branching structure over a shared page representation. Under a region-sufficiency assumption, we derive a prefix-conditioned factorization in which the layout stream and regional content branches advance concurrently, reducing the decoding depth to the longest layout-content path. We realize this factorization within a single MLLM: packed variable-length ancestor attention preserves the visibility under standard next-token training, while masked parallel decoding creates branches that the evaluated vLLM backend serves as concurrent requests with cache-resident shared-prefix reuse. On OmniDocBench Full, PaDoc attains an Overall layout F1 of 91.1 and, among end-to-end parsers, a top-tier Overall score of 94.24 together with the best Text Edit (0.038) and Formula CDM (95.59). On a 384-page subset and one A800 GPU, it is the fastest end-to-end parser at five concurrency levels, improving valid-page throughput by 67.4-118% and reducing P95 latency by 39.2-54.9% relative to a same-backbone Sequential SFT baseline. Code is available at https://github.com/Longin-Yu/Padoc

Introduction

PaDoc reformulates document parsing as layout-grounded branching over a shared page representation, exposing regional parallelism while preserving full-page context. Its single-MLLM implementation combines ancestor attention, packed variable-length training, and cache-aware concurrent serving, achieving strong accuracy and efficiency.

  • Core formulation: PaDoc derives a prefix-conditioned factorization that makes region content conditionally independent while retaining a shared page-image and layout prefix.The resulting logical decoding depth is the longest root-to-leaf layout-content path rather than the sum of regional content lengths.
  • Implementation: Packed variable-length ancestor attention preserves the required branch visibility under standard next-token training without auxiliary prediction heads or losses.At inference, the evaluated vLLM backend serves branch streams as concurrent requests with cache-resident image and layout prefix reuse.
  • Efficiency: 67.4 ∼118% throughput improvement is reported relative to the same-backbone Sequential SFT baseline on an A800 GPU.PaDoc is evaluated across five concurrency levels and is described as the fastest end-to-end parser at each level.
  • Results: 91.1 Overall layout F1 is achieved on OmniDocBench.The evaluation covers layout analysis, end-to-end parsing, and inference efficiency.
  • Results: 94.24 Overall score is achieved among end-to-end parsers, alongside the best Text Edit and Formula CDM.The passage identifies these as top-tier end-to-end parsing results.

Related Work

Related work follows three main directions: multimodal generation for unified document outputs, layout-first localization with region-level recognition, and structured generation for parallel decoding. Layout-first systems improve local efficiency but can weaken page-level coherence when regions are recognized independently.

  • Document parsing as multimodal generation: Multimodal generation converts document images or PDF pages directly into Markdown, structured text, or task-specific markup.This unified output space accommodates heterogeneous elements including text, tables, formulas, figures, and layout structures.
  • Layout detection and instance-level recognition: Layout-first pipelines localize page elements before recognizing them at the region level, using efficient structural front-ends such as DocLayout-YOLO and PP-DocLayout.DocBed, Dolphin, and HSD likewise indicate efficiency benefits from separating localization from recognition.
  • Layout detection and instance-level recognition: Independent region recognition can weaken reading order, section hierarchy, table-caption relations, and cross-region references, creating tension between local parallelism and global coherence.The limitation arises because regions are recognized largely independently rather than with full page-level context.
  • Structured parallel generation: Structured generation represents outputs as sets, trees, graphs, or schemas, enabling parallel decoding over meaningful units instead of individual tokens.SoT expands skeleton points in parallel, while APAR forks hierarchical autoregressive branches using learned control tokens.

Method

PaDoc factorizes document parsing into a layout stream and region-content branches conditioned on shared page and layout prefixes. Ancestor attention and masked parallel decoding realize this structure while preserving region-level concurrency without per-region crop materialization.

  • Task formulation: Document parsing predicts canonically ordered layout-content pairs, decomposing their distribution into layout analysis and content recognition conditioned on layout.Each region Bk is paired with associated textual or structured content Yk.
  • Native structural properties: Under region-specific content sufficiency, each region’s content depends on the image and its current layout region, not other regions or previously decoded contents.Crop sufficiency separately states that a region crop preserves all visual evidence needed for recognizing its associated content.
  • Shared-prefix parameterization: PaDoc replaces per-region crop conditions with a shared document-image prefix plus the generated layout prefix B≤k for each content branch.This retains region-level parallelism while avoiding crop extraction, data movement, and repeated visual prefills that prevent shared page-prefix reuse.
  • Shared-prefix parameterization: After layout prefix B≤k is available, PaDoc can decode the next layout region and current-region content concurrently, reducing dependency depth to the longest layout-content path.The factorization pairs both predictors at step k with the same prefix and prevents either variable from conditioning on the other.
  • Implementation: A purpose-built ancestor-attention mask trains the factorization in one shared sequence, while masked parallel decoding forks content streams that retain only their ancestor context.Content branches cannot observe later layout predictions or sibling contents, and cannot affect the continuing layout trajectory.

Experiments

Experiments evaluate PaDoc’s document-parsing quality on OmniDocBench and serving efficiency under controlled concurrency on one A800 GPU. PaDoc achieves top-tier end-to-end quality while substantially improving throughput and P95 latency over a same-backbone sequential baseline.

  • Evaluation Setup: 1,651 pages from OmniDocBench v1.6 measure quality, while a uniformly sampled 384-page subset evaluates serving efficiency across shared concurrency settings.Each page includes layout, reading order, and region-level annotations for text, formulas, tables, and figures.
  • Parsing Quality: 94.24 Overall parsing score places PaDoc in the top tier, exceeding Qianfan-OCR by 0.34 points and the strongest evaluated general VLM by 0.62 points.PaDoc also attains the best end-to-end Text Edit of 0.038.
  • Serving Efficiency: 67.4–118% higher throughput and 39.2–54.9% lower P95 latency than the same-backbone Sequential SFT baseline distinguish PaDoc across every tested concurrency.PaDoc reaches 1.722 pages per second per GPU at 64 concurrent requests and remains near that level at higher concurrency.
  • Serving Efficiency: At every tested concurrency level, PaDoc has the highest valid-page throughput and lowest P95 latency among end-to-end parsers on one A800 GPU.Figure 2 reports five concurrency levels and shows PaDoc approaching compact two-stage systems despite using a larger model.

Conclusion

PaDoc factorizes regional content generation under shared page-image and layout prefixes, exposing region-level parallelism without repeated visual prefills. Its packed ancestor attention and concurrent serving enable efficient end-to-end parsing with shared-prefix reuse.

  • Method: PaDoc factorizes regional content generation under shared page-image and layout prefixes, exposing region-level parallelism without repeated visual prefills.The design treats regional generation as parallel branches grounded in shared page context and layout information.
  • Method: Packed variable-length ancestor attention trains the factorization with a standard next-token objective.This training mechanism preserves the intended factorization while using standard next-token supervision.
  • Efficiency: Concurrent serving reuses shared prefixes to accelerate PaDoc inference.The shared page-image and layout prefixes remain available across concurrent regional-generation branches.

A Derivations … B.2 Soft Category Modes

The appendix derives PaDoc’s prefix-conditioned sufficiency from region-specific sufficiency and specifies PageIoU’s taxonomy-robust coverage representation and soft category modes. These definitions preserve valid category variation while distinguishing missing, duplicate, and overlapping coverage.

  • A.1 From Region-Specific Sufficiency to Prefix-Conditioned Sufficiency: PaDoc’s prefix-conditioned sufficiency follows from region-specific sufficiency via the chain rule of conditional mutual information.The derivation applies the chain rule to separate prior layout information from later layout and content variables.
  • A.1 From Region-Specific Sufficiency to Prefix-Conditioned Sufficiency: The resulting condition is the weaker prefix-conditioned consequence used by PaDoc, not an equivalent reformulation of the original region-specific assumption.The converse fails because the prefix-conditioned form does not constrain dependence on B<k given (X, Bk).
  • B Details of PageIoU: PageIoU maps heterogeneous predictions and taxonomies into broad classes with core include and ambiguous acceptable categories.This avoids conflating taxonomy mismatch with localization error during cross-model comparison.
  • B Details of PageIoU: PageIoU measures complete-page coverage without one-to-one matching or an IoU threshold, separating binary coverage from countable coverage.Countable coverage retains overlapping-box multiplicity, while strict hard matching is avoided to preserve semantically valid outputs.
  • B.1 Coverage Representation: Coordinates are independently normalized to Ω = [0, 1]^2, and OmniDocBench quadrilaterals are converted to axis-aligned rectangles.Coverage is represented separately for core ground truth, acceptable ground truth, and predictions mapped to each broad class.
  • B.1 Coverage Representation: Binary coverage clips pointwise counts so only box unions matter, whereas countable coverage retains counts and penalizes duplicate or overlapping predictions.Include and acceptable category sets remain disjoint even when their rectangles overlap spatially.
  • B.2 Soft Category Modes: Acceptable mode credits predicted acceptable coverage without treating omitted acceptable coverage as a false negative.The effective ground-truth coverage is selected within the acceptable interval to be as consistent with prediction coverage as possible.
  • B.2 Soft Category Modes: Ignore acceptable mode removes locations covered exclusively by acceptable ground-truth regions from IoU computation while preserving penalties for overlapping coverage in countable mode.Predictions may exceed core coverage by at most the available acceptable coverage in the prediction denominator.

B.3 Per-Page Metrics and Aggregation · B.4 Category Mapping · B.5 Exact Computation

The evaluation aggregates per-page metrics over class-specific support pages, uses soft category mapping, and computes reported areas exactly with coordinate compression. It also defines handling for empty supports, zero-valued F1 cases, missing pages, and a raster cross-check.

  • B.3 Per-Page Metrics and Aggregation: Support pages contain an include box, acceptable box, or mapped prediction for class q; pages lacking all three are omitted.The reported metric is the page-level macro average over the support-page set S_q.
  • B.3 Per-Page Metrics and Aggregation: Page-level macro averaging prevents pages with large canvases or regions from dominating the score.Reported F1 is derived from macroaveraged precision and recall.
  • B.3 Per-Page Metrics and Aggregation: F1 is set to zero when both aggregated precision and recall are zero.The ratio construction treats agreement on an empty support as perfect when 0 ≤ x ≤ y.
  • B.3 Per-Page Metrics and Aggregation: The primary Table 1 result uses countable coverage with acceptable category handling, preserving category ambiguity tolerance while penalizing overlapping predictions.The other binary/countable and acceptable/ignore-acceptable combinations are reported in Section B.6.
  • B.4 Category Mapping: Table 4 defines a soft mapping from OmniDocBench ground-truth categories to five evaluation classes.This mapping supplies the category assignments used by the evaluation metrics.
  • B.5 Exact Computation: The default backend computes axis-aligned rectangle area sums exactly using coordinate compression and cellwise coverage counts from a two-dimensional difference array.Include, acceptable, and prediction boundaries partition Ω into cells where a, b, and c are constant, with each contributing exact area ∆x∆y.
  • B.5 Exact Computation: A raster backend cross-checks the equations on a configurable grid, while all reported results use coordinate-compressed exact areas.The OmniDocBench raster grid is 1000 × 1000; pages align by image basename, and missing prediction pages are evaluated as empty layouts.

B.6 Additional PageIoU Results · C Ablations of the Training Paradigm · C.1 Attention Backends

Tables 5–7 extend PageIoU evaluation under three additional protocols, while the training-paradigm ablation compares three attention backends that preserve the same token visibility and next-token objective. The implementations differ in sparse-attention representation and execution, with Tree-varlen decomposing the packed tree into variable-length causal problems.

  • B.6 Additional PageIoU Results: Tables 5–7 report three additional layout protocols using per-class IoU, F1, precision, recall, and Overall macro-aggregation.Overall uses the same macro-aggregation as Table 1.
  • C Ablations of the Training Paradigm: The attention-backend ablation compares implementations that preserve identical token visibility and next-token optimization without changing the probabilistic factorization.The comparison covers memory, single-node efficiency, and multi-node scaling.
  • C.1 Attention Backends: The packed attention relation contains one causal problem for shared stream M and one causal problem for each branch Yk.For branch Yk, M≤fk denotes the prefix visible at its fork.
  • C.1 Attention Backends: Dense SDPA materializes a B × S × S Boolean mask, providing a simple reference but blocking standard causal FlashAttention-2 and incurring quadratic mask growth.The arbitrary two-dimensional mask prevents text layers from using the standard causal FlashAttention-2 path.
  • C.1 Attention Backends: Flex Attention encodes the same ancestor relation with a compiled block mask, skipping fully masked query–key blocks without constructing a dense mask.Its tradeoff includes graph-compilation and block-scheduling overhead, especially during the first optimizer step.
  • C.1 Attention Backends: The first Flex Attention optimizer step is excluded from steady-state timing comparisons because it includes compilation.This exclusion isolates steady-state efficiency from cold-start compilation cost.
  • C.1 Attention Backends: Tree-varlen decomposes the packed tree into variable-length causal problems and executes them with FlashAttention-2 varlen while computing projections once in packed order.It gathers visible shared-prefix and private-history K/V rows, sums branch gradients into shared-prefix rows, and scatters outputs back.
  • C.1 Attention Backends: FlashAttention’s bottom-right causal alignment lets each branch see its full prefix and only non-future private tokens, with temporaries scaling by concatenated varlen K/V rows.This avoids materializing an S × S mask.

C.2 Experimental Protocol

The protocol benchmarks the Qwen3-VL-2B backbone under controlled hardware, precision, sharding, and decoder-backend settings. Training scales across cluster sizes while timing excludes cold-start effects and averages subsequent optimizer steps with end-to-end work included.

  • Experimental setup: The experiments benchmark Qwen3-VL-2B with a frozen vision encoder, bf16, FSDP full sharding, and gradient checkpointing.Each node contains eight A800 GPUs.
  • Experimental setup: Dense and Flex configurations use their named backends, while tree-varlen applies to the text decoder and FlashAttention-2 remains on the frozen vision encoder.The backends follow each configuration’s actual end-to-end training path.
  • Scaling protocol: The global batch size is fixed at 256 while cluster size varies over {1, 2, 4, 8, 16} nodes.The protocol evaluates scaling across multiple cluster sizes.
  • Measurement protocol: Each run executes four optimizer steps, discarding step 1 from timing because it includes cold data/kernel startup and Flex graph compilation.Reported timing uses steps 2–4.
  • Measurement protocol: Each measured step includes data loading, accumulation microsteps, forward and backward passes, FSDP communication, gradient clipping, and the optimizer update.CUDA synchronization determines maximum wall time and peak memory over all ...

C.3 Results and Backend Selection · D Controlled Quality and Structural Efficiency Analyses

Tree-varlen is selected as the training backend because it delivers the most consistent steady-state efficiency improvement with low compilation overhead and exact ancestor-mask realization. Controlled analyses use a systematically sampled 384-page OmniDocBench subset on A800 hardware to isolate parallel training, structural efficiency, and complexity effects.

  • C.3 Results and Backend Selection: 1.163–1.185× faster than dense SDPA, tree-varlen provides the most consistent 16K training speedup across all cluster sizes.At 8K, its gain is comparable to or slightly better than the baselines.
  • C.3 Results and Backend Selection: 0.754× and 0.519× SDPA training speed are achieved by Flex Attention on 16 nodes at 16K and 8K, respectively.Its excluded compilation cold step takes 258.951 seconds at 16K and 423.558 seconds at 8K.
  • C.3 Results and Backend Selection: 91.2% efficiency is reached by tree-varlen across the cleaner 8-to-16-node transition, compared with 90.5% for SDPA and 80.7% for Flex.The one- and two-node 8K timings have high variance because some prompts exceed the 8K cap.
  • C.3 Results and Backend Selection: 75.5–81.8% efficiency is sustained by tree-varlen at each 16K node doubling.The 8K one- and two-node transitions should not be interpreted as pure communication efficiency because of prompt-length outliers.
  • C.3 Results and Backend Selection: Tree-varlen is consistently fastest in multi-node training, while Flex Attention remains slower despite avoiding the dense mask.Figure 3 reports mean optimizer-step time over steps 2–4 with fixed global batch size 256; error bars show sample standard deviation.
  • C.3 Results and Backend Selection: Tree-varlen is used for all main experiments because of its steady-state speedup, low compilation overhead, and exact realization of the ancestor mask.The backend choice follows the reported trade-off between speed, compilation cost, and masking fidelity.
  • D Controlled Quality and Structural Efficiency Analyses: The controlled analyses isolate the parallel training objective under an identical backbone and data configuration.They also quantify autoregressive forward-step reduction, paired wall-clock speedups, and speedup variation with page complexity.
  • D Controlled Quality and Structural Efficiency Analyses: The efficiency analyses use a systematically sampled 384-page OmniDocBench subset on A800 hardware unless otherwise noted.This controlled setup complements the cross-model results in the main text.

D.1 Same-Backbone Quality Comparison … E.2 Request-Level vLLM Realization

Controlled experiments show that PaDoc improves layout grounding while preserving end-to-end parsing quality, and substantially reduces decoding steps and measured latency. Its masked-parallel design is realized through isolated request branches in vLLM with shared-prefix reuse, while measured efficiency includes serving overheads rather than guaranteeing the ideal critical-path ratio.

  • D.1 Same-Backbone Quality Comparison: 1.460 points: PaDoc’s Overall layout F1 improves under the same Qwen3-VL-2B backbone, training corpus, and PageIoU protocol, with formulas gaining 2.231 points.Precision rises for every class, including 5.574 points on images and 3.729 points Overall, while recall decreases for Overall, Image, Table, and Full page.
  • D.1 Same-Backbone Quality Comparison: −0.345 points: PaDoc’s Overall end-to-end quality change is small, while Text Edit and both table metrics improve and Reading Order Edit and Formula CDM remain close to Sequential SFT.The controlled comparison indicates that the efficiency gain does not require sacrificing layout grounding or regional content recognition.
  • D.2 Forward-Step Reduction and Measured Speedup: 58.4%: At C64, PaDoc reduces the median per-page forward-step count from 725.0 sequentialized steps to 297.5 critical-path steps.The interquartile range of per-page reduction is 47.7–68.2%; this is a decoding-iteration measure, not a wall-clock speedup estimate.
  • D.2 Forward-Step Reduction and Measured Speedup: 1.453× to 1.937×: Median paired end-to-end speedup spans the five concurrency levels, corresponding to median latency reductions of 31.2–48.4%.PaDoc and Sequential SFT are evaluated on the same pages, hardware, vLLM version, and document concurrencies, with valid requests paired by page.
  • E.1 Masked Parallel Decoding: Masked parallel decoding processes all live streams in one logical batched step while ancestor visibility and private histories prevent streams from observing one another’s suffixes.When a layout region completes, PaDoc copies the layout visibility and creates a content stream with a learned branch-initialization input; completed streams are retired while private cache positions remain invisible.
  • E.2 Request-Level vLLM Realization: Request-level vLLM realization preserves branch isolation by giving each child the original multimodal prompt, pre-boundary layout prefix, and private continuation.The layout and child requests run in one shared vLLM runtime, where automatic prefix caching can reuse matching resident paged key–value cache blocks; continuous batching overlaps live branches.
  • E.2 Request-Level vLLM Realization: The vLLM implementation is visibility-equivalent to the logical algorithm but does not materialize the ancestor mask or execute every stream in one fused forward pass.Its latency includes request admission, residual prefix processing, kernel and host overhead, and idle work from unequal branch lengths, so the critical-path ratio is an ideal structural quantity rather than a wall-clock guarantee.

E.3 Additional Serving Metrics

On the fixed 384-page workload, PaDoc completed every request normally and improved sustained output-token throughput over Sequential SFT across all tested concurrency settings. Its sustained rate peaked at C64, while peak short-window throughput continued increasing through C256.

  • Validity: All 384 PaDoc requests completed normally across five concurrency settings, without service errors, timeouts, truncations, or early-stop events.Sequential SFT produced 359–364 valid pages per setting; 20–25 outputs reached the generation limit.
  • Output-token throughput: 26.9–44.4% higher sustained output-token throughput was achieved by PaDoc than Sequential SFT across all tested concurrency settings.PaDoc also achieved higher peak output throughput throughout the controlled same-backbone comparison.
  • Output-token throughput: 1,410.0 tokens/s/GPU at C16 increased to 1,551.2 at C64, then remained within 2.2% of that value at C128–C256.C64 was the highest sustained throughput operating point in the tested range.
  • Output-token throughput: 2,655.7 tokens/s/GPU at C16 increased to 3,576.5 at C256 for peak 10-second output throughput.Higher concurrency increased short-window token throughput even after sustained throughput peaked at C64.
Loading 2608.06146v1…