PaperlayerView exact revision on arXiv

Source-linked AI summary

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

Qian Chen, Xianyin Zhang, Lifan Guo, Chi Zhang

arXiv:2610.18238v1cs.CVcs.AI

TL;DR

Long-document heading reconstruction must preserve useful visual evidence despite VLM input-resolution and context-length limits. The paper incrementally predicts heading levels with level-wise multimodal memory and evaluates the method on four benchmarks. Results are particularly strong on complex, long documents.

  • Problem

    VLMs cannot jointly process all page images in long documents within input-resolution and context-length limits while preserving fine-grained visual evidence.

    Source: §1 · pp. 1–2

  • Method

    The framework incrementally predicts heading levels using compact memory anchored by the most recent node at each observed hierarchy level.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Experiments on four benchmarks show particularly strong results on complex, long documents; on CLDoc-HHRBench, the score of 91.07 exceeded the next-best score of 83.61 by 7.46 points.

    Source: §5.3 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    CLDoc-HHRBench provides evaluation coverage across 250 documents and 11,212 pages, including long documents and deep heading hierarchies.

    Source: §1 · p. 2

    The task assumes headings are already detected, placed in reading order, and transcribed by upstream processing.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page-level elements. However, downstream applications such as retrieval-augmented generation (RAG) also require coherent document-level structure. Although these models indicate heading levels using different numbers of preceding ``\#'' symbols, page-by-page prediction often lacks document-level consistency. In practice, input-resolution and context-length constraints make it infeasible for a VLM to jointly process all page images in documents containing tens or hundreds of pages. To address this limitation, we propose a VLM-based framework that incrementally reconstructs heading hierarchies with a \emph{level-wise multimodal memory}. The memory retains a compact subset of all previously processed pages, selected by anchors. Our framework supports flexible anchor selection, and in this work we select the most recent node at each level, allowing the VLM to observe the visual, textual, and layout patterns of headings at every level. For comprehensive evaluation, we also introduce the Complex and Long Document Heading Hierarchy Reconstruction Benchmark (CLDoc-HHRBench), comprising 250 documents and 11,212 pages with expert-verified annotations. Experiments on CLDoc-HHRBench and three existing benchmarks demonstrate the effectiveness of our method.

1 Introduction

Heading hierarchies organize documents for retrieval and understanding, but reconstructing them consistently across long, visually varied documents is difficult for VLMs constrained by input resolution and context length. The paper proposes level-wise multimodal memory and introduces CLDoc-HHRBench to evaluate complex and long documents.

  • Heading hierarchies provide a navigational backbone for document organization and support downstream retrieval-augmented generation.Prior heuristic, machine-learning, and VLM approaches have limitations involving specialized components, generalization, visual detail, or consistency on ultra-long documents.Source: §1 · p. 1
  • Long-document reconstruction must balance fine-grained visual evidence against VLM input-resolution and context-length limits.Combining pages into one image can reduce image count but may compress page-level visual cues.Source: §1 · pp. 1–2
  • The framework incrementally reconstructs heading hierarchies using compact multimodal memory, retaining the most recent heading node at each observed level as an anchor.Selected pages provide visual styles, heading text, layout patterns, and predicted structural relationships; the framework supports prediction from a single heading to multiple consecutive pages.Source: §1 · p. 2
  • CLDoc-HHRBench covers 250 documents and 11,212 pages, addressing the limited length and hierarchy complexity of existing benchmarks.It spans diverse layouts and document types, including a document of 387 pages and documents with hundreds or over a thousand headings.Source: §1 · p. 2
  • Extensive experiments validate the proposed method and provide a comprehensive evaluation of heading hierarchy reconstruction on CLDoc-HHRBench.Source: §1 · p. 2

2 Related Work

Related work ranges from general and OCR-focused VLMs to rule-based, machine-learning, and VLM-based hierarchy reconstruction. Existing benchmarks broaden document types, but length and hierarchy depth remain limited.

2.1.1 General VLMs

  • General VLMs, including closed- and open-source families, show strong OCR capabilities across text, formulas, and tables.Source: §2.1.1 · pp. 2–3

2.1.2 OCR VLMs

  • OCR-oriented VLMs parse pages or regions through unified or two-stage frameworks, but do not explicitly address document-level heading hierarchy reconstruction.Source: §2.1.2 · p. 3

2.2.1 Rule-based Methods

  • Rule-based methods typically identify table-of-contents pages, extract entries, and link each entry to its document section.Source: §2.2.1 · p. 3

2.2.2 ML-based/DL-based Methods

  • Machine- and deep-learning methods combine textual, visual, and layout features; D-O-C predicts parent–child and sibling scores and inserts headings into the current rightmost subtree.Source: §2.2.2 · p. 3

2.2.3 VLM-based Methods

  • VLM-based methods use cross-modal interpretation, with MinerU-Popo applying dynamic chunking and synchronization and LingDT-VL-OCR aggregating heading crops into Pseudo-ToC images.Source: §2.2.3 · p. 3

2.3 Heading Hierarchy Reconstruction Benchmarks

  • Heading-hierarchy benchmarks expanded beyond scientific papers to varied document types, yet document length and hierarchy depth remain limited.The surveyed benchmarks include arXivdocs, HRDoc, Comp-HRDoc, E-Periodica, DocHieNet, and MPDocBench-Parse.Source: §2.3 · p. 3

3 Method

The framework incrementally predicts heading levels with a level-wise multimodal memory, then uses the ordered levels to construct a hierarchy tree. It selects historical pages through heading anchors and supports both single-heading and grouped prediction.

3.1 Problem Definition

  • Each heading’s predicted level forms an ordered sequence from which the tree is built by attaching it to the most recent preceding heading one level above, or to a virtual root at level 1.The task assumes headings have already been detected, placed in reading order, and transcribed.Source: §3.1 · pp. 3–4

3.2 Level-Wise Multimodal Memory

  • The memory starts on the first page containing a heading, assigns its first heading level 1, and then predicts subsequent headings, including possible later level-1 headings.This differs from relation-prediction methods that treat the first heading as the root and cannot assign a later heading to that same level.Source: §3.2 · p. 4
  • The memory selects historical pages through one anchor per observed level; this work uses the most recent predicted node at each level.Selected pages retain all processed headings on them as structural references, and the number of selected pages is bounded by the current tree depth.Source: §3.2 · p. 4

3.3 Level Prediction

  • For single-heading prediction, the VLM uses visual, textual, and structural information, and a new heading attaches to the latest preceding heading at the immediately shallower level, or to the virtual root at level 1.The level may stay the same, increase by one, or return to any shallower level.Source: §3.3 · pp. 4–5
  • CLDoc-HHRBench has the highest median document length and a larger share of long documents, as well as a higher median tree depth and a broader tail toward deeper hierarchies.Document length is measured in pages containing headings.Source: Figure 2 · p. 5, Figure 3 · p. 5
  • Grouped prediction reuses a fixed historical memory for headings on one or more consecutive target pages, constraining each later predicted level to rise by at most one.Later headings in the group may provide visual, semantic, and structural cues for earlier predictions.Source: §3.3 · p. 5

4 Benchmark

CLDoc-HHRBench covers long, structurally complex documents across multiple document types. Its headings were independently annotated and expert-adjudicated.

4.1 Benchmark Construction

  • CLDoc-HHRBench evaluates heading hierarchy reconstruction on complex, long documents and comprises 250 documents totaling 11,212 pages across 10 document types, including long documents and deep heading hierarchies.Source: §4.1 · p. 5

4.2 Annotation

  • Two annotators labeled each document independently, followed by expert adjudication; six annotators and three experts spent approximately 800 person-hours.Annotations cover heading bounding boxes, text, reading order, and hierarchy levels.Source: §4.2 · p. 5

5 Experiments

The experiments compare heading-hierarchy strategies on shared annotated inputs using STEDS, with incremental methods relying on their own previous predictions. The proposed method leads overall, and further tests examine document length, prediction granularity, memory selection, multimodal context, and historical-error sensitivity.

5.1 Experiment Setup

  • Baselines include rule-based PaddleX, heading-sequence MinerU, and two VLM strategies reproduced with a shared backbone to isolate reconstruction strategies.A uniform comparison using the original VLM-based implementations was not feasible.Source: §5.1 · p. 6

5.2 Evaluation Protocols

  • Incremental methods use their own earlier predictions as history; the evaluation isolates hierarchy reconstruction and excludes upstream parsing errors.Methods receive the same annotated headings and predict only hierarchy levels; Micro-STEDS is reported unless otherwise specified.Source: §5.2 · pp. 6–7

5.3 Experimental Results

  • 91.07 on CLDoc-HHRBench beats the next-best 83.61 by 7.46 points; overall Macro-STEDS/Micro-STEDS are 84.73/88.68, beating strongest competitors by 1.09/5.97.Fine-tuning the framework yields 85.32 Overall Micro-STEDS with Qwen3.5-0.8B.Source: §5.3 · p. 7
  • The method’s advantage is most pronounced for CLDoc-HHRBench documents with at least 25 heading-bearing pages.Figure 4 groups documents by heading-bearing-page count and reports Micro-STEDS.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4.1 Effect of Prediction Granularity

  • Micro-STEDS remains relatively stable across multi-page prediction units, though the effect and optimal granularity vary by benchmark.The evaluated units include one heading, one page, or two or four consecutive pages.Source: Figure 5 · p. 7, §5.4.1 · p. 8

5.4.2 Comparison of Memory Strategies

  • Most Recent achieves 88.68 Micro-STEDS, versus 87.47 for Previous-n and 78.22 for Random, supporting selection of useful historical context.Previous-n retains the three immediately preceding headings; Most Recent selects the latest heading at each observed level.Source: §5.4.2 · p. 8, Table 3 · p. 8

5.4.3 Effect of Multimodal Context

  • The multimodal method consistently outperforms a text-only variant that retains heading text, bounding-box coordinates, and predicted levels.The text-only variant excludes page images while keeping the same memory selection and prediction procedure.Source: §5.4.3 · p. 8

5.4.4 Sensitivity to Historical Hierarchy Information

  • Overall parent accuracy is 83.89% with predicted memory and 88.73% with fully ground-truth-guided memory, a moderate aggregate gap.The diagnostic changes anchor selection and historical level labels to ground truth.Source: §5.4.4 · p. 8, Table 5 · p. 8

6 Conclusion

The conclusion highlights level-wise multimodal memory for VLM-based heading hierarchy reconstruction and reports strong results across four benchmarks, particularly on complex, long documents.

6 Conclusion

  • Retaining the most recent heading at each observed level provides compact visual and structural context for heading-level and multi-page prediction.Source: §6 · p. 8
  • The paper presents CLDoc-HHRBench as a diverse benchmark of long documents and deep hierarchies.Source: §6 · p. 8
  • Experiments on four benchmarks show particularly strong results on complex, long documents.Source: §6 · p. 8

Ethical Considerations

CLDoc-HHRBench uses publicly available online documents, with permissions secured for inclusion and screening to exclude sensitive or personally identifiable information.

Ethical Considerations

  • The benchmark sources publicly available PDF and Microsoft Word documents found online, with permissions secured for inclusion.Source: Ethical Considerations · p. 8
  • Annotators manually annotate the source documents, with expert adjudication.Source: Ethical Considerations · p. 8
  • The team screens documents and annotations to exclude sensitive or personally identifiable information, characterizing the benchmark as low ethical risk.Source: Ethical Considerations · p. 8

A Benchmark Domain Breakdown

CLDoc-HHRBench spans ten document types, with Research Reports accounting for 43.6% of the benchmark and divided into five subcategories.

A Benchmark Domain Breakdown

B Annotation Protocol

The annotation protocol labels headings and their locations, text, order, and normalized hierarchy levels, followed by independent labeling, expert adjudication, and validation.

B Annotation Protocol

  • Annotators identify headings and label each heading’s bounding box, text, reading order, and normalized hierarchy level using the original pages.Skipped levels were prohibited: a heading could be at most one level deeper than its predecessor.Source: B Annotation Protocol · p. 11
  • Before adjudication, annotators agreed at 0.9611 F1 for heading detection, 0.9334 mean box IoU, 0.9761 normalized edit similarity for reading order, and 0.9673 hierarchy STEDS.Source: B Annotation Protocol · p. 11
  • Two blinded annotators labeled each document; experts adjudicated disagreements, and deterministic validators checked consistency before correction and revalidation.Six annotators and three experts spent approximately 800 person-hours.Source: B Annotation Protocol · p. 11

C Prompts

The prompts adapt each strategy’s visual inputs while keeping a shared heading-level output format. Variants also change the target headings, historical context, or use of images as needed.

C Prompts

  • All strategies ask the VLM to return JSON mapping each target heading ID to its hierarchy level.The reproduced Dynamic Chunking prompt adds this output instruction for the shared backbone, while leaving the rest unchanged.Source: C Prompts · p. 11
  • Our method supplies original page images with heading boxes, while LingDT-VL-OCR uses aggregated heading crops and MinerU-Popo concatenates chunk pages vertically.These are the strategy-specific meanings of the visual input in each prompt.Source: C Prompts · p. 11
  • Prompts vary the number of target headings and their pages, and memory-selection variants replace the page images and heading information with those for the selected historical nodes.The text-only variant excludes page images and serializes bounding boxes; the diagnostic replaces memory inputs with ground-truth hierarchy levels.Source: C Prompts · p. 11

D Implementation Details

The implementation details include method-specific configurations and prompts for the method and reproduced baselines. The shown prompts provide strategy-specific inputs and request heading-level outputs.

D Implementation Details

  • The implementation section introduces method-specific configurations and retry and validation settings, and identifies PaddleX as reproduced using its source code.The configurations are introduced as a summary; the supplied passage does not enumerate their contents.Source: D Implementation Details · p. 11
  • The method prompt provides page images, known title levels with image IDs, and target titles with image IDs for hierarchy prediction.The example includes two known H1 titles and two target titles.Source: D Implementation Details · p. 12
  • The Pseudo-ToC prompt presents page images and a list of titles to predict, while the Dynamic Chunking prompt supplies title-level entries with page and box information.Both prompt examples request hierarchy parsing and specify JSON output mapping IDs to levels.Source: D Implementation Details · p. 12
  • The illustrated Dynamic Chunking inputs include title-level entries with page, box, and content fields for headings across the example pages.The example includes headings such as “SUPERGRAVITY” and “DISCUSSION AND SUMMARY.”Source: D Implementation Details · p. 12

Related papers

Loading 2610.18238v1…