Source-linked AI summary
LongDocBench: Benchmarking TOC Hierarchy and Contextual Relationship Recovery in Long Documents
Yuefeng Zou, Yichen Lu, Jingxiao Yang, Bingtao Fu, Gaoyang Zhang, Xiongfei Bai, Tian Chen, Xiang Qi
TL;DR
Existing benchmarks do not directly evaluate long-document TOC hierarchy and contextual relationship recovery. LongDocBench benchmarks both tasks and shows their structures improve question answering while parsers remain limited at recovering them.
Problem
Existing benchmarks emphasize page-level parsing and only partially evaluate cross-page TOC hierarchies and typed table-or-figure contextual relationships.
Method
LongDocBench evaluates TOC hierarchy and contextual relationship recovery using 85 real-world documents spanning 2,582 pages with human-verified annotations.
Results
32.98% to 54.29%: jointly using human-verified TOC hierarchies and contextual relationships improves question-answering accuracy, while parsers reach only 0.55 Macro TEDS and 0.63 Macro.
Takeaways & Limitations
LongDocBench establishes document-level organization recovery as a useful, insufficiently addressed capability.
Takeaways & Limitations
LongDocBench is limited in scale and coverage.
Abstract
from arXiv · showhide
Parsing visual documents into machine-readable representations is fundamental to document intelligence. Existing benchmarks focus on page-level element recognition, reading order, formula recognition, and table structure. Long documents, however, also require document-level structure recovery. This includes reconstructing cross-page table-of-contents (TOC) hierarchies and identifying typed links from tables and figures to their captions, notes, and sources, often in one-to-many form. Because these structures are covered only partially or subsumed within broader parsing protocols, existing benchmarks cannot directly evaluate two key document-level tasks: \emph{Table-of-Contents Hierarchy Recovery} and \emph{Contextual Relationship Recovery}. To benchmark these two tasks, we introduce \textsc{LongDocBench}, comprising 85 real-world financial reports, textbooks, and academic papers spanning 2,582 pages, with up to 105 pages per document. It provides human-verified annotations for 3,937 heading nodes (mean node depth 3.55; maximum depth 9) and 3,258 contextual relationships annotated across 2,680 table and figure objects. We further evaluate both the downstream utility and recoverability of these structures. Long-document question-answering experiments show that human-verified TOC hierarchies and contextual relationships improve reasoning, with their combination providing complementary benefits. Meanwhile, representative document parsers remain limited on both recovery tasks despite strong page-level performance. To support further progress, we publicly release \textsc{LongDocBench} and its evaluation protocol and reproducible testbed for advancing document-level structure recovery in long documents.
1 Introduction
LongDocBench targets two under-evaluated document-level structure-recovery tasks: reconstructing cross-page TOC hierarchies and recovering typed contextual links among tables, figures, captions, notes, and sources. The benchmark provides human-verified annotations, evaluates downstream question answering and parser performance, and supports fine-grained reproducible evaluation.
- Motivation: Long documents require recovering deeply nested TOC hierarchies and spatially distant, potentially one-to-many links from tables and figures to captions, notes, and sources.Incorrect TOC parent–child assignments alter sectional scope, while contextual links may connect objects to distant text.
- Motivation: Existing benchmarks emphasize page-level recognition, reading order, formulas, and tables, providing only partial diagnosis of document-level organization.Prior work indicates that multilevel structure and image–text associations improve retrieval, generation, and question answering, motivating direct evaluation.
- Benchmark: 85 documents spanning 2,582 pages, with up to 105 pages per document, form LongDocBench across financial reports, textbooks, and academic papers.The benchmark is designed around Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery.
- Benchmark: 3,937 heading nodes and 3,258 typed contextual relationships involving 2,680 table and figure objects receive human-verified annotations.Heading annotations have mean node depth 3.55 and maximum depth 9.
- Findings: Both TOC hierarchies and contextual relationships improve long-document question-answering accuracy, while their joint use yields further gains.Representative parsers remain limited on complete TOC hierarchy and typed contextual relationship recovery despite strong, tightly clustered page-level performance.
2 LongDocBench
LongDocBench benchmarks Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery in long documents. It contains 85 documents spanning 2,582 pages, with human-verified annotations for 3,937 heading nodes and 3,258 typed contextual relationships across 2,680 table and figure objects.
- Benchmark Tasks: LongDocBench evaluates TOC hierarchy reconstruction and typed links between tables or figures and their captions, notes, and sources.TOC recovery reconstructs headings distributed across pages into complete, ordered hierarchies, while contextual recovery supports one-to-many relationships.
- Annotation Construction: Annotations combine parser-based pre-annotation, full-document manual correction, automated validation, and two-stage expert review before release.Annotators correct heading text, order, depth, and coordinates; revise contextual links; and verify one-to-many, shared, and cross-page relationships.
- Document Distribution: 85 documents span 2,582 pages, averaging 30.38 pages per document, with finance largest at 42 documents and 1,331 pages.The longest document is 105 pages; textbooks average 49.11 pages across 18 documents, while papers average 14.68 pages across 25 documents.
- TOC Hierarchies: 3,937 human-verified heading nodes span finance, textbook, and paper domains, with finance forming the largest and deepest hierarchies at mean depth 3.81 and maximum depth 9.The textbook domain contains 910 nodes with mean depth 3.31 and maximum depth 6, while the paper domain contains 445 nodes with mean depth 2.49.
- Contextual Relationships: 3,258 typed contextual relationships connect 1,169 tables and 1,511 figures, including 1,703 caption, 961 note, and 594 source relationships.Among the 2,680 annotated objects, 904 link to multiple contextual elements, demonstrating prevalent one-to-many associations.
3 Experiments
Experiments evaluate TOC hierarchy and contextual relationship recovery, their downstream reasoning utility, and the gap between strong page-level parsing and document-level structure recovery. Results show substantial benefits from verified structures, while automatic recovery remains limited.
- TOC Hierarchy Recovery: TextIn achieves 0.49 Weighted TEDS and 0.55 Macro TEDS under the with-ignorable setting, outperforming systems clustered at 0.44–0.45 and 0.51–0.52.TEDS jointly reflects heading content, order, and parent–child structure, with scores reported using document-macro and page-count-weighted aggregation.
- TOC Hierarchy Recovery: Macro TEDS reaches 0.69–0.82 on papers but only 0.44–0.48 on financial documents and 0.37–0.46 on textbooks.Weighted TEDS is consistently lower than Macro TEDS, indicating worse recovery as document length increases.
- TOC Hierarchy Recovery: Accounting for ignorable headings improves Macro TEDS by only 0.02–0.04 and Weighted TEDS by 0.02–0.03, without materially changing rankings or domain trends.Remaining errors primarily involve heading order, depth, and parent–child structure.
- Contextual Relationship Recovery: GPT-5.6-Sol achieves the highest contextual-recovery Macro score of 0.63, followed by Qwen3.5-397B with thinking at 0.60, while no model dominates every relation type.Source scores range from 0.61 to 0.85, captions from 0.53–0.65, and notes from 0.16–0.42; thinking raises Qwen3.5-397B from 0.59 to 0.60.
- Page-Level Versus Document-Level Recovery: Despite Overall page-level scores of 93.54–95.96, the best systems reach only 0.55 Macro TEDS and 0.63 Macro for document-level recovery.Strong page-element recognition and local structure do not imply reliable cross-page hierarchies or typed table/figure relationships.
- Downstream Long-Document Reasoning: Verified combined structures achieve 54.29% answer accuracy versus 32.98% for Fixed + BM25, while recovered structures achieve only 37.86% when combined.Verified TOC and Relations individually reach 45.95% and 47.62%; automatic TextIn TOC and Qwen3.5 Relations alone reach 30.83% and 31.90%.
4 Conclusion
LongDocBench introduces a human-verified benchmark for recovering table-of-contents hierarchies and contextual relationships in long documents. Its annotated corpus establishes document-level organization recovery as a useful but insufficiently addressed capability.
- Conclusion: LongDocBench evaluates Table-of-Contents Hierarchy Recovery and Contextual Relationship Recovery as document-level structure-recovery tasks.The benchmark is human-verified.
- Conclusion: 85 documents spanning 2,582 pages contain 3,937 heading nodes and 3,258 typed contextual relationships across 2,680 table and figure objects.These annotations cover real-world documents.
- Conclusion: Document-level organization recovery is a useful, insufficiently addressed capability.LongDocBench establishes this capability as an important benchmark target.