PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Hierarchical Structures for Agentic Document Analysis

Ruiying Ma, Yiming Lin, Aditya G. Parameswaran

arXiv:2609.33322v1cs.DBcs.AIcs.CLcs.IR

TL;DR

Prior structure-inference methods lack broad guarantees, despite the value of document hierarchies for efficient analysis. SHED infers robust, compact structures and characterizes the document classes for which its approaches guarantee robustness. It improves structure-inference F-1 scores and agentic QA accuracy while reducing costs against baselines.

  • Problem

    Prior approaches for inferring document hierarchies provide no broad guarantees about matching the true structure, despite the need to analyze long documents efficiently.

    Source: §1 · p. 1

  • Method

    SHED is a two-stage workflow that infers semantic depths and assembles a robust, compact SHT, with a pluggable family of first-stage approaches.

    Source: §1 · p. 3, §3.1 · p. 4

  • Results

    SHED improves SHT-inference F-1 by 13%–68% over non-LLM baselines and 9%–15% over expensive LLM-based approaches, while agentic QA accuracy improves by 3%–23% with costs reduced by up to 10×.

    Source: §10 · p. 14

  • Takeaways & Limitations

    SHED-inferred structures support agents' section-by-section retrieval, with higher average accuracy and lower costs than baselines across four datasets.

    Source: §1 · p. 3

    SHED focuses on documents with rich visual formatting, such as PDF and Word files.

    Source: §2 · p. 3

Abstract

from arXiv · show

Large Language Models (LLMs) enable us to better understand text documents, including PDFs and Word documents. However, LLMs, as well as more modern LLM agents, i.e., those with tool-calling abilities, typically treat such documents as plain text, ignoring the fact that they are often organized hierarchically into sections and subsections. Extracting this structure, while difficult, can improve efficiency and effectiveness for agents (and humans)---since only sections relevant to a given task need to be processed. Unfortunately, prior work on structure extraction provides no formal guarantees on how well the inferred structure matches the true one. Instead, we target a robust and compact variant that is feasible to infer and useful in practice. Robustness ensures that the text under each subsection header is a superset of the text under the same header in the true structure. Compactness seeks to minimize this superset, reducing agentic cost (or human cognitive load). We propose SHED, a two-stage workflow for inferring a robust and compact structure. The first stage is pluggable with an infinite family of approaches, each guaranteeing robustness for a specific document class. We theoretically characterize the document space using these classes and their hierarchical relationships. Empirically, SHED improves F-1 scores (measuring the robustness--compactness trade-off) by 13%--68% over non-LLM baselines and 9%--15% over expensive LLM-based approaches. Finally, we show how SHED-inferred structures are valuable for agentic document analysis: agents using SHED outperform baselines, achieving 3%--23% higher accuracy while being up to 10x cheaper.

1 Introduction

SHED targets robust, compact document hierarchies because existing tools often ignore structure and prior inference methods lack broad guarantees. Its inferred structures improve both structure-inference scores and agentic document analysis outcomes.

Motivation

  • Prior SHT-inference approaches provide no broad guarantees that inferred structures match the true structure, while LLM-based approaches are also expensive.Identifying the true SHT is described as unrealistic because of heterogeneous layouts, semantic ambiguity, and document length.Source: §1 · p. 1

Robustness and Compactness

  • Robustness requires each header's text span to include the corresponding true span, while compactness minimizes the extra context to reduce processing cost or human cognitive load.The paper compares this objective to ensuring 100% recall while maximizing precision when retrieving subsections.Source: §1 · p. 2

SHED

  • SHED uses header visual patterns in a two-stage workflow: it clusters headers and infers cluster relationships, then derives node edges to assemble a compact SHT.Its first stage infers semantic depths; the second assembles edges from their relative ordering.Source: §1 · p. 2

Characterization of Robustness Classes

  • SHED characterizes document classes for which approaches guarantee robustness, and generalizes semantic-depth inference to an infinite family of approaches.Local-first and global-first do not dominate one another; depth-aligned documents are a class for which local-first guarantees robustness but global-first does not.Source: §1 · p. 3

Application to Agentic Document Analysis

  • 3%–23% higher average accuracy and up to 10× lower total costs over baselines across four datasets are reported for agents using SHED-inferred structures.The structures let agents retrieve a section or subsection at a time.Source: §1 · p. 3

Contributions

  • SHED's local-first approach improves F-1 by 13%–68% over non-LLM baselines and 9%–15% over expensive LLM-based approaches.F-1 measures the robustness–compactness trade-off.Source: §1 · pp. 2–3

2 Robust Semantic Hierarchical Trees

The paper represents document hierarchy as a Semantic Hierarchical Tree (SHT) and targets robust trees whose extra text is minimized. Robustness preserves each header’s true text span as a subset of its inferred span, while compactness penalizes excess text.

2.1 Preliminaries

  • Documents are serialized as labeled phrases, and visual patterns such as font, alignment, capitalization, and list type provide cues for inferring hierarchy.Header phrases are identified by their section_header label.Source: §2.1 · p. 3
  • An SHT is a rooted tree whose nodes correspond one-to-one with header phrases and whose preorder traversal preserves their document order.The set of headers is fixed, but multiple edge structures may be valid; the true SHT is the structure humans consider best.Source: §2.1 · pp. 3–4
  • A node’s text span runs from its header through the phrase before the next non-descendant header in the tree.This span defines the content associated with that node.Source: §2.1 · p. 4

2.2 Robust and Compact SHTs

  • A robust SHT requires every inferred node span to contain that node’s span in the true SHT.Equivalently, every true parent-child relationship must remain an ancestor-descendant relationship in the inferred tree.Source: §2.2 · p. 4
  • Compactness minimizes the extra text included in inferred node spans beyond their true spans.The target is a robust tree with the excess-span quantity C minimized.Source: §2.2 · p. 4
  • The objective favors a robust SHT over a deep tree that captures excessive text or a wide tree that omits true context.Constructing the true hierarchy may be unrealistic because of semantic ambiguity, so a robust alternative can suffice.Source: §2.2 · p. 4

3 SHT Inference: Two Approaches

SHED infers semantic depths from header visual patterns, then assembles a tree using those depths and text-span coverage. Its workflow guarantees an SHT and bounds its depth, while local-first and global-first approaches provide robustness for different document classes.

3.1 Overall Workflow

  • SHED first clusters headers by visual pattern and infers cluster semantic depths, then assembles tree edges so each parent has smaller semantic depth than its child.Semantic depth constrains relationships; it does not directly assign exact tree levels.Source: §3.1 · p. 4
  • The two inference approaches guarantee robustness for different document classes, and neither local-first nor global-first dominates the other.SHED combines them into an infinite family of approaches, each guaranteeing robustness for a particular class.Source: §3.1 · p. 5

3.2 Semantic Depth Inference: Local-First

  • Local-first assigns each cluster a depth one greater than the depth of the cluster containing the header immediately before its indexing node.This conservative choice preserves robustness but can reduce compactness when that preceding node is below the true parent.Source: §3.2 · p. 5

3.3 Semantic Depth Inference: Global-First

  • Global-first assigns each cluster a depth one greater than the maximum depth among all clusters preceding its indexing node.Considering all preceding nodes covers the worst case when the true parent is unknown; it generalizes ZenDB’s algorithm beyond well-formatted documents.Source: §3.3 · p. 5

3.4 SHT Assembly

  • Restricting candidate parents by semantic depth reduces the rightmost-path candidates while retaining the true parent and improving compactness.Choosing the latest remaining candidate preserves robustness because it is the true parent or its descendant.Source: §3.4 · p. 6
  • Assembly processes headers in document order and selects the latest candidate parent that covers the header and has smaller semantic depth.The selected node is added as the parent’s rightmost child, preserving reading order.Source: §3.4 · p. 6

3.5.1 SHT Correctness

  • The assembly algorithm preserves reading order, so its output is an SHT.Theorem 2 attributes this property to inserting each node under the required conditions.Source: §3.5.1 · p. 6

3.5 Correctness and Compactness

  • The inferred tree is an SHT, and its depth is bounded by the number of distinct header-pattern clusters, unlike a single-chain tree whose depth equals its node count.As depth ratio to the true SHT rises from ≤1 to 2.2, average compactness drops from 0.92 to 0.83; bounded depth is a proxy, not an optimality guarantee.Source: §3.5.1 · p. 6, §3.5.2 · p. 7

4 Robustness Analysis and Hierarchy

This section characterizes when SHED’s inferred semantic depths guarantee robust trees, compares depth-inference approaches, and organizes the document classes for which guarantees hold.

4.1 Semantic Depth Conformance

  • SD conformance is necessary and sufficient for robustness: every true parent must have a smaller inferred semantic depth than its child.The condition applies to semantic depths assigned by any inference approach.Source: §4.1 · p. 7

4.2.1 Illustrative Scenarios

  • Local-first and global-first can each produce a robust SHT where the other fails, depending on whether their inferred depths satisfy SD conformance.The examples reverse which approach succeeds across two documents.Source: §4.2.1 · p. 7, Figure 6 · p. 8

4.2.2 A Document Class Favoring Local-First Inference

  • For depth-aligned documents, local-first assigns each node its true depth and guarantees a robust SHT; this class favors local-first over global-first.Depth alignment requires each cluster’s indexing node to be its parent’s leftmost child and same-cluster nodes to share a level.Source: §4.2.2 · p. 7

4.3 Semantic Depth Inference Family

  • An infinite family of SD inference approaches interpolates local-first and global-first using cluster-specific K mappings.Setting K(C) to 1 for every cluster recovers local-first; setting it to 0 recovers global-first.Source: §4.3 · pp. 7–8

4.4 Hierarchy of Robustness

  • Well-formatted documents are a strict subset of loosely-formatted documents, which guarantee robustness for every approach in K; 60.17% of 600 documents were loosely formatted versus 23.39% well-formatted.Loosely-formattedness relaxes the sibling-cluster requirement and uses the first ancestor-descendant occurrence rather than the first parent-child occurrence.Source: §4.4 · pp. 8–9

4.5 Relationship to ZenDB

  • SHED generalizes ZenDB’s well-formatted-document scope to broader document classes, with robustness guarantees for more than 60% rather than fewer than 25% of real-world documents.The paper also frames SHED as a robust-and-compact SHT formulation and a two-stage framework that generalizes ZenDB’s algorithm.Source: §4.5 · p. 9

5 Evaluation Setup

The evaluation uses composite real-world PDFs to test structure inference and document QA, then implements SHED with extracted visual patterns and local-first depth inference.

5.1 Datasets

  • The evaluation draws on civic agenda reports, legal contracts, financial filings, and scientific papers from existing document QA benchmarks.The original datasets were considered insufficiently challenging because documents or questions were often simple.Source: §5.1 · p. 9
  • Each question’s source document is concatenated with randomly selected same-dataset documents, making its identification within the composite necessary to answer the question.The composite SHT is built by manually structuring constituent documents and concatenating their roots.Source: §5.1 · p. 9
  • Finance documents are longest, followed by Papers; Civic has complex nested structures, while Contracts has dense layouts with short spans between headers.Dataset statistics include average document size, document count, and question count.Source: §5.1 · p. 9
  • Three additional law, telecom, and environment datasets with complex hierarchies use synthesized questions and manually labeled ground-truth answers for large-scale evaluation.Source: §5.1 · p. 9

5.2 SHED Implementation

  • SHED identifies headers with CPU-runnable Huridocs models, extracts visual patterns with PyMuPDF, clusters headers, and applies local-first depth inference to construct SHTs.Local-first is chosen because more documents satisfy its SD conformance and it infers more robust and compact SHTs than global-first in the cited analysis.Source: §5.2 · p. 9

6 Evaluation of SHT Inference

Across four datasets, SHED achieves higher average robustness–compactness F-1 than both non-LLM and more expensive LLM-based baselines. The evaluation compares multiple structure-inference approaches and measures text-span and ancestor-header agreement.

6 Evaluation of SHT Inference

  • 13%–68% higher average F-1 than non-LLM baselines and 9%–15% higher than significantly more expensive LLM-based approaches are reported for SHED across four datasets.F-1 measures the robustness–compactness trade-off; Table 3 reports the two metric families and their results.Source: §6 · p. 9, Table 3 · p. 11

6.1.1 Baselines

  • Five baselines span rule-based Deep and Wide trees, GROBID, and one-shot LLM-text and batched-page LLM-vision structure inference.GROBID is evaluated on composite documents using individual-document SHTs, which the authors treat as an optimistic upper bound.Source: §6.1.1 · pp. 9–10

6.1.2 Evaluation Metrics

  • Top-down metrics compare inferred and true text spans, while bottom-up metrics compare their ancestor-header lists; recall, precision, and F-1 represent robustness, compactness, and their trade-off.Scores are averaged over true-SHT nodes for a common evaluation basis.Source: §6.1.2 · p. 10

6.2 SHT Inference Results

  • Header identification is consistently accurate for SHED, while GROBID performs poorly outside its training domain and LLM approaches perform poorly on Contracts and Finance.LLM-text and LLM-vision incur total costs of $157 and $164 across datasets, respectively; SHED and the other baselines incur no LLM cost.Source: §6.2 · p. 10, Table 2 · p. 11
  • Deep has the highest recall and Wide the highest precision across almost all datasets, but both have low F-1; SHED follows them with minor recall and precision reductions.LLM-vision has higher average F-1 than LLM-text under both metric families; the authors attribute this to added visual information.Source: §6.2 · p. 10, Table 3 · p. 11
  • Using true rather than extracted headers improves average recall, precision, and F-1 by only 3%–6% under both top-down and bottom-up metrics.Subsequent evaluations therefore use extracted headers as the realistic setting.Source: §6.2 · p. 10

7 Evaluation of Agentic Question Answering

The evaluation compares SHT-assisted and non-SHT strategies, then tests retrieval tools and SHT inference approaches. SHED-based agents achieve strong accuracy-cost results across these comparisons.

7.1.1 Evaluated Strategies

  • The evaluation compares full-document in-context processing and tool-based retrieval, each with and without SHT assistance.The SHT-based agent receives the true SHT and uses read_section to retrieve a selected section; the vanilla-grep agent searches matching chunks.Source: §7.1.1 · p. 11

7.1.2 Evaluation Metrics

  • Evaluation uses task-specific metrics: F-1 for Civic, 0-1 accuracy for Contracts, and LLM-as-judge for Finance and Papers; total end-to-end inference and QA cost is reported.Table 4 reports average accuracy and total cost by dataset, with stratified averages and totals.Source: §7.1.2 · pp. 11–12, Table 4 · p. 12

7.2 Usefulness of SHTs

  • Across the four strategies, SHT assistance improves average accuracy; the SHT-based agent is 10× and 9× cheaper than SHT-augmented-context and vanilla-in-context, respectively.On Finance, it is 30× and 29× cheaper than those two strategies, while costing more than vanilla-grep but achieving 8% higher accuracy.Source: §7.2 · p. 12, Table 4 · p. 12

7.3 Agentic Retrieval Ablation

  • The SHT-based agent is 3%–23% more accurate and 1.3–2× cheaper than all agentic retrieval baselines.SHTs also improve accuracy for grep and embedding-based retrieval compared with their vanilla counterparts.Source: §7.3 · p. 12, Table 5 · p. 12

7.4 SHT Ablation

  • SHED achieves the highest average accuracy and lowest total cost among SHT inference approaches, performs comparably to true SHTs, and exceeds other inference approaches by 6%–19% in average accuracy.Deep costs 4×–18× more than SHED and has up to a 55% accuracy drop; LLM-text and LLM-vision are each about 9× more expensive and have 8% and 6% lower average accuracy, respectively.Source: §7.4 · pp. 12–13, Table 6 · p. 12

8 Scalability Analysis

The scalability evaluation tests inference latency, growing document composition sizes, and three datasets of complex documents. SHED-based agents retain accuracy at low cost across these settings.

8.1 SHT Inference Latency

  • SHED is estimated to process 955 documents—about 54K pages and 30M tokens—sequentially in 5 hours.GROBID takes 4 hours but infers poor SHTs; LLM-based approaches take 2–5 hours and are costly, while SmolDocling-256M-preview takes about 11.4 days.Source: §8.1 · p. 13

8.2 Document Composition

  • As composition size doubles, SHED’s cost rises only 1.4×; its agent remains accurate at 0.71—0.73 average accuracy and costs $12–$28 total.It is up to 7%, 19%, and 27% more accurate than baselines at sizes 1, k+1, and 2(k+1), respectively, and is on par with true SHTs.Source: §8.2 · p. 13

8.3 Complex Documents with Synthetic Q&A

  • On three new complex-document datasets, SHED has 7%–18% higher QA average accuracy at up to 15× lower cost than baselines.The evaluation used 119 documents and 142 synthetic Q&A pairs across CFR, ETSI, and FERC.Source: §8.3 · p. 13

9 Related Work

Related work spans document analytics, structure inference, agentic document processing, and retrieval-augmented generation. The paper positions SHED as a structure-aware approach applicable beyond template-specific systems.

Systems for Document Analytics

  • Document analytics systems often treat documents as linear text; ZenDB considers structure but targets collections sharing a template, while SHED broadens applicability with a new formulation and general two-stage framework.Table-extraction systems [5] [40] [79] could complement this work.Source: §9 · p. 13

Document Structure Inference

  • For PDF and Word structure inference, rule-based methods can fail on heterogeneous layouts, learning-based methods may generalize poorly, and LLM approaches lack robustness and compactness guarantees.SHED uses learning-based methods for header identification while providing robustness guarantees for structure inference.Source: §9 · p. 13

Agentic Document Processing

  • Document agents often process full documents or retrieve relevant portions while ignoring hierarchy; SHED can potentially recover hierarchies for agent skills without manual specification.Related knowledge-management systems organize document collections hierarchically rather than structuring a single document.Source: §9 · p. 13

Retrieval-Augmented Generation (RAG)

  • Most RAG approaches use chunks, summary trees, or knowledge graphs rather than inherent document hierarchy; hierarchical RAG methods infer structure with LLMs at higher cost and without guarantees.SHED could be used to improve RAG approaches.Source: §9 · pp. 13–14

10 Conclusion

The conclusion presents SHED as a two-stage workflow for robust SHT inference, with theoretical document-class guarantees and empirical gains in structure inference and agentic QA.

10 Conclusion

  • SHED’s two-stage workflow guarantees robustness with good compactness, and its agentic QA application improves accuracy by 3%–23% while reducing cost by up to 10×.For SHT inference, it improves F-1 by 13%–68% over non-LLM baselines and 9%–15% over expensive LLM-based approaches.Source: §10 · p. 14

A.1 Proof for Theorem 1

  • The proof establishes that an SHT is robust exactly when each true parent-child relationship remains an ancestor-descendant relationship in the inferred tree.The forward direction shows true text spans are contained in inferred spans; the converse derives the ancestry condition from robustness.Source: §A.1 · p. 17

A.2 Proof for Theorem 2

  • The insertion proof shows that adding each node as a rightmost child on the rightmost path preserves pre-order traversal in increasing node-index order.Source: §A.2 · p. 17

A.3 Proof for Theorem 3

The proofs relate robustness to semantic-depth ordering and establish document conditions under which local-first or a broader inference family guarantees robust SHTs. They also connect SHED’s global-first workflow to ZenDB’s construction and well-formattedness definitions.

A.3 Proof for Theorem 3

  • Robustness is equivalent to semantic-depth conformance: each parent cluster must have smaller inferred depth than its child cluster.The proof uses this ordering to show that the true parent is a candidate when each node is inserted, preserving robustness.Source: §A.3 · p. 17

A.4 Depth-Aligned Documents

  • For depth-aligned documents, local-first recovers every node’s true depth and therefore guarantees robustness, while global-first can fail.Depth alignment requires the indexing node to be the leftmost child and clusters to contain nodes at the same true level.Source: §A.4 · p. 17

A.5 Proof for Theorem 5

  • Well-formatted documents are loosely formatted, but the converse does not follow because well-formattedness cannot infer parent-child relations from ancestor-descendant occurrences.Source: §A.5 · p. 17

A.6 Proof for Theorem 6

  • The SD inference family guarantees robustness on loosely formatted documents, using semantic-depth conformance as the robustness condition.Its inference rule interpolates between local-first and global-first semantic-depth values.Source: §A.6 · pp. 17–18, B SD Inference Family · p. 18

C.1 SHT Construction

  • Global-first SD inference followed by SHT assembly is equivalent to ZenDB’s SHT construction algorithm [41].The proof reduces both procedures to selecting the latest earlier node whose cluster index is smaller.Source: C Relationship to ZenDB · p. 18, §C.1 · p. 18

C.2 Well-Formattedness

  • The paper’s parent-child consistency condition is equivalent to ZenDB’s original requirement that nodes in a cluster share identical visual-prefix sequences [41].Source: §C.2 · p. 18

D Datasets

The evaluation datasets combine real-world documents and questions from varied domains, lengths, and templates. The appendix introduces additional evaluation results.

D Datasets

  • Four datasets cover civic reports, NDAs, financial filings, and 416 NLP research papers, with document names placed on prepended title pages.Finance questions may require identifying a hashed constituent document, and serialized financial tables are reformatted as Markdown sections and list items.Source: D Datasets · p. 19

E Additional Evaluations and Results

E.1 True Header Ablation

SHED’s inferred structures remain robust and compact as documents are composed, while supporting accurate, lower-cost QA. True headers improve most metrics, and greater inferred depth is associated with lower compactness.

E.1 True Header Ablation

  • With true headers, SHED reaches 96%–99% M↓ recall; the remaining 1%–4% comes from documents that violate SD conformance.Across all metrics, the average gap between true and extracted headers is moderate at 3%–6%, so the evaluation uses extracted headers as the realistic setting.Source: §E.1 · p. 19

E.2 Compactness-Depth Correlation

  • Average bottom-up compactness declines from 0.92 to 0.83 for extracted headers and from 0.95 to 0.91 for true headers across increasing depth-ratio bins, indicating deeper SHTs tend to be less compact.The three bins are (0.5, 1], (1, 1.2], and (1.2, 2.2].Source: §E.2 · p. 19, Figure 8 · p. 19

E.3 Scalability Analysis: Document Composition

  • 13%–71% higher M↓ F-1 and 15%–55% higher M↑ F-1 than baselines at composition size 2(k+1); SHED-based QA reaches 0.71—0.73 average accuracy at $12–$28 total cost.At that size, QA accuracy is up to 27% higher than baselines; costs rise only 1.4× as composition size doubles. SHED’s QA costs are up to 28× lower than baselines.Source: §E.3 · p. 19

E.4 Scalability Analysis: Complex Documents with Synthetic Q&A

The evaluation adds complex, individually structured CFR, ETSI, and FERC documents with structure-dependent synthetic questions. SHED outperforms baselines on QA accuracy and structure metrics, with results varying by dataset and query complexity.

E.4 Scalability Analysis: Complex Documents with Synthetic Q&A

  • The evaluation selects CFR volumes with at least six agency sections, ETSI documents of at least 40 pages, and FERC documents from 2020–2025.Questions are generated from structure-dependent templates instantiated with document entities; embedded tables of contents provide the true SHTs.Source: §E.4 · p. 20
  • SHED’s QA accuracy is 7%–18% higher on average than three baselines, at nearly Wide’s cost and 15× below vanilla-in-context.On CFR, missed headers are mostly lowest-level regulations whose content agents can recover from the parent regime span; FERC accuracy is low across strategies because queries require complex cross-subsection analysis.Source: §E.4 · p. 20
  • Across datasets, SHED exceeds Wide by 8%–15% in M↓ recall and 1%–13% in M↓ F-1, with larger gains under M↑.The corresponding M↑ gains are 16%–26% in recall and 12%–20% in F-1.Source: §E.4 · p. 20

E.5 Scalability Analysis: SHT Inference Latency

The passages cover SHED’s scalability and inference latency, including its performance on tested datasets and additional real-world documents. They also describe a rubric for judging finance answers against ground-truth references based on correctness, completeness, and avoiding unsupported claims.

E.5 Scalability Analysis: SHT Inference Latency

  • SHED processes 955 documents in about 5 hours sequentially, while GROBID, LLM-text, and LLM-vision take 4, 2, and 5 hours but have quality or cost drawbacks.The 955 documents comprise nearly 54K pages and 30M tokens; LLM-based methods are estimated at ~$0.2M per 1M documents, and SmolDocling-256M-preview is estimated at ~11.4 days. Header identification accounts for 90%–94% of SHED’s time.Source: Table 14 · p. 22, §E.5 · p. 22

E.6 An Advanced SHT Inference Baseline

  • SmolDocling-256M-preview’s SHTs average up to 56% below SHED, with mostly flat structures and less accurate header identification cited as possible causes.78 of 80 sampled documents have only 1-level non-root nodes; header identification recall/precision/F-1 ranges from 0.20/0.74/0.31 on Contracts to 0.85/0.46/0.58 on Finance.Source: Table 15 · p. 22, §E.6 · p. 22

E.7 Sensitivity of SHED

  • On the four datasets, global-first and local-first differ by less than 1% in average recall, precision, and F-1, while local-first guarantees recall=1 and yields 4% and 5% higher average F-1 on M↓ and M↑ for 40 real-world documents.The authors attribute the main-dataset stability partly to over 60% of documents being loosely formatted, for which either inference approach guarantees robustness.Source: Table 16 · p. 22, Table 17 · p. 22, §E.7 · p. 22

F.1 LLM-based SHT Inference

  • The LLM-based SHT prompt requires identifying all headings, preserving their wording and order, and representing hierarchy in Markdown without inventing or omitting headings.The LLM-vision variant substitutes document-page images in reading order for serialized text.Source: §F.1 · pp. 22–23

F.2 LLM-as-judge

  • The judge scores each model answer against each ground-truth answer independently, then uses the maximum score as the final score.It outputs the final score, matched reference index, per-reference scores, and a rationale as JSON.Source: §F.2 · p. 24
  • The 0–5 rubric ranges from fully correct answers with all key facts and no unsupported claims to incorrect, unrelated, or unjustified factual answers when the reference says unanswerable.Intermediate scores reflect omissions, imprecision, contradictions, or unsupported details.Source: §F.2 · pp. 24–25
  • Scoring checks whether required facts are present, including exact quantities and units, complete item sets, and the procedure or criteria in definition questions.The rubric also requires an answer to match an explicit Yes/No or “metric not relevant” stance in the reference.Source: §F.2 · pp. 24–25

Related papers

Loading 2609.33322v1…