PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Template Inference for Data Extraction from Documents

Yiming Lin, Mawil Hasan, Rohan Kosalge, Alvin Cheung, Aditya G. Parameswaran

arXiv:2501.06659v2cs.DBcs.CV

TL;DR

Templatized documents contain structured data but can have complex, mixed layouts that challenge extraction tools and require substantial human effort. TWIX infers a visual template before extracting data, outperforming comparison systems on accuracy and large-collection efficiency.

  • Problem

    Existing extraction approaches struggle with complex layouts, may require labeled examples and field specifications, and can incur high latency and cost on large document collections.

    Source: §1 · p. 2

  • Method

    TWIX infers fields from recurring location patterns, assembles them into a template using visual constraints, and extracts data from the resulting records and blocks.

    Source: §1 · pp. 3–4

  • Results

    Across two benchmarks, TWIX achieves around 90% precision and recall, outperforming each baseline by over 25% in both metrics, and is 520× faster and 3786× cheaper on collections with 2000+ pages.

    Source: §1 · p. 5

  • Takeaways & Limitations

    Template-based extraction preserves relationships between values without requiring users to specify attributes or provide labeled data.

    Source: §1 · p. 3

    When OCR splits a phrase spanning multiple table rows, TWIX may need visual cues or LLM semantic checks to merge the fragments.

    Source: §7.2 · p. 23

Abstract

from arXiv · show

Many templatized documents are programmatically generated from structured data following a visual template. Such documents include invoices, tax documents, financial reports, and purchase orders. Effective data extraction from these documents is crucial to support downstream analytical tasks. Current data extraction tools often struggle with complex document layouts, incur high latency and/or cost on large datasets, and require significant human effort. The key insight of our tool, TWIX, is to infer the underlying template used to create such documents, and then extract the data, rather than extracting directly from documents. To do so, TWIX first infers the underlying fields, such as columns of tabular portions or keys in co-located key-value pairs, by leveraging their consistent location patterns (e.g., two fields in the same template repeatedly co-occur within a fixed distance apart across multiple records). TWIX then assembles these fields into a template by enforcing visual constraints, such as vertically aligning table rows with their column headers for tabular regions, and horizontally aligning keys with their values for key-value pairs. TWIX then uses this inferred template to accurately and efficiently extract data from templatized documents at a low cost. On one benchmark with 34 diverse real-world datasets, TWIX outperforms state-of-the-art structured data extraction tools (Evaporate, Textract, and Azure Document Intelligence), and vision-based LLMs like GPT-4-Vision, by over 25% in precision and recall. Another benchmark with 30 large datasets demonstrates TWIX's scalability: it is 520X faster and 3,786X cheaper than the most competitive compared tool, for extracting data from large document collections with over 2000 pages.

1 Introduction

TWIX addresses the difficulty of extracting structured data from templatized documents with mixed, nested layouts by inferring their visual template before extraction. It uses repeated location patterns to infer fields, then assembles fields into structures constrained by visual alignment.

1 Introduction

  • Templatized documents mix tabular and key-value blocks, may nest blocks, and require reasoning about both vertical and horizontal alignments.These layouts complicate extraction, while existing approaches may require labeled examples or lose relationships between extracted values.Source: §1 · pp. 2–3
  • TWIX reconstructs a template from documents that share it, then decomposes records into simple table or key-value blocks for extraction.Template-based extraction preserves relationships between values without requiring users to specify attributes or provide labeled data.Source: §1 · pp. 3, 5
  • TWIX infers fields by clustering phrases with consistent location patterns and uses LLM semantic knowledge when visual cues are insufficient.The LLM role is restricted to simple questions such as whether a phrase is a field or a value.Source: §1 · p. 4
  • TWIX assembles fields by labeling rows and enforcing structural constraints, including vertical alignment between table keys and values.The row-labeling problem is formalized as an ILP; the paper proves it NP-hard and provides efficient pruning strategies.Source: §1 · p. 5
  • TWIX achieves around 90% precision and recall, outperforming each baseline by over 25% in both metrics across two benchmarks.It also exceeds the best baseline by over 31% in structure precision and 22% in structure recall, and is 520× faster and 3786× cheaper on collections with 2000+ pages.Source: §1 · p. 5

2 Background and Overview

TWIX operates on concatenated documents generated from a shared template, using OCR-extracted phrases and their bounding boxes. It infers fields and row labels, then uses the resulting template to extract data.

2 Background and Overview

  • TWIX infers a template for a concatenated document whose constituent documents share a template, provided OCR can extract phrases and bounding boxes.The pipeline uses the inferred template to extract data from the entire document.Source: §2 · pp. 5–6
  • A row groups horizontally aligned phrases, while a block is a consecutive list of rows; fields are table column names or key-value keys.TWIX greedily scans phrases in location order to form rows.Source: §2 · p. 6
  • TWIX applies OCR to extract phrases with bounding boxes and locations, then clusters phrases with consistent location patterns to infer fields.LLMs provide semantic knowledge to improve robustness, and an ILP assigns row labels subject to visual alignment constraints.Source: §2 · p. 6

3 Template Formalization

TWIX represents a document template as an ordered tree whose nodes specify data-block types and fields. The tree captures block order, repetition, and nesting, which correspond to records’ visual structure.

3 Template Formalization

  • A template is an ordered directed tree with an artificial root; each other node specifies a Table or Key-Value block and its fields.The tree defines which fields are populated, how they are grouped, and the order of blocks.Source: §3 · pp. 6–7
  • Each node generates one or more data blocks in the template’s predefined order, and every field appears once in each generated block.A record consists of a sequence of blocks generated by the template’s nodes.Source: §3 · p. 7
  • An invoice template can nest repeated blocks: its child node’s blocks overlap the parent block and occur within its sequence.By contrast, police-record template nodes at the same level correspond to non-overlapping blocks.Source: §3 · p. 8
  • TWIX infers the template from identified fields before extracting data by dividing documents into records and simple table or key-value blocks.Once the template is inferred, those blocks can be extracted individually.Source: §3 · p. 8

4 Field Inference

TWIX identifies fields using repeated phrase-location patterns, then uses matching properties and a clustering procedure to handle ordinary and recurring-block cases. LLM semantics help distinguish fields from values when location patterns alone are ambiguous.

4 Field Inference

  • TWIX first infers fields from phrases that tend to appear in similar locations across records.It bases location patterns on OCR extraction order rather than physical bounding boxes, which may be brittle under OCR noise.Source: §4 · p. 8, §4.2 · p. 9

4.1 Location Vectors and Matches

  • Fields sharing a template block tend to have location vectors related by a constant shift, formalized as a perfect match when their vectors have equal length and consistent distance.Partial perfect matches allow a matching subsequence when a field appears in multiple blocks; for example, Line partially matches Start Date.Source: §4.1 · p. 8

4.2 Analysis of Location-based Matches

  • For fields in a unique table node, equal-length location vectors imply a perfect match; unequal lengths can yield a partial match through the relevant subsequence.For key-value fields, the stated matching property assumes consistent filling patterns; partial matches can also occur when a field is consistently present or missing across a sequence of blocks.Source: §4.2 · p. 9

4.3 Field Inference Algorithm

  • TWIX clusters perfect matches, prunes likely value clusters using cluster size and LLM semantic judgments, then recovers clusters that partially match identified field clusters.LLM semantics address cases such as a value that is constant across records and therefore resembles its field in location patterns.Source: §4.3 · pp. 9–10

5 Template Inference

The section introduces template inference for a concatenated document populated with records, and identifies phrase clustering, cluster pruning, and cluster recovery as steps of the field-inference algorithm.

5 Template Inference

  • The section considers inferring template T from a concatenated document populated with records generated by T.The accompanying algorithm labels identify phrase clustering, cluster pruning, and cluster recovery as its three steps.Source: §5 · pp. 10–11

21 Return 𝐹

TWIX labels rows using field-pattern probabilities and visual alignment, then assembles those labels into a template structure. To limit a hard optimization problem, it uses a short row subsequence that is guaranteed to include a complete record under a stated field condition.

5.1.1 Row Label Probabilities and Alignment.

  • A row’s label probabilities derive from adjacent phrase-pair patterns, while row alignment checks whether phrases overlap vertically without one phrase aligning to multiple phrases in the other row.Bounding-box overlap is described as robust to minor OCR inaccuracies.Source: §5.1.1 · pp. 11–12

5.1.2 The Row Label Assignment Problem.

  • Row labeling maximizes the product of row-label probabilities subject to constraints requiring each row to have one label and Key and Value rows to align in order.The row-labeling problem is NP-hard and is formulated as an integer linear program after linearizing its objective.Source: §5.1.2 · p. 13, §5.1.3 · p. 13
  • The Metadata label, assigned a low probability, prevents rows lacking aligned Key or Value counterparts from forcing a zero-probability assignment.The authors say metadata rows do not affect template structure and are typically misaligned with true Key rows.Source: §5.1.2 · p. 13

5.1.3 Solving Row Label Assignment: ILP and Hardness.

  • TWIX uses the smallest consecutive row subsequence in which every inferred field appears at least twice, rather than solving row labeling over all document rows.Theorem 3 states that this subsequence contains a full record when a true field was correctly inferred and occurs once in every record.Source: §5.1.3 · p. 13
  • If a correctly inferred true field appears exactly once in every record, the pruned row subsequence R′ contains at least one complete record.Theorem 3 gives this guarantee for the smallest consecutive subsequence in which each inferred field appears at least twice.Source: §5.1.3 · p. 13, Figure 8 · p. 14

5.2 Template Inference

  • TWIX assembles labeled rows into a template tree by creating table nodes from Key rows, merging consecutive Key-Value rows, and assigning Value rows to the closest preceding aligned Key row.Nodes with the same type and fields are not inserted more than once.Source: §5.2 · p. 14

6 Data Extraction

TWIX uses its inferred template to divide documents into records and data blocks, then extracts table and key-value data while preserving structural relationships.

6 Data Extraction

  • Template-guided extraction separates each document into records, identifies data blocks within each record, and extracts table and key-value contents.The pipeline also assembles extracted data into a tree and preserves metadata locations for later use.Source: §6 · p. 15

6.1 Record Separation

  • A record is the smallest block that visits every template node at least once in pre-order traversal.A node may be visited multiple times, as can occur with nested structures.Source: §6.1 · p. 15

6.2 Block Separation

  • Block separation labels rows as table keys, key-value rows, or values, then assigns value rows to aligned preceding keys and merges consecutive key-value rows.Unmatched value rows become metadata; aligned rows may attach to a preceding non-leaf key when the closest key is not aligned.Source: §6.2 · pp. 15–16

6.3 Data Extraction

  • For table blocks, vertically aligned phrases supply field values, and a field with no aligned phrase is recorded as missing.For a key-value block, consecutive phrase pairs are scanned against the inferred fields to identify values or missing values.Source: §6.3 · p. 16

12 Return 𝐾𝑉

TWIX extracts block contents into structured results and preserves metadata so users can interpret its spatial relationship to extracted data.

12 Return 𝐾𝑉

  • TWIX stores each metadata phrase’s bounding box and page number, preserving its spatial association with extracted data.Users can use the alignment to transform metadata into table fields, but TWIX leaves that interpretation to them.Source: §12 · p. 17
  • In the example, the preserved alignment suggests that “Complaint #1” corresponds to the first extracted table row, enabling a user to add a Complaint column with value 1.The paper also states that end-to-end correctness is established for a commonly observed class called compliant documents.Source: §12 · p. 17

7 Experimental Evaluation

TWIX is evaluated on 64 real-world templatized document collections, with separate benchmarks for extraction quality and scalability. It outperforms the compared tools on extraction quality and scales to large collections with low latency and cost.

7.2 Experimental Results

  • TWIX exceeds the best baseline by 31% in S-Precision and 22% in S-Recall, and achieves 80% S-Precision and 76% S-Recall on Hard datasets.It also reports higher precision and recall than the baselines across Medium and Hard datasets; Table 2 reports precision/recall and Table 3 reports structure metrics.Source: Table 3 · p. 19, Table 2 · p. 19, §7.2 · p. 20
  • TWIX takes around 5 seconds per Q-Benchmark dataset, making it 2× faster than AzureDI, 4× faster than Textract, 7× faster than Evaporate, and 28.6× faster than vision-based approaches.Figure 10 presents end-to-end latency on the left and cost on the right.Source: §7.2 · p. 20, Figure 10 · p. 20
  • TWIX processes an S-Benchmark dataset in around 4 minutes for $0.014, while vision-based LLMs take over 30 hours and cost over $50 for around 2000+ pages.The text reports TWIX as 12× to 174× faster and 1500× to 2423× cheaper than baselines other than Eva-C; Table 4 presents latency and cost comparisons.Source: §7.2 · p. 21, Table 4 · p. 21
  • On the smaller Q-Benchmark, template inference consumes around 85% of TWIX’s time; on the larger S-Benchmark, phrase extraction becomes the main latency component as document volume grows.Figure 11 breaks down the percentage of latency by component; TWIX’s template-inference costs are incurred during that stage.Source: §7.2 · p. 21, Figure 11 · p. 22
  • In Easy datasets, inferred fields have around 0.95 precision and 0.98 recall; on Medium and Hard datasets, precision falls to around 0.86 while recall remains close to 0.95.Figure 12 reports field-inference precision and recall.Source: §7.2 · p. 21, Figure 12 · p. 22
  • Template-structure inference corrects some field errors by rejecting metadata that violates row-alignment constraints and false-positive fields in rows labeled as values.The authors describe these corrections as evidence of robustness in field inference.Source: §7.2 · p. 22
  • Compared tools exhibit six recurring error types, including column misalignment, header or data-block misidentification, OCR phrase errors, metadata misclassification, and field-inference errors.Table 5 summarizes frequent and occasional error patterns across tools.Source: §7.2 · p. 22, Table 5 · p. 23
  • TWIX’s results remain bounded by OCR and bounding-box imperfections, and phrases spanning multiple table rows may require merging with visual cues or semantic checks.The authors describe semantic correction and improved OCR as outside TWIX’s scope.Source: §7.2 · p. 23

8 Related Work

Prior document extraction approaches use pretrained tools, user-specified fields, or LLM-generated scripts, while web extraction often relies on HTML structure. TWIX targets visually complex document layouts by inferring templates and preserving relationships among extracted fields.

Document Data Extraction

  • General-purpose document tools extract tables or entities, but the paper characterizes LLM-generated extraction scripts as heuristic and not robust for reliable extraction.The related-work discussion includes Evaporate [16] and industrial tools.Source: §8 · p. 23
  • Learning-based extraction requires human-labeled fields and does not capture relationships among values, whereas TWIX’s unsupervised template inference preserves those relationships.Parthasarathy et al. [47] use landmarks to narrow regions for user-specified fields.Source: §8 · p. 23

Web Data Extraction

  • Web extraction often uses HTML or XML tags to represent structure, but such tags are often unavailable in PDFs and similar documents.Prior approaches also extract web tables or logs, while TWIX handles nested combinations of tables and key-value structures.Source: §8 · p. 24

Document Layout Analysis

  • Document layout analysis detects coarse components but struggles with fine-grained mixtures of key-value and table blocks; combining it with TWIX is proposed as future work.The proposed combination could first locate structured portions in long documents containing text or images.Source: §8 · p. 24

9 Conclusion

TWIX infers flexible visual templates to organize templatized documents into nested records and data blocks, then extracts structured data from those blocks. Experiments show that it outperforms baselines across accuracy, latency, and cost.

9 Conclusion

  • TWIX infers flexible visual templates that separate documents into nested records and data blocks, enabling extraction from each block.It combines optimization for template discovery with targeted use of LLM semantic knowledge.Source: §9 · p. 24
  • The paper demonstrates hardness for the underlying problems while providing correctness guarantees.Source: §9 · p. 24
  • Experiments show that TWIX outperforms baselines across accuracy, latency, and cost.Source: §9 · p. 24

A.1 Proof of Proposition 1

Proposition 1 relates matching phrase locations to whether two fields share a label in the true template. Equal location vectors imply a perfect match, while unequal vectors imply a partial perfect match.

A.1 Proof of Proposition 1

  • Equal location vectors for two fields in the unique table node imply that the fields are a perfect match.The proof uses the assumption that each phrase has a unique label and the table schema is consistent across records.Source: §A.1 · p. 28
  • When the location vectors differ, the proof establishes a partial perfect match between the fields.Source: §A.1 · p. 28
  • For equal labels, the proof derives consistent relative positions for the two phrases across the table's blocks.Source: §A.1 · p. 28

A.2 Proof of Proposition 2

The material covers proof conditions for field matching and extraction correctness, the hardness of row labeling, and the representation and evaluation of extracted structures, while also specifying LLM-comparison prompts. It notes that the supplied passages do not state Proposition 2’s proof and include manuscript dates.

A.2 Proof of Proposition 2

  • For a key-value node, equal-length consistently present fields are perfect matches; when one field has more locations, its matching subsequence is a partial perfect match.The proof relies on consistent field ordering across blocks generated from the same template node.Source: §A.2 · p. 28

A.3 Proof of Theorem 1

  • Taking logarithms preserves the objective’s ordering, and the one-label-per-row constraint yields the linear objective in Eq. (4).The proof first transforms the objective using the monotonicity of logarithms, then uses the row-label constraint to linearize it.Source: §A.3 · pp. 28–29

A.4 Proof of Theorem 2

  • A reduction from Minimum Vertex Cover proves row labeling is NP-hard: a graph has a vertex cover of size at most k exactly when the constructed instance admits a feasible labeling meeting its threshold.The construction forces edge rows to label V and uses K-labeled vertex rows to encode a cover.Source: §A.4 · pp. 29–30

A.5 Proof of Theorem 3

  • If a correctly inferred field appears exactly once in every record and the candidate rows contain every field at least twice, the consecutive row list includes a complete record.This is the condition under which Theorem 3 guarantees at least one record created from the true template.Source: §A.5 · p. 30

A.6 Representation of Data Extraction Results

  • TWIX’s extraction objects form a tree of data blocks, with each object storing fields, type, and content; parent blocks overlap and precede their children, while siblings are disjoint.Table content stores headers and tuples, while key-value content stores pairs.Source: §A.6 · pp. 30–31

A.7 Correctness Analysis of TWIX in Compliant Documents

  • For compliant documents with correctly extracted phrases and an correctly inferred template, TWIX’s extraction objects are correct when metadata rows are not aligned with key rows; 91% of documents across 34 datasets are compliant.Removing 3% of non-compliant phrases makes the remaining cases compliant.Source: §A.7 · p. 32

A.8.1 Example of Structure Evaluation.

  • Structure evaluation matches tuples by minimum distance, preserves their order, flattens nested tables, and full-outer-joins data blocks while retaining unmatched fields as NULLs.Key-value blocks are evaluated as one-row tables, and precision and recall are reported over concatenated true and predicted tables.Source: §A.8.1 · pp. 32–33

A.8.2 Table Match Algorithm.

  • The table-matching algorithm uses tuple Jaccard distance and dynamic programming to find a minimum-cost order-preserving alignment, then backtracks to recover matched tuples.It considers earlier predicted tuples when computing the minimum cost for each alignment.Source: §A.8.2 · p. 33

A.9 Prompts of vision-based LLMs

  • A prompt fragment shows a key-value pair followed by an ellipsis indicating additional pairs.The fragment appears in the subsection on prompts for vision-based LLMs.Source: §A.9 · p. 36
  • The manuscript was received in April 2025, revised in July 2025, and accepted in August 2025.Source: §A.9 · p. 36
Loading 2501.06659v2…