VRDU: A Benchmark for Visually-rich Document Understanding

1 INTRODUCTION

• Existing benchmarks can miss practical challenges including diverse data types, complex target schemas, hierarchical entities, unseen templates, and small training sets.Layout information is critical to understanding visually rich documents, unlike plain-text documents.Source: §1 · pp. 1–2
• VRDU collects FCC political ad-buy forms and FARA registration forms to create two datasets intended to reflect practical challenges and reduce extraneous evaluation factors.The paper describes the collected and annotated data in a later section.Source: §1 · p. 2
• The benchmark defines three increasingly difficult template-learning tasks, varies training-sample counts, and evaluates entities with type-aware matching.The matching functions account for entity type rather than relying only on string equality.Source: §1 · p. 2
• Baseline models perform worse on unseen layouts than on seen-template tasks, few-shot extraction remains difficult, and hierarchical or repeated entities are harder than simple fields.The authors describe these as areas of opportunity across commonly used baselines.Source: §1 · p. 2
• VRDU contributes benchmark desiderata, datasets with high-quality OCR and annotations, application-oriented tasks, and an evaluation toolkit.The toolkit uses a type-aware matching algorithm.Source: §1 · p. 2

2 BENCHMARK DESIDERATA

• A useful benchmark should include rich schema, layout-rich documents, diverse templates, high-quality OCR, and token-level annotation.These five desiderata are intended to measure progress on structured extraction from visually rich documents in real applications.Source: §2 · p. 3

2.1 Rich Schema

• Practical document schemas contain varied data types and required, optional, repeated, or hierarchical fields, unlike datasets that treat entities as simple strings.A US address, for example, can combine address lines, city, state, and ZIP code.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Layout-rich Documents

• Documents should include complex layout elements such as tables, key-value pairs, multiple columns, varying fonts, images, captions, and footnotes.These features distinguish visually rich documents from pages organized mainly as prose.Source: §2.2 · p. 3, Figure 3 · p. 3

2.3 Diverse Templates

• A benchmark should include diverse templates and test generalization to unseen layouts rather than permit extraction by memorizing one template.The paper illustrates the practical need with invoices from new vendors that may use different templates.Source: §2.3 · p. 3, Figure 4 · p. 4

2.4 High-quality OCR Results

• Using the same high-quality OCR engine helps isolate document-understanding performance from variation in OCR quality.Different engines across benchmarks make comparisons less consistent and can obscure whether gains come from models or OCR.Source: §2.4 · pp. 3–4

2.5 Token-level Annotation

• Benchmarks should provide entity token spans because matching entity text to document tokens can be ambiguous and sequence-labeling models require token-level annotations.Repeated text values may appear in a document even when only one occurrence is the target entity.Source: §2.5 · p. 4, Figure 5 · p. 4

3 RELATED WORK

• Existing benchmarks have mixed limitations: FUNSD and SROIE have relatively simple schemas or limited templates, while CORD has limited template diversity and OCR artifacts.Kleister-NDA documents are mostly plain-text paragraphs and chapters with few layout elements.Source: §3 · p. 4
• Kleister-Charity and DeepForm include layout-rich documents and multiple templates but lack token-level annotations and hierarchical or repeated fields.VRDU's Ad-buy Forms use the same source documents as DeepForm and add bounding boxes and token-level annotations.Source: §3 · pp. 4–5
• VRDU's Ad-buy Forms provide hierarchical entity annotations, a practical structural extraction task not explored in the other benchmarks described.The datasets also include rich schemas, layout-rich documents, diverse templates, high-quality OCR, and token-level annotations.Source: §3 · p. 5

4 VRDU BENCHMARK

• VRDU comprises Ad-buy Forms and Registration Forms with rich schemas, hierarchical repeated fields, complex layouts, multiple templates, high-quality OCR, and token-level ground-truth annotations.The section introduces dataset collection and annotation, three extraction tasks with prescribed splits, and a type-aware matching algorithm for evaluation.Source: §4 · p. 5

4.1 Data Collection

• The authors collected public FCC ad-buy forms and FARA registration forms because privacy limits the suitability of sensitive in-house documents for public research.They rebuilt the Ad-buy Forms dataset from the DeepForm source, annotating documents from scratch.Source: §4.1 · p. 5

4.1.1 Ad-buy Forms.

• The Ad-buy Forms dataset contains 641 political-advertisement invoices or receipts with tables, multi-column layouts, key-value pairs, and detailed advertisement information.The documents are signed between TV stations and campaign groups.Source: §4.1.1 · p. 5

4.1.2 Registration Forms.

• The Registration Forms dataset contains 1,915 foreign-agent registration documents using three templates: Amendment, Short Form, and Dissemination Report.The forms share a topic and use the same schema.Source: §4.1.2 · p. 5

4.2.1 Labeling Schema.

• VRDU uses six unrepeated entity names for Registration Forms and nine unrepeated names plus one hierarchical repeated entity for Ad-buy Forms.A line_item groups repeated components such as description, start/end dates, and sub-price for a TV program.Source: §4.2.1 · p. 6

4.2.2 Labeling Team.

• The annotation team comprised 30 annotators and three experts, who reviewed annotator labels; estimated annotation time was six minutes per document.The authors report typical times of 6–8 minutes for a single-page document with fewer than 20 fields and 10–30 minutes for a multi-page document with 25 fields.Source: §4.2.2 · p. 6

4.2.3 Labeling Protocol.

• Annotators marked entity bounding boxes and categories, which the system used with OCR to create token-level labels; experts reviewed and corrected the first-pass results.For hierarchical labels, annotators marked component entities and a larger box grouping them; when unrepeated entities appeared multiple times, all instances were labeled but evaluation required only one extraction.Source: §4.2.3 · p. 6
• The protocol targets common annotation errors such as confusing flight dates with invoice periods and omitting the final line of multi-line addresses.Previous annotated documents served as references, and an expert checked the results.Source: §4.2.4 · p. 6

4.3.1 Task 1: Single Template Learning (STL).

• Single Template Learning places training, validation, and test documents within one template to assess performance on a fixed layout.The authors expect very high F1 scores in this simplest setting.Source: §4.3.1 · p. 6
• Mixed Template Learning draws training, validation, and test documents from the same set of templates while preserving each template’s distribution across splits.Source: §4.3.2 · pp. 6–7

4.3.2 Task 2: Mixed Template Learning (MTL).

• Unseen Template Learning tests generalization across disjoint training, validation, and test template sets, such as training on two Registration Form templates and testing on the third.The authors describe this as a setting not explicitly provided by previous benchmarks and datasets.Source: §4.3.3 · p. 7

4.3.3 Task 3: Unseen Template Learning (UTL).

• Each task uses a 300-document test set and training sets of 10, 50, 100, or 200 samples to assess data efficiency.The prescribed splits are published for apples-to-apples comparisons.Source: §4.3.4 · p. 7

• Type-aware fuzzy matching applies entity-specific comparison functions and reports macro and micro F1 scores.Source: §4.4 · p. 7
• The entity-grouping procedure selects designated component types and groups their ordered occurrences into hierarchical entities.It tracks component types while traversing the entity list and starts a new group when a type repeats.Source: §4.4 · p. 7
• For entity-specific matching, the method converts price values to numeric form before comparison and parses date strings, using a standard date-equality function to handle different formats.Source: §4.4 · p. 7

4.5 Post-processing for Evaluation Toolkit

• Unrepeated entity names require one value; when models lack confidence scores, the toolkit keeps the first extracted entity.For example, a series ID should have one value rather than several candidates.Source: §4.5 · p. 7
• Hierarchical entities group related individual entities into spans, with a split when an entity type appears for the second time.The toolkit orders extracted entities by OCR reading order before applying this heuristic; for [A, B, C, B, C], the split is at the second B.Source: §4.5 · p. 7

5 EXPERIMENTS

• The experiments assess whether baseline models can handle VRDU's practical extraction challenges; comparing model architectures comprehensively is explicitly outside the paper's goal.Three random-seed training sets are averaged for each setting; Ad-buy Form has no Single Template Learning task because it has many templates and few documents per template.Source: §5 · p. 7, §5.2 · p. 8

5.1 Baselines

• The evaluated baselines are LayoutLM, LayoutLMv2, LayoutLMv3, and FormNet, which incorporate different combinations of text, layout, visual, and graph-based features.The paper uses these commonly used models to highlight shared challenges, not to provide a comprehensive architecture comparison.Source: §5.1 · pp. 7–8

5.2 Experiment Results

• At |D| = 200, the highest micro-F1 is around 90% on Registration Form and around 45% on Ad-buy Form; CORD and SROIE scores are higher.The passage reports 97.21% for FormNet and 96.01% for LayoutLMv2 on CORD, and 97.81% for LayoutLMv2 on SROIE.Source: Table 4 · p. 8, §5.2 · p. 8
• FormNet's micro-F1 on Registration Form in the simplest STL setting is 15.16 points lower at |D| = 10 than at |D| = 50; this gap persists across tasks and models.Even in that simple setting, few-shot performance reaches only 74.22% micro-F1.Source: §5.2 · p. 8
• At 200 training documents, UTL micro-F1 is 13–17 percentage points below MTL across three models, while STL and MTL exceed 80% micro-F1 and macro-F1 on both datasets.The authors attribute the STL and MTL performance to testing layouts being represented in training; UTL tests unseen templates.Source: §5.2 · p. 8

5.3 Performance on Hierarchical Entities

• For FormNet on Ad-buy Form, hierarchical-entity micro-F1 trails other entities by 60–70 points across training sizes, and the authors attribute the low score to heuristic grouping.The results also report much higher macro-F1 than micro-F1 on Ad-buy Form because hierarchical repeated entities have very low F1.Source: §5.2 · pp. 8–9, §5.3 · p. 9, Figure 7 · p. 9

5.4 Case Study

• Selected FormNet errors include incomplete address and handwritten-name extraction, attributed respectively to a hidden P.O. Box and difficulty grouping differently sized handwritten characters.Figure 6 marks ground truth in green and model extractions in red; the selected cases also include Ad-buy Form keyword-confusion and unseen-template errors.Source: §5.4.1 · p. 9, Figure 6 · p. 9, §5.4 · p. 9

5.4.1 Incomplete Extraction.

• In another case, similar keywords led the model to extract Invoice ID instead of Order ID, while an unseen template's use of “Station” led it to miss Property.The authors suggest using the training-set association between “WBTW” and Property to address the rare unseen-template case.Source: §5.4.2 · p. 9

6 CONCLUSIONS AND FUTURE STUDY

• VRDU responds to five benchmark desiderata with two datasets, application-oriented tasks, high-quality OCR and annotations, and a type-aware evaluation toolkit.The authors make both datasets, all train/validation/test splits, and the toolkit publicly available.Source: §6 · p. 9
• Future work will evaluate models incorporating multimodal features and explore approaches for unknown templates, limited training samples, and hierarchical entities.The authors identify template generalization, few-shot extraction, and complex hierarchical-repeated fields as areas of opportunity.Source: §6 · pp. 9–10