Source-linked AI summary
Document Understanding Dataset and Evaluation (DUDE)
Jordy Van Landeghem, Rubén Tito, Łukasz Borchmann, Michał Pietruszka, Paweł Józiak, Rafał Powalski, Dawid Jurkiewicz, Mickaël Coustaty, Bertrand Ackaert, Ernest Valveny, Matthew Blaschko, Sien Moens, Tomasz Stanisławek
TL;DR
Document AI benchmarks often use simplified or narrowly scoped datasets that do not capture real-world document complexity, leaving generalization across domains, layouts, and tasks insufficiently evaluated. DUDE introduces a multi-page, multi-domain, multi-industry DocVQA benchmark with diverse question types and layout-navigation challenges, and finds that current models still lag far behind human performance. The benchmark therefore exposes the need for more effective joint modeling of language, vision, and document layout, while remaining limited to English documents and a relatively small sample of the world's documents.
Problem
Existing Document Understanding benchmarks do not simultaneously address multi-page documents, visual layouts, diverse document types, and cross-domain generalization.
Method
DUDE constructs a large-scale multi-page, multi-domain, multi-industry DocVQA benchmark with diverse questions, layout-navigation tasks, and multi-stage annotation.
Results
Current state-of-the-art textual and multimodal models still lag behind human performance on DUDE, with visual and layout understanding remaining difficult.
Takeaways & Limitations
DUDE provides a more practical evaluation setting for studying generalization across document types, domains, layouts, and question-answer formats.
Takeaways & Limitations
The benchmark contains only English-language documents and remains a relatively small sample of the documents found in the real world.
Abstract
from arXiv · showhide
We call on the Document AI (DocAI) community to reevaluate current methodologies and embrace the challenge of creating more practically-oriented benchmarks. Document Understanding Dataset and Evaluation (DUDE) seeks to remediate the halted research progress in understanding visually-rich documents (VRDs). We present a new dataset with novelties related to types of questions, answers, and document layouts based on multi-industry, multi-domain, and multi-page VRDs of various origins, and dates. Moreover, we are pushing the boundaries of current methods by creating multi-task and multi-domain evaluation setups that more accurately simulate real-world situations where powerful generalization and adaptation under low-resource settings are desired. DUDE aims to set a new standard as a more practical, long-standing benchmark for the community, and we hope that it will lead to future extensions and contributions that address real-world challenges. Finally, our work illustrates the importance of finding more efficient ways to model language, images, and layout in DocAI.
1. Introduction
DUDE addresses the gap between simplified document datasets and real-world document complexity by introducing a multi-page, multi-domain DocVQA benchmark. It evaluates generalization across document types, layouts, and domains while showing that current models remain far behind human performance.
- Simplified datasets often fail to reflect real-world document complexity, limiting the practical suitability of resulting models.
- Document Understanding still faces unresolved domain generalization, task adaptation, and long-tailed data challenges across document types.
- The benchmark tests whether systems can navigate visual layouts and generalize across document types and domains.
- DUDE introduces a multi-page, multi-domain, multi-industry DocVQA benchmark for evaluating document understanding under realistic conditions.
- Current state-of-the-art models lag far behind human baselines, highlighting the need for more holistic language, vision, and layout modeling.
2. Related Work
Prior document VQA benchmarks commonly focus on single-page, extractive, domain-specific settings and omit non-answerable or list-answer questions. DUDE builds on this landscape by targeting broader question types and document conditions.
- Document Understanding benchmarks span layout analysis, classification, information extraction, table extraction, and visual question answering, but are often task-specific and domain-limited.
- Prior document VQA research has mainly used single-page documents, with MP-DocVQA as a rare multi-page exception.
- Existing datasets commonly emphasize extractive questions and generally omit non-answerable questions.
- No prior VQA dataset was known to include questions requiring list answers, although some text-only QA datasets supported that answer type.
- Many related datasets are domain-specific, limiting evaluation across diverse document topics and settings.
3. DUDE Dataset
DUDE is designed as a diverse, practical DocVQA benchmark spanning document types, domains, layouts, question forms, and evaluation needs. Its dataset and annotation choices support fine-grained analysis of model behavior in realistic document-understanding settings.
- Documents: DUDE covers diverse document types, sources, dates, domains, and layouts to evaluate models on visually varied documents.The dataset includes medical, legal, technical, and financial material with varied text arrangements, font sizes, and styles.
- Questions: DUDE includes compositional, arithmetic, comparison, and multi-hop questions that require semantic understanding and sequential reasoning across document content.Its multi-hop examples connect information across pages, while arithmetic and comparison questions target numerical and date reasoning.
- Answers: The dataset combines extractive and abstractive answers, including text, numbers, dates, yes/no responses, lists, and no-answer cases.This answer design extends beyond plain extractive text and includes questions whose answers are not entailed by the document.
- Annotation Process: DUDE uses staged annotation involving crowdworkers and qualified linguists, with non-answerable questions predominantly handled by linguists.The test set can receive an additional qualified-linguist review that corrects questions and answers and adds diagnostic metadata.
- Evaluation: The benchmark adds diagnostic metadata, ANLS evaluation, calibration, and confidence-ranking measures to support detailed and practically oriented model assessment.AURC combines estimator performance with confidence ranking and evaluates selective prediction when systems can abstain and defer to humans.
- Analysis & Discussion: Current models remain well below the human baseline, with the reported best result reaching 46.04 ANLSall versus 74.76 human ANLS points.The paper also reports that extending context from 512 to 8192 tokens increases T5 and T5+2D scores by 4.4–5.0 points.
4. Conclusion
DUDE is presented as a large-scale benchmark for real-world document understanding, covering long and varied documents while exposing substantial gaps between models and human performance. Its scope is constrained by English-only documents and a still-limited sample of the broader document universe.
- Figure 5 compares average ANLS for human experts with the best-performing model in each diagnostic category as a ceiling analysis.
- The benchmark evaluates baseline performance on the full test set and a diagnostic subset using zero-shot, few-shot, and fine-tuned setups, including Max Conf. and Concat inference.
- DUDE benchmarks document understanding across multi-page, multi-domain, and multi-industry documents with visual semantics such as tables, charts, figures, lists, checkboxes, and stamps.
- State-of-the-art textual and multimodal models still lag behind human performance, indicating a need for improved visual understanding in document understanding systems.
- The benchmark uses only English-language documents and, despite being large-scale, represents a relatively small sample of the documents found in the real world.
Supplementary Materials
DUDE exposes distinct weaknesses in current document models: encoder-only systems struggle with abstractive answers, while visual reasoning and long-document processing remain substantial challenges.
- A.1.1 Encoder vs. Decoder: <10 ANLS% on abstractive questions was reported for BigBird, Longformer, BERT, and LayoutLMv3, despite average extractive-question scores.These results support the need for generative models on DUDE’s considered question types.
- A.1.1 Encoder vs. Decoder: Humans scored ANLS ±82% on abstractive questions versus ±68% on extractive questions, while machine baselines showed the reverse pattern.Answer-format differences may confound ANLS comparisons between extractive and abstractive answers.
- A.1.2 Incorporating Layout & Vision: T5 with 2D position embeddings improved most on ‘evidence table or list’, ‘complexity simple’, and ‘evidence plain’ diagnostic categories.
- A.1.2 Incorporating Layout & Vision: 37% ANLS was achieved by both GPT3 (4-shot) and T5-2d-large-8K on visual categories, compared with 72% for humans.The gap indicates that text access alone does not provide the visual evidence required by these questions.
- A.1.3 Toward Long Document Processing: 1832±2545 tokens is DUDE’s average document length, and extending T5 sequence length from 512 to 8192 produced a ∼5% ANLS improvement.BERT-concat underperformed Longformer and BigBird, reinforcing the importance of long-sequence processing.
- A.1.3 Toward Long Document Processing: HiVT5’s compressed question-conditioned document representation and tuned 10 [PAGE] tokens per page suggest hierarchical processing as a promising direction for visually rich documents.DUDE may require better fine-tuning of [PAGE] tokens because its images contain richer colored graphics and layouts.
A.1.4 Diagnosis of LLM Results
LLM baselines show selective strengths on lists, unanswerable questions, and multi-hop questions, but substantial weaknesses on abstraction, visual understanding, and arithmetic-like operations.
- Strengths: GPT3 achieved ANLS=36-40% on list-type questions, while ChatGPT (4-shot) reached ANLS=77.45% on not-answerable questions.
- Strengths: GPT3 (4-shot) outperformed other tested baselines on ‘complexity multi-hop’ questions with ANLS=52.51%.
- Weaknesses: LLMs scored ANLS=22% on abstractive questions, about half the T5-base-512 score of ANLS=47%.Even after 4-shot fine-tuning, LLMs predicted ‘Not-answerable’ for 55% of abstractive questions, versus 10% in reality.
- Weaknesses: Arithmetic, counting, and comparison operations remained below 25% ANLS for the tested LLMs.
- Weaknesses: Both tested LLMs averaged ANLS=21% on questions requiring visual understanding, substantially below the human baseline.The models were text-only, limiting access to the visual evidence needed by these questions.
- Evaluation caveat: Zero-shot performance may be inflated because DUDE’s public-license web documents could have appeared in the LLMs’ pre-training corpora.
A.2. Assessing Confidence
Confidence evaluation distinguishes calibration from performance-confidence ranking and reveals trade-offs among sequence length, 2D embeddings, and page aggregation strategies.
- Confidence metrics: ECE measures confidence calibration, whereas AURC combines performance with confidence ranking for selecting models in real-world applications.Low-confidence answers can be manually reviewed to help prevent undesired outcomes from wrong predictions.
- Calibration results: T5-base-512 achieved ECE=10.82, while T5-2D-large-8K had the highest ANLS but worse calibration at ECE=14.4.
- Calibration results: Extending maximum sequence length generally worsened calibration, whereas adding 2D position embeddings improved ANLS.
- Calibration results: T5-2D-large-8K achieved the highest AURC among the tested baselines.
- Aggregation strategies: Max Conf. produced poor overall calibration compared with Concat, while ANLS did not consistently favor either aggregation strategy.
B. Baseline Experiments Setup
The benchmark evaluates encoder, decoder, and GPT-3.5-based systems with task-specific prompting, confidence estimation, and metrics adapted to diverse document answers. Its evaluation uses ANLS for answer similarity, ECE for calibration, and AURC for confidence-risk trade-offs.
- Prompted LLM evaluation: GPT-3.5 models are evaluated in 0-shot and 4-shot settings, with four demonstration question types in the latter.The 4-shot prompt includes extractive, abstractive, list, and not-answerable examples from one training document.
- Prompted LLM evaluation: Temperature is set to 0.0 to make GPT-3.5 outputs more deterministic and focused.
- Confidence estimation: Confidence is defined as the top-1 predicted probability, typically the largest softmax-normalized logit value.
- Evaluation metrics: ANLS extends normalized Levenshtein similarity to lists, multiple valid references, and not-answerable questions, while accommodating OCR and formatting variation.For list questions, Hungarian matching pairs ground-truth and predicted answer parts; the threshold is τ = 0.5.
- Evaluation metrics: ECE measures the Lp-norm difference between predicted posterior confidence and the true likelihood of correctness.In DUDE, correctness is based on whether ANLS exceeds the threshold τ.
- Evaluation metrics: AURC measures the trade-off between coverage and error risk by sorting predictions according to confidence.With ANLS thresholding, the best possible AURC is constrained by test error and test-set size.
C. Qualitative Examples
The qualitative examples span arithmetic, visual, handwriting, counting, map-based multi-hop, extractive, and abstractive questions. They show that answer quality varies substantially across systems and question demands, with some models returning incorrect or not-answerable responses.
- Qualitative examples: Arithmetic questions require extracting relevant table values, dividing them, and correcting subject–verb agreement.
- Qualitative examples: A visual chart question combines chart and line-color recognition, page navigation, and numerical comparison.
- Qualitative examples: A handwriting question requires recognizing a handwritten date and locating the relevant page.The human answer matches the ground truth exactly with ANLS 1.0.
- Qualitative examples: The map-based multi-hop question requires visual map comprehension and linking legend information to depicted regions.The ground truth is ID | SD | KS, while several model outputs are incorrect or not answerable.
D.1. Answer Types
DUDE contains diverse answer types that are only weakly associated with question types. Aside from expected pairings such as “None” with not-answerable questions, answer categories remain broadly distributed across question forms.
- Answer-type correlations: Date and duration answers are equally likely for extractive and abstractive questions.
- Answer-type correlations: Cramer’s V values below 0.1 indicate little correlation between answer types and question types.
D.2. Dataset Diversity
DUDE is analyzed for diversity across question types, answer distributions, and document visual embeddings. Its answer-type inventory is larger than those reported for several related DocVQA datasets.
- Answer and visual diversity: The visual-embedding comparison relates DUDE’s first-page document diversity to that of other DocVQA datasets.
- Answer and visual diversity: Figure 7 reports the distribution of answer types for each question type in DUDE.
- Answer and visual diversity: DUDE contains 24 answer types, compared with 13 in TAT-DQA, 20 in InfographicsVQA and SP-DocVQA, and 23 in VisualMRC.
- Answer and visual diversity: Figure 9 visualizes first-page document-image similarities across datasets using t-SNE over ResNet101 features from 1,000 documents.