Source-linked AI summary
Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts
Tomasz Stanisławek, Filip Graliński, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, Przemysław Biecek
TL;DR
Key Information Extraction lacks well-defined benchmarks reflecting real-world business documents, complicating robust evaluation. Kleister introduces two datasets of long formal documents and evaluates existing KIE systems on them. The strongest baseline scores 81.77/83.57 F1 on NDA/Charity, below the cited SROIE result.
Problem
Public KIE benchmarks cover relatively few well-defined problems, and their disparity from real-world business needs makes robust evaluation difficult.
Source: §1 · pp. 1–2Method
The paper introduces two datasets of long, complex-layout business documents and evaluates KIE architectures using textual and structural layout features.
Source: §1 · pp. 2–3Results
81.77/83.57 F1-score on Kleister NDA/Charity, respectively, is below the cited 98.17 SROIE result and shows weakness in current state-of-the-art KIE models.
Source: §6 · pp. 13–14Takeaways & Limitations
The datasets are intended to help investigate how document length, complex layouts, and OCR quality affect KIE performance.
Source: §6 · p. 13For Charity income and spending, about 5% of examples require inferring monetary scale from document context, which sequence-labeling-only models do not handle competitively.
Source: §5 · p. 12
Abstract
from arXiv · showhide
The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.
1 Introduction
Public KIE datasets do not yet reflect the diversity and complexity of real-world business documents, making robust evaluation difficult. Kleister introduces two long-document datasets and evaluates existing KIE systems on them.
1 Introduction
- Existing public KIE datasets differ from real-world business needs, leaving robust evaluation difficult.Prior datasets address areas such as scanned receipts, form understanding, and visual question answering.Source: §1 · p. 2
- The datasets include challenges from complex layouts, business logic, OCR quality, long documents, noisy training data, and normalization.The paper also evaluates KIE systems and analyzes business-domain challenges.Source: §1 · p. 2
- Kleister introduces two datasets of long documents with complex layouts, totaling 3,328 documents, 64,872 pages, and 23,772 entities.The datasets address real-life business problems using diverse scanned and born-digital documents.Source: §1 · p. 2
- The study evaluates state-of-the-art NER architectures, including Flair, BERT, RoBERTa, LayoutLM, and LAMBERT.It also presents a semi-supervised dataset-collection method intended to reduce manual work and potentially be reused for similar tasks.Source: §1 · p. 3
2 Related Work
Related work spans visually rich documents and one-dimensional text, but prior datasets differ in the extraction tasks and document contexts they cover. The paper distinguishes KIE from NER and discusses datasets ranging from receipts and forms to invoices and Wikipedia articles.
2 Related Work
- KIE research addresses business-document challenges including complex layouts, business logic, OCR quality, document-level extraction, and normalization.The authors identify these challenges as motivation for preparing a new dataset.Source: §2 · p. 3
2.1 KIE from Visually Rich Documents (publicly available)
- SROIE covers scanned receipts, while FUNSD focuses on structuring form text rather than extracting particular entities from varied layouts.SROIE includes annotated company name, date, address, and total amount paid.Source: §2.1 · p. 3
2.2 KIE from Visually Rich Documents (publicly unavailable)
- Invoice datasets extract fields such as invoice date, invoice number, net amount, and vendor name using NLP and computer vision.Their documents are usually short, so extended document context is less often needed.Source: §2.2 · p. 3
2.3 Information Extraction from one-dimensional documents
- WikiReading predicts textual values from Wikidata by reading corresponding Wikipedia articles, with some values extracted and others inferred.The task is suited to end-to-end models handling longer documents.Source: §2.3 · p. 4
- KIE targets specific actionable data points and does not require span retrieval, unlike general-entity NER.The passage uses the CoNLL 2003 NER challenge as an example.Source: §2.3 · p. 4
3 Kleister: New Datasets
Kleister introduces datasets of long formal US non-disclosure agreements and UK charity annual reports, including born-digital and mostly scanned documents. The section describes their collection and annotation procedures and the varied entity and document-length challenges they represent.
3 Kleister: New Datasets
- Kleister comprises long formal US NDAs and UK charity annual reports, mixing born-digital and mostly scanned documents with PDF/text inputs and entity lists.The NDA documents are born-digital, while the Charity reports needed OCR.Source: §3 · pp. 4–5
3.1 NDA Dataset
- NDA annotation used similarity-selected contexts for 315 documents and full manual annotation for 195; the approach was about 50% quicker than fully manual annotation.A super-annotator checked all documents, with Cohen’s κ = 0.971.Source: §3.1 · p. 5
3.2 Charity Dataset
- Charity labels came from Commission website descriptions only for entities also present in reports; filtering removed non-English, low-quality, and multi-organization documents.Address information was split into postal code, postal town, and street or road name; dev/test documents were manually annotated, with κ = 0.9 on duplicate annotations of 100 test documents.Source: §3.2 · pp. 6–7
3.3 Statistics and Analysis
- Entity presence and multiplicity vary: NDA term appears in 36% of documents and party can have seven gold values; document lengths and entity occurrences also create long-context extraction challenges.Five of twelve entities appear once in a document, while party, charity_number, and report_date each average more than ten mentions; entity spans may recur in different contexts, and jurisdiction may require canonicalization.Source: §3.3 · p. 8, Manual Annotation of Text Spans · p. 8, Comparison with Existing Resources · p. 9
4 Experiments
The experiments use a pipeline that detects entity spans, normalizes them, and aggregates candidates into one value per entity type. They train NER models on autotagged data and account for long inputs and alternative PDF text-extraction tools.
4 Experiments
- The Pipeline approach uses NER to find entity occurrences, normalizes them, then aggregates candidates to return one value for each entity type.Autotagging prepares span-level training data from document-level annotations; it is not applied at prediction time.Source: §4 · p. 10, §4.1 · p. 10
4.1 Document Processing Pipeline
- The pipeline trains Flair, BERT, RoBERTa, LayoutLM, or LAMBERT on autotagged spans, then normalizes detected values to canonical forms and selects the highest-scoring normalized group.For example, date variants are normalized to YYYY-MM-DD; grouped candidates’ scores are summed before selection.Source: §4.1 · p. 10
4.2 Experimental Setup
- Because Kleister documents exceeded model input limits, documents were split into overlapping 300-word chunks for Flair or 510 BPE-token chunks for BERT/RoBERTa/LayoutLM/LAMBERT; different PDF text-extraction tools were also tested for their effect on pipeline scores.Scores for each token in overlapping parts were aggregated by averaging.Source: §4.2 · pp. 10–11
5 Results
LAMBERT leads the tested baselines, while performance remains challenging across several entity types and document conditions. The analysis highlights difficulties with autotagging, monetary normalization, document length, layout, and OCR.
5 Results
- 0.77 overall F1-score improvement on Kleister NDA and 2.04 on Kleister Charity; LAMBERT's advantage on born-digital NDA documents was not substantial.On Charity, its gains over sequential models were +4.03 F1-score on income and +5.60 on spending, which appears mostly in table-like structures.Source: §5 · p. 11, Table 3 · p. 12
- The most challenging entities for all models are effective_date, party, term, post_town, postcode, street_line, charity_name, income, and spending.Source: §5 · p. 12
- Although autotagging underperforms nearly all document-level models on text spans, NDA party is especially difficult because organization names appear as acronyms or shortened forms.The passage notes that a general regular-expression rule does not readily capture forms such as “Emerson Electric Co.” and “Emerson.”Source: §5 · p. 12
- About 5% of Charity income and spending examples require inferring monetary scale, such as thousand or million, from document context.The figure illustrates an income amount in a table that should be multiplied by 1000.Source: §5 · p. 12, Figure 5 · p. 13
- All models perform worse on longer documents in the Kleister Charity test set.Source: §5 · p. 12, Figure 6 · p. 13
- LAMBERT's results support using textual (1D) and layout (2D) features; charity_number and report_date score higher when they appear in sequential contexts.Source: §5 · pp. 12–13
- Commercial OCR engines outperform Tesseract on scanned Charity documents, while born-digital PDFs should be processed with a dedicated PDF tool such as pdf2djvu.The authors report that good PDF processing can add several F1-score points.Source: §5 · p. 13
6 Conclusions
The conclusion introduces Kleister NDA and Kleister Charity and presents baseline solutions whose best scores are lower than those reported for similar-domain datasets. The datasets are intended to support investigation of document length, complex layouts, and OCR quality in KIE.
6 Conclusions
- The paper introduces Kleister NDA and Kleister Charity and details their preparation for Key Information Extraction tasks.Source: §6 · p. 13
- The authors intend the datasets to help investigate how document length, complex layouts, and OCR quality affect KIE performance.Source: §6 · p. 13
- 81.77/83.57 F1-score on Kleister NDA and Kleister Charity, respectively, was achieved by the best baseline, below similar-domain results such as 98.17 for SROIE.Source: §6 · pp. 13–14, Table 4 · p. 14