Source-linked AI summary
FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
Guillaume Jaume, Hazim Kemal Ekenel, Jean-Philippe Thiran
TL;DR
Form understanding requires extracting and structuring information from varied, noisy forms, but comprehensive public data for this task was lacking. The paper introduces FUNSD, a fully annotated dataset with a formalized pipeline, baselines, and evaluation metrics. Google Vision captures approximately 95% of textual content when words are correctly identified, while the dataset’s variability limits guaranteed generalization.
Problem
Comprehensive publicly available datasets for extracting, structuring, and linking information in noisy, varied forms were lacking.
Method
The paper constructs FUNSD from 199 annotated noisy forms and formalizes form understanding as linked tasks over labeled semantic entities.
Results
≈95% textual content is captured by Google Vision when words are correctly identified on FUNSD.
Takeaways & Limitations
FUNSD provides a starting point for progress in document understanding using baselines and metrics across form-understanding tasks.
Takeaways & Limitations
The dataset may not contain enough variability for a generic and generalizable form-understanding application, and most textual content is machine-written.
Abstract
from arXiv · showhide
We present a new dataset for form understanding in noisy scanned documents (FUNSD) that aims at extracting and structuring the textual content of forms. The dataset comprises 199 real, fully annotated, scanned forms. The documents are noisy and vary widely in appearance, making form understanding (FoUn) a challenging task. The proposed dataset can be used for various tasks, including text detection, optical character recognition, spatial layout analysis, and entity labeling/linking. To the best of our knowledge, this is the first publicly available dataset with comprehensive annotations to address FoUn task. We also present a set of baselines and introduce metrics to evaluate performance on the FUNSD dataset, which can be downloaded at https://guillaumejaume.github.io/FUNSD/.
I. INTRODUCTION
The paper defines form understanding as extracting, structuring, and linking information from forms, and introduces FUNSD with baselines and evaluation metrics for this challenge.
- Form understanding extracts and structures written information by identifying questions, answers, headers, and links among entities.
- FUNSD addresses noisy scanned forms whose structures and appearances vary widely across fields.The dataset contains 199 fully annotated, one-page forms rendered at low resolution and corrupted by real noise.
- Unlike template-specific commercial systems requiring manually annotated zones, FUNSD targets template-agnostic and semantically meaningful form representations.Such structured information can be stored in databases for data analysis.
- The paper formalizes form understanding as a pipeline that produces labeled semantic entities and interlinks them.
- FUNSD supports text detection, OCR, spatial layout analysis, and entity linking through bottom-up annotations.
- The authors provide baselines defining current benchmark results and metrics for evaluating the form-understanding pipeline.
II. DATASET DESCRIPTION
FUNSD was constructed from real, noisy document images selected from RVL-CDIP and manually filtered before annotation.
- FUNSD uses a subset of RVL-CDIP to capture real forms with varied structures and realistic scanning noise.RVL-CDIP contains low-resolution grayscale documents affected by successive scanning and printing procedures.
- From 25,000 form-category images, manual checking produced 3,200 eligible documents, from which 199 were randomly sampled for annotation.
- The source documents span fields including scientific research, marketing, and advertising.
B. Annotation procedure
FUNSD represents forms as interlinked semantic entities annotated with labels, bounding boxes, words, and relations, using JSON ground-truth files.
- Annotation process: The annotation tool was specifically designed for form understanding, while text-detection annotations were performed by mechanical turks.
- Data representation: Each form is encoded as a JSON list of interlinked semantic entities.Entities group words that belong together semantically and spatially.
- Entity annotations: Each semantic entity contains a unique identifier, label, bounding box, links, and word list.Labels include question, answer, header, and other.
- Entity annotations: Each word is represented by its textual content and bounding box, with boxes encoded as [xleft, ytop, xright, ybottom].
- Dataset statistics: The dataset statistics and semantic-entity class distribution are reported in Tables I and II.
- Linking example: The example links “Registration No.” as a question to “533” as an answer, forming a question–answer pair.
D. Limitations of the FUNSD dataset
FUNSD broadens coverage across fields but cannot ensure sufficient variability for a generic, generalizable form-understanding application, and most text is machinewritten.
- Dataset variability: FUNSD selects forms from marketing, science, advertisement, and other fields to attenuate variation in form appearance.
- Dataset variability: The authors cannot ensure that FUNSD captures enough examples to create a generic and generalizable form-understanding application.
- Handwriting coverage: Most textual content is machinewritten, although some forms contain handwritten signatures and dates.
III. BASELINES AND METRICS
The paper reports baseline evaluations for word-level text detection using four systems and evaluates them with precision, recall, and F1 at IoU = 0.5.
- Text detection: Four baselines evaluate word-level text detection: Tesseract, EAST, Google Vision API, and Faster R-CNN.
- Text detection: Tesseract, EAST, and Google Vision are tested without retraining, whereas Faster R-CNN is retrained for the FUNSD task.
- Metrics: Precision, recall, and F1 score are computed on the FUNSD test set at IoU = 0.5.
- Results: Faster R-CNN yields the best overall performance by F1-score, while Google Vision also performs well without task-specific retraining.
B. Text recognition with optical character recognition
OCR quality is evaluated with case-sensitive Levenshtein similarity under two normalization schemes, with Google Vision substantially outperforming Tesseract on the noisy forms.
- Evaluation: OCR output is scored using Levenshtein similarity between each predicted word and its ground-truth word.The metric accounts for character differences and checkbox recognition.
- Evaluation: The evaluation reports two metrics: text detection + OCR normalized by ground-truth words, and OCR normalized by identified words.
- Evaluation: The OCR engines receive the documents without preprocessing before recognition.
- Results: Google Vision captures textual content at approximately 95% similarity when words are correctly identified.
- Results: Tesseract performs poorly on FUNSD, potentially because the dataset does not meet its stated minimum quality of 300 dpi.
C. Form understanding
Form understanding is decomposed into grouping words into semantic entities, labeling those entities, and linking related entities. Figure 2 illustrates grouping and labels for questions, headers, and answers.
- C. Form understanding: FoUn comprises word grouping, semantic-entity labeling, and entity linking.These tasks structure form text as labeled semantic entities and relations.
- C. Form understanding: Word grouping aggregates words belonging to the same semantic entity.
- C. Form understanding: Semantic-entity labeling assigns each entity one of four categories: question, answer, header, or other.
- C. Form understanding: Entity linking predicts relations between semantic entities.
- C. Form understanding: Tesseract and Google Vision textline extraction provide naive baselines for grouping words.
1) Word grouping:
Word grouping is evaluated as clustering semantic entities, using OCR textline outputs as naive baselines. These baselines perform poorly because they ignore spatial layout and textual content.
- 1) Word grouping:: The baselines perform poorly because textline extraction ignores spatial layout and textual content.The paper anticipates learned grouping algorithms as a route to more competitive methods.
- 1) Word grouping:: Word grouping is evaluated as clustering, with words as data points and ground-truth entity count fixing the number of clusters.
- 1) Word grouping:: Adjusted Rand Index measures grouping, where 0 denotes random assignment and 1 denotes perfect clustering.
2) Semantic entity labeling:
Semantic entity labeling uses a learned MLP baseline combining language, spatial, and sequence-length features. Evaluation assumes known grouping, locations, and text to isolate labeling.
- 2) Semantic entity labeling:: The baseline combines BERT semantic features, bounding-box spatial features, and sequence-length meta features.Together, these features produce a 733-dimensional representation for each semantic entity.
- 2) Semantic entity labeling:: Each entity is independently classified by an MLP with two 500-unit ReLU hidden layers and a softmax output.
- 2) Semantic entity labeling:: The labeling algorithms assume known word grouping, locations, and textual content, isolating the specific task.Results are reported in Table VI, whose precision and recall are expressed as percentages.
3) Entity linking:
Entity linking is formulated as binary classification over possible entity pairs using reused entity features. The paper evaluates predicted relations with precision, recall, and F1-score and calls for graph-aware baselines.
- 3) Entity linking:: Entity linking determines whether a relation exists between each possible pair of semantic entities.
- 3) Entity linking:: The model concatenates the two entities’ feature representations and classifies each pair with a two-hidden-layer ReLU MLP.
- 3) Entity linking:: Precision, recall, and F1-score evaluate whether predicted links correctly exist among identified and labeled entities.Some entities have no relations, such as page-number descriptions or unanswered questions.
- 3) Entity linking:: The paper suggests stronger baselines should represent semantic-entity relations as a graph.