PaperlayerView exact revision on arXiv

Source-linked AI summary

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents

Guillaume Jaume, Hazim Kemal Ekenel, Jean-Philippe Thiran

arXiv:1905.13538v2cs.IRcs.CVcs.LGstat.ML

TL;DR

Form understanding requires extracting and structuring information from varied, noisy forms, but comprehensive public data for this task was lacking. The paper introduces FUNSD, a fully annotated dataset with a formalized pipeline, baselines, and evaluation metrics. Google Vision captures approximately 95% of textual content when words are correctly identified, while the dataset’s variability limits guaranteed generalization.

  • Problem

    Comprehensive publicly available datasets for extracting, structuring, and linking information in noisy, varied forms were lacking.

    Source: §I · p. 1

  • Method

    The paper constructs FUNSD from 199 annotated noisy forms and formalizes form understanding as linked tasks over labeled semantic entities.

    Source: §I · p. 1, A. A subset of the RVL-CDIP dataset · p. 2

  • Results

    ≈95% textual content is captured by Google Vision when words are correctly identified on FUNSD.

    Source: B. Text recognition with optical… · pp. 3–4

  • Takeaways & Limitations

    FUNSD provides a starting point for progress in document understanding using baselines and metrics across form-understanding tasks.

    Source: §I · pp. 1–2, §IV · p. 5

    The dataset may not contain enough variability for a generic and generalizable form-understanding application, and most textual content is machine-written.

    Source: D. Limitations of the FUNSD dataset · p. 3

Abstract

from arXiv · show

We present a new dataset for form understanding in noisy scanned documents (FUNSD) that aims at extracting and structuring the textual content of forms. The dataset comprises 199 real, fully annotated, scanned forms. The documents are noisy and vary widely in appearance, making form understanding (FoUn) a challenging task. The proposed dataset can be used for various tasks, including text detection, optical character recognition, spatial layout analysis, and entity labeling/linking. To the best of our knowledge, this is the first publicly available dataset with comprehensive annotations to address FoUn task. We also present a set of baselines and introduce metrics to evaluate performance on the FUNSD dataset, which can be downloaded at https://guillaumejaume.github.io/FUNSD/.

I. INTRODUCTION

The paper defines form understanding as extracting, structuring, and linking information from forms, and introduces FUNSD with baselines and evaluation metrics for this challenge.

  • Form understanding extracts and structures written information by identifying questions, answers, headers, and links among entities.Source: §I · p. 1
  • FUNSD addresses noisy scanned forms whose structures and appearances vary widely across fields.The dataset contains 199 fully annotated, one-page forms rendered at low resolution and corrupted by real noise.Source: §I · p. 1
  • Unlike template-specific commercial systems requiring manually annotated zones, FUNSD targets template-agnostic and semantically meaningful form representations.Such structured information can be stored in databases for data analysis.Source: §I · p. 1
  • The paper formalizes form understanding as a pipeline that produces labeled semantic entities and interlinks them.Source: §I · p. 1
  • FUNSD supports text detection, OCR, spatial layout analysis, and entity linking through bottom-up annotations.Source: §I · pp. 1–2
  • The authors provide baselines defining current benchmark results and metrics for evaluating the form-understanding pipeline.Source: §I · p. 2

II. DATASET DESCRIPTION

FUNSD was constructed from real, noisy document images selected from RVL-CDIP and manually filtered before annotation.

B. Annotation procedure

FUNSD represents forms as interlinked semantic entities annotated with labels, bounding boxes, words, and relations, using JSON ground-truth files.

Annotation process

  • The annotation tool was specifically designed for form understanding, while text-detection annotations were performed by mechanical turks.Source: B. Annotation procedure · p. 2

Data representation

Entity annotations

Dataset statistics

Linking example

D. Limitations of the FUNSD dataset

FUNSD broadens coverage across fields but cannot ensure sufficient variability for a generic, generalizable form-understanding application, and most text is machinewritten.

Dataset variability

Handwriting coverage

III. BASELINES AND METRICS

The paper reports baseline evaluations for word-level text detection using four systems and evaluates them with precision, recall, and F1 at IoU = 0.5.

Text detection

  • Four baselines evaluate word-level text detection: Tesseract, EAST, Google Vision API, and Faster R-CNN.Source: A. Text detection · p. 3
  • Tesseract, EAST, and Google Vision are tested without retraining, whereas Faster R-CNN is retrained for the FUNSD task.Source: A. Text detection · p. 3

Metrics

Results

  • Faster R-CNN yields the best overall performance by F1-score, while Google Vision also performs well without task-specific retraining.Source: A. Text detection · p. 3

B. Text recognition with optical character recognition

OCR quality is evaluated with case-sensitive Levenshtein similarity under two normalization schemes, with Google Vision substantially outperforming Tesseract on the noisy forms.

Evaluation

Results

C. Form understanding

Form understanding is decomposed into grouping words into semantic entities, labeling those entities, and linking related entities. Figure 2 illustrates grouping and labels for questions, headers, and answers.

C. Form understanding

1) Word grouping:

Word grouping is evaluated as clustering semantic entities, using OCR textline outputs as naive baselines. These baselines perform poorly because they ignore spatial layout and textual content.

1) Word grouping:

  • The baselines perform poorly because textline extraction ignores spatial layout and textual content.The paper anticipates learned grouping algorithms as a route to more competitive methods.Source: 1) Word grouping · p. 4
  • Word grouping is evaluated as clustering, with words as data points and ground-truth entity count fixing the number of clusters.Source: 1) Word grouping · p. 4
  • Adjusted Rand Index measures grouping, where 0 denotes random assignment and 1 denotes perfect clustering.Source: 1) Word grouping · p. 4, Table V · p. 4

2) Semantic entity labeling:

Semantic entity labeling uses a learned MLP baseline combining language, spatial, and sequence-length features. Evaluation assumes known grouping, locations, and text to isolate labeling.

2) Semantic entity labeling:

  • The baseline combines BERT semantic features, bounding-box spatial features, and sequence-length meta features.Together, these features produce a 733-dimensional representation for each semantic entity.Source: 2) Semantic entity labeling · p. 4
  • Each entity is independently classified by an MLP with two 500-unit ReLU hidden layers and a softmax output.Source: 2) Semantic entity labeling · p. 4
  • The labeling algorithms assume known word grouping, locations, and textual content, isolating the specific task.Results are reported in Table VI, whose precision and recall are expressed as percentages.Source: 2) Semantic entity labeling · p. 4, Table VI · p. 4

3) Entity linking:

Entity linking is formulated as binary classification over possible entity pairs using reused entity features. The paper evaluates predicted relations with precision, recall, and F1-score and calls for graph-aware baselines.

3) Entity linking:

  • Entity linking determines whether a relation exists between each possible pair of semantic entities.Source: 3) Entity linking · p. 4
  • The model concatenates the two entities’ feature representations and classifies each pair with a two-hidden-layer ReLU MLP.Source: 3) Entity linking · p. 4
  • Precision, recall, and F1-score evaluate whether predicted links correctly exist among identified and labeled entities.Some entities have no relations, such as page-number descriptions or unanswered questions.Source: 3) Entity linking · pp. 4–5
  • The paper suggests stronger baselines should represent semantic-entity relations as a graph.Source: 3) Entity linking · p. 5, §IV · p. 5

Related papers

Loading 1905.13538v2…