PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal deep networks for text and image-based document classification

Nicolas Audebert, Catherine Herold, Kuider Slimani, Cédric Vidal

arXiv:1907.06370v1cs.CV

TL;DR

Fine-grained document classification can depend on text that visual analysis alone does not capture. The paper combines OCR-derived word embeddings with document-image features in a multimodal network. It reports improved performance over single-modality baselines on two datasets, while noting limits in dataset realism and OCR quality.

  • Problem

    Fine-grained document recognition often depends on textual content as well as visual appearance, making classification from document images alone insufficient for some cases.

    Source: §1 · p. 2

  • Method

    The network combines OCR-extracted text represented with FastText word embeddings and visual features learned with MobileNetV2 for end-to-end classification.

    Source: §6 · p. 7

  • Results

    Using text and document appearance improves both single-modality baselines by several percentage points on two document-recognition datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The chosen document embedding appears at least partially robust to OCR noise and preserves enough content information to further boost CNN accuracy on document-image classification.

    Source: §5.1 · p. 6

    Public datasets do not capture the range of orientations, image conditions, and user-generated quality found in practice, so data augmentation is necessary for industrial-grade applications.

    Source: §5.3 · p. 7

Abstract

from arXiv · show

Classification of document images is a critical step for archival of old manuscripts, online subscription and administrative procedures. Computer vision and deep learning have been suggested as a first solution to classify documents based on their visual appearance. However, achieving the fine-grained classification that is required in real-world setting cannot be achieved by visual analysis alone. Often, the relevant information is in the actual text content of the document. We design a multimodal neural network that is able to learn from word embeddings, computed on text extracted by OCR, and from the image. We show that this approach boosts pure image accuracy by 3% on Tobacco3482 and RVL-CDIP augmented by our new QS-OCR text dataset (https://github.com/Quicksign/ocrized-text-dataset), even without clean text information.

1 Introduction

Document recognition supports filing workflows, but fine-grained classification often depends on document text as well as appearance. The paper introduces a multimodal network that learns from OCR-extracted text and document images.

1 Introduction

  • Document recognition helps determine the class-specific rules used to file submitted documents, such as subscription forms, IDs, and proof of residence.Automating this task can assist workers with document-processing burdens.Source: §1 · pp. 1–2
  • Fine-grained document recognition often depends on textual content, since documents with similar layouts can differ in what they say.For example, tax forms may share layouts, logos, and templates while their content differs.Source: §1 · p. 2
  • The proposed multimodal network learns from both document images and textual content automatically extracted by OCR.The pipeline is designed for end-to-end heterogeneous feature extraction and fusion under time and cost constraints.Source: §1 · p. 2
  • Using text and appearance improves both single-modality baselines by several percentage points on two document-recognition datasets.The reported datasets are Tobacco3482 and RVL-CDIP.Source: §1 · p. 2

2 Related work

Prior document-classification work has emphasized visual or textual features, while multimodal approaches have explored combining both. This paper proposes a hybrid deep model for OCRized document images using an end-to-end fusion network.

2 Related work

  • Earlier document-image classification largely used visual features such as layout, geometry, color, and shape, while text-classification research used methods including word embeddings and recurrent models.Deep CNN studies followed the introduction of the RVL-CDIP dataset.Source: §2 · pp. 2–3
  • Prior multimodal methods include OCR-derived keyword boxes, text and visual feature maps on clean synthetic data, and shallow bag-of-words and visual-words models.The synthetic-data approach assumes clean text and images, while the shallow-model study reports gains when OCR is unreliable.Source: §2 · p. 3
  • The paper proposes a hybrid deep baseline that jointly learns from text and image using off-the-shelf feature extractors and an efficient end-to-end network.Its fusion strategy improves performance on two document-image datasets.Source: §2 · p. 3

3 Learning on text and image

The method combines a lightweight visual CNN with OCR-derived text features, then fuses their representations for classification. Text features use character-aware embeddings to accommodate noisy OCR and out-of-vocabulary words.

3.1 Visual features

  • MobileNetV2 supplies visual features through an ImageNet-pretrained, lightweight CNN built from inverted bottleneck blocks.Each block expands channels, applies a depthwise convolution, and projects features; the architecture is described as faster than VGG-16 with on-par accuracy.Source: §3.1 · p. 3

3.2 Textual features

  • FastText character-based embeddings address OCR noise and out-of-vocabulary words by producing word vectors from subword information.On Tobacco3482, an average of 23% of words were absent from GloVe embeddings and 26% from the US English dictionary; FastText was chosen over ELMo partly because it is faster at inference.Source: §3.2 · p. 4
  • The text branch tokenizes OCR output with SpaCy and uses pretrained FastText embeddings, either averaged into a document vector or retained as a word sequence.The sequence option supports models that handle variable-length inputs.Source: §3.2 · p. 4

3.3 Multimodal features

  • The multimodal network concatenates text and image feature vectors so a fusion MLP can combine them without requiring aligned feature spaces.Both fusion strategies are differentiable, but preliminary experiments found that summation underperformed the image baseline, so the paper uses concatenation.Source: §3.3 · p. 4

4 Experimental setup

The experiments use Tobacco3482 and RVL-CDIP with OCR-extracted text, then compare single-modality models and a multimodal fusion model. The setup includes a sequence-based text classifier, an image baseline, and preliminary augmentation and tuning choices.

4.1 Datasets

  • Tobacco3482 contains 3,482 documents in 10 classes, while RVL-CDIP contains 400,000 documents in 16 classes with a standard 320,000/40,000/40,000 train/validation/test split.Tobacco3482 uses 800 training documents and the remainder for testing, averaged over three runs.Source: Tobacco3482 · p. 4, RVL-CDIP · p. 5
  • Tesseract OCR text was extracted from both datasets and released as QS-OCR for replication and further post-OCR or multimodal classification experiments.Figure 4 notes that Tesseract output is noisy and does not fully match common word-embedding training assumptions.Source: Text generation · p. 5, Figure 4 · p. 5

4.2 Models

  • The text baseline uses a 1D CNN over word sequences rather than averaged embeddings, because it can select relevant content despite noisy OCR output.The authors report that the CNN significantly outperforms the MLP on Tobacco3482 and choose it as the TEXT component.Source: Text baseline · p. 5
  • The IMAGE baseline uses pretrained MobileNetV2 on grayscale pages resized to 384×384, duplicating the grayscale channel for ImageNet initialization.The aspect ratio is warped; prior work cited in the passage reports better accuracy than padding at the same resolution.Source: Image baseline · p. 5
  • JPEG artifacts, saturation, and contrast augmentation did not significantly change Tobacco3482 accuracy, and the authors omit multi-crop inference for speed.They attribute the augmentation result to low variability among the grayscale document images.Source: Image baseline · p. 6
  • FUSION maps the image and text representations to 128-dimensional vectors before multimodal training.The image representation comes from global average pooling of convolutional features; fusion is trained for 200 epochs with SGD and momentum.Source: Fusion · p. 6

5 Discussion

Fusion improves image-only document classification on Tobacco3482 and RVL-CDIP, with processing times relevant to interactive use. However, the public datasets underrepresent real user-generated capture conditions.

5.2 Processing time

  • On an Intel Core i7-8550U, FUSION takes ≃360ms and OCR-plus-classification <≃1300ms; batching averages ≤750ms on GPU and ≤1000ms on CPU.The CPU batching figures use two Tesseract processes; batching improves global throughput, not one user's perceived delay.Source: §5.2 · pp. 6–7

5.1 Performances

5.3 Limitations

  • Tobacco documents are right-oriented and professionally scanned; industrial use needs augmentation for varied capture conditions, especially rotations.Post-OCR embeddings also contain noisy or wrong words that generate OOV errors; SpaCy tokenization and named entity recognition support partial spellchecking, while robust embeddings remain future work.Source: §5.3 · p. 7

6 Conclusion

The authors combine OCR-derived text embeddings with visual features in an end-to-end classifier, reporting consistent gains on Tobacco3482 and RVL-CDIP. They also note that clean text is unavailable for document-image classification and identify further investigation as future work.

6 Conclusion

  • Consistent gains were reported on both Tobacco3482 and RVL-CDIP, spanning small and large datasets.The conclusion describes the gains as evidence of interest in hybrid image/text approaches even when clean text is unavailable.Source: §6 · p. 7
  • The classifier fuses character-based FastText embeddings of noisy Tesseract OCR output with visual features learned by MobileNetv2.The end-to-end network jointly learns text and image features for final classification.Source: §6 · p. 7
  • Clean text is unavailable for document-image classification, and the authors identify further investigation as future work.Source: §6 · pp. 7–8

Related papers

Loading 1907.06370v1…