PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval

Adam W. Harley, Alex Ufkes, Konstantinos G. Derpanis

arXiv:1502.07058v1cs.CVcs.IRcs.LGcs.NE

TL;DR

Document image classification and retrieval must use visual structure despite intra-class layout variation and interclass similarity. The paper evaluates CNN-based representations, including transfer learning, holistic and region-based models, and compression, and finds them superior to handcrafted alternatives across the studied tasks. CNN features also transfer from non-document images, remain effective under compression, and need not use region-specific guidance given sufficient training data.

  • Problem

    Document analysis must classify and retrieve images despite intra-class variability, interclass similarity, and information unavailable from OCR alone.

  • Method

    The paper evaluates holistic and region-based CNN representations, pretrained transfer, feature compression, and CNN activations for document classification and retrieval.

  • Results

    CNN features outperform handcrafted alternatives across classification and retrieval; ImageNet pretraining transfers well, compression causes little loss, and holistic CNNs can match region-based ensembles.

  • Takeaways & Limitations

    Deep CNN features provide a strong general representation for document image classification and retrieval, while whole-image learning can avoid explicit region-specific guidance with sufficient training data.

Abstract

from arXiv · show

This paper presents a new state-of-the-art for document image classification and retrieval, using features learned by deep convolutional neural networks (CNNs). In object and scene analysis, deep neural nets are capable of learning a hierarchical chain of abstraction from pixel inputs to concise and descriptive representations. The current work explores this capacity in the realm of document analysis, and confirms that this representation strategy is superior to a variety of popular hand-crafted alternatives. Experiments also show that (i) features extracted from CNNs are robust to compression, (ii) CNNs trained on non-document images transfer well to document analysis tasks, and (iii) enforcing region-specific feature-learning is unnecessary given sufficient training data. This work also makes available a new labelled subset of the IIT-CDIP collection, containing 400,000 document images across 16 categories, useful for training new CNNs for document analysis.

I. INTRODUCTION

Document images can reveal category through visual structure and layout even when text is unreadable, but classification and retrieval must handle substantial variation within and across categories. The paper therefore evaluates deep CNN features for these tasks.

  • I. INTRODUCTION: Letter documents often share recognizable visual structure despite illegible text and variation in exact layout.Addresses and dates typically appear near the top, while signatures typically appear near the bottom.
  • I. INTRODUCTION: Document image analysis supports indexing before OCR, improves OCR preprocessing, and preserves visual information that OCR loses.Images retain typeface, graphics, and layout for storage and indexing.
  • I. INTRODUCTION: Wide intra-class layout variability makes rigid template matching difficult, while interclass visual similarity complicates classification and retrieval.Documents within a category may omit or rearrange components, and different categories can look substantially alike.
  • I. INTRODUCTION: The paper extensively evaluates CNNs for document classification and retrieval, including transfer learning and region-focused feature learning.It reports that deep CNN features exceed popular alternative features on both tasks by a large margin.

A. Related Work

Prior document analysis compared region-based, holistic, spatial-pyramid, and machine-learned representations, while CNN retrieval remained unexplored. This paper builds on CNN successes in other vision domains and transfers their learned representations to documents.

  • A. Related Work: Region-based methods exploit consistent spatial configurations and can approach template matching for rigid documents.Their assumptions are strongest for forms and business letters with stable component locations.
  • A. Related Work: Holistic bag-of-visual-words methods seek landmark features regardless of position, trading geometric invariance for weaker discrimination on rigid templates.They support broader document coverage than template-based approaches.
  • A. Related Work: Spatial pyramid matching combines whole-image and increasingly localized pooled features to represent both global and local layout characteristics.For document retrieval, this representation was described as the current state of the art.
  • A. Related Work: Machine-learned document representations had replaced many handcrafted alternatives, but CNNs had not yet been applied to document retrieval.The paper positions CNN retrieval as an open gap following CNN advances in object recognition and related tasks.
  • A. Related Work: CNN research recommends large-problem pretraining to reduce overfitting and considers region-specific networks when spatial information matters.These practices motivate the paper’s initialization and region-based experiments.
  • A. Related Work: Activations near the top of classification-trained CNNs can serve as descriptive feature vectors for retrieval.The paper applies this transfer from image retrieval to documents.

B. Contributions

The paper evaluates deep CNN features, transfer learning, compression, and region-specific guidance for document analysis, and releases a large labelled document dataset. Its experiments favor CNN representations over handcrafted alternatives.

  • B. Contributions: CNN features outperform all handcrafted competitors and can be compressed to very short codes with negligible performance loss.The contribution covers CNN design, training, feature processing, and compression.
  • B. Contributions: CNNs trained on non-document images transfer well to document-related tasks.This establishes the usefulness of pretrained visual representations beyond their original domain.
  • B. Contributions: Region-specific guidance produces little to no improvement in classification and retrieval over a basic holistic CNN.The result suggests holistic CNNs may learn region-specific or better features automatically.
  • B. Contributions: The released IIT-CDIP subset contains 400,000 document images across 16 categories.It is intended for training new CNNs for document analysis.

II. TECHNICAL APPROACH

The technical approach learns document representations with either a holistic CNN or an ensemble of region-based CNNs, using alternative initialization strategies and CNN activations for retrieval. Region guidance targets subtle local differences between visually similar document types.

  • B. Region-based guidance: The paper trains either one holistic CNN or an ensemble of region-based CNNs to learn informative document features.It compares random initialization with initialization based on prior training.
  • II. TECHNICAL APPROACH: CNNs process pixel matrices through convolutional layers and classify the resulting representations with fully connected layers.The architecture provides hierarchical representational capacity for document images.
  • II. TECHNICAL APPROACH: The approach retains spatial specificity in CNN activations because document analysis can benefit from location-sensitive features.This differs from applications where geometric invariance is usually desirable.
  • II. TECHNICAL APPROACH: High-level CNN activations become retrieval vectors whose dimensionality can be reduced with PCA without significantly affecting discriminative power.The passage gives 4096 dimensions as an example that can be reduced to 128.
  • B. Region-based guidance: Letters and memos may require region-focused learning because their consistent difference can be confined to the address section.Memos have short “To” and “From” fields, whereas letters have full addresses.

B. Region-based guidance

The paper evaluates region-specific CNN guidance and transfer learning for document analysis. Region features are combined into a compressed descriptor, while ImageNet pre-training supports learning when document data are limited.

  • B. Region-based guidance: Region-specific guidance assigns CNNs to document regions so their features can be extracted and combined for discrimination.The proposed representation uses region-tuned CNNs alongside a holistic CNN.
  • B. Region-based guidance: Five CNNs process one holistic image representation and four regions: header, left body, right body, and footer.The resulting PCA-compressed vectors are concatenated for classification or used directly for retrieval.
  • C. Transfer learning: Transfer learning pre-trains CNN weights on a complementary, larger task before training on the target document problem.The paper investigates whether ImageNet features transfer to documents and whether pre-training improves over random initialization.
  • C. Transfer learning: ImageNet-trained networks have been shown to provide general-purpose features for other vision challenges without target-task fine-tuning.This prior result motivates testing transfer from natural-image recognition to document analysis.

A. Datasets

The experiments use two IIT-CDIP-derived datasets with different sizes and category counts. SmallTobacco supports comparison with prior work, while BigTobacco provides a newly sampled 400,000-image, 16-category benchmark.

  • A. Datasets: The IIT-CDIP collection contains over seven million scanned documents with hand-labelled tags, although some tags are missing or erroneous.The collection was assembled from public records of lawsuits against American tobacco companies.
  • A. Datasets: SmallTobacco contains 3482 images from the collection, each assigned one of ten labels.It was selected and labelled in earlier work and used in related document-analysis studies.
  • A. Datasets: BigTobacco contains 400000 labelled images sampled randomly from 16 IIT-CDIP categories.The sample includes classes such as letters, memos, forms, invoices, presentations, and scientific publications.
  • A. Datasets: SmallTobacco uses 800 training images, 200 validation images, and the remainder for testing across 10 random splits.Reported performance is the median across splits; retrieval split selection uses mAP@10.

B. Implementation details

The implementation compares CNN, bag-of-words, brightness, and retrieval representations across holistic and region-based designs. CNN features are extracted from fully connected layers, while PCA compresses large descriptors before classification or retrieval.

  • B. Implementation details: The CNNs use an N-way softmax output, where N is the number of learned document categories.The networks were implemented in Caffe.
  • B. Implementation details: Most CNNs use the ImageNet reference architecture with five convolutional and three fully connected layers, producing 4096-dimensional extracted features.Features come from the first fully connected layer.
  • B. Implementation details: The ensemble of CNNs compresses each 4096-dimensional region vector to 640 dimensions before concatenating them for classification.Its classification architecture is written as 3200 × 4096 × N.
  • B. Implementation details: The region representation crops header, footer, and left and right body areas after resizing images to 780 × 600.Each extracted region is resized to 227 × 227 before CNN processing.
  • B. Implementation details: Baseline representations include GIST, average brightness, ensemble-of-regions brightness, and spatially partitioned SURF bag-of-words features.The BoW features use spatial pyramids and horizontal-vertical partitioning schemes.
  • B. Implementation details: Retrieval ranks training documents by Euclidean distance after compressing feature vectors larger than 128 dimensions to 128 dimensions with PCA.The sorted distances produce a ranked list for each test query.

C. Classification results

CNN-based approaches achieve strong classification performance on both dataset scales, with the best architecture depending on available training data. Region ensembles lead on SmallTobacco, while a fine-tuned holistic CNN leads on BigTobacco.

  • C. Classification results: 79.9% classification accuracy makes the ensemble of region-based CNNs the best approach on SmallTobacco.The previous best reported result was 65.4% with a randomly initialized Small CNN.
  • C. Classification results: The holistic CNN performs only slightly worse than the region ensemble on SmallTobacco.This suggests that holistic processing may learn some information that region-based analysis was expected to add.
  • C. Classification results: ImageNet fine-tuning substantially improves the larger CNNs, whereas randomly initialized CNNs perform similarly to or worse than bag-of-words approaches.Among the bag-of-words baselines, spatial-pyramid pooling performs best.
  • C. Classification results: On BigTobacco, the ImageNet-fine-tuned holistic CNN outperforms every other approach, including the ensemble of CNNs.The results associate this outcome with sufficient training data eliminating the advantage of region-tuned analysis.
  • C. Classification results: 95% accuracy is achieved by the CNN trained to classify only letters and memos.This result concerns the two-category LetterMemo task.

D. Retrieval results

Retrieval performance favors CNN-based representations, with the strongest approach depending on dataset size and training setup. CNN vectors also remain effective under substantial PCA compression.

  • On SmallTobacco, the ensemble of region-tuned CNNs performs best, followed by a holistic CNN fine-tuned from ImageNet.The generic ImageNet descriptor also exceeds most other descriptors.
  • The spatial-pyramid-pooled BoW performs best among BoW approaches on both SmallTobacco and BigTobacco.GIST performs approximately as well as BoW on SmallTobacco but exceeds BoW by a large margin on BigTobacco.
  • On BigTobacco, the holistic CNN performs best, exceeding the region-tuned ensemble by a small margin and most other approaches by a large margin.Every CNN vector outperforms the BoW approaches, including the generic ImageNet vector.
  • Representative BigTobacco retrievals show top-ranked documents sharing memo authors or letterheads with the query despite content and typeface variation.The authors caution that dataset biases, such as few letterheads or memo authors, may contribute to these results.
  • CNN vectors show almost no mAP@10 loss until PCA reduction to 16 dimensions, while the holistic CNN outperforms every other approach at all compression levels.The compression experiment was conducted on BigTobacco.

IV. CONCLUSION

The paper establishes CNN-based document image representations as a new state-of-the-art for classification and retrieval. CNN features outperform handcrafted alternatives, transfer from ImageNet, tolerate compression, and make region-specific ensembles unnecessary with sufficient data.

  • Deep CNN features establish a new state-of-the-art for document image classification and retrieval.The paper reports that CNN-based representations exceed handcrafted alternatives.
  • Generic ImageNet CNN features outperform state-of-the-art alternatives, while fine-tuning on document images pushes results higher.
  • With sufficient training data, a single holistic CNN performs approximately as well as an ensemble of region-specific CNNs.
Loading 1502.07058v1…