Source-linked AI summary
PHOCNet: A Deep Convolutional Neural Network for Word Spotting in Handwritten Documents
Sebastian Sudholt, Gernot A. Fink
TL;DR
Word spotting needs methods that avoid distorted word images and limitations of fixed-class, softmax-based CNNs. The paper introduces PHOCNet, which predicts PHOC representations from arbitrary-sized word images for QbE and QbS retrieval. Across multiple datasets and scripts, PHOCNet reportedly outperforms existing approaches while training and testing quickly.
Problem
Existing CNN word-spotting methods crop or rescale word images, use ImageNet pretraining, and cannot readily handle out-of-vocabulary QbS queries.
Method
PHOCNet is a CNN trained with PHOC labels, using sigmoid-based multi-label prediction and spatial pyramid pooling for arbitrary-sized inputs.
Results
PHOCNet outperforms current state-of-the-art approaches, ImageNet-finetuned CNNs, and Attribute SVMs across QbE and QbS datasets in Latin and Arabic scripts.
Takeaways & Limitations
The approach provides a single word-image representation usable for both QbE and QbS and supports competitive performance with short training and test times.
Abstract
from arXiv · showhide
In recent years, deep convolutional neural networks have achieved state of the art performance in various computer vision task such as classification, detection or segmentation. Due to their outstanding performance, CNNs are more and more used in the field of document image analysis as well. In this work, we present a CNN architecture that is trained with the recently proposed PHOC representation. We show empirically that our CNN architecture is able to outperform state of the art results for various word spotting benchmarks while exhibiting short training and test times.
I. INTRODUCTION
Word spotting indexes document images when direct recognition is unreliable, but existing CNN-based approaches distort word images or rely on mismatched pretraining. PHOCNet addresses these issues with a word-spotting CNN for both QbE and QbS.
- Earlier CNN word spotting cropped words to unit width and height and used ImageNet pretraining from a different domain.
- PHOCNet is a CNN architecture specifically designed for word spotting and achieves state-of-the-art performance in QbE and QbS across different datasets.
- Word spotting retrieves relevant word images from document collections when recognition does not produce reliable results.
- QbE uses a word-image query, whereas QbS accepts a textual query and returns matching word images.
- QbE can require users to identify a query image, while QbS requires annotated word images to learn text-to-image mappings.
B. Convolutional Neural Networks
CNNs became prominent through strong computer-vision performance, large datasets, and efficient hardware, but their use for word spotting remained limited and often required fixed-size inputs.
- Large-scale datasets and optimized GPU implementations made training CNNs with thousands of parameters practical.
- CNN use in word spotting was limited, with prior work fine-tuning a pretrained network to learn word-image classes.
- Fixed input sizes in prior CNN approaches required most word images to be cropped or rescaled.
- An alternative ensemble used character and n-gram CNNs for unconstrained text recognition, but resized word images and supported words up to 23 characters.
III. METHOD
A CNN combines convolutional feature extraction with pooling and fully connected classification, then trains predictions using output activations and cross-entropy loss.
- Convolutional layers apply filters to input images and produce feature maps for subsequent CNN layers.
- Activation functions introduce nonlinearity, while pooling aggregates filter responses by downsampling feature maps.
- Deep CNNs commonly use fully connected layers stacked as a multilayer perceptron after convolutional feature extraction.
- Softmax converts the final-layer output into predicted pseudo-class probabilities for single-label classification.
- Cross-entropy loss compares one-hot labels with predicted probabilities, and the error is backpropagated through the network.
B. PHOCNet Architecture
PHOCNet uses PHOC-based multi-label prediction, spatial pyramid pooling, and a retrieval representation to support arbitrary-sized word images in QbE and QbS.
- PHOCNet Architecture: PHOCNet uses 3 × 3 convolutions with ReLU activations and increasing filter counts in higher layers.
- PHOCNet Architecture: A 3-level spatial pyramid max-pooling layer produces constant output size from arbitrarily sized input images, avoiding cropping or resizing.
- PHOCNet Architecture: PHOC is a binary pyramidal representation that encodes whether character attributes occur in particular string splits.
- PHOCNet Architecture: Using lowercase Latin characters, ten digits, and frequent bigrams, the PHOC dimensionality is 604.
- PHOCNet Architecture: Sigmoid activations replace softmax because multiple PHOC attributes can be positive, making training a multi-label classification task.
- PHOCNet Architecture: The trained network outputs an estimated PHOC used with nearest-neighbor retrieval for QbE and QbS.
C. Regularization
PHOCNet uses regularization to reduce overfitting from its large parameter count, combining dropout with augmented training word images.
- C. Regularization: Large parameter count makes PHOCNet prone to overfitting, motivating multiple regularization techniques.
- C. Regularization: Dropout with probability 0.5 is applied to every fully connected layer except the last.Dropout randomly sets activations to zero and acts as a CNN regularizer.
- C. Regularization: Training images are augmented with random affine transformations to balance word-image classes and provide additional regularization.
D. Training
PHOCNet training uses fixed stochastic-gradient-descent settings, variance-based weight initialization, and execution on a single Titan X GPU with Caffe.
- D. Training: PHOCNet is trained with stochastic gradient descent using batch size 10, momentum 0.9, weight decay 5 · 10^-5, and learning rate 10^-4.Training runs for 80,000 iterations, with the learning rate divided by 10 after 70,000 iterations.
- D. Training: Weights are initialized from a zero-mean uniform distribution with variance 2/n, where n is the number of parameters in the layer.Layer biases are initialized to 0.
- D. Training: The authors report slightly better results with uniform rather than Gaussian weight initialization.
- D. Training: Training is performed on a single Nvidia GeForce Titan X GPU using the Caffe framework.
A. Datasets
PHOCNet is evaluated on four word-spotting datasets spanning single-writer and multi-writer handwriting, English and Arabic scripts, and multiple evaluation partitions.
- A. Datasets: The George Washington dataset contains 4,860 words across 20 pages and is evaluated with fourfold cross-validation.It is a homogeneous single-writer correspondence dataset with no official train-test partition.
- A. Datasets: IAM contains 115,320 words written by 657 writers and uses the official writer-independent text-line recognition partition.Official stop words are excluded as queries but retained as distractors for comparison with prior work.
- A. Datasets: Esposalles is a multi-writer marriage-license register from 1451–1905, using 32,052 training images and 12,048 test images.
- A. Datasets: IFN/ENIT contains handwritten Arabic city names, with subsets a–c for training and subset d for testing.The partition contains 19,724 training images and 6,735 test images.
- A. Datasets: Arabic PHOCs use a reduced character set that maps character shapes and Shadda variants to representative forms.The resulting character set has size 50 and produces 800-dimensional PHOC representations.
- A. Datasets: Table I reports QbE and QbS experiment results in mean average precision percentages.
B. Protocol
The evaluation uses segmentation-based Query-by-Example and Query-by-String protocols, ranking test word images with PHOCNet outputs and Bray-Curtis dissimilarity.
- B. Protocol: Evaluation is performed in segmentation-based QbE and QbS scenarios using the same protocol as prior work.Ground-truth bounding boxes provide perfect segmentation before training and querying.
- B. Protocol: For QbE, each test word image is used once as a query to rank the remaining test-set word images.Queries appearing only once in the test set are discarded.
- B. Protocol: Bray-Curtis dissimilarity is used as the distance measure for retrieval.
- B. Protocol: PHOCNet is compared with state-of-the-art literature results and a Softmax CNN sharing its architecture but using softmax output activation.The Softmax CNN requires considerably more training iterations in pre-experiments.
C. Results
PHOCNet achieves strong word-spotting performance across datasets while remaining efficient to train and robust across parameter settings. Results also show high performance on small training partitions, though comparisons using different protocols are not direct.
- Some included methods use different evaluation protocols, so their results cannot be compared directly with PHOCNet.Examples include retrieving entire lines or using fivefold cross-validation, which can make retrieval easier.
- 86.59 mAP is achieved on GW using one training fold and one testing fold, with 1215 training images and 488 training classes on average.This result uses the same cross-validation splits as the full GW experiments.
- PHOCNet outperforms other methods on the small GW dataset using data augmentation and common regularization techniques.The setup uses 3645 training images and 964 training classes on average.
- Training on IAM terminates in less than 17 hours, while estimating a PHOC for one word image takes less than 28 ms.Training Attribute SVMs on IAM takes roughly two days by comparison.
- Highly competitive results can already be achieved after 40 000 iterations when training speed is the primary concern.Figure 4 tracks mAP over training iterations for four QbE experiments.
- PHOCNet uses the same parameter set across all experiments and performs robustly on IAM in a multi-writer scenario.The authors describe robustness with respect to parametrization and writing style.
V. CONCLUSION
The paper introduces PHOCNet, a deep CNN for word spotting that accepts arbitrary-sized word images and predicts PHOC representations. Empirically, it outperforms current state-of-the-art approaches across datasets and scripts.
- PHOCNet is a deep CNN architecture for word spotting that processes input images of arbitrary size and predicts their corresponding PHOC representations.The architecture is trained using PHOC labels.
- Figure 4 displays mAP across training iterations for four QbE experiments, with cross-validation standard error shown for GW.The figure supports comparison of performance progression during training.
- PHOCNet outperforms current state-of-the-art approaches on various datasets.The conclusion reports this as an empirical finding.
- PHOCNet produces vastly better results than an ImageNet-pretrained CNN fine-tuned on word images and outperforms Attribute SVMs in QbE and QbS scenarios.The reported advantage holds for both Latin and Arabic script.