PaperlayerView exact revision on arXiv

Source-linked AI summary

Online and Offline Handwritten Chinese Character Recognition: A Comprehensive Study and New Benchmark

Xu-Yao Zhang, Yoshua Bengio, Cheng-Lin Liu

arXiv:1606.05763v1cs.CV

TL;DR

HCCR systems must learn discriminative representations while exploiting domain knowledge and handling writer-specific style variation. This paper combines normalized directional maps with an 11-layer convNet, establishes benchmarks for both online and offline HCCR, and adds unsupervised writer adaptation that further improves accuracy.

  • Problem

    HCCR must address large character vocabularies, similar-character confusion, handwriting-style variation, and mismatch when deep models transfer across domains or writers.

  • Method

    The paper integrates normalization-cooperated direction-decomposed directMaps with an 11-layer convNet and adds a source-layer adaptation layer trained through unsupervised self-training.

  • Results

    The framework sets new benchmarks for both online and offline HCCR on ICDAR-2013, while writer adaptation consistently improves accuracy across 60 writers.

  • Takeaways & Limitations

    Domain-specific representation and writer adaptation remain effective additions to deep HCCR systems, including systems that already surpass human performance.

  • Takeaways & Limitations

    The adaptation evaluation assumes writer-consistent groups and uses the writer index as additional information, rather than treating test samples as independent.

Abstract

from arXiv · show

Recent deep learning based methods have achieved the state-of-the-art performance for handwritten Chinese character recognition (HCCR) by learning discriminative representations directly from raw data. Nevertheless, we believe that the long-and-well investigated domain-specific knowledge should still help to boost the performance of HCCR. By integrating the traditional normalization-cooperated direction-decomposed feature map (directMap) with the deep convolutional neural network (convNet), we are able to obtain new highest accuracies for both online and offline HCCR on the ICDAR-2013 competition database. With this new framework, we can eliminate the needs for data augmentation and model ensemble, which are widely used in other systems to achieve their best results. This makes our framework to be efficient and effective for both training and testing. Furthermore, although directMap+convNet can achieve the best results and surpass human-level performance, we show that writer adaptation in this case is still effective. A new adaptation layer is proposed to reduce the mismatch between training and test data on a particular source layer. The adaptation process can be efficiently and effectively implemented in an unsupervised manner. By adding the adaptation layer into the pre-trained convNet, it can adapt to the new handwriting styles of particular writers, and the recognition accuracy can be further improved consistently and significantly. This paper gives an overview and comparison of recent deep learning based approaches for HCCR, and also sets new benchmarks for both online and offline HCCR.

1. Introduction

HCCR has progressed from traditional approaches to deep learning, but domain-specific knowledge remains useful. The paper combines directMap with convNet for both online and offline recognition and adds writer adaptation for style variation.

  • HCCR must handle many character classes, visually similar characters, and substantial handwriting-style differences across writers.
  • Deep learning methods gradually became dominant in HCCR competitions, with convNet systems winning both online and offline ICDAR-2013 tasks.
  • Shape normalization and direction-decomposed feature maps provide domain knowledge that can improve deep learning representations for HCCR.
  • The proposed directMap+convNet framework uses an 11-layer convNet and achieves state-of-the-art performance for both online and offline HCCR under one framework.Its embedded domain knowledge eliminates the need for data augmentation and model ensemble.
  • A writer-specific adaptation layer reduces the mismatch between training and test handwriting styles through unsupervised adaptation.Experiments on 60 writers show consistent and significant accuracy increases for both online and offline HCCR.
  • The paper also reviews recent deep-learning progress in HCCR and establishes new benchmarks for online and offline recognition.

2. Related Works

Related work shows a transition from traditional HCCR methods to convNets and other deep models. Prior results also indicate that domain knowledge and writer adaptation remain important for improving recognition.

  • ConvNets changed HCCR from traditional methods, with MCDNN succeeding offline and sparse convNet achieving the best online ICDAR-2013 result.
  • Combining handcrafted features or domain knowledge with deep networks has produced high accuracy in both offline and online HCCR.
  • Deep learning has also been applied to writer identification, hybrid models, confidence analysis, legal-amount recognition, text spotting, and online recognition with HMMs or RNNs.
  • Earlier writer adaptation focused mainly on traditional classifiers, while this work adapts convNets using a simple layer, unsupervised learning, and small amounts of writer-specific data.

3. Direction Decomposed Feature Map

DirectMap incorporates shape normalization and directional decomposition into compact representations for online and offline handwritten characters. Its sparse directional structure preserves character shape while supporting efficient processing.

  • Shape normalization maps original and normalized character coordinates, while direction decomposition can operate in either representation.
  • Offline directMaps use normalization-cooperated LDPI processing, whereas online directMaps use P2DBMN because LDPI does not apply to trajectories.
  • Each directMap is an 8 × 32 × 32 tensor containing eight directional maps and an average map for visualization.The average map reflects contour information offline and reconstructs local strokes online.
  • DirectMap sparsity reaches 92.41% zeros online and 79.01% offline, enabling efficient storage and reuse without losing shape information at smaller map sizes.
  • The traditional framework represents directMaps through sampling, Gaussian blurring, Box-Cox transformation, dimensionality reduction, and classification.

4. From Traditional HCCR to ConvNet

Traditional HCCR pipelines already contain operations analogous to components of a simplified convNet. This connection explains how handcrafted processing can be related to learned neural-network architectures.

  • Traditional HCCR samples each directional map after Gaussian blurring, applies a Box-Cox transformation, then performs dimensionality reduction and classification.
  • Gaussian blurring acts like a predefined convolution mask, while Box-Cox acts like a nonlinear activation in the traditional HCCR architecture.
  • The traditional pipeline also contains fully connected and classification layers, linking its design philosophy to deep neural networks.

5. Convolutional Neural Network

The paper uses an 11-layer convNet that processes normalized direction-decomposed feature maps for both online and offline HCCR. Its architecture combines convolution, pooling, fully connected layers, dropout, leaky-ReLU activation, and signal-rescaling initialization before softMax classification.

  • 5.1. Architecture: An 11-layer convNet processes directMap inputs for HCCR, with architecture sized to the 8 × 32 × 32 representation.The directMap is passed through convolutional layers with 3 × 3 filters and unit stride.
  • 5.1. Architecture: Eight convolutional layers and four max-pooling layers produce a 1600-dimensional vector, followed by fully connected layers of 900 and 200 units.The final softMax layer performs 3755-way classification.
  • 5.2. Regularization: Dropout is applied to every layer except layer-1 and layer-10, with probabilities increasing as network depth increases.Layer-10 is the final fully connected feature extractor before softMax.
  • 5.3. Activation: Leaky-ReLU supplies network nonlinearity and is described as potentially improving convergence and performance over sigmoid and tanh.Its form is f(x) = max(x, 0) + λ min(x, 0), with standard ReLU corresponding to λ = 0.
  • 5.4. Initialization: To avoid signal shrinking before softMax, the method rescales inputs by v = −ln 0.8/∆ after estimating ∆ from training data.The rescaling is preserved through max-pooling and leaky-ReLU because both are semi-linear and biases are initialized to zero.
  • 5.5. Training: Training minimizes multiclass negative log-likelihood with mini-batch gradient descent and momentum, without data augmentation.The mini-batch size is 1000, momentum is 0.9, and the initial learning rate is 0.005.

6. Adaptation of ConvNet

The adaptation method addresses mismatch between pretrained ConvNet features and new handwriting styles by learning a regularized affine layer from unlabeled data. It iteratively updates this layer using confidence-weighted pseudo-labels and can be efficiently reverted or implemented.

  • 6.1. Motivation: Domain adaptation is introduced to reduce mismatch between training and test data when transferring deep-network models to new domains.Direct fine-tuning is described as requiring significant labeled data.
  • 6.2. Adaptation Layer: The proposed adaptation layer is an activation-free affine transform placed after a fully connected source layer, with weight matrix A and offset vector b.Class-specific means on the source layer are estimated from training data before adaptation.
  • 6.3. Unsupervised Adaptation: Unlabeled adaptation samples receive ConvNet predictions and softMax confidences, which supply pseudo-labels and weights for learning the adaptation parameters.The procedure is summarized as estimating class means, adding the layer, initializing A = I and b = 0, then iterating prediction, confidence estimation, and parameter updates.
  • 6.4. Objective: The adaptation objective moves each source-layer feature toward its predicted class mean, weighting each transformation by softMax confidence.Regularization constrains A toward the identity matrix and b toward zero.
  • 6.5. Implementation: The learned output is fed into the network’s next layer, and placing adaptation after the 200-unit bottleneck reduces the sizes of A and b.This placement is intended to make adaptation more efficient and effective.
  • 6.6. Properties: The adaptation problem is a convex quadratic program with a closed-form solution and requires no sophisticated optimization tricks.Setting A = I and b = 0 returns the network to its unadapted state; adaptation data need not cover all classes.

7. Experiments

Experiments evaluate directMap+convNet for online and offline HCCR on ICDAR-2013 data, using CASIA datasets for training and competition data from different writers for testing. The offline results establish a new benchmark with a single compact network and without data augmentation.

  • Experimental setup: The experiments compare online and offline HCCR methods on ICDAR-2013 competition databases after training on CASIA datasets.The test writers differ from those in the training datasets.
  • Training of ConvNet: Two convNets are trained separately for online and offline HCCR using directMaps tailored to their respective input types.The training uses SGD with momentum and a learning-rate decay schedule.
  • Training of ConvNet: After the first learning-rate drop, training and test accuracy improve significantly following an initial plateau.The learning rate starts at 0.005 and is reduced three times before termination.
  • Offline HCCR results: DirectMap+convNet achieves a new offline HCCR benchmark with a single network and lower memory usage than the compared systems.The proposed model uses 23.5MB, whereas a previous best result based on five networks uses 950MB.
  • Offline HCCR results: The proposed offline system outperforms compared methods while using a single objective and no data augmentation.The comparison includes systems using deeper networks, distortion-based sample generation, multi-supervised training, or model ensembles.

7.4. Online HCCR Results

Online HCCR results show that directMap+convNet establishes a new benchmark with a single, memory-efficient network. The method also supports high top-N accuracy and competitive performance with limited training data, while GPU use remains important for speed.

  • 97.51% was the best reported online HCCR result before directMap+convNet, achieved using an ensemble of nine networks.
  • DirectMap+convNet sets a new online HCCR benchmark using a single network with better accuracy and memory usage than the best single-network method.
  • Top-2 accuracies exceed 99% for both online and offline HCCR, supporting later language-model correction in handwritten Chinese text recognition.
  • The system is faster than the ICDAR-2013 winners for online and offline HCCR and uses memory efficiently enough for environments such as handheld devices.
  • 96.55% online-related comparison performance is obtained with only HWDB1.1 using a single network without data augmentation.

7.7. Comparison of Different Representations

The paper argues that representation design strongly affects HCCR performance and efficiency. Integrating domain knowledge into directMap reduces variation and enables compact networks without data distortion, while preprocessing choices have smaller effects.

  • Domain-specific representation reduces within-class variation, allowing smaller convNets and eliminating the need for data distortion.
  • The 8 × 32 × 32 directMap is more compact than competing representations using 17 offline feature maps or 30 online feature maps.
  • A preprocessing mismatch between Matlab training and OpenCV testing caused one competing system to lose first place in the ICDAR-2013 offline competition.
  • Nonlinear gray-level normalization performs slightly better than linear or absent normalization, but the advantage is not significant.

7.9. Effectiveness of Dropout

Dropout is evaluated as a way to control overfitting in the deep HCCR network. Without dropout, training accuracy reaches 100% quickly but test performance suffers, whereas larger networks with dropout perform better than smaller ones without it.

  • Without dropout, training accuracy reaches 100% after 30 epochs, but test accuracy is not promising because of overfitting.
  • The network uses smaller dropout probabilities in bottom layers and larger probabilities at greater depth because lower layers are harder to train.
  • A large network with dropout outperforms a small network without dropout because dropout acts like an ensemble of many subnetworks.

7.11. Model Ensemble Results

The study compares model ensembling with writer adaptation and finds that adaptation provides a more efficient route to personalized high-accuracy recognition. It also shows that normalization-cooperated directMaps outperform normalization-based directMaps because they preserve original stroke directions while benefiting from normalization.

  • Writer-Level Results: The figures report error reduction rates for 60 writers in offline and online HCCR.These writer-level comparisons are presented separately for the two recognition settings.
  • Model Ensemble Results: Model ensemble improves classification accuracy, but gains from adding a third model are already vanishing.Using more models increases processing time and memory usage; even two models double both resources.
  • Model Ensemble Results: Writer adaptation is more efficient and effective than model ensemble for high-accuracy personalized handwriting recognition.The comparison favors a single adapted network over many networks in practical applications.
  • DirectMap Comparison: Normalization-cooperated directMaps significantly outperform normalization-based directMaps when training the offline HCCR convNet.They preserve original stroke direction while also benefiting from shape normalization, avoiding the stroke-shape distortion introduced by normalization.

8. Conclusion

The paper combines domain-specific directMaps with deep convolutional networks to set new online and offline HCCR benchmarks. It further shows that an efficient adaptation layer improves deep-network recognition for individual writers, while positioning these components for broader handwritten-text and visual-recognition tasks.

  • Conclusion: Integrating directMap with an 11-layer convNet sets new benchmarks for online and offline HCCR on ICDAR-2013.The framework achieves the highest accuracies without data augmentation or model ensemble and uses less memory than previous state-of-the-art methods.
  • Conclusion: Writer adaptation consistently and significantly improves deep-network HCCR despite deep learning methods already surpassing traditional approaches.The adaptation layer reduces training–test mismatch and can operate with small adaptation datasets in an unsupervised manner.
  • Conclusion: Further progress is still possible through improved convNet architectures and training algorithms.The conclusion specifically identifies super-deep networks and better training procedures as future directions.
  • Conclusion: The proposed directMap, convNet, and adaptation components are intended for extension to handwritten Chinese text recognition and related document-analysis tasks.The paper also mentions possible applications in document retrieval, natural-scene text recognition, and historical-document analysis.
Loading 1606.05763v1…