Source-linked AI summary
Building Fast and Compact Convolutional Neural Networks for Offline Handwritten Chinese Character Recognition
Xuefeng Xiao, Lianwen Jin, Yafeng Yang, Weixin Yang, Jun Sun, Tianhai Chang
TL;DR
Portable deployment of CNN-based HCCR is limited by computational cost and parameter storage. The paper combines Global Supervised Low-rank Expansion, Adaptive Drop-weight pruning, and Connection Redundancy Analysis to address both constraints. The resulting system reduces computation ninefold and parameter storage eighteenfold with a 0.21% accuracy drop, while using 2.3 MB and processing a character in 9.7 ms on a CPU.
Problem
CNN-based HCCR offers strong recognition performance, but computational cost and storage requirements hinder deployment on portable devices.
Method
The paper combines Global Supervised Low-rank Expansion for convolutional acceleration with Adaptive Drop-weight pruning and Connection Redundancy Analysis for compact HCCR models.
Results
9× lower computational cost and 18× lower parameter storage were achieved with a 0.21% accuracy drop, while the model required 2.3 MB and processed an offline character in 9.7 ms on a CPU.
Takeaways & Limitations
The framework produced a fast and compact CNN for large-scale HCCR involving 3,755 Chinese-character classes.
Takeaways & Limitations
Connection Redundancy Analysis evaluates layers separately, making all-layer joint pruning difficult; its goal of maximizing compression under tolerable accuracy loss requires further research.
Abstract
from arXiv · showhide
Like other problems in computer vision, offline handwritten Chinese character recognition (HCCR) has achieved impressive results using convolutional neural network (CNN)-based methods. However, larger and deeper networks are needed to deliver state-of-the-art results in this domain. Such networks intuitively appear to incur high computational cost, and require the storage of a large number of parameters, which renders them unfeasible for deployment in portable devices. To solve this problem, we propose a Global Supervised Low-rank Expansion (GSLRE) method and an Adaptive Drop-weight (ADW) technique to solve the problems of speed and storage capacity. We design a nine-layer CNN for HCCR consisting of 3,755 classes, and devise an algorithm that can reduce the networks computational cost by nine times and compress the network to 1/18 of the original size of the baseline model, with only a 0.21% drop in accuracy. In tests, the proposed algorithm surpassed the best single-network performance reported thus far in the literature while requiring only 2.3 MB for storage. Furthermore, when integrated with our effective forward implementation, the recognition of an offline character image took only 9.7 ms on a CPU. Compared with the state-of-the-art CNN model for HCCR, our approach is approximately 30 times faster, yet 10 times more cost efficient.
1. Introduction
Offline HCCR has reached strong performance with CNNs, but their computational and storage demands hinder portable deployment. This paper combines low-rank expansion, adaptive pruning, and redundancy analysis to build a faster, more compact classifier.
- CNNs achieve state-of-the-art HCCR performance, but computational cost and storage requirements hinder portable-device deployment.
- Existing acceleration and compression methods rarely combine speed and storage optimization, and their feasibility for HCCR with more than 3,700 classes remained underexplored.
- The proposed framework unifies low-rank expansion for convolutional acceleration with pruning to remove redundant connections and reduce parameter storage.
- Adaptive Drop-weight dynamically increases pruning thresholds, while Connection Redundancy Analysis guides layer-specific pruning ratios under an accuracy constraint.
- 9× lower computational cost and 18× lower parameter storage were achieved with a 0.21% accuracy drop, while surpassing the reported best single-network CNN on ICDAR 2013 HCCR.
2. Related Work
Related work established strong CNN performance for offline HCCR and explored specialized acceleration and compression techniques. The paper targets the remaining need to jointly reduce convolutional computation and network storage.
- CNN-based HCCR progressed from MCDNN and competition systems to models reporting accuracy above 96%.
- The proposed framework combines low-rank expansion with network pruning to address computational cost and storage capacity together.
- Convolutional layers typically dominate computation, whereas fully connected layers contain most parameters, motivating separate acceleration and compression strategies.
- Fast convolution methods include Strassen multiplication, FFTs, and Winograd filtering, while im2col with BLAS supports efficient CPU matrix multiplication.
- Fully connected layers have been compressed using hashing, fixed-point or binary representations, pruning, quantization, and Huffman coding.
3. Architecture of Convolutional Neural Network
The baseline HCCR-CNN9Layer is a nine-layer CNN with seven convolutional and two fully connected layers, using BN and PReLU before final classification over 3,755 classes.
- The network has seven convolutional layers and two fully connected layers, with pooling arranged after the first three and then after each pair of convolutions.
- The first fully connected layer contains 1,024 neurons, and the final layer contains 3,755 neurons for classification.
- PReLU supports convergence and reduces overfitting risk, while batch normalization stabilizes nonlinear inputs and permits higher learning rates.
- The BN- and PReLU-equipped baseline is named HCCR-CNN9Layer and achieves state-of-the-art HCCR performance despite its simple CNN structure.
4. Adaptive Drop-Weight
Adaptive Drop-weight gradually removes redundant connections by increasing pruning thresholds, while layer-specific redundancy analysis determines suitable pruning ratios. Weight quantization further compresses the pruned network.
- Adaptive Drop-weight gradually prunes weighted connections by dynamically increasing each layer’s pruning threshold.
- Fixed thresholds can prune too aggressively and sharply reduce performance or prune too conservatively and miss the desired compression ratio.
- Connections are pruned every I iterations by increasing the pruned count toward a target ratio, while pruned weights remain zero during later updates.
- K-means weight quantization is applied after pruning, followed by fine-tuning that may improve network performance.
- The pruning algorithm updates nonzero weights, selects a threshold from the smallest absolute weights, and sets weights below that threshold to zero.
5. Connection Redundancy Analysis
Connection Redundancy Analysis evaluates each layer’s redundant connections separately to set layer-specific pruning ratios while keeping accuracy loss tolerable.
- Deep networks contain varying redundancy across layers, especially between convolutional and fully connected layers, so a uniform pruning ratio is inappropriate.
- Prior pruning methods used a fixed weight threshold, requiring experiments to tune threshold parameters for each setting.
- CRA analyzes each layer’s redundancy to determine a suitable pruning ratio r_i for that layer.
- The method fixes other layers, progressively prunes one layer with Adaptive Drop-weight, and uses accuracy degradation to guide further pruning.
- CRA’s layer-wise procedure cannot directly analyze simultaneous pruning of all layers, so maximizing compression under tolerable accuracy loss remains for further research.
6. Grobal Supervised Low Rank Expansion
The decomposition scheme replaces a convolution with sequential vertical and horizontal low-rank filters, while GSLRE trains each decomposition using the network’s global label-guided loss. This reduces computation, but the method’s training rationale and layer-wise procedure are central to preserving classification performance.
- 6.1. Decomposition Scheme: The original layer maps X ∈ R^C×H×W to Y ∈ R^N×H′×W′ using a 4D K × K kernel W.
- 6.1. Decomposition Scheme: O(CNK^2H′W′) is the computational cost of direct convolution.
- 6.1. Decomposition Scheme: Low-rank expansion replaces the square filter with sequential vertical and horizontal filters.
- 6.1. Decomposition Scheme: The first convolution uses T ∈ R^C×D×K×1 and costs O(CDKH′W).
- 6.1. Decomposition Scheme: The intermediate map M is convolved with V ∈ R^D×N×1×K, and the second convolution costs O(NDKH′W′).
- 6.1. Decomposition Scheme: Together, the two low-rank convolutions cost O(DKH′(NW′ + CW)), with D chosen according to the desired acceleration factor.
- 6.2. Training scheme: GSLRE uses the true label and the entire network’s loss to train each low-rank expansion layer by layer, unlike local supervision based on reconstruction error.
- 6.2. Training scheme: During layer-specific training, other convolutional layers remain fixed, while batch normalization helps gradients pass into lower layers.
7. Experiment
Experiments evaluated GSLRE and ADW across MNIST and offline HCCR, measuring acceleration, compression, accuracy, redundancy, and CPU runtime. The combined framework substantially reduced computational cost and storage while retaining competitive recognition accuracy.
- Global Supervised Low-rank Expansion: Fourfold acceleration was achieved for the baseline network with negligible accuracy loss using Global Supervised Low-rank Expansion training.The decomposing training scheme also outperformed direct training of the decomposed architecture.
- Adaptive Drop-weight: 250× and 113× compression ratios were obtained for LeNet-5 and LeNet-300-100, respectively, without accuracy loss after pruning and weight quantization.Adaptive Drop-weight alone achieved 133× and 60× compression for the two networks.
- Adaptive Drop-weight: Fourfold-accelerated offline HCCR was compressed to approximately one-quarter of its size with ADW, then reduced approximately 14-fold after weight quantization with a 0.18% accuracy drop.The evaluated network contained 13 convolutional and two fully connected layers.
- Connection Redundancy Analysis: Connection Redundancy Analysis guided layer-specific pruning by allowing a 0.1% accuracy drop per layer, while each layer’s pruning ratio reflected its redundancy.Accuracy was initially less sensitive to pruning, then decreased sharply at higher pruning ratios.
- Accuracy and efficiency: The nine-layer HCCR network reached 97.30% accuracy, while GSLRE and ADW reduced computational cost ninefold and parameter storage 18-fold with a 0.21% drop.The compressed model surpassed the best single-network result reported in the comparison and used lower storage and computational cost.
- Accuracy and efficiency: A larger HCCR-CNN12Layer achieved 97.59% accuracy and, after GSLRE and ADW, retained this performance with 16-fold lower computational cost and 10-fold lower parameter storage, losing 0.19% accuracy.Forward implementation used BLAS-based dense operations and sparse matrix-vector multiplication for fully connected layers; insufficient sparsity could reduce performance.
8. Conclusion
The paper presents an approach for accelerating and compressing CNNs for large-scale HCCR, combining GSLRE, ADW, and CRA. The authors report a fast, compact model and identify applications beyond HCCR as future work.
- The approach targets acceleration and compression of CNNs for HCCR involving 3,755 Chinese-character classes.
- GSLRE accelerates convolutional-layer calculations, while ADW removes redundant connections through dynamically increasing pruning thresholds.
- CRA analyzes layer-wise connection redundancy to guide CNN pruning without compromising network performance.
- 9.7 ms/char recognition and 2.3 MB storage were achieved using compression and sparse matrix-vector multiplication for fully connected layers.
- Future work includes applying the framework to image classification, object detection, and deep recurrent neural networks for online handwritten recognition.