Source-linked AI summary
Spatially-sparse convolutional neural networks
Benjamin Graham
TL;DR
Deep CNNs can be costly to train and test, especially for sparse online handwriting inputs and challenging recognition tasks. The paper develops spatially sparse CNNs that exploit sparse representations and padding, applies them to handwriting and image recognition, and enables architectures otherwise rejected as too slow.
Problem
Training and testing deep CNNs is limited by budget and time constraints, motivating efficient processing for challenging handwriting and image-recognition tasks.
Method
The paper develops spatially sparse CNNs that exploit sparse input representations and padding, including deep architectures with network-in-network layers.
Results
The approach was applied to handwriting and image recognition and allowed use of CNN architectures that would otherwise have been rejected as too slow.
Takeaways & Limitations
Spatial sparsity enables efficient processing of sparse inputs while supporting substantial increases in spatial resolution for challenging recognition datasets.
Takeaways & Limitations
Meaningful comparison with other papers is difficult because data-augmentation practices differ; the authors restrict augmentation for some evaluations.
Abstract
from arXiv · showhide
Convolutional neural networks (CNNs) perform well on problems such as handwriting recognition and image classification. However, the performance of the networks is often limited by budget and time constraints, particularly when trying to train deep networks. Motivated by the problem of online handwriting recognition, we developed a CNN for processing spatially-sparse inputs; a character drawn with a one-pixel wide pen on a high resolution grid looks like a sparse matrix. Taking advantage of the sparsity allowed us more efficiently to train and test large, deep CNNs. On the CASIA-OLHWDB1.1 dataset containing 3755 character classes we get a test error of 3.82%. Although pictures are not sparse, they can be thought of as sparse by adding padding. Applying a deep convolutional network using sparsity has resulted in a substantial reduction in test error on the CIFAR small picture datasets: 6.28% on CIFAR-10 and 24.30% for CIFAR-100.
1 Introduction
The paper introduces spatially sparse CNNs for online character recognition, using sparsity to reduce computation and enable more flexible padding and data preparation. It extends this idea to image recognition and deep architectures.
- CNNs process three-dimensional feature arrays through convolution and pooling before softmax classification.The spatial dimensions are represented by N × N, while M stores features at each location.
- Sparse inputs can accelerate convolutional computation and provide greater freedom in preparing input data.As layers advance, spatial size decreases while the number of features increases.
- Online isolated character recognition represents touchscreen or stylus input as paths and is challenging for languages with large character sets.The task can serve as a building block for reading cursive handwriting.
- Handwritten characters become sparse binary images because they contain O(N) non-zero pixels among N^2 total pixels.This sparsity allows the first hidden layer to be calculated much more quickly.
- Conventional valid-mode convolution poorly handles image boundaries, motivating padding through input augmentation, layer padding, or related approaches.The listed strategies include zero-padding inputs and adding padding to convolutional layers.
- Spatial sparsity can combine these advantages by evaluating the whole object at once while adding substantial padding at no extra cost.The paper presents this as a motivation for spatially sparse convolutional networks.
2 Deep convolutional networks and spatial sparsity
The paper develops deep CNN architectures that combine slow max-pooling with sparse computation, preserving spatial information while reducing work for sparse inputs. It also extends these networks with Network-in-Network layers and combines high-resolution location and direction representations for online character recognition.
- DeepCNet(ℓ, k): Slow max-pooling retains more spatial information, which is better suited to highly structured handwriting, but is computationally expensive for general inputs.For sparse inputs, preserved sparsity in early hidden layers offsets this cost and supports operation on low-power tablet computers.
- DeepCNet(ℓ, k): DeepCNet(ℓ, k) alternates ℓ+1 convolutional layers with ℓ 2 × 2 max-pooling layers, using 3 × 3 filters initially and 2 × 2 filters thereafter.The n-th convolutional layer uses nk filters, and the input size is chosen so the final convolution produces a fully connected hidden layer.
- DeepCNiN: DeepCNiN adds 1 × 1 Network-in-Network layers after each max-pooling layer and the final convolution without changing spatial structure substantially.Dropout is applied only to convolutional layers, and leaky rectified linear units use x/3 for negative inputs rather than x/100.
- Spatial sparsity for convolutional networks: Sparse forward propagation computes hidden variables only where they differ from a ground state, using feature and pointer matrices to represent active locations.The feature matrix stores ground-state and active-location vectors, while the pointer matrix maps spatial locations to feature rows.
- Online Character Recognition: Combining high-resolution pen-location images with directional histograms yields a sparse (1+8)×n×n representation that preserves both location and stroke-direction information.With n = 64, the representation is suitable for CNN input, while added features have negligible performance impact for sparse inputs because they mainly affect the first hidden layer.
3 Results
The results evaluate sparse convolutional networks across handwriting and image-recognition settings, emphasizing augmentation, network parameters, and performance under computational constraints.
- Handwriting recognition: Increasing the input scale through additional max-pooling levels changes the number of active spatial locations; at ℓ = 6, about 6% of sites are active.
- Handwriting recognition: Training-data translations are crucial for Assamese recognition, while more general affine transformations also improve results.
- Handwriting recognition: Adding input features is relatively cheap for sparse CNNs because it mainly increases the cost of evaluating the first layer, and it can improve generalization.
- CASIA recognition: CASIA experiments show that sparsity supports good performance at relatively low computational cost, while larger networks provide good performance.
- CASIA recognition: 2.61% test error won the ICDAR2013 task 3 competition, compared with 3.13% for the second-best entry and 4.81% reported human performance.
- Offline image recognition: DeepCNet(5, 60) achieved a 0.31% MNIST test error, while DeepCNiN(5,300) achieved 6.28% on CIFAR-10 and 24.30% on CIFAR-100.
4 Conclusion
The paper demonstrates spatially sparse CNNs for handwriting and image recognition, enabling CNN architectures that would otherwise be too slow. It also identifies extensions to sparse representations, segmentation, and higher-dimensional convolution.
- Spatially sparse CNNs apply to handwriting and image recognition, enabling architectures otherwise rejected as too slow.The authors report implementation and application of the approach across both task types.
- The online-character system extends dense 8-directional histograms into a sparse representation.The authors note that other sparse representations, such as path curvature features, could be explored.
- Larger sparse CNNs could make training on uncropped images and more flexible data augmentation practical, while potentially improving accuracy or efficiency.This is presented as a potential benefit relative to related large CNNs trained with multiple GPUs or computer networks.
- Sparse CNNs could classify irregular foreground and background segments efficiently and test whether neighboring segments form a larger object.This proposed use combines sparsity with image segmentation.
- Higher-dimensional sparse CNNs could analyze 3D objects and objects moving through 2+1 or 3+1 dimensional space-time.The paper motivates this extension because lines and surfaces form sparse sets when embedded in higher-dimensional spaces.