Source-linked AI summary
High-Performance Neural Networks for Visual Object Classification
Dan C. Cireşan, Ueli Meier, Jonathan Masci, Luca M. Gambardella, Jürgen Schmidhuber
TL;DR
The paper addresses CNN computational speed and inflexible architecture design by presenting a fast, flexible GPU implementation. Its networks achieve state-of-the-art results across MNIST, NORB, and CIFAR10, while NORB’s limited training variability constrains generalization.
Problem
CNN computational speed remains limiting, while prior GPU implementations were hard-coded and architectural choices were commonly set by trial and error.
Method
The paper presents flexible, fully parameterizable GPU-based CNN variants with adaptable architecture parameters and max-pooling layers.
Results
The best networks achieved recognition test error rates of 0.35% on MNIST, 2.53% on NORB, and 19.51% on CIFAR10.
Takeaways & Limitations
The implementation combines state-of-the-art generalization capabilities with flexibility and speed across digit, 3D object, and natural-image benchmarks.
Takeaways & Limitations
NORB’s five training instances per class produce poor training-set variability, so networks learn quickly but generalize badly and show inconsistent results across initializations.
Abstract
from arXiv · showhide
We present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants. Our feature extractors are neither carefully designed nor pre-wired, but rather learned in a supervised way. Our deep hierarchical architectures achieve the best published results on benchmarks for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with error rates of 2.53%, 19.51%, 0.35%, respectively. Deep nets trained by simple back-propagation perform better than more shallow ones. Learning is surprisingly rapid. NORB is completely trained within five epochs. Test error rates on MNIST drop to 2.42%, 0.97% and 0.48% after 1, 3 and 17 epochs, respectively.
IDSIA / USI-SUPSI
IDSIA is a joint Swiss institute associated with USI and SUPSI, founded in 1988 by the Dalle Molle Foundation.
- The institute is located in Galleria 2, 6928 Manno, Switzerland.
- IDSIA is jointly associated with the University of Lugano and the University of Applied Sciences of Southern Switzerland.
- The Dalle Molle Foundation founded IDSIA in 1988 to promote quality of life.
- The work was partially supported by the Swiss Commission for Technology and Innovation through Project n. 9688.1 IFF.
- The paper is titled “High-Performance Neural Networks for Visual Object Classification.”
1 Introduction
The paper addresses the computational difficulty of flexible CNN architectures by presenting a fast, fully online GPU implementation evaluated on MNIST, NORB, and CIFAR10.
- Deep hierarchical models learn localized image features through convolution with filters, unlike standard models using fixed filters.
- Computational speed limits CNN research because architectures contain many building blocks often selected by trial and error.
- The implementation is flexible and fully online, enabling systematic exploration of CNN structural parameters.
- The study evaluates networks on MNIST, NORB, and CIFAR10 classification benchmarks.
2 Convolutional neural networks
The implementation uses configurable convolutional, optional image-processing, max-pooling, and classification layers, with learned weights and adaptable connectivity and spatial parameters.
- CNNs alternate convolutional and subsampling layers, with optional max-pooling after each CNN layer in this implementation.
- The optional image-processing layer supplies fixed-filter information such as edges and gradients during training.
- A convolutional layer is parameterized by map counts, kernel sizes, skipping factors, and a connection table.
- 2.3 Max-pooling layer: Max-pooling takes the maximum activation over non-overlapping rectangular regions, providing local position invariance and downsampling.
- 2.4 Classification layer: The top layer is fully connected and contains one output unit per class label.
- 2.4 Classification layer: Figure 1 depicts fully connected convolutional layers using 5 x 5 kernels and skipping factors of 1.
3 GPU implementation
The GPU implementation organizes forward propagation, backward propagation, and weight adjustment around parallel thread grids and data structures for flexible map connectivity, prioritizing efficient handling of shared weights and deltas.
- 3.6 GPU optimizations: Switching from GTX 285 to GTX 480 makes the CNN program 2-3 times faster.
- 3.6 GPU optimizations: The implementation uses precomputed expressions, template-kernel loop unrolling, strided matrices, and registers for optimization.
- 3.5 Data structures: Connection matrices store forward and backward map links, while weight-index arrays locate weights for bias handling.
- 3.1 Forward propagation: GPU kernels exploit parallel threads, shared weights, and shared inputs to compute map neurons efficiently.
- 3.3 Backward propagation: Pulling deltas avoids the serialization and memory-access costs associated with pushing deltas in shared-weight architectures.
- 3.3 Backward propagation: Backward propagation computes each previous-layer delta by aggregating connected current-layer deltas and applying the activation derivative.
- 3.4 Adjusting weights: The adjusting-weights thread grid assigns blocks to map connections and threads to individual kernel weights.
4 Experiments
Experiments evaluate the flexible GPU CNN implementation across MNIST, NORB, and CIFAR10, testing depth, preprocessing, translations, and network size. The networks achieve low error rates while GPU execution substantially accelerates training.
- MNIST: 0.35% MNIST test error was achieved with deeper randomly connected CNNs, while the best test error reached 0.27%.Networks with additional layers performed better than shallower alternatives.
- MNIST: 2.42%, 0.97%, and 0.48% MNIST test errors were obtained after one, three, and seventeen epochs, respectively.A four-hidden-layer network reached below 0.50% test error after 34 epochs.
- NORB: 2.53% NORB average test error was obtained with contrast-extraction preprocessing and translations, improving the previous 2.87% result.The corresponding average error without translation was 3.94%.
- CIFAR10: 19.51% CIFAR10 error was achieved with 300 maps per layer, improving on the previous 20.40% state-of-the-art result.Training time increased exponentially as network width grew, while test error decreased to its minimum at 300 maps per layer.
- GPU speed: More than 60× GPU speedup was obtained for huge networks, with one epoch taking 35 GPU minutes versus more than 35 CPU hours.Speedup was smaller for small networks because they fit better in CPU cache and underused GPU resources.
5 Conclusion
The paper presents flexible, high-performance GPU CNN variants that achieve strong results across MNIST, NORB, and CIFAR10, while requiring labeled data but no unsupervised pretraining.
- 0.35% MNIST, 2.53% NORB, and 19.51% CIFAR10 test error rates were achieved by the reported CNNs.
- The CNNs use sparse random connectivity, making them more computationally efficient and biologically plausible than fully connected CNNs.
- All structural CNN parameters are adaptable, including input size, hidden layers, maps per layer, kernel sizes, skipping factors, and connection tables.
- The networks require a labeled dataset of sufficient size, but no unsupervised pretraining is required.
- Big and deep sparsely connected CNNs are computationally prohibitive on CPUs but feasible on current GPUs.