Source-linked AI summary
Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
Dan Claudiu Ciresan, Ueli Meier, Luca Maria Gambardella, Juergen Schmidhuber
TL;DR
Plain, deep MLPs have been difficult to evaluate on MNIST because training them is computationally demanding. The paper trains large networks with online back-propagation and reports a 0.35% error rate, using hardware acceleration and extensive image deformation.
Problem
The paper asks whether complex MNIST methods are necessary when plain MLPs are trained at much larger depth and scale.
Method
The paper trains large, deep plain MLPs with online back-propagation, numerous deformed training images, and graphics-card acceleration.
Results
0.35% error was achieved on the MNIST handwritten-digit benchmark.
Takeaways & Limitations
The results show that ongoing hardware progress may enable training big and deep neural networks with online back-propagation.
Takeaways & Limitations
Single-precision SSE offers a maximum theoretical speedup of four, while MNIST’s 60,000 images exceed processor cache capacity.
Abstract
from arXiv · showhide
Good old on-line back-propagation for plain multi-layer perceptrons yields a very low 0.35% error rate on the famous MNIST handwritten digits benchmark. All we need to achieve this best result so far are many hidden layers, many neurons per layer, numerous deformed training images, and graphics cards to greatly speed up learning.
1 Introduction
The introduction asks whether plain, deep MLPs can match more complex MNIST methods and explains why testing them has been difficult. It proposes overcoming the computational barriers with large, deep MLP training.
- Research question: The central question is whether complexifications of plain MLPs are necessary or whether sufficiently large plain MLPs can perform well on MNIST.The introduction specifically asks why large plain MLPs have received little attention.
- Training challenge: Deep MLPs are difficult to train because back-propagated gradients quickly vanish as the number of layers increases.Earlier successful deep networks used weight sharing or unsupervised layer-wise pre-training.
- Contribution: The paper presents a way to train large, deep MLPs despite these optimization and hardware obstacles.The stated approach uses large, deep MLPs and is introduced as overcoming the preceding problems.
2 Data
The experiments use MNIST’s standard training and test datasets, while online deformation allows the full undeformed training set to serve as validation. Images are normalized pixel arrays fed into the network.
- Dataset: MNIST contains 60,000 training images and 10,000 test images.Many studies instead split the training set into 50,000 training and 10,000 validation images.
- Validation: Continually generated deformed training images allow the entire undeformed training set to be used for validation without withholding training images.The deformations are generated online during training.
- Preprocessing: Each 28 × 28 image has 784 pixels, whose intensities are mapped from [0, 255] to real values in [−1.0, 1.0].The normalized pixel values are fed into the neural-network input layer.
3 Architectures
The study trains five plain MLPs spanning depths from 2 to 9 hidden layers and sizes from 1.34 to 12.11 million parameters. All use standard online back-propagation with a decaying learning rate and scaled tanh activations.
- Architectures: Five MLPs contain 2 to 9 hidden layers with varying numbers of hidden units.The number of hidden units usually decreases toward the output layer.
- Architectures: 1.34 to 12.11 million free parameters are used across the networks.The paper refers to these parameters equivalently as weights or synapses.
- Activation: Each neuron uses the scaled hyperbolic tangent y(a) = A tanh Ba, with A = 1.7159 and B = 0.6666.The activation specification follows the cited MLP design.
4 Deforming images to get more training instances
The training pipeline continually deforms MNIST images to create additional instances and improve robustness to within-class variation. It combines affine transformations with elastic distortions controlled by explicit parameters.
- Deformation types: The method combines rotation, scaling, horizontal shearing, and elastic deformations.Elastic distortions emulate uncontrolled oscillations of hand muscles.
- Affine parameters: β is a random angle in [−β, +β] for rotation or horizontal shearing, while tan β gives the horizontal-displacement ratio for shearing.The deformation parameters are sampled to vary the training images.
- Scaling parameters: Horizontal and vertical scaling use γx and γy, each randomly selected around 1 within a range determined by γ.The cited parameterization expresses scaling variability around the original image size.
- Online augmentation: Training images are deformed at the beginning of every epoch, continually increasing the number of training instances.This supports training networks with many weights and aims to make them insensitive to in-class variability.
5 Results
The plain MLP achieves very low MNIST error rates when scaled to many layers and parameters, with continual deformations supplying training variation. GPU acceleration makes training and deformation substantially faster.
- GPU acceleration speeds deformation by a factor of 10 and forward- and back-propagation by a factor of 40.
- 0.35% is the best network’s MNIST test error, corresponding to 35 errors out of 10,000 digits.
- 0.32% is the MLP’s best test error and may represent the network’s maximum capacity.
- Adding hidden layers and more units per layer improves performance across the evaluated networks.
- Networks with up to 12 million weights reach test errors below 1% after 20–30 epochs in under 2 hours of training.
- Continual training-image deformations generate a virtually infinite supply of examples, so the network rarely sees an identical training image.
Conclusion
GPU-based plain neural networks surpass previously reported MNIST results, including results from more complex methods. The authors argue that continuing hardware progress may matter more than advances in algorithms and software, while the approach may extend beyond handwriting.
- The results suggest ongoing hardware progress may be more important than advances in algorithms and software.
- Current GPUs are already more than 40 times faster than standard microprocessors for training big and deep neural networks with on-line back-propagation.
- GPU-based neural networks surpass all previously reported results on the competitive MNIST benchmark.
- The approach is not limited to handwriting and may apply to many visual and other pattern-recognition problems.
Graphics Processing Unit
CUDA enabled general-purpose GPU implementations after earlier GPU neural-network studies showed only modest speedups. GPUs provide much higher speed and memory bandwidth than CPUs, supporting fast MLP implementations.
- Earlier GPU implementations covered two-layer MLPs and CNNs, but their speedups were relatively modest.
- More recent batch-trained GPU CNNs are two orders of magnitude faster than CPU-based CNNs.
- CUDA, introduced by NVIDIA in 2007, provided a general programming framework for GPUs.
- GPU speed and memory bandwidth are vastly superior to those of CPUs and are crucial for fast MLP implementations.
- In CUDA terminology, the CPU is the host and the graphics card is the device or GPU.
Deformations
Elastic image deformations enlarge MNIST training data, and the most expensive convolution step is ported to the GPU. This reduces deformation time from 93 to 9 seconds, while deformation remains a small fraction of epoch computation.
- The Gaussian-kernel convolution used for elastic distortions is ported to the GPU.
- The MNIST training set uses 600 sequentially processed batches and scales digits from 28 × 28 to 29 × 29 pixels.
- The deformation grid is zero-padded to 300 × 300 so a 21 × 21 Gaussian convolution kernel avoids margin effects.
- GPU blocks share the Gaussian kernel and random-field parts, with 21 × 10 threads computing vertical convolution strips.
- Generating the elastic displacement field takes 3 seconds, while deforming the full training set takes 9 seconds instead of 93.
- Deformations consume only 5–15% of one epoch’s total computation time, depending on MLP size.
Training algorithm
The implementation follows standard back-propagation while separating delta propagation and weight updates to increase parallelism. GPU kernels use tiled reductions and shared memory, achieving substantial throughput but reduced utilization for small final layers.
- Back-propagation of deltas and weight updates are disentangled and performed sequentially, allowing more parallelism within each routine.
- Forward propagation: Forward propagation partitions the weight matrix across blocks, computes partial dot products in parallel, stores them temporarily, and reduces them into neuron activations.Each block has 256 threads, with each thread computing a partial dot product of 32-component vectors.
- Forward propagation: 115GB/s average memory bandwidth is achieved because many relatively small blocks keep the GPU busy.
- Forward propagation: Shared memory stores previous-layer activations for reuse by all threads, while precomputed indices and minimized instructions reduce kernel overhead.The first 32 threads read activations that are then used by all 256 threads in the block.
- Back-propagating deltas: Back-propagating deltas uses two reduction kernels that communicate partial results through global memory, sum them, apply the activation derivative, and write new deltas.Weight patches and input-delta patches are read into shared memory before partial results are computed and written out.