Source-linked AI summary
Multi-column Deep Neural Networks for Image Classification
Dan Cireşan, Ueli Meier, Juergen Schmidhuber
TL;DR
Recognizing handwritten characters and traffic signs remained difficult for traditional computer-vision and machine-learning methods, while training deep networks was computationally expensive. The paper introduces multi-column convolutional networks trained with GPU acceleration and reports state-of-the-art results across image-classification benchmarks, including a 0.23% MNIST error rate near human performance.
Problem
Traditional computer-vision and machine-learning methods struggled to match human performance, while training deep networks required substantial computational resources.
Method
The method combines deep convolutional winner-take-all neural columns trained on differently preprocessed inputs, averages their predictions, and uses GPU acceleration for training.
Results
The method set improved records across multiple image-classification benchmarks, improving the state of the art by 30–80% and reaching a 0.23% MNIST error rate.
Takeaways & Limitations
The results demonstrate human-competitive performance on widely used computer-vision benchmarks using fully supervised training without additional unlabeled data.
Takeaways & Limitations
Recognition remains difficult for visually similar characters and for separating letters from similar-looking digits.
Abstract
from arXiv · showhide
Traditional methods of computer vision and machine learning cannot match human performance on tasks such as the recognition of handwritten digits or traffic signs. Our biologically plausible deep artificial neural network architectures can. Small (often minimal) receptive fields of convolutional winner-take-all neurons yield large network depth, resulting in roughly as many sparsely connected neural layers as found in mammals between retina and visual cortex. Only winner neurons are trained. Several deep neural columns become experts on inputs preprocessed in different ways; their predictions are averaged. Graphics cards allow for fast training. On the very competitive MNIST handwriting benchmark, our method is the first to achieve near-human performance. On a traffic sign recognition benchmark it outperforms humans by a factor of two. We also improve the state-of-the-art on a plethora of common image classification benchmarks.
IDSIA / USI-SUPSI
IDSIA is a joint institute of USI and SUPSI, founded in 1988 by the Dalle Molle Foundation. The section identifies the paper and its authors as Ueli Meier and Jürgen Schmidhuber.
- Institutional affiliation: The institute is located at Galleria 2, 6928 Manno, Switzerland.
- Institutional affiliation: IDSIA is jointly affiliated with the University of Lugano (USI) and the University of Applied Sciences of Southern Switzerland (SUPSI).The institute was founded in 1988 by the Dalle Molle Foundation, which promoted quality of life.
- Paper identification: The paper is titled “Multi-column Deep Neural Networks for Image.”
- Paper identification: The listed authors are Ueli Meier and Jürgen Schmidhuber.
1 Introduction
The paper trains deep convolutional neural networks by simple online back-propagation and reports greatly improved records across several image-classification benchmarks. It addresses the computational burden of training large, deep networks, which can require weeks to years on CPUs.
- 1 Introduction: Simple online back-propagation yields greatly improved records on MNIST, Latin letters, Chinese characters, traffic signs, NORB, and CIFAR10 benchmarks.The reported benchmarks include jittered, cluttered NORB.
- 1 Introduction: Deep convolutional neural networks have demonstrated strong performance on handwritten digits, handwritten characters, 3D toys, and faces.The passage states that DNNs fully unfold their potential when they are big and deep.
- 1 Introduction: Training big and deep DNNs on CPUs can require weeks, months, or even years.High data-transfer latency also limits the benefits of multi-threading and multi-CPU code.
2 Architecture
The architecture uses deep, biologically inspired networks with 6-10 nonlinear layers, winner-take-all processing, online training of only winner neurons, and democratically averaged predictions from multiple columns.
- Depth: 6-10 nonlinear layers are stacked to create a deep architecture comparable to the layers between macaque retina and visual cortex.The design is inspired by the Neocognitron and contrasts with shallow neural networks.
- Computation: 50-100× additional speedup over serial standard-computer code is obtained through carefully designed massively parallel GPU implementations.The paper also notes that contemporary computers were more than 60000 times faster than those of the early 1990s.
- Winner-take-all processing: Winner-take-all neurons use overlapping receptive fields with shared weights, while max pooling selects the most active neuron in each local inhibition region.Winning neurons form smaller, down-sampled layers with lower resolution for subsequent processing.
- Receptive fields: Near-minimal 2x2 or 3x3 receptive fields produce near-maximal depth of layers with non-trivial two-dimensional winner-take-all regions.After down-sampling reaches a one-dimensional layer, the hierarchy becomes a standard multilayer perceptron.
- Training: Only winner neurons are trained through fully online updates after each gradient computation step, reducing synaptic changes and energy consumption.Other neurons cannot forget what they have learned so far, although peripheral weight changes may affect them.
- Multi-column ensemble: Predictions from several randomly initialized DNN columns are democratically averaged, with differently preprocessed inputs reducing error rate and required column count.Columns may be trained on the same inputs or on inputs preprocessed in different ways.
3 Experiments
Across handwritten, Latin, Chinese, traffic-sign, and object-recognition benchmarks, MCDNNs improve recognition performance through multiple specialized columns and preprocessing schemes. The experiments report near-human MNIST performance, large gains on difficult character tasks, human-beating traffic-sign recognition, and state-of-the-art error rates on additional benchmarks.
- MNIST: 0.23% error on MNIST improves the state of the art by at least 34% and approaches the ≈0.2% human error rate.The 35-column MCDNN outperforms constituent DNNs except for original images.
- MNIST: 0.52% error from one DNN trained on all seven datasets is worse than both MCDNNs and individual DNNs, isolating the benefit of the MCDNN.A 60-net MCDNN reaches 0.24%, similarly to the 35-net model, indicating additional preprocessing does not further improve recognition.
- Latin characters: 1.5-5 times better recognition rates than any published result are achieved on all evaluated Latin-character tasks.The test set contains 82,000 characters, including 58,000 digits and 24,000 letters; digit and letter error rates are 3% and 33%, respectively.
- Traffic signs: 0.54% test error makes the 25-column traffic-sign MCDNN the only artificial method reported to outperform humans, who produced twice as many errors.Five DNNs are trained for each dataset using preprocessing designed to address illumination and contrast variation.
- Object recognition benchmarks: 2.70% error further improves the state of the art on NORB, while considering second predictions reduces the error to 0.42%.The model processes one sample in 1.3ms; 84% of errors have a correct second prediction, with over half involving car–truck confusions.
4 Conclusion
The MCDNN achieves human-competitive results across widely used computer-vision benchmarks, improving the state of the art by 30–80% on many image-classification datasets. Single DNNs already suffice for new state-of-the-art results, while combining them into MCDNNs produces further performance gains without additional unlabeled data.
- Conclusion: Human-competitive results are reported for the first time on widely used computer-vision benchmarks.The paper’s conclusion characterizes these benchmark results as human-competitive.
- Conclusion: Single DNNs obtain new state-of-the-art results, while combining them into MCDNNs yields further dramatic performance boosts.The comparison is made within the same fully supervised approach.
- Conclusion: The method is fully supervised and uses no additional unlabeled data source.This specifies the training-data condition accompanying the reported results.
5 Supplementary Material
The supplementary analyses identify characteristic confusion patterns across character-recognition tasks and describe contrast-normalization and network-architecture choices for image classification. Errors commonly arise from visually similar characters, while context and preprocessing are potential remedies.
- Character confusion analysis: Most errors in the 62-class task involve digit–letter and lowercase–uppercase confusions.The experiments also identify digits 0 and 1 as difficult to separate from letters O and I.
- Character confusion analysis: Four–nine confusions are the most common errors in the digit task.
- Character confusion analysis: Case confusions dominate the 52-letter task, with additional difficulty distinguishing q–g and l–i.For uppercase and lowercase tasks, errors primarily involve similarly shaped letters such as D–O, V–U, g–q, and v–u.
- Character confusion analysis: 1.82% is the total error for uppercase letters, versus 7.47% for lowercase letters.The uppercase task struggles with similarly shaped letters, while the lowercase task has analogous shape-based confusions.
- Character confusion analysis: 7.99% is the merged-case task’s error, shrinking by almost threefold as identical-letter case confusions disappear.The merged-case task contains 37 classes and mainly confuses similarly shaped letters such as l and i.
- Preprocessing and architectures: Four contrast normalizations are tested: Image Adjustment, Histogram Equalization, Adaptive Histogram Equalization, and Contrast Normalization.Adaptive equalization uses 8 nonoverlapping 6x6-pixel tiles, while Contrast Normalization applies a 5x5-pixel difference-of-Gaussians filter; the other methods operate in Lab space before conversion back to RGB.