Source-linked AI summary
A Taxonomy of Deep Convolutional Neural Nets for Computer Vision
Suraj Srinivas, Ravi Kiran Sarvadevabhatla, Konda Reddy Mopuri, Nikita Prabhu, Srinivas S S Kruthiventi, R. Venkatesh Babu
TL;DR
Computer vision had relied heavily on hand-crafted features, creating a need to understand learned deep representations and suitable CNN designs. The paper surveys CNNs from AlexNet through task-specific variants, finding a broad range of adaptations for vision applications while acknowledging coverage and tuning constraints.
Problem
Computer vision traditionally depended on hand-engineered features, while a computer-vision-specific survey of deep CNN architectures was missing.
Method
The paper surveys CNN principles starting from AlexNet and organizes modifications for tasks including fine-tuning, detection, multimodal learning, recurrent integration, captioning, and visual question answering.
Results
The survey describes CNN adaptations that replace complicated hand-tuned algorithms with end-to-end trained monolithic models across computer vision applications.
Takeaways & Limitations
The article is intended as a recipe-style guide for vision researchers and beginning practitioners applying deep learning to computer vision.
Takeaways & Limitations
The paper does not cover the entire range of deep learning research, and detailed unsupervised-learning discussion is beyond its scope.
Abstract
from arXiv · showhide
Traditional architectures for solving computer vision problems and the degree of success they enjoyed have been heavily reliant on hand-crafted features. However, of late, deep learning techniques have offered a compelling alternative -- that of automatically learning problem-specific features. With this new paradigm, every problem in computer vision is now being re-examined from a deep learning perspective. Therefore, it has become important to understand what kind of deep networks are suitable for a given problem. Although general surveys of this fast-moving paradigm (i.e. deep-networks) exist, a survey specific to computer vision is missing. We specifically consider one form of deep networks widely used in computer vision - convolutional neural networks (CNNs). We start with "AlexNet" as our base CNN and then examine the broad variations proposed over time to suit different applications. We hope that our recipe-style survey will serve as a guide, particularly for novice practitioners intending to use deep-learning techniques for computer vision.
1 Introduction
Computer vision traditionally depended on increasingly complex hand-engineered features, while deep learning offered models that learn features from raw images. This survey organizes CNN principles and task-specific modifications as a guide for practitioners.
- Traditional vision systems relied on hand-engineered features such as SIFT, HoG, and Bag-of-visual-words descriptors followed by learning algorithms.Their performance therefore depended crucially on the selected features.
- Early deep learning models learned features through multiple layers of nonlinear transformations and showed promise on small datasets.Examples included Restricted Boltzmann Machines, Deep Belief Networks, and Stacked Autoencoders.
- The paper surveys CNNs beginning with AlexNet, then examines modifications designed for different computer vision problems.It develops general CNN principles, discusses task-specific variations, and identifies open problems and future directions.
2 Introduction to Convolutional Neural Networks
CNNs address the parameter and spatial-structure limitations of fully connected image models through local, shared convolutions, pooling, and nonlinearities. Their design choices must match the underlying image problem.
- A fully connected model for a 200 × 200 image with 20,000 hidden units requires 800 Million input weights and ignores 2D spatial structure.
- Convolutions make image modeling tractable by learning small filters while naturally preserving two-dimensional structure.
- Local connectivity restricts filters to small image regions, while weight sharing applies the same local filters across many locations.
- Location-independent convolutional detectors are not fully suitable for registered images, where different spatial locations may require different detectors.Locally connected networks without weight sharing address this setting.
- Max-pooling selects the strongest local activation for limited spatial invariance and reduces the next activation size by a factor of n^2.
- Nonlinearities between convolutional layers increase expressiveness beyond a linear model, with modern CNNs commonly using ReLU.ReLU networks were reported to train faster than networks using traditional sigmoid or tanh nonlinearities.
2.2 Depth
CNN depth improves representational efficiency and performance, but successful training also depends on supervised optimization, regularization, augmentation, and ensembling.
- A deep network can represent some functions with far fewer neurons than a shallow network, and deeper networks have been shown to perform better in training studies.
- Greedy layerwise pre-training enabled training networks deeper than three layers, while supervised training later proved sufficient for powerful computer vision models.
- CNN classification commonly uses softmax or sigmoid cross-entropy losses, whereas regression uses Euclidean error.
- SGD approximates gradient descent for large datasets and has been found to generalize better, while momentum accelerates its convergence.
- Dropout combats overfitting by randomly removing neurons during training and scaling activations when all neurons are used at test time.The procedure was described as training an ensemble of networks with shared parameters.
- Data augmentation typically boosts performance by about 3%, while CNN ensembles typically improve accuracy by 1-2%.
2.5 Putting it all together: AlexNet
AlexNet combines convolutional, pooling, normalization, dropout, and fully connected layers, and its pretrained weights enabled adaptation to new tasks. Its success motivated deeper and differently designed CNNs.
- AlexNet: AlexNet applies max-pooling in layers 1, 2, and 5, dropout in the final two fully connected layers, and local response normalization in layers 1 and 2.The cited evidence reports that removing local response normalization does not affect performance.
- Transferable features: AlexNet’s later layers learned semantic attributes, and FC7 activations became generic descriptors that outperformed SIFT or HoG on various tasks.
- AlexNet: AlexNet’s pretrained model could be downloaded and slightly modified for applications beyond its original image-classification task.
- Fine-tuning: Fine-tuning initializes a new task with trained weights and typically uses a lower learning rate than the original training.Similar tasks can freeze earlier layers, while different tasks may require relearning all layers or training from scratch.
- Later CNN designs: OverFeat jointly classified, localized, and detected objects, reflecting CNN extensions motivated by AlexNet’s performance.
- Later CNN designs: VGG replaced larger filters with stacks of smaller filters, producing efficiency and a regularization-like effect on parameters.
- Object detection: Region-based detection used roughly 2000 category-independent proposals, CNN features for each warped region, and category-specific linear SVMs.
3.1 Region-based CNNs
Region-based CNN approaches address object detection and localization by generating candidate regions and applying CNN features to them, rather than classifying an entire image as one object. Subsequent variants improve region selection, feature robustness, and multi-label prediction.
- Object detection must classify and precisely localize multiple objects that may vary in size within an image.Standard image-recognition CNNs can predict only a single class even when multiple objects are present.
- Region proposals reduce the overhead of exhaustive sliding-window search by producing hundreds of class-agnostic patches at multiple scales.Classification networks can then process patches individually for localization.
- R-CNN generates around 2000 selective-search proposals, warps each proposal, and extracts fixed-size high-level semantic CNN features.The extracted region features support downstream object localization.
- Objectness-based patch extraction aggregates CNN features across selected scales and locations to build a more robust image representation for image search.This avoids exhaustive consideration of all image patches.
- A shared CNN can predict labels for an arbitrary number of region proposals, with pooling producing the final multi-label prediction.The architecture extends a single-label CNN to multi-label image prediction.
3.2 Fully Convolutional Networks
Fully convolutional networks adapt CNNs to per-pixel vision tasks such as semantic segmentation by replacing non-translation-invariant inner-product layers with spatially consistent operations. Related architectures preserve resolution, use structured prediction, and extend end-to-end dense prediction to saliency and joint tasks.
- 3.2 Fully Convolutional Networks: Semantic segmentation and scene parsing assign a label to every pixel, making them structured-prediction problems rather than whole-image classification.Earlier approaches used region-level hand-crafted features and classifiers for per-pixel labeling.
- 3.2 Fully Convolutional Networks: Figure 3 depicts AlexNet modified into a fully convolutional network for semantic object segmentation on PASCAL VOC 2012 with 21 classes.The figure presents the architecture in the context of dense segmentation.
- 3.2 Fully Convolutional Networks: Long et al. proposed a fully convolutional architecture that learns per-pixel tasks such as semantic segmentation end to end.Convolutional and max-pooling layers provide location-invariant operations, preserving spatial correspondence up to scaling.
- 3.2 Fully Convolutional Networks: Removing densely connected layers eliminates their translation-invariance problem and removes the fixed input-size constraint imposed by classification CNNs.Fully convolutional networks can therefore operate on images without those inner-product-layer restrictions.
- 3.2 Fully Convolutional Networks: Resolution-preserving variants remove stride where possible and model the network output as unary terms in a CRF with pairwise smoothness constraints.This formulates per-pixel classification as inference over the original-resolution image grid.
- 3.2 Fully Convolutional Networks: DeepSaliency jointly learns saliency detection and semantic object segmentation, showing that collaborative learning across related tasks can boost overall performance.DeepFix instead uses inception blocks and large-receptive-field convolutions to capture multi-scale structure and global context.
3.3 Multi-modal networks
Multi-modal CNNs extend visual representation beyond still RGB images by combining complementary signals such as appearance, depth, motion, audio, or text. For video, two-stream networks separately process frames and optical flow before fusing their predictions.
- 3.3 Multi-modal networks: RGB-D methods combine color and depth information, extending CNN representations to perception modalities beyond standard RGB vision.Early work used CNNs to exploit both modalities for classification.
- 3.3 Multi-modal networks: Video supplies motion information in addition to image appearance, but early CNN extensions often processed raw frames, making learning more difficult.This motivated representations that explicitly separate spatial and temporal information.
- 3.3 Multi-modal networks: The two-stream network separates video into a spatial stream for frame appearance and a temporal stream for motion represented by optical flow.The spatial stream processes individual frames, while the temporal stream receives stacked optical flow from consecutive frames.
- 3.3 Multi-modal networks: The two CNN streams are trained separately on their corresponding inputs, and their predictions are fused for action recognition.This architecture is depicted in Figure 4.
- 3.3 Multi-modal networks: Later video methods focus on selecting active spatio-temporal volumes or pooling deep features along trajectories for efficient representation.Examples use motion-salient proposal subsets or trajectory-constrained pooling.
- 3.3 Multi-modal networks: Multi-modal CNNs can also incorporate audio or text alongside image data to complement visual content representation.The survey presents this as an extensible direction for other modalities.
3.4 CNNs with RNNs
CNNs are poorly suited to sequence learning because they lack memory and feedback, motivating recurrent extensions for temporal computer-vision tasks. CNN-RNN models combine visual representations with recurrent state for action recognition, captioning, and visual question answering.
- Motivation: CNNs are not well suited to learning sequences because they lack memory of previous states and feedback mechanisms.RNN feedback connections provide internal memory for learning temporal patterns.
- Recurrent neural networks: A simple RNN stores previous inputs and predicts the cumulative sum of a sequence up to the current timestep.The example uses one hidden layer and ReLU nonlinearities.
- Long short-term memory: LSTMs address the vanishing-gradient problem through a recurrent architecture designed for sequence learning.They subsequently became widely used in speech and handwriting recognition.
- Video understanding: For video action recognition, LRCN jointly trains an AlexNet variant and a 256-unit LSTM using RGB and optical-flow features, then averages frame predictions.Frames are sampled, classified individually, and aggregated across time.
- Image captioning and visual question answering: CNN-RNN systems encode visual inputs and generate textual descriptions or answers conditioned on images and questions.Image captioning uses a CNN encoder with a language-generating RNN, while visual question answering combines question and image representations.
3.5 Hybrid learning methods
Hybrid learning methods extend CNNs through shared, multi-task, task-specific, metric-learning, and ranking architectures. These designs adapt representations to related objectives such as detection, localization, segmentation, retrieval, and fine-grained similarity.
- Multi-task learning: Multi-task learning exploits shared representations across objectives to improve generalization compared with single-task models.CNN implementations include multi-task loss functions and auxiliary-task supervision.
- Multi-task learning: Multi-task CNNs jointly optimize primary and auxiliary tasks, such as classification with bounding-box regression or expression recognition with landmark localization.Auxiliary tasks are typically intended to improve the primary task, and some methods adaptively weight their contributions.
- Task-specific architectures: Other hybrid designs substantially modify pretrained networks to support simultaneous classification, localization, and detection.One example replaces pretrained layers with a regression network and fine-tunes the resulting model.
- Task-specific architectures: Task-specific sub-networks support joint point regression, body-part detection, localization, segmentation, and related objectives.Serially stacked subnetworks can pass localization outputs and the original image to a segmentation network.
- Metric and rank learning: Siamese networks compare two images using identical-weight branches whose combined activations produce a distance-like similarity metric.Triplet networks extend this structure for rank learning and fine-grained image similarity.
4 Open problems
The paper identifies unresolved challenges in CNN training, robustness, efficiency, online learning, and unsupervised learning for computer vision.
- Training and optimization: CNN training requires tuning many hyper-parameters, including architectural choices, while automated tuning can be time consuming and impractical for large networks.Automated tuning requires training multiple models.
- Adversarial examples: CNNs can be induced to produce high-confidence false predictions, and imperceptible image changes can yield entirely different classification labels.Attempts to reduce adversarial effects have not established complete elimination.
- Geometric invariance: CNNs are robust to small geometric transformations, but invariance under extreme deformation remains largely unstudied.The open question concerns extending robustness toward invariance under more severe transformations.
- Model efficiency: CNN models require many data points and are relatively slow to evaluate because they are large, while training nonredundant models remains unclear.Existing work has focused on reducing parameter counts, but does not resolve how to train nonredundant models initially.
- Online learning: CNNs are currently trained in a one-shot manner, motivating online training methods for robotics applications.The proposed direction targets settings where data arrive continuously.
- Unsupervised learning: Unsupervised learning could exploit large quantities of unlabeled web images, but the paper does not provide a detailed discussion of this area.Recent work exists, while a fuller treatment remains an open direction.
5 Concluding remarks
The paper surveys convolutional neural networks for computer vision and describes their use in replacing complicated hand-tuned algorithms with end-to-end trained monolithic models. It is intended especially as a guide for vision researchers beginning to use deep learning, while acknowledging that it cannot cover the entire computer-vision deep-learning literature.
- Scope and contribution: The survey focuses on convolutional neural networks for computer vision and their variations across applications.It is more focused on computer vision than broader deep-learning reviews.
- Scope and contribution: End-to-end training enables complicated hand-tuned algorithms to be replaced by single monolithic algorithms.The conclusion presents this as the practical significance of the surveyed paradigm.
- Scope and contribution: The paper acknowledges that it may not capture the entire range of deep-learning research, even within computer vision.It directs readers to broader reviews for general deep learning.