Source-linked AI summary
Multi-scale recognition with DAG-CNNs
Songfan Yang, Deva Ramanan
TL;DR
The paper addresses the limitation of single-layer CNN features for recognition tasks that require different levels of invariance and detail. It extracts multi-scale features from multiple layers in end-to-end trainable DAG-CNNs, achieving consistent improvements and state-of-the-art results across three scene benchmarks.
Problem
CNN recognition pipelines typically use a single output layer, although coarse and fine-grained classification require different levels of invariance and detail.
Method
DAG-CNNs combine features from multiple CNN layers in a feed-forward directed acyclic graph and train the multi-scale predictor end to end.
Results
State-of-the-art performance is reported on SUN397, MIT67, and Scene15, including a 23.9% reduction in the best-previously reported error on MIT67.
Takeaways & Limitations
Multi-scale features can be shared between coarse and fine-grained classification tasks, and can be extracted from a single CNN encoding without separate image patches.
Takeaways & Limitations
Adding all layers can hurt test performance because redundant or correlated nearby-layer features may cause over-fitting.
Abstract
from arXiv · showhide
We explore multi-scale convolutional neural nets (CNNs) for image classification. Contemporary approaches extract features from a single output layer. By extracting features from multiple layers, one can simultaneously reason about high, mid, and low-level features during classification. The resulting multi-scale architecture can itself be seen as a feed-forward model that is structured as a directed acyclic graph (DAG-CNNs). We use DAG-CNNs to learn a set of multiscale features that can be effectively shared between coarse and fine-grained classification tasks. While fine-tuning such models helps performance, we show that even "off-the-self" multiscale features perform quite well. We present extensive analysis and demonstrate state-of-the-art classification performance on three standard scene benchmarks (SUN397, MIT67, and Scene15). In terms of the heavily benchmarked MIT67 and Scene15 datasets, our results reduce the lowest previously-reported error by 23.9% and 9.5%, respectively.
1. Introduction
The paper argues that recognition benefits from combining features across scales because coarse and fine-grained tasks require different levels of invariance and detail. It introduces end-to-end trainable DAG-CNNs that extract multi-scale features from multiple layers and reports consistent benchmark improvements.
- Coarse recognition needs deformation-robust invariant features, whereas fine-grained recognition needs detailed shape cues.
- Typical CNN classifiers use features from a single output layer, limiting prediction to one representation scale.
- The proposed architecture trains a predictor on features extracted from multiple layers, which are already computed during one feed-forward pass.
- Multi-scale features can be pooled across spatial locations to control the dimensionality that otherwise makes linear-model training difficult and prone to over-fitting.
- DAG-CNNs remain feed-forward models but replace a chain structure with a directed acyclic graph that supports end-to-end discriminative training.
- Across SUN397, MIT67, and Scene15, multi-scale DAG variants consistently improve performance regardless of the underlying CNN architecture.
2. Motivation
Empirical analysis shows that classification benefits from combining features across selected CNN layers because different classes favor different levels of invariance. Pooling and forward selection make multi-scale representations practical, while indiscriminately adding lower-layer features can hurt test performance.
- Empirical setup: Off-the-shelf Caffe and Deep19 CNNs were evaluated on MIT67 using 10-fold cross-validation.The analysis examined pre-trained models without task-specific fine-tuning.
- Single-scale models: Higher-level features generally improve single-scale accuracy, with slight gains at successive ReLU layers.The ReLU operation introduces nonlinear rectification, unlike the linear convolutional and pooling layers.
- Scale-varying classification: Over a third of categories perform better with mid-level than high-level features, indicating class-dependent preferences for feature invariance.Figure 4 reports that layer 20 is optimal for 15 classes, whereas layer 18 is optimal for 26 classes.
- Spatial pooling: Average-pooled, L2-normalized features outperform unpooled full-dimensional features, with the latter performing worse on test data because of over-fitting.Pooling reduces each H × W × F layer representation to a 1×1×F feature.
- Multi-scale classification: Adding intermediate layers improves multi-scale classification, but lower-layer features can saturate or reduce performance.The analysis associates the reduced test performance of additional layers with over-fitting and finds high- and mid-level features more useful for scene classification.
- Multi-scale selection: Greedy forward selection approximates the optimal layer combination because exhaustive search over N ReLU layers requires evaluating 2^N combinations.The selected scales reject low-level features and are reused across benchmarks, suggesting cross-dataset generalization.
3. Approach
The approach represents multi-scale CNN prediction as a feed-forward DAG, enabling end-to-end gradient-based learning through layers with multiple outputs and inputs. It specifies corresponding back-propagation rules, efficient special cases, and an implementation whose runtime is essentially that of the chain model.
- Gradient computation: End-to-end learning uses the chain rule and partial derivatives to generalize back-propagation to layers with multiple parents or inputs.Multiple back-propagation signals are combined and passed to lower layers.
- Model: The multi-scale model is equivalent to feed-forward processing of a DAG-structured CNN, replacing the earlier hinge-loss predictor with a softmax for training.Typical CNNs use Conv., ReLU, normalization, and pooling layers, followed by a K-way softmax.
- Gradient computation: At ReLU layers, duplicated outputs allow multiple back-propagation signals to be summed before local-gradient scaling, reducing multiplications by C.The same ReLU values feed both the next chain layer and the current-layer pooling operation.
- Model: Multi-scale output connections pool activations at each layer, normalize them, apply K-output fully connected layers, and sum scores before softmax prediction.This connects multiple feature scales directly to the K output classes.
- Gradient computation: At the Add layer, the same partial gradient is reused for every input, and the standard chain-structured back-propagation signal is replicated across its parents.This special case also saves C multiplications.
- Implementation: The implementation uses a custom Add layer and DAG data structure, while training and testing are essentially as fast as in the chain model.The modifications were implemented in the MatConNet codebase.
- Training dynamics: Multi-scale connections provide lower layers with direct output links, ensuring stronger learning signals and addressing vanishing gradients during training.The paper reports that Figure 9 experimentally verifies this property, with DAG gradients consistently 10× larger for Layer-1 convolution.
4. Experimental Results
Across SUN397, MIT67, and Scene15, multi-scale DAG models outperform single-scale and prior methods, with strong performance even in off-the-shelf settings. Fine-tuning improves results further, while the multiscale representation remains only about twice as large as single-scale features.
- SUN397: 56.2% accuracy made Deep19-DAG the highest-performing method reported on SUN397.The next-best method scored 54.3% and used both an ImageNet-trained CNN and a custom-trained CNN.
- MIT67: 77.5% accuracy on MIT67 reduced the best-previously reported error by 23.9%.Deep19-DAG computes the representation from a single end-to-end trainable DAG CNN rather than extracting and rescaling image patches.
- Scene15: Deep19-DAG outperformed all prior work on Scene15, continuing the performance pattern across the three benchmarks.The next-best method used a custom 7-million-image scene dataset for training.
- Off-the-shelf and fine-tuned models: Fine-tuned models consistently outperform off-the-shelf models, while DAG models show a large improvement over chain models.The comparison uses both single-scale chain and multi-scale DAG variants of the Caffe backbone.
- Off-the-shelf and fine-tuned models: Off-the-shelf multiscale features provide a strong cheap baseline: they require no additional computation time and are only twice as large to store.The conclusion also reports state-of-the-art classification performance on all three scene benchmarks, sometimes by a significant margin.