Source-linked AI summary
Deep Layer Aggregation
Fisher Yu, Dequan Wang, Evan Shelhamer, Trevor Darrell
TL;DR
Existing skip connections fuse network layers only shallowly, leaving how to aggregate layers and modules effectively as an open architectural question. Deep Layer Aggregation introduces iterative and hierarchical fusion, improving performance and parameter efficiency across recognition and dense prediction tasks, including about 30% fewer parameters and ∼1 point of improvement in top-1 error rate for DLA-34 versus ResNet-34.
Problem
How to connect and aggregate layers and modules to better fuse semantic and spatial information remains insufficiently explored.
Method
Deep Layer Aggregation introduces iterative and hierarchical architectures that compose nonlinear, shared aggregations across depths, resolutions, and scales.
Results
Across architectures and tasks, DLA improves performance and parameter efficiency over baseline networks; DLA-34 uses about 30% fewer parameters and achieves ∼1 point of improvement in top-1 error rate versus ResNet-34.
Takeaways & Limitations
Deep layer aggregation provides a backbone-compatible way to make more efficient use of layers, parameters, and computation for visual recognition and dense prediction.
Takeaways & Limitations
The architectural formulation assumes that the order of arguments for N follows Equation 2.
Abstract
from arXiv · showhide
Visual recognition requires rich representations that span levels from low to high, scales from small to large, and resolutions from fine to coarse. Even with the depth of features in a convolutional network, a layer in isolation is not enough: compounding and aggregating these representations improves inference of what and where. Architectural efforts are exploring many dimensions for network backbones, designing deeper or wider architectures, but how to best aggregate layers and blocks across a network deserves further attention. Although skip connections have been incorporated to combine layers, these connections have been "shallow" themselves, and only fuse by simple, one-step operations. We augment standard architectures with deeper aggregation to better fuse information across layers. Our deep layer aggregation structures iteratively and hierarchically merge the feature hierarchy to make networks with better accuracy and fewer parameters. Experiments across architectures and tasks show that deep layer aggregation improves recognition and resolution compared to existing branching and merging schemes. The code is at https://github.com/ucbdrive/dla.
1. Introduction
Deep layer aggregation (DLA) addresses the need to better connect network layers by hierarchically and iteratively fusing semantic and spatial information. It introduces backbone-independent IDA and HDA structures and reports improved performance, parameter count, and memory usage across architectures and tasks.
- Motivation: Increasing nonlinearity, capacity, and receptive fields can improve accuracy but creates optimization and computation challenges.Existing architectures use blocks and connections such as bottlenecks, residual, gated, and concatenative designs to manage these quantities.
- Motivation: Deeper layers tend to extract more semantic and global features, but the last layer is not necessarily the ultimate representation for every task.This motivates further exploration of how layers and modules should be connected.
- Contribution: IDA and HDA extend shallow skip connections with deeper, shared aggregation structures that fuse semantic and spatial information.The framework is independent of the backbone, supporting compatibility with current and future networks.
- Contribution: IDA focuses on fusing resolutions and scales, whereas HDA merges features from all modules and channels.Together, they iteratively and hierarchically refine representations for recognition and localization.
- Results: DLA improves performance, parameter count, and memory usage over baseline ResNet, ResNeXt, and DenseNet architectures across classification, fine-grained recognition, semantic segmentation, and boundary detection.DLA achieves state-of-the-art results among compact classification models.
2. Related Work
Prior work improves visual recognition through architectural depth, width, connectivity, and feature fusion. Deep layer aggregation unifies semantic fusion across channels and depths with spatial fusion across resolutions and scales, extending DenseNet- and FPN-like ideas.
- Architectural context: Visual-recognition architectures have advanced through deeper and wider networks, bottlenecks, shortened paths, channel mixing, and connectivity mechanisms including skipping, gating, branching, and aggregation.Examples include ResNets [16] and high-way networks for depth, ResNeXT [41] and Fractal-Net [25] for width, and Network-in-Network for channel mixing.
- Fusion axes: Deep layer aggregation unifies semantic fusion for inferring what with spatial fusion for inferring where across channels, depths, resolutions, and scales.Semantic fusion aggregates channels and depths, whereas spatial fusion aggregates resolutions and scales.
- Semantic fusion: DenseNets [19] propagate features and losses through concatenated skip connections, while hierarchical deep aggregation extends this reuse across stages with deeper tree-based fusion.The aggregation structures share DenseNets’ emphasis on short paths and feature reuse but fuse beyond simple stage-local concatenation.
- Spatial fusion: FPNs [30] equalize resolution and standardize semantics through top-down and lateral connections, whereas iterative deep aggregation raises resolution through nonlinear, progressive fusion.The paper contrasts FPN’s linear connections with deeper aggregation intended to strengthen earlier, semantically weaker levels.
3. Deep Layer Aggregation
Deep layer aggregation combines network layers across depths, resolutions, and scales through compositional, nonlinear structures that repeatedly merge representations. The framework includes iterative aggregation for progressively refining stages and hierarchical aggregation for combining blocks across the feature hierarchy.
- 3. Deep Layer Aggregation: Deep aggregation is compositional and nonlinear, with the earliest aggregated layer passing through multiple aggregations across network depths, resolutions, and scales.The framework groups aggregations over layers throughout a network and focuses on blocks and stages organized by feature resolution.
- Iterative Deep Aggregation: IDA progressively merges shallow, small-scale features with deeper, larger-scale stages, refining shallow representations as they propagate through aggregation.Existing skips such as FCN [35], U-Net [33], and FPN [30] are described as linear and shallow, aggregating the shallowest layers least.
- Hierarchical Deep Aggregation: HDA hierarchically merges shallow and deep blocks and stages in a tree, combining a broader feature hierarchy than sequential IDA.Its branching structure spans layers and preserves richer combinations across the network’s feature hierarchy.
- Hierarchical Deep Aggregation: HDA feeds each aggregation output back into the backbone for the next subtree and merges same-depth aggregation nodes to propagate prior blocks efficiently.This propagates all previous block aggregations rather than only the immediately preceding block.
- Aggregation Nodes: Aggregation nodes compress inputs into a single output dimension, use convolution, batch normalization, and nonlinearity, and support architecture-specific 1×1 or 3×3 convolutions.IDA nodes are binary, HDA nodes accept depth-dependent arguments, and the architecture family imposes no requirements on block or stage internals.
4. Applications
DLA networks apply iterative aggregation to classification and dense prediction, fusing semantic and spatial information without ensembles, context modeling, or dilation. IDA and HDA extend across stages and resolutions, while learned interpolation enables high-resolution outputs for image-to-image tasks.
- Applications: Aggregation fuses semantic and spatial information for both classification and dense prediction without ensembles, context modeling, or dilation.The study isolates the aggregated representation through linear prediction without additional machinery.
- Classification: DLA applies IDA and HDA to ResNet and ResNeXT-style staged classifiers, using global average pooling followed by linear softmax prediction.Stages halve spatial resolution through six levels, from input resolution to 32× downsampling.
- Architecture: DLA connects stages with IDA and aggregates within and across stages with HDA, sharing aggregation nodes by combining the hierarchy roots.Stages are downsampled using max pooling with size 2 and stride 2, except the earliest stages, which use strided convolution.
- Dense prediction: For segmentation, contour detection, and other image-to-image tasks, fully convolutional DLA combines interpolation with additional IDA to reach task-specific output resolutions.This conversion from classification DLA is described as simple and consistent with other architectures.
- Dense prediction: IDA increases depth and resolution by projecting stages 3–6 to 32 channels, interpolating them to stage 2 resolution, and iteratively aggregating the results.Projection and upsampling parameters are learned jointly, initialized with bilinear interpolation, and subsequently optimized.
5. Results
DLA improves or rivals specialized networks across image classification, fine-grained recognition, semantic segmentation, and contour detection, often using fewer parameters or less computation. It establishes new state-of-the-art results on several fine-grained datasets and boundary-detection metrics, while high output resolution is critical for localization.
- Results: DLA improves or rivals special-purpose networks across ImageNet classification, fine-grained recognition, semantic segmentation, and contour detection, transferring the classification architecture with little to no modification.These evaluations span ILSVRC, multiple fine-grained datasets, and dense prediction tasks.
- ImageNet classification: DLA-34 uses about 30% fewer parameters than ResNet-34 while improving top-1 error by approximately 1 point.The comparison uses similar layer counts and the same basic convolutional blocks.
- ImageNet classification: DLA-X-102 has nearly half as many parameters as ResNeXt-101 while differing in error rate by only 0.2%.DLA retains an accuracy and parameter-efficiency advantage when using split convolutional blocks.
- Fine-grained recognition: DLA establishes new state-of-the-art results on Car, Plane, and Food recognition without additional annotations or task-specific modules, while remaining competitive with only several million parameters.Results improve or rival the state of the art on the evaluated fine-grained datasets, but do not surpass it on Birds, where fewer instances per class may require further regularization.
- Boundary detection: DLA achieves state-of-the-art ODS and OIS scores on both BSDS and PASCAL boundaries, while experiments show that higher output resolution is critical for accurate boundary detection.Inference uses a single forward pass without ensembles or multi-scale testing; deeper networks do not continue improving BSDS performance.
6. Conclusion
The conclusion identifies aggregation as decisive for architecture and addresses the need for deeper aggregation through iterative and hierarchical deep aggregation, yielding models that are more accurate and use parameters and computation more efficiently than baseline networks.
- 6. Conclusion: Iterative deep aggregation and hierarchical deep aggregation address the need for deeper aggregation across channels, scales, and resolutions.The paper frames aggregation and module connectivity as increasingly important as architectures incorporate more modules.
- 6. Conclusion: The resulting models are more accurate and use parameters and computation more efficiently than baseline networks.