Source-linked AI summary
Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yannis Kalantidis, Marcus Rohrbach, Shuicheng Yan, Jiashi Feng
TL;DR
CNN feature maps contain spatial redundancy across neighboring locations and mix information at different spatial frequencies. The paper factorizes these maps into high- and low-frequency groups, processes the low-frequency group at reduced resolution with OctConv, and uses it as a direct convolution replacement. Across image and video recognition tasks, OctConv improves performance while reducing memory and computation, with reported gains depending on the frequency allocation ratio.
Problem
CNN feature maps contain spatial redundancy because neighboring locations can share information, but conventional representations store feature descriptors independently.
Method
OctConv factorizes feature maps into high- and low-frequency groups, stores low-frequency maps at half spatial resolution, and directly processes the resulting representation.
Results
OctConv consistently improves image and video recognition performance while reducing memory and computational cost across multiple CNN backbones.
Takeaways & Limitations
OctConv offers a generic, plug-and-play way to reduce spatial redundancy and improve the recognition-efficiency trade-off without changing backbone architectures.
Takeaways & Limitations
Reported GFLOPs are training measurements at spatial size 224^2, whereas one comparison reports 36.1 GFLOPs at spatial size 256^2.
Abstract
from arXiv · showhide
In natural images, information is conveyed at different frequencies where higher frequencies are usually encoded with fine details and lower frequencies are usually encoded with global structures. Similarly, the output feature maps of a convolution layer can also be seen as a mixture of information at different frequencies. In this work, we propose to factorize the mixed feature maps by their frequencies, and design a novel Octave Convolution (OctConv) operation to store and process feature maps that vary spatially "slower" at a lower spatial resolution reducing both memory and computation cost. Unlike existing multi-scale methods, OctConv is formulated as a single, generic, plug-and-play convolutional unit that can be used as a direct replacement of (vanilla) convolutions without any adjustments in the network architecture. It is also orthogonal and complementary to methods that suggest better topologies or reduce channel-wise redundancy like group or depth-wise convolutions. We experimentally show that by simply replacing convolutions with OctConv, we can consistently boost accuracy for both image and video recognition tasks, while reducing memory and computational cost. An OctConv-equipped ResNet-152 can achieve 82.9% top-1 classification accuracy on ImageNet with merely 22.2 GFLOPs.
1. Introduction
OctConv addresses spatial redundancy in CNN feature maps by separating information into frequency groups and processing low-frequency maps at reduced resolution. It provides a generic convolution replacement that reduces resource use while supporting information exchange and recognition improvements across image and video models.
- Motivation: CNN feature maps contain spatial redundancy because neighboring locations may share information that dense independent descriptors do not exploit.This redundancy is distinct from redundancy in model parameters or channel dimensions.
- Motivation: Octave feature representation separates high-frequency fine details from smoothly varying low-frequency maps and stores the latter in a low-resolution tensor.The low-frequency maps are defined one octave lower, at half the spatial resolution of the high-frequency maps.
- Method: OctConv directly operates on the factorized representation, updates each frequency group, and enables information exchange between groups.It avoids decoding low-frequency maps back to high resolution before convolution.
- Method: OctConv is a plug-and-play replacement for vanilla convolution that can be used without changing network architectures or tuning additional hyperparameters.The operation is designed to complement topology, channel-redundancy, and parameter-redundancy reduction methods.
- Results: OctConv consistently improves recognition across popular 2D image and 3D video CNN backbones while reducing memory and computational cost.The reported experiments cover ImageNet image recognition and Kinetics video action recognition.
2. Related Work
Prior efficiency methods reduce redundancy through network topology, channel connectivity, or model parameters, while multi-scale methods use explicit multi-scale designs. OctConv instead targets spatial redundancy and is intended to complement these approaches without requiring backbone architecture changes.
- CNN efficiency methods: ResNet and DenseNet improve CNN topology with shortcut connections, while ResNeXt and ShuffleNet reduce interchannel redundancy using group convolutions.Xception and MobileNet further reduce connection density with depth-wise convolutions.
- Multi-scale methods: Multi-scale approaches provide related context, but OctConv operates on spatial dimensions and can complement temporal multi-scale pathways such as SlowFast Networks.The supplied passage specifically contrasts OctConv’s spatial operation with SlowFast’s temporal pathways.
- OctConv positioning: OctConv focuses on spatial redundancy and replaces vanilla convolution without requiring backbone architecture adjustments.The paper reports top results for OctConv-equipped CNNs on challenging benchmarks.
3. Method
OctConv factorizes feature maps into high- and low-frequency groups, processes them directly at different resolutions, and exchanges information between the groups. This design reduces spatial redundancy while supporting efficient integration into convolutional networks.
- 3.1. Octave Feature Representation: Octave feature representation factorizes feature maps into high- and low-frequency channel groups, with low-frequency maps stored at half the high-frequency spatial resolution.The ratio α controls the channels allocated to the low-frequency group.
- 3.3. Implementation Details: The multi-frequency representation reduces memory and computational cost, while OctConv can also be applied to group and depth-wise convolutions.When low-frequency compression is omitted, group OctConv and depth-wise OctConv reduce to their respective vanilla versions.
- 3.2. Octave Convolution: OctConv replaces vanilla convolution by separately updating high- and low-frequency maps while enabling communication in both directions.Its four paths are high-to-high, low-to-high, low-to-low, and high-to-low.
- 3.2. Octave Convolution: Average pooling is used for high-to-low communication because strided convolution causes misalignment when aggregating information across scales.Upsampling for the opposite direction is folded into the convolution rather than explicitly storing upsampled maps.
- 3.2. Octave Convolution: 2× larger effective receptive fields for low-frequency maps allow each OctConv layer to capture contextual information from more distant locations.This enlargement results from applying k × k kernels to maps represented at half the spatial resolution.
- 3.3. Implementation Details: OctConv is a plug-and-play replacement for vanilla convolution that can be inserted into existing networks without special architectural adjustments.The first and last OctConv layers convert to and from the multi-frequency representation using α_in = 0 or α_out = 0.
4. Experimental Evaluation
Experiments evaluate OctConv across image and video recognition backbones, testing efficiency–accuracy trade-offs, frequency behavior, and compatibility with compact and related architectures. Results show consistent accuracy gains alongside reduced computation, with benefits extending to practical inference and 3D CNNs.
- Experimental setup: Experiments replace regular convolutions with OctConv across popular 2D CNNs and compare standard ImageNet validation accuracy under matched training and testing settings.The study evaluates ResNet, ResNeXt, DenseNet, MobileNet, and SE-Net backbones, excluding only the first convolution before max pooling.
- ImageNet ablations: OctConv’s ResNet-50 FLOPs–accuracy curve is concave, with α = 0.5 roughly halving FLOPs at similar or better accuracy and α = 0.125 achieving 1.2% higher accuracy than baseline.At α = 0.125, the reported accuracy peak is 1.2% above the baseline; 75% of feature maps can also be compressed to half resolution with only a 0.3% accuracy drop.
- ImageNet ablations: OctConv-equipped networks show similar concave FLOPs–accuracy behavior across ResNet, ResNeXt, DenseNet, and SE-ResNet, with performance peaks commonly at α = 0.125 or α = 0.25.The reported consistency supports using OctConv as a replacement for vanilla convolution across different depths, widths, and topologies.
- Medium models: OctConv improves medium-model comparisons, including 78.8% versus 78.4% accuracy with 31% lower computational cost for Oct-ResNeXt-50 than the Elastic-based method.Other reported comparisons include higher accuracy with substantially lower FLOPs for Oct-ResNet and reduced FLOPs with better accuracy when added to GloRe.
- Video recognition: In video action recognition, adding OctConv to C2D and I3D consistently decreases FLOPs while improving accuracy, and remains complementary to Non-local modules.The evaluation uses Kinetics-400 and Kinetics-600 with standard 3D CNN baselines.
5. Conclusion
Octave Convolution addresses spatial redundancy by separating low- and high-frequency features while serving as an in-place replacement for regular convolution. The paper reports efficiency and recognition-performance gains across image and video recognition settings.
- Octave Convolution stores and processes low- and high-frequency features separately to reduce spatial redundancy and improve model efficiency.
- Octave Convolution can replace regular convolutions in most 2D and 3D CNNs without adjusting the model architecture.
Appendix A. The Misalignment Problem
Strided convolution followed by up-sampling can misalign feature maps when odd-sized filters are used. The resulting shift causes problems when combining shifted and unshifted maps.
- Odd-filter strided convolution followed by up-sampling shifts the entire feature map toward the lower right.
- Adding the shifted up-sampled map to an unshifted map becomes problematic because their spatial locations no longer align.
OctConv
OctConv represents feature maps at two spatial frequencies and reduces the cost of processing smoothly varying information at lower resolution. Its computation combines within-frequency processing with cross-frequency communication.
- The low-frequency tensor uses 2× lower spatial resolution, costing 75% less space for low-frequency maps than the conventional representation.
- OctConv computes four paths: H →H, H →L, L →H, and L →L.
- The multi-frequency representation stores smoothly changing low-frequency maps in a low-resolution tensor to reduce spatial redundancy.
- Pooling and nearest-neighbor up-sampling are treated as negligible computational costs in the cost analysis.
- Three of OctConv’s four internal convolution operations run on lower-resolution tensors, excluding the first high-to-high convolution.
Appendix C. ImageNet Ablation Study Results
ImageNet ablations examine how OctConv’s gains vary with image resolution and how down-sampling and inter-octave connectivity affect performance. The reported results emphasize larger gains at higher resolution and the value of both communication paths.
- OctConv’s gain over baseline models increases as the test image resolution grows.
- The larger receptive field of each OctConv is offered as an explanation for better detection of large objects.
- The ImageNet ablation confirms the importance of retaining both inter-frequency communication paths.
- Pooling methods outperform strided convolution for down-sampling, with average pooling performing best in the ablation.
- Table 7 evaluates ImageNet classification accuracy with a ResNet-50 backbone trained using 256 × 256 crops.
- Table 8 reports ablations of down-sampling and inter-octave connectivity on ImageNet, while Table 9 provides tabulated values corresponding to Figure 4.