Source-linked AI summary
Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, Kurt Keutzer
TL;DR
Spatial convolutions provide spatial aggregation but are costly because computation and model size grow quadratically with kernel size. The paper replaces them with zero-FLOP, zero-parameter shifts fused with point-wise convolutions, using expansion E to trade efficiency against accuracy. Across classification, face verification, and style transfer, ShiftNet achieves competitive performance with fewer parameters, while the authors note that channel allocation among shift groups remains largely uninformed.
Problem
Spatial convolutions impose computation and model-size costs that are problematic for CNNs deployed on memory-constrained devices.
Method
The paper combines parameter-free spatial shifts with point-wise convolutions in trainable shift-based modules, with expansion E controlling the efficiency–accuracy tradeoff.
Results
ShiftNet achieves competitive performance with significantly fewer parameters across image classification, face verification, and style transfer.
Takeaways & Limitations
Shift-based architectures can replace spatial convolutions while retaining competitive multi-task performance under substantially smaller model-size constraints.
Takeaways & Limitations
Shift-group construction assigns a fixed number of channels to each group, but this allocation is largely uninformed and may be improved by better heuristics.
Abstract
from arXiv · showhide
Neural networks rely on convolutions to aggregate spatial information. However, spatial convolutions are expensive in terms of model size and computation, both of which grow quadratically with respect to kernel size. In this paper, we present a parameter-free, FLOP-free "shift" operation as an alternative to spatial convolutions. We fuse shifts and point-wise convolutions to construct end-to-end trainable shift-based modules, with a hyperparameter characterizing the tradeoff between accuracy and efficiency. To demonstrate the operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters; we additionally demonstrate the operation's resilience to parameter reduction on ImageNet, outperforming ResNet family members. We finally show the shift operation's applicability across domains, achieving strong performance with fewer parameters on classification, face verification and style transfer.
1. Introduction and Related Work
CNNs support many vision tasks but spatial convolutions impose substantial computation and model-size costs, motivating an alternative that preserves spatial aggregation with fewer resources. The paper introduces shift-based modules that combine parameter-free spatial shifts with point-wise convolutions and expose an accuracy–efficiency tradeoff.
- Motivation: CNNs serve image classification, object detection, face recognition, and style transfer, including applications on memory-constrained mobile and IoT devices.Over-the-air updates may be restricted to 100-150MB, constraining deployable CNN size.
- Motivation: Spatial convolutions aggregate spatial information but their computation and model size grow quadratically with kernel size.In VGG-16, 3x3 convolutions account for 15 million parameters, while the effectively 7x7 fc1 layer accounts for 102 million.
- Shift operation: The paper sidesteps spatial convolutions with a shift operation that moves each input channel spatially and requires zero FLOPs and zero parameters.Shift operations are paired with point-wise convolutions that mix information across channels.
- Shift-based modules: Shift-based modules introduce expansion E as a hyperparameter controlling the tradeoff between FLOPs or parameters and accuracy.This lets practitioners select models according to device or application requirements.
- Evaluation: The resulting ShiftNet family is evaluated on image classification, face verification, and style transfer, achieving competitive performance with significantly fewer parameters.The method is presented as orthogonal to model compression, tensor factorization, and low-bit networks, so these techniques can be composed with it.
2. The Shift Module and Network Design
The shift module replaces spatial aggregation with parameter-free, FLOP-free shifts interleaved with point-wise convolutions, while expansion and related design choices control receptive field and efficiency.
- Motivation: Spatial convolution cost grows quadratically with kernel size, while depth-wise convolution can become memory-access bound despite lower theoretical computation.The paper motivates shifts partly by the practical cost of memory accesses on depth-wise implementations.
- Shift kernel construction: A shift kernel can expose every position in a DK × DK window when M is at least DK^2, after which a point-wise convolution exchanges channel information.Channels are divided among shift directions, with remaining channels assigned to a central unshifted group.
- Shift operation: Shift operations move each channel in a designated spatial direction, avoiding the parameters and FLOPs required by spatial or depth-wise convolutions.A shift kernel assigns one nonzero position per channel; point-wise convolutions then mix information across channels.
- Trainable construction: Point-wise convolutions before and after the shift make channel permutations trainable end-to-end and render different shift permutations equivalent.The construction replaces discrete permutation optimization with learned point-wise convolutions around the shift.
- Shift-based modules: The CSC module applies point-wise convolution, shift, and point-wise convolution, while SC2 adds a preceding shift to further increase receptive field.Both point-wise-convolution sets use batch normalization and ReLU; residual or pooling-concatenation paths handle shape-preserving or downsampling cases.
- Module design: Kernel size, dilation rate, stride, and expansion rate control shift-module behavior, with expansion trading intermediate channel capacity against efficiency.Because shift kernel size does not change parameter size or FLOPs, larger intermediate channel sizes can be used to gather nearby information.
3. Experiments
The experiments evaluate shift-based modules across classification, face verification, and style transfer, emphasizing accuracy–efficiency tradeoffs under reduced parameters and computation. ShiftResNet and ShiftNet generally retain competitive performance while using substantially fewer parameters than convolutional alternatives.
- Classification: ShiftResNet preserves accuracy better than reduced ResNet models at matched parameter counts, including 8% better resilience on CIFAR100.On ImageNet, ShiftResNet also achieves better accuracy with millions fewer parameters.
- Classification: 1.5x fewer parameters still leaves ShiftResNet consistently outperforming ResNet across the evaluated ResNet models.The CIFAR100 accuracy–parameter and accuracy–FLOP curves likewise show superior ShiftResNet tradeoffs.
- Classification: ShiftNet-A is 34X smaller than VGG-16 with a 1.4% top-1 accuracy drop, while ShiftNet-C achieves better accuracy than SqueezeNet with 2/3 as many parameters.ShiftNet-C is also 77X smaller than AlexNet; ShiftNet-B uses fewer parameters than MobileNet but has 2.5% worse top-1 accuracy.
- Face Verification: ShiftFaceNet reduces FaceNet’s parameter size by 35X with at most a 2% accuracy drop across three face-verification benchmarks.ShiftFaceNet contains 0.78 million parameters, compared with 28.5 million in the original FaceNet-based model.
4. Discussion
The discussion finds that fixed channel-to-shift assignments are largely uninformed, motivating allocation heuristics based on within-group redundancy and channel contributions. Analysis shows anisotropic shift contributions, with the largest contributions forming a cross.
- Allocation Strategies: Fixed channel assignments are largely uninformed, motivating more informed allocations and potential improvements.The current construction assigns a fixed number of channels to each shift.
- Allocation Strategies: An ideal allocation avoids redundant features within each shift group and gives every shifted feature a nontrivial output contribution.Redundancy is measured through within-group activation correlation, while contribution is approximated by the l2 norm of a point-wise kernel row.
- Channel Correlations: Correlation matrices from a CIFAR100 validation analysis estimate redundancy among channels within each shift group.The analysis examines a CSC module with 16 input/output channels and expansion 9 from a trained ShiftResnet20 model.
- Channel Contributions: Shift contributions are anisotropic, with the largest contributions forming a cross of horizontal and vertical shifts.Contributions are estimated from normalized l2 norms of rows in the second point-wise convolution kernel.
- Channel Contributions: Figure 7 ranks normalized channel contributions across nine groups of 16 and compares normalized totals for each shift pattern.The horizontal axis groups channels by shift, while yellow indicates the highest magnitude in the shared color map.
- Implication: Better channel-allocation heuristics may yield higher per-FLOP and per-parameter accuracy.This is presented as a potential improvement rather than an established experimental result.
5. Conclusion
The paper presents shift as a zero-FLOP, zero-parameter alternative to convolutions, combining it with point-wise convolutions in trainable modules. Replacing ResNet convolutions improves accuracy under matched budgets and supports competitive performance across several tasks with around 4 million parameters.
- Contribution: Shift is a zero-FLOP, zero-parameter, easy-to-implement alternative to convolutions for spatial-information aggregation.The paper constructs end-to-end-trainable modules by combining shift operations with point-wise convolutions.
- Results: Replacing ResNet convolutions increases accuracy by up to 8% at the same parameter/FLOP budget and recovers accuracy with one-third the budget.The paper varies model-size constraints and constructs a family of shift-based networks.
- Results: Shift-based networks attain competitive performance with around 4 million parameters on classification, face verification, and style transfer.These results demonstrate applicability across multiple tasks while using significantly fewer parameters.