Source-linked AI summary
CondConv: Conditionally Parameterized Convolutions for Efficient Inference
Brandon Yang, Gabriel Bender, Quoc V. Le, Jiquan Ngiam
TL;DR
CNNs traditionally share convolutional kernels across examples, limiting how capacity can be increased under inference-cost and latency constraints. CondConv makes kernels input-dependent by combining expert kernels, and experiments show improved performance–inference-cost trade-offs across classification and detection, including 78.3% ImageNet accuracy with 413M multiply-adds for CondConv-EfficientNet-B0.
Problem
Shared convolutional kernels and conventional capacity scaling limit efficient increases in CNN capacity under computational and latency constraints.
Method
CondConv computes each example's convolutional kernel as a learned input-dependent linear combination of expert kernels, requiring one expensive convolution.
Results
CondConv improves the performance–inference-cost trade-off across several CNN architectures on ImageNet classification and COCO object detection, including 78.3% accuracy with 413M multiply-adds for CondConv-EfficientNet-B0.
Takeaways & Limitations
CondConv provides a way to increase model size and capacity while maintaining efficient inference across the evaluated classification and detection settings.
Abstract
from arXiv · showhide
Convolutional layers are one of the basic building blocks of modern deep neural networks. One fundamental assumption is that convolutional kernels should be shared for all examples in a dataset. We propose conditionally parameterized convolutions (CondConv), which learn specialized convolutional kernels for each example. Replacing normal convolutions with CondConv enables us to increase the size and capacity of a network, while maintaining efficient inference. We demonstrate that scaling networks with CondConv improves the performance and inference cost trade-off of several existing convolutional neural network architectures on both classification and detection tasks. On ImageNet classification, our CondConv approach applied to EfficientNet-B0 achieves state-of-the-art performance of 78.3% accuracy with only 413M multiply-adds. Code and checkpoints for the CondConv Tensorflow layer and CondConv-EfficientNet models are available at: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/condconv.
1 Introduction
CondConv addresses the tension between increasing CNN capacity and maintaining efficient inference by making convolutional kernels input-dependent rather than static. The approach is presented as an efficient alternative for latency-sensitive vision applications and is evaluated across classification and detection architectures.
- Increasing CNN capacity has improved performance but makes inference computationally expensive and often incompatible with strict latency constraints.Larger models can consume substantial datacenter capacity and may be infeasible for latency-sensitive applications.
- Standard convolutional layers apply the same kernels to every example, while conventional capacity scaling enlarges kernels or adds layers at computational cost proportional to the input size.
- CondConv computes convolutional kernels as input-dependent linear combinations of multiple experts, challenging the static-kernel paradigm.The expert weights are learned functions of the input.
- Increasing the number of CondConv experts can raise model capacity more efficiently than enlarging convolutional kernels because experts are combined once per input before convolution.
- CondConv is a drop-in replacement that improves capacity and performance while maintaining efficient inference on ImageNet classification and COCO object detection.
2 Related Work
Related work increases capacity or adapts processing through conditional computation, generated weights, multi-branch convolutions, activation scaling, and input-dependent filters. CondConv differs by producing different convolutional weights for each example without discrete routing and with one convolution.
- Conditional computation activates only part of a network per example to increase capacity without proportional computation, but discrete routing can make training difficult.
- CondConv enables each individual example to be processed with different weights, extending prior computation approaches beyond processing examples with different subnetworks.
- Weight-generating networks use generated CNN weights shared across dataset examples, achieving lower parameter counts but worse performance than the original network.
- A CondConv layer is mathematically equivalent to a multi-branch convolutional layer with weighted branch aggregation, but requires only one convolution.
- Activation-scaling methods conditionally modulate activations, whereas CondConv changes the convolutional weights themselves.
- Prior vision and language work has used input-dependent filters or offsets, while CondConv targets example-dependent convolutional weights for CNN layers.
3 Conditionally Parameterized Convolutions
CondConv replaces static convolutional kernels with input-dependent combinations of expert kernels. This preserves the capacity of a mixture-of-experts formulation while requiring one expensive convolution and using a lightweight routing function based on global context.
- A CondConv layer computes its output by applying an activation to a convolution with an input-dependent linear combination of expert kernels.Each expert kernel retains the dimensions of the original convolutional kernel.
- The routing weights are example-dependent scalars produced by a learned routing function, with n denoting the number of experts and σ the activation function.
- CondConv increases capacity by adding experts instead of enlarging kernel dimensions or channel counts, avoiding multiply-add costs that scale with input feature-map size.
- CondConv matches the capacity of a linear mixture of experts while requiring only one expensive convolution rather than one convolution per expert.
- The routing function computes expert weights through global average pooling, a fully connected layer, and a Sigmoid activation.
- Using global pooled inputs allows the routing function to adapt local convolutional operations using global context.
- CondConv can replace any convolutional layer and extend to depth-wise convolutions and fully connected layers.
4 Experiments
Experiments evaluate CondConv across ImageNet classification, COCO detection, routing designs, and layer placements. CondConv generally improves performance relative to inference cost, including strong EfficientNet and SSD results.
- ImageNet classification: All baseline architectures improve ImageNet accuracy with a small relative inference-cost increase below 10%.The comparison uses eight experts per CondConv layer across several architectures.
- Experimental setup: CondConv models replace baseline convolutions, increase experts per layer, and use shared routing weights within architectural blocks.Some models also replace the final classification layer with a 1x1 CondConv layer.
- ImageNet classification: CondConv improves MobileNetV1 accuracy relative to inference cost, and increasing experts yields monotonically increasing performance with sufficient regularization.This comparison is against static scaling through channel width and input size.
- ImageNet classification: 78.3% accuracy with 413M multiply-adds is achieved by CondConv-EfficientNet-B0, reported as state-of-the-art against the MixNet frontier.The depth-scaled variant reaches 79.5% accuracy with 614M multiply-adds, versus EfficientNet-B1 at 79.2% with 700M multiply-adds under the same search space.
- COCO object detection: CondConv with eight experts improves COCO detection at every model size, with relative inference-cost increases below 5%.The 0.75x model exceeds the 1.0x baseline by 0.7 mAP at 60% of its inference cost, while the 1.0x model gains 2.1 mAP at similar cost.
- Ablation studies: Sigmoid routing outperforms Softmax, suggesting that multiple experts are often useful for a single example.Routing at multiple network depths also outperforms routing only once later in the network.
5 Analysis
CondConv routing becomes increasingly class-specific at deeper layers, while experts specialize semantically and exhibit sparse, example-dependent activation patterns.
- Routing-weight distributions are similar across classes in early layers but become increasingly class-specific at deeper layers.The analysis compares cliff, pug, goldfish, and plane at Layers 12, 26, and the final fully connected layer.
- Final-layer routing weights cluster near 0 or 1, suggesting sparse expert activation and expert specialization without regularization.
- Within a class, some kernels remain consistently highly weighted, yet routing weights can vary substantially across examples.
- Experts specialize in semantically and visually meaningful ways, as shown by the classes receiving the highest mean routing weights.The analysis visualizes the top 10 classes for four experts using exemplar images with the highest routing weights.
6 Conclusion
The paper introduces CondConv to replace shared convolutional kernels with input-conditioned kernels, offering a way to increase capacity while maintaining efficient inference. It frames kernel-generating complexity as a more efficient scaling direction and identifies relationships between examples as an open research question.
- CondConv computes convolutional kernels as functions of the input rather than sharing static kernels across all examples.
- Increasing the size and complexity of the kernel-generating function can raise model capacity more efficiently than adding or expanding convolutions.The kernel is computed once and then convolved across the input.
- CondConv motivates studying how relationships between examples can be uncovered, represented, and leveraged for model performance.
- Future work includes larger datasets, more complex kernel-generating functions, architecture search, and the limitations of CondConv.
A ImageNet Architectures
Across ImageNet architectures, CondConv improves or matches accuracy at competitive multiply-add budgets, with gains also appearing when scaling EfficientNet-B0 against static compound scaling.
- CondConv-MobileNetV1: 71.6% accuracy at 190M multiply-adds matches the 71.7% accuracy of MobileNetV1 (1.0x) at 571M multiply-adds.The CondConv-MobileNetV1 model uses 32 experts per CondConv layer and a 0.5x width setting.
- CondConv-MobileNetV2: CondConv-MobileNetV2 (1.0x) achieves 74.6% accuracy at 329M multiply-adds, similar to the wider static MobileNetV2 (1.4x) architecture.
- CondConv-MnasNet-A1: CondConv-MnasNet-A1 achieves 76.2% accuracy with 329M multiply-adds, versus 74.9% with 312M multiply-adds for baseline MnasNet-A1.The static MnasNet-A2 reaches 75.6% accuracy with 340M multiply-adds.
- CondConv-ResNet-50: 78.6% accuracy at 4213 multiply-adds improves on ResNet-50’s 77.7% accuracy at 4096M multiply-adds.The paper reports this improvement with sufficient regularization.
- CondConv-EfficientNet-B0: 78.3% accuracy with 413M multiply-adds improves on EfficientNet-B0’s 77.2% accuracy with 391M multiply-adds.
- CondConv-EfficientNet-B0-depth: 79.5% accuracy with 614M multiply-adds exceeds EfficientNet-B1’s 79.2% accuracy with 700M multiply-adds under matched training conditions.The comparison uses CondConv-EfficientNet-B0-depth with a 1.1x depth multiplier.