Source-linked AI summary
Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections
Elian Iluk, Gil Ben-Artzi
TL;DR
Duplicated convolutional experts that share inputs can collapse to nearly identical filters, limiting the value of operator-level MoE in CNNs. MoCE replaces dense pointwise projections with learned static sparse channel supports and input-adaptive temperature mixing, achieving comparable or better accuracy with lower computation and latency. Its main design conclusion is that adapting how fixed channels are mixed is more useful and schedulable than adapting which channels are read.
Problem
Parallel convolutional experts sharing the same input can learn nearly identical filters, raising the question of how to obtain useful specialization without inefficient operator duplication.
Method
MoCE replaces dense pointwise projections with one output-channel expert per channel, using learned static top-k supports, input-dependent temperature mixing, residual coverage, and load balancing.
Results
Across evaluated ResNet, EfficientViT, transfer, and classification settings, MoCE preserves or improves accuracy while reducing computation and deployment cost; ResNet-50 MACs fall by 16.7%.
Takeaways & Limitations
Input dependence is more valuable for adjusting how a fixed channel set is mixed than for changing which channels are read, while static supports remain schedulable.
Takeaways & Limitations
MoCE removes arithmetic but not the Θ(CHW) activation traffic, making network-level speedup smaller than MAC reduction and leaving execution bandwidth-dominated.
Abstract
from arXiv · showhide
Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.
1 Introduction
Pointwise projections are pervasive and costly, while duplicated convolutional experts sharing inputs tend to learn nearly identical filters. MoCE instead specializes through learned sparse channel supports with limited input-adaptive mixing and retains the efficiency benefits in measured evaluations.
- Motivation: Pointwise projections mix channels at every spatial location across modern vision backbones, with cost CinCoutHW that grows with channel width.They occur in ResNet bottlenecks, inverted bottlenecks, ConvNeXt-style blocks, and vision-transformer feed-forward projections.
- Motivation: 0.88 mean off-diagonal cosine similarity was observed among eight convolutional experts sharing inputs in a ResNet-50/CIFAR-100 diagnostic.Most expert pairs fell between 0.85 and 0.97, indicating strongly aligned learned kernels.
- MoCE: MoCE replaces duplicated operators with one aggregation unit per output channel, each using a learned top-k input-channel support and convex mixing.Supports are static at inference, enabling fixed gathers and packed execution.
- MoCE: +0.03 points accuracy was obtained by the dynamic routing variant, but its routing-and-gather path was 7.13× slower; removing temperature gating reduced accuracy by 1.19 points.These results position static supports with adaptive mixing as the evaluated operating point.
- Results: 17–21% fewer MACs were reported across ResNet and EfficientViT backbones while preserving or improving the dense accuracy–efficiency operating point.Deployed-parameter reductions on ResNet variants were also 17–21%, while end-to-end speedup was smaller than MAC reduction.
2 Related Work
MoCE connects sparse expert routing with structured sparse pointwise projections and input-dependent channel reweighting. Its distinctive combination is static learned supports, adaptive mixing, residual coverage, and hardware-oriented execution.
- Sparse experts and routing: Sparse MoE makes routing balance and stability central by activating only a subset of experts for each input.Related approaches include expert-side capacity assignment and differentiable or soft alternatives to hard routing.
- Structured sparse projection: Structured sparsity methods learn and freeze grouped 1 × 1 connectivity, providing the closest precedent for MoCE’s static-support principle.MoCE further constrains each routed output to normalized nonnegative weights and adds adaptive mixing.
- MoCE: Figure 2’s design uses E −1 routed experts over learned static supports together with one residual aggregate.The residual component complements routed channel aggregation in the layer design.
- Conditional computation in CNNs: MoCE differs from dynamic channel-selection methods by combining a sparse static support with an input-dependent mixing rule.The paper frames it as a sparse projection with an adaptive mixing rule rather than a replacement for broader structured pruning.
3 Mixture of Channel Experts
MoCE replaces dense pointwise channel projections with fixed sparse supports, adaptive convex mixing, and a residual summary, retaining efficient inference without data-dependent branching. Its cost scales favorably with channel width, producing substantial MAC and deployment savings, while measured speedups are constrained by activation traffic.
- Architecture: MoCE replaces a dense pointwise map with E−1 routed experts and one residual aggregate, applying to projections with E ≤ C but not spatial convolutions.Each routed expert uses a learned top-k support; inference precomputes supports and requires neither top-k search nor data-dependent branching.
- Adaptive mixing: Each expert combines its selected channels using routing preferences and an input-dependent temperature that ranges from concentrated selection toward uniform averaging.The temperature changes the entropy of fixed routing preferences while preserving their ordering.
- Residual coverage: Every input channel contributes through either a routed expert or a residual summary, with 7.7% of channels uncovered on average across replaced layers.Replacing the residual mean with temperature-gated aggregation changed accuracy negligibly, so the cheaper residual form is retained.
- Residual coverage: The coverage loss equalizes aggregate soft channel usage but does not guarantee disjoint hard supports, whose overlap and usage are measured empirically.The regularizer also supplies gradients to unselected logits.
- Cost analysis: Relative to dense cost CEHW, MoCE’s leading spatial-work ratio is approximately k/C because pooling and residual overheads scale linearly in C.Using measured |U| ≈ 0.077C tightens one overhead term to ≈1.08/E, although reported MACs use the conservative bound.
- Measured efficiency: 16.7% of the whole ResNet-50 network’s MACs are saved, reducing replaced-projection cost from 706.5 M to 20.1 M; deployment also replaces 4.33 M dense weights with approximately 30.1 k retained logits and 60 KB of indices.ResNet-152 savings reach 20.8%; training retains the full routing matrix, so the parameter reduction is a deployment-time property.
- Measured efficiency: A 128× MAC reduction on the deepest replaced projection yields a 3.0× measured speedup because sparsification removes arithmetic but not Θ(CHW) activation traffic.The roofline-style attribution is explanatory rather than profiler-verified, and further traffic reduction offers more headroom than reducing k alone.
4 Experiments
Experiments evaluate MoCE across ResNet, CIFAR-100, transfer learning, EfficientViT, ablations, and measured execution. MoCE generally preserves or improves accuracy while reducing computation, parameters, and selected routing overhead, with end-to-end gains smaller than MAC reductions.
- Setup: MoCE replaces dense ResNet bottleneck entrance projections while retaining the surrounding block structure and using learned routing, shared temperature gates, and coverage regularization.The default settings use k = 8, bounded temperature, and dataset-specific coverage weights.
- ImageNet-1K: 0.73 points higher Top-1 accuracy accompanies 16.7% fewer MACs and 16.8% fewer deployed learned parameters on ResNet-50.ResNet-101 preserves accuracy within observed variation at 19.7% lower MACs, while ResNet-152 gains 0.46 points at 20.8% lower MACs.
- Conditional Channel Methods: 17% fewer MACs and 6.8 M fewer deployed learned parameters let MoCE reach SE-level accuracy, while MoCE also exceeds Pick-or-Mix’s accuracy point estimate at comparable measured batch time.These comparisons use methods retrained under the same ResNet-50 protocol.
- CIFAR-100 and Transfer: 17.2–21.0% lower MACs accompany improved ResNet-50 CIFAR-100 accuracy and preserved ResNet-152 accuracy in both from-scratch and ImageNet-to-CIFAR transfer settings.ImageNet-learned supports remain fixed during transfer fine-tuning and remain effective after the label space changes.
- EfficientViT: 18–22% lower MACs with accuracy preservation extend MoCE beyond ResNet to EfficientViT’s second feed-forward projection.The replaced layer is token-wise linear, corresponding to a pointwise projection.
- Where Input Dependence Pays: Removing input-conditioned temperature reduces accuracy by 1.19 points, whereas the more expensive dynamic-support router changes accuracy by only +0.03 points and increases routing-and-gather time 7.13-fold.The dynamic variant rises from 1.9 to 13.5 ms for the measured routing-and-gather stages.
- Sparsity, Coverage, and Temperature: k = 8 is selected by representation rather than compute parity: k = 4 limits capacity, k ≥10 reduces specialization, and break-even occurs at k⋆≥63.The default coverage weight λ = 5 × 10−4 gives the best point estimate; extreme weights permit concentration or over-constrain routing.
- Measured Batch Execution Time: 19.63 to 15.60 ms lower execution time across replaced operators, while the network-level gain is 4.6%, smaller than the MAC reduction.A representative 2048 →512 projection runs at 0.331× dense time, but bandwidth traffic remains because the sparse path still reads the input activation tensor.
5 Conclusion
MoCE replaces dense pointwise projections with learned, frozen channel supports and input-dependent temperatures, preserving or improving accuracy while reducing computation, storage, and execution time.
- MoCE learns channel supports during training, freezes them afterward, and uses one input-dependent temperature per expert.
- MoCE preserves or improves accuracy across evaluated settings while reducing computation, deployment storage, and measured execution time.
- Input dependence is more useful for adjusting how fixed channel sets are mixed than for changing which channels are read.
- A strictly more expressive input-conditioned router gained nothing measurable at 7.13× the routing-and-gather cost, while removing the temperature gate reduced accuracy by 1.19 points in the same run.
- Once sparsified, the layer is bounded by memory traffic rather than arithmetic.