Source-linked AI summary
WeightNet: Revisiting the Design Space of Weight Networks
Ningning Ma, Xiangyu Zhang, Jiawei Huang, Jian Sun
TL;DR
Designing convolutional weights remains a key CNN design issue, and the relationship between SENet and CondConv motivates a unified weight-space framework. WeightNet adds a grouped fully connected layer to generate convolutional weights directly. The paper reports improved trade-offs and performance across ImageNet classification and COCO detection, while noting that other weight-space structures remain possible.
Problem
Convolution-weight design is a key CNN issue, and the relationship between SENet and CondConv motivates investigating their common ground in weight-network space.
Source: §1 · pp. 1–2Method
WeightNet adds a grouped fully connected layer to the attention vector to generate convolutional weights directly, with input size and group number as adjustable design choices.
Source: §1 · p. 2, §3.3 · p. 7Results
Experiments report WeightNet’s superiority on ImageNet classification and COCO detection, with adjustable layer settings controlling the parameter–FLOPs–accuracy trade-off.
Source: §1 · p. 2, §5 · p. 14Takeaways & Limitations
The weight-space framework offers a simple structure for future research, and more complex structures within it may further improve performance.
Source: §5 · p. 14The explored weight-space design leaves open structures beyond fully connected layers and sparse-matrix forms beyond those shown in Fig. 4.
Source: §1 · p. 3
Abstract
from arXiv · showhide
We present a conceptually simple, flexible and effective framework for weight generating networks. Our approach is general that unifies two current distinct and extremely effective SENet and CondConv into the same framework on weight space. The method, called WeightNet, generalizes the two methods by simply adding one more grouped fully-connected layer to the attention activation layer. We use the WeightNet, composed entirely of (grouped) fully-connected layers, to directly output the convolutional weight. WeightNet is easy and memory-conserving to train, on the kernel space instead of the feature space. Because of the flexibility, our method outperforms existing approaches on both ImageNet and COCO detection tasks, achieving better Accuracy-FLOPs and Accuracy-Parameter trade-offs. The framework on the flexible weight space has the potential to further improve the performance. Code is available at https://github.com/megvii-model/WeightNet.
1 Introduction
Designing convolutional weights is a key CNN design issue, and prior weight-generating methods offer a flexible design space. WeightNet links SENet and CondConv by adding a grouped fully connected layer to generate convolutional weights, with adjustable settings that shape trade-offs in experiments on ImageNet and COCO.
1 Introduction
- The paper frames weight-generating methods as a design space for convolutional weights and seeks a simple, effective framework inspired by conditional networks.It identifies convolution-weight design as a key issue in CNNs.Source: §1 · p. 1
- WeightNet adds a grouped fully connected layer to the attention vector, directly generating convolutional weights and recasting SENet and CondConv operations as particular cases.The layer’s input size and group number are variable.Source: §1 · p. 2
- The added layer’s input size and group number distinguish the SENet and CondConv cases and jointly affect the parameter–FLOPs–accuracy trade-off.CondConv uses a minimum group number of one and a small input; SENet uses a group number equal to the input-channel count.Source: §1 · p. 2
- Experiments on ImageNet classification and COCO detection report WeightNet’s superiority, with its settings controlling the parameter–FLOPs–accuracy trade-off.Figure 1 compares accuracy against FLOPs and parameter count on ImageNet using ShuffleNetV2.Source: §1 · p. 2, Figure 1 · p. 2
- The authors identify potential for weight-space structures beyond fully connected layers and for additional sparse-matrix forms.They present these as directions for future work.Source: §1 · p. 3
2 Related Work
Related work covers networks that generate weights, conditional CNNs that adapt computation to input samples, and attention mechanisms that adjust networks based on input. WeightNet is distinguished as operating in kernel space rather than feature space.
2 Related Work
- Earlier weight-generating approaches include fast weights for recurrent connections, filter-generating networks, HyperNetworks, and meta networks.These methods provide a design space for generating network weights.Source: §2 · p. 3
- Conditional CNNs use input-conditioned dynamic kernels, widths, or depths, while dynamic-depth methods skip layers for different samples.The passage reports improvement for conditional CNNs and notes CondConv’s mixture of sample-selected experts.Source: §2 · p. 3
- Attention mechanisms adjust networks based on input; examples include SENet’s channel-wise gating, GaterNet’s binary masks, and kernel selection.WeightNet is designed in kernel space rather than feature space.Source: §2 · p. 3
3 WeightNet
WeightNet unifies SENet and CondConv in weight space by adding a grouped fully-connected layer that directly generates convolution weights. Its grouping and input-size choices provide a flexible design space for balancing parameter count and accuracy.
3 WeightNet
- WeightNet adds a grouped fully-connected layer after the attention activation vector to generate convolution weights directly, rather than applying the vector in feature space.The framework is composed of grouped or standard fully-connected layers.Source: §3 · pp. 3–4
3.3 WeightNet Structure
- GAP reduces feature dimensions before weight generation, making weight-branch FLOPs relatively negligible; reshaping batches enables memory-conserving training and inference.The weight branch has O(MCCkhkw/G) FLOPs, compared with O(hwCCkhkw) for the convolution branch.Source: §3 · p. 4, Complexity analysis · p. 8, Training with batch dimension · p. 8
3.1 Rethinking CondConv
- The CondConv reformulation avoids memory-consuming training and the batch-size-one restriction when m > 4, allowing efficient training with large batches.The original restriction arose when increasing the number of experts m.Source: §3.1 · p. 6
3.2 Rethinking SENet
- At kernel level, SENet’s channel scaling becomes a grouped sparse operation with C inputs, C × C × kh × kw outputs, and C groups.The prose contrasts this kernel-level operation with scaling features before or after convolution.Source: §3.2 · p. 6
3.3 WeightNet Structure
- The general grouped layer spans group counts from 1 to the input-channel count C, making CondConv and SENet its two extreme cases.CondConv uses the minimum group count of one; SENet uses the maximum, C.Source: §3.3 · pp. 6–7, Figure 4 · p. 7
- M controls the input size and G the group count; together they control the parameter-accuracy tradeoff, with CondConv and SENet represented by specific M,G settings.CondConv uses M = m/C and G = 1/C; SENet uses M = 1 and G = 1. The added layer takes M × C inputs and has G × C groups.Source: §3.3 · p. 7, Implementation details · p. 7
4 Experiments
Experiments compare WeightNet with conditional modules and standard convolution on ImageNet classification and COCO detection, then examine parameter, grouping, placement, pooling, and generated-weight choices.
4.1 Classification
- Under matched budgets, WeightNet outperforms SE and CondConv across ShuffleNetV2 and ResNet50 on ImageNet.ShuffleNetV2 (0.5×) gains 3% Top-1 accuracy without additional computation budgets; gains are 5.7% and 2.0% Top-1 accuracy for ShuffleNetV2 (0.5×) and (2×), respectively. ShuffleNetV2 (2×) also outperforms ResNet50 with 40% of its parameters and 14.6% of its FLOPs.Source: ShuffleNetV2 · pp. 9–10, Table 4 · p. 10, ResNet50 · p. 11
4.2 Object Detection
- 4.6 mAP improvement over standard convolution is reported for ShuffleNetV2 (0.5×) with WeightNet (4×) on RetinaNet at the same FLOPs.WeightNet also outperforms CondConv uniformly under the same computational budgets on COCO detection.Source: Table 5 · p. 11, §4.2 · p. 11, Table 6 · p. 11
4.3 Ablation Study and Analysis
- Optimal λ values for ShuffleNetV2 {0.5×, 1×, 1.5×, 2×} are {8, 4, 4, 4}; capacity is upper-bounded as λ increases.The experiments report an optimal λ for performance rather than continued gains from increasing the parameter multiplier.Source: The influence of · pp. 11–12, Table 7 · p. 12
- Layer-wise GAP has Top-1 err. 35.04, versus block-wise GAP 35.47 and stage-wise GAP 37.01; adding GAP operators improves model capacity.The comparison uses ShuffleNetV2 0.5× on ImageNet.Source: The influence of · p. 12, Table 10 · p. 12, The number of the global average… · p. 13
- Increasing G improves model capacity; the last stage has the largest influence, and performance is best when WeightNet replaces convolutions in the last two stages.The study also compares G values {1, 2, 4} while holding λ at 1.Source: The influence of · p. 12, WeightNet on different stages · p. 12, Table 8 · p. 12
- Generated weights form roughly 20 class-associated t-SNE clusters, while WeightNet kernels show lower channel similarity than standard convolution.The t-SNE analysis uses weights from 1,000 validation samples across 20 classes; the cosine-similarity comparison uses five distinct-class samples.Source: Figure 5 · p. 13, Weight similarity among distinct samples · p. 13, Figure 6 · p. 14
5 Conclusion and Future Works
WeightNet unifies SENet and CondConv in weight space, with their grouping schemes as two extremes of a framework controlled by M and G. The authors see further performance improvement as a possibility for more complex structures.
5 Conclusion and Future Works
- WeightNet unifies SENet and CondConv in weight space, with their grouping manners as two extreme cases.The framework consists entirely of grouped or ungrouped fully-connected layers.Source: §5 · p. 14
- Adjusting M and G provides a straightforward structure with better trade-off results.The authors describe these as previously uninvestigated hyperparameters.Source: §5 · p. 14
- More complex structures in the framework have the potential to further improve performance.The authors identify the flexible weight-space framework as an area for future work.Source: §5 · p. 14