IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks

• Portable, accurate vision models motivate both compressing pretrained networks and designing lightweight networks from scratch.The design approach in this section focuses on less-redundant kernels for lightweight networks.Source: §1 · p. 1
• Low-rank kernel sequences and sparse-kernel sequences are two established ways to compose convolutional kernels.Examples include bottlenecks and inverted residual blocks for low-rank designs, and interleaved group convolutions, MobileNetV1, Xception, and deep roots for sparse designs.Source: §1 · pp. 1–2
• IGCV3 combines low-rank and sparse kernels in a modular block to compose a dense convolutional kernel.It introduces low-rank design into group pointwise convolutions, expanding and then projecting the feature dimension.Source: §1 · p. 2
• The loose complementary condition imposes complementarity over super-channels, handling unequal input and output channel counts while avoiding over-wide convolutions.The paper describes this as equivalent to IGCV2’s strict complementary condition when imposed over super-channels.Source: §1 · p. 2
• IGCV3 is reported superior to IGCV2 and MobileNetV2 on CIFAR and ImageNet classification and COCO object detection.The reported results concern the combination of low-rank and sparse kernels.Source: §1 · p. 2

• Kernel redundancy reduction includes lowering numerical precision and removing superfluous weights.Examples include binarization [3], trinarization [21] [45] [46], and quantization [44].Source: §2 · p. 2
• Sparse-kernel approaches impose sparsity constraints, while low-rank approaches prune weights or channels or factorize larger kernels.Group convolution is described as a predefined structured-sparse matrix; spatial factorization examples include 3 × 3 kernels decomposed into 3 × 1 and 1 × 3 convolutions.Source: §2 · p. 2
• Compositions of sparse kernels retain dense connectivity through permutations, while bottlenecks can be viewed as low-rank approximations along the output-channel domain.The passage names IGCV1, IGCV2, Xception [2], and bottleneck designs [10] [14] [31].Source: §2 · pp. 2–3

3.1 Prior Works

• IGCV2 extends IGCV1 with more structured sparse matrices, while MobileNetV1 combines channel-wise spatial and pointwise convolutions and MobileNetV2 uses an inverted bottleneck.In MobileNetV2, the first dense pointwise convolution increases width and the second reduces it.Source: §3.1 · p. 4

3.2 Interleaved Low-Rank Group Convolutions

• IGCV3 replaces IGCV2's group convolutions with low-rank group pointwise convolutions around a channel-wise spatial convolution.The two pointwise stages reduce and then expand width; the composed pointwise kernel is designed to be dense.Source: §3.2 · pp. 4–5
• The loose complementary condition tests branch coverage over super-channels, relaxing the strict condition defined over channels.The paper motivates the relaxation by observing that over-sparse convolutions produce wider response maps and higher memory cost without higher accuracy.Source: §3.2 · pp. 5–6

3.3 Discussions and Analysis

• The paper uses linear IGCV3 for very deep networks and reports that nonlinear IGCV3 performs better for networks with only 10+ blocks.The linear version has no intermediate ReLU; the nonlinear version has one or more intermediate ReLUs.Source: §3.3 · p. 6
• The inverted IGCV3 block places the identity connection between low-dimensional representations and is adopted to save training and inference memory.For comparisons with MobileNetV2, the paper also studies networks that are wider at equal depth or deeper at equal width.Source: §3.3 · p. 6

4.1 Datasets

• CIFAR-10 and CIFAR-100 each contain 60,000 32×32 images, while ImageNet uses over 1.2 million training images across 1,000 categories.COCO detection training uses the union of 80k training images and a 35k validation subset, with evaluation on test-dev.Source: §4.1 · pp. 6–7

4.2 Comparisons with IGCV1 and IGCV2.

• IGCV3 improves ImageNet classification accuracy by about 1.5% over prior IGC blocks at the same parameter count.The Table 1 comparison reports mean±std across five runs; parameter counts exclude the final FC-layer.Source: Table 1 · p. 7, §4.2 · p. 7
• On CIFAR, IGCV3 slightly outperforms IGCV1 and IGCV2, whose comparisons use models with the same number of parameters.The CIFAR results for IGCV1 and IGCV2 were reproduced by the authors.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Comparisons with Other Mobile Networks.

• On CIFAR, IGCV3 outperforms MobileNetV2 with similar parameter counts; the 50%-parameter version also performs better at the same depth.The authors reproduced MobileNetV2 on four GPUs and used the best reproduced settings to train IGCV3.Source: §4.3 · pp. 7–8, Table 2 · p. 8
• IGCV3-D (1.4) reaches 74.55% vs 73.76% under the same ImageNet training settings, but is marginally worse than original MobileNetV2 (1.4).The comparison counts Multiply-Adds; IGCV3-D has different parameter counts across datasets because of the final FC-layer.Source: Table 3 · p. 8, §4.3 · p. 8
• On COCO, IGCV3 is slightly better than MobileNetV2 with fewer parameters and exceeds YOLOV2 by 0.6% mAP with many fewer parameters.The comparison evaluates methods at different input sizes; reproduced MobileNetV2 and IGCV3 use 320×320 inputs.Source: §4.3 · p. 8, Table 4 · p. 9

Deeper and wider networks

• The deeper version significantly outperforms the wider version in the comparisons on CIFAR and ImageNet.The authors attribute this to redundancy in feature dimensions and improved performance as bottleneck-stacked networks deepen.Source: Table 5 · p. 9, §5 · p. 9

Intermediate ReLUs

• Among tested ReLU placements, the IGCV3 block (second block) has obvious advantages; the first uses twice as many ReLUs and may perform worse.The third block is equivalent to a convolution followed by ReLU and is reported as worse than IGC blocks.Source: §5 · pp. 9–10, Table 6 · p. 9

Branches in group convolutions

• The first group convolution prefers to be denser, while reducing kernels in the projecting group convolution may have little effect on performance.The authors attribute the latter to information loss when projecting high-dimensional features back to a low-dimensional space.Source: §5 · p. 10, Table 7 · p. 10
• Allowing IGCV3 blocks to use different settings can yield a deeper, better-performing network; experiments use G1 = 2 and G2 = 2 to reduce memory cost while retaining good performance.The text notes many possible group-count combinations satisfying the loose complementary condition, with G1 × G2 dividing both input and output channel counts.Source: §5 · p. 10, Table 7 · p. 10

6 Conclusion

• The proposed interleaved low-rank group convolution block combines low-rank and sparse convolutions.The design introduces super-channels and a loose complementary condition to guide construction.Source: §6 · p. 10
• Empirical results show IGCV3 models are efficient and superior to IGCV2 and MobileNetV2 on several benchmarks.The conclusion names image classification and object detection among the evaluated tasks.Source: §6 · p. 10
• The paper presents interleaved low-rank group convolutions as a novel block that benefits from both low-rank and sparse convolutions.Its design uses super-channels and a loose complementary condition.Source: §6 · p. 10