Source-linked AI summary
Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
Chun-Fu Chen, Quanfu Fan, Neil Mallinar, Tom Sercu, Rogerio Feris
TL;DR
CNN accuracy often grows with computational cost, while many compression approaches operate on single-resolution inputs. Big-Little Net fuses branches with different resolutions and computational complexity; it reports reduced computation with improved object-recognition accuracy and better speech word error rates.
Problem
CNN accuracy usually grows with complexity and computational cost, and many compression methods use single-scale inputs despite the success of multi-scale representations.
Source: §1 · p. 1Method
Big-Little Net combines features from branches operating at different image scales and computational complexities through repeated fusion.
Source: §1 · pp. 1–2, §3.1 · p. 3Results
The approach reduces computation by 1/3 in models such as ResNet and ResNeXt while improving accuracy by over 1 percentage point, and reports better speech word error rates with 30% fewer FLOPs.
Source: §1 · p. 2Takeaways & Limitations
The authors state that bL-Net can be combined with CNN acceleration approaches to make models more compact and efficient.
Source: §5 · p. 9
Abstract
from arXiv · showhide
In this paper, we propose a novel Convolutional Neural Network (CNN) architecture for learning multi-scale feature representations with good tradeoffs between speed and accuracy. This is achieved by using a multi-branch network, which has different computational complexity at different branches. Through frequent merging of features from branches at distinct scales, our model obtains multi-scale features while using less computation. The proposed approach demonstrates improvement of model efficiency and performance on both object recognition and speech recognition tasks,using popular architectures including ResNet and ResNeXt. For object recognition, our approach reduces computation by 33% on object recognition while improving accuracy with 0.9%. Furthermore, our model surpasses state-of-the-art CNN acceleration approaches by a large margin in accuracy and FLOPs reduction. On the task of speech recognition, our proposed multi-scale CNNs save 30% FLOPs with slightly better word error rates, showing good generalization across domains. The codes are available at https://github.com/IBM/BigLittleNet
1 INTRODUCTION
CNN accuracy often grows with model complexity and computational cost, challenging real-time and low-memory deployment. Big-Little Net combines branches at different image scales and computational costs, reporting reduced computation with improved recognition performance.
1 INTRODUCTION
- CNN accuracy usually grows with model complexity and computational cost, creating deployment challenges for real-time and low-memory applications.Examples include self-driving vehicles, mobile-device interaction, and robotics.Source: §1 · p. 1
- Many compression methods use single-scale inputs, while multi-scale networks have had less attention to computational complexity.The paper contrasts pruning, quantization, and low-rank factorization with multi-resolution processing.Source: §1 · p. 1
- Big-Little Net pairs a costly, accurate branch at low image resolution with a cheaper, less accurate branch at high resolution, frequently merging their features.Figure 1 depicts stacked modules with branches representing different image scales.Source: §1 · pp. 1–2, Figure 1 · p. 2
- The approach reduces computation by 1/3 in models such as ResNet and ResNeXt while improving accuracy by over 1 percentage point, and reports better word error rates with 30% fewer FLOPs on speech recognition.The paper also reports computational savings up to 1/2 without losing accuracy.Source: §1 · p. 2
2 RELATED WORK
Prior efficiency approaches and multi-scale feature methods address different aspects of the problem. Big-Little Net uses branches with different computational capacities across scales and fuses them at multiple network levels.
Model Compression and Acceleration
- Pruning, quantization, sparse models, compact architectures, distillation, structured matrices, hashing, and dynamic routing are described as methods operating on single-resolution inputs; the authors say bL-Net could complement them.The proposed approach processes multi-resolution data and could be used in tandem with these methods to improve efficiency.Source: §2 · p. 2
Multi-Resolution Feature Representations
- Some multi-scale methods use equal-complexity subnetworks or incur rising cost as the number of scales grows, whereas bL-Net assigns different capacities to scales and fuses features at multiple levels.The passage contrasts this design with hierarchical multi-resolution and multiple-network approaches.Source: §2 · p. 3
- Related multi-scale branching approaches gain efficiency mostly through input-dependent early exits, while bL-Net speeds up a network regardless of the input image.Source: §2 · p. 3
3 OUR APPROACH
Big-Little Net builds multi-scale representations with branches whose computational cost is inversely related to image resolution. Its modules align and merge branch features, with branch capacity and merging choices controlling the design.
3 OUR APPROACH
- Each branch represents one image resolution, and its computational cost is inversely proportional to that scale, assigning higher complexity to lower-resolution branches.Source: §3 · p. 3
3.1 BIG-LITTLE NET
- A Big-Little Module combines branch features at each scale by weighted sum after branch outputs are matched in size and feature-map count.The module may apply a final fusion layer; branches are merged at the end of every module, retaining the largest intermediate spatial size and feature-map count.Source: §3.1 · p. 3
- The two-branch implementation uses a deeper, lower-resolution Big-Branch and a shallower, higher-resolution Little-Branch, with upsampling and channel adjustment before fusion.For the ResNet-based design, fusion uses unit weights and a residual block.Source: §3.1 · p. 4
- The Little-Branch's width and depth are controlled by α and β, respectively; larger reduction factors lower complexity, and its cost can be 1/6 of bL-Net and 1/12 of the original network.Source: §3.1 · p. 4
3.2 NETWORK MERGING
- Linear combination merges features by addition after matching feature-map size and channel count, whereas concatenation joins channels and may require a 1×1 convolution afterward.Source: §3.2 · p. 4
- The authors choose linear combination because it provides immediate exchange between branches with little overhead and better accuracy than concatenation in object recognition.The passage notes that concatenation delays exchange until later layers.Source: §3.2 · p. 4
4 EXPERIMENTAL RESULTS
The experiments evaluate Big-Little networks on object and speech recognition, then compare them with other accelerated and efficient architectures. Across these evaluations, the results report improved accuracy or word error rate alongside computational savings, with design observations about branch size and merging.
4 EXPERIMENTAL RESULTS
- The experiments evaluate bL-Net with ResNet as the primary backbone and also integrate it with ResNeXt and SEResNeXt; speech models are trained on Switchboard+Fisher and evaluated on Hub5 portions.The speech evaluation uses cross-entropy training and decoding with a small language model.Source: §4 · p. 4, §4.1 · p. 5, §4.2 · p. 7
4.1 OBJECT RECOGNITION
- All ResNet-50-based models improve performance over the baseline with less computation; a Little-Branch under 15% of network size can compensate for low-resolution features.The paper uses α = 2 and β = 4 as the default setting for subsequent experiments.Source: §4.1 · p. 5, Table 1 · p. 5, Table 2 · p. 5
- At 256 × 256, bL-ResNeXt-101 (64×4d) improves top-1 performance by 1.1% with 40% fewer FLOPs, and GPU speedups track theoretical FLOP reductions.The paper also reports 2× speedups for bL-ResNeXt-101 and bL-SEResNeXt-101, with accuracy gains of 0.12% and 0.13%, respectively.Source: Table 2 · p. 5, §4.1 · p. 6
4.1.1 COMPARISON WITH RELATED WORK
- bL-ResNet-101 improves accuracy over pruning and dynamic-routing approaches at lower or equal FLOPs, and outperforms several methods at matched FLOPs.The reported matched-FLOP gains include 3.7% over BlockDrop, 2.2% over SkipNet, 1.2% over ConvNet-AIG, and 2.2% over SPPoint.Source: §4.1.1 · p. 6, Figure 2 · p. 6
- bL-ResNeXt has a better efficiency–accuracy tradeoff than Inception networks; it is 2% better than DenseNet at the same running speed and 2× faster than DualPathNet at the same performance.NASNet has lower FLOPs and higher accuracy than bL-Net, but slower GPU running time; bL-ResNeXt is 0.5% better and 1.5× faster than NASNet’s lowest-FLOP configuration.Source: §4.1.1 · p. 7, Figure 3 · p. 7
4.2 SPEECH RECOGNITION
- The best bL-ResNet-22 setting reduces FLOPs and modestly improves WER over baseline; shorter full-resolution Little-Branches reduce FLOPs without WER degradation, while more feature maps support performance.The authors identify α = 2 and β = 3 as the best setting; concatenation performs similarly to addition but increases FLOPs, and extra lower-layer branches may not justify their added parameters and computation.Source: §4.2 · p. 8, Table 3 · p. 8
4.3 DISCUSSION ON bL-Net
- Across both tasks, the authors report that a light Little-Branch can suffice, wider and shallower Little-Branches perform better, and addition merging is more effective than concatenation.They attribute addition’s advantage to better regularization that supports complementary branch features.Source: §4.3 · p. 8
5 CONCLUSION
Big-Little Net combines branches operating at different image scales for object and speech recognition. The paper reports lower computation with competitive or improved performance, and notes that the architecture can extend to other CNNs.
5 CONCLUSION
- Object recognition achieves approximately 2× speedup over baselines while improving accuracy; speech recognition saves 30% FLOPs alongside a reported 0.2% WER gain.The authors also report that reduced FLOPs consistently speed up GPU runtime.Source: §5 · p. 8
- Big-Branches work at low resolution to reduce computation while extracting meaningful features, and Little-Branches enrich them with features from high-resolution input.The paper describes bL-Net as integrating multiple networks for multi-scale feature representation.Source: §5 · p. 8
Appendix
- The appendix provides experiment details for object and speech recognition, along with additional ablation studies.Source: Appendix · p. 13
A.1 EXPERIMENTAL SETUP
The appendix describes the ImageNet training and validation setup and the configurations used to build bL-ResNets from ResNets. It also details shared transitions and stage redistribution for deeper models.
A.1 EXPERIMENTAL SETUP
- Models were trained on ImageNet for 110 epochs with batch size 256, weight decay 0.0001, momentum 0.9, and initial learning rate 0.1.Training used cosine learning-rate scheduling; validation resized the shorter image side to 256 before a center 224 × 224 crop.Source: §A.1 · p. 13
- The reported vanilla ResNet, ResNeXt, and SEResNeXt validation results differ from their original papers: ResNet is better, while ResNeXt and SEResNeXt are slightly worse.Source: §A.1 · p. 13
A.2 NETWORK STRUCTURE
- For bL-ResNet-50, α = 2 and β = 4; Big- and Little-Branches share a transition residual block, and the Little-Branch has one block per stage.The block described uses 1 × 1, 3 × 3, and 1 × 1 convolutions, with C/4, C/4, and C kernels, respectively.Source: §A.2 · pp. 13–14
- For bL-ResNet-101 and bL-ResNet-152, some blocks move from the 4th stage to the 2nd and 3rd stages to distribute computation across stages.The authors motivate this change by noting that the original models concentrate most convolutions in the 4th stage, limiting Big-Branch access to information at larger scales in earlier stages.Source: §A.2 · p. 14, Table 5 · p. 14
A.3 PERFORMANCE ON LOW RESOLUTION INPUT
The low-resolution baseline reduces computation substantially but has unacceptable accuracy; bL-ResNet-50 provides a better balance. Examples also show its predictions retaining confidence or small object details where the low-resolution model does not.
A.3 PERFORMANCE ON LOW RESOLUTION INPUT
- When both models predict correctly, bL-ResNet-50 gives higher confidence; it also captures small object details that let it predict correctly when ResNet-50-lowres is wrong.The examples include a bird's beak shape and ladybug spots.Source: §A.3 · p. 14, Figure 4 · p. 15
- bL-ResNet-50 balances accuracy and performance better than ResNet-50-lowres, whose computation falls substantially but whose accuracy is unacceptable.The authors report a similar trend for ResNet-101-lowres and characterize its performance as reasonable given an almost 3–4× computation reduction.Source: §A.3 · p. 14, Table 6 · p. 15
A.4 ABLATION STUDY ON NETWORK MERGING AND MULTI-BRANCH
The ablations compare feature-merging schemes, branch scales, and merge counts. They favor addition over concatenation, two scales over the tested three-scale setup, and merging before feature dimensions change.
A.4 ABLATION STUDY ON NETWORK MERGING AND MULTI-BRANCH
- Concatenation is more efficient, but performs almost 1.5% worse than addition, which the authors favor for both visual and speech tasks.Source: Table 7 · p. 15, §A.4 · p. 15
- The tested three-scale bL-ResNet-50 requires more FLOPs and parameters than the two-scale model and does not outperform it.The authors attribute this to merge overhead and the Big-Branch's 4× downsampling, which degrades its feature representation.Source: Table 7 · p. 15, §A.4 · p. 15
- For bL-ResNet-50, fewer merges (m = 1 or 2) give significantly worse accuracy than m = 4 without saving many FLOPs or parameters; extra merges in bL-ResNet-101 do not improve performance and require more FLOPs.The authors identify merging before a feature dimension changes as the most effective setting; extra merges at identical dimensions can be redundant.Source: Table 8 · p. 16, §A.4 · p. 16
A.5 bL-Net FOR HIGH-EFFICIENCY NETWORK
The authors apply bL-Net to ShuffleNetV2 and report improved accuracy under similar FLOPs.
A.5 bL-Net FOR HIGH-EFFICIENCY NETWORK
- bL-ShuffleNetV2@256 outperforms ShuffleNetV2 by up to 0.5% point under similar FLOPs.The result is presented in Table 9.Source: §A.5 · p. 16, Table 9 · p. 16
- Table 9 compares ShuffleNetV2 and bL-ShuffleNetV2@256 with α = 2 and β = 2.Source: Table 9 · p. 16
- The authors report that all models were trained by themselves.Source: §A.5 · p. 16
A.6 bL-Net FOR OBJECT DETECTION
The authors evaluate bL-Net as a FasterRCNN+FPN backbone on PASCAL VOC and MS COCO, reporting competitive detection performance with about 1.5× FLOPs savings.
A.6 bL-Net FOR OBJECT DETECTION
- The evaluation uses bL-Net as the backbone for FasterRCNN+FPN on PASCAL VOC and MS COCO.Source: §A.6 · p. 16
- bL-Net achieves competitive detection performance while saving about 1.5× FLOPs.The authors suggest this indicates transferability to other vision tasks.Source: §A.6 · pp. 16–17, Table 10 · p. 17
- The reported FLOPs use 600×1024 input images for one setting and 800×1344 for another, with 300 proposals in each.Source: §A.6 · p. 17
A.7 COMPARISON OF MEMORY REQUIREMENT
In the memory comparison, bL-Net is reported as most memory-efficient during training; test-phase memory varies across its models and comparison networks.
A.7 COMPARISON OF MEMORY REQUIREMENT
- For bL-Net, the benchmark uses 224×224 input images during training and 256×256 during testing.The image size for other models matches the experiment reported in Figure 3.Source: §A.7 · p. 17
- At test time, bL-SEResNeXt outperforms all approaches, while bL-ResNeXt uses more memory than Inception-ResNet-v2 and Inception-v4 at the same accuracy.Source: §A.7 · p. 17
- NASNet and PNASNet are described as not memory friendly, largely because they use 331×331 images and have many layers.Source: §A.7 · p. 17
- bL-Net is the most memory-efficient approach for training among those compared.Figure 5 reports training and test memory requirements; the benchmark uses batch size 8.Source: Figure 5 · p. 17, §A.7 · p. 17
B.1 EXPERIMENTAL SETUP
The speech experiments use log-mel inputs and a ResNet-22 baseline, then describe bL-Net branch configurations and how their features are aligned and merged.
B.1 EXPERIMENTAL SETUP
- Speech inputs have dimensions batch_size×3×64×49, and outputs are projected to 32k tied CD states for central-frame phone prediction.The input stacks log-mel filterbanks with first and second derivatives; results are reported on Hub5’00 after decoding with a 4M n-gram language model.Source: §B.1 · p. 18
B.2 NETWORK STRUCTURES
- The ResNet-22 baseline has four stages of residual blocks, followed by a bottleneck projection to the 32k-dimensional output.The model avoids padding and striding in time; striding is applied only in frequency.Source: §B.2 · p. 18
- The bL-ResNet-22 baseline uses two-branch Big-Little Modules parameterized by α and β, with residual transition layers between modules.The transition layer uses a stride-2 convolution, shifting one convolution from the preceding stage’s last residual block.Source: §B.2 · p. 18
- Branches are merged by linear combination with unit weights; frequency upsampling and channel matching align their feature maps before merging.For β > 1, shallower branches are cropped in time to match the deeper Big-Branch before merging.Source: §B.2 · pp. 18–19
- The explored β = 2 and β = 3 variants reduce Little-Branch convolution counts from five in the Big-Branch to three and two, respectively, in the first three stages.The final-stage Little-Branch has two convolutions in both variants.Source: §B.2 · p. 19
- The pyramidal variant uses four, three, then two branches across its first three stages and matches the ResNet-22 fourth stage.Its first-stage Big-Branch has 256 channels; the three Little-Branches have 64, 16, and 4 channels.Source: §B.2 · p. 20