Source-linked AI summary
Multi-Fiber Networks for Video Recognition
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, Jiashi Feng
TL;DR
The paper addresses the high computational cost of 3D convolutional networks for video recognition while seeking to preserve state-of-the-art accuracy. It introduces sparse Multi-Fiber networks with multiplexers for cross-fiber information flow, and reports lower computation with state-of-the-art performance across major benchmarks.
Problem
3D spatio-temporal models provide strong video recognition but remain computationally expensive relative to 2D counterparts.
Method
The Multi-Fiber architecture slices residual units into lightweight fibers and adds multiplexers to share representations across them.
Results
The model achieves state-of-the-art accuracy on Kinetics, UCF-101, and HMDB51 with significantly lower computational cost than related 3D CNNs.
Takeaways & Limitations
Multi-Fiber designs improve the efficiency of convolutional networks for both image and video recognition tasks.
Takeaways & Limitations
On the authors’ server, MF-Net was only about 30% faster than an optimized ResNet-50 implementation because CuDNN group convolutions were not optimized.
Abstract
from arXiv · showhide
In this paper, we aim to reduce the computational cost of spatio-temporal deep neural networks, making them run as fast as their 2D counterparts while preserving state-of-the-art accuracy on video recognition benchmarks. To this end, we present the novel Multi-Fiber architecture that slices a complex neural network into an ensemble of lightweight networks or fibers that run through the network. To facilitate information flow between fibers we further incorporate multiplexer modules and end up with an architecture that reduces the computational cost of 3D networks by an order of magnitude, while increasing recognition performance at the same time. Extensive experimental results show that our multi-fiber architecture significantly boosts the efficiency of existing convolution networks for both image and video recognition tasks, achieving state-of-the-art performance on UCF-101, HMDB-51 and Kinetics datasets. Our proposed model requires over 9x and 13x less computations than the I3D and R(2+1)D models, respectively, yet providing higher accuracy.
1 Introduction
Video recognition benefits from temporal modeling, but 3D convolutions impose substantial computational costs. Multi-Fiber networks target this efficiency gap with sparse fibers and multiplexers while preserving recognition accuracy.
- Motivation: 3D convolutions learn spatial and temporal video representations but carry high computational overhead.They remain costly even though decomposed and grouped convolutions reduce some computation.
- Multi-Fiber architecture: The proposed Multi-Fiber architecture slices each unit into lightweight 3D convolutional fibers, reducing computational cost by approximately N times.Each fiber is independent, creating a sparsely connected network.
- Multi-Fiber architecture: Multiplexer modules redirect information across fibers with minimal overhead, increasing the model’s representational capacity.The modules are attached at the head of each residual block.
- Contributions: The architecture improves efficiency in both 2D image models and spatio-temporal video models.The authors report performance gains when embedding it in common compact image-recognition models and approximately an order of magnitude lower cost for 3D models.
- Results: The model outperforms recent related methods on Kinetics, UCF-101, and HMDB51 while requiring several times lower computational cost.The introduction describes this as a multi-benchmark video-recognition result.
- Computational context: I3D and R(2+1)D-34 require 108 GFLOPs and 152 GFLOPs per frame, respectively, compared with 11 GFLOPs for ResNet-152 and 15 GFLOPs for VGG-16.These figures illustrate the computational gap between representative 3D and 2D models.
2 Related Work
Video-recognition methods use either efficient 2D convolutions or more expressive 3D convolutions that learn motion directly from raw frames. Multi-Fiber extends sparse, lightweight network design to this setting while using multiplexers to restore cross-path information flow.
- 2D convolutional models: 2D video models are computationally efficient but commonly rely on pre-computed motion features to capture temporal dependencies.Two-stream models use RGB frames and optical flow, while CoViAR averages 4.2 GFLOPs per frame using compressed-frame motion information.
- 3D convolutional models: 3D convolutional networks learn spatial and temporal information end to end from raw video frames.Their 3D kernels model motion and appearance jointly, enabling more complex relations to be captured.
- Sparse network designs: Sparse connections have also been explored in low-power networks and through group convolutions or learned connectivity.The paper distinguishes its video-recognition strategies from these related approaches while noting benefits for compact models such as MobileNet-v2.
- Multi-Fiber design: The multi-fiber design differs from the conventional multi-path design by using separated lightweight residual units called fibers and a multiplexer for cross-fiber information transfer.The multiplexer uses linear projection layers for dimension reduction and expansion.
3 Multi-Fiber Networks
The Multi-Fiber architecture reduces convolutional connections by slicing residual units into lightweight fibers while using multiplexers to restore information flow. It applies to both 2D and 3D CNNs, improving image-recognition efficiency and extending the design to video models.
- Motivation: 3D spatio-temporal convolutions improve video representations but impose substantially higher computational costs than 2D convolutions.Their kernels and input-output tensors are multiple times larger.
- Multi-Fiber Unit: The multi-fiber unit slices a residual unit into N parallel, separated fibers, preserving overall width while reducing connections by a factor of N.Each fiber is an isolated lightweight convolutional path.
- Multi-Fiber Unit: The multiplexer gathers features from all fibers and redirects them to specific fibers, addressing the information-flow limitation caused by isolated paths.It uses two 1 × 1 convolution layers to reduce routing overhead.
- ImageNet-1k Results: 2.9%/1.9% Top-1/Top-5 accuracy gains accompany smaller size and lower FLOPs when multi-fiber units replace ResNet-18 residual units.The MF model uses 9.6M vs. 11.7M parameters and 1.6G vs. 1.8G FLOPs.
- ImageNet-1k Results: 0.8%/0.3% Top-1/Top-5 accuracy gains also occur for MobileNet-v2, with 6.0M vs. 6.9M parameters and 578M vs. 585M FLOPs.The training curves additionally report consistently higher training and validation accuracy for the MF model.
- Ablation Studies: Increasing the fiber count improves performance, whereas removing the multiplexer causes a significant drop, demonstrating the importance of cross-fiber information sharing.The 2D multi-fiber network can perform as well as the larger ResNet-50.
- 3D Extension: The 3D extension follows a ResNet-34 design with adjusted channels for video processing and uses 3D convolutions to preserve temporal information.Figure 3 presents the overall network and internal multi-fiber-unit structure.
4 Experiments
Experiments evaluate MF-Net on Kinetics, UCF-101, and HMDB51, measuring recognition accuracy alongside computational efficiency. Across these benchmarks, the model maintains or improves accuracy while substantially reducing computation, with category-level analysis identifying short actions and weak visual cues as difficult cases.
- Kinetics: 11.1 GFLOPs enabled MF-Net to match or exceed existing 3D CNN accuracy while using substantially less computation and the smallest model size on Kinetics.Compared with R(2 + 1)D-RGB, MF-Net was over 13× faster, used 8× fewer parameters, and achieved 0.8% higher Top-1 accuracy.
- Discussion: Visualization of learned filters showed that inflated 2D filters developed diverse temporal patterns after Kinetics training.The filters began with identical temporal sub-kernels but evolved along the temporal dimension during learning.
- Transfer evaluation: 96.0% Top-1 accuracy on UCF-101 was achieved with 11.1 GFLOPs, versus 152.4 GFLOPs for the compared state-of-the-art methods.Against Res3D, MF-Net improved Top-1 accuracy from 85.8% to 96.0% while reducing computational cost by 42%.
- Transfer evaluation: 74.6% versus 59.1% accuracy gave MF-Net more than a 15% improvement over RGB-only 2D CNN models on the evaluated transfer task.The comparison also reports an improvement of over 5% against methods using additional optical information, while Figure 6 shows an order-of-magnitude efficiency advantage.
- Discussion: 190 of 400 Kinetics categories exceeded 80% accuracy, while 17 remained below 30%, revealing uneven recognition performance across actions.High-accuracy categories tend to include distinctive objects, backgrounds, or long-duration actions; low-accuracy categories often lack distinguishing objects and contain brief target actions.
5 Conclusion
The proposed multi-fiber architecture reduces computational resources for 3D video networks while retaining state-of-the-art accuracy, and also benefits image-classification networks.
- The multi-fiber architecture introduces sparse connections inside residual blocks and multiplexers to compensate for information loss.These designs reduce model redundancy and computational cost.
- The model achieves state-of-the-art video-recognition accuracy on Kinetics, UCF-101, and HMDB51 with significantly lower resource usage than existing 3D CNNs.
- The architecture is generic and can also improve existing networks for image classification.