PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Classification with Channel-Separated Convolutional Networks

Du Tran, Heng Wang, Lorenzo Torresani, Matt Feiszli

arXiv:1904.02811v4cs.CVcs.AI

TL;DR

Video classification relies on computationally expensive 3D convolutions, motivating study of efficient group-convolution designs and the role of channel interactions. The paper introduces CSNs that separate channel and spatiotemporal interactions, finding improved accuracy, lower cost, and competitive or state-of-the-art results across major benchmarks.

  • Problem

    3D video-classification convolutions are costly, and the factors governing accuracy and computation trade-offs in 3D group-convolution networks are not well established.

  • Method

    CSNs factorize 3D convolutions into 1×1×1 convolutions for channel interaction and depthwise 3×3×3 convolutions for spatiotemporal interaction.

  • Results

    CSNs achieve higher accuracy with about 2.5-3× FLOP savings when sufficient channel interaction is retained, and outperform or match state-of-the-art methods across Sports1M, Kinetics, and Something-Something.

  • Takeaways & Limitations

    Channel separation can reduce computation while preserving or improving accuracy, and its factorization acts as a regularizer that improves generalization.

Abstract

from arXiv · show

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M, Kinetics, and Something-Something, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.

1. Introduction

The paper examines whether 3D group convolution can reduce video-classification cost while preserving accuracy, focusing on which kernel factors and channel interactions matter. It proposes channel-separated 3D convolutions as a promising factorization for this trade-off.

  • Motivation: 3D video convolutions are expensive because their cost scales with the number of frames as O(CTHW), versus O(CHW) for 2D convolutions.T denotes frames, H and W spatial dimensions, and C channels.
  • Research Questions: The paper asks which parameters in large 4D video-convolution kernels matter most.
  • Prior Factorizations: Prior video architectures factor 3D convolutions into spatial 2D and temporal 1D convolutions to increase accuracy and/or reduce computation.Examples include P3D, R(2+1)D, and S3D.
  • Proposed Direction: CSNs separate channel interactions from spatiotemporal interactions, retaining high channel interaction while reducing FLOPs and parameters.The paper studies interaction-reduced and interaction-preserved factorizations.

2. Related Work

The paper situates CSNs among group- and depthwise-convolution architectures developed for efficient image and video models. Its contribution is a systematic study of channel interactions alongside the proposed channel-separated designs.

  • Efficient Convolutions: Group convolution and depthwise convolution have been used to reduce memory, model size, and computational cost in architectures such as AlexNet, MobileNet, Xception, and ResNeXt.
  • Relationship to Prior Work: CSNs extend Xception’s channel-and-space factorization from 2D object classification to 3D channel-and-space-time factorization for action recognition.CSNs use bottleneck blocks, whereas Xception uses simple blocks.
  • Contribution: The paper contributes a systematic empirical study of channel interactions in addition to designing CSN architectures.The study links channel interaction with compute reduction, accuracy, and regularization.

3. Channel-Separated Convolutional Networks

CSNs factor 3D convolutions into separate channel and spatiotemporal operations, using group structure to trade computational cost against channel interaction. The paper develops progressively grouped blocks and two bottleneck factorizations.

  • 3.1. Background: Group convolution partitions filters into groups, restricting each filter to channels within its group and reducing compute and parameters by the number of groups.Depthwise convolution is the extreme case, with one channel per group; grouping also reduces feature interactions.
  • 3.1. Background: The paper measures channel interaction by counting pairs of input channels connected through any output filter.For a grouped layer, each filter connects to Cin/G input channels; example interaction counts are 24, 4, and 0 for conventional, two-group, and depthwise convolutions.
  • 3.2. Channel Separation: CSNs decompose 3D convolution into 1×1×1 conventional convolutions for channel interaction and k×k×k depthwise convolutions for local spatiotemporal interaction.Typically k = 3, and all convolutional layers except conv1 use one of these two forms.
  • 3.3. Example: Channel-Separated Bottleneck Block: Interaction-preserved bottlenecks add a 1×1×1 convolution before depthwise 3×3×3 convolution, whereas interaction-reduced bottlenecks remove that extra channel-mixing layer.The preserved variant reduces cost while retaining channel interactions; the reduced variant loses channel interaction in the factorized 3×3×3 convolution.
  • 3.4. Channel Interactions in Convolutional Blocks: The paper constructs a spectrum of blocks by progressively applying grouping to 3×3×3 layers and then to 1×1×1 layers.Examples include simple-G, simple-D, Bottleneck-G, Bottleneck-D, and Bottleneck-DG.

4. Ablation Experiment

The ablations show that channel interactions, rather than FLOPs alone, organize accuracy in CSNs. Preserving interactions reduces compute with small shallow-network losses and deeper-network gains, while channel separation can regularize deeper models.

  • Main findings: Interaction-preserving blocks reduce compute while retaining interactions, causing only a slight accuracy loss in shallow networks and higher accuracy in deeper networks.The reported pattern spans the shallow and deep configurations examined against ResNet3D.
  • Main findings: Channel interactions predict performance within the CSN family more reliably than FLOPs, which can decrease substantially while interactions remain preserved.The study identifies interaction count as a central design factor in the ablation.
  • 4.2. Reducing FLOPs, preserving interactions: 3.6x FLOPs savings with a 2.9% accuracy drop occur for 26-layer ir-CSN, whereas 2.9x savings with a 0.7% drop occur for 26-layer ip-CSN.The shallow models have low interaction counts; ip-CSN preserves about 0.42 giga-pairs, compared with 0.27 giga-pairs for ir-CSN.
  • 4.2. Reducing FLOPs, preserving interactions: 0.9–1.4% higher accuracy than ResNet3D is reported for both ir-CSN and ip-CSN in deeper settings.The accuracy gap between the two CSNs also becomes smaller because their deeper configurations have nearly the same channel interactions.
  • 4.3. What makes CSNs outperform ResNet3D?: Higher training error but lower testing error for ip-CSN-101 suggests channel separation regularizes the model and prevents overfitting.This comparison is made against ResNet3D-101 on Kinetics.
  • 4.4. The effects of different blocks in group convolutional networks: Bottleneck-D, also called ir-CSN, provides the best computation/accuracy trade-off, while its interaction changes explain the turning points in bottleneck curves.Applying group convolution to 3×3×3 layers reduces bottleneck interactions marginally, but applying it to 1×1×1 layers causes a sharp accuracy drop.

5. Comparison with the State-of-the-Art

Across Sports1M, Kinetics, and Something-Something, CSNs match or surpass competing video architectures while substantially reducing computation.

  • Results on Sports1M: 2-4x faster: CSNs with 101 or 152 layers outperform previous Sports1M models by good margins.The ir-CSN-152 exceeds C3D, P3D, Conv Pool, and R(2+1)D on video top-1 accuracy.
  • Results on Kinetics: Multiple times faster: ir-CSN-152 is better than or comparable with previous Kinetics models under RGB-only evaluation.The Kinetics comparison is restricted to models trained on RGB input for fair evaluation.
  • Results on Kinetics: 1.2% and 1.3% higher video top-1 accuracy: ip-CSN-152 and ir-CSN-152 outperform R(2+1)D-152 while being 3.0–3.4 times faster.This comparison uses pre-training and fine-tuning on the same large-scale dataset.
  • Results on Something-Something: 53.3% top-1 accuracy: ip-CSN-152 achieves a new state-of-the-art record on Something2-v1 validation when pretrained on IG-65M.Under the same pretraining and finetuning setting, ir-CSN-152 and ip-CSN-152 exceed R(2+1)D-152 by 0.5% and 1.7%.
  • Results on Something-Something: Good margins: CSNs outperform previous Something2-v1 methods when all models use only RGB input.The comparison is reported on the Something2-v1 benchmark.

6. Conclusion

The paper presents CSNs as a factorization of 3D convolutions that reduces computation and can improve accuracy when channel interactions are sufficient. Across three major benchmarks, CSNs achieve state-of-the-art-level accuracy and run multiple times faster than competing networks.

  • CSNs factorize 3D convolutions to reduce computational cost and improve accuracy when networks have enough channel interactions.
  • ir-CSN and ip-CSN significantly outperform existing methods and achieve state-of-the-art accuracy on Sports1M, Kinetics, and Something-Something.
  • CSNs are multiple times faster than current competing networks.

Appendix: Visualization of CSN filters

The appendix visualizes filters learned by ir-CSN-152, distinguishing the initial normal 3D convolution from later depthwise convolutions across network groups.

  • ir-CSN-152 contains one initial normal 3D convolution and 50 bottleneck-D convolutional blocks arranged into four output-size groups.Figures 8–11 show filters from conv 1 and selected blocks in the first three convolutional groups.
  • Depthwise filters in later blocks learn motion and/or texture patterns within the feature channel supplied by the preceding layer.The initial conv 1 filters capture texture, motion, and color-changing patterns through cross-channel processing.
Loading 1904.02811v4…