PaperlayerView exact revision on arXiv

Source-linked AI summary

Gate-Shift Networks for Video Action Recognition

Swathikiran Sudhakaran, Sergio Escalera, Oswald Lanz

arXiv:1912.00381v2cs.CV

TL;DR

Video action recognition needs temporal reasoning, while deep 3D CNNs can be costly and difficult to train without sufficiently large datasets. The paper introduces GSM, which adds spatial gating and temporal feature routing to 2D CNNs with minimal overhead. GSM achieves state-of-the-art results on Something Something-V1 and Diving48, competitive results on EPIC-Kitchens, and a +32% accuracy gain over TSN with 0.48% more parameters and 0.55% more FLOPs.

  • Problem

    Deep 3D CNNs provide joint spatiotemporal representations but may under-perform when their large parameter and computation requirements exceed available training data.

  • Method

    GSM adds learnable spatial gating to a 2D CNN, routing gated features through temporal shifts or convolution while adding residual features through a skip connection.

  • Results

    GSM achieves state-of-the-art results on Something Something-V1 and Diving48, competitive results on EPIC-Kitchens, and +32% accuracy on Something Something-V1 with TSN using 0.48% additional parameters and 0.55% additional FLOPs.

  • Takeaways & Limitations

    GSM provides an efficient spatio-temporal feature extractor that can be plugged into 2D CNNs with far less model complexity than competing approaches.

Abstract

from arXiv · show

Deep 3D CNNs for video action recognition are designed to learn powerful representations in the joint spatio-temporal feature space. In practice however, because of the large number of parameters and computations involved, they may under-perform in the lack of sufficiently large datasets for training them at scale. In this paper we introduce spatial gating in spatial-temporal decomposition of 3D kernels. We implement this concept with Gate-Shift Module (GSM). GSM is lightweight and turns a 2D-CNN into a highly efficient spatio-temporal feature extractor. With GSM plugged in, a 2D-CNN learns to adaptively route features through time and combine them, at almost no additional parameters and computational overhead. We perform an extensive evaluation of the proposed module to study its effectiveness in video action recognition, achieving state-of-the-art results on Something Something-V1 and Diving48 datasets, and obtaining competitive results on EPIC-Kitchens with far less model complexity.

1. Introduction

Video action recognition requires temporal reasoning, but deep 3D CNNs can be difficult to train effectively because of their parameter and computation costs. GSM adds learnable spatial gating to a lightweight 2D-CNN-based spatio-temporal extractor, with strong benchmark results and minimal overhead.

  • Video action recognition remains behind image recognition, despite applications in surveillance, media analysis, and robotics.
  • Deep 3D CNNs model joint spatiotemporal features but may under-perform when datasets are insufficiently large for scale training.Their 3D kernels increase parameters and computation because they operate across densely sampled spatial and temporal dimensions.
  • Existing decomposition methods reduce 3D-kernel complexity by separating spatial, temporal, or channel interactions, but their interaction patterns remain structurally constrained.The cited approaches include channel-wise, spatial-then-temporal, and grouped spatial-temporal decompositions.
  • GSM introduces spatial gating that splits 2D-CNN outputs into gated and residual features, applies temporal processing to the gated branch, and skip-connects the residual.It uses group spatio-temporal convolution for gating and hard-coded time shifts instead of learnable temporal convolution.
  • +32 percentage points in accuracy are obtained on Something Something-V1 when GSM is plugged into TSN, with 0.48% additional parameters and 0.55% additional FLOPs.
  • The paper reports a lightweight plug-in module, extensive ablations, and state-of-the-art or competitive benchmark results with fewer parameters and FLOPs.

2. Related Work

Prior video-recognition work ranges from two-stream and frame-aggregation methods to short-term temporal modeling and full 3D convolutions. GSM addresses the hard-wired interactions in these efficient decompositions by adaptively routing features through time.

  • Fusing appearance and flow: Two-Stream architectures combine an RGB appearance stream with an optical-flow motion stream, but rely on externally computed flow and limited RGB-frame context.
  • Video as a set or sequence of frames: Frame-based methods apply sampled RGB frames to 2D CNNs and integrate frame-level features through pooling, multilayer perceptrons, recurrent aggregation, or attention.
  • Modeling short-term temporal dependencies: Short-term temporal methods model dependencies through feature differencing, filtering, or differentiable optical-flow extraction.
  • Video as a space-time volume: Full 3D CNNs learn directly in space-time but require many parameters, substantial computation, and large-scale datasets for pre-training.
  • Spatial-temporal modeling: Efficient decompositions such as 2D-plus-1D convolution and channel shifting reduce complexity, yet previous approaches hard-wire spatial, temporal, and channel-wise interactions.
  • Spatial-temporal modeling: GSM learns to adaptively route and combine features through time while controlling interactions in spatial-temporal decomposition.

3. Gate-Shift Networks

Gate-Shift Networks turn a 2D CNN into a spatio-temporal feature extractor by learnably routing selected spatial features through time while retaining residual features. The module uses lightweight gating, temporal shifts, and residual fusion within existing CNN architectures.

  • Gate-Shift Module: GSM replaces hard-wired channel splitting with learnable spatial gating that selectively routes features through temporal shifts and merges them with a spatial residual.This design is inspired by GST and TSM while adaptively combining spatial and temporal information.
  • Gate-Shift Module: GSM first applies spatial convolution, splits channels into two groups, gates each group, shifts gated features forward and backward in time, and fuses them with the residual.The fused output is propagated to the next layer.
  • Gate-Shift Module: A single 3D convolution with tanh activation computes the gating, using short-range spatio-temporal information and gating values in (−1, +1).The paper reports that tanh performs better than sigmoid in ablation experiments.
  • Gate-Shift Module: The GSM implementation adds 27·C parameters, far fewer than a typical C2D block, while using 2D kernels and parameter-free temporal shifts.Its output equations combine shifted gated features with residual terms in a residual-learning formulation.
  • Gate-Shift Architecture: GSM is inserted into selected BN-Inception or InceptionV3 branches, with the least-convolution branch performing best empirically; clip scores are averaged across frame-level predictions.Adding GSM to all branches reduced performance in the reported experiments.

4. Experiments and Results

Experiments evaluate GSM across three action-recognition benchmarks, ablations, feature visualizations, and accuracy-complexity trade-offs. GSM improves recognition substantially while adding little parameter or computational overhead, and supports temporal reasoning beyond orderless frame pooling.

  • Benchmark evaluation: GSM is evaluated on Something-V1, Diving48, and EPIC-Kitchens, three benchmarks requiring spatio-temporal reasoning.Something-V1 has 100K videos and 174 fine-grained actions; Diving48 has around 18K videos and 48 dive classes; EPIC-Kitchens contains 34K egocentric videos.
  • Ablation analysis: The best ablation places GSM in the Inception branch with the fewest convolution layers; inserting it in every branch reduces performance.The authors retain this branch-selection rule for InceptionV3 because applying GSM broadly can adversely affect spatial modeling capacity.
  • Ablation analysis: 47.24% accuracy is achieved with GSM in all Inception blocks, a +30% absolute gain over TSN with only 0.48% parameter and 0.55% complexity overhead.Increasing the number of GSMs consistently improves recognition in the ablation.
  • Ablation analysis: Tanh gating outperforms sigmoid gating by 3% absolute, reaching 47.24% versus 44.75%.The comparison supports tanh for calibrating the gate.
  • State-of-the-art comparison: +32% absolute accuracy over TSN raises Something-V1 performance from 17.52% to 49.56%, while GSM uses considerably fewer FLOPs than heavier alternatives.An ensemble reaches 55.16% recognition accuracy, and GSM remains competitive with top-performing methods at lower computational cost.
  • State-of-the-art comparison: 40.27% accuracy on Diving48 improves the previous state of the art by +1.3%, where actions require strong spatio-temporal reasoning.The model uses 16 frames for training, two clips during inference, and InceptionV3 as backbone.
  • Temporal reasoning: Reversing frames reduces GSM performance from 47.24% to 15.38%, whereas TSN performance does not change.This experiment tests whether GSM encodes temporal order rather than treating frames as an orderless set.
  • Feature analysis: GSM features show lower intra-class and higher inter-class variability than TSN features in t-SNE visualizations.The analysis samples 1,800 validation videos and includes the ten action groups described in the figure.

5. Conclusion

GSM turns a 2D-CNN into an efficient spatio-temporal feature extractor and performs strongly across three video action-recognition benchmarks. Its TSN integration illustrates substantial accuracy gains with minimal added complexity.

  • GSM turns a 2D-CNN into a highly efficient spatio-temporal feature extractor.
  • GSM achieves state-of-the-art results on Something Something-V1 and Diving48, with competitive results on EPIC-Kitchens.
  • +32% absolute recognition accuracy is obtained on Something Something-V1 when GSM is plugged into TSN.The integration adds 0.48% parameters and 0.55% FLOPs.

Appendix

The appendix provides supplementary analyses, visualizations, and implementation resources for the GSM models. It also points readers to released code and models.

  • The appendix includes additional t-SNE plots for the GSM models.
  • It reports Something Something-V1 results using an ensemble of models.
  • The appendix provides saliency-tube visualization samples, a supplementary video, and links to code and models.

A. Architecture Details

This appendix section provides architectural details for the CNN backbones used in GSM models.

  • The section documents the CNN architectures used in GSM models.
  • The documented architectures serve as backbones for GSM models.
  • These architecture details support interpretation of the GSM model implementations.

A.1. BN-Inception

The appendix describes GSM BN-Inception and GSM InceptionV3 architectures, including layer output sizes and an input-size change for InceptionV3. The BN-Inception table specifies convolutional-layer conventions and the class-count symbol.

  • BN-Inception: Tab. 6 presents the GSM BN-Inception architecture and output size after each layer.
  • BN-Inception: Table 6 specifies that convolution layers are followed by batch normalization and ReLU, with C denoting the number of classes.
  • InceptionV3: GSM InceptionV3 uses 229 × 229 inputs instead of 299 × 299 to reduce computational complexity without affecting performance.The GSM InceptionV3 architecture and output sizes are listed in Tab. 7.

B. t-SNE

The t-SNE analysis examines how GSM changes feature organization across models and backbone layers. Adding GSM improves class separation, with semantic separation increasing toward higher layers.

  • Feature organization: Adding GSM reduces intra-class variability and increases inter-class variability in the learned features.
  • Layerwise separation: With 10 GSMs in BN-Inception, semantic separation increases toward the top layers of the backbone.The comparison uses t-SNE plots from the last four Inception blocks.

C. Ensemble Results

The ensemble combines GSM InceptionV3 models trained with different input-frame counts to encode videos at multiple temporal resolutions. An ensemble using 8, 12, 16, and 24 frames reaches 55.16% recognition accuracy.

  • Ensemble results: 55.16% recognition accuracy is achieved by ensembling GSM models trained on 8, 12, 16, and 24 frames.The ensemble combines prediction scores from models trained with different numbers of input frames.
  • Ensemble results: Adding more models increases ensemble accuracy on Something Something-V1.The table reports individual-model results with 1 and 2 clips, followed by results for different ensemble configurations.
  • Architecture: The GSM InceptionV3 architecture is illustrated through its placement within Inception blocks.

D. Visualization

The visualization compares TSN and GSM saliency tubes on Something Something-V1 validation videos. GSM focuses more on active objects at the times actions occur, whereas TSN can remain focused on scene objects across frames.

  • Visualization setup: Saliency tubes compare TSN on the left with GSM on the right using sample Something Something-V1 validation videos.The visualizations use BN-Inception models trained with 16 frames, with action labels shown by column.
  • Saliency comparison: GSM enables temporal reasoning by focusing on active objects where and when an action takes place.The compared classes require strong temporal reasoning for understanding the action.
  • Example comparison: For putting something in front of something, TSN tracks the pen and cup across early and later frames, while GSM uses frames where the cup is introduced.
  • Example comparison: For taking one of many similar things on the table, TSN focuses on the matchbox throughout, while GSM relies on frames where the action occurs.
Loading 1912.00381v2…