PaperlayerView exact revision on arXiv

Source-linked AI summary

SlowFast Networks for Video Recognition

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He

arXiv:1812.03982v3cs.CV

TL;DR

Video recognition must capture slowly changing semantic content and rapidly changing motion without treating space and time symmetrically. SlowFast networks use separate low- and high-frame-rate pathways, achieving state-of-the-art accuracy for action classification and detection across major benchmarks.

  • Problem

    Video recognition lacks architectures that separately model slowly evolving spatial semantics and rapidly changing motion at their respective temporal resolutions.

  • Method

    SlowFast networks combine a low-frame-rate Slow pathway for semantics with a high-frame-rate, lightweight Fast pathway for fine-grained motion.

  • Results

    SlowFast networks achieve state-of-the-art accuracy for action classification and detection on Kinetics-400, Kinetics-600, Charades, and AVA.

  • Takeaways & Limitations

    Contrasting temporal speeds across pathways provides a supported architecture design for video action recognition.

Abstract

from arXiv · show

We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Our models achieve strong performance for both action classification and detection in video, and large improvements are pin-pointed as contributions by our SlowFast concept. We report state-of-the-art accuracy on major video recognition benchmarks, Kinetics, Charades and AVA. Code has been made available at: https://github.com/facebookresearch/SlowFast

1. Introduction

The paper introduces SlowFast, a two-pathway video-recognition architecture that separates slowly evolving spatial semantics from rapidly changing motion. It uses different temporal rates and a lightweight Fast pathway, achieving strong results across major video-recognition benchmarks.

  • Motivation and model: SlowFast separates slowly evolving spatial semantics from rapidly changing motion using low- and high-temporal-resolution pathways.The Slow pathway captures semantic information from images or sparse frames, while the Fast pathway models rapidly changing motion.
  • Architecture: The Slow pathway operates at low frame rates, whereas the Fast pathway operates at high temporal resolution and uses a fraction β of channels, such as 1/8.Lateral connections fuse the two pathways.
  • Architecture: The lightweight Fast pathway can avoid temporal pooling and preserve temporal fidelity, while the slower pathway emphasizes spatial processing and semantics.Different temporal rates allow the pathways to specialize in their respective domains.
  • Relation to prior work: Unlike Two-Stream networks, SlowFast explores different temporal speeds, uses a lightweight Fast pathway, avoids optical flow, and learns end-to-end from raw video.The cited passage contrasts SlowFast with Two-Stream’s shared backbone across streams.
  • Evaluation: SlowFast is evaluated on Kinetics-400, Kinetics-600, Charades, and AVA, with ablations demonstrating efficacy and state-of-the-art results on all datasets.The passage attributes significant gains to SlowFast relative to previous systems.

2. Related Work

Prior video-recognition methods model spatiotemporal structure through filtering, pooling, separable convolutions, optical flow, or two-stream inputs. SlowFast instead separates modeling expertise across two different temporal speeds.

  • Spatiotemporal filtering: Spatiotemporal methods capture actions with oriented filtering, 3D ConvNets, long-term filtering and pooling, or decomposed spatial and temporal convolutions.These approaches extend image models into the spatiotemporal domain or use temporal strides and separable processing.
  • Spatiotemporal filtering: SlowFast pursues a more thorough separation of modeling expertise by using two different temporal speeds.This distinguishes the approach from spatiotemporal filtering and separable versions.
  • Optical flow for video recognition: Before deep learning, hand-crafted optical-flow features—including flow histograms, motion-boundary histograms, and trajectories—achieved competitive action-recognition performance.These methods form a classical branch of spatiotemporal-feature research.
  • Optical flow for video recognition: Two-stream networks use optical flow as another input modality and underpin many competitive results, but flow is hand-designed and often not learned jointly end-to-end.The methodological concern is that the optical-flow representation is not learned end-to-end with the network.

3. SlowFast Networks

SlowFast Networks use complementary Slow and Fast pathways operating at different frame rates: the Slow pathway captures spatial semantics, while the lightweight Fast pathway preserves fine-grained temporal information. Lateral connections fuse Fast features into the Slow pathway to form the complete network.

  • Slow pathway: The Slow pathway processes one out of every τ frames, typically using τ = 16, to model video at a low frame rate.For 30-fps video, this corresponds to roughly 2 sampled frames per second.
  • Fast pathway: The Fast pathway samples αT frames at stride τ/α, typically with α = 8, giving it eight times the temporal sampling density of the Slow pathway.The frame-rate ratio explicitly creates different temporal speeds and drives complementary pathway expertise.
  • Fast pathway: The Fast pathway maintains αT-frame feature tensors without temporal downsampling until global pooling, preserving temporal fidelity throughout the hierarchy.It uses no temporal pooling or time-strided convolutions before the final global pooling layer.
  • Fast pathway: The Fast pathway uses β < 1 of the Slow pathway’s channels, typically β = 1/8, making it lightweight and computation-effective.Reduced channel capacity weakens spatial modeling while strengthening temporal modeling, a tradeoff supported by the model’s good results.
  • Fusion: Lateral connections fuse features from the Fast pathway into the Slow pathway, with one connection per stage and unidirectional Fast-to-Slow fusion.The pathways’ differing temporal dimensions require transformations before fusion.

4. Experiments: Action Classification

SlowFast networks improve action-classification accuracy across Kinetics-400 and Charades, with gains from the lightweight, high-temporal-resolution Fast pathway. These improvements are achieved at comparatively low inference cost and remain robust across pathway-fusion and channel-capacity settings.

  • Kinetics-400: 79.8% top-1 accuracy is 5.9% higher than the previous best result without ImageNet pre-training, while the best model reaches 81.8%.The best model also provides 2.1% higher top-1 accuracy than the previous state of the art.
  • Kinetics-400: 36.1 GFLOPs per spacetime view demonstrates that SlowFast can remain accurate with low inference-time cost and without many temporal clips.The inference setup uses 30 views: 10 temporal clips and 3 spatial crops per clip.
  • Kinetics-400: SlowFast consistently improves its Slow-only counterpart across variants, confirming benefits from adding the Fast pathway at comparatively low cost.The comparison varies backbones and sample rates, with model capacity measured for a single 256^2 input clip.
  • Charades: 42.1 mAP improves over the 39.0 mAP Slow-only baseline by 3.1 mAP on Charades, while Kinetics-600 pretraining reaches 45.2 mAP.Adding the extra Nonlocal component contributes an additional 0.4 mAP.
  • Kinetics-400 ablations: 3.0% better top-1 accuracy than Slow-only is achieved with the default T-conv lateral connection, which is selected as the standard fusion method.The Fast-only pathway reaches 51.7% accuracy, yet contributes up to 3.0% improvement when combined with Slow.
  • Kinetics-400 ablations: 1.6% improvement is obtained with β=1/32 while adding only 1.3 GFLOPs, showing that the Fast pathway remains effective at very small channel capacity.All tested β values from 1/32 to 1/4 improve over the Slow-only counterpart; β=1/8 is the default.

5. Experiments: AVA Action Detection

On AVA action detection, SlowFast substantially improves over Slow-only and prior systems, reaching state-of-the-art validation and challenge results through lightweight fast-pathway motion modeling and proposal-based training.

  • Dataset: AVA v2.1 contains 211k training and 57k validation video segments with per-person bounding boxes and potentially multiple actions annotated once per second.The dataset uses spatiotemporal action localization, with action detection posing the main difficulty.
  • Comparison with prior work: 26.3 mAP is achieved using only Kinetics-400 pre-training, exceeding the previous best 21.7 mAP under similar settings by 5.6 mAP.This result is also 7.3 mAP higher than the result without optical flow.
  • Proposal training: 28.2 mAP single-crop validation accuracy is obtained by training with predicted proposals having IoU > 0.9 alongside ground-truth boxes.The paper identifies this as a new state-of-the-art on AVA.
  • AVA v2.2: 29.0 mAP is achieved on AVA v2.2, rising to 29.8 mAP with the 16×8 model and 30.7 mAP with multi-scale and horizontal-flip testing.AVA v2.2 provides more consistent annotations.
  • Challenge result: 34.3 mAP is achieved by a seven-model SlowFast++ ensemble on the AVA action detection challenge 2019 test set, ranking first.The ensemble was submitted to the official ActivityNet challenge test server.
  • SlowFast ablation: SlowFast improves the Slow-only baseline by 5.2 mAP, from 19.0 to 24.2, across 57 of 60 action categories.The largest gains include hand clap (+27.7 AP), swim (+27.4 AP), and run/jog (+18.8 AP).

6. Conclusion

The paper treats time as a special dimension by contrasting processing speeds along it, achieving state-of-the-art accuracy for video action classification and detection while encouraging further research.

  • Conclusion: The architecture contrasts processing speed along the time axis, which the paper identifies as a special dimension.The design investigates differing speeds along the temporal dimension.
  • Conclusion: The approach achieves state-of-the-art accuracy for video action classification and detection.The reported performance covers both classification and detection tasks.
  • Conclusion: The authors hope the SlowFast concept will foster further research in video recognition.The conclusion positions the concept as a basis for continued investigation.

A. Appendix

The appendix describes backbone choices, initialization strategies, and dataset-specific training procedures for SlowFast models. It covers synchronized large-scale training on Kinetics and fine-tuning protocols for Charades.

  • Implementation details: The models use ResNet-50 or ResNet-101 backbones, optionally augmented with non-local blocks.R-101 models use scale jittering over [256, 340].
  • Implementation details: T×τ = 16×8 models initialize from 8×8 counterparts and train for half the epochs, while non-local models initialize from models without non-local blocks.These initialization choices are intended to reduce training time and facilitate convergence.
  • Kinetics training: Kinetics training uses synchronized SGD across 128 GPUs with 8 clips per GPU, yielding a total mini-batch size of 1024.The authors report accuracy comparable to typical training on one 8-GPU machine while scaling out better.
  • Kinetics-600 training: For Kinetics-600, training epochs and the learning-rate schedule are extended by 2×, with base learning rate η set to 0.8.This dataset-specific configuration extends the standard training schedule.
  • Charades training: Charades models fine-tune Kinetics models using per-class sigmoid outputs, 24k iterations, batch size 16, and temporal max-pooling at inference.The base learning rate is 0.0375 for Kinetics400-pre-trained models and 0.02 for Kinetics-600-pre-trained models, with 10× step-wise decay when validation error saturates.
Loading 1812.03982v3…