PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Study of Deep Video Action Recognition

Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, Mu Li

arXiv:2012.06567v1cs.CVcs.MM

TL;DR

Video action recognition faces challenges in long-range temporal modeling, computational cost, and incomparable evaluations caused by dataset and protocol variation. This survey synthesizes over 200 deep-learning papers, reviews datasets and model evolution, benchmarks methods, and releases reproducible code. Its comparisons highlight efficiency advances and persistent limitations in training cost, deployment, and dataset fairness while identifying open research opportunities.

  • Problem

    Video action recognition lacks a comprehensive deep-learning survey, while long-range temporal modeling, computational cost, and inconsistent datasets and evaluations remain challenges.

  • Method

    The paper reviews over 200 approaches chronologically, introduces influential datasets, benchmarks methods for accuracy and efficiency, and releases implementations for reproducibility.

  • Results

    The survey reports that efficient student networks reduce inference time by 30% and FLOPs by approximately 90% with negligible performance drop, while 3D models often achieve similar accuracy around 75%.

  • Takeaways & Limitations

    The synthesis identifies dataset bias, deployment constraints, and open problems including zero-shot and fine-grained action recognition as directions for future research.

  • Takeaways & Limitations

    Training a standard SlowFast network on Kinetics400 can require 10 days on a high-end 8-GPU machine, limiting accessibility.

Abstract

from arXiv · show

Video action recognition is one of the representative tasks for video understanding. Over the last decade, we have witnessed great advancements in video action recognition thanks to the emergence of deep learning. But we also encountered new challenges, including modeling long-range temporal information in videos, high computation costs, and incomparable results due to datasets and evaluation protocol variances. In this paper, we provide a comprehensive survey of over 200 existing papers on deep learning for video action recognition. We first introduce the 17 video action recognition datasets that influenced the design of models. Then we present video action recognition models in chronological order: starting with early attempts at adapting deep learning, then to the two-stream networks, followed by the adoption of 3D convolutional kernels, and finally to the recent compute-efficient models. In addition, we benchmark popular methods on several representative datasets and release code for reproducibility. In the end, we discuss open problems and shed light on opportunities for video action recognition to facilitate new research ideas.

1. Introduction

Video action recognition is a central video-understanding task with broad applications, rapid dataset and model growth, and persistent gaps in comprehensive synthesis and reproducible comparison.

  • Video action recognition supports applications including behavior analysis, video retrieval, human-robot interaction, gaming, and entertainment.
  • Dataset scale expanded from 7K videos across 51 classes in HMDB51 to 8M videos across 3,862 classes in YouTube8M.
  • Dataset releases increased from 3 during 2011–2015 to 13 during 2016–2020.
  • Deep video models progressed from early CNN adaptations through two-stream approaches and later representative architectures summarized chronologically.
  • The survey addresses the absence of a comprehensive deep-learning-focused review by covering over 200 papers, benchmarking methods for accuracy and efficiency, and releasing implementations.
  • It also discusses challenges, open problems, and opportunities for future video action recognition research.

2. Datasets and Challenges

Video action recognition datasets vary in scale, labeling, and visual bias, while modeling must handle temporal information, action variation, and fair evaluation constraints.

  • Large-scale annotated datasets are needed because deep learning methods generally improve in accuracy as training-data volume grows.
  • Dataset construction typically defines action labels, collects videos, manually annotates temporal boundaries, and removes noisy classes or samples.
  • Kinetics400 contains approximately 240K training and 20K validation videos across 400 categories, while Kinetics600 and Kinetics700 expanded to 480K and 650K videos.
  • Datasets differ in task structure, including Charades for concurrent multi-label activities, HVU for multi-label multi-task understanding, and Something-Something for object-interaction actions.
  • Scene-focused datasets can permit recognition from static context, whereas Something-Something requires temporal reasoning and Moment in Time includes actor and environment variation.
  • Benchmark comparisons are constrained by ambiguous annotation, inconsistent downloaded data, and the resulting impossibility of fair method comparisons.
  • Models must address strong intra- and inter-class variation while jointly capturing short-term motion and long-range temporal information.

3. An Odyssey of Using Deep Learning for Video Action Recognition

The survey reviews deep-learning methods for video action recognition from 2014 onward and places earlier related work in context.

  • The review covers deep-learning methods for video action recognition from 2014 to the present and introduces earlier work for context.

3.1. From hand-crafted features to CNNs

Before deep learning, hand-crafted features dominated video understanding because of their accuracy and robustness; CNN-based methods then adapted image models to video frames and temporal connectivity.

  • Hand-crafted features, particularly Improved Dense Trajectories, dominated before 2015 because of high accuracy and robustness.
  • Their heavy computational cost and limited scalability made hand-crafted features difficult to deploy.
  • DeepVideo applied a single 2D CNN independently to video frames and investigated late, early, and slow temporal fusion patterns.

3.2. Two-stream networks

Two-stream networks augment RGB appearance modeling with optical-flow motion modeling, helping CNNs recognize temporal patterns in videos. The section traces their fusion strategies, temporal-segment extension, and recurrent variants.

  • Two-stream motivation: Optical flow provides motion information complementary to RGB appearance, especially when backgrounds are cluttered.Its colors encode motion directions, and the representation describes action-specific movement patterns.
  • Two-stream architecture: Two-stream networks use a spatial stream for RGB frames and a temporal stream for stacked optical-flow images.The two streams model visual appearance and motion between video frames before their predictions are combined.
  • Two-stream results: 88.0% vs 87.9% on UCF101 and 59.4% vs 61.1% on HMDB51 marked CNN-based performance comparable to hand-crafted IDT features.The results supported the importance of motion information and the difficulty of learning temporal information directly from raw frames.
  • Spatial-temporal fusion: Late fusion averages stream predictions, whereas early fusion enables earlier interactions between appearance and motion pathways during learning.Fusion studies vary the operator, fusion layer, and use of 2D or 3D convolution for temporal fusion.
  • Recurrent neural networks: LSTM-based two-stream models aggregate frame-level CNN features into video-level predictions, but their empirical improvement over the two-stream baseline was unclear.A later study found that LSTMs require proper care, including pre-segmented data, to exploit temporal information effectively.
  • Long-range temporal modeling: Temporal Segment Networks address two-stream networks’ inability to capture long-range information by sampling frames from multiple video segments and aggregating them.Shared network weights process the sampled frames, followed by segmental consensus operations such as average or max pooling.

3.3. The rise of 3D CNNs

3D CNNs model video as a spatiotemporal tensor, avoiding reliance on separately precomputed optical flow but introducing optimization and computational challenges. The section follows their development from C3D and I3D to factorized, non-local, and efficient architectures.

  • 3D convolutional modeling: 3D CNNs treat video as a tensor with two spatial dimensions and one temporal dimension, using 3D kernels to model temporal information.This approach responds to the computational and storage burden of precomputing optical flow.
  • Early 3D CNNs: C3D extended early 3D networks with a deeper VGG16-like design, but its benchmark performance was unsatisfactory despite strong generalization as a feature extractor.Its training nevertheless demonstrated the potential and practical difficulty of deep 3D architectures.
  • Optimization challenges: 3D networks are hard to optimize, and C3D training took weeks to converge even with the large-scale Sports1M dataset.Many users therefore adopted C3D as a feature extractor rather than modifying or fine-tuning it.
  • I3D: I3D inflated ImageNet-pretrained 2D weights into 3D models and, with Kinetics400 pretraining, achieved 95.6% on UCF101 and 74.8% on HMDB51.Its later two-stream version reached 98.0% on UCF101 and 80.9% on HMDB51, combining 3D CNNs with the two-stream framework.
  • 3D factorization and hybrid designs: P3D and R2+1D factorize 3D kernels into separate spatial and temporal convolutions to reduce training complexity.Other simplifications mix 2D and 3D convolutions or use top-heavy designs, as in MiCTNet, ARTNet, S3D, and ECO.
  • Long-range temporal modeling: Long-range temporal modeling remains difficult because later deep-network stages may lose information from temporally distant frames.LTC uses longer temporal convolutions, T3D preserves temporal information through dense connections, and Non-local adds a dedicated building block.
  • Enhancing 3D efficiency: CSN factorizes channel interactions from spatiotemporal interactions and achieves state-of-the-art performance while running 2 to 3 times faster than previous best approaches.This efficiency direction targets GFLOPs, parameter count, and latency in 3D CNNs.
  • Efficient architectures: SlowFast uses a low-frame-rate pathway for detailed semantics and a high-temporal-resolution pathway for rapidly changing motion, linked through lateral connections.X3D further expands 2D architectures along multiple axes to provide efficient models at different complexity levels.

3.4. Efficient Video Modeling

Efficient video modeling addresses the computational and deployment costs of optical flow and 3D convolution by exploring alternatives that preserve temporal reasoning while reducing overhead.

  • Efficiency constraints: 3D CNNs remain difficult to train and deploy: a standard SlowFast model on Kinetics400 takes 10 days on a high-end 8-GPU machine, while platform support is weaker than for 2D convolution.3D CNNs also require more input frames, increasing I/O costs.
  • Efficiency constraints: Since 2018, researchers have investigated alternatives that improve accuracy and efficiency simultaneously for video action recognition.The survey organizes these alternatives into categories of efficient video modeling methods.
  • Motion without optical flow: Optical flow is computationally expensive, storage demanding, and prevents end-to-end training, motivating motion encoding without optical flow during inference.Two-stream methods require pre-computed optical flow; Kinetics400 optical-flow storage is given as 4.5TB, and pre-computation prevents real-time operation.
  • Motion without optical flow: Motion-estimation networks can remove optical-flow computation during inference, although early approaches still require optical flow during training.Hidden two-stream networks instead use a lightweight unsupervised MotionNet to learn motion information.
  • Temporal modeling without 3D convolution: Temporal modeling without 3D convolution includes channel shifting and attention-based modules that exchange information across neighboring frames or encode motion features.TSM inserts temporal shifts into residual branches of 2D CNNs, while STM and TEA use spatiotemporal and motion modules.

3.5. Miscellaneous

The survey reviews diverse directions beyond standard CNN, two-stream, and 3D-CNN approaches, including trajectory and rank-based representations, compressed video, selective sampling, and visual-tempo modeling.

  • Trajectory-based methods: Trajectory-constrained pooling combines deep convolutional features with trajectory information to produce motion-aware descriptors that performed strongly on UCF101 and HMDB51.A trajectory tracks pixels through time, combining merits of hand-crafted and deep-learned features.
  • Rank pooling: Rank pooling represents temporal evolution by learning a ranking function that orders video frames, with later work making the operation differentiable for end-to-end learning.Dynamic images apply rank pooling to raw pixels to produce one RGB image per video.
  • Compressed video: Compressed-video methods exploit I-frames, motion vectors, and residuals to reduce redundant raw-video processing, but current approaches cannot handle B-frames because of added complexity.Motion vectors may be inaccurate, motivating knowledge distillation or adversarial refinement in different methods.
  • Frame and clip sampling: Uniformly processing every frame can hurt performance and waste inference computation because discriminative actions often occur only during a few moments.Key-volume and adaptive frame-sampling methods select informative content, while teacher-student designs target efficient inference.
  • Frame and clip sampling: A see-very-little student reduced inference time by 30% and FLOPs by approximately 90% with negligible performance drop, while SCSampler achieved state-of-the-art results on Kinetics400 and Sports1M.SCSampler uses compressed representations to sample salient clips and reported higher accuracy than using all video frames.
  • Visual tempo: Visual tempo helps distinguish actions with similar appearance, such as walking, jogging, and running, and is also used as supervision in self-supervised video learning.Methods explore multiple temporal rates to improve temporal modeling.

4. Evaluation and Benchmarking

The survey standardizes benchmark comparisons across dataset types and evaluates methods by accuracy and efficiency, showing that model strengths depend on temporal demands, data scale, and deployment costs.

  • Evaluation protocol: The evaluation covers scene-focused, motion-focused, and multi-label benchmarks, using accuracy for single-label recognition and mAP for multi-label recognition.The survey also compares parameter count, FLOPs, latency, and frames per second.
  • Evaluation protocol: Standard inference uses 25 frames with ten-crop augmentation for 2D CNNs and 10 clips with three-crop augmentation for 3D CNNs, though protocols evolve by architecture and dataset.The 2D scheme uses 250 frames per video, while the 3D scheme is called the 30-view strategy.
  • Scene-focused datasets: Scene-focused datasets contain short videos often recognizable from static appearance, and early results show that adding motion or temporal modeling improves over frame-only DeepVideo.Trajectory pooling and global feature encoding transfer useful ideas from traditional recognition pipelines into deep models.
  • Scene-focused datasets: Factorized 3D convolutions and I3D weight inflation achieve performance comparable to the best two-stream methods without optical flow, while later 3D models can improve further with more data, pathways, or architecture search.C3D underperformed concurrent approaches, possibly because optimizing 3D kernels was difficult.
  • Scene-focused datasets: Efficient models exceed two-stream accuracy and approach 3D-CNN performance while using 2D CNNs without optical flow; most are real-time, and some support online recognition.The survey identifies 2D CNNs with temporal modeling as a promising efficiency-oriented direction.
  • Motion-focused datasets: Longer inputs generally help Something-Something, and explicit temporal modules such as TSM, TEA, and MSNet outperform stacked 3D kernels on this motion-focused benchmark.These results indicate a need for strong temporal motion reasoning alongside spatial semantics.
  • Multi-label datasets: On Charades, 3D models generally outperform 2D alternatives, longer inputs help, and stronger pre-trained backbones perform better because the medium-sized dataset lacks diversity.Assembled models and added human-object interaction information substantially outperform single-model fine-tuning.
  • Speed comparison: 2D models have much lower latency than 3D variants, while most 3D models achieve similar accuracy around 75%; larger-scale pre-training can significantly increase performance.FLOPs are not strongly correlated with actual inference latency, so deployment comparisons require more than FLOPs alone.

5. Discussion and Future Work

The survey identifies open directions spanning evaluation, data augmentation, domain adaptation, neural architecture search, and deployment efficiency. It emphasizes that accuracy gains coexist with plateauing benchmark performance and substantial real-world deployment constraints.

  • Benchmark performance has plateaued, but video action recognition still has many active and promising research directions.
  • Current datasets can be biased toward static representations, so experiments may produce erroneous conclusions and limit progress.
  • Video action recognition uses relatively basic augmentation, motivating investigation of techniques such as color jittering and random rotation.
  • Generalization across datasets or domains remains less explored despite improving accuracy on standard datasets, while existing video domain-adaptation studies often use small-scale settings.
  • Neural architecture search is promising but resource-intensive; TVN jointly optimizes parameters and runtime and reaches competitive accuracy with faster execution.TVN runs within 37 to 100 ms on a CPU and 10 ms on a GPU per 1-second video clip.
  • Deployment is constrained by offline inputs, unmet real-time requirements, and incompatibility of 3D convolutions or other non-standard operators on edge devices.Efficient 2D architectures, compression, quantization, pruning, and mobile networks are identified as promising directions, while online-stream benchmarking remains underdeveloped.

5.6. New datasets

The survey highlights dataset needs around temporal modeling, robustness, security, zero-shot recognition, weak supervision, and fine-grained action understanding. These directions address biases, annotation costs, unseen categories, and contextual overfitting.

  • Most datasets favor spatial representations, motivating datasets that require long-term temporal modeling rather than single-frame recognition.
  • Video models remain vulnerable to adversarial attacks, and investigating both attacks and defenses is important for securing video-service APIs.
  • Zero-shot action recognition transfers learned knowledge to previously unseen categories, addressing expensive annotation and the large space of possible human actions.
  • Standard zero-shot approaches map visual embeddings into semantic space and classify new classes by nearest-neighbor matching, while later work explores universal and end-to-end representations.
  • Weakly-supervised recognition seeks to reduce dataset curation costs by using webly gathered or partially annotated data and by eliminating manual trimming.
  • Fine-grained datasets test whether models capture action-specific information rather than overfit to contextual cues in scenes.

5.11. Egocentric action recognition

The survey connects egocentric recognition and multimodal or self-supervised learning to richer video representations. It reports that recent self-supervised methods approach supervised pretraining in downstream accuracy.

  • 5.11. Egocentric action recognition: Egocentric recognition requires fine-grained modeling of hand motion and interacting objects in complex environments.
  • 5.11. Egocentric action recognition: Multimodal video understanding combines scene, object, motion, audio, or text information either to enrich representations or to supervise pretraining.
  • 5.11. Egocentric action recognition: Joint video-text embedding spaces can support zero-shot action recognition by combining textual and visual data.
  • 5.11. Egocentric action recognition: Audio and text provide supervisory signals for self-supervised video representation learning, with datasets such as HowTo100M pairing 136M clips from 1.22M videos with text.
  • 5.11. Egocentric action recognition: Video self-supervision exploits temporal structure through tasks including frame ordering, future-frame prediction, motion forecasting, speed prediction, and spatial-temporal contrast.
  • 5.11. Egocentric action recognition: Recent self-supervised methods achieve much better linear-evaluation accuracy and fine-tuning accuracy comparable to supervised pretraining, making better video representations a promising direction.

6. Conclusion

The survey reviews more than 200 deep-learning approaches to video action recognition. It aims to provide an accessible tutorial and discussion of research directions.

  • The survey presents a comprehensive review of 200+ recent deep-learning approaches to video action recognition.
  • It is intended as an easy-to-follow tutorial for newcomers and an inspiring discussion for researchers seeking new directions.
Loading 2012.06567v1…