PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Analysis of CNN-based Spatio-temporal Representations for Action Recognition

Chun-Fu Chen, Rameswar Panda, Kandan Ramakrishnan, Rogerio Feris, John Cohn, Aude Oliva, Quanfu Fan

arXiv:2010.11757v4cs.CV

TL;DR

Action-recognition research lacks clear evidence about what improves spatiotemporal modeling and how 2D- and 3D-CNN approaches differ, amid confounded comparisons. The paper develops a shared framework for comparing these models. Its analysis finds recent gains chiefly in efficiency rather than accuracy, and similar spatiotemporal representation and transferability across the model families.

  • Problem

    The paper examines what contributes to improved spatiotemporal representations and whether 2D- and 3D-CNN models differ in temporal modeling, while noting that evaluation is confounded by dataset and experimental differences.

    Source: §1 · p. 1

  • Method

    The authors provide a common framework for comparative analysis of representative 2D- and 3D-CNN models and analyze their spatiotemporal effects.

    Source: §1 · p. 2

  • Results

    Recent progress is mostly in efficiency rather than accuracy, and 2D- and 3D-CNN models behave similarly in spatiotemporal representation and transferability.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analysis provides insights into the spatiotemporal effects of action models across backbones and architectures.

    Source: §7 · p. 8

Abstract

from arXiv · show

In recent years, a number of approaches based on 2D or 3D convolutional neural networks (CNN) have emerged for video action recognition, achieving state-of-the-art results on several large-scale benchmark datasets. In this paper, we carry out in-depth comparative analysis to better understand the differences between these approaches and the progress made by them. To this end, we develop an unified framework for both 2D-CNN and 3D-CNN action models, which enables us to remove bells and whistles and provides a common ground for fair comparison. We then conduct an effort towards a large-scale analysis involving over 300 action recognition models. Our comprehensive analysis reveals that a) a significant leap is made in efficiency for action recognition, but not in accuracy; b) 2D-CNN and 3D-CNN models behave similarly in terms of spatio-temporal representation abilities and transferability. Our codes are available at https://github.com/IBM/action-recognition-pytorch.

1. Introduction

The paper asks what drives improved spatiotemporal representations and whether 2D- and 3D-CNNs differ in temporal modeling, against a background of inconsistent comparisons. It introduces a unified framework and reports that recent progress is chiefly in efficiency, while the model families behave similarly in spatiotemporal representation and transferability.

1. Introduction

  • The paper asks what improves spatiotemporal representations and whether 2D- and 3D-CNNs differ in their temporal modeling.The authors note that 3D models report better results on Kinetics, while 2D models are superior on Something-Something.Source: §1 · p. 1
  • Comparisons are confounded by differences in datasets, backbones, model inputs, and training and evaluation protocols.The authors note that Kinetics-400 is biased toward spatial modeling and is therefore inappropriate for validating spatiotemporal modeling capability.Source: §1 · pp. 1–2
  • The paper provides a common ground for comparing 2D- and 3D-CNN models without bells and whistles.Its analysis compares representative models across three large-scale benchmark datasets.Source: §1 · p. 2
  • After removing nonstructural differences, 2D- and 3D-CNN models behave similarly in spatiotemporal representation and transferability.Temporal pooling tends to suppress temporal modeling, but provides a significant performance boost to TSN.Source: §1 · p. 2
  • Recent progress in action recognition is mostly in efficiency, not accuracy; I3D remains on par with state-of-the-art approaches.The paper also warns that input sampling strategy should be considered when comparing models.Source: §1 · p. 2

2. Related Work

Related work spans two-stream, 2D-CNN, and 3D-CNN approaches to action recognition, with methods varying in how they extract and aggregate temporal information. Prior work has also studied temporal information, but the cited analyses cover only a limited set of backbones and temporal modeling methods.

2. Related Work

  • Two-stream models process RGB frames and optical flow in separate CNNs, then fuse their outputs in upper layers.Other 2D-CNN approaches extract frame-level features and model temporal relationships or aggregate features across frames.Source: §2 · p. 2
  • 2D-CNN methods model time after extracting frame-level features, using consensus, feature relationships, channel shifts, or temporal aggregation.Examples include TSN, TRN, TSM, and TAM.Source: §2 · p. 2
  • 3D-CNNs learn spatiotemporal features jointly, and their features have also been applied to other vision tasks.High computational load has motivated approaches that decompose 3D convolution or use group convolution.Source: §2 · p. 2
  • Prior studies of temporal information have examined action complexity or motion, but cover only a limited set of backbones and temporal modeling methods.The cited work includes per-category analysis and an ablation of motion effects on C3D.Source: §2 · p. 3

3. Challenges of Evaluating Action Models

Action-model comparisons are difficult because studies use different datasets and experimental settings. The paper examines variation in backbones, input lengths, training protocols, and evaluation protocols as sources of unfair or unclear comparisons.

3. Challenges of Evaluating Action Models

  • About 60% of papers use different datasets or evaluation settings, while Kinetics-400 and Something-Something are used by around 60% and 50% of papers, respectively.The paper surveyed 37 papers from 2015 to 2020.Source: §3 · p. 3, Figure 2 · p. 3
  • About 70% of papers compare models with different backbones, which can lead to incorrect conclusions about temporal modeling.Using a stronger backbone for I3D improves its Kinetics-400 result by 4.0%.Source: §3 · p. 3
  • About 80% of papers compare models using different numbers of input frames, potentially favoring either the proposed or reference method.Methods may prefer different frame counts, complicating direct comparison.Source: §3 · p. 3
  • About 60% of papers use different training protocols, and about 60% evaluate models differently when comparing them.Differences include training duration and ImageNet initialization, as well as sampling strategies and input lengths at evaluation.Source: §3 · p. 3

4. 2D-CNN and 3D-CNN Approaches

The paper compares representative 2D- and 3D-CNN approaches under a shared framework to reduce differences unrelated to their spatiotemporal modules. The compared models vary in convolution design, inputs, temporal pooling, and temporal aggregation.

4. 2D-CNN and 3D-CNN Approaches

  • The study includes I3D, ResNet3D, S3D, R(2+1)D, TSN, and TAM as representative approaches for comparison.These methods also serve as building blocks for later approaches such as SlowFast and CSN.Source: §4 · p. 3
  • I3D and ResNet3D are pure 3D-CNNs, while S3D and R(2+1)D factorize 3D filters into spatial and temporal filters.TSN uses 2D convolution without temporal modeling; TAM adds depthwise temporal aggregation to TSN.Source: §4 · p. 3
  • The approaches differ not only in convolution kernels but also in model input, temporal pooling, and temporal aggregation.These dimensions are presented as part of the comparison of the 2D- and 3D-CNN approaches in the study.Source: §4 · p. 4, Table 1 · p. 4
  • A shared framework represents models as stacked spatiotemporal modules with optional temporal pooling, making the module their distinguishing component.The re-implementations allow configurations such as backbone, temporal pooling, and temporal aggregation to be varied.Source: §4 · p. 4

5. Datasets, Training, Evaluation Protocols

The experiments use shared data, training, and evaluation protocols to compare models across full and reduced video datasets.

  • Experiments cover Something-Something V2, Kinetics-400, and Moments-in-Time, plus mini versions with half the categories for SSV2 and Kinetics and one-eighth of MiT videos.Source: §5 · p. 4
  • Models are progressively trained with 8, 16, 32, and 64 input frames, starting from an ImageNet-pretrained model or an 8-frame starter model.Source: §5 · p. 4
  • The framework represents input frames as a 3D tensor for 2D models and a 4D tensor for 3D models.Source: Figure 3 · p. 4
  • Clip-level accuracy uses one clip, while video-level accuracy combines multiple clips and is usually higher; the paper reports clip-level accuracy by default.Source: §5 · p. 4

6. Experimental Results and Analysis

The experiments compare 2D and 3D action models across mini and full datasets, varying backbones, input sampling, temporal pooling, and temporal modules. They find that stronger backbones and evaluation choices materially affect comparisons, while spatial and temporal contributions vary by dataset.

6.1. Performance Analysis on Mini Datasets

  • Across spatiotemporal modules, ResNet50 > InceptionV1 > ResNet18, and gains from longer inputs become insignificant beyond 32 frames on all three mini-datasets.The mini-dataset analysis trains 288 model configurations across modules, backbones, pooling settings, datasets, and input lengths.Source: §6.1 · p. 4
  • Uniform sampling yields higher clip-level accuracy than dense sampling, while dense sampling gains 6%–15% on Mini-Kinetics and 5%–20% on Mini-SSV2 with multi-clip evaluation.Ten clips multiply FLOPs tenfold, so the analysis uses uniform sampling and clip-level evaluation unless stated otherwise.Source: §6.1 · p. 5, Figure 5 · p. 5
  • Temporal pooling generally hurts action-model performance, but TSN gains over 20% on Mini-SSV2 and 3%–5% on Mini-MiT.On Kinetics, a 32-frame TSN-R50 with pooling reaches 74.9% top-1 accuracy, 5.1% above its version without pooling.Source: §6.1 · p. 5, Figure 6 · p. 5

6.2. Benchmarking of SOTA Approaches

  • I3D-ResNet50 without temporal pooling is on par with state-of-the-art approaches across three datasets; removing pooling adds 1.1% on Kinetics and 6% on SSV2.A stronger backbone adds 4.0% on Kinetics; adding squeeze-excitation makes I3D surpass SlowFast by 0.8%.Source: §6.2 · p. 6, Figure 7 · p. 6, Table 2 · p. 6
  • Uniform sampling makes SlowFast about 5% less accurate than dense sampling, whereas I3D and TAM show only small differences between the sampling strategies on Kinetics.The authors conjecture this relates to SlowFast's dual-path architecture and its use of sparsely sampled frames.Source: §6.2 · p. 6, Table 3 · p. 6
  • I3D, TAM, and SlowFast differ by less than 2% on downstream transfer tasks; I3D reaches 97.12% top-1 accuracy on UCF101.The transfer study evaluates UCF101, HMDB51, Jester, and Mini-SSV2.Source: §6.2 · p. 7, Table 4 · p. 7

6.3. Analysis of Spatio-temporal Effects

  • TAM and TSM outperform I3D, 1D regular convolution, and non-local modules despite using simpler temporal aggregation strategies.The authors attribute this pattern to depthwise convolution's handling of sparsely sampled frames, while noting that non-local modules may fare poorly on Mini-SSV2 because its spatial features between frames are weak.Source: §6.3 · p. 7, Table 5 · p. 7
  • Across three representations, spatial modeling contributes more than temporal modeling on Mini-Kinetics and Mini-MiT, but only slightly more on Mini-SSV2.The analysis also reports that temporal pooling reduces temporal modeling's effectiveness across approaches, especially on Mini-Kinetics.Source: §6.3 · p. 8, Table 7 · p. 8

7. Conclusion

The conclusion highlights a comparative analysis of representative CNN action-recognition approaches across backbones and temporal aggregations. It reports insights into 2D- and 3D-CNN spatio-temporal effects and the efficacy of spatio-temporal representations.

7. Conclusion

  • The analysis compares representative CNN-based action-recognition approaches with different backbones and temporal aggregations.It examines differences and spatio-temporal effects across 2D-CNN and 3D-CNN approaches.Source: §7 · p. 8
  • The analysis provides insights into differences and spatio-temporal effects between 2D-CNN and 3D-CNN approaches.Source: §7 · p. 8
  • The work provides insights regarding the efficacy of spatio-temporal representations for action recognition.Source: §7 · p. 8

A. More Results on SOTA Models

The supplemental results show that input sampling affects comparisons among models, while runtime and memory measurements reveal differences not captured by FLOPs alone.

A. More Results on SOTA Models

  • With uniform sampling, TAM and TSM are slightly less accurate than SlowFast at the same FLOPs, while all three work well with uniform and dense sampling.Source: A. More Results on SOTA Models · p. 12
  • SlowFast achieves better speed and memory usage than the other models in the reported runtime benchmark.Throughput was measured at maximum batch size on an NVIDIA V100 32G GPU.Source: A. More Results on SOTA Models · p. 12
  • Despite having half the FLOPs of I3D-R50, TAM-R50 has similar throughput because its 3D-depthwise convolution is not yet optimized.Source: A. More Results on SOTA Models · p. 12

B. Effects of Pretrained Models

The authors compare I3D and TAM models trained with and without ImageNet weights on Kinetics. After 196 epochs, training from scratch has little effect on I3D and only a small, nonsignificant accuracy cost for TAM.

B. Effects of Pretrained Models

C. Datasets

The paper uses SSV2, Kinetics, and MiT, datasets with differing video content and temporal demands. Frames are extracted and resized before analysis.

C. Datasets

  • SSV2 contains 192K videos of 174 human-object interactions and is a setting where temporal reasoning is essential for recognition.Source: C. Datasets · p. 12
  • Kinetics has 240K training videos and 20K validation videos across 400 action categories, and TSN's strong results suggest temporal modeling is less important on this dataset.Videos last 6–10 seconds.Source: C. Datasets · p. 12
  • MiT contains one million labeled videos across 339 classes, with each clip trimmed to 3 seconds.Source: C. Datasets · p. 12
  • The datasets span different video types and are used to study varied spatio-temporal representations.Source: C. Datasets · p. 12
  • The shorter side of each extracted video frame is resized to 256 pixels while its aspect ratio is maintained.Source: C. Datasets · p. 12

D.1. Mini-Datasets

The appendix describes training and evaluation protocols for mini and full datasets, followed by the transfer-learning setup. It also identifies the comparisons shown in the mini-dataset figures and table.

D.1. Mini-Datasets

  • Table 10 covers temporal aggregation strategies without temporal pooling, and Table 11 provides an overview of the datasets.Table 12 is identified as the mini-dataset training protocol.Source: Table 10 · p. 13, Table 11 · p. 13, Table 12 · p. 13
  • Mini-Something-Something and Mini-Kinetics400 are formed by randomly sampling half of the classes; training uses multi-scale augmentation, temporal jittering, and ImageNet mean and standard deviation normalization.Most models use one node with 6 V100 GPUs; large models may use up to 18 GPUs with synchronized batch normalization.Source: §D.1 · pp. 12–13
  • Mini-dataset evaluation compares uniform and dense sampling, uses 10 clips for video-level accuracy, and includes figures varying clip counts and temporal pooling.Figure 9 uses three 256×256 spatial crops; Figure 10 compares top-1 accuracy with and without temporal pooling on three mini-datasets.Source: §D.1 · p. 13, Figure 9 · p. 13, Figure 10 · p. 14

D.2. Full Datasets

  • For Kinetics, models train on 32 sampled frames for 196 epochs and evaluate with 10 clips and three spatial crops; SSV2 evaluation uses two clips.Training uses 128 GPUs with total batch size 1024; video predictions average 30 crop-clip predictions.Source: §D.2 · p. 14

D.3. Transfer Learning

  • Transfer learning uses 32-frame models trained on Kinetics as pretrained weights, then fine-tunes for 45 epochs with cosine annealing.Fine-tuning uses batch size 48 and synchronized batch normalization on a six-GPU machine.Source: §D.3 · p. 14

E. Model Implementation

The implementation standardizes several 2D backbones and video architectures, including their initialization and temporal-pooling placement. It describes the specific construction choices for I3D, S3D, TAM, and TSN.

E. Model Implementation

  • The common framework uses InceptionV1, ResNet18, and ResNet50 backbones with I3D, S3D, TAM, and TSN architectures, aligning temporal-pooling locations across backbones.ImageNet-pretrained weights initialize models by inflation for I3D and S3D or direct loading for TAM and TSN; inflation copies weights along time.Source: E. Model Implementation · p. 14
  • I3D converts 2D convolutions to 3D convolutions with temporal kernel size 3 while retaining spatial kernel sizes.In I3D-ResNet50, bottleneck 3×3 convolutions become 3×3×3 convolutions.Source: E. Model Implementation · p. 14
  • S3D factorizes each I3D 3D convolution into a 2D spatial convolution followed by a 1D temporal convolution, except for the first convolution.For InceptionV1, temporal stride is removed from the first convolution and a temporal-pooling layer is added to preserve the temporal downsampling ratio.Source: E. Model Implementation · p. 14
  • TAM modules are inserted on residual blocks’ non-identity paths or after Inception modules, while TSN uses 2D models without temporal modeling.For ResNet S3D, output channels remain at their original size to permit direct loading of ImageNet-pretrained weights.Source: E. Model Implementation · p. 15

Related papers

Loading 2010.11757v4…