PaperlayerView exact revision on arXiv

Source-linked AI summary

TEA: Temporal Excitation and Aggregation for Action Recognition

Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, Limin Wang

arXiv:2004.01398v1cs.CV

TL;DR

Action recognition requires temporal modeling across both adjacent frames and distant frames, but existing approaches face limitations in motion integration, long-range aggregation, or computational efficiency. TEA addresses this with complementary ME and MTA modules integrated into ResNet blocks. The resulting network achieves stronger benchmark performance at low computational cost.

  • Problem

    Action recognition needs both short-range motion and long-range temporal aggregation, while existing methods leave gaps in modeling complex temporal structure efficiently.

  • Method

    TEA combines motion excitation, which uses feature-level temporal differences to enhance motion-sensitive channels, with multiple temporal aggregation, which enlarges the temporal receptive field without additional parameters.

  • Results

    TEA slightly increases computation to 35G FLOPs (1.06×) while improving performance to 48.8% versus 43.4% for TSN8f and TSM8f.

  • Takeaways & Limitations

    The complementary ME and MTA modules provide effective and efficient short- and long-range temporal modeling within standard ResNet blocks.

Abstract

from arXiv · show

Temporal modeling is key for action recognition in videos. It normally considers both short-range motions and long-range aggregations. In this paper, we propose a Temporal Excitation and Aggregation (TEA) block, including a motion excitation (ME) module and a multiple temporal aggregation (MTA) module, specifically designed to capture both short- and long-range temporal evolution. In particular, for short-range motion modeling, the ME module calculates the feature-level temporal differences from spatiotemporal features. It then utilizes the differences to excite the motion-sensitive channels of the features. The long-range temporal aggregations in previous works are typically achieved by stacking a large number of local temporal convolutions. Each convolution processes a local temporal window at a time. In contrast, the MTA module proposes to deform the local convolution to a group of sub-convolutions, forming a hierarchical residual architecture. Without introducing additional parameters, the features will be processed with a series of sub-convolutions, and each frame could complete multiple temporal aggregations with neighborhoods. The final equivalent receptive field of temporal dimension is accordingly enlarged, which is capable of modeling the long-range temporal relationship over distant frames. The two components of the TEA block are complementary in temporal modeling. Finally, our approach achieves impressive results at low FLOPs on several action recognition benchmarks, such as Kinetics, Something-Something, HMDB51, and UCF101, which confirms its effectiveness and efficiency.

1. Introduction

The paper targets temporal modeling for action recognition by addressing gaps in short-range motion encoding and long-range temporal aggregation. It introduces complementary ME and MTA modules that provide efficient short- and long-range modeling within ResNet blocks.

  • Temporal modeling must capture both short-range motion between adjacent frames and long-range aggregation across distant frames.
  • Optical-flow-based two-stream methods are time-consuming, storage demanding, and separate spatial and temporal feature learning until late fusion.
  • Simple temporal pooling can lose or confuse temporal information, while stacked local convolutions make long-range modeling difficult to optimize.
  • The TEA network combines lightweight, complementary modules in standard ResNet blocks, using only 1.06× the FLOPs of 2D ResNet.
  • The ME module integrates short-range motion modeling into whole spatiotemporal feature learning.
  • The MTA module efficiently enlarges the temporal receptive field for long-range temporal modeling.

2. Related Works

Prior action-recognition methods use frame-wise fusion, optical-flow streams, or computationally heavier temporal operators, with limitations in motion modeling, feature integration, or efficiency. TEA instead inserts ME and MTA into a 2D ResNet framework to combine efficient motion excitation with temporal aggregation.

  • Frame-wise 2D CNNs and simple temporal pooling can lose temporal information, while recurrent models isolate frame feature extraction.
  • Two-stream methods separately train spatial RGB and temporal optical-flow networks, averaging their final video predictions.
  • STM adds motion encoding directly to spatiotemporal features, whereas TEA uses motion features to recalibrate and enhance motion-sensitive patterns.
  • 3D and (2+1)D CNNs jointly model spatial and temporal features but generally require higher computation than 2D-based approaches.
  • TEA samples T frames sparsely, applies a 2D ResNet backbone with inserted ME and MTA modules, and averages action predictions temporally.
  • MTA enlarges temporal modeling without extra parameters or time-consuming operations by replacing local processing with grouped aggregation.

3. Our Method

TEA integrates motion excitation and multiple temporal aggregation into ResNet blocks to model short- and long-range temporal evolution efficiently. ME enhances motion-sensitive channels from feature-level differences, while MTA hierarchically enlarges temporal receptive fields without additional parameters.

  • TEA Block Integration: TEA stacks temporal excitation and aggregation blocks within a 2D CNN ResNet framework for spatiotemporal video modeling.The network samples one frame from each of T temporal segments and constructs the model by stacking TEA blocks.
  • Motion Excitation (ME): ME identifies motion-sensitive channels by computing transformed feature differences between adjacent frames.Features are channel-reduced for efficiency, transformed channel-wise, differenced across neighboring frames, spatially pooled, and expanded back to the original channel dimension.
  • Motion Excitation (ME): ME uses residual excitation to enhance motion information while preserving static background information useful for action recognition.Its output combines the input feature with channel-wise motion-attentive modulation rather than suppressing static content completely.
  • Multiple Temporal Aggregation (MTA): MTA splits features and local convolutions into channel-wise fragments processed through hierarchical residual sub-convolutions, enlarging the temporal receptive field without additional parameters.Successive sub-convolutions aggregate information from preceding fragments, while temporal and spatial operations are applied to the later fragments.
  • Multiple Temporal Aggregation (MTA): MTA produces spatiotemporal representations covering different temporal ranges, unlike a single local temporal convolution.The final outputs from the fragments are concatenated to form the module output.

4. Experiments

The experiments evaluate TEA on four action-recognition datasets, with ablations and comparisons testing its temporal modeling, accuracy, and computational efficiency. TEA performs strongly across benchmarks, particularly on Something-Something, while using relatively low FLOPs.

  • Datasets: TEA is evaluated on Something-Something V1, Kinetics400, HMDB51, and UCF101, with Something-Something emphasized because its interactions require stronger temporal understanding.Something-Something V1 contains 174 categories, while Kinetics400, HMDB51, and UCF101 provide broader benchmark coverage.
  • Ablation Study: The ablation study compares altered ResNet-based blocks, including (2+1)D ResNet, (2+1)D Res2Net, ME, MTA, and TEA variants.The comparisons isolate motion excitation, temporal aggregation, residual connections, and alternative block designs.
  • Ablation Study: 46.5% vs. 46.0% shows that frame-wise SE excitation provides only limited improvement over the (2+1)D ResNet baseline.The SE block processes each video frame independently and does not consider temporal information.
  • Ablation Study: 48.9% is achieved by TEA after adding MTA to ME, while removing residual connections reduces performance from 48.1% to 47.2%.The results support complementary short- and long-range modeling and the need to retain static scene information through residual connections.
  • State-of-the-art Comparisons: 35G FLOPs and 48.8% accuracy give TEA a 5.4% relative improvement over 43.4% methods with similar efficient inference on Something-Something.TEA uses 1.06× the FLOPs of comparable 2D ResNet-style computation.
  • State-of-the-art Comparisons: 52.3% on Something-Something with 16 frames and full resolution×10 clips surpasses NL I3D+GCN at 46.1% and RGB-plus-flow methods at 49.5% and 50.2%.The reported test-set result is 46.6%, which outperforms most existing methods.
  • State-of-the-art Comparisons: On Kinetics400, TEA reaches 76.1%, outperforming TSM16f and STM16f at 74.7% and 73.7% but remaining below SlowFast at 79.8%.With 8 frames, TEA reaches 75.0% versus TSM’s 74.1%.
  • State-of-the-art Comparisons: TEA obtains 73.3% on HMDB51 and 96.9% on UCF101 under the accuracy inference protocol, outperforming most existing methods except I3D.I3D uses 3D CNNs and an additional input modality, resulting in substantially higher computational FLOPs.

5. Conclusion

TEA combines motion excitation and multiple temporal aggregation to model both short- and long-range temporal relationships. Its MTA module enlarges the equivalent temporal receptive field through grouped sub-convolutions.

  • TEA consists of motion excitation (ME) and multiple temporal aggregation (MTA) modules for short- and long-range temporal modeling.
  • The ME module inserts motion encoding into spatiotemporal feature learning and enhances motion patterns in the features.
  • The MTA module deforms local convolutions into sub-convolutions to establish long-range temporal relationships over an enlarged equivalent receptive field.

6. Ackonwledgement

The work was supported by Tencent’s Video Understanding Middle Platform within the Platform and Content Group. The authors also thanked Wei Shen for helpful suggestions.

  • The work was supported by Tencent’s Video Understanding Middle Platform of the Platform and Content Group.
  • The authors thanked Wei Shen for helpful suggestions.
  • The acknowledgement identifies both institutional support and individual technical assistance.

A. Temporal Convolutions in TEA

This section motivates learnable temporal convolutions and examines their initialization in (2+1)D ResNet. Part-shift initialization improves performance while preserving spatial learning and is applied in TEA.

  • Motivation: 3D convolutions are computationally expensive, while jointly optimizing spatial and temporal modeling can make video models harder to optimize.
  • TSM background: TSM exchanges neighboring-frame information by shifting feature channels and leaves most channels unchanged to balance temporal modeling with spatial representation learning.
  • Temporal convolutions: TSM can be represented as a 1D temporal convolution with fixed, pre-designed kernel weights, motivating learnable temporal kernels.
  • Experiments: The compared (2+1)D ResNet variants differ only in their 1D temporal convolution type and are evaluated on Something-Something V1 with eight input frames.
  • Results: The general temporal-convolution baseline performs poorly, while channel-wise convolution reaches 43.6% versus TSM’s 43.4%.
  • Results: 46.0% top-1 accuracy is obtained by part-shift initialization, substantially exceeding TSM and motivating its use throughout TEA temporal convolutions.

B. Training Details

The section introduces detailed training configurations for the TEA network across datasets. It also notes that code and related experimental logs would be made available.

  • Training configurations are described for the TEA network on different datasets.
  • The section concerns implementation details for training the TEA network.
  • The authors state that codes and related experimental logs would be made available.

B.1. Model Initializations

The model uses ImageNet-pretrained 2D CNNs for initialization, with Res2Net selected for the MTA-based architecture. Motion differences are computed after a transformation convolution to better align displaced regions.

  • ImageNet-pretrained 2D CNNs initialize the action-recognition networks, including a released Res2Net50 model for the proposed architecture.
  • Res2Net is selected because the MTA module is based on an architecture different from standard ResNet.
  • 46.2% vs. 46.0%: (2+1)D Res2Net provides only slight action-recognition improvement over (2+1)D ResNet.The passage contrasts stronger image-task backbones with the smaller gains observed for video action recognition.
  • The training setup follows prior work and uses dataset-specific fine-tuning procedures for Kinetics, Something-Something, HMDB51, and UCF101.The supplied passages specify batch normalization, learning-rate, weight-decay, dropout, and batch-size settings for these procedures.
  • A 3×3 convolution transforms features at time step t + 1 before temporal differences are computed.The transformation aims to capture matched regions after motion-induced spatial displacement.

D. Runtime Analysis

Runtime analysis compares TEA with other methods and isolates the timing costs of its components. MTA contributes more additional runtime than ME because its stages are processed sequentially.

  • TEA achieves higher accuracy than STM with similar efficiency, while TEA8F is both more effective and efficient than TSM16F and I3D.
  • 19.7% vs. 48.9%: the 2D ResNet baseline runs nearly 1.8x faster than TEA but performs far worse.
  • 0.0062s: MTA adds more runtime than ME, whose added time is 0.0033s.The MTA stages require sequential processing, whereas ME has a smaller measured increase.
  • Feature differences are computed between adjacent timestamp groups by subtracting f_t from f_t1.For input features shaped N×T×C×H×W, the operation produces T−1 temporal differences.

E. The Location of the TEA Block

An ablation study examines where TEA blocks are inserted across ResNet stages, while feature visualizations show that ME selectively amplifies motion-sensitive channels and preserves background channels weakly.

  • The Location of the TEA Block: TEA blocks replace all ResNet blocks within a selected stage, while the remaining stages retain their original ResNet blocks.The study evaluates insertion at stages such as conv2 while leaving conv3–conv5 unchanged.
  • ME Visualization: Figure 5 visualizes input and output features of ME in the Conv2 1 block.
  • ME Visualization: A=0.90/0.58: ME enhances swimmer-related channels 11 and 25 with large attention weights.
  • ME Visualization: A=0.08/0.05: ME assigns low attention weights to channels 14 and 42 that preserve background information.

G. Experimental Results on Something-Something V2

On Something-Something V2, TEA outperforms existing state-of-the-art methods on validation data and remains comparable on the test set under the full-resolution ten-clip protocol.

  • Dataset and Evaluation: Something-Something V2 contains 168,913 training videos, 24,777 validation videos, and 27,157 videos in its test set.The dataset is described as twice the size of Something-Something V1.
  • Dataset and Evaluation: The accuracy protocol samples ten clips at full resolution, and final predictions average the clips' scores.
  • Validation and Test Results: 65.1%: TEA outperforms existing state-of-the-art methods on the Something-Something V2 validation set.
  • Validation and Test Results: 63.2% vs. 63.5%: TEA is comparable to the state-of-the-art result on the test set.
Loading 2004.01398v1…