Directional Temporal Modeling for Action Recognition

1 Introduction

• 3D backbones can have local effective receptive fields, while global average pooling discards temporal ordering and relationships between sub-clips.The introduction notes that full-clip theoretical receptive fields do not necessarily provide full-clip motion understanding.Source: §1 · p. 1
• CIDC progressively aggregates backbone features across the clip in both forward and backward directions, preserving temporal order between sub-clips.Its output represents increasingly larger portions of the clip, from early frames through the full clip.Source: §1 · p. 2
• The lightweight CIDC network stacks units on the last three backbone blocks to aggregate features from multiple spatial-temporal scales.It can attach to activity-recognition backbones such as I3D and I3D-NL.Source: §1 · pp. 2–3
• The activation example contrasts I3D and I3D-NL responses to moving audiences with CIDC's focus on the hurdling athletes.The figure shows representative activation maps for sub-clips of a 32-frame Kinetics-400 video.Source: §1 · p. 2, Figure 1 · p. 2
• The multi-scale CIDC network improves state-of-the-art activity recognition models on UCF-101, HMDB51, Something-Something V2, and Kinetics-400.The introduction reports consistent improvements but provides no numerical results here.Source: §1 · pp. 2–3

2 Related Work

• ConvNet-2D feature aggregation, two-stream optical flow, and ConvNet-3D are established approaches, with 3D networks handling spatial and temporal dimensions.The related-work section also describes TSM and TAM channel shifting, SlowFast's fast-motion and slow-appearance branches, and LFB self-attention aggregation.Source: §2 · p. 3
• Temporal modeling approaches include LSTMs, temporal rank pooling, and self-attention, each with distinct constraints or reported benefits.The section notes that rank pooling requires flattened features that may compromise spatial arrangement, limiting its insertion mid-network.Source: §2 · p. 3
• CIDC models clip-level temporal associations, is described as complementary to approaches capturing local spatiotemporal information, and outperforms LSTM and self-attention blocks in accuracy.The comparison is reported empirically in the related-work discussion.Source: §2 · p. 3

3.1 Channel Independent Directional Convolution

• CIDC applies directional spatial convolutions independently per channel, progressively adding temporal context so output positions summarize increasingly long portions of the clip.Changing the output temporal length provides soft temporal resizing; grouped convolutions implement the channel-independent operation efficiently.Source: §3.1 · p. 4, Channel Independent Directional… · p. 5

3.2 Multi-scale CIDC Network

• The network fuses backbone features with CIDC features before the final classification layer, using either concatenation or summation.The fusion function determines how the two feature representations are combined.Source: §3.2 · p. 6
• Multiple CIDC branches aggregate feature maps across scales, passing higher-resolution features to lower-resolution stages through dimension reduction and element-wise addition.The architecture uses one-way cross-scale aggregation rather than bidirectional aggregation.Source: Multi-scale aggregation · pp. 6–7, Figure 3 · p. 6
• Later-stage activation maps are upsampled to weight early-stage features, directing CIDC toward semantically important regions before temporal modeling.The attention operation uses bilinear interpolation and retains the original early-stage feature through a residual addition.Source: Spatial-attention propagation · p. 7

3.3 Implementation Details

• A masked, softmax-normalized weight matrix enforces directional temporal convolution while keeping its weights compatible with SGD optimization.When input and output temporal lengths differ, the triangular matrix is resized by bilinear interpolation; the operation can also be made bidirectional by flipping the input in time.Source: Efficient CIDC Unit · pp. 7–8
• Training ran 100 epochs with SGD, a 0.01 learning rate decayed at epochs 40 and 80, and 32-frame clips; inference averages 30 clip-crop predictions.Training also used dropout and spatial augmentation; inference samples 10 clips and takes three crops per clip.Source: Training Details · p. 8

4.1 Dataset

• Evaluation covers UCF-101, HMDB51, Something-Something V2, and Kinetics-400, with top-1 accuracy reported on all four and top-5 also reported on the latter two.UCF-101 and HMDB51 use split 1; Something-Something V2 and Kinetics-400 use validation sets.Source: §4.1 · p. 8

4.2 Comparison with state-of-the-art

• CIDC achieves state-of-the-art on UCF101/HMDB51, cutting 3D-network error 61% and 11%–19%, and consistently improves 2D/3D methods on Kinetics-400 and Something-Something V2.On Something-Something V2, error falls 7.3% on R2D and 13.2% on I3D baselines.Source: §4.2 · p. 9, Table 1 · p. 9, Table 2 · p. 10

4.3 Ablation Study

• On UCF-101, HMDB-51, and Kinetics-400, each added configuration component boosts performance, and multi-scale CIDC benefits from aggregating early-stage feature maps.The comparisons add multi-scale CIDC to single-scale CIDC, then add spatial attention.Source: §4.3 · p. 10, Table 4 · p. 11
• Bi-directional CIDC performs best; relative to non-directional CIDC, it reduces error by 52.5% on UCF-101, 8.7% on HMDB-51, and 2.2% on Kinetics-400.The directional variants compare non-directional, uni-directional, and bi-directional temporal masking.Source: §4.3 · pp. 10–11, Table 4 · p. 11
• Temporal concatenation is the best tested feature fusion function S and is selected for CIDC, while LSTM trails I3D and self-attention trails CIDC.The fusion comparison is on Kinetics-400; the temporal-model comparison uses Table 5.Source: §4.3 · pp. 11–12, Table 5 · p. 11
• CIDC boosts top-1 accuracy across 2D and 3D backbones on all four datasets, including backbones with non-local temporal mechanisms.The authors describe the learned temporal associations as generalizing across backbones and complementing other temporal modeling techniques.Source: §4.3 · p. 12

4.4 Error Analysis

• On Kinetics-400, CIDC helps classes with large target-object motions and similar-looking activities distinguished by motion, but some classes show little benefit.The authors note garbage collection and springboard diving as cases without benefit, attributing them to object cues and subtle motion, respectively.Source: §4.4 · p. 12, Table 6 · p. 13

4.5 Visualizing CIDC activations

• Activation maps show CIDC focusing on action-explanatory regions in clapping and peeling, whereas I3D and I3D-NL also activate irrelevant regions.The examples use Kinetics-400 clips and representative frames from 8-frame sub-clips.Source: §4.5 · pp. 13–14, Figure 4 · p. 14

• CIDC learns temporal associations among local features directionally, encoding action temporal ordering into feature maps.Source: §5 · p. 14
• A lightweight network built from CIDC units models clip-level temporal associations of local spatial and spatial-temporal features.Source: §5 · p. 14
• The method achieves state-of-the-art performance on four datasets, and the ablation study validates CIDC's effectiveness for temporal modeling in action recognition.Source: §5 · p. 14
• Activation-map visualizations show that the CIDC network generally focuses on moving targets performing actions.Source: §5 · p. 14