PaperlayerView exact revision on arXiv

Source-linked AI summary

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition

M. Esat Kalfaoglu, Sinan Kalkan, A. Aydin Alatan

arXiv:2008.01232v3cs.CV

TL;DR

TGAP at the end of 3D CNNs averages temporal features despite potentially meaningful differences in their importance or order. The paper replaces TGAP with BERT-based late temporal modeling, reporting improvements across several 3D CNN architectures and top-1 results of 85.10% on HMDB51 and 98.69% on UCF101.

  • Problem

    TGAP averages temporal features without accounting for their differing importance or order, potentially leaving temporal information underused.

    Source: §1 · p. 2

  • Method

    The paper replaces TGAP at the end of 3D CNN architectures with BERT’s attention mechanism for late temporal modeling.

    Source: §1 · p. 2

  • Results

    BERT improves ResNeXt101, I3D, SlowFast, and R(2+1)D, with 85.10% Top-1 on HMDB51 and 98.69% Top-1 on UCF101 using BERT R(2+1)D.

    Source: §1 · p. 2

  • Takeaways & Limitations

    BERT performs better than average pooling, concatenation pooling, and standard LSTM pooling, while further parameter-efficient implementations remain a future direction.

    Source: §5 · p. 16

    R(2+1)D results use 34-layer basic blocks and IG65M pretraining, and are limited to RGB, so comparisons with the other architectures require these conditions to be considered.

    Source: R(2+1)D Architecture · p. 14

Abstract

from arXiv · show

In this work, we combine 3D convolution with late temporal modeling for action recognition. For this aim, we replace the conventional Temporal Global Average Pooling (TGAP) layer at the end of 3D convolutional architecture with the Bidirectional Encoder Representations from Transformers (BERT) layer in order to better utilize the temporal information with BERT's attention mechanism. We show that this replacement improves the performances of many popular 3D convolution architectures for action recognition, including ResNeXt, I3D, SlowFast and R(2+1)D. Moreover, we provide the-state-of-the-art results on both HMDB51 and UCF101 datasets with 85.10% and 98.69% top-1 accuracy, respectively. The code is publicly available.

1 Introduction

The paper replaces temporal global average pooling in 3D CNNs with BERT-based late temporal modeling to better use temporal information. It reports improvements across several architectures and state-of-the-art results on HMDB51 and UCF101.

1 Introduction

  • The work combines hierarchical temporal relationships from 3D convolution with late temporal modeling typically applied to 2D CNN features.Late temporal modeling commonly uses recurrent architectures such as LSTM or Conv LSTM on features extracted from selected frames.Source: §1 · pp. 1–2
  • TGAP can discard useful temporal distinctions because it averages features even when temporal regions differ in importance or order.The authors describe features before TGAP as representing different temporal regions, with effective receptive fields focusing on different regions.Source: §1 · p. 2
  • BERT’s multi-head attention mechanism is proposed to identify which temporal features matter more than TGAP does.Source: §1 · p. 2
  • 85.10% and 98.69% Top-1 performances were obtained on HMDB51 and UCF101, respectively, with BERT R(2+1)D.The authors also report improved performance for ResNeXt101, I3D, SlowFast, and R(2+1)D on HMDB51 split 1.Source: §1 · p. 2

2 Related Work on Action Recognition

Related work covers temporal integration strategies and 3D CNN architectures. The paper identifies an unresolved need for effective temporal fusion at the end of 3D CNNs.

2.1 Temporal Integration Using Pooling, Fusion or Recurrent Architectures

  • Pooling, fusion, and recurrent architectures are established approaches for integrating temporal information in action recognition.Pooling examples include concatenation and averaging; recurrent methods include LSTMs applied to 2D CNN frame features.Source: §2.1 · pp. 2–3

2.2 3D CNN Architectures

  • 3D CNNs model temporal information hierarchically throughout the network, but require greater computation and memory than 2D counterparts.SlowFast combines two streams with different frame rates and spatial-versus-temporal capabilities.Source: §2.2 · p. 3
  • Despite their capacity, 3D CNNs still lack an effective temporal fusion strategy at the architecture’s end.Source: §2.2 · p. 3

3 Proposed Methods

The methods replace end-of-network TGAP with BERT-based temporal pooling, add feature-reduction blocks to manage BERT’s parameter cost, and adapt the design to SlowFast’s two streams.

3 Proposed Methods

  • The proposed methods cover BERT-based temporal modeling, feature-reduction blocks, and SlowFast-specific implementations.Source: §3 · p. 4

3.1 BERT-based Temporal Modeling with 3D CNNs for Action Recognition

  • Selected frames pass through a 3D CNN without TGAP; positional encodings and a learned classification embedding are input to BERT, whose classification vector feeds a classifier.The method uses the classification embedding rather than the pooled backbone feature used by the cited video action transformer network.Source: §3.1 · pp. 4–6
  • BERT’s attention combines temporal visual embeddings, while a learnable classification embedding learns how to attend to those features.The embeddings include positional encoding; the resulting classification vector is passed to a final linear layer.Source: §3.1 · p. 5

3.2 Proposed Feature Reduction Blocks: FRAB & FRMB

  • A single-layer BERT module has about 3 million parameters at feature dimension 512 and about 50 million at dimension 2048.The authors state that BERT’s computational complexity increases quadratically with the CNN backbone’s output feature dimension.Source: §3.2 · p. 6
  • FRMB replaces the CNN’s final unit block to reduce feature dimension, whereas FRAB appends an additional reduction block.FRMB offers better computational complexity and parameter efficiency than FRAB, but may forgo larger-dataset pretraining weights for the replaced block when reduction is introduced only during fine-tuning.Source: §3.2 · pp. 6–7

3.3 Proposed BERT Implementations on SlowFast Architecture

  • SlowFast’s different stream resolutions prevent directly applying the standard BERT design, so the paper proposes early- and late-fusion variants.Early fusion aligns fast-stream temporal resolution to the slow stream and uses one BERT; late fusion uses one BERT per stream and concatenates their outputs.Source: §3.3 · p. 7

4 Experiments

The experiments evaluate BERT temporal pooling through dataset and implementation choices, pooling ablations, tests across four 3D CNN architectures, and comparisons with state-of-the-art methods. The reported results include state-of-the-art performance on HMDB51 and UCF101 and examine how pooling design, architecture, and pretraining affect outcomes.

4.1 Dataset

  • HMDB51 and UCF101 have three splits used for averaged performance; the study also uses Kinetics-400 and IG65M for pretraining before fine-tuning.HMDB51 has about 7k clips and 51 classes; UCF101 has about 13k clips and 101 classes.Source: §4.1 · pp. 7–8

4.2 Implementation Details

  • Fine-tuning uses architecture-matched normalization, multi-scale cropping, averaged scores from non-overlapping test clips, and architecture-specific optimizers and BERT settings.Standard architectures use SGD, while BERT architectures use ADAMW; learning rates vary by architecture.Source: §4.2 · pp. 8–9

4.3 Ablation Study

  • Among six FRMB pooling alternatives, BERT performs best and improves Top-1 accuracy by 3% over ResNeXt101 average pooling while retaining better computational complexity and parameter efficiency.The BERT layer adds about 3M parameters with negligible computational complexity relative to the ResNeXt101 backbone.Source: Table 1 · p. 9, §4.3 · p. 10
  • ADAMW raises average-pooling performance 1.5% over SGD; FRMB beats FRAB by about 0.5% but trades about 1% Top-1 against the standard backbone for efficiency.FRMB has lower computational complexity and better parameter efficiency than the standard backbone.Source: Table 1 · p. 9, §4.3 · p. 10
  • Replacing average-pooled features with a learnable classification token boosts Top-1 accuracy 1.42%; eight-head attention gains 0.52% over single-head, while two layers reduce performance 0.25%.Each additional BERT layer costs about 3M in memory trade-off.Source: §4.3 · pp. 10–11, Table 2 · p. 11

4.4 Results on Different 3D CNN Architectures

4.5 Comparison with State-of-the-Art

  • IG65M-pretrained R(2+1)D BERT (64f) is reported as state-of-the-art on HMDB51 and UCF101; IG65M-pretrained RGB R(2+1)D (32f) beats Kinetics-400 by about 6% and 1.4% Top-1, respectively.The paper also reports BERT gains over standard two-stream ResNeXt101 and RGB R(2+1)D (32f) on both datasets.Source: §4.5 · pp. 14–15, Table 7 · p. 15

5 Conclusions

The study replaces TGAP at the output of 3D CNNs with BERT-based late temporal modeling, and reports improvements over state-of-the-art techniques on HMDB51 and UCF101. It identifies late temporal pooling as its main contribution while outlining several directions for future work.

5 Conclusions

  • The study replaces TGAP at the output of 3D CNNs with BERT to better employ temporal information.The authors describe this as the first study to remove TGAP and use BERT for late temporal modeling in 3D CNN architectures.Source: §5 · p. 15
  • The method reports significant improvements over state-of-the-art techniques on HMDB51 and UCF101.It evaluates the approach on ResNeXt, I3D, SlowFast, and R(2+1)D.Source: §5 · pp. 15–16
  • The authors identify late temporal pooling as the main contribution and propose more parameter-efficient BERT implementations and unsupervised methods as future work.They also note potential applications to temporal and spatial action localization and video captioning.Source: §5 · p. 16

Related papers

Loading 2008.01232v3…