PaperlayerView exact revision on arXiv

Source-linked AI summary

TALLFormer: Temporal Action Localization with a Long-memory Transformer

Feng Cheng, Gedas Bertasius

arXiv:2204.01680v2cs.CV

TL;DR

Long-video TAL requires both useful short-term features and long-range boundary localization, but memory limits often constrain feature extraction, especially with video transformers. TallFormer reduces repeated clip processing by using long-term memory, and reports higher benchmark results with RGB input alone and no external recognition classifier.

  • Problem

    Memory costs for long untrimmed videos lead TAL methods to freeze backbones or reduce spatial resolution, while video transformers add quadratic self-attention costs.

    Source: §1 · p. 2

  • Method

    TallFormer processes a fraction of clips with a short-term Transformer and retrieves other clip features from long-term memory, updating the cache with newly computed features.

    Source: §1 · p. 3, §3 · p. 5

  • Results

    59.1% average mAP on THUMOS14 and 35.6% on ActivityNet-1.3; TallFormer outperforms prior state-of-the-arts by 7.1% and 1.2%, respectively.

    Source: §1 · p. 4

  • Takeaways & Limitations

    TallFormer predicts boundaries and categories in one RGB-only model without an external action recognition classifier, while supporting long, high-resolution end-to-end training.

    Source: §1 · p. 4, §5 · p. 14

Abstract

from arXiv · show

Most modern approaches in temporal action localization divide this problem into two parts: (i) short-term feature extraction and (ii) long-range temporal boundary localization. Due to the high GPU memory cost caused by processing long untrimmed videos, many methods sacrifice the representational power of the short-term feature extractor by either freezing the backbone or using a small spatial video resolution. This issue becomes even worse with the recent video transformer models, many of which have quadratic memory complexity. To address these issues, we propose TALLFormer, a memory-efficient and end-to-end trainable Temporal Action Localization Transformer with Long-term memory. Our long-term memory mechanism eliminates the need for processing hundreds of redundant video frames during each training iteration, thus, significantly reducing the GPU memory consumption and training time. These efficiency savings allow us (i) to use a powerful video transformer feature extractor without freezing the backbone or reducing the spatial video resolution, while (ii) also maintaining long-range temporal boundary localization capability. With only RGB frames as input and no external action recognition classifier, TALLFormer outperforms previous state-of-the-arts by a large margin, achieving an average mAP of 59.1% on THUMOS14 and 35.6% on ActivityNet-1.3. The code is public available: https://github.com/klauscc/TALLFormer.

1 Introduction

Temporal action localization must extract useful short-term features and use them to locate action boundaries across long videos, but memory costs often force weaker feature extraction. TallFormer uses long-term memory to reduce repeated processing, and reports stronger results on both benchmarks with RGB input alone.

1 Introduction

  • Memory limits lead prior methods to freeze the backbone or use small spatial resolutions, degrading feature quality and TAL accuracy.On THUMOS14, reducing resolution from 168 × 168 to 112 × 112 drops mAP by 2.3%; freezing the backbone drops it by about 8–9%.Source: §1 · p. 2
  • Video transformers intensify the challenge because quadratic self-attention requires substantial GPU memory, even with checkpointing or mixed precision.This makes adapting them to TAL difficult, given TAL’s existing memory demands.Source: §1 · pp. 2–3
  • TallFormer processes only a fraction of randomly selected clips per training iteration and retrieves the remaining features from long-term memory.Newly computed features replace older cached features as the short-term extractor evolves.Source: §1 · p. 3
  • 59.1% average mAP on THUMOS14 and 35.6% on ActivityNet-1.3; TallFormer outperforms prior state-of-the-arts by 7.1% and 1.2%, respectively.The model uses RGB inputs and does not rely on an external action recognition classifier.Source: §1 · p. 4

2 Related Work

Related work covers short-term action recognition features, TAL systems, and techniques for reducing memory use. The paper contrasts reliance on pre-extracted features or low-resolution end-to-end training with its goal of long, high-resolution end-to-end TAL.

2 Related Work

  • Video transformers have improved action recognition over CNNs, but quadratic self-attention makes them slow and memory-intensive for long-term tasks such as TAL.Pretrained action-recognition models are widely used as TAL feature extractors.Source: §2 · p. 4
  • Many TAL methods use pre-extracted recognition features; AFSD and DaoTAD train end-to-end but use low spatial resolutions that reduce TAL accuracy.The authors state that, to their knowledge, no existing method combines end-to-end training with both high spatial resolution and long temporal extent.Source: §2 · p. 4
  • Two-stage TAL detectors use external recognition classifiers, whereas one-stage detectors localize and classify together; the former typically have higher detection accuracy.The paper aims to achieve strong benchmark results without an external classifier.Source: §2 · pp. 4–5
  • Gradient checkpointing and mixed precision reduce GPU memory use by about 50%, and the paper describes its approach as complementary to them.Low-rank attention approximations such as LinFormer and Performer are reported to work poorly for video recognition.Source: §2 · p. 5

3 TallFormer

TallFormer combines a short-term Transformer encoder, long-term feature memory, temporal consistency refinement, and a boundary-localization module. It uses sampled clips and cached features to train on long, high-resolution videos while predicting action boundaries and categories.

3 TallFormer

  • TallFormer combines a short-term Transformer encoder, long memory, temporal consistency, and temporal boundary localization to predict action boundaries and categories.The model’s action instances include start and end timestamps, a class, and an instance probability.Source: §3 · p. 5

3.1 Short-term Transformer Encoder

  • The short-term encoder processes a fixed number of randomly sampled clips, while the remaining clips use long-memory features, enabling end-to-end training on long, high-resolution video.The approach avoids reducing spatial resolution, freezing the backbone, or using a weak short-term feature extractor.Source: §3.1 · p. 6

3.2 Long Memory Module

  • The Long Memory Module caches encoder features for all clips, supplies features for clips not processed online, and updates the cache with newly computed features.The pretrained short-term encoder evolves more slowly than other modules, so cached features approximate its current features while saving training time and GPU memory.Source: §3.2 · p. 7

3.3 Temporal Consistency Module

  • The Temporal Consistency Module refines online and cached features together because features from different training iterations may have different distributions.Its self-attention layers incorporate temporal information across the untrimmed video to produce refined features.Source: §3.3 · pp. 7–8

3.4 Temporal Boundary Localization Module

  • The boundary-localization module predicts action boundaries and categories from refined clip features, using a shared linear classifier rather than an external recognition classifier.For ActivityNet, the classifier operates on temporally averaged features with 50% dropout.Source: §3.4 · p. 8

4 Experiments

Experiments compare TallFormer with constrained baselines and prior methods across THUMOS14, ActivityNet-1.3, and HACS-Segment. Ablations test the effects of clip sampling rate, temporal consistency, and temporal support.

4.1 Datasets and Evaluation Metrics

  • Evaluation covers THUMOS14, ActivityNet-1.3, and HACS-Segment; THUMOS14 uses validation videos for training and test videos for evaluation, while the other datasets use their training and validation splits.Results are reported as mAP using dataset-specific IoU thresholds.Source: §4.1 · pp. 8–9

4.2 Implementation Details

  • Experiments use a pretrained Video Swin Transformer, RGB input with 224 × 224 crops, gradient checkpointing, and four RTX A6000 GPUs.THUMOS14 uses 480 frames at 15fps; ActivityNet-1.3 videos are resized to 768 RGB frames.Source: §4.2 · p. 9

4.3 Comparison with Short-term and Long-term Baselines

  • Under 32GB, 12-second ST-E2E is 4.1% lower in average mAP than TallFormer; sparse 32-second ST-E2E is 2.9% worse, and LT-Frozen has 6.5% lower accuracy.TallFormer has the best accuracy-memory trade-off under both 12GB and 32GB constraints.Source: Table 2 · p. 10, §4.3 · pp. 10–11

4.4 Comparison to the State-of-the-Art

  • On THUMOS14, same-memory TallFormer-12 with I3D and Swin-B beats AFSD by 1.9% and 7.0%; on ActivityNet-1.3 it beats prior approaches by 1.2% using RGB alone.The comparisons are summarized in Table 3.Source: §4.4 · pp. 11–12, Table 3 · p. 12
  • On HACS-Segment, TallFormer exceeds GTAD and BMN by 9.1% and 0.7% average mAP, respectively, without an external classifier.The results are reported in Table 4.Source: §4.4 · p. 12, Table 4 · p. 13

4.5 Ablation Study

  • At r=0.15, mAP drops only 1.0% while memory falls to 12GB from >45GB at r=1.0; TCM adds 1.5% average mAP, and longer temporal support raises mAP.The sampling-rate results use Swin-B at 224×224; the TCM and temporal-support results use Swin-T at 112 × 112.Source: Table 5 · p. 13, §4.5 · p. 14

4.4 Comparison to the State-of-the-Art

  • On ActivityNet-1.3, TallFormer inference takes 1.58s/video versus AFSD’s 11.74s/video while outperforming AFSD by 1.2% mAP.The timing comparison uses an RTX A6000 GPU.Source: §4.4 · p. 13

5 Discussion

TALLFormer is presented as a simple, flexible framework for efficiently training on long, high-resolution videos. The authors report that it significantly outperforms previous TAL approaches on THUMOS14 and ActivityNet-1.3.

5 Discussion

  • TALLFormer significantly outperforms previous TAL approaches on THUMOS14 and ActivityNet-1.3.Source: §5 · p. 14
  • The framework can be efficiently trained on long, high-resolution videos for temporal action localization.Source: §5 · p. 14
  • The authors argue that rising demands for higher-resolution analysis and larger models sustain GPU-memory challenges, and plan to extend the framework to multimodal visual-language settings.They also report combining TALLFormer with existing memory-saving techniques in their experiments.Source: §5 · p. 14

A Implementation Details

The implementation samples clips for short-term encoding and uses long-term memory for the remaining clips. A Temporal Consistency Module refines the combined features before they enter temporal boundary localization.

A Implementation Details

  • The implementation details cover the Long Memory Module and Temporal Consistency Module, with Algorithm 1 outlining short-term feature extraction and memory sampling.Source: A Implementation Details · pp. 18–19

A.1 Long Memory Module

A.2 Temporal Boundary-Localization Module

The Temporal Boundary-Localization Module maps refined features to action boundaries and categories, using different detection designs for THUMOS14 and ActivityNet-1.3.

A.2 Temporal Boundary-Localization Module

  • Given refined features, the Temporal Boundary-Localization Module produces action boundaries and categories, with distinct designs for THUMOS14 and ActivityNet-1.3.Source: §A.2 · p. 20
  • For THUMOS14, the module uses a Feature-Pyramid Network and a DaoTAD Detection Head, with Focal loss for classification and DIoU loss for regression.Source: §A.2 · p. 20
  • For ActivityNet-1.3, the design retains the Long Memory Module, Temporal Consistency Module, and FPN, while adopting an AFSD Detection Head and video-level classifier.The ActivityNet classifier assigns action categories to proposals; the figure caption confirms the shared modules and FPN.Source: §A.2 · p. 20, Figure 4 · p. 21

B.1 Importance of Temporal Consistency Module

The feature visualization compares short-term-extractor and long-memory features before and after the Temporal Consistency Module. The authors report greater similarity between the two feature sources after refinement.

B.1 Importance of Temporal Consistency Module

  • The visualization applies PCA and plots the first two components for short-term and long-memory features, each before and after the Temporal Consistency Module.Source: §B.1 · p. 21
  • Short-term and long-memory features are more similar after the Temporal Consistency Module than before it.Source: §B.1 · p. 21
  • The authors interpret the increased similarity as evidence that the Temporal Consistency Module reduces inconsistency between the two feature sources.Source: §B.1 · p. 21

B.2 Ablating Different Short-term Transformer Encoders

TallFormer can use different short-term transformer encoders as its clip-level backbone. Experiments with Swin variants report high average mAP across all tested backbones.

B.2 Ablating Different Short-term Transformer Encoders

  • TallFormer’s flexibility allows any short-term transformer encoder to serve as its clip-level backbone.The experiments demonstrate this generalization with Swin-Tiny, Swin-Small, and Swin-Base.Source: §B.2 · pp. 21–22
  • Swin-Tiny, Swin-Small, and Swin-Base backbones all achieve pretty high average mAP with TallFormer.Source: §B.2 · p. 22

B.3 Ablating Temporal Support

The temporal-support ablation evaluates how support duration affects TallFormer’s average mAP on THUMOS14. Longer temporal supports lead to consistently higher average mAP.

B.3 Ablating Temporal Support

  • Longer temporal supports lead to consistently higher average mAP on THUMOS14.The ablation uses TallFormer with a Swin-T backbone at 112 × 112 spatial resolution.Source: §B.3 · p. 22, Table 7 · p. 22
  • The ablation measures temporal support in seconds, motivated by actions that can last 30 seconds.Source: §B.3 · p. 22
Loading 2204.01680v2…