PaperlayerView exact revision on arXiv

Source-linked AI summary

ActionFormer: Localizing Moments of Actions with Transformers

Chenlin Zhang, Jianxin Wu, Yin Li

arXiv:2202.07925v2cs.CV

TL;DR

Temporal action localization is challenging, while prior methods often rely on complex proposals or anchor designs. ActionFormer applies local self-attention and multiscale features in a single-stage anchor-free model that classifies moments and regresses boundaries, achieving strong results across major benchmarks, though its boundary regression remains less accurate in visualization.

  • Problem

    Temporal action localization remains challenging, and existing methods often achieve accuracy through increasingly complex proposals, anchor designs, losses, architectures, and decoding processes.

  • Method

    ActionFormer combines local self-attention with multiscale feature representations and a lightweight decoder to classify every moment and regress action boundaries without proposals or predefined anchor windows.

  • Results

    ActionFormer achieves state-of-the-art results across major TAL benchmarks, including 71.0% mAP at tIoU=0.5 on THUMOS14 and strong performance on ActivityNet 1.3 and EPIC-Kitchens 100.

  • Takeaways & Limitations

    The combination of local self-attention and multiscale feature representation provides the reported power of this simple anchor-free TAL design.

  • Takeaways & Limitations

    Visualized outputs show that action-boundary regression is less accurate, and the authors conjecture that the regression heads can be further improved.

Abstract

from arXiv · show

Self-attention based Transformer models have demonstrated impressive results for image classification and object detection, and more recently for video understanding. Inspired by this success, we investigate the application of Transformer networks for temporal action localization in videos. To this end, we present ActionFormer -- a simple yet powerful model to identify actions in time and recognize their categories in a single shot, without using action proposals or relying on pre-defined anchor windows. ActionFormer combines a multiscale feature representation with local self-attention, and uses a light-weighted decoder to classify every moment in time and estimate the corresponding action boundaries. We show that this orchestrated design results in major improvements upon prior works. Without bells and whistles, ActionFormer achieves 71.0% mAP at tIoU=0.5 on THUMOS14, outperforming the best prior model by 14.1 absolute percentage points. Further, ActionFormer demonstrates strong results on ActivityNet 1.3 (36.6% average mAP) and EPIC-Kitchens 100 (+13.5% average mAP over prior works). Our code is available at http://github.com/happyharrycn/actionformer_release.

1 Introduction

Temporal action localization remains challenging, and prior accuracy often requires increasingly complex proposals, anchors, losses, architectures, and decoding. ActionFormer addresses this with a minimalist Transformer-based, single-stage anchor-free design that classifies each moment, regresses action boundaries, and achieves strong benchmark results.

  • Temporal action localization identifies action instances in time and recognizes their categories, but remains challenging in video understanding.
  • Prior TAL methods commonly use action proposals or anchor windows, with accuracy often accompanied by increasing modeling complexity.The complexity includes proposal generation, anchor design, loss functions, network architectures, and output decoding.
  • ActionFormer uses local self-attention to model temporal context, classifies every moment, and regresses corresponding action boundaries.It is trained with standard classification and regression losses.
  • ActionFormer is a single-stage anchor-free model that decodes moment-level action candidates into labels and temporal boundaries.Its feature pyramid locations serve as action candidates, while a lightweight convolutional decoder performs classification and boundary regression.
  • 71.0% mAP at tIoU=0.5 on THUMOS14 surpassed the best prior model by 14.1 absolute percentage points.The model also reached 36.6% average mAP on ActivityNet 1.3 and boosted average mAP by over 13.5 absolute percentage points on EPIC-Kitchens 100.
  • The paper presents ActionFormer as a simple Transformer-based TAL model supported by favorable empirical results, extensive ablations, and state-of-the-art benchmark performance.The authors also position it as a solid baseline for TAL.

2 Related Works

Related TAL work includes two-stage proposal-based systems and single-stage approaches using anchors, convolutional, recurrent, or graph models. ActionFormer differs by applying a Transformer to single-stage anchor-free TAL, while spatial-temporal localization addresses a different task.

  • Two-stage TAL: Two-stage TAL methods generate action proposals before classifying them and refining their temporal boundaries.Proposal generation has used anchor windows, boundary detection, graph representations, and Transformers.
  • Single-stage TAL: Single-stage TAL seeks to localize actions in one shot, although many prior methods remain anchor-based.Prior examples use convolutional networks, recurrent memory, or Gaussian kernels to optimize anchor scale.
  • Single-stage TAL: ActionFormer formulates TAL as sequence labeling by classifying every moment and regressing action boundaries in a single-stage anchor-free model.Its key distinction from earlier formulations is the use of a Transformer network for action localization.
  • Spatial-temporal Action Localization: Spatial-temporal action localization detects actions both temporally and spatially as moving actor bounding boxes, unlike TAL.ActionFormer processes video-frame sequences, whereas the cited related Transformer method uses 2D object proposals.
  • Object Detection: ActionFormer’s multiscale feature representation and convolutional decoder are inspired by feature pyramid networks and RetinaNet.Its center-sampling training strategy also draws from single-stage object detectors.
  • Vision Transformer: Vision Transformers originated in NLP and subsequently achieved strong results across vision tasks, motivating Transformer-based video models.The cited progression includes ViT, DeiT, T2T-ViT, Swin Transformer, Focal Transformer, and PVT.

3 ActionFormer: A Simple Transformer Model for Temporal Action Localization

ActionFormer formulates temporal action localization as anchor-free sequence labeling: every time step is classified and assigned distances to action boundaries. A Transformer encoder builds a multiscale feature pyramid with local self-attention, while a lightweight convolutional decoder produces classification and regression outputs.

  • Action representation: Each video is represented as feature vectors over discretized time steps, and the model predicts action instances with labels, starts, and ends.The anchor-free formulation converts structured action-instance prediction into sequence labeling by classifying moments and regressing boundary distances.
  • Action representation: Every time step is treated as an action candidate, classified into an action category or background, and assigned onset and offset distances when foreground.The boundary distances are undefined for background time steps.
  • Transformer encoder: The encoder embeds clip features and maps them through Transformer blocks into a multiscale feature pyramid with varying temporal resolutions.The embedding begins with a convolutional projection, after which the Transformer produces the pyramid; optional downsampling supports multiple temporal scales.
  • Transformer encoder: Local self-attention restricts attention to a temporal window, reducing the cost of vanilla self-attention for long videos while modeling nearby temporal context.Vanilla self-attention has O(T^2D + D^2T) memory and time complexity, motivating the local-window design.
  • Decoder and design: A lightweight convolutional decoder uses classification and regression heads to decode the feature pyramid into sequence labels at each time step.The model’s feature-pyramid design uses 2x downsampling between levels and expands regression ranges approximately by 2 accordingly.

4 Experiments and Results

Experiments show that ActionFormer achieves strong temporal action localization results across major benchmarks, while ablations identify multiscale feature pyramids as critical and local attention as an efficient design choice. Visualizations also reveal remaining boundary-regression errors.

  • Results on THUMOS14: 71.0% mAP at tIoU=0.5 on THUMOS14, with 66.8% average mAP across [0.3:0.1:0.7].The model also reaches 43.9% mAP at tIoU=0.7 and exceeds previous methods by +14.1% and +12.8% mAP at those thresholds.
  • Results on ActivityNet-1.3: 36.6% average mAP on ActivityNet 1.3 with TSP pre-training, outperforming the best method using the same features by +0.7%.With I3D features alone, ActionFormer reaches 35.6% average mAP and exceeds previous methods using the same features by at least 0.6%.
  • Results on EPIC-Kitchens 100: 23.5% and 21.9% average mAP for verb and noun localization on EPIC-Kitchens 100, exceeding BMN and G-TAD by over 13.5 absolute percentage points.All compared methods use the same SlowFast features; the dataset contains many more action instances per video than ActivityNet-1.3.
  • Results on EPIC-Kitchens 100: 21.36% / 20.95% average mAP on the EPIC-Kitchens 100 2022 Action Detection challenge validation / test sets ranked ActionFormer second.The gap to the top solution was 0.32 average mAP after combining SlowFast and ViViT features.
  • Ablation Experiments: Reducing the local self-attention window lowers MACs while maintaining similar accuracy, but Transformer models run roughly 2x slower than the convolutional baseline.Global attention increases MACs by 26% versus the baseline; the implementation used PyTorch primitives without customized CUDA kernels.
  • Ablation Experiments: Removing the feature pyramid causes a -19.3% drop in average mAP, while performance generally improves with more pyramid levels and saturates at six levels.For three pyramid levels, the best initial regression range is [0, 4); larger ranges reduce mAP.
  • Ablation Experiments: Visualizations show action scores at each time step and score-weighted onset and offset histograms, while regression of action boundaries appears less accurate.The model produces a strong peak near action centers, potentially due to center sampling during training.

5 Conclusion and Discussion

ActionFormer is a simple single-stage, anchor-free Transformer for temporal action localization. Its combination of local self-attention and multiscale features produces strong results across THUMOS14, ActivityNet-1.3, and EPIC-Kitchens 100.

  • Conclusion and Discussion: ActionFormer combines local self-attention with multiscale feature representation to model longer-range temporal context in videos.The method is presented as a Transformer-based, single-stage anchor-free approach with a simple design.
  • Conclusion and Discussion: The model achieves impressive results across major TAL benchmarks, including egocentric videos in EPIC-Kitchens 100.The authors position ActionFormer as a baseline that may inform temporal action localization and video understanding.

6 Appendix

The appendix provides supplementary experiments, analyses, implementation details, visualizations, and limitations or future directions for ActionFormer.

  • The appendix contains additional ablation experiments in Section A.
  • Section B presents further error analysis of the reported results.
  • Section C documents implementation details and reproduction procedures.
  • Section D adds visualizations, while Section E discusses limitations and future directions.

A Additional Ablation Experiments

Additional ablations examine loss weighting, training sequence length, and temporal feature resolution on THUMOS14, finding generally stable performance within tested settings.

  • Loss Weight: A broad range of λreg values produces stable performance, with a maximum average-mAP gap of 1.4%.λreg = 1 gives the best results and is used in all experiments.
  • The ablations report mAP at tIoU=0.5 and 0.7 alongside average mAP on THUMOS14.The studies vary λreg, maximum training input length, and feature stride.
  • Maximum Input Sequence Length during Training: A 512-step training sequence causes only a minor 1.1% drop in average mAP and still significantly outperforms.
  • Temporal Feature Resolution: A feature stride of 8 causes only a slight 0.5% drop in average mAP, whereas stride 16 causes larger degradation.The results remain favourable even with the larger reduction in temporal resolution.

B Further Error Analyses

Further THUMOS14 analyses show that ActionFormer is most challenged by extreme action durations, crowded videos, and complex contexts, while remaining robust across categories.

  • Results and Analyses: False negatives rise for extra-short or extra-long actions and videos containing many action instances.
  • Results and Analyses: False-negative rates are similar across coverage categories but substantially higher for extreme lengths and large numbers of instances.
  • Results and Analyses: Normalized mAP is higher in simpler contexts and lower in more complicated scenarios across coverage, length, and instance-count categories.
  • Results and Analyses: The model remains robust across coverage, length, and instance-count categories with small variances.
  • Results and Analyses: Localization error and background confusion are the two leading false-positive error types.The profiling considers predictions for the top-10 ground-truth instances and separately reports the impact of error types.

C Implementation Details

The implementation uses a Transformer encoder with a lightweight convolutional decoder, supports varied video inputs, and reports deterministic single-machine results with small cross-environment variation.

  • Training Details: Training supports fixed- and variable-length inputs, capping variable-length sequences at 2304 clips and randomly sampling consecutive subsets.Position embeddings are disabled by default except for ActivityNet.
  • Inference Details: Inference feeds the full sequence to the model for ActivityNet-1.3, THUMOS14, and EPIC-Kitchens 100.
  • Network Architecture: The architecture combines a Transformer encoder with a lightweight convolutional decoder and classification and regression heads.Table D lists layer parameters, inputs, outputs, and regression ranges.
  • Score Fusion: Score fusion optionally multiplies ActionFormer action scores by external top-2 video-level classification scores on THUMOS14 and ActivityNet-1.3.Each detected action produces two fused action instances.
  • Reproducibility of Our Results: Using the same seed and environment reproduces results deterministically, while average mAP varies by up to 0.5% across machines on THUMOS14.

D Additional Visualizations

ActionFormer detects most action occurrences and estimates their temporal boundaries, while failures mainly arise from classification confusion or inaccurate boundary regression.

  • D Additional Visualizations: Most visualized cases show successful action detection and temporal-boundary estimation.The successful examples appear in the first column of Fig. D.
  • D Additional Visualizations: The main failure mode is incorrect classification of action centers, producing background confusion.
  • D Additional Visualizations: Another failure mode is inaccurate regression of action onsets and offsets, causing localization errors.
  • D Additional Visualizations: Fig. D displays input frames, per-time-step action scores, and score-weighted onset and offset histograms.

E Limitations and Future Work

The method relies on pre-extracted video features, human-labeled training videos, and a predefined action vocabulary; proposed future work targets less supervised alternatives.

  • E Limitations and Future Work: A main limitation is reliance on pre-extracted video features.
  • E Limitations and Future Work: Training requires many human-labeled videos, limiting the method’s supervision setting.
  • E Limitations and Future Work: The method is constrained by a predefined vocabulary of actions.
  • E Limitations and Future Work: Future directions include pre-training for action localization and learning from video-text corpora without human labels.
Loading 2202.07925v2…