PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-end Temporal Action Detection with Transformer

Xiaolong Liu, Qimeng Wang, Yao Hu, Xu Tang, Shiwei Zhang, Song Bai, Xiang Bai

arXiv:2106.10271v4cs.CV

TL;DR

Temporal action detection requires locating and labeling every action in untrimmed videos, but prior systems rely on complex, hand-designed pipelines. TadTR formulates detection as end-to-end set prediction with action queries and locality-aware Transformer components, achieving strong benchmark performance with lower computation cost. It reports 56.7% mAP on THUMOS14, 32.09% mAP on HACS Segments, and 36.75% mAP on ActivityNet-1.3 with an extra classifier.

  • Problem

    Prior temporal action detectors use complex pipelines, hand-crafted components, and sometimes separate classifiers, which restrict flexibility and prevent end-to-end learning.

  • Method

    TadTR maps learnable action queries to action instances using an encoder-decoder Transformer with temporal deformable attention and locality-aware refinement mechanisms.

  • Results

    TadTR achieves state-of-the-art or competitive performance on HACS Segments, THUMOS14, and ActivityNet-1.3 with lower computation costs; it reaches 36.75% mAP on ActivityNet-1.3 with an extra classifier.

  • Takeaways & Limitations

    The framework simplifies temporal action detection by removing hand-crafted components while retaining strong performance and efficiency.

  • Takeaways & Limitations

    TadTR may perform worse on short actions and can miss actions when a video contains more true actions than its number of queries.

Abstract

from arXiv · show

Temporal action detection (TAD) aims to determine the semantic label and the temporal interval of every action instance in an untrimmed video. It is a fundamental and challenging task in video understanding. Previous methods tackle this task with complicated pipelines. They often need to train multiple networks and involve hand-designed operations, such as non-maximal suppression and anchor generation, which limit the flexibility and prevent end-to-end learning. In this paper, we propose an end-to-end Transformer-based method for TAD, termed TadTR. Given a small set of learnable embeddings called action queries, TadTR adaptively extracts temporal context information from the video for each query and directly predicts action instances with the context. To adapt Transformer to TAD, we propose three improvements to enhance its locality awareness. The core is a temporal deformable attention module that selectively attends to a sparse set of key snippets in a video. A segment refinement mechanism and an actionness regression head are designed to refine the boundaries and confidence of the predicted instances, respectively. With such a simple pipeline, TadTR requires lower computation cost than previous detectors, while preserving remarkable performance. As a self-contained detector, it achieves state-of-the-art performance on THUMOS14 (56.7% mAP) and HACS Segments (32.09% mAP). Combined with an extra action classifier, it obtains 36.75% mAP on ActivityNet-1.3. Code is available at https://github.com/xlliu7/TadTR.

I. INTRODUCTION

TadTR addresses the complexity and limited end-to-end learning of prior temporal action detection pipelines with a simple Transformer-based set prediction framework. It enhances locality awareness through temporal deformable attention and related refinements while achieving strong performance at lower computation cost.

  • Temporal action detection identifies each action’s label, start time, and end time in an untrimmed video, supporting applications such as surveillance, home care, and video editing.
  • Prior TAD methods use complex pipelines with hand-designed anchors, non-maximum suppression, grouping, and sometimes standalone classifiers, restricting flexibility and blocking end-to-end learning.
  • TadTR directly maps learnable action queries to action instances in parallel through an encoder-decoder Transformer that models snippet, action-snippet, and inter-action relations.
  • Temporal deformable attention selectively attends to sparse, dynamically learned key snippets around reference locations, extracting adaptive context while preserving temporal locality awareness.
  • Segment refinement iteratively improves predicted boundaries, while an actionness head estimates action-ground-truth IoU using local features for more reliable detection ranking than classification scores alone.

II. RELATED WORK

Prior temporal action detection methods use complex, hand-designed pipelines, while Transformer-based approaches provide a basis for modeling video context. TadTR adapts set prediction and Transformer architectures toward simpler, end-to-end TAD.

  • Temporal Action Detection: Top-down methods generate candidate segments or anchors, while bottom-up methods classify frames and group them into segments.These categories include multi-stage, two-stage, one-stage, and bottom-up pipelines.
  • DETR and Deformable DETR: On THUMOS14, TadTR is reported to achieve state-of-the-art performance while running significantly faster than recent methods.The comparison considers average mAP and speed.
  • Transformers and Context in Video Understanding: Transformers model long-range context and dynamically adjust attention weights according to the input, supporting video-context modeling.Prior video applications include joint video-text representation, video classification, captioning, and spatio-temporal action detection.
  • Temporal Action Detection: Existing TAD pipelines commonly rely on NMS, grouping, anchor settings, or standalone classifiers, restricting flexibility and end-to-end learning.These hand-designed operations also block gradient flow.
  • DETR and Deformable DETR: TadTR uses DETR-style set prediction with learnable action queries and adapts deformable attention for direct temporal action prediction.The architecture decodes action predictions in parallel from video features and action queries.

III. TADTR

TadTR is an encoder-decoder Transformer detector that maps video features and learnable action queries to temporal segments, confidence scores, and semantic labels.

  • III. TADTR: TadTR takes CNN-extracted video features and learnable action queries as input, then outputs action predictions represented by segments, confidence scores, and labels.Its Transformer encoder models snippet interactions, while the decoder predicts action segments.

A. Architecture

TadTR enhances Transformer locality awareness through temporal deformable attention, iterative segment refinement, and actionness regression. Its decoder uses query-specific context to predict and refine action instances.

  • Architecture: The encoder models interactions between video snippets and produces temporally contextualized features.It uses temporal deformable attention and feed-forward networks.
  • Architecture: TadTR replaces dense attention with sparse temporal deformable attention because temporal redundancy and ambiguous boundaries require stronger locality awareness.Dense attention also has higher computation cost and slower convergence.
  • Architecture: Temporal deformable attention samples a sparse set of key elements using query-dependent attention weights and temporal offsets.Sampled features are linearly interpolated when offsets fall between sequence positions.
  • Architecture: The decoder combines query self-attention, temporal deformable cross-attention, and feed-forward networks to model query, snippet, and action relations.Learnable reference points provide initial action-center estimates and are shared across input videos.
  • Segment Refinement: Segment refinement iteratively adjusts predicted boundaries and updates cross-attention reference points using preceding decoder-layer segments.These strategies align attention more closely with local action features.
  • Actionness Regression: Actionness regression extracts features aligned to each predicted interval and estimates localization quality for more reliable ranking.Temporal RoIAlign is applied to encoder features, with context slightly expanded around segment boundaries.

B. Training and Inference

TadTR trains with one-to-one set matching and infers from the final decoder layer, avoiding duplicate predictions and NMS while using sparse action queries.

  • Training: Bipartite matching assigns each ground-truth action to one detection by minimizing a cost based on classification probabilities and segment distance.The matching problem is solved with the Hungarian algorithm using classification, L1, and IoU terms.
  • Training: Set-based matching avoids duplicate predictions, eliminates NMS, and enables end-to-end training with sparse queries instead of dense candidate segments.The paper gives 10 queries as an example, versus tens of thousands for BMN and G-TAD.
  • Inference: During inference, TadTR keeps only the final decoder layer and computes each detection confidence as the predicted class probability multiplied by actionness.The final-layer action predictions are used for detection.

A. Experimental Setup

Experiments evaluate TadTR on THUMOS14, HACS Segments, and ActivityNet-1.3 using established video features and mAP across IoU thresholds. Training uses AdamW with dataset-specific schedules and an improved setting involving focal loss and instance filtering.

  • Datasets and Evaluation Metrics: Experiments use THUMOS14, HACS Segments, and ActivityNet-1.3 to evaluate TadTR.THUMOS14 contains 20 sports action classes; ActivityNet-1.3 and HACS Segments share 200 daily-activity classes.
  • Datasets and Evaluation Metrics: mAP is reported at dataset-specific IoU thresholds, with average mAP computed over the prescribed threshold ranges.THUMOS14 uses 0.3–0.7; the other datasets use 0.5, 0.75, 0.95 and 0.5–0.95 averages.
  • Video Feature Extraction: TadTR takes offline video features as input, including official I3D features on HACS Segments and commonly used features on the other datasets.The HACS features are extracted with I3D trained on Kinetics at 2FPS.
  • Implementation Details: The default implementation uses 2 encoder layers, 4 decoder layers, 8 attention heads, and 4 sampling points.The implementation also sets loss weights λiou=2, λcoord=5, and λact=5.
  • Implementation Details: Training uses AdamW with an initial learning rate of 2 × 10^-4, dataset-specific epoch schedules, and batch size 16.The learning rate is reduced by 0.1 after the specified milestones, and projection-layer learning rates are multiplied by 0.1.
  • Implementation Details: An improved training setting uses focal loss for classification and integrity-based instance filtering on THUMOS14, reducing the training schedules.The resulting schedules are 16, 12, and 20 epochs on THUMOS14, ActivityNet-1.3, and HACS Segments.

B. Main Results

TadTR delivers strong detection accuracy and low computational cost across three benchmarks, including state-of-the-art self-contained results on THUMOS14 and HACS Segments. It can also be combined with external classifiers or BMN, while its efficiency comparison excludes frame-input methods.

  • THUMOS14: TadTR* achieves the best mAP at all reported IoU thresholds on THUMOS14 among the compared methods.The * variant uses focal loss and integrity-based instance filtering; TadTR without cross-window fusion also reaches state-of-the-art performance.
  • THUMOS14: TadTR is 8× faster than the competitive single-network detector A2Net and requires fewer FLOPs.The reported THUMOS14 runtime is 155 ms per video.
  • Self-contained Detection: A single unified TadTR network performs action detection without the extra classifier required by several proposal-generation methods.This distinguishes TadTR from methods whose proposal generation and classification networks are independently trained.
  • HACS Segments: 30.83% average mAP on HACS Segments exceeds SSN by 11.86% and G-TAD by 3.35%.The improved training setting reaches 32.09% mAP.
  • HACS Segments: 455× fewer GFLOPs and 19 ms per video make TadTR substantially cheaper and faster than G-TAD on HACS Segments.G-TAD takes 941 ms per video including post-processing, or 49.5× TadTR’s runtime.
  • ActivityNet-1.3: 28.21% average mAP on ActivityNet-1.3 is stronger than all other compared methods, while the improved setting adds 1.69% mAP.The improved variant is 2.56% above PCG-TAL and 9.68% above TAL-Net.
  • ActivityNet-1.3: 34.64% average mAP results when TadTR detections are fused with an ensemble of action classifiers.This combination is better than the other compared methods in average mAP.
  • Combined Systems: 34.55% mAP is achieved by TadTR+BMN, a 0.7% improvement over BMN.The configuration connects TadTR’s encoder to BMN’s detection head.

C. Ablation Study

Ablations show that adaptive context, temporal deformable attention, actionness regression, and segment refinement materially improve TadTR’s detection quality. Moderate architectural settings provide favorable accuracy–cost trade-offs, while deeper or denser alternatives can hurt performance or efficiency.

  • Context Information: Removing the Transformer encoder drops average mAP by 3.89% on HACS Segments, 6.93% on THUMOS14, and 0.87% on ActivityNet.Removing instance-level context also reduces average mAP by 1.13%, 2.66%, and 1.98% on those datasets.
  • Encoder Comparison: Replacing the Transformer encoder with a 1D CNN lowers average mAP by 2.88% without NMS and 1.41% with NMS.Deeper CNNs and larger kernels provide no improvement and use much higher FLOPs.
  • Temporal Attention: Dense attention reduces THUMOS14 average mAP by 23.77% and increases computation cost 5.7× versus temporal deformable attention.The paper attributes this to insufficient locality awareness and over-smoothing of similar background frames.
  • Component Ablations: Actionness regression improves HACS Segments mAP by 3.06%, 2.26%, 1.55%, and 2.32% at IoU 0.5, 0.75, 0.95, and average mAP.Average mAP also improves by 2.83% on THUMOS14 and 2.08% on ActivityNet.
  • Component Ablations: Segment refinement improves HACS Segments mAP at IoU 0.95 by 2.24%, with average-mAP gains of 3.85% on THUMOS14 and 0.81% on ActivityNet.Other HACS thresholds also improve consistently.
  • Segment Refinement: Incremental refinement and reference-point adjustment contribute 1.31% and 1.85% mAP, respectively, on the evaluated ablations.Removing either component decreases performance.
  • Efficiency Ablations: Adding actionness regression increases THUMOS14 runtime from 130 ms to 141 ms, while segment refinement increases it to 155 ms.The two components have little impact on FLOPs.
  • Hyper-parameters: The best action-query counts are 40 on THUMOS14, 30 on HACS Segments, and 10 on ActivityNet-1.3.These settings correspond to average action-instance counts of 15.4, 2.8, and 1.5 per video, respectively.

D. Analysis

Visualizations show that TadTR’s attention and queries capture structured temporal and instance-level context, while its main weaknesses concern short actions and videos containing more actions than available queries.

  • Attention visualization: Different temporal attention heads focus on distinct regions and scales, with decoder sampling often spanning most of an action prediction.Encoder sampling covers a shorter temporal extent, while decoder sampling provides a larger receptive field.
  • Attention visualization: Query self-attention concentrates on semantically related predicted instances, indicating instance-level context among action queries.The topmost query in each example attends most to queries with semantically related predictions.
  • Action-query visualization: Action queries learn to specialize in particular prediction locations and temporal lengths, providing a more flexible alternative to hand-crafted anchors.Different locations and scales are covered by a small number of queries.
  • Limitations: TadTR may fail on some short actions, where MUSES partially detects two examples and quantitative results also show lower short-action performance.The paper suggests that Transformer may be inferior to 1D CNNs for short-term dependency modeling.
  • Limitations: TadTR can miss actions when a video contains more true actions than its Nq queries, although the paper describes such cases as potentially rare.Increasing Nq while maintaining performance is identified as future work.

V. CONCLUSION

The paper concludes that TadTR is a simple, end-to-end Transformer detector that treats temporal action detection as direct set prediction. It reports competitive performance and lower computation costs across three datasets, while identifying joint encoder learning and purely Transformer-based detectors as future directions.

  • V. CONCLUSION: TadTR maps learnable embeddings to action instances in parallel by adaptively extracting temporal context from video.The method removes hand-crafted anchor setting and post-processing from the TAD pipeline.
  • V. CONCLUSION: TadTR uses three improvements to enhance Transformer locality awareness for temporal action detection.The conclusion states that extensive experiments validate the method’s components, performance, and efficiency.
  • V. CONCLUSION: TadTR achieves state-of-the-art or competitive performance on HACS Segments, THUMOS14, and ActivityNet-1.3 with lower computation costs.
  • V. CONCLUSION: The current implementation uses offline extracted CNN features, with future work targeting joint video-encoder learning and purely Transformer-based temporal action detectors.

APPENDIX

The appendix presents supplementary visualizations of dense attention and temporal deformable attention to illustrate their behavior in additional examples.

  • APPENDIX: The appendix supplements the main-document visualizations with dense-attention smoothing results and additional temporal deformable-attention examples.
Loading 2106.10271v4…