PaperlayerView exact revision on arXiv

Source-linked AI summary

Relaxed Transformer Decoders for Direct Action Proposal Generation

Jing Tan, Jiaqi Tang, Limin Wang, Gangshan Wu

arXiv:2102.01894v3cs.CV

TL;DR

Temporal proposal generation is constrained by hand-crafted anchors or boundary matching, while video features are slow-changing and boundaries ambiguous. RTD-Net adapts Transformer-based direct set prediction with boundary-attentive features, relaxed matching, and completeness-aware decoding, outperforming prior methods on THUMOS14 and matching them on ActivityNet-1.3 across proposal generation and action detection.

  • Problem

    Existing temporal proposal methods rely on pre-defined anchors or heuristic boundary matching, while slow-changing video features and ambiguous boundaries complicate direct Transformer-based generation.

  • Method

    RTD-Net directly generates proposal sets using boundary-attentive representations, a relaxed Transformer decoder and matcher, and a three-branch head that estimates completeness.

  • Results

    RTD-Net outperforms state-of-the-art methods on THUMOS14 and achieves comparable performance on ActivityNet-1.3 for both proposal generation and action detection.

  • Takeaways & Limitations

    Parallel global-context decoding removes hand-crafted proposal designs and NMS post-processing while providing a more efficient proposal generation pipeline.

Abstract

from arXiv · show

Temporal action proposal generation is an important and challenging task in video understanding, which aims at detecting all temporal segments containing action instances of interest. The existing proposal generation approaches are generally based on pre-defined anchor windows or heuristic bottom-up boundary matching strategies. This paper presents a simple and efficient framework (RTD-Net) for direct action proposal generation, by re-purposing a Transformer-alike architecture. To tackle the essential visual difference between time and space, we make three important improvements over the original transformer detection framework (DETR). First, to deal with slowness prior in videos, we replace the original Transformer encoder with a boundary attentive module to better capture long-range temporal information. Second, due to the ambiguous temporal boundary and relatively sparse annotations, we present a relaxed matching scheme to relieve the strict criteria of single assignment to each groundtruth. Finally, we devise a three-branch head to further improve the proposal confidence estimation by explicitly predicting its completeness. Extensive experiments on THUMOS14 and ActivityNet-1.3 benchmarks demonstrate the effectiveness of RTD-Net, on both tasks of temporal action proposal generation and temporal action detection. Moreover, due to its simplicity in design, our framework is more efficient than previous proposal generation methods, without non-maximum suppression post-processing. The code and models are made available at https://github.com/MCG-NJU/RTD-Action.

1. Introduction

Temporal action proposal generation must localize action segments in untrimmed videos, but existing anchor-based and boundary-based methods depend on hand-crafted designs. RTD-Net directly generates proposals with Transformer-based global temporal modeling and three adaptations for video, achieving strong benchmark performance.

  • Temporal action detection localizes action instances and durations in untrimmed videos, combining proposal generation with action classification.
  • Anchor-based methods struggle to cover action instances with highly variable durations under reasonable computation, while boundary-based methods rely on local context and can produce incomplete proposals.
  • RTD-Net uses global temporal context and parallel decoding to model inter-proposal relationships and streamline proposal generation without hand-crafted matching, anchor placement, or NMS.
  • The framework addresses video-specific feature slowness and ambiguous boundaries with boundary-attentive representations, relaxed matching, and a three-branch head with completeness estimation.
  • RTD-Net outperforms existing state-of-the-art methods on THUMOS14 and achieves comparable performance on ActivityNet-1.3 for proposal generation and action detection.

2. Related Work

Prior work uses anchors, local boundary grouping, or structured long-range context for temporal proposals. RTD-Net instead uses decoder-only Transformer processing to exploit global temporal context and generate proposals directly.

  • Action recognition extracts semantic labels or snippet-level features, while temporal action proposal generation seeks flexible and precise segments in untrimmed videos.
  • Anchor-based methods use dense, multi-scale anchors but are inflexible and may not cover all action instances; boundary-based methods pair predicted endpoints using local information and are noise-sensitive.
  • Graph-based methods model long-range context through pre-defined graph structures whose construction depends heavily on human design.
  • Transformers model long-range sequence dependencies through self-attention, and RTD-Net applies a decoder-only Transformer to generate proposals directly from global temporal context.

3. Method

RTD-Net directly generates temporal action proposals with a Transformer-alike pipeline adapted to slow features, ambiguous boundaries, sparse annotations, and proposal completeness.

  • Three-Branch Detection Head: The three-branch head predicts proposal boundaries, foreground confidence, and completeness to incorporate temporal localization quality into proposal confidence.The completeness branch predicts overlap with targets using a tIoU-based training loss.
  • Feature Encoding: RTD-Net extracts appearance and motion features with two-stream networks, concatenates them, and arranges them into a temporal feature sequence.Features are extracted using a sliding window with temporal stride τ.
  • Boundary-Attentive Representations: The boundary-attentive module multiplies short-term features by action-starting and action-ending scores to preserve discriminative boundary information.An MLP encoder transforms the enhanced representation into a compact form for decoding.
  • Training and Inference: RTD-Net uses set prediction loss and directly evaluates its proposal set without non-maximum suppression post-processing.Training includes localization, overlap, classification, and completeness components, with a two-step strategy for the completeness head.
  • Relaxed Transformer Decoder: A relaxed matching scheme marks multiple predictions as positive when their tIoU with a ground-truth instance exceeds a threshold.This addresses sparse annotations and ambiguous temporal boundaries; tIoU is intersection over union between prediction and target.

4. Experiments

Experiments evaluate RTD-Net for proposal generation and action detection on THUMOS14 and ActivityNet-1.3, including component ablations and efficiency comparisons. RTD-Net achieves the best reported proposal-generation performance on THUMOS14, comparable results on ActivityNet-1.3, and improved high-tIoU detection performance without NMS.

  • Temporal Action Proposal Generation: RTD-Net achieves the best proposal-generation performance on THUMOS14 and comparable results on ActivityNet-1.3.The comparison uses AR@AN and AUC; RTD-Net is free of NMS.
  • Temporal Action Proposal Generation: RTD-Net predicts one complete proposal covering an action instance, whereas BSN produces two incomplete predictions in the qualitative example.The paper attributes this contrast to RTD-Net’s global context modeling versus bottom-up methods’ local context.
  • Action Detection with RTD Proposals: The direct proposal-generation pipeline avoids time-consuming non-maximum suppression post-processing.Inference speed was evaluated on one RTX 2080Ti GPU.
  • Ablation Study: Boundary-attentive representation improves average recall by up to 4% on THUMOS14, with scaling factor αr = 2 producing the largest improvement.The scaling factor sets the threshold used to enhance or suppress features according to boundary probability.
  • Ablation Study: MLP encoders outperform Transformer encoders on THUMOS14, while smaller-receptive-field Transformer features improve performance to around 36%@50 but remain worse than MLP.The analysis associates the Transformer drop with self-attention over-smoothing.
  • Ablation Study: Relaxed matching improves AR and AUC by converting some high-quality proposals from negative to positive training samples.Training first uses strict bipartite matching, followed by fine-tuning with relaxed matching.
  • Action Detection with RTD Proposals: RTD-Net improves temporal action detection especially at high tIoU settings, and proposals combined with P-GCN improve mAP over other proposal-generation methods at all thresholds.RTD-Net makes 100 predictions, whereas BSN and BMN generate nearly 900 proposals per video before selecting 100.

5. Conclusion

The paper presents RTD-Net, a direct action proposal-generation pipeline that adapts Transformer-like decoding to video through boundary-aware representations, relaxed matching, and a three-branch head. It outperforms prior state-of-the-art methods on THUMOS14, performs comparably on ActivityNet-1.3, and avoids NMS post-processing.

  • 5. Conclusion: RTD-Net adapts a Transformer-like architecture for direct action proposal generation with boundary-attentive representation, relaxed matching, and a three-branch prediction head.Parallel decoding provides explicit context modeling for multiple proposals.
  • 5. Conclusion: RTD-Net outperforms previous state-of-the-art methods for proposal generation and action detection on THUMOS14.The conclusion attributes the result to parallel decoding of multiple proposals with explicit context modeling.
  • 5. Conclusion: RTD-Net achieves comparable performance on ActivityNet-1.3 while remaining free of NMS post-processing.The conclusion characterizes the detection pipeline as more efficient than previous methods.

A.1. Boundary Attentive Module

The boundary-attentive module is evaluated through projection placement, positional embedding, and receptive-field ablations, with results linking encoder behavior to over-smoothing and temporal information.

  • A.1. Boundary Attentive Module: Pre-enhancement projection outperforms post-enhancement and channel-concatenated boundary weighting on THUMOS14.The comparison uses AR@AN.
  • A.1. Boundary Attentive Module: Concatenating positional embeddings with video features explicitly encodes relative temporal order for the MLP encoder.The ablation evaluates removing positional embeddings or adding them directly into the encoder.
  • A.1. Boundary Attentive Module: Transformer-encoder performance improves with smaller receptive fields because reduced context alleviates self-attention over-smoothing.This contrasts with the MLP encoder, whose performance decreases with smaller receptive fields.

A.2. Relaxed Transformer Decoder

The relaxed decoder uses a two-step top-1-to-top-k matching strategy to improve proposal recall while preserving sparse predictions that avoid NMS.

  • A.2. Relaxed Transformer Decoder: The two-step scheme first uses strict bipartite matching, then fine-tunes with relaxed matching to improve overall recall.The first stage produces sparsely distributed, minor-overlapped positive samples.
  • A.2. Relaxed Transformer Decoder: During fine-tuning, tIoU-based positive assignment updates classification and localization losses while other modules remain frozen.The passage describes thresholding predictions by tIoU and updating the matching permutation.
  • A.2. Relaxed Transformer Decoder: Relaxed matching improves AR and AUC by increasing the confidence of top-k proposals around groundtruth instances.Top-1 proposal performance remains strong while additional nearby proposals receive improved scores.
  • A.2. Relaxed Transformer Decoder: RTD-Net reverses the comparison method’s top-k-to-top-1 order, with both strategies removing heuristic NMS through label-assignment optimization.The comparison method freezes its detector and trains a PSS head during fine-tuning.

A.3. Non-Maximum Suppression

RTD-Net does not require non-maximum suppression because its two-step training produces sparse, minimally overlapping predictions.

  • A.3. Non-Maximum Suppression: RTD-Net achieves similar results with and without NMS on THUMOS14.Its predictions are relatively sparse and minor-overlapped under the two-step training strategy.
  • A.3. Non-Maximum Suppression: Dense proposal generators such as BSN and BMN require NMS because they produce highly overlapped proposals with similar confidence.NMS suppresses these redundant proposals.

B. Visualization

Visualizations examine how boundary attention modifies slow video features and how decoder queries interact during inter-proposal modeling.

  • B. Visualization: Input features show vertical patterns across temporal locations, while boundary attention introduces horizontal darker patterns after starting and ending scores are applied.The visualization interprets the input pattern as feature slowness and the altered pattern as boundary-information enhancement.
  • B. Visualization: Transformer self-attention exhibits diagonal activation with short vertical patterns, indicating over-smoothing across temporal locations.The visualization uses input temporal locations on the x-axis and output temporal locations on the y-axis.
  • B. Visualization: Ending-region activations indicate that proposal queries focus on features near action-instance endings.The passage connects these activations to proposal-query focus.
  • B. Visualization: Decoder query self-attention has high activations along the y-axis, showing that queries learn from selected well-predicted queries during inter-proposal modeling.The 14th query is both the highest-ranked and a well-predicted proposal in the example.

C. Additional Comparisons with SOTA

RTD-Net produces more precise and complete proposals than BSN across tIoU thresholds while using fewer predictions and avoiding post-processing. Its proposals remain competitive on unseen action categories, and its decoder models proposal relationships globally.

  • RTD-Net outperforms BSN under every tIoU threshold, especially with fewer proposals, and maintains higher recall at high tIoU thresholds.High-tIoU performance indicates stronger localization under stricter true-positive criteria.
  • RTD-Net predicts 640 proposals for a 3-minute video without post-processing, whereas BSN outputs about 3k predictions requiring SNMS.RTD-Net uses 1,519 MB GPU memory versus 7,152 MB for BMN.
  • RTD-Net predicts fewer proposals than BSN but achieves higher average recall across all metrics on THUMOS14.Pair-wise decoder modeling reduces redundant, highly overlapping proposals.
  • RTD-Net remains competitive on unseen ActivityNet-1.3 categories and predicts accurate proposals regardless of semantics.The evaluation separates seen and unseen action subsets.
  • The encoder-decoder attention map displays action queries against encoder time steps, with blue indicating stronger activation and orange marking action intervals.The decoder self-attention visualization is averaged across multiple heads.

D. Performance on HACS Segments

On HACS Segments, RTD-Net is evaluated with AR@AN and AUC and is the only compared method without NMS. It achieves strong performance at small proposal counts, while BSN leads at AR@100 and matches its AUC.

  • RTD-Net is the only method in the HACS Segments comparison that is free of NMS.The comparison includes TAG+NMS, BSN+SNMS, and RTD-Net.
  • RTD-Net achieves AR@1 (val) of 16.34, while BSN achieves higher AR@100 (val) of 63.62 versus 61.11.At AR@100, TAG+NMS records 55.88.
  • BSN and RTD-Net achieve the same AUC (val) of 53.41, while TAG+NMS reaches 49.15.AUC and AR@AN are the reported HACS Segments metrics.
  • The comparison demonstrates that RTD-Net achieves higher average recall at small average numbers of proposals, such as AR@1.BSN evaluates its top-100 proposals, whereas RTD-Net directly generates fewer proposals.
  • Bottom-up methods first estimate boundary confidence and match starting and ending points, with 2D CNN features preserving local information better than I3D features.The passage connects feature choice with boundary recall and bottom-up performance.
Loading 2102.01894v3…