Source-linked AI summary
TriDet: Temporal Action Detection with Relative Boundary Modeling
Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, Dacheng Tao
TL;DR
Temporal action detection is hindered by ambiguous action boundaries and feature similarity that can cause self-attention rank loss. TriDet uses a Trident-head for relative boundary distributions and an SGP-based feature pyramid, achieving state-of-the-art results across three benchmarks.
Problem
Ambiguous action boundaries and self-attention rank loss limit precise temporal action detection.
Method
TriDet is a one-stage detector using a Trident-head that models relative boundary probability distributions and an SGP layer that captures multiple temporal granularities.
Results
TriDet achieves state-of-the-art performance on THUMOS14, HACS, and EPIC-KITCHEN 100.
Takeaways & Limitations
Relative boundary modeling and the SGP-based feature pyramid provide a convolutional-based framework with high generalization across challenging temporal action detection benchmarks.
Takeaways & Limitations
Self-attention can increase feature similarity through convex combinations and incurs high computational overhead.
Abstract
from arXiv · showhide
In this paper, we present a one-stage framework TriDet for temporal action detection. Existing methods often suffer from imprecise boundary predictions due to the ambiguous action boundaries in videos. To alleviate this problem, we propose a novel Trident-head to model the action boundary via an estimated relative probability distribution around the boundary. In the feature pyramid of TriDet, we propose an efficient Scalable-Granularity Perception (SGP) layer to mitigate the rank loss problem of self-attention that takes place in the video features and aggregate information across different temporal granularities. Benefiting from the Trident-head and the SGP-based feature pyramid, TriDet achieves state-of-the-art performance on three challenging benchmarks: THUMOS14, HACS and EPIC-KITCHEN 100, with lower computational costs, compared to previous methods. For example, TriDet hits an average mAP of $69.3\%$ on THUMOS14, outperforming the previous best by $2.5\%$, but with only $74.6\%$ of its latency. The code is released to https://github.com/sssste/TriDet.
1. Introduction
TriDet addresses ambiguous temporal action boundaries with relative boundary modeling and replaces self-attention in its feature pyramid with an efficient SGP layer. The resulting detector achieves state-of-the-art performance across THUMOS14, HACS, and EPIC-KITCHEN 100.
- Ambiguous action boundaries make temporal action detection challenging because boundary-adjacent instants can receive relatively high classifier responses.
- Existing approaches either use global segment features or regress boundaries from individual instants, leaving adjacent-instant relative probabilities underused.
- TriDet’s Trident-head estimates a relative boundary probability distribution and computes offsets from expected values of neighboring bins.
- The SGP layer addresses self-attention’s rank loss and computational overhead through branches that improve instant-level discrimination and capture multiple temporal receptive-field scales.
- TriDet surpasses previous detectors and achieves state-of-the-art performance across THUMOS14, HACS, and EPIC-KITCHEN 100.
2. Related Work
Related work in temporal action detection includes two-stage proposal-based methods, one-stage methods, and Transformer-based approaches. Object-detection techniques also inform TAD architecture design.
- Temporal action detection: Temporal action detection methods are commonly divided into two-stage proposal-generation/classification systems and one-stage systems.
- Temporal action detection: Many prior methods emphasize proposal generation, including approaches that predict action-boundary probabilities and densely match start and end instants.
- Object detection: Object-detection research contributes relevant ideas, including distributional bounding-box regression, depthwise convolution, and branched architectures.
- Transformer-based methods: Transformer-based TAD methods use attention in decoder or encoder architectures, but many operate primarily on local behavior.
3. Method
TriDet is a one-stage temporal action detector that combines an SGP-based feature pyramid with a Trident-head for relative action-boundary modeling. SGP replaces self-attention to address rank loss and computational overhead, while the Trident-head estimates boundaries from start, end, and center-offset information.
- Overall Architecture: TriDet uses a video backbone, an SGP feature pyramid, and a boundary-oriented Trident-head in its one-stage detection architecture.Backbone features are processed across pyramid levels, whose outputs share a detection head containing classification and boundary components.
- Motivation: Self-attention can increase temporal feature similarity through convex combinations and adds dense pair-wise computational overhead.These effects are described as rank loss and reduced inference speed, respectively.
- Feature Pyramid with SGP Layer: SGP replaces self-attention with a fully convolutional layer and the successive Layer Normalization with Group Normalization.Its instant-level branch increases feature discriminability, while its window-level branch captures temporal information at different scales.
- Feature Pyramid with SGP Layer: The SGP layer combines video-level average features, pointwise fully connected processing, and depth-wise convolutions with scalable temporal windows.The branches interact through element-wise multiplication with the mainstream feature, and the resulting pyramid is reported to be more efficient than a transformer-based pyramid.
- Trident-head with Relative Boundary Modeling: TriDet addresses ambiguous action boundaries by using relations among adjacent temporal features rather than relying only on global segment features or single-instant regression.The approach is designed for boundary localization when boundary-adjacent instants receive similarly high classifier responses.
- Trident-head with Relative Boundary Modeling: The Trident-head models each boundary through relative probabilities across neighboring instants instead of directly regressing offsets from a single instant.Its three components locate start boundaries, end boundaries, and the temporal action center, and the offset is computed from the expected value of the predicted bin distribution.
4. Experiments
TriDet is evaluated on four challenging datasets using mAP at specified IoU thresholds, achieving strong results across benchmarks and demonstrating gains from its components and efficient design.
- Main Results: 69.3% average mAP on THUMOS14, outperforming all previous methods, including recent Transformer-based detectors.The experiments use I3D backbone features and report performance across IoU thresholds from 0.3 to 0.7.
- Main Results: 36.8% average mAP on HACS with official I3D features, exceeding the previous best TadTR by about 4.7%.Replacing I3D with SlowFast further increases average mAP by 1.8%.
- Main Results: 25.4% verb and 23.8% noun average mAP on EPIC-KITCHEN 100, surpassing ActionFormer with identical features by 1.9% on each sub-task.TriDet achieves state-of-the-art performance on this dataset.
- Main Results: TriDet outperforms the second-best ActivityNet model using the same feature, while remaining below a two-stage model using unavailable SlowFast features.The evaluation combines detection scores with video classification scores predicted by UntrimmedNet.
- Ablation Study: 6.2% absolute average-mAP improvement comes from SGP over the implemented baseline, while Trident-head adds 1.0% over instant-level regression.SGP also exceeds ActionFormer by 1.5%, and Trident-head yields a 1.6% improvement at IoU 0.7.
- Ablation Study: TriDet uses lower computation and latency than ActionFormer while retaining higher average mAP.With Trident-head, the model remains 74.6% of ActionFormer latency and uses 1.6G fewer GMACs.
5. Conclusion
TriDet is a simple one-stage convolutional detector with relative boundary modeling. Across four datasets, it achieves state-of-the-art results on three and comparable results on ActivityNet.
- TriDet combines a one-stage convolutional framework with relative boundary modeling for temporal action detection.
- Experiments cover THUMOS14, HACS, EPIC-KITCHEN and ActivityNet, demonstrating the method’s generalization capability.
- TriDet achieves state-of-the-art performance on THUMOS14, HACS and EPIC-KITCHEN.
- TriDet achieves comparable results on ActivityNet, while extensive ablations evaluate each proposed component.
A.1. Network Architecture in Feature Pyramid
The feature-pyramid analysis compares convolutional and Transformer baselines, then studies convolutional replacements, normalization choices, and Trident-head variants. The results show a substantial Transformer advantage over the basic CNN baseline, while convolutional layers narrow that gap.
- Baseline comparison: The study builds convolutional and Transformer baselines to analyze feature-pyramid module design.The Transformer baseline uses local-window self-attention, while the convolutional baseline follows prior one-stage detectors.
- Baseline comparison: The Transformer baseline exceeds the CNN baseline by about 8.1% in average mAP on THUMOS14.
- Ablation design: The ablation evaluates three variants with both normal regression heads and Trident-heads.
- Convolutional replacement: Replacing local self-attention with a same-receptive-field 1D convolution increases average mAP by about 6.2% over the CNN baseline.The convolution uses a kernel size of 19 and remains about 1.9% behind the Transformer baseline.
A.2. The rank loss problem in Transformer.
The appendix explains rank loss as feature homogenization caused by self-attention’s convex combinations, especially for already similar video features. It motivates analyzing convolutional alternatives and using SGP to preserve discrimination while aggregating temporal information.
- Rank loss motivation: Self-attention can drive input features toward a rank-1 matrix at a double-exponential rate, while MLPs and residual connections only partially slow convergence.
- Empirical analysis: Figure 7 compares two baseline models with three convolutional-based variants, while Table 9 reports their THUMOS14 results.
- Geometric explanation: Because self-attention outputs are convex combinations of value features, the largest feature angle after attention cannot exceed the input maximum.This result is established when the input convex hull does not contain the origin.
- Geometric explanation: The appendix uses convex-hull geometry: maximum angular separation is attained by extreme points rather than interior convex combinations.
- TAD implication: In TAD, highly similar backbone features and Layer Normalization often produce a convex hull that excludes the origin, aggravating feature similarity.
A.3. Error Analysis
The THUMOS14 error analysis examines sensitivity across action lengths and instance counts, false positives by prediction group, and false negatives. Performance is generally balanced, with localization regression and extreme-duration actions remaining notable challenges.
- Sensitivity analysis: TriDet’s performance is balanced across most action lengths, except for extremely long action instances.Sensitivity is evaluated using coverage, length, and the number of instances across bins from XS to XL.
- False positives: Regression error, including localization and background error, is the false-positive category that most warrants attention.
- False negatives: False-negative rates are 9.0% for extremely short instances and 13.5% for extremely long instances.These two duration extremes have relatively higher miss-detection rates.
- False negatives: For videos containing one action instance, TriDet records a 0.0% miss-detection rate.
A.4. Qualitative Analysis
Fig. 11 qualitatively shows that TriDet accurately localizes action boundaries, while the Trident-head produces a concentrated and smooth relative boundary-probability distribution.
- TriDet accurately predicts the action’s start and end instants on a THUMOS14 test-set example.
- The Trident-head assigns relatively high probability only to the bin around the boundary.
- Low and smooth probabilities in the other bins indicate that the Trident-head converges to a reasonable result.