Source-linked AI summary
Video Self-Stitching Graph Network for Temporal Action Localization
Chen Zhao, Ali Thabet, Bernard Ghanem
TL;DR
Short actions are common in temporal action localization datasets but are localized less accurately because their limited temporal information is vulnerable to loss or distortion. VSGN combines original and magnified clips with cross-scale graph-pyramid aggregation, and reports improved short-action localization and state-of-the-art overall performance on THUMOS-14 and ActivityNet-v1.3.
Problem
Short actions dominate many datasets but have inferior localization performance because their small temporal scales contain fewer frames that can be lost or distorted.
Method
VSGN magnifies short clips, stitches them with original-scale clips, and uses a cross-scale graph pyramid to aggregate same-scale and cross-scale features.
Results
VSGN achieves 52.4% mAP@0.5 on THUMOS-14 and 35.07% average mAP on ActivityNet-v1.3, surpassing previous best scores under the same features.
Takeaways & Limitations
VSGN improves short-action localization and overall performance while generating more short training samples and positive anchors.
Abstract
from arXiv · showhide
Temporal action localization (TAL) in videos is a challenging task, especially due to the large variation in action temporal scales. Short actions usually occupy a major proportion in the datasets, but tend to have the lowest performance. In this paper, we confront the challenge of short actions and propose a multi-level cross-scale solution dubbed as video self-stitching graph network (VSGN). We have two key components in VSGN: video self-stitching (VSS) and cross-scale graph pyramid network (xGPN). In VSS, we focus on a short period of a video and magnify it along the temporal dimension to obtain a larger scale. We stitch the original clip and its magnified counterpart in one input sequence to take advantage of the complementary properties of both scales. The xGPN component further exploits the cross-scale correlations by a pyramid of cross-scale graph networks, each containing a hybrid module to aggregate features from across scales as well as within the same scale. Our VSGN not only enhances the feature representations, but also generates more positive anchors for short actions and more short training samples. Experiments demonstrate that VSGN obviously improves the localization performance of short actions as well as achieving the state-of-the-art overall performance on THUMOS-14 and ActivityNet-v1.3.
1. Introduction
Temporal action localization must handle widely varying action durations, yet short actions dominate datasets and are localized less accurately. VSGN addresses this imbalance by combining original and temporally magnified clips with cross-scale feature aggregation.
- Temporal action localization identifies action categories and their start and end times in untrimmed videos.
- Short actions dominate ActivityNet-v1.3 but perform worse than longer actions across temporal action localization methods.Actions shorter than 30 seconds are the dominant duration group.
- Short actions are difficult because their few frames can be lost or distorted as videos pass through deep networks.Methods that process videos without regard to duration can sacrifice short-action performance.
- VSGN magnifies short clips, stitches them with original-scale clips, and aggregates features across scales through video self-stitching and a cross-scale graph pyramid network.The framework also produces more short training samples and positive anchors for short actions.
- 52.4% mAP@0.5 on THUMOS-14 and 35.07% average mAP on ActivityNet-v1.3 establish VSGN's reported state-of-the-art overall performance under the same features.The THUMOS-14 result is compared with 40.4%, and the ActivityNet-v1.3 result with 34.26%.
2. Related Work
Prior TAL approaches address temporal scale through fixed-length rescaling, sliding windows, or multi-scale features, but these strategies can weaken short-action information. VSGN instead combines original and up-scaled clips and models both same-scale and cross-scale snippet relations in a multi-level graph pyramid.
- Multi-scale solution in object detection: Feature pyramid networks generate multi-scale features through encoder and decoder pyramids, inspiring VSGN's temporal cross-scale graph pyramid.VSGN embeds cross-scale graph networks at the pyramid levels.
- Multi-scale solution in object detection: Mosaic augmentation motivates VSGN's strategy of stitching the same video clip at different temporal scales.The related augmentation aims to reduce overemphasis on large objects and improve small-object performance.
- Temporal action localization: Fixed-length rescaling is efficient but can down-scale short actions and lose or distort their information, especially in long videos.Architectural constraints also make up-scaling non-trivial for some TAL methods.
- Temporal action localization: Sliding-window methods preserve original frame information and obtain multi-scale features through pooling or strided convolution.VSGN differs by using both the original clip and its up-scaled counterpart in one framework.
- Graph neural networks for TAL: VSGN extends snippet graphs beyond same-scale relations by adding cross-scale edges and multiple graph levels in a pyramid.This design differs from G-TAD's single-level, same-scale graph modeling.
3. Video Self-Stitching Graph Network
VSGN addresses short-action localization by combining original and temporally magnified clips, then modeling their within- and cross-scale relationships with graph networks.
- Overview: VSGN contains video self-stitching, cross-scale graph pyramid processing, and scoring and localization components.The framework is described as a multi-level architecture for handling short actions.
- Video self-stitching: Video self-stitching extracts snippet features, cuts long videos into short clips, temporally up-scales them, and stitches original and enlarged clips with a zero-filled gap.The gap helps prevent predictions from spanning across the two stitched clips.
- Video self-stitching: Short clips contain no more than γL snippets during training, while inference does not cut sequences and long actions remain detectable through original sequences.Actions longer than γL are excluded from self-stitching but are still processed through the original input path.
- Video self-stitching: The shorter a clip is, the longer its up-scaled counterpart becomes, filling the remaining input space and emphasizing shorter clips.A clip with M snippets is up-scaled to length L − G − M, where G is the gap length.
- Cross-scale graph pyramid: The cross-scale graph network aggregates temporal-neighborhood, intra-scale, and cross-scale features before activation and pooling to produce the next pyramid level.Its graph is built over features from both original and up-scaled clips using free and cross-scale edges.
- Cross-scale graph pyramid: Free edges connect feature-similar nodes without clip restrictions, whereas cross-scale edges connect nodes from different clips.Each node uses K/2 edges of each type.
4. Experiments
Experiments on THUMOS-14 and ActivityNet-v1.3 show that VSGN improves short-action localization and overall performance through VSS, xGPN, and complementary multi-scale predictions.
- Comparison with State-of-the-Art: VSGN achieves 52.4% mAP at tIoU 0.5 on THUMOS-14 and 35.07% average mAP on ActivityNet-v1.3.These results are reported as state-of-the-art and outperform methods using the same pre-extracted features.
- Comparison with State-of-the-Art: Short actions constitute 99.7% of THUMOS-14 actions and 54.4% of ActivityNet-v1.3 actions, with VSGN showing especially evident gains on them.Short actions are defined as instances shorter than 30 seconds.
- Ablation Study: VSS improves short-action performance on both datasets, while xGPN further improves overall performance and benefits actions of different lengths.On ActivityNet-v1.3, VSS can sacrifice long-action performance, an effect xGPN is designed to mitigate.
- Ablation Study: Clip O performs better at lower tIoU thresholds, Clip U has an advantage at higher thresholds, and combining both predictions performs best.The result supports using the complementary properties of the original and enlarged clips.
- Ablation Study: Using xGN at all encoder levels achieves the best performance, while the proposed mixture of free and cross-scale edges outperforms single edge types.The comparisons evaluate xGN placement across levels and alternative edge constructions.
- Inference Efficiency: VSGN has lower inference-time requirements than end-to-end frameworks while using pre-extracted features and avoiding complex 3D convolutions.Its VSS computation is negligible, and its xGPN cost is similar to comparable pre-extracted-feature methods.
5. Conclusions
The conclusion presents VSGN as a multi-level cross-scale solution for short actions in TAL, combining stitched clips with cross-scale feature aggregation. It reports improvements in both short-action and overall performance.
- Conclusion: VSGN targets large action-scale variation by focusing on short actions in temporal action localization.The paper identifies this as a challenging problem and frames VSGN as its proposed solution.
- Conclusion: VSS stitches an enlarged clip with the original-scale clip to exploit their complementary properties.The enlarged clip provides a larger temporal scale, while the original clip preserves the original-scale information.
- Conclusion: xGPN aggregates features across different scales and within the same scale.Together with VSS, this forms the paper’s multi-level cross-scale framework.
- Conclusion: VSGN significantly improves short-action performance as well as overall performance.The conclusion states this result without restricting it to a single dataset.
A. More Localization Results using Different Features
Additional ActivityNet-v1.3 experiments evaluate VSGN with R2+1d and TSP features generated from 15-fps video using one-second, non-overlapping snippets.
- Different Features: VSGN is evaluated on ActivityNet-v1.3 using R2+1d and TSP features in addition to the primary feature setting.Both feature types use RGB streams and videos sampled at 15 fps.
- Different Features: The additional features are extracted from non-overlapped 16-frame snippets, each covering approximately one second.This setup is used to generate both R2+1d and TSP representations.
B. More Ablation Results
Further ablations test the two stitched clip types and the two xGN branches, showing that their combination improves performance over using individual components alone.
- Clip Ablation: Using both Clip O and Clip U outperforms using either clip type alone.When only one clip type is used, zeros fill the other clip’s positions in the stitched sequence.
- Branch Ablation: Combining the temporal and graph branches in xGN outperforms using either branch alone for both overall and short-action performance.The two branch outputs are summed to form the xGN output.
C. Performance at Different Temporal Scales
VSGN performs best at the shortest temporal scales, which contain the most action instances, while remaining competitive for longer actions. Its advantage is especially clear for the shortest duration group.
- Performance across temporal scales: VSGN obviously outperforms other methods in accumulated mAP for the shortest duration group while maintaining high ranks at longer durations.Accumulated mAP weights each duration group by its number of instances and average-mAPN.
- Performance across temporal scales: VSGN achieves its best mAP at the shortest scales, which contain the most instances, and competitive scores at long scales.
D. Visualization of Localization Results
Visualizations show that VSGN can accurately localize very short and long action instances, including consecutive instances. However, precise boundaries remain difficult when consecutive short actions merge or the background resembles the action.
- Localization examples: VSGN accurately localizes very short and long action instances, even when multiple consecutive instances occur in one video.
- Localization examples: VSGN sometimes fails to generate precise boundaries when consecutive short actions are mistaken for one long action or when background resembles the actions.The paper identifies these cases as requiring further exploration.