Source-linked AI summary
Enriching Local and Global Contexts for Temporal Action Localization
Zixin Zhu, Wei Tang, Le Wang, Nanning Zheng, Gang Hua
TL;DR
Temporal action localization must balance fine-grained temporal discrimination for boundary localization with visual invariance for action classification. ContextLoc combines local and global context with inter-proposal relations, and outperforms recent state-of-the-art methods on THUMOS14 and ActivityNet v1.3.
Problem
Temporal action localization needs representations that balance fine-grained temporal discrimination for localization with visual invariance for action classification.
Source: §1 · p. 1Method
ContextLoc uses L-Net to retrieve snippet-level local context, G-Net to adapt video-level global context to proposals, and P-Net to model inter-proposal relations.
Source: §1 · p. 2Results
54.3% mAP at tIoU 0.5 on THUMOS14 and 56.01% mAP at tIoU 0.5 on ActivityNet v1.3 outperform prior best methods.
Source: §1 · p. 2Takeaways & Limitations
Ablations indicate that local context, global context, and inter-proposal relations are complementary, and the components improve strong baselines across P-Net instantiations.
Source: §4.1 · p. 7
Abstract
from arXiv · showhide
Effectively tackling the problem of temporal action localization (TAL) necessitates a visual representation that jointly pursues two confounding goals, i.e., fine-grained discrimination for temporal localization and sufficient visual invariance for action classification. We address this challenge by enriching both the local and global contexts in the popular two-stage temporal localization framework, where action proposals are first generated followed by action classification and temporal boundary regression. Our proposed model, dubbed ContextLoc, can be divided into three sub-networks: L-Net, G-Net and P-Net. L-Net enriches the local context via fine-grained modeling of snippet-level features, which is formulated as a query-and-retrieval process. G-Net enriches the global context via higher-level modeling of the video-level representation. In addition, we introduce a novel context adaptation module to adapt the global context to different proposals. P-Net further models the context-aware inter-proposal relations. We explore two existing models to be the P-Net in our experiments. The efficacy of our proposed method is validated by experimental results on the THUMOS14 (54.3\% at tIoU@0.5) and ActivityNet v1.3 (56.01\% at tIoU@0.5) datasets, which outperforms recent states of the art. Code is available at https://github.com/buxiangzhiren/ContextLoc.
1. Introduction
Temporal action localization must balance fine-grained temporal discrimination with visual invariance for classification. ContextLoc enriches proposal features with local and global context alongside inter-proposal relations, and reports stronger results than prior best methods on two benchmarks.
1. Introduction
- Prior two-stage methods max-pool snippet features within proposals, discarding fine-grained temporal information needed to locate action boundaries.Source: §1 · p. 2
- Video-level context supplies background and high-level activity cues that can distinguish similar-looking actions, but existing TAL models have largely ignored it.Source: §1 · p. 2
- ContextLoc differs from P-GCN by enriching proposals with local and global contexts, which the authors report are complementary to inter-proposal relations.Source: §1 · p. 2
- 54.3% mAP at tIoU 0.5 on THUMOS14 and 56.01% mAP at tIoU 0.5 on ActivityNet v1.3 outperform the respective prior best methods, PBRNet.Source: §1 · p. 2
- ContextLoc combines L-Net, G-Net, and P-Net to enrich proposal features with snippet-level local context, adapted video-level global context, and context-aware inter-proposal relations.L-Net uses proposals to query snippets for fine-grained temporal information; G-Net adapts global context to different proposals, while P-Net models inter-proposal relations.Source: §1 · p. 2
2. Related Work
Prior TAL work has modeled temporal structure in snippets, proposals, or snippet graphs, while ContextLoc enhances proposal features with snippet-level local and video-level global context. Its contextual modeling is hierarchical, with P-Net also connecting those contexts across proposals.
2. Related Work
- ContextLoc extracts snippet features with a two-stream I3D network pretrained on Kinetics, fusing predictions from RGB frames and optical flow.Source: §2 · p. 2
- Earlier snippet-focused TAL methods model action temporal structure with pyramids, 3D ConvNets, or graph convolutional networks.Examples cited include Zhao et al. [46], Shou et al. [31], and Xu et al. [42].Source: §2 · p. 3
- Proposal-focused methods refine boundaries or select reliable proposals, whereas ContextLoc refines proposal features with snippet-level and video-level context.The cited proposal methods include Lin et al. [21] and Chen et al..Source: §2 · p. 3
- ContextLoc enhances proposal features using snippet-level local context and video-level global context, rather than treating snippets as graph nodes for sub-graph localization as G-TAD does.Source: §2 · p. 3
- ContextLoc models context hierarchically: snippets are local to proposals, proposals are local to videos, and P-Net connects contexts across proposals.Source: §2 · p. 3
3. The ContextLoc Model
ContextLoc enriches proposal representations with local and global video context, then models relations among proposals. Its three networks address different context levels, while its extended-proposal design links the original proposal to its surrounding regions.
3.1. Overview
- ContextLoc starts from snippet-, proposal-, and video-level features and uses L-Net, G-Net, and P-Net to enhance proposals through local context, global context, and inter-proposal relations.The final representations support action classification and completeness prediction with temporal boundary refinement.Source: §3.1 · p. 4
3.2. L-Net (Local Context)
- L-Net retrieves and aggregates snippets relevant to each proposal, preserving fine-grained temporal information that temporal max pooling can discard.It matches proposal features to within-proposal snippet features and uses attention weights to aggregate transformed snippet values.Source: §3.2 · p. 4
3.3. G-Net (Global Context)
- G-Net adapts video-level context to each proposal because proposals need different context and the video representation can contain irrelevant noise.It attends the video representation to proposal and within-proposal snippet features before combining adapted context with the L-Net representation.Source: §3.3 · pp. 4–5
3.4. P-Net (Inter-proposal Relations)
- P-Net models relations among context-enriched proposals using either P-GCN, which connects overlapping or nearby proposals, or a non-local network with similarity-based edge weights.The authors present P-Net as complementary to local and global context, rather than claiming a specific P-Net architecture as their contribution.Source: §3.4 · p. 5
3.5. Extended Proposals
- For L-Net and G-Net, ContextLoc represents an extended proposal as the original proposal plus two side regions, each half its duration, processes them separately with shared weights, then concatenates their representations.This design includes the original proposal in extended-proposal processing, reducing model size and computational complexity while reflecting their connection.Source: §3.5 · pp. 5–6, Figure 3 · p. 5
4. Experiments
On the THUMOS14 test set and ActivityNet v1.3 validation set, ContextLoc outperforms the compared methods at key tIoU thresholds, while its ablations show gains from local and global context modeling. Qualitative examples also show more precise boundary and class predictions than P-GCN.
4.1. Ablation Study
- With non-local P-Net, adding L-Net improves average mAP by 2.88% and tIoU 0.5 by 4.65%; with P-GCN, the gains are 2.69% and 4.07%, respectively.The complete ContextLoc further improves performance with either P-Net instantiation.Source: §4.1 · p. 6
- P-GCN outperforms the non-local network as P-Net both alone and with other sub-networks, and the context components improve performance with either instantiation.The authors use P-GCN as P-Net in the remaining experiments.Source: §4.1 · p. 7
- Both global context adaptation and interactions between adapted global contexts improve performance, with adaptation helping RGB more and interactions helping flow more.The ablation reports results on the THUMOS14 test set.Source: Table 2 · p. 6, §4.1 · p. 7
- Processing extended proposals as three proposals outperforms single-proposal baselines at both 1024 and 3072 feature dimensions.The 3072-dimensional baseline performs better than the 1024-dimensional one, but uses significantly more parameters and flops while remaining inferior to the proposed processing.Source: §4.1 · p. 7, Table 3 · p. 7
- ContextLoc uses 6.7M parameters and 3.1G flops versus P-GCN’s 4.6M and 1.7G, while its light version beats P-GCN by 3.22% at tIoU@0.5 with 3.4M parameters and 1.6G flops.Adding graph convolutional layers to P-GCN does not improve performance.Source: §4.1 · p. 7, Table 4 · p. 7
- For high-quality proposals, L-Net contributes more to localization accuracy and G-Net more to classification accuracy.High-quality proposals are defined as having tIoU greater than 0.7 with ground truth.Source: §4.1 · p. 7, Table 5 · p. 7
4.2. Comparison with State-of-the-Art Methods
- At tIoU 0.5, ContextLoc beats PBRNet by 3.0% on THUMOS14, while ContextLoc* reaches 56.01% mAP on ActivityNet v1.3, 2.05% above PBRNet.At tIoU 0.75, ContextLoc* is 0.22% above PBRNet; at 0.95, it does not match previous methods. Results use the THUMOS14 test set and ActivityNet v1.3 validation set.Source: §4.2 · pp. 7–8, Table 6 · p. 8, Table 7 · p. 8
4.3. Qualitative Results
- In the qualitative examples, ContextLoc predicts more precise temporal boundaries and classifications than P-GCN.Figure 4 uses blue for ground truth, orange for ContextLoc, and green for P-GCN.Source: §4.3 · p. 8, Figure 4 · p. 8
5. Conclusion
ContextLoc jointly models local and global context with context-aware inter-proposal relations. Ablations indicate each component is effective, and results on two datasets outperform state-of-the-art TAL methods.
5. Conclusion
- ContextLoc outperforms state-of-the-art TAL methods on two datasets.Source: §5 · p. 8
- ContextLoc unifies local and global context modeling with context-aware inter-proposal relations for TAL.Source: §5 · p. 8
- Controlled ablation experiments indicate the effectiveness of each ContextLoc component.Source: §5 · p. 8