Source-linked AI summary
Graph Convolutional Networks for Temporal Action Localization
Runhao Zeng, Wenbing Huang, Mingkui Tan, Yu Rong, Peilin Zhao, Junzhou Huang, Chuang Gan
TL;DR
Temporal action localization systems commonly process proposals independently, leaving proposal-proposal relations underused even though actions may span multiple proposals. P-GCN constructs a proposal graph with contextual and surrounding edges and applies GCNs to jointly support classification and localization. It outperforms state-of-the-art methods on THUMOS14 and shows efficacy on ActivityNet augmentation experiments.
Problem
Existing action localization systems process proposals individually and therefore do not explicitly exploit proposal-proposal relations during learning.
Method
P-GCN constructs a graph whose nodes are proposals, establishes contextual and surrounding edges, and applies GCNs to produce enhanced features for category and temporal-box prediction.
Results
49.1% mAP at tIoU = 0.5 on THUMOS14 outperformed the 42.8% state-of-the-art result, and ActivityNet augmentation experiments verified proposal-relation efficacy.
Takeaways & Limitations
Modeling proposal-proposal interactions with GCN message passing outperformed state-of-the-art methods on THUMOS14 and ActivityNet v1.3.
Takeaways & Limitations
The approach assumes that action proposals have already been obtained by another method before prediction.
Abstract
from arXiv · showhide
Most state-of-the-art action localization systems process each action proposal individually, without explicitly exploiting their relations during learning. However, the relations between proposals actually play an important role in action localization, since a meaningful action always consists of multiple proposals in a video. In this paper, we propose to exploit the proposal-proposal relations using Graph Convolutional Networks (GCNs). First, we construct an action proposal graph, where each proposal is represented as a node and their relations between two proposals as an edge. Here, we use two types of relations, one for capturing the context information for each proposal and the other one for characterizing the correlations between distinct actions. Then we apply the GCNs over the graph to model the relations among different proposals and learn powerful representations for the action classification and localization. Experimental results show that our approach significantly outperforms the state-of-the-art on THUMOS14 (49.1% versus 42.8%). Moreover, augmentation experiments on ActivityNet also verify the efficacy of modeling action proposal relationships. Codes are available at https://github.com/Alvin-Zeng/PGCN.
1. Introduction
Temporal action localization must classify actions and locate their temporal boundaries in untrimmed videos, but proposal-wise prediction neglects relations among proposals. P-GCN addresses this gap by representing proposals as graph nodes and modeling contextual and surrounding relations with GCNs.
- Untrimmed-video localization requires both action classification and prediction of each action instance’s start and end times.
- Most two-stage methods classify and regress boundaries for each proposal independently, neglecting semantic relations between proposals.
- Overlapping proposals can describe different parts of one action, so neighboring proposal features provide additional context for individual predictions.
- P-GCN represents proposals as graph nodes and uses contextual edges for related parts of an action and surrounding edges for nearby distinct actions.
- 49.1% mAP at tIoU = 0.5 on THUMOS14 surpassed the 42.8% state-of-the-art result, while ActivityNet augmentation experiments supported proposal-relation modeling.
2. Related work
Prior temporal action localization work includes frame- or segment-level methods and two-stage proposal-based methods, while GCNs provide a framework for convolution on non-grid structures. P-GCN applies this graph-based perspective to proposal relations.
- Temporal action localization methods include frame- or segment-level classification approaches that require smoothing and merging to obtain temporal boundaries.
- Two-stage approaches first generate action proposals and then perform classification and temporal boundary regression.
- P-GCN builds on the two-stage paradigm by applying graph convolutions to proposal relations.
- GCNs define convolutions on non-grid structures and have been applied in areas including skeleton-based action recognition, person re-identification, and video classification.
3. Our Approach
The approach represents action proposals as graph nodes linked by temporal relations, then applies graph convolutions to enhance proposal features for action localization. It uses separate GCNs for classification and boundary regression and samples neighborhoods to reduce training cost.
- 3.1. Notation and Preliminaries: Each action proposal is treated as a graph node, with edges representing relations between proposals.The proposal graph and proposal features jointly support graph convolution over proposal neighborhoods.
- 3.2. General Scheme of Our Approach: P-GCN applies graph convolution so each proposal aggregates neighborhood information before predicting its action category and temporal bounding box.The enhanced proposal features are used to jointly predict category labels and temporal positions.
- 3.3. Proposal Graph Construction: Contextual edges connect proposals with tIoU above θctx, selecting highly overlapping proposals whose non-overlapping portions provide contextual information.This context is intended to help refine detection boundaries and improve detection accuracy.
- 3.3. Proposal Graph Construction: Surrounding edges connect nearby proposals with zero overlap when d(pi, pj) < θsur, allowing messages to pass across distinct action instances.The distance uses proposal center coordinates and complements contextual edges with additional temporal cues.
- 3.4. Graph Convolution for Action Localization: Two GCNs separately predict action labels and temporal boundaries using original and temporally extended proposal features.The label branch uses original proposal features, while the boundary branch uses features extracted after extending each proposal boundary.
- 3.5. Efficient Training by Sampling: Neighborhood sampling reduces the computation and memory required to train graph convolutions on videos containing thousands of proposals.SAGE samples fixed-size neighborhoods layer by layer during training, while no sampling is used at testing.
4. Experiments
Experiments evaluate P-GCN on THUMOS14 and ActivityNet v1.3 using standard action-localization metrics, with comparisons across methods and model configurations.
- Datasets: THUMOS14 contains 13,320 training videos, with 1,010 validation, 1,574 testing, and 2,500 background videos.
- Datasets: ActivityNet v1.3 contains around 10K training videos and 5K validation videos spanning 200 activities.Each video averages 1.65 action instances.
- Evaluation Metrics: mAP evaluates proposals across temporal IoU thresholds, requiring category agreement with the ground-truth instance.THUMOS14 uses thresholds from 0.1 to 0.5; ActivityNet uses 0.5, 0.75, 0.95, plus averaged thresholds from 0.5 to 0.95.
- Comparison with state-of-the-art results: P-GCN achieves the highest mAP across THUMOS14 thresholds and improves over TAL-Net by 6.3% at tIoU = 0.5.The same passage reports improvements over the second-best result by more than 12.2% at that threshold.
- Comparison with state-of-the-art results: On ActivityNet v1.3 average mAP, P-GCN exceeds SSN, CDC, and TAL-Net by 3.01%, 3.19%, and 6.77%, respectively.
5. Ablation Studies
The ablations test whether proposal relations, edge designs, self-addition, and neighborhood sampling improve P-GCN over independent or pooled baselines. Results support relation modeling, two edge types, self-addition, and efficient sampling, while qualitative examples show gains in boundaries and classification.
- Proposal-proposal relations: P-GCN is compared with an MLP that removes the adjacency matrix and processes each proposal independently.This isolates the contribution of message passing along proposals.
- Proposal-proposal relations: P-GCN consistently improves over MLP across feature and proposal combinations, indicating effectiveness beyond a specific backbone.The combinations include BSN+I3D, BSN+2D, TAG+I3D, and TAG+2D.
- Edge weights and self-addition: Self-addition improves mAP by at least 1.7% on both RGB and Flow streams, whereas embedded cosine similarity changes mAP by less than 0.4%.The comparison evaluates edge-weight functions and the self-feature term.
- Edge types: Removing either contextual or surrounding edges causes a remarkable performance drop, while surrounding edges alone still improve over MLP.The ablation supports using both relation types and indicates that distinct actions can provide useful information.
- Sampling efficiency: When Ns = 4, neighborhood sampling achieves higher mAP than the full model while reducing training time per iteration by 76%.Sampling sizes from 1 to 5 and 10 are compared with the full model.
- Qualitative results: Qualitative results show more precise temporal boundaries and correct classification where MLP confuses “Shotput” with “Throw Discus”.In the top example both methods classify correctly, but P-GCN localizes the boundary more precisely.
6. Conclusions
The paper concludes that proposal-proposal interactions can be modeled with a proposal graph and GCN message passing for temporal action localization. It reports large-margin improvements on THUMOS14 and ActivityNet v1.3, while identifying object detection as future work.
- Conclusions: P-GCN constructs a graph of proposals and applies GCN message passing to exploit proposal-proposal interactions for temporal action localization.The conclusion describes the graph-based interaction mechanism as the paper’s central approach.
- Conclusions: P-GCN outperforms state-of-the-art methods by a large margin on THUMOS14 and ActivityNet v1.3.The conclusion names both evaluated benchmarks.
- Future work: Extending P-GCN to object detection is identified as future work.The paper leaves this extension for subsequent research.
B. Network Architectures
The P-GCN architecture uses graph-based processing followed by task-specific fully connected layers, while the MLP and mean-pooling baselines remove or replace graph message passing.
- P-GCN: P-GCN uses three fully connected layers after the GCN for boundary regression, action classification, and completeness classification.The regression layer has Nclass × 2 outputs; the classification layers have Nclass outputs.
- Mean-pooling baseline: The mean-pooling baseline applies mean pooling to MLP outputs over adjacent proposals using the same graph as GCN.Its architecture otherwise follows the MLP baseline.
C. Training Details
Training categorizes proposals using tIoU and overlap criteria, samples fixed proportions within single-video mini-batches, and uses graph neighborhoods with fixed contextual and surrounding connections. The model jointly learns classification, completeness, and boundary regression.
- Training samples: Training samples are labeled foreground, incomplete, or background using best tIoU and best overlap thresholds.Foreground requires tIoU ≥ θ1; incomplete requires OL ≥ θ2 and tIoU ≤ θ3; background requires tIoU ≤ θ4.
- Mini-batches: Each mini-batch draws examples from one video with foreground:incomplete:background fixed at 1:6:1.Batch sizes are 32 on THUMOS14 and 64 on ActivityNet v1.3.
- Neighborhoods: Each node uses 10 sampled neighborhoods, selecting contextual edges by largest relevance and surrounding edges by smallest distance in a 4:1 ratio.This fixed neighborhood design is introduced for efficiency.
- Edge weights: Negative edge weights are set to zero because this empirically produces better results.The condition is Ai,j < 0.
- Multi-task learning: The multi-task loss combines action classification, completeness classification, and location regression, with λ1 = λ2 = 0.5.Completeness loss applies to non-background proposals, while regression applies to foreground complete proposals.
E. Details of Augmentation Experiments on ActivityNet
The ActivityNet augmentation incorporates external video-level action labels into proposal predictions and uses action, completeness, and BSN confidence scores for mAP computation.
- External video-level labels replace predicted action classes in the augmentation, assigning UntrimmedNet’s top-2 classes to every proposal.Each proposal therefore receives two action classes from the input video.
- The augmentation uses UntrimmedNet to predict the top-2 video-level action classes for each input video.
- mAP scoring combines each proposal’s action and completeness scores with BSN confidence and UntrimmedNet action scores.
F. Explanation and ablation study of θctx
The contextual-edge threshold θctx controls graph construction, with the method performing well across several high threshold settings in the THUMOS14 Flow ablation.
- Explanation of θctx: Contextual edges connect proposals when their relevance exceeds θctx, which can be selected from [0, 1).
- Ablation study: The method performs well when θctx = 0.7, 0.8, or 0.9.
- Ablation study: The THUMOS14 Flow ablation evaluates different θctx values in Table B.
G. Ablation study of boundary regression
The boundary-regression ablation supports retaining boundary regression, while the runtime comparison indicates that P-GCN’s additional computation is relatively small compared with MLP.
- Ablation study of boundary regression: Boundary-regression ablation results on THUMOS14 validate the necessity of using boundary regression.
- Ablation study of boundary regression: Table C reports the ablation results of boundary regression on THUMOS14.
- Additional runtime compared to [52]: P-GCN incurs relatively little additional runtime compared with MLP while significantly boosting performance.The runtime comparison excludes proposal generation and feature extraction.