Source-linked AI summary
Boundary Content Graph Neural Network for Temporal Action Proposal Generation
Yueran Bai, Yingying Wang, Yunhai Tong, Yang Yang, Qiyue Liu, Junhui Liu
TL;DR
Temporal action proposal generation needs precise boundaries and high-quality action content, yet existing approaches have limited flexibility for varied action durations. BC-GNN models boundaries as nodes and proposal content as edges, updates both through graph reasoning, and combines them to generate proposals. Experiments report superior performance in both temporal action proposal generation and temporal action detection on ActivityNet-1.3 and THUMOS14.
Problem
Generating temporal proposals with both precise boundaries and rich action content remains challenging, while fixed windows and anchors lack flexibility for varied action durations.
Method
BC-GNN constructs a boundary-content graph, applies graph reasoning to update node and edge features, and uses connected nodes and edges to generate proposals.
Results
BC-GNN outperforms existing state-of-the-art methods in both temporal action proposal generation and temporal action detection tasks.
Takeaways & Limitations
Modeling interactions between proposal boundaries and content supports high-performing temporal proposal generation and detection across the evaluated datasets.
Takeaways & Limitations
The proposal framework is evaluated on untrimmed video represented as a sequence of frames and uses fixed-interval temporal locations for start and end locations.
Abstract
from arXiv · showhide
Temporal action proposal generation plays an important role in video action understanding, which requires localizing high-quality action content precisely. However, generating temporal proposals with both precise boundaries and high-quality action content is extremely challenging. To address this issue, we propose a novel Boundary Content Graph Neural Network (BC-GNN) to model the insightful relations between the boundary and action content of temporal proposals by the graph neural networks. In BC-GNN, the boundaries and content of temporal proposals are taken as the nodes and edges of the graph neural network, respectively, where they are spontaneously linked. Then a novel graph computation operation is proposed to update features of edges and nodes. After that, one updated edge and two nodes it connects are used to predict boundary probabilities and content confidence score, which will be combined to generate a final high-quality proposal. Experiments are conducted on two mainstream datasets: ActivityNet-1.3 and THUMOS14. Without the bells and whistles, BC-GNN outperforms previous state-of-the-art methods in both temporal action proposal and temporal action detection tasks.
1 Introduction
Temporal action proposal generation must jointly achieve precise boundaries and rich action content, but fixed windows and anchors limit temporal flexibility. BC-GNN links boundary locations and proposal content in a graph, updating both to produce proposals, and is reported to outperform existing methods across proposal generation and detection.
- Temporal proposals must localize action instances with both precise boundaries and rich action content.
- Fixed sliding windows and temporal anchors cannot precisely bound proposals or flexibly retrieve actions with varied durations.
- BC-GNN models temporal locations as graph nodes and the content between valid start–end pairs as edges.
- A novel graph reasoning operation updates edge and node features so boundary probabilities and content confidence can be jointly estimated.
- BC-GNN combines two connected nodes with their updated edge to generate candidate proposals from boundary probabilities and content confidence.
- Experiments report that BC-GNN outperforms existing state-of-the-art methods in temporal action proposal generation and temporal action detection.
2 Related work
Related work covers action-recognition features and graph neural networks, while temporal proposal methods seek high-recall, high-precision segments despite varied action durations. Existing approaches include sliding windows, multi-scale anchors, and Temporal Actionness Grouping.
- Action Recognition: Action recognition methods provide feature sequences for analyzing long, untrimmed videos, including CNN-based multimodal and spatiotemporal approaches.
- Graph Neural Network: Graph neural networks extend deep learning to graph-structured data, with prior work addressing node convolutions, attention, and edge features.
- Temporal Action Proposal Generation: Temporal action proposal generation retrieves segments containing action instances with high recall and precision.
- Temporal Action Proposal Generation: Prior proposal methods use temporal sliding windows or multi-scale anchors, but these approaches neglect the varied durations of ground-truth actions.
3 Our Approach
BC-GNN encodes video features, constructs a boundary-content graph, exchanges information between graph nodes and edges, and predicts proposal scores from updated graph features.
- Framework: BC-GNN processes encoded video features through Base, Graph Construction, Graph Reasoning, and Output modules before generating proposals.The Output Module predicts boundary and content confidence scores, followed by score fusion and Soft-NMS.
- Feature Encoding: The two-stream encoder separately captures appearance from RGB frames and motion from optical flow frames.Its outputs are concatenated into a feature sequence after processing regularly sampled video snippets.
- Base Module: The Base Module expands the receptive field and normalizes variable-length videos using fixed-length temporal observation windows.It applies temporal convolutions to extract local semantic features from each input window.
- Graph Construction Module: GCM forms candidate start-end pairs, represents intervening content as edge features, and connects each valid start node only to a later end node.Content between temporal locations is interpolated to a fixed number of vectors and projected to the node-feature dimension.
- Graph Reasoning Module: GRM updates graph edges and nodes by exchanging information between connected features through directed edge representations.Each undirected edge is split into opposite directions because information from start to end can differ from information from end to start.
4 Experiment
Experiments evaluate BC-GNN on two benchmark datasets for proposal generation and action detection, using standard recall- and precision-based metrics. BC-GNN achieves strong comparisons with existing methods, while ablations examine its graph components and detection performance.
- Experimental setup: Experiments use ActivityNet-1.3 and THUMOS-14, with AR@AN and AUC for proposal quality and mAP for action detection.ActivityNet-1.3 uses AR@AN and AUC; THUMOS-14 uses AR@AN for proposals, while detection is evaluated with mAP.
- Temporal action proposal generation: BC-GNN outperforms leading proposal-generation methods by a large margin on ActivityNet-1.3, particularly at AR@100.The comparison is reported on the validation set using AR@AN and AUC.
- Temporal action proposal generation: BC-GNN outperforms other state-of-the-art proposal methods by a large margin on THUMOS-14 AR@AN.The comparison uses the testing set and includes flow, two-stream, and C3D features for fair evaluation.
- Analysis: The experiments attribute BC-GNN's performance improvement to explicitly modeling interactions between boundaries and action content.This interaction is the central relationship represented by the boundary-content graph.
- Ablation study: Ablation experiments evaluate GCN-based graphs, edge updates, and directed edges as components of the graph reasoning module.The study compares these graph designs and reports that both examined strategies are effective and essential.
- Temporal action detection: BC-GNN reaches state-of-the-art action detection performance on ActivityNet-1.3 and surpasses existing two-stage methods on THUMOS-14 mAP.Detection uses proposals generated by BC-GNN within a two-stage framework.
5 Conclusion
BC-GNN introduces a boundary-content graph and graph reasoning operation for temporal action proposal generation. Experiments show promising performance in both temporal proposal generation and temporal action detection.
- BC-GNN models interactions between boundary probabilities and proposal confidence with a boundary content graph.The graph reasoning operation updates node and edge features, while an output module generates proposals from the strengthened features.
- A graph reasoning operation updates node and edge features, followed by an output module that generates proposals.
- BC-GNN achieves promising performance in both temporal proposal generation and temporal action detection tasks.