Source-linked AI summary
Multi-granularity Generator for Temporal Action Proposal
Yuan Liu, Lin Ma, Yifeng Zhang, Wei Liu, Shih-Fu Chang
TL;DR
Temporal action proposal generation must localize human-action segments in untrimmed video, while existing multi-stage methods lack overall optimization and neglect temporal position information. MGG combines position-aware video representations with coarse segment proposals and fine frame actionness in an end-to-end model, achieving state-of-the-art proposal performance on THUMOS-14 and ActivityNet-1.3 and improved video detection performance.
Problem
Temporal action proposal generation localizes human-action segments in untrimmed video, but existing multi-stage methods lack overall optimization and neglect temporal position information; capturing varied action durations with high recall is also challenging.
Method
MGG combines position-embedded video features, bilinear matching, coarse segment proposal generation through SPP, and fine frame-actionness evaluation through FAP in an end-to-end architecture.
Results
MGG achieves state-of-the-art temporal action proposal performance on THUMOS-14 and ActivityNet-1.3, while generated proposals also support superior video detection performance.
Takeaways & Limitations
Analyzing videos from coarse and fine perspectives yields proposals with high recall and more precise boundary information, with effectiveness further demonstrated in video detection.
Abstract
from arXiv · showhide
Temporal action proposal generation is an important task, aiming to localize the video segments containing human actions in an untrimmed video. In this paper, we propose a multi-granularity generator (MGG) to perform the temporal action proposal from different granularity perspectives, relying on the video visual features equipped with the position embedding information. First, we propose to use a bilinear matching model to exploit the rich local information within the video sequence. Afterwards, two components, namely segment proposal producer (SPP) and frame actionness producer (FAP), are combined to perform the task of temporal action proposal at two distinct granularities. SPP considers the whole video in the form of feature pyramid and generates segment proposals from one coarse perspective, while FAP carries out a finer actionness evaluation for each video frame. Our proposed MGG can be trained in an end-to-end fashion. By temporally adjusting the segment proposals with fine-grained frame actionness information, MGG achieves the superior performance over state-of-the-art methods on the public THUMOS-14 and ActivityNet-1.3 datasets. Moreover, we employ existing action classifiers to perform the classification of the proposals generated by MGG, leading to significant improvements compared against the competing methods for the video detection task.
1. Introduction
Temporal action proposal generation must localize action-containing intervals while balancing recall, overlap, and false alarms. MGG addresses limitations of prior approaches by combining coarse segment proposals with fine frame actionness in an end-to-end, position-aware architecture.
- Temporal action proposal localizes intervals likely to contain actions in untrimmed videos and supports recognition, summarization, grounding, and captioning.
- Existing segment-based methods generate proposals of varied spans but can miss true actions because regularly distributed segments receive low confidence on long video segments.
- Prior boundary-sensitive and complementary methods combine local boundary detection with global proposal ranking or multiple refinement stages, but train stages independently and neglect temporal position information.
- MGG integrates position-embedded video features, bilinear local matching, and coupled segment proposal and frame actionness producers for multi-granularity proposal generation.
- SPP uses a U-shape feature-pyramid architecture for high-recall proposals across temporal spans, while FAP predicts each frame’s starting, ending, and middle-point probabilities.
- By adjusting segment boundaries with complementary frame actionness, MGG achieves state-of-the-art temporal action proposal performance on THUMOS-14 and ActivityNet-1.3.
2. Related Work
Related work commonly uses proposal-plus-classification frameworks for temporal action detection, with proposal and classification stages trained separately or jointly.
- Many temporal action detection methods apply classifiers to a smaller set of class-agnostic segment proposals in a proposal-plus-classification framework.
- The proposal and classification stages in these methods can be trained separately or jointly.
3. Our Approach
MGG generates temporal action proposals at coarse segment and fine frame granularities using position-aware video representations and bilinear local matching. Its SPP uses a U-shape feature pyramid for varied proposal durations, while FAP provides frame actionness for finer evaluation and boundary adjustment.
- 3. Our Approach: MGG combines coarse segment proposals with fine frame actionness and temporally adjusts proposal boundaries to generate refined action proposals.The architecture integrates SPP and FAP outputs through temporal boundary adjustment.
- 3.1. Video Representation: MGG combines video visual features with sinusoidal position embeddings to represent both appearance and temporal ordering.The position embedding is concatenated with each visual feature to form the final video representation.
- 3.2. BaseNet: The BaseNet uses two temporal convolutions and factorized bilinear matching to capture local interactions between representations at different semantic levels.Factorization reduces parameters relative to traditional bilinear matching, making training easier.
- 3.3. Segment Proposal Producer: SPP uses a U-shape architecture with lateral connections to build a feature pyramid whose levels localize proposals across different temporal spans.The architecture addresses the challenge of capturing actions with substantially varying durations while retaining high-level semantic information across pyramid levels.
- 3.3. Segment Proposal Producer: SPP predicts proposal presence and boundary offsets through separate classification and boundary-regression branches applied to regularly distributed anchors.The two branches form the anchor prediction module at each feature-pyramid location.
- 3.4. Frame Actionness Producer: FAP densely evaluates every frame with separate probabilities for being an action starting point, ending point, or middle point.Its three probability sequences provide finer-grained frame information than the segment proposals generated by SPP.
4. Training and Inference
MGG jointly trains a shared BaseNet with SPP for coarse segment proposals and FAP for fine frame actionness, then fuses their outputs to refine temporal boundaries during inference. SPP uses multi-task actionness and boundary regression, while TBA combines proposal-level and frame-level evidence to improve boundary accuracy.
- Training: MGG jointly trains a shared BaseNet, SPP, and FAP end-to-end with a weighted sum of their objectives.The total loss combines the SPP and FAP objectives, with β empirically set to 0.1.
- Training: SPP assigns binary labels to anchor segments using tIoU thresholds and filters anchors that are neither positive nor negative.Anchors are positive above 0.7 tIoU or when having the highest tIoU with a ground-truth proposal, negative below 0.3, and otherwise discarded.
- Training: SPP uses multi-task learning to predict anchor actionness and regress proposal boundaries with cross-entropy and smooth L1 losses.Boundary regression is activated only for positive anchors; the trade-off parameter γ is set to 0.001 empirically.
- Training: FAP labels starting, ending, and middle regions around annotated actions and trains three probability sequences with cross-entropy losses and inverse-frequency weighting.The starting and ending regions are defined around action boundaries, while the middle region spans the annotated action.
- Inference: TBA first adjusts SPP proposal boundaries using FAP starting and ending scores, then replaces proposals whose tIoU with grouped high-middle-probability regions exceeds 0.8.SPP supplies coarse proposals, whereas FAP provides finer frame-level boundary sensitivity; the second stage improves overall boundary accuracy.
5. Experiments
Experiments evaluate MGG on THUMOS-14 and ActivityNet-1.3, including proposal quality, component ablations, boundary adjustment, training strategy, and video detection. MGG generally outperforms competing methods, while combining coarse and fine proposal information improves boundaries and performance.
- Temporal Proposal Generation: MGG consistently improves AR@AN over competing methods on THUMOS-14 across AN values from 50 to 500 with two-stream features, and from 50 to 1000 with C3D features.BSN+NMS performs better at AN=1000 when using two-stream features.
- Temporal Proposal Generation: 37.01% versus 33.02% at AN=40, while MGG also achieves higher recall than other methods for tIoU values from 0.5 to 1.These results are reported on the THUMOS-14 testing split.
- Temporal Proposal Generation: MGG outperforms the compared methods on both validation and testing splits of ActivityNet-1.3 in AUC and AR@100.On validation AR@100, MGG improves over BSN from 74.16 to 74.54.
- Ablation Study: The full MGG outperforms MGG-P, MGG-B, MGG-U, MGG-F, and MGG-S, supporting the effectiveness of its position embedding, bilinear matching, U-shape, SPP, and FAP components.Removing the U-shape architecture particularly reduces recall for short proposals.
- Ablation Study: End-to-end training outperforms stagewise training under the same settings, demonstrating the importance of jointly optimizing SPP and FAP with a shared BaseNet.The comparison is reported in Table 5 on ActivityNet-1.3.
- Video Detection: Using the same action classifier, MGG yields better video-detection performance than competing proposal generators, including CTAP and BSN, on THUMOS-14.The evaluation uses mAP on the testing set.
6. Conclusion
The conclusion presents MGG as an end-to-end temporal action proposal architecture with coarse and fine branches. It reports improved proposal quality and performance on THUMOS-14 and ActivityNet-1.3, along with stronger video detection using the generated proposals.
- Conclusion: MGG couples SPP for coarse whole-video segment proposals with FAP for finer frame-level actionness evaluation in an end-to-end model.The architecture analyzes videos from both coarse and fine perspectives.
- Conclusion: MGG generates proposals with high recall and more precise boundary information, outperforming state-of-the-art methods on THUMOS-14 and ActivityNet-1.3.The conclusion attributes this to analyzing whole videos from coarse and fine perspectives.
- Conclusion: Video detection using MGG-generated proposals also achieves better performance than competing methods.The conclusion presents this as further evidence of MGG’s effectiveness.
A. Supplementary Material
The supplementary experiments show that MGG improves recall for short proposals and produces more accurate refined proposals, while failures remain for very short actions and poor-quality videos.
- Recall Rates: MGG outperforms DAP, TURN, CTAP, and BSN in recall for short temporal proposals on THUMOS-14.The evaluation uses temporal spans from 1 to 60 frames with AN=100 and tIoU=0.75.
- Recall Rates: Temporal boundary adjustment improves proposal boundary accuracy and overlap with ground-truth proposals.The supplementary results identify TBA as one reason for MGG's stronger short-proposal recall.
- Recall Rates: MGG's U-shape architecture provides high-level semantic information to lower layers, helping capture short-duration proposals.
- Qualitative Results: Qualitative results on ActivityNet-1.3 and THUMOS-14 show that refined proposals have higher accuracy.
- Qualitative Results: MGG produces false negatives for ground-truth proposals with short temporal durations, and poorer video-frame quality further reduces performance.