PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs

Zheng Shou, Dongang Wang, Shih-Fu Chang

arXiv:1601.02129v2cs.CV

TL;DR

Temporal action localization seeks action start and end times in unconstrained long videos that may contain multiple actions and background content. Segment-CNN uses three stages of 3D ConvNets, including overlap-aware localization. It reports higher mAP than state-of-the-art systems on MEXaction2 and THUMOS 2014 at an overlap threshold of 0.5.

  • Problem

    Untrimmed videos may contain multiple action instances alongside background scenes or other activities, making it important to determine when a specific action starts and ends.

    Source: §1 · p. 1

  • Method

    Segment-CNN uses 3D ConvNets to propose candidate segments, train action classifiers to initialize localization, and localize actions with a loss that considers temporal overlap.

    Source: §1 · p. 1

  • Results

    At an overlap threshold of 0.5, mAP increases from 1.7% to 7.4% on MEXaction2 and from 15.0% to 19.0% on THUMOS 2014.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation supports Segment-CNN as an effective multi-stage framework for temporal action localization in untrimmed long videos.

    Source: §5 · p. 8

    The stated training setup uses both a set of trimmed videos and a set of untrimmed videos.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

We address temporal action localization in untrimmed long videos. This is important because videos in real applications are usually unconstrained and contain multiple action instances plus video content of background scenes or other activities. To address this challenging issue, we exploit the effectiveness of deep networks in temporal action localization via three segment-based 3D ConvNets: (1) a proposal network identifies candidate segments in a long video that may contain actions; (2) a classification network learns one-vs-all action classification model to serve as initialization for the localization network; and (3) a localization network fine-tunes on the learned classification network to localize each action instance. We propose a novel loss function for the localization network to explicitly consider temporal overlap and therefore achieve high temporal localization accuracy. Only the proposal network and the localization network are used during prediction. On two large-scale benchmarks, our approach achieves significantly superior performances compared with other state-of-the-art systems: mAP increases from 1.7% to 7.4% on MEXaction2 and increases from 15.0% to 19.0% on THUMOS 2014, when the overlap threshold for evaluation is set to 0.5.

1. Introduction

Temporal localization must find action boundaries in unconstrained long videos, where multiple actions and background content coexist. Segment-CNN uses multi-stage 3D ConvNets and an overlap-aware localization loss; it reports higher mAP than state-of-the-art systems on two benchmarks.

1. Introduction

  • The paper identifies reliance on manually selected features and the lack of motion information in 2D ConvNets as limitations of prior approaches.The authors use 3D ConvNets to capture motion characteristics in video.Source: §1 · p. 1
  • Segment-CNN uses 3D ConvNets in a multi-stage framework to propose candidate segments, classify actions, and localize temporal boundaries.The proposal network removes unlikely candidates; classification initializes localization, which is used with the proposal network at prediction.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • A novel localization loss rewards segments with greater temporal overlap, producing confidence scores better suited to post-processing than classification scores alone.Classification may score a segment highly despite little overlap with the ground-truth action, potentially causing NMS to discard a better-localized segment.Source: §1 · pp. 1–2
  • The authors present their work as the first use of 3D ConvNets with multi-stage processing for temporal localization in untrimmed long videos.They describe the proposal network as improving efficiency and the localization network as key to boosting temporal accuracy.Source: §1 · p. 2
  • At an overlap threshold of 0.5, mAP rises from 1.7% to 7.4% on MEXaction2 and from 15.0% to 19.0% on THUMOS 2014.The authors also state that they did not evaluate on THUMOS Challenge 2015 because its ground truth is withheld for future evaluation.Source: §1 · p. 2

2. Related work

Prior work on temporal action localization includes weakly supervised methods and approaches using temporally annotated videos. Related work also covers simultaneous spatial-temporal localization and object-detection methods, which differ in their targets, annotation needs, and localization strategies.

Temporal action localization

  • Weakly supervised localization uses video-level category labels without temporal annotations, while another line of work learns from temporally annotated action boundaries.The former selects key instances; the latter commonly treats sliding windows as action candidates for classification.Source: §2 · p. 2
  • For temporally annotated videos, leading approaches included dense trajectory features and CNN-based video representations, alongside work on efficiency and temporal precision.Actom-based methods model action-sequence structure but require manual actom annotations, which the authors describe as subjective and burdensome.Source: §2 · pp. 2–3

Spatio-temporal localization

  • Spatiotemporal localization outputs action regions in space and time, whereas temporal localization needs only action start and end times.The cited spatiotemporal approaches assume human or predefined objects and require exhaustive per-frame object annotations.Source: §2 · p. 3

Object detection

  • Unlike R-CNN-style object detection, this work addresses diverse temporal annotations, learns action-versus-background proposals, and replaces time-shift regression with an overlap-aware loss.The authors report that regression for segment time shift and duration did not work well in their experiments, possibly because actions lack consistent start/end patterns.Source: §2 · p. 3

3. Detailed descriptions of Segment-CNN

Segment-CNN processes multi-scale video segments through proposal, classification, and localization stages, with overlap-aware training for localization. At prediction, proposal filtering precedes localization scoring and NMS.

3.1. Problem setup

  • Untrimmed videos yield multi-scale sliding-window candidates (16–512 frames, 75% overlap), sampled into 16-frame inputs for 3D ConvNets initialized from Sports-1M C3D.The network architecture follows C3D and uses 3D convolutions and pooling across spatial and temporal dimensions.Source: §3.1 · pp. 3–4

3.2. Training procedure

  • The proposal network filters background segments; the classification network then predicts K action categories plus background and initializes localization training.The proposal network has two output nodes, for background and being-action segments.Source: §3.2 · pp. 4–5
  • Proposal training assigns positives above IoU 0.7 and backgrounds below 0.3, with a fallback positive above 0.5 for a ground-truth instance lacking a higher-IoU segment.The training set also includes randomly sampled background segments, approximately matching the number of positive segments.Source: §3.2 · p. 4
  • The localization network is fine-tuned from classification weights and combines softmax with an overlap-aware loss that rewards higher-IoU positive segments.For positive untrimmed segments, the overlap measurement is their IoU with the associated ground-truth instance; for trimmed-video segments, it is set to 1.Source: §3.2 · pp. 4–5

3.3. Prediction and post-processing

  • At prediction, windows are filtered by proposal score ≥0.7, scored by the localization network, and redundant detections are removed by NMS.The NMS overlap threshold is set to θ − 0.1, where θ is the evaluation overlap threshold.Source: §3.3 · pp. 5–6

4. Experiments

Experiments evaluate Segment-CNN on MEXaction2 and THUMOS 2014 and compare its component networks. On THUMOS 2014, it improves mAP over other systems and the classification and localization networks improve localization performance.

4.1. Datasets and setup

  • MEXaction2 includes two action classes and both trimmed clips and approximately 77 hours of untrimmed INA video; THUMOS 2014 evaluates 20 actions in untrimmed videos.MEXaction2 has 1,336 training, 310 validation, and 329 test instances; THUMOS 2014 uses trimmed UCF101 videos for training and annotated untrimmed videos for validation and testing.Source: §4.1 · p. 6

4.2. Comparison with state-of-the-art systems

  • They evaluate temporal action localization as a retrieval problem using average precision; a prediction is correct only with the correct category and IoU above the overlap threshold, and redundant detections are disallowed.For THUMOS 2014, the authors re-evaluate all teams’ submission results using the updated toolkit.Source: §4.2 · p. 6
  • On MEXaction2, Segment-CNN achieves a large performance gain for BullChargeCape and competitive performance for HorseRiding against dense trajectory features.The evaluation overlap threshold is 0.5; the supplied table caption identifies average precision but gives no cell values.Source: §4.2 · p. 6, Table 1 · p. 6
  • mAP rises from 15.0% to 19.0% on THUMOS 2014, with Segment-CNN outperforming other systems for 14 of 20 actions.The comparison uses the updated evaluation toolkit; Table 2 reports mAP across overlap IoU thresholds.Source: Table 2 · p. 6, §4.2 · p. 6

4.3. Impact of individual networks

  • Keeping the proposal network improves precision by filtering background segments and reduces computation on them.Table 3 compares mAP with and without the proposal network at overlap threshold 0.5.Source: §4.3 · p. 7, Table 3 · p. 7
  • Across top-κ selection depths, fine-tuning localization on the classification network outperforms removing classification, while localization improves over the no-localization baseline.The localization loss favors segments with greater overlap with ground truth; performance is stable for overlap-loss α values from 0.25 to 1.0.Source: §4.3 · pp. 7–8, Figure 7 · p. 7

5. Conclusion

Segment-CNN addresses temporal action localization in untrimmed long videos. The conclusion highlights candidate proposal, classification initialization, and an overlap-aware loss for localizing action instances.

5. Conclusion

  • Segment-CNN uses candidate proposal, classification initialization, and a new localization loss to precisely localize action instances in time.The conclusion describes evaluations supporting each network’s contribution, the classification network’s role in initializing localization, and the loss’s effectiveness.Source: §5 · p. 8
  • The authors identify extending the framework to events and activities composed of multiple actions as future work.Source: §5 · p. 8
  • Precisely localizing action instances in time can help recognize and detect events and activities.Source: §5 · p. 8
Loading 1601.02129v2…