PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Shot Temporal Action Detection

Tianwei Lin, Xu Zhao, Zheng Shou

arXiv:1710.06236v1cs.CV

TL;DR

Temporal action detection must locate action categories and boundaries in long, untrimmed videos, but proposal-based methods fix proposal boundaries during classification. The paper introduces SSAD, a temporal-convolutional single-shot detector that directly predicts action instances, and reports higher mAP on THUMOS 2014 and MEXaction2.

  • Problem

    Temporal action detection requires recognizing action categories and temporal boundaries in untrimmed videos, while proposal-based methods classify proposals with fixed boundaries.

  • Method

    SSAD uses a multi-granularity temporal convolutional network to directly predict action boundaries and confidence scores without proposal generation, while exploring architectures and feature configurations.

  • Results

    At IoU 0.5, SSAD increases mAP from 19.0% to 24.6% on THUMOS’14 and from 7.4% to 11.0% on MEXaction2.

  • Takeaways & Limitations

    SSAD directly predicts action instances in untrimmed video and significantly outperforms compared state-of-the-art systems on both reported benchmarks.

Abstract

from arXiv · show

Temporal action detection is a very important yet challenging problem, since videos in real applications are usually long, untrimmed and contain multiple action instances. This problem requires not only recognizing action categories but also detecting start time and end time of each action instance. Many state-of-the-art methods adopt the "detection by classification" framework: first do proposal, and then classify proposals. The main drawback of this framework is that the boundaries of action instance proposals have been fixed during the classification step. To address this issue, we propose a novel Single Shot Action Detector (SSAD) network based on 1D temporal convolutional layers to skip the proposal generation step via directly detecting action instances in untrimmed video. On pursuit of designing a particular SSAD network that can work effectively for temporal action detection, we empirically search for the best network architecture of SSAD due to lacking existing models that can be directly adopted. Moreover, we investigate into input feature types and fusion strategies to further improve detection accuracy. We conduct extensive experiments on two challenging datasets: THUMOS 2014 and MEXaction2. When setting Intersection-over-Union threshold to 0.5 during evaluation, SSAD significantly outperforms other state-of-the-art systems by increasing mAP from 19.0% to 24.6% on THUMOS 2014 and from 7.4% to 11.0% on MEXaction2.

1 INTRODUCTION

Temporal action detection must identify action categories and temporal boundaries in long, untrimmed videos. SSAD addresses limitations of proposal-based detection by directly predicting multi-scale action instances and achieves higher mAP on two benchmarks.

  • Motivation: Temporal action detection identifies action categories and temporal boundaries for multiple instances in untrimmed videos.Such methods can support applications including surveillance video analysis and intelligent home care.
  • Motivation: Proposal-based detection separates proposal generation from classification, leaving proposal boundaries fixed during classification.The framework is commonly trained in separate stages, although joint training would be preferable.
  • Approach: SSAD skips proposal generation and directly predicts boundaries and confidence scores for multiple action categories using a multi-granularity temporal convolutional network.Its base, anchor, and prediction layers respectively shorten feature sequences, associate features with anchor instances, and predict categories, offsets, and overlap scores.
  • Approach: SAS features concatenate snippet-level category probabilities from multiple action classifiers and serve as SSAD inputs.The classifiers provide multiple granularities of action-recognition features.
  • Study design: The study searches SSAD architectures and evaluates input-feature and post-processing configurations to improve temporal action detection.The paper conducts experiments on THUMOS’14 and MEXaction2.
  • Results: At IoU 0.5, SSAD increases mAP from 19.0% to 24.6% on THUMOS’14 and from 7.4% to 11.0% on MEXaction2.These results are reported as outperforming other state-of-the-art systems.

2 RELATED WORK

Related work frames temporal action detection as temporal object detection and reviews proposal-based, recurrent, convolutional, and single-shot approaches. SSAD follows single-shot detection by predicting action instances without proposal generation.

  • Action recognition: Action-recognition models provide features for temporal detection, including two-stream networks that use frames and stacked optical flow and C3D networks using 3D convolutions.These models are reviewed as feature-extraction components for action detection.
  • Temporal action detection: Temporal action detection parallels object detection by locating and classifying multiple action instances over time.Datasets such as THUMOS 2014 and MEXaction2 contain untrimmed videos with multiple categories and complex backgrounds.
  • Temporal action detection: Many temporal methods use extracted features with SVMs to classify segment proposals or sliding windows, while others focus specifically on proposal generation.SSAD differs by containing no proposal-generation step.
  • Sequence modeling: RNN-based approaches encode feature sequences for per-frame action predictions, whereas temporal convolution offers an alternative for sequence encoding.The related-work passage notes that RNNs have difficulty retaining long time-period memory in practice.
  • Object detection: R-CNN methods generate region proposals before classification, sharing the proposal-based detection pattern adapted by many temporal action methods.The analogy is between spatial regions in images and temporal segments in videos.
  • Object detection: Single-shot detectors predict objects directly: YOLO jointly predicts category probabilities, confidence scores, and location offsets, while SSD uses multi-scale feature maps and default boxes.SSAD combines characteristics of these single-shot detection methods for temporal action detection.

3 OUR APPROACH

SSAD processes snippet-level action-score sequences with a 1D temporal convolutional network that directly predicts action instances using multi-scale temporal feature maps and anchors.

  • Extracting of Snippet-level Action Scores: Snippet-level action scores concatenate category probabilities from multiple action classifiers and form the input feature sequence for SSAD.The classifiers include two-stream and C3D networks; each score vector includes action categories and background.
  • SSAD Network: SSAD uses base, anchor, and prediction layers, with temporal convolution and pooling shortening feature maps while expanding receptive fields.ReLU activations and temporal max pooling are used outside the convolutional prediction layers.
  • SSAD Network: The base-layer search varies temporal shortening operations, network depth, and layer sizes while keeping anchor and prediction layers fixed.Five base-layer architectures are designed because no widely used pre-trained 1D temporal convolutional model exists.
  • SSAD Network: Three anchor convolutional layers progressively reduce temporal resolution, enabling predictions from multiple-resolution feature maps.Their output sizes are Tw/32 × 512, Tw/64 × 512, and Tw/128 × 512.
  • SSAD Network: Lower anchor layers target short actions and higher layers target long actions using feature maps with different resolutions and receptive fields.Each feature-map cell is associated with multiple-scale anchor instances.
  • SSAD Network: Convolutional prediction layers simultaneously output class scores, overlap scores, and center- and width-offsets for each anchor instance.The prediction vector contains K′ classification scores, one overlap score, and two location offsets; offsets adjust anchor centers and widths.

4 EXPERIMENTS

Experiments evaluate SSAD on THUMOS’14 and MEXaction2, comparing datasets, features, architectures, post-processing, and state-of-the-art systems. SSAD achieves higher mAP at IoU 0.5 on both benchmarks and benefits from complementary classifiers and selected architectural and post-processing configurations.

  • Dataset and setup: SSAD is evaluated on THUMOS’14 and MEXaction2 using Average Precision per category and mean Average Precision under temporal IoU thresholds.THUMOS’14 uses 20 temporally annotated action categories, while MEXaction2 contains HorseRiding and BullChargeCape.
  • Comparison with state-of-the-art systems: 24.6% mAP is achieved by SSAD on THUMOS’14 at IoU 0.5, compared with 19.0% for the state-of-the-art result.SSAD outperforms other state-of-the-art methods for 7 of 20 action categories.
  • Comparison with state-of-the-art systems: 11.0% mAP is achieved by SSAD on MEXaction2 at overlap threshold 0.5, compared with 7.4%.The reported gain occurs across all action categories in the comparison.
  • Model analysis: Combining two-stream and C3D classifiers produces the best THUMOS’14 performance among the evaluated action-classifier variants.The passage attributes this result to the complementary SAS features produced by the two classifier types.
  • Model analysis: Architecture B performs best among the evaluated SSAD base-layer configurations, with max pooling and kernel size 9 identified as favorable choices.The configurations share input and output sizes, enabling a fair comparison.
  • Model analysis: The complete post-processing strategy gives the best SSAD performance, with pclass outperforming the averaged SAS category score and pover further improving performance.The evaluation compares multiple late-fusion strategies for generating the final prediction score.

5 CONCLUSION

The paper concludes that SSAD directly predicts action instances without proposal generation and performs strongly on both evaluated datasets. It also identifies separate feature extraction and detection as a design characteristic, with end-to-end learning from raw video proposed as future work.

  • 5 CONCLUSION: SSAD directly predicts action instances in untrimmed video while omitting proposal generation.The network predicts temporal boundaries and confidence scores for action instances.
  • 5 CONCLUSION: 24.6% and 11.0% mAP are reported at IoU 0.5 on THUMOS’14 and MEXaction2, respectively.The corresponding comparison values are 19.0% and 7.4%.
  • 5 CONCLUSION: Feature extraction and action detection are conducted separately, allowing SSAD to handle concise high-level features and be easily trained.The paper proposes combining these procedures into an end-to-end framework trained directly from raw video.
Loading 1710.06236v1…