PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoLoc: Weakly-supervised Temporal Action Localization

Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, Shih-Fu Chang

arXiv:1807.08333v2cs.CV

TL;DR

Weakly-supervised TAL avoids costly temporal-boundary annotations but prior methods mainly threshold noisy CAS outputs. AutoLoc directly predicts action boundaries and uses OIC loss to discover segment-level supervision, improving mAP on THUMOS’14 and ActivityNet while reaching performance comparable to some fully supervised methods.

  • Problem

    Weakly-supervised TAL must localize action boundaries using only video-level training labels, while CAS thresholding is vulnerable to activation noise.

  • Method

    AutoLoc directly predicts temporal boundaries and uses a differentiable OIC loss to automatically discover segment-level supervision from video-level annotations.

  • Results

    At IoU 0.5, mAP rises from 13.7% to 21.2% on THUMOS’14 and from 7.4% to 27.3% on ActivityNet, comparable to some fully supervised methods.

  • Takeaways & Limitations

    AutoLoc demonstrates that weakly-supervised TAL can directly predict segment boundaries and substantially outperform prior weakly-supervised methods on two standard benchmarks.

  • Takeaways & Limitations

    A visualization example reports that an AutoLoc ending boundary was not well aligned with the annotation, although the underlying action ending was ambiguous.

Abstract

from arXiv · show

Temporal Action Localization (TAL) in untrimmed video is important for many applications. But it is very expensive to annotate the segment-level ground truth (action class and temporal boundary). This raises the interest of addressing TAL with weak supervision, namely only video-level annotations are available during training). However, the state-of-the-art weakly-supervised TAL methods only focus on generating good Class Activation Sequence (CAS) over time but conduct simple thresholding on CAS to localize actions. In this paper, we first develop a novel weakly-supervised TAL framework called AutoLoc to directly predict the temporal boundary of each action instance. We propose a novel Outer-Inner-Contrastive (OIC) loss to automatically discover the needed segment-level supervision for training such a boundary predictor. Our method achieves dramatically improved performance: under the IoU threshold 0.5, our method improves mAP on THUMOS'14 from 13.7% to 21.2% and mAP on ActivityNet from 7.4% to 27.3%. It is also very encouraging to see that our weakly-supervised method achieves comparable results with some fully-supervised methods.

1 Introduction

AutoLoc addresses weakly-supervised temporal action localization, where training uses only video-level labels but testing requires action classes and temporal boundaries. It directly predicts boundaries using OIC loss and substantially improves benchmark performance over weakly-supervised methods.

  • Fully supervised TAL requires expensive, time-consuming temporal-boundary annotations for long untrimmed videos.
  • Weakly-supervised TAL uses easier video-level labels during training while still predicting each action instance’s class and start and end times.
  • CAS thresholding can over-segment actions when activations dip or include irrelevant background when the threshold is too low.
  • AutoLoc directly predicts action boundaries by estimating each instance’s center location and duration instead of thresholding CAS.
  • OIC loss automatically discovers segment-level supervision from video-level annotations by contrasting activations inside predicted boundaries with surrounding activations.
  • At IoU 0.5, mAP improves from 13.7% to 21.2% on THUMOS’14 and from 7.4% to 27.3% on ActivityNet.

2 Related Works

Prior TAL research includes fully supervised segment-level localization and several weak-supervision settings. Video-level supervision is attractive because boundary annotations require substantial effort, while other work explores ordering, point-level labels, and weakly supervised vision methods.

  • Video analysis backbones and sequence models include 3D ConvNets, LSTM, two-stream networks, I3D, and Temporal Segment Networks.
  • Large-scale TAL datasets require substantial effort to annotate temporal boundaries for fully supervised training.
  • Video-level annotations are easier to collect and have been used for weakly-supervised TAL, including methods that discover additional supervision from web images.
  • Fully supervised TAL methods localize at the segment level through candidate proposals or direct single-shot boundary prediction.
  • Other weak-supervision approaches use action order or point-level supervision for action detection.
  • Weakly-supervised deep learning has also been studied for image object detection, semantic segmentation, video captioning, and visual relation detection.

3 Outer-Inner-Contrastive Loss

The OIC loss evaluates predicted action segments by comparing average CAS activation in an inflated outer region with activation inside the predicted inner boundary. Its differentiability allows this discovered supervision to train boundary predictions.

  • AutoLoc computes OIC loss during the forward pass and proves it differentiable for back-propagation through the boundary predictor.
  • For each predicted segment, OIC loss uses an inner action boundary, an inflated outer boundary, and an action class.
  • The loss is average outer-region activation minus average inner-region activation, favoring high activation inside and low activation outside.
  • Boundary values are rounded to the nearest snippet to retrieve class activations from the CAS.
  • The loss gradients provide physical boundary-adjustment signals based on activations at inner and outer endpoints relative to regional averages.

4 AutoLoc

AutoLoc combines a classification branch that generates CAS with a localization branch that directly predicts class-agnostic action boundaries. Its pipeline generates anchors, regresses inner boundaries, inflates them to outer boundaries, and uses OIC to train and filter segment predictions.

  • Input Data Preparation and Feature Extraction: AutoLoc first trains an UntrimmedNet backbone and extracts concatenated 2048-dimensional spatial and temporal features for each video snippet.The input video is divided into non-overlapping 15-frame snippets before feature extraction.
  • Classification Branch: The classification branch generates a Class Activation Sequence by combining action-category and snippet-attention predictions.The activation generator is built on UntrimmedNet and produces category scores together with attention scores for snippets.
  • Localization Branch: The localization branch directly predicts segment boundaries by applying a temporal model to each feature-map position and multiple anchor scales.Each anchor receives regression outputs for its center location and temporal length.
  • Boundary Prediction Procedure: Boundary prediction sequentially generates anchors, regresses inner boundaries, and inflates them to obtain outer boundaries.Regression uses tx to shift anchor centers and tw to scale anchor lengths; inflation expands the inner boundary by ratio α.
  • OIC Layer: The OIC layer evaluates candidate segments against CAS, removes unlikely segments during inference, and back-propagates the total OIC loss during training.The loss is computed from kept segment predictions and updates the underlying boundary predictor.
  • Final Predictions: At test time, each kept prediction outputs an action class, confidence score, and start and end times derived from its inner boundary.Confidence is set to 1 minus the OIC loss, and snippet-level boundaries are converted to continuous video time.

5 Experiments

Experiments evaluate AutoLoc on THUMOS’14 and ActivityNet using weak video-level supervision, standard mAP metrics, comparisons, and ablations. Results support direct boundary prediction and the OIC loss, including robustness to noisy CAS.

  • Experimental Setup: AutoLoc is evaluated on THUMOS’14 and ActivityNet, with training using video-level labels and testing using temporal localization metrics.THUMOS’14 uses 200 validation videos for training and 213 test videos; ActivityNet v1.2 uses 4,819 training and 2,383 validation videos.
  • Experimental Setup: Predictions are scored by mAP, requiring both the correct action class and temporal overlap IoU above the evaluation threshold.Each prediction includes an action class, start time, end time, and confidence score.
  • State-of-the-art Comparisons: AutoLoc significantly outperforms weakly-supervised TAL methods on THUMOS’14 and is comparable to some fully-supervised methods.The method directly predicts segment boundaries while incorporating contextual information, rather than relying on simple CAS thresholding.
  • State-of-the-art Comparisons: AutoLoc achieves substantial improvements on ActivityNet v1.2 over the compared methods.The comparison is reported in Table 2; Wang et al.’s localization results were additionally evaluated from released models and source code.
  • OIC Loss Analysis: OIC Selection improves over state-of-the-art weakly-supervised results by favoring segments with high inner and low outer activations.The selected low-OIC-loss segments are usually well aligned with ground-truth segments.
  • OIC Loss Analysis: Removing the outer-inner contrast causes a large performance drop, showing that contextual contrast is important for training the boundary predictor.The Inner Only Loss encourages high activations inside the segment but ignores the surrounding contextual area.
  • OIC Loss Analysis: Direct Optimization performs worse than AutoLoc because optimizing boundaries against each testing video’s potentially inaccurate CAS can overfit its noise.AutoLoc trains on multiple videos, making its boundary predictor more robust to CAS noise.

6 Conclusion and Future Works

The paper presents AutoLoc as a weakly-supervised TAL framework that directly predicts temporal boundaries in a single shot. It uses OIC loss to provide segment-level supervision without boundary annotations.

  • AutoLoc directly predicts temporal boundaries in a single-shot weakly-supervised TAL framework.
  • The OIC loss provides segment-level supervision when only video-level annotations are available.

8 Supplementary Materials

Supplementary visualizations show that AutoLoc handles noisy Class Activation Sequences more robustly than simple thresholding, while its OIC loss contrasts activations inside and outside predicted segments.

  • 8.1 Visualization Examples: AutoLoc correctly localizes action instances when thresholding either over-segments one action or merges consecutive instances.These behaviors are illustrated on PoleVault and GolfSwing examples from THUMOS’14.
  • 8.1 Visualization Examples: Low PoleVault activations caused by incomplete camera coverage make thresholding split one ground-truth segment, whereas AutoLoc predicts it as a whole.AutoLoc was trained to predict at the segment level rather than relying only on thresholded CAS responses.
  • 8.1 Visualization Examples: High GolfSwing activations during a scene transition make thresholding merge two consecutive instances, while AutoLoc separates them.The transition occurs after the first instance ends and before the second begins.
  • 8.2 Detailed Derivation of the OIC Back-propagation: When boundaries exceed the video range, the supplementary derivation explains gradients for the inner and outer boundaries after clipping.The displayed gradients correspond to boundary-specific back-propagation terms.
  • 8.2 Detailed Derivation of the OIC Back-propagation: The OIC loss is the average activation in the outer area minus the average activation in the inner area of a predicted segment.Each prediction has inner boundary [x1, x2] and inflated outer boundary [X1, X2].
  • 8.2 Detailed Derivation of the OIC Back-propagation: OIC effectively convolves CAS with a zero-integral step function, favoring segment centers with high inner and low outer activations.The minimum loss is achieved at the center of a likely action segment under this contrast.

B A C

The clipping supplement explains why zero-padding is used when predicted boundaries exceed the video range and how this preserves useful optimization behavior.

  • B A C: Clipping a predicted inner boundary to [1, T] can produce an undesirable gradient direction because endpoint snippets may have nonzero activations.The implementation treats [x1, x2] as including the snippets at both boundaries.
  • B A C: The zero-padding scheme allows optimization to move an oversized predicted boundary back toward candidate locations inside the video.This is the stated purpose of the clipping design when predictions exceed the video range.
  • B A C: Inflating inner boundaries provides contextual information, but very short segments can make rounded inner and outer boundaries coincide.The supplement addresses this issue by extending the outer boundary by at least one snippet.

Details of Inflation

For very short predicted segments, rounding can collapse the inner and outer boundaries, so the OIC layer enforces a minimum outer-boundary extension.

  • Details of Inflation: The OIC layer extends the outer boundary by at least one snippet when rounding makes it coincide with the inner boundary.This preserves a contrast between inner and contextual outer areas.
  • Details of Inflation: Algorithm 1 summarizes how the OIC layer decides whether to retain each candidate segment prediction during training.Its inputs include CAS and candidate segment predictions.

Exploration Study

The exploration study selects the inflation ratio α through grid search and compares OIC with an inner-only alternative for weakly supervised localization.

  • Exploration Study: The inflation ratio α is explored by hypothesizing a reasonable range and conducting grid search on training data.The ratio controls how far the inner boundary is inflated to obtain contextual outer boundaries.
  • Exploration Study: α should remain between 1/8 and 1/2 to balance contextual coverage against inclusion of irrelevant area.The supplement motivates the lower and upper bounds using contextual sufficiency and noise avoidance.
  • Exploration Study: Across the explored range, results are acceptable and comparable on THUMOS’14 and ActivityNet, with α = 1/4 slightly better overall.The paper therefore uses α = 1/4 in all experiments.
  • Exploration Study: The Inner Only Loss encourages high activations inside the segment without considering the contextual outer area.It is examined as an alternative to the OIC loss.

Formulation of the Inner Only Loss

The Inner Only Loss uses activation inside a predicted action interval, while the broader comparison shows why direct anchor-based optimization is preferable to enumerating segments for long videos.

  • Formulation of the Inner Only Loss: The Inner Only Loss considers only the predicted inner boundary when computing gradients.The gradients are back-propagated with respect to the inner boundary.
  • Direct Optimization versus OIC Selection: Direct Optimization outperforms OIC Selection even without training on the training videos.This supports the advantage of combining anchor mechanisms with boundary regression over enumeration with OIC selection.
  • Direct Optimization versus OIC Selection: Direct Optimization generates a small set of anchors at each CAS position and selects the anchor with the lowest OIC loss.OIC Selection instead enumerates a large set of possible segments before selecting candidates.
  • Direct Optimization versus OIC Selection: OIC Selection can retain multiple false alarms because distinct candidates may survive selection and NMS despite differing substantially in overlap.The example describes candidates with overlap IoU 0.2 that are both likely to be kept.
  • Computational consideration: Enumeration is impractical for long untrimmed videos because of its speed cost.TAL videos are usually untrimmed and long, making this computational constraint especially relevant.
Loading 1807.08333v2…