Source-linked AI summary
Temporal Action Localization by Structured Maximal Sums
Zehuan Yuan, Jonathan C. Stroud, Tong Lu, Jia Deng
TL;DR
Temporal action localization must detect action instances and their boundaries while accounting for temporal evolution and scaling to long videos. The paper uses structured maximal sums over arbitrary-length windows with separate start, middle, and end scores, and reports competitive performance on THUMOS’14.
Problem
Temporal action localization must determine whether an action occurs and where it starts and ends, while modeling temporal evolution and scaling to arbitrarily long videos.
Method
The paper models localization as a structured maximal-sum problem over arbitrary-length windows, separately scoring start, middle, and end frames with end-to-end learned CNN features.
Results
The system achieves competitive performance on the THUMOS’14 action detection benchmark and outperforms state-of-the-art at overlap thresholds σ = 0.1, 0.2, and 0.3.
Takeaways & Limitations
Modeling temporal evolution can improve localization performance while retaining efficient temporal localization for arbitrary-length videos.
Takeaways & Limitations
The training videos are assumed to contain only a single action instance, and short or rapidly successive actions can receive low average precision or be merged.
Abstract
from arXiv · showhide
We address the problem of temporal action localization in videos. We pose action localization as a structured prediction over arbitrary-length temporal windows, where each window is scored as the sum of frame-wise classification scores. Additionally, our model classifies the start, middle, and end of each action as separate components, allowing our system to explicitly model each action's temporal evolution and take advantage of informative temporal dependencies present in this structure. In this framework, we localize actions by searching for the structured maximal sum, a problem for which we develop a novel, provably-efficient algorithmic solution. The frame-wise classification scores are computed using features from a deep Convolutional Neural Network (CNN), which are trained end-to-end to directly optimize for a novel structured objective. We evaluate our system on the THUMOS 14 action detection benchmark and achieve competitive performance.
1. Introduction
Temporal action localization must identify whether and when actions occur in videos whose actions have meaningful temporal evolution and whose lengths may be arbitrarily large. The paper addresses these challenges with structured temporal modeling, efficient localization, and end-to-end learned features.
- Actions have inherent temporal structure, but prior approaches either fail to model it or incur significant computational cost.The need is especially acute for videos that may last hours or days.
- The paper models actions through their temporal evolution while developing a provably efficient localization algorithm.Its framework represents action structure through start, middle, and end components while retaining efficient temporal localization.
- Each temporal window contains one start frame, one or more middle frames, and one end frame, without imposing restrictive assumptions on the rest of the action sequence.This captures useful temporal structure without requiring additional annotations or unrealistic composition assumptions.
- At test time, localization searches for the structured maximal sum of start, middle, and end frame-wise confidence scores, using an algorithm designed for arbitrary-length videos.The underlying naive search is quadratic over possible start-end pairs, motivating the proposed dynamic-programming solution.
- The system separately classifies action components with two-stream CNN features, trains end-to-end with a structured loss, and achieves competitive THUMOS’14 results.The paper also reports ablation studies of the pipeline components.
2. Related work
Prior temporal action localization methods primarily use sliding windows or frame-wise classification, each offering temporal information with important computational or modeling trade-offs. The paper distinguishes its end-to-end structured-window approach from these methods and related structured prediction work.
- Sliding Window: Sliding-window methods use contextual temporal information but require multiple window sizes, causing redundant computation and expensive scaling to long videos.Their success nevertheless demonstrates that contextual information is important for accurate localization.
- Frame-wise Classification: Frame-wise approaches incorporate some temporal context during post-processing but typically rely on hand-designed features or optimize frame-level losses.The paper instead directly optimizes a structured loss over temporal action windows.
- Other Approaches: The paper differs from prior approaches by training end-to-end and learning features for structured temporal-window localization.This contrasts with methods based on frame-level objectives or hand-designed frame features.
- Other Approaches: The most related prior work performs joint spatial-temporal localization using structured prediction and efficient max-path inference.The paper’s stated distinction is its modeling of the temporal evolution of actions.
3. Localization as Structured Prediction
The paper formulates localization as maximizing summed frame-wise confidence over contiguous temporal windows, then extends this formulation to separately score start, middle, and end action components. This structure supports efficient inference and aims to improve temporal boundary consistency.
- 3. Localization as Structured Prediction: A temporal window is a contiguous video segment whose confidence is the sum of signed frame-wise scores, and the predicted window maximizes that confidence.The frame-wise score may be positive or negative and can depend in practice on features from the entire video.
- 3. Localization as Structured Prediction: Because the score decomposes into frame-wise terms, the maximal-sum localization problem admits an O(n)-time solution instead of naive quadratic search over start-end pairs.The same formulation can be extended to finding k-best windows for multiple action instances.
- 3.1. Temporal Evolution Model: The temporal-evolution model assigns separate signed confidence scores to a single start frame, arbitrary-length middle frames, and a single end frame.The start and end components are emphasized because they define action-instance boundaries.
- 3.1. Temporal Evolution Model: The component weights λs, λm, and λe control the relative importance of start, middle, and end scores, and are set equally to 1 in the experiments unless stated otherwise.
- 3.1. Temporal Evolution Model: Separate component matching penalizes poor start or end matches, discourages merging adjacent or partial instances, and encourages detections spanning the full action.The formulation uses existing temporal action annotations and imposes no restrictive assumptions on complex action structure.
4. Structured Maximal Sums
Structured Maximal Sums treats detections as top-k temporal windows built from separate start, middle, and end scores, and finds them efficiently with a single-pass dynamic program.
- Problem formulation: The method represents detections as top-k temporal windows ordered by their confidence scores, using separate frame-wise start, middle, and end scores.The notation assumes these classifier scores are precomputed in ordered lists, and the algorithm can recover window endpoints with bookkeeping.
- SMS algorithm: Top-k structured temporal windows are computed by maintaining sorted complete windows and incomplete windows ending at the current frame.The algorithm updates these lists through insertion, continuation, completion, and merging operations.
- SMS algorithm: Each incomplete window at the next frame is either newly started or continued from one of the K-best incomplete windows at the previous frame.This recurrence is implemented by merging the new start score with continued windows after adding the next middle score.
- SMS algorithm: Each complete window at the next frame is either an incomplete window completed by the end score or one of the previously best complete windows.Merging these candidates preserves the K-best complete windows.
- Efficiency: The SMS algorithm computes the K-best temporal windows in O(nK) time for a video of length n.Each list update and merge takes O(K) time, and the algorithm processes all n frames.
5. Training
The training framework learns frame-wise confidence scores with two-stream CNNs and optimizes a structured objective for temporal localization. It combines localization and classification losses, with efficient loss-augmented inference during end-to-end training.
- 5.1. Network Architecture: Two-stream CNNs with VGG backbones extract spatial and motion features, average their scores, and concatenate them across frames.The spatial stream processes RGB frames, while the motion stream processes dense optical flow.
- 5. Training: The model is trained end-to-end with a video-level structured loss rather than a frame-wise recognition loss.This objective allows frame-wise scores to account for temporal action evolution during localization.
- 5.2. Structured Loss: The localization loss compares the highest-scoring temporal window with the ground-truth action window using a hinge loss and overlap-sensitive penalty.The penalty is weakened for windows with high overlap with the ground truth.
- 5.2. Structured Loss: The classification loss enforces higher scores for the ground-truth action class than for estimated windows of other actions.A distance threshold prevents penalizing detections whose distance is already below the selected margin.
- 5.2. Structured Loss: Loss-augmented inference modifies the structured maximal-sum algorithm to find the required best window in linear time during training.The resulting subgradients can be propagated through the CNN parameters using backpropagation.
6. Experiments
On THUMOS ’14, the system is evaluated against existing methods, ablations, and per-class performance, achieving state-of-the-art mAP at lower overlap thresholds and competitive results overall.
- 6.1. Results on THUMOS 2014: The evaluation uses THUMOS ’14, with training clips, background, and validation videos for training and untrimmed test videos for evaluation.THUMOS ’14 contains 20 sports action classes and 1574 untrimmed test videos.
- 6.1. Results on THUMOS 2014: The system achieves the best performance on 5 of the 20 THUMOS ’14 action classes at overlap threshold σ = 0.5.Per-class average precision is reported in Figure 5.
- 6.1. Results on THUMOS 2014: State-of-the-art mAP is achieved at overlap thresholds σ = 0.1, 0.2, and 0.3, with competitive results at σ = 0.4 and 0.5.The reported comparison uses Table 1 and evaluates mean average precision across varying overlap thresholds.
- 6.1. Results on THUMOS 2014: The full model outperforms all other configurations in ablations of the structured objective and two-stream architecture.The ablation variants include removing the structured localization loss, temporal evolution, or components of the two-stream setup.
- 6.1. Results on THUMOS 2014: Jointly training the two streams performs better than late fusion, whose confidence scores may be incompatible across the separate networks.This comparison is reported in the two-stream architecture ablation.