Source-linked AI summary
End-to-end Learning of Action Detection from Frame Glimpses in Videos
Serena Yeung, Olga Russakovsky, Greg Mori, Li Fei-Fei
TL;DR
Action detection in long videos must localize when actions occur without relying on exhaustive frame-level search. The paper uses a recurrent agent that observes selected frames, refines temporal hypotheses, and learns observation and emission policies with REINFORCE. It reports state-of-the-art results on THUMOS’14 and ActivityNet while observing 2% or less of video frames.
Problem
Long-video action detection requires determining both whether an action occurs and its temporal extent, but existing exhaustive frame-level approaches model localization indirectly and inefficiently.
Method
A recurrent neural network-based agent sequentially chooses frames to observe, refines action hypotheses, and decides when to emit predictions, using REINFORCE for non-differentiable policies.
Results
2% or less of video frames are observed while achieving state-of-the-art performance on THUMOS’14 and ActivityNet.
Takeaways & Limitations
Direct temporal-bound reasoning with learned frame-observation policies can support effective action detection with substantially less exhaustive video processing.
Takeaways & Limitations
The learned duration priors can be overly strong in some cases, producing predictions with slightly longer durations than the ground-truth action.
Abstract
from arXiv · showhide
In this work we introduce a fully end-to-end approach for action detection in videos that learns to directly predict the temporal bounds of actions. Our intuition is that the process of detecting actions is naturally one of observation and refinement: observing moments in video, and refining hypotheses about when an action is occurring. Based on this insight, we formulate our model as a recurrent neural network-based agent that interacts with a video over time. The agent observes video frames and decides both where to look next and when to emit a prediction. Since backpropagation is not adequate in this non-differentiable setting, we use REINFORCE to learn the agent's decision policy. Our model achieves state-of-the-art results on the THUMOS'14 and ActivityNet datasets while observing only a fraction (2% or less) of the video frames.
1. Introduction
The paper frames action detection as direct temporal-bound prediction through iterative observation and hypothesis refinement, replacing exhaustive frame-level search with a learned agent policy. The model jointly learns where to observe and when to emit predictions, achieving strong benchmark performance while examining at most 2% of video frames.
- Long-video action detection requires identifying both whether an action occurs and its temporal extent, while exhaustive frame-level classification with post-processing is computationally inefficient and indirect.
- The proposed approach directly predicts action temporal bounds by sequentially observing frames and refining hypotheses about action locations.The observation-and-refinement process can skip forward or backward to narrow localization.
- The model is a recurrent neural network-based agent trained end-to-end with backpropagation and REINFORCE to form and refine detection hypotheses.
- The agent decides both which frame to observe next and when to emit a prediction, addressing the structured, variable-sized outputs of action detection.
- 2% or less of video frames are observed while the model achieves state-of-the-art performance on THUMOS’14 and ActivityNet.
2. Related Work
Prior work in video analysis and temporal action detection spans handcrafted and learned features, sliding-window systems, web-image priors, and structured action grammars. Related spatio-temporal methods reason over proposals, human tracks, surveillance scenes, or multi-camera settings.
- Temporal action detection methods commonly use dense trajectories, frame-level CNN features, and sound features within sliding-window frameworks on unconstrained, untrimmed videos.
- Related approaches improve temporal detection with hand-centric or object-centric features, web-image priors, or grammars that model complex actions and their temporal sub-components.
- Spatio-temporal action detection research includes action proposals, human tracks, group reasoning, multi-camera analysis, and rule-based quadratic-programming formulations.
3. Method
The method treats action detection as sequential observation and hypothesis refinement: a recurrent agent chooses where to look next and when to emit temporal action predictions. It combines differentiable candidate learning with REINFORCE for non-differentiable observation and emission decisions.
- Model overview: The agent receives video frames, processes a fixed proportion of them, and learns to use these frame glimpses to reason about action temporal bounds.An observation network encodes frames, while a recurrent network processes the resulting observations sequentially.
- Observation network: The observation network encodes each observed frame together with its normalized temporal location into a feature vector for the recurrent network.The frame representation can include both what was seen and where in the video it was observed.
- Recurrent network: At each timestep, the recurrent network produces a candidate detection, a prediction indicator, and the temporal location of the next observation.The next location may move either forward or backward through the video, allowing sequential observation and refinement.
- Candidate detections and emission: Candidate detections encode action hypotheses as normalized start, end, and confidence values, while the prediction indicator determines when a candidate enters the prediction set.Separating candidate formation from emission reduces noise and false positives; their combination makes non-maximum suppression a learnable component.
- Candidate detection training: Candidate detections are matched to nearby ground-truth instances and optimized with classification and localization losses using backpropagation.The confidence uses cross-entropy, and matched segment boundaries use an L2 regression loss.
- Policy learning: REINFORCE trains the non-differentiable observation-location and prediction-indicator policies by increasing action probabilities associated with higher future reward.The policy conditions decisions on the interaction history, including previous states and actions.
4. Experiments
Experiments on THUMOS’14 and ActivityNet show that the end-to-end model outperforms existing methods while observing 2% or less of video frames. Ablations and qualitative examples indicate that learned frame selection and direct temporal reasoning are central to precise action localization.
- Overall results: 2% or less of video frames were processed while the model outperformed existing methods on THUMOS’14 and ActivityNet.The evaluation covered both datasets, with comparisons against existing work.
- THUMOS’14 results: The model outperformed existing methods at all IOU thresholds α on THUMOS’14.Its relative margin increased as α decreased, indicating more frequent detection near ground-truth instances despite imperfect localization.
- Ablation experiments: Removing learned observation locations lowered performance because uniform sampling provided insufficient resolution for localizing action boundaries.Removing the observation policy harmed performance more than removing the prediction indicator.
- Ablation experiments: Removing localization regression produced the poorest ablation performance at α = 0.5, even below the CNN baseline.This result highlights the contribution of temporal regression to action detection.
- Per-class breakdown: The model outperformed the THUMOS’14 leaderboard comparison method on 12 out of 20 classes.Improvements were notable for challenging classes including basketball dunk, diving, and frisbee catch.
- Observation policy analysis: Learned policies increased observation frequency near action onset and sometimes stepped backward near boundaries to refine predictions.The examples include javelin throw and diving, with the diving sequence containing two closely spaced instances.
- Observation policy analysis: The diving example separated two rapidly successive instances despite difficult frame-level CNN signals, but its predictions were somewhat longer than ground truth.Motion blur complicated one observation, while the model still discerned the separate instances.
- ActivityNet results: The model outperformed ActivityNet comparison work in 13 out of 21 Sports classes and 10 out of 15 Work classes.The improvement was particularly large on the Work subset, where activities had weaker, more diffuse frame-level CNN probabilities.
5. Conclusion
The approach achieves state-of-the-art action-detection results on THUMOS’14 and ActivityNet while directly predicting temporal bounds and observing only a fraction of video frames.
- State-of-the-art results are achieved on the THUMOS’14 and ActivityNet action-detection datasets.
- The model directly predicts the temporal bounds of action instances.
- The ActivityNet Sports and Work subsets report per-class breakdowns and mAP at IOU α = 0.5.
- The method observes only a fraction of video frames while performing action detection.