Source-linked AI summary
SMART Frame Selection for Action Recognition
Shreyank N Gowda, Marcus Rohrbach, Laura Sevilla-Lara
TL;DR
Action recognition is computationally expensive, and the paper asks whether frame selection can improve accuracy even for short, trimmed videos. It proposes SMART, which selects frames jointly using attention and relational information, and reports improved accuracy with substantially lower computation across trimmed and untrimmed benchmarks. The paper also notes that potential applications require analysis of algorithmic bias.
Problem
The paper addresses frame selection for computationally expensive action recognition in trimmed videos, where selecting useful frames must account for their diversity and temporal distribution.
Method
SMART combines individual frame scores with relational and temporal attention so frames are evaluated jointly rather than independently.
Results
SMART outperforms baselines across three trimmed action-classification datasets, reduces computation by up to 4 times, and improves accuracy over prior frame-selection approaches on untrimmed videos.
Takeaways & Limitations
Joint frame selection can serve as an accuracy-improving and computationally efficient step for action recognition across trimmed and untrimmed settings.
Takeaways & Limitations
Potential applications require analysis of algorithmic biases from training data and inherited pretrained models; harmful-content data were not evaluated.
Abstract
from arXiv · showhide
Action recognition is computationally expensive. In this paper, we address the problem of frame selection to improve the accuracy of action recognition. In particular, we show that selecting good frames helps in action recognition performance even in the trimmed videos domain. Recent work has successfully leveraged frame selection for long, untrimmed videos, where much of the content is not relevant, and easy to discard. In this work, however, we focus on the more standard short, trimmed action recognition problem. We argue that good frame selection can not only reduce the computational cost of action recognition but also increase the accuracy by getting rid of frames that are hard to classify. In contrast to previous work, we propose a method that instead of selecting frames by considering one at a time, considers them jointly. This results in a more efficient selection, where good frames are more effectively distributed over the video, like snapshots that tell a story. We call the proposed frame selection SMART and we test it in combination with different backbone architectures and on multiple benchmarks (Kinetics, Something-something, UCF101). We show that the SMART frame selection consistently improves the accuracy compared to other frame selection strategies while reducing the computational cost by a factor of 4 to 10 times. Additionally, we show that when the primary goal is recognition performance, our selection strategy can improve over recent state-of-the-art models and frame selection strategies on various benchmarks (UCF101, HMDB51, FCVID, and ActivityNet).
Introduction
The paper targets computationally expensive action recognition in trimmed videos, where selecting informative and diverse frames can improve accuracy while reducing computation. SMART selects frames jointly by combining individual frame value with relationships among frames, and it outperforms baselines across several datasets.
- Motivation: Trimmed videos contain redundant, unevenly informative frames, motivating selection to reduce computation and improve classification.Some frames may be difficult because of blur or occlusion, while an oracle frame selection can outperform using the entire video.
- Gap: Existing frame-selection methods mainly target long, untrimmed videos where irrelevant content is easier to discard.The paper instead focuses on standard trimmed activity-classification clips.
- Method: SMART considers frame discriminativeness and inter-frame relations jointly rather than selecting frames one at a time.An attention and relational network evaluates frames together, aiming to preserve diversity and represent the action across the video.
- Results: SMART outperforms baselines on trimmed datasets while reducing computational cost by 4 to 10 times, depending on the dataset.The evaluation includes Something-something, UCF101, and subsets of Kinetics.
- Results: The method also achieves higher accuracy than previous frame-selection work on ActivityNet and FCVID and state-of-the-art results on UCF101 and HMDB51.The latter results use selected frames as a preprocessing step for deep action-recognition models.
Related Work
Prior action-recognition work includes recurrent, reinforcement-learning, attention, and relational approaches, with frame selection largely developed for untrimmed videos. SMART differs by selecting frames jointly and by extending relational attention with temporal attention.
- Frame Selection: Frame-selection methods commonly use reinforcement-learning agents that inspect frames sequentially and decide how many frames to skip.Examples include approaches that use LSTMs, memory, confidence, or multiple agents to guide observation.
- Frame Selection: Several methods target salient clips in long videos by discarding irrelevant content, sometimes using audio or deep reinforcement learning.SCSampler uses compressed-video and audio features, while attention-aware sampling trains an agent without extra labels.
- Frame Selection: Most prior frame-selection approaches focus on untrimmed videos, where unnecessary content is easier to discard than frames from trimmed videos.SMART instead considers frames jointly rather than one at a time.
- Attention and Relational Models: Attention models assign relevance weights to sequence elements or image regions, while relational models add global feature representations to local representations.The paper extends relation attention with relation-temporal attention for temporal global representation.
SMART Frame Selection
SMART selects frames using both individual-frame usefulness and relations across frames, with attention and temporal modeling refining the selection. Its lightweight selector is designed to reduce selection cost before expensive classification.
- Lightweight frame representation: SMART uses a lightweight frame representation so selection consumes only a small portion of the overall computational cost.MobileNet extracts visual features, which can be concatenated with language features to form each frame representation Xi.
- Two-stream selection: An individual-frame stream assigns each frame a usefulness score δi, while a second stream scores frame pairs using attention and relational modeling.The two scores are multiplied to produce each frame’s final selection score.
- Temporal relations: The multi-frame stream pairs each frame with a randomly selected subsequent frame, allowing temporal relationships at different distances to be represented.The paired representation is processed by attention and produces temporal relation-attention weights γ1, γ2, .., γN.
- Attention and relation modeling: Self-attention aggregates local frame features into a global representation, which then refines attention by modeling relations between local features and the video as a whole.The relation module further estimates frame attention weights before temporal modeling.
- Temporal aggregation: An LSTM captures sequential frame changes, after which relational-temporal weights produce an attended content vector for classification.The module feeds the attended vector into an MLP and minimizes classification loss using the ground-truth labels.
Experimental Analysis
The experiments evaluate SMART across multiple datasets, ablations, and sampling behaviors. Results examine component choices, frame-count effects, selector complementarity, temporal consistency, and qualitative frame selection.
- Experimental Setup: SMART is evaluated across Something-something-v2, Kinetics subsets, UCF101, HMDB51, ActivityNet, and FCVID.The study uses Something-something-v2 for extensive ablations, Kinetics-Temporal and Kinetics-Static for generality tests, UCF101 and HMDB51, and ActivityNet and FCVID for comparison with untrimmed-video methods.
- Ablation Study: Adding semantic language features improves frame-selection performance compared with standalone visual features.The ablation combines MobileNet visual features with GloVe categorical features before examining selector choices.
- Ablation Study: The combined single-frame and global selectors perform best, indicating complementary selection behavior.The study compares the single-frame selector, global selector, and their combination on Something-something-v2.
- Ablation Study: Using frame pairs improves the global selector, with standard deviations of 0.067 using Inception v3 and 0.082 using ResNet-152 across 10 random runs.The pair-based input adds another mechanism for considering frames jointly; reported accuracies are averaged because the frames are randomly sampled.
- Frame-Count Analysis: SMART outperforms random and uniform sampling across selected-frame counts from 10 to 50, while accuracy peaks before gradually declining as more frames are added.The reported behavior supports a sweet spot in frame count, since additional frames can include harder-to-classify content.
- Selection Behavior: Global-selector scores vary more temporally consistently than single-frame scores, and selected frames can capture the action’s story.For “pushing something from left to right,” the selected frames illustrate the action, while semantically related pushing classes show similar importance-score patterns.
Quantitative results analysis
SMART consistently outperforms frame-sampling baselines across trimmed and untrimmed video benchmarks, while using fewer frames or reducing computation. It also improves recognition when used with other action-recognition models.
- Trimmed datasets: SMART outperforms random, uniform, and full-video sampling on UCF101 and Kinetics at the optimal frame count.On UCF101, full-video performance is exceeded except when using 10 frames; the same pattern appears on temporal and static Kinetics subsets.
- Untrimmed datasets: Fewer frames than recent methods still produce higher accuracy on ActivityNet and FCVID.The comparison includes AdaFrame, FastForward, FrameGlimpse, LiteEval, MARL, and DSN, though accessing all frames makes SMART slower than some frame-selection approaches.
- Extensions to other models: SMART combined with TSN achieves state-of-the-art results on UCF101 and HMDB51.The comparison includes two-stream networks, DynaMotion, I3D, and Knowledge Integration approaches.
- Extensions to other models: Using SMART-selected frames improves I3D, STM-ResNet, and ISTPAN performance at inference.The paper reports these comparisons in Table 5.
Conclusion
The paper concludes that SMART jointly selects frames for trimmed-video action recognition, improving accuracy across datasets while reducing computation. It also transfers to untrimmed-video comparisons and preprocessing for other recognition models.
- Conclusion: SMART makes frame-selection decisions globally by considering all frames in a video jointly rather than individually.This design targets better coverage of the action while selecting frames.
- Conclusion: SMART outperforms baselines on three action-classification datasets while reducing computation cost by up to 4 times.The conclusion states the trimmed-video result at the dataset level.
- Conclusion: SMART outperforms recent frame-selection approaches in accuracy on untrimmed videos.The supplied table caption identifies ActivityNet and FCVID as the untrimmed-video benchmarks.
- Conclusion: SMART preprocessing yields state-of-the-art accuracy on two benchmarks.The supplied caption describes extensions to state-of-the-art deep-learning approaches.
Potential Ethical Impact
The paper notes that video-recognition applications may have positive user impacts, but these uses require analysis of potential algorithmic bias.
- Potential ethical impact: Potential applications include video search and harmful-content removal, but the paper does not experiment with harmful-content data.The authors recommend analyzing biases from training and inherited pretrained models before pursuing such uses.