Source-linked AI summary
AdaFrame: Adaptive Frame Selection for Fast Video Recognition
Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Socher, Larry S. Davis
TL;DR
Video recognition systems often process many expensive frames even though informative content and recognition difficulty vary across videos. AdaFrame uses a global-memory LSTM trained with policy gradients to select frames and estimate future utility for adaptive inference. It matches all-frame performance with substantially fewer computations on FCVID and ActivityNet, while frame usage reflects recognition difficulty.
Problem
Video recognition needs scalable computation, but uniform sampling can analyze expensive irrelevant frames and frame importance is unsupervised under weak sequence-level labels.
Method
AdaFrame uses a global-memory LSTM with policy-gradient training to select the next frame, predict, and estimate future utility for adaptive inference.
Results
58.9% and 63.3% fewer computations are required on FCVID and ActivityNet, respectively, with similar or better accuracy than uniform sampling.
Takeaways & Limitations
Learned frame usage varies across categories and instances, with easier videos using fewer frames and more difficult cases receiving additional inference steps.
Takeaways & Limitations
The framework treats a frame broadly as an RGB image, stacked RGB images, or stacked optical-flow images.
Abstract
from arXiv · showhide
We present AdaFrame, a framework that adaptively selects relevant frames on a per-input basis for fast video recognition. AdaFrame contains a Long Short-Term Memory network augmented with a global memory that provides context information for searching which frames to use over time. Trained with policy gradient methods, AdaFrame generates a prediction, determines which frame to observe next, and computes the utility, i.e., expected future rewards, of seeing more frames at each time step. At testing time, AdaFrame exploits predicted utilities to achieve adaptive lookahead inference such that the overall computational costs are reduced without incurring a decrease in accuracy. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet. AdaFrame matches the performance of using all frames with only 8.21 and 8.65 frames on FCVID and ActivityNet, respectively. We further qualitatively demonstrate learned frame usage can indicate the difficulty of making classification decisions; easier samples need fewer frames while harder ones require more, both at instance-level within the same class and at class-level among different categories.
1. Introduction
AdaFrame addresses the computational cost of video recognition by selecting informative frames adaptively for each input video. It uses a memory-augmented LSTM and policy-gradient training to guide frame selection and adaptive inference.
- Video recognition remains computationally expensive because state-of-the-art systems analyze many uniformly sampled or even every frame with high-capacity backbones.
- Recognition difficulty varies across categories and within-category instances, so different videos can require different numbers of frames.
- AdaFrame learns to select a small number of informative frames per video despite weak sequence-level labels and no supervision identifying important frames.
- AdaFrame uses an LSTM augmented with global memory to guide temporal exploration, frame selection, prediction, and future-utility estimation.
- 58.9% and 63.3% fewer computations are required on FCVID and ActivityNet, respectively, while achieving similar or better accuracy than uniform sampling.
2. Related Work
Prior video-recognition work largely improves representations or aggregates predictions from multiple frames, while AdaFrame focuses on per-video selection of relevant frames and temporal modeling for efficient recognition.
- Most video-recognition methods extend 2D convolutions or model motion information, whereas relatively few methods target efficient video classification.
- Existing efficient approaches commonly mean-pool scores or features from multiple uniformly sampled or agent-selected frames to classify a video clip.
- AdaFrame selects a small number of relevant frames per video and models their temporal relations with an LSTM; its framework can also accept stacked-frame features from 3D CNNs.
- Policy-search frame selection is harder for classification than detection because classification lacks ground-truth temporal boundaries indicating whether viewed frames are relevant.
- Unlike confidence-based early-exit approaches, AdaFrame learns frame selection and stopping within a sequential policy for video recognition.
3. Approach
AdaFrame adaptively selects frames per video using a memory-augmented LSTM that predicts, chooses where to look next, and estimates the utility of future observations. Global context guides temporal exploration, while adaptive lookahead inference accommodates different computational needs.
- AdaFrame selects frames per video and captures the temporal dynamics of those selected frames for efficient recognition.
- Memory-augmented LSTM: The memory-augmented LSTM processes the current frame, previous hidden and cell states, and global context to produce updated states.
- Memory-augmented LSTM: The prediction, selection, and utility networks respectively classify the video, choose the next frame, and estimate expected future rewards.
- Global memory: Global memory stores lightweight representations of spatially and temporally downsampled frames, with positional encoding and attention producing context for frame exploration.
- Optimization: The classification objective uses predictions from the last time step while constraining Te ≪T to use as few frames as possible.
- Optimization: The reward increases when the current ground-truth margin exceeds historical margins, serving as a proxy for information gained from observing another frame.
- Optimization: Policy-gradient optimization combines differentiable classification and exploration terms, using a utility baseline to reduce gradient-estimation variance.
4. Experiments
Experiments evaluate AdaFrame on FCVID and ACTIVITYNET using mAP, computation, ablations, and analyses of adaptive frame usage. AdaFrame maintains accuracy with substantially fewer frames and computations, while its global memory, reward, and utility-based stopping designs improve adaptive inference.
- Datasets and evaluation: FCVID contains 91,223 videos across 239 classes, while both datasets are evaluated with mean average precision on untrimmed videos.FCVID videos average 167 seconds; untrimmed content makes redundant-frame reduction especially relevant.
- Main results: 8.21 and 8.65 frames match all-frame performance on FCVID and ACTIVITYNET, respectively.These results verify that AdaFrame learns frame-selection policies while maintaining the accuracies obtained using all frames.
- Computational savings: 58.9% and 63.3% less computation than AVGPOOLING and LSTM using 25 frames is achieved by AdaFrame-10 on FCVID and ACTIVITYNET, respectively.AdaFrame-10 uses 80.2 versus approximately 195 GFLOPs on FCVID and 71.5 versus approximately 195 GFLOPs on ACTIVITYNET.
- Main results: AdaFrame outperforms alternative frame-selection methods and remains better than competing baselines when fewer frames are used.With fewer frames, AdaFrame reaches 66.1% versus 64.2% on FCVID and 56.3% versus 53.0% on ACTIVITYNET.
- Learned policies: Early predictions tend to be more accurate, while later steps handle difficult decisions and classify more samples when the computational budget increases.Frame usage also varies across classes and within classes, and AdaFrame revisits past information in 42.8% of FCVID videos.
- Ablations: Using 16 global-memory frames offers the best trade-off between computational overheads and accuracies, while global memory improves the non-memory model.The ablation study varies the number of frames used to generate the global memory on FCVID.
- Ablations: Reward functions modeling prediction differences over time outperform prediction-only rewards, and adding a ground-truth-versus-other-class margin improves results further.The comparison includes the proposed reward and PREDICTION TRANSITION REWARD against PREDICTION REWARD.
- Stopping criterion: Predicted utilities provide smoother stopping signals than entropy because they explicitly consider future information from the first step.Entropy produced high early values and extremely low values near the end, limiting adaptive inference across thresholds.
5. Conclusion
AdaFrame derives per-video frame usage policies to reduce computational cost while maintaining strong recognition performance. It combines global context, policy-gradient training, and adaptive inference to select frames effectively.
- AdaFrame derives an effective frame usage policy that selects a small number of frames per video to reduce overall computational cost.
- The framework augments an LSTM with global memory to inject global context information into frame selection.
- Policy-gradient training teaches AdaFrame to predict which frame to use and calculate future utilities, which guide adaptive inference during testing.