Source-linked AI summary
AR-Net: Adaptive Frame Resolution for Efficient Action Recognition
Yue Meng, Chung-Ching Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, Rogerio Feris
TL;DR
Action-recognition models can be computationally expensive for resource-limited deployment, especially when all frames use the same resolution. AR-Net learns an input-conditioned policy to select per-frame resolutions or skips jointly with recognition, achieving lower computation and improved benchmark accuracy.
Problem
Action-recognition accuracy often grows with computational complexity, while existing methods commonly process all frames at one resolution, limiting efficient deployment.
Method
AR-Net uses a lightweight policy network with Gumbel Softmax to select each frame’s resolution or skip it and trains the policy jointly with recognition through back-propagation.
Results
AR-Net achieves 73.8% mAP on ActivityNet and 81.3% mAP on FCVID while using 17%–64% less computation than compared methods.
Takeaways & Limitations
Adaptive resolution learning supports a broad range of accuracy–speed operating points and can be applied to different backbone architectures and other video-understanding tasks.
Abstract
from arXiv · showhide
Action recognition is an open and challenging problem in computer vision. While current state-of-the-art models offer excellent recognition results, their computational expense limits their impact for many real-world applications. In this paper, we propose a novel approach, called AR-Net (Adaptive Resolution Network), that selects on-the-fly the optimal resolution for each frame conditioned on the input for efficient action recognition in long untrimmed videos. Specifically, given a video frame, a policy network is used to decide what input resolution should be used for processing by the action recognition model, with the goal of improving both accuracy and efficiency. We efficiently train the policy network jointly with the recognition model using standard back-propagation. Extensive experiments on several challenging action recognition benchmark datasets well demonstrate the efficacy of our proposed approach over state-of-the-art methods. The project page can be found at https://mengyuest.github.io/AR-Net
1 Introduction
AR-Net addresses the computational cost of action recognition by selecting frame resolutions or skipping frames adaptively. It jointly learns this policy with the recognition model and reports improved efficiency and accuracy across benchmark datasets.
- Action-recognition accuracy typically increases with model complexity and computation, limiting deployment on resource-constrained platforms.
- Most existing efficient methods process every video frame at the same resolution, motivating input-conditioned per-frame resolution selection.
- AR-Net uses a policy network to choose each frame’s resolution or skip it, conditioned on the input.
- Gumbel Softmax enables joint policy and recognition-model training through standard back-propagation despite discrete decisions.
- AR-Net is reported to reduce computation while maintaining or improving recognition accuracy, including about 45% less computation than a state-of-the-art method on ActivityNet-v1.3.
- Experiments cover ActivityNet-v1.3, FCVID, and Mini-Kinetics to evaluate the approach against state-of-the-art methods.
2 Related Works
Prior efficient action-recognition work reduces computation through lightweight architectures or input-conditioned frame and clip selection. AR-Net is positioned among adaptive sampling and multi-resolution approaches while remaining agnostic to the recognition architecture.
- Efficient Action Recognition: Efficient action-recognition methods commonly pursue lightweight architectures or select salient frames and clips conditioned on the input.
- Efficient Action Recognition: AR-Net is most closely related to adaptive data-sampling methods and is designed to remain agnostic to the recognition network architecture.
- Efficient Action Recognition: LiteEval-related work emphasizes flexibility for multiple actions in one step and scalability to long untrimmed videos through multi-step skipping.
- Adaptive Computation: Prior adaptive-computation methods dynamically choose network exits, layers, or information routes to reduce inference cost.
- Multi-Resolution Processing: Multi-resolution representations have long been used in computer vision, but relatively few methods explore multi-scale recognition for efficient video understanding.
3 Proposed Method
AR-Net jointly learns a lightweight per-frame policy that selects resolutions or skips frames, routes frames to resolution-specific backbones, and averages predictions for video classification. Gumbel Softmax makes the discrete policy differentiable, while the loss balances classification accuracy, computational cost, and policy usage.
- Adaptive Resolution: AR-Net uses a policy network to select each frame's resolution or skip action, then routes resized frames to corresponding backbone networks.The policy network uses a lightweight feature extractor and LSTM; frame-level predictions are averaged into the video-level prediction.
- Adaptive Resolution: During inference, the policy selects actions before frames are resized and processed, and the network averages the resulting frame-level predictions.The policy network and resizing operations add negligible cost relative to the original recognition models.
- Adaptive Resolution: The action space combines descending frame resolutions with skipping operations that omit the current and subsequent frames from prediction.The lowest-resolution backbone can share computation with the policy network, and skipped frames are treated as a special resolution choice.
- Learning the Adaptive Resolution Policy: Gumbel Softmax replaces nondifferentiable discrete sampling with a differentiable relaxation, enabling backpropagation through policy decisions.During the forward pass decisions are sampled discretely, while the backward pass uses gradients from the continuous softmax relaxation; temperature is annealed during training.
- Loss Functions: AR-Net trains with classification, computational, and policy-usage objectives to balance recognition quality, GFLOPS, and action-distribution regularization.Classification uses cross-entropy; computational cost is estimated from an offline GFLOPS lookup table, while a regularizer encourages balanced policy usage and higher action-distribution entropy.
4 Experiments
Experiments evaluate AR-Net across multiple datasets, backbones, baselines, policy settings, and training strategies. The results show improved recognition–efficiency trade-offs through adaptive resolution selection and frame skipping.
- Experimental Setup: AR-Net is evaluated on ActivityNet-v1.3, FCVID, and Mini-Kinetics using ResNet and EfficientNet backbones.The experiments compare adaptive policies with simple baselines and state-of-the-art efficient recognition methods.
- Evaluation: The experiments measure recognition with mAP or Top-1 accuracy and efficiency with GFLOPS per frame and per video.The evaluation includes baseline comparisons, policy ablations, loss studies, and training-strategy studies.
- Main Results: 50% less computation is achieved with better classification performance than traditional Uniform and LSTM baselines.This comparison is reported for ActivityNet and FCVID across ResNet and EfficientNet architectures.
- Policy Analysis: AR-Net matches Multi-Scale classification performance while using 70% less computation, and learned policies outperform random policies on both datasets.For ResNet, learned policies reach 73.8% versus 65.0% mAP on ActivityNet and 81.3% versus 75.3% on FCVID.
- Main Results: 73.8% mAP on ActivityNet and 81.3% mAP on FCVID are achieved with 17%–64% less computation than competing approaches.With EfficientNet, AR-Net gains 5.9% mAP on ActivityNet and 3.1% on FCVID while using 54%–60% less computation than AR-Net with ResNet.
- Qualitative Analysis: Adaptive resolution selection and skipping retain indicative frames at original resolution while resizing or skipping irrelevant or low-quality frames.Qualitative examples use eight uniformly sampled frames from ActivityNet and FCVID testing sets.
5 Conclusion
AR-Net applies adaptive resolution learning per frame for efficient video action recognition. It spans accuracy-speed operating points and is reported as model-agnostic for broader video understanding tasks.
- AR-Net learns adaptive frame resolution for efficient video action recognition.
- The approach covers accuracy-speed operating points, including versions that are faster and more accurate than comparable visual-only models.
- An EfficientNet-based version achieves state-of-the-art accuracy-throughput performance.
- The learning framework is model-agnostic and can be applied to sophisticated backbone networks and other video understanding tasks.
Adaptive Resolution for Efficient Action Recognition (Supplementary)
The supplementary material includes an overview of the supplementary contents. The supplied passage identifies this overview as Table 1.
- Table 1 provides an overview of the supplementary material.
- The supplementary overview is presented as a table.
- The table is labeled “Supplementary Material Overview.”
A Mini-Kinetics
Mini-Kinetics is a smaller dataset created from Kinetics-400 by randomly selecting half its categories. It contains 121K training videos and 10K testing videos, each lasting 6–10 seconds.
- Mini-Kinetics is created by randomly selecting half of the categories from Kinetics-400.
- Mini-Kinetics contains 121K training videos and 10K testing videos.
- Each Mini-Kinetics video lasts 6–10 seconds.
- The dataset splits will be made publicly available to enable future comparisons.
B GFLOPS Estimation
GFLOPS estimation combines online policy distributions with offline measurements for modules and resolutions. The supplementary material presents the resulting GFLOPS table and explains the measurement and approximation procedures.
- Table 2 lists GFLOPS for different backbones and resolutions.
- Overall GFLOPS are computed as a weighted sum using the online policy distribution and an offline GFLOPS lookup table.
- The lookup table measures GFLOPS by instantiating each network and resolution configuration and profiling it with THOP.
- LSTM GFLOPS are approximated by the square of the input feature dimension and are negligible when feature dimensions are around 1000–2000.
C Distributions
AR-Net’s policy usage varies across datasets and action categories, adapting resolution and frame skipping to the apparent informativeness and complexity of videos.
- Policy distributions: AR-Net uses a policy distribution over 224×224, 168×168, 112×112, and frame-skipping choices, with 84×84 omitted because it affects fewer than 1% of frames.Videos are uniformly sampled in 8 frames.
- Dataset-level usage: Mini-Kinetics videos receive relatively more frame skipping than ActivityNet and FCVID, indicating lower motion informativeness.
- Category-level usage: Complex procedures such as making a sandwich use more high-resolution frames than static scenes or scene-related actions such as lightning, ballet, or building cabinet.
- Figure organization: Figure 1 reports overall policy distribution and relative high-resolution usage across ActivityNet, FCVID, and Mini-Kinetics.
D Additional Qualitative Analysis
Qualitative analyses on ActivityNet, FCVID, and Mini-Kinetics show AR-Net allocating computation according to example difficulty, appearance clarity, and motion.
- Qualitative findings: Across the three datasets, AR-Net saves computation greatly for examples with clear appearance or actions involving less motion.
- Difficulty categories: Examples are grouped as Easy, Medium, or Hard according to their computation budgets.
- Visualization setup: The qualitative examples uniformly sample videos into 8 frames and display the original inputs alongside AR-Net’s selected resolutions or skipping decisions.
E RL vs Gumbel Softmax in Policy Learning
The study compares reinforcement-learning and Gumbel Softmax approaches for learning AR-Net’s policy on ActivityNet-v1.3, with results reported in Table 3.
- Experimental comparison: The reinforcement-learning comparison uses policy gradients with the same training procedures and epoch count as the Gumbel Softmax experiment.
- Experimental comparison: The RL-based method receives further learning-rate tuning, changing joint-training from 0.001 to 0.002 and finetuning from 0.0005 to 0.001.
- Reported results: Table 3 reports performances for the different policy-learning approaches on ActivityNet-v1.3.