PaperlayerView exact revision on arXiv

Source-linked AI summary

Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition

Yulin Wang, Haoji Zhang, Yang Yue, Shiji Song, Chao Deng, Junlan Feng, Gao Huang

arXiv:2412.11228v1cs.CVcs.AIcs.LG

TL;DR

Deep video recognition can be computationally costly, and this paper investigates whether spatial, temporal, and sample-wise redundancy can be used to reduce that cost. It introduces AdaFocus and extends it as Uni-AdaFocus; experiments report improved efficiency against competitive baselines across benchmarks and application scenarios.

  • Problem

    Deep video-recognition models achieve strong performance but incur high computational costs, motivating investigation of spatial redundancy alongside temporal and sample-wise computation.

    Source: §1 · p. 1

  • Method

    AdaFocus uses a lightweight global encoder and policy to select task-relevant regions for a high-capacity local encoder, and Uni-AdaFocus integrates spatial, temporal, and sample-wise dynamic computation.

    Source: §1 · pp. 1–2

  • Results

    Uni-AdaFocus consistently outperforms competitive baselines across seven benchmark datasets and three application scenarios, achieving state-of-the-art theoretical computational efficiency and practical inference speed.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Uni-AdaFocus can be deployed with off-the-shelf backbones such as TSM and X3D, and its inference cost can be adjusted online without additional training.

    Source: §1 · p. 2, §6 · p. 15

    AdaFocusV1’s three-stage reinforcement-learning training is nontrivial, time-consuming, may yield sub-optimal solutions, and depends substantially on implementation configurations.

    Source: §3.3 · pp. 4–5

Abstract

from arXiv · show

This paper presents a comprehensive exploration of the phenomenon of data redundancy in video understanding, with the aim to improve computational efficiency. Our investigation commences with an examination of spatial redundancy, which refers to the observation that the most informative region in each video frame usually corresponds to a small image patch, whose shape, size and location shift smoothly across frames. Motivated by this phenomenon, we formulate the patch localization problem as a dynamic decision task, and introduce a spatially adaptive video recognition approach, termed AdaFocus. In specific, a lightweight encoder is first employed to quickly process the full video sequence, whose features are then utilized by a policy network to identify the most task-relevant regions. Subsequently, the selected patches are inferred by a high-capacity deep network for the final prediction. The full model can be trained in end-to-end conveniently. Furthermore, AdaFocus can be extended by further considering temporal and sample-wise redundancies, i.e., allocating the majority of computation to the most task-relevant frames, and minimizing the computation spent on relatively "easier" videos. Our resulting approach, Uni-AdaFocus, establishes a comprehensive framework that seamlessly integrates spatial, temporal, and sample-wise dynamic computation, while it preserves the merits of AdaFocus in terms of efficient end-to-end training and hardware friendliness. In addition, Uni-AdaFocus is general and flexible as it is compatible with off-the-shelf efficient backbones (e.g., TSM and X3D), which can be readily deployed as our feature extractor, yielding a significantly improved computational efficiency. Empirically, extensive experiments based on seven benchmark datasets and three application scenarios substantiate that Uni-AdaFocus is considerably more efficient than the competitive baselines.

1 INTRODUCTION

The introduction frames video recognition efficiency as a response to the high computational cost of deep models, and explores spatial redundancy alongside prior temporal-selection approaches. It presents AdaFocus and its extensions, then Uni-AdaFocus, with reported efficiency gains across benchmarks and application scenarios.

1 INTRODUCTION

  • Deep video-recognition models can be computationally costly, while prior efficiency work emphasizes selecting informative frames; spatial redundancy in video recognition is comparatively underexplored.The paper connects computation to power consumption, carbon emissions, and practical latency.Source: §1 · p. 1
  • AdaFocus uses a lightweight model and policy network to select task-relevant frame regions for a high-capacity local encoder, allocating computation unevenly while preserving accuracy.The introduction motivates this approach by noting that a 96x96 patch requires about 18% of the computation of a 224x224 image.Source: §1 · pp. 1–2
  • The paper improves AdaFocus with end-to-end training, deep-feature-based region-selection policies, and deformable patches that adapt to different scales, shapes, and locations.The end-to-end formulation eliminates the earlier three-stage reinforcement-learning procedure.Source: §1 · p. 2
  • Uni-AdaFocus combines spatial, temporal, and sample-wise dynamic computation and supports off-the-shelf backbones such as TSM and X3D, with inference cost adjustable online without additional training.The paper identifies fluctuating computational resources and flexible performance targets as relevant settings.Source: §1 · p. 2
  • Uni-AdaFocus consistently outperforms competitive baselines across seven benchmark datasets and three application scenarios, with state-of-the-art theoretical computational efficiency and practical inference speed.The paper also describes Uni-AdaFocus as integrating spatial, temporal, and sample-wise dynamic computation.Source: §1 · p. 2

2 RELATED WORKS

Related work spans efficient video-recognition architectures and methods for reducing temporal or spatial computation. The paper positions AdaFocus as spatially adaptive and Uni-AdaFocus as combining spatial, temporal, and sample-wise computation.

2 RELATED WORKS

  • Video-recognition work includes 3D convolution, frame-wise feature extraction with temporal aggregation, two-stream models, and self-attention, while deep-network cost constrains practical use.The cited examples include C3D, I3D, X3D, recurrent networks, temporal channel shift, and video Transformers.Source: §2 · p. 2
  • Prior temporal-redundancy methods adapt frame processing, resolution, precision, or stopping time; this paper adds spatially focused AdaFocus, dynamic frame sampling, and a unified spatial-temporal-sample-wise framework.The paper states that integrating the three forms yields state-of-the-art computational efficiency.Source: §2 · p. 3
  • Unlike OCSampler, which uses reinforcement learning for sequential weighted frame sampling, this method optimizes expected loss with a differentiable Monte Carlo formulation and reports lower complexity and training time.The passage also reports significantly improved performance relative to.Source: §2 · p. 3
  • Spatially adaptive video recognition remains relatively unexplored; unlike spatial transformer networks, AdaFocus localizes task-relevant video regions to improve computational efficiency.The paper says a straightforward interpolation-based implementation does not yield competitive results for its problem and requires improved designs.Source: §2 · p. 3
  • AdaFocus’s pipeline uses a lightweight global encoder, a policy to select a frame region, a high-capacity local encoder, and a classifier that aggregates features across frames.The overview describes an online recognition setting in which predictions may be retrieved after processing any number of frames.Source: Figure 2 · p. 3

3 ADAPTIVE FOCUS NETWORK (ADAFOCUS)

AdaFocus reduces spatially redundant computation by selecting task-relevant frame patches with a policy network and processing them with a high-capacity local encoder. The section develops this design from AdaFocusV1’s three-stage reinforcement-learning training to AdaFocusV2’s differentiable end-to-end approach and training techniques.

3.1 Network Architecture

  • A lightweight global encoder and policy network guide a high-capacity local encoder to process selected patches, while a classifier aggregates frame features for recognition.The global features also support patch localization, enabling feature reuse.Source: §3.1 · pp. 3–4

3.2 AdaFocusV1: Three-stage Reinforcement Learning

  • Because patch cropping is non-differentiable, AdaFocusV1 formulates localization as sequential discrete decisions and uses a three-stage reinforcement-learning training procedure.The policy distribution is parameterized by global features, with candidate patches selected from that distribution during training.Source: §3.2 · p. 4

3.3 AdaFocusV2: Differentiable End-to-end Training

  • AdaFocusV2 replaces AdaFocusV1’s practitioner-unfriendly, potentially time-consuming training procedure with end-to-end training that reduces training cost and achieves better performance.The comparison starts both procedures from the same initial backbone network.Source: §3.3 · p. 5, Table 1 · p. 5

3.3.1 Interpolation-based Patch Selection

  • Bilinear interpolation constructs patches at continuous policy-predicted coordinates, allowing gradients to pass through the selected patch to the policy network.The patch coordinates are constrained so the selected patch remains inside the frame.Source: Figure 4 · p. 5, §3.3.1 · p. 5

3.3.2 Training Techniques

  • Straightforward end-to-end training degrades performance, so AdaFocusV2 combines auxiliary encoder supervision, diversity augmentation, and stop-gradient training for stability.The authors report consistent improvements across datasets, backbone architectures, and model configurations without adding tunable hyper-parameters.Source: §3.3.2 · p. 6

4 UNIFIED ADAFOCUS (UNI-ADAFOCUS)

This section presents Uni-AdaFocus, which extends AdaFocusV2 by combining improved spatial computation with spatial, temporal, and sample-wise dynamic computation while retaining efficient end-to-end training and broad backbone compatibility. It also describes conditional-exit thresholds for meeting computational budgets and adjusting computation without additional training.

4.1.1 Training the Policy Network with Feature Interpolation

  • Deep-feature interpolation gives π semantic-level guidance from global features, improving on AdaFocusV3 with negligible training overhead and no local-encoder train/test input discrepancy.The method adds only a linear-classification cost during training.Source: §4.1.1 · p. 7

4.1.2 Deformable Patches

  • Deformable patches let π adapt patch centers, heights, and widths to each frame; regularization prevents patches collapsing to a point, enabling flexible regions and considerable accuracy improvements.Patches are resized to P ×P for parallel processing; the reported flexibility outweighs the resulting geometric distortion.Source: §4.1.2 · pp. 7–8

4.2 Reducing Temporal Redundancy – Dynamic Frame Sampling

  • π jointly selects TL task-relevant frames and their informative patches, combining temporal and spatial dynamic computation in an end-to-end framework.The lightweight global encoder first processes TG uniformly sampled frames; the local encoder processes only the selected frames and patches.Source: §4.2 · pp. 8–9
  • Dynamic frame sampling selects TL frames from T0 by weighted sampling without replacement, and a differentiable Monte Carlo objective enables efficient end-to-end policy training.With M =128, the objective has complexity O(TL(T0 + TL log T0) · M), and is more efficient than directly solving the sampling objective.Source: §4.2 · pp. 8–9

4.3 Reducing Sample-wise Redundancy – Conditional-exit

  • A conditional early-exit algorithm reduces computation on easier videos and integrates with the existing training paradigm without an additional training phase.At test time, it compares the per-frame softmax prediction entropy with a predefined threshold to determine whether to terminate inference.Source: §4.3 · p. 10
  • Changing conditional-exit thresholds adjusts computational cost without additional training, with threshold values selected for a given budget and an infinite threshold at the final frame.The thresholds are solved using validation-set accuracy and FLOPs; the implementation instead solves the optimization on the training set using, which performs on par with cross-validation.Source: §4.3 · p. 10

5 EXPERIMENTS

The experiments evaluate Uni-AdaFocus across benchmark datasets, lightweight backbones, application scenarios, and component analyses. Results report improved accuracy-efficiency trade-offs, including gains over efficient baselines and earlier AdaFocus versions.

5.1 Comparisons with State-of-the-art Efficient Video Understanding Approaches

  • At the same state-of-the-art mAP, Uni-AdaFocus requires approximately 4.8× fewer GFLOPs/video than OCSampler on ActivityNet.Uni-AdaFocus (128²) also reaches 80.4% mAP versus OCSampler’s 76.9% with 18.1 versus 21.7 GFLOPs; end-to-end variants train ∼2× faster than AdaFocusV1, and Uni-AdaFocus cuts AdaFocusV3 inference cost by 2.3–2.8× without sacrificing performance.Source: §5.1 · pp. 11–12

5.2 Building on Top of Existing Efficient Backbones

  • On Sth-Sth V2, Uni-AdaFocus reduces TSM+ computation 3.99× while improving Top-1 accuracy by 2.9%; on Kinetics-400, it cuts X3D-L computation 3.76× and improves accuracy by 0.5%.Compared with MoViNet-A2, Uni-AdaFocus-X3D uses 2.10× fewer GLOPs/video at the same 75.0% accuracy; on Sth-Sth, Uni-AdaFocus also reduces computation 2.7–3.5× at matched accuracy versus preliminary AdaFocus versions.Source: §5.2 · pp. 12–13

5.3 Real-world Application Scenarios

  • Across diving classification, MRI-based Alzheimer’s and Parkinson’s diagnosis, and online violence recognition, Uni-AdaFocus reports large-margin gains over existing works and new state-of-the-art performance.The application results are reported in Appendix D.Source: §5.3 · p. 13

Effectiveness of the techniques for end-to-end training

  • The proposed training techniques improve performance across settings and enable end-to-end training; reusing global features adds 0.8–2.8% accuracy with negligible computation.The feature-reuse result is reported in Table 8; Table 6 evaluates training techniques without sample-wise dynamic computation.Source: Effectiveness of the techniques for… · pp. 13–14, Table 6 · p. 13, Table 8 · p. 14
  • Deep-feature guidance improves patch selection by ∼1%, deformable patches add gains especially at 96² local-encoder input, and dynamic frame sampling further enhances accuracy.The learned spatial-temporal policies outperform alternatives; the authors report that regularization supports effective deformable-patch policies and their frame sampler beats MG Sampler.Source: Effectiveness of the techniques for… · pp. 13–14
  • Adaptive conditional exit saves computation at matched mAP versus fixed-length and random early-exit baselines, while allowing inference cost to change online without retraining.The method adjusts cost by changing early-exit thresholds.Source: §5.1 · p. 11, Effectiveness of the techniques for… · p. 13, Figure 12 · p. 14

5.4.2 Visualization Results

  • Visualizations show Uni-AdaFocus selecting task-relevant regions, including dancers, dogs, skateboards, and human hands, in informative video frames.Figure 13 marks selected patches with blue boxes; additional visualizations and failure-case analysis are in Appendix F.Source: §5.4.2 · p. 14, Figure 13 · p. 14

5.4.3 Discussions

  • Spatial, temporal, and sample-wise computation are compatible, with spatial adaptiveness yielding the largest efficiency gains; allowing location, scale, and shape variation helps, while more flexible warps slightly reduce accuracy.The authors suggest the warps may distort local-encoder inputs from the natural distribution and increase policy-network training complexity.Source: §5.4.3 · p. 15

6 CONCLUSION

Uni-AdaFocus allocates computation dynamically across spatial regions, video frames, and samples. The paper reports improved efficiency across backbones and evaluations, and identifies multimodal models as a direction for future study.

6 CONCLUSION

  • Uni-AdaFocus dynamically allocates computation across spatial, temporal, and sample dimensions and can be deployed on lightweight backbones such as TSM and X3D.The paper reports state-of-the-art generalization, theoretical computational efficiency, and practical inference speed across seven benchmarks and three real-world scenarios.Source: §6 · p. 15
  • The conclusion presents efficient end-to-end training and improved inference efficiency as properties of the Uni-AdaFocus framework.The reported evaluation spans seven large-scale benchmark datasets and three application scenarios.Source: §6 · p. 15
  • The authors propose exploring dynamic computation in multimodal large language models by selecting video frames and image regions conditioned on text or audio prompts.They also identify fine-tuning pretrained multimodal models as important given the expensive cost of training large models.Source: §6 · p. 15

APPENDIX A SUMMARY OF CHANGES

The appendix summarizes extensions to AdaFocus in spatial, temporal, and sample-wise computation, alongside updated benchmark evaluations and analyses.

APPENDIX A SUMMARY OF CHANGES

  • The spatial algorithm adds deep-feature-based policy training and deformable patches that adapt to regions with diverse shapes, sizes, and scales.The deep-feature approach incurs negligible additional cost while improving final accuracy.Source: Appendix A · p. 19
  • Dynamic frame sampling extends AdaFocus to combine spatial, temporal, and sample-wise dynamic computation in Uni-AdaFocus.The framework is reported to achieve strong computational efficiency across settings.Source: Appendix A · p. 19
  • Updated experiments report state-of-the-art Uni-AdaFocus results on six benchmark datasets and evaluate its use with X3D on Kinetics-400.The comparisons include earlier AdaFocus works,,.Source: Appendix A · p. 19
  • The expanded evaluation adds three application scenarios: fine-grained diving classification, MRI-based Alzheimer’s and Parkinson’s diagnosis, and online violence recognition.The appendix reports encouraging results for these applications.Source: Appendix A · p. 19
  • Additional analyses cover new-component ablations, findings about the method, comparisons with existing works, and visualizations.The appendix identifies Tables 7–9 and Figure 12 for ablations, Tables 10–11 for discussions, Tables 15–16 for comparisons, and Figures 13–15 for visualizations.Source: Appendix A · p. 19

APPENDIX B THREE STAGE TRAINING OF ADAFOCUSV1

AdaFocusV1 uses three training stages because recognition and patch localization involve continuous and discrete optimization. The stages first train the recognition components, then the patch-selection policy, and finally fine-tune the model with that policy.

APPENDIX B THREE STAGE TRAINING OF ADAFOCUSV1

  • AdaFocusV1 alternates continuous recognition and discrete patch-localization optimization through a three-stage training procedure.The standard end-to-end paradigm cannot be directly applied to this formulation.Source: Appendix B · p. 19
  • Stage I initializes the global and local encoders and classifier, then trains on randomly sampled patches using cross-entropy.This stage teaches the model to extract task-relevant information from frame patches, preparing for policy training.Source: Appendix B · p. 19
  • Stage II fixes the Stage I encoders and classifier, then trains a randomly initialized policy network with reinforcement learning to select informative patches.The policy maximizes discounted rewards; the implementation uses PPO with γ = 0.7.Source: Appendix B · p. 19
  • The policy reward compares ground-truth prediction confidence for a selected patch against the expected confidence from a random patch, encouraging confident predictions with fewer frames.Subtracting the random-patch term yields an expected reward of zero and is reported to stabilize training.Source: Appendix B · p. 19
  • Stage III fine-tunes the local encoder and classifier, or only the classifier, using the learned policy; this stage further improves performance..Source: Appendix B · p. 19

APPENDIX C IMPLEMENTATION OF DYNAMIC FRAME SAMPLING

Dynamic frame sampling uses global-encoder features to approximate the sampling process at low additional cost, then makes inference deterministic through weighted quantiles.

APPENDIX C IMPLEMENTATION OF DYNAMIC FRAME SAMPLING

  • The T0-to-TL sampling process is approximated with TG-to-⌊TL T0/TG⌋ weighted sampling using down-sampled weights.The implementation minimizes Eq. (21) without notable additional cost.Source: Appendix C · pp. 19–20
  • Frame losses come from the global encoder’s auxiliary linear classifier; with temporal fusion, each loss represents a short clip centered on a sampled frame.The test-time procedure takes TL uniform quantiles of the weighted-sampling cumulative distribution to select positions deterministically.Source: Appendix C · p. 20

APPENDIX D RESULTS IN REAL-WORLD APPLICATION SCENAR-

Across three realistic application scenarios, Uni-AdaFocus reports strong results in diving classification, neurodegenerative disease diagnosis, and online violence recognition.

Fine-grained diving action classification

  • 7.7% higher test accuracy (88.1% v.s. 80.4%) with 1.87x less computation than TSM+ on Diving48.The passage attributes the result in part to Diving48’s less biased setting, where filtering less informative regions and redundant frames may be lossless or beneficial.Source: Table 12 · p. 20, Appendix D · p. 20

Alzheimer’s and Parkinson’s diseases diagnosis with MRI

  • Uni-AdaFocus outperforms state-of-the-art medical imaging frameworks in accuracy for Alzheimer’s and Parkinson’s diagnosis on ADNI, OASIS, and PPMI.Many comparison methods use generative models, Transformers, or 3D convolution and are described as more computationally intensive.Source: Table 13 · p. 20, Appendix D · p. 20

Violence recognition for online videos

  • 98.5% test accuracy on RLVS and 23.0x less computational cost than the previous best baseline for violence recognition.The passage reports a new state-of-the-art result for this scenario.Source: Table 14 · p. 20, Appendix D · p. 20

APPENDIX E ADDITIONAL DISCUSSIONS

The discussion contrasts Uni-AdaFocus with AdaFocusV3 and OCSampler, emphasizing training design, optimization complexity, and empirical performance.

Comparisons with OCSampler

  • 2.5x less real training time and higher mAP (77.4% v.s. 74.9%) than OCSampler.Uni-AdaFocus eliminates the factorial time complexity of the algorithm while working without OCSampler’s multi-stage training and reinforcement learning techniques.Source: Table 16 · p. 21, Appendix E · p. 21

Comparisons with AdaFocusV3

  • Uni-AdaFocus keeps local-encoder input size consistent across training and testing, avoiding their discrepancy and additional training cost.It uses off-the-shelf global features to estimate policy gradients at no extra cost; these features are more effective than local features for guiding the policy.Source: Appendix E · p. 21, Table 15 · p. 21

Comparisons with OCSampler

  • For dynamic frame sampling, Uni-AdaFocus optimizes expected loss through a differentiable decomposition solved with Monte Carlo for end-to-end training.The comparison with OCSampler concerns algorithms for learning temporal dynamic computation strategies.Source: Appendix E · p. 21, Table 16 · p. 21

APPENDIX F ADDITIONAL VISUALIZATION RESULTS

The visualizations show both failure modes and successful localization of interaction-critical regions; failures include rare actions and tasks requiring more contextual reasoning.

Failure cases

  • Failures include rare, atypical tail actions and video-understanding tasks with greater reasoning demands.Figure 14 presents representative failures, with blue boxes marking selected patches and labels showing ground truth and predictions.Source: Appendix F · p. 21, Figure 14 · p. 21

Additional visualization results

  • For videos with multi-person or object interactions, the visualizations show Uni-AdaFocus capturing regions critical to recognizing the action.Figure 15 marks selected patches with blue boxes.Source: Figure 15 · p. 21, Appendix F · p. 22

Failure cases

  • In examples, the model identifies hand–potato interactions but misses cookie-baking context, or localizes regions while misreading a dog’s role.These examples illustrate errors beyond simply locating task-relevant regions.Source: Appendix F · p. 21

APPENDIX G DATASETS

The appendix describes evaluation across seven video benchmarks, the model configurations used with efficient backbones, and applications in fine-grained action recognition, disease diagnosis, and violence recognition. It also records efficiency-oriented preprocessing and design choices for short videos.

APPENDIX G DATASETS

  • Experiments use the official training-validation splits of seven widely used video-understanding benchmarks.Source: Appendix G · p. 22
  • At test time, videos use one centre-cropped clip, with three-crop prediction also reported on Kinetics-400; more inference views are excluded to limit cost.Source: Appendix G · p. 22

H.1 Details of Network Architectures

  • For ActivityNet, FCVID, and Mini-Kinetics, MobileNet-V2 and ResNet-50 serve as global and local encoders, with temporal accumulated maxpooling as classifier.Source: §H.1 · p. 22
  • The policy network separates temporal and spatial decisions into two branches that pool or compress features to reduce redundant information and computational overhead.Source: §H.1 · p. 22
  • For Sth-Sth, Jester, and Kinetics-400, Uni-AdaFocus uses TSM or X3D backbones and compares against TSM+ or X3D under the stated backbone settings.Source: §H.1 · p. 22
  • Because these videos are short and adjacent inputs need similar visual angles, Uni-AdaFocus uses one common patch location and size per video while retaining sample-wise activation of the local encoder.Source: §H.1 · pp. 22–23

H.2 Real-world Application Scenarios

  • The application evaluations cover Diving48 fine-grained action classification, MRI-based Alzheimer’s and Parkinson’s diagnosis, and violence recognition on RLVS.Source: §H.2 · p. 23

H.3 Training Hyper-parameters

Training hyperparameters are selected using a held-out mini-validation set, then the selected settings are used to train on the full training set. Configurations vary by dataset group and backbone.

H.3 Training Hyper-parameters

  • Hyperparameters are selected on 25% of training samples held out as a mini-validation set, then the model is retrained on the full training set for official validation or test evaluation.Source: §H.3 · p. 23
  • For ActivityNet, FCVID, and Mini-Kinetics, all Uni-AdaFocus components train end-to-end with SGD and cosine learning-rate annealing; batch size and regularization differ by dataset group.Source: §H.3 · p. 23
  • Sth-Sth and Jester TSM experiments adjust learning rates and regularization, while Kinetics-400 X3D experiments use official pretrained backbones and their own specified learning rates and regularization.Source: §H.3 · p. 23

Related papers

Loading 2412.11228v1…