Adaptive Focus for Efficient Video Recognition

1. Introduction

• Video recognition models incur high computational costs, which translate into power consumption, carbon emissions, and practical latency.The paper frames minimizing these costs as important under economic and safety considerations.Source: §1 · p. 1
• Spatial redundancy is an underexplored source of computation in video recognition, beyond prior efforts to reduce temporal redundancy by selecting informative frames.The paper notes that a 96x96 patch requires about 18% of the computation of a 224x224 image.Source: §1 · pp. 1–2
• AdaFocus uses a lightweight CNN to glance at each frame, a recurrent policy to select a task-relevant region, and a high-capacity CNN to process that patch.The policy is trained with reinforcement learning because localizing task-relevant regions is non-differentiable.Source: §1 · p. 2
• AdaFocus+ adds a policy network to skip less valuable frames, extending vanilla AdaFocus, which processes every frame with the same computation.The paper describes the extension as compatible with temporal-based techniques.Source: §1 · p. 2
• Across five video-recognition benchmarks, AdaFocus outperforms baselines, while AdaFocus+ further improves efficiency; it uses 2-3x less FLOPs than AR-Net at the same accuracy.The method can also be deployed on top of state-of-the-art networks such as TSM to improve computational efficiency.Source: §1 · p. 2

2. Related Works

• In this paper, FLOPs means the number of multiply-add operations.The paper uses this term for computational cost.Source: §2 · p. 2
• Video-recognition research includes 3D CNNs that jointly model space and time, and methods that aggregate frame-level features using temporal averaging, LSTMs, or channel shifting.Expensive CNN computation, especially for 3D CNNs, limits applicability.Source: §2 · p. 2
• Efficiency approaches include lightweight architectures and dynamic computation, with temporal-redundancy methods using early stopping, conditional computing, or frame and clip sampling.AdaFocus differs by allocating most computation to task-relevant regions and is compatible with temporal-redundancy reduction.Source: §2 · p. 2
• Image-classification work attends to informative image regions, and video attention studies show that frame regions contribute unequally; prior work had not exploited spatial redundancy to improve video-recognition efficiency.The paper positions AdaFocus as addressing this gap.Source: §2 · p. 2
• AdaFocus’s overview proceeds from a quick full-frame glance to policy-selected patches, local CNN features, and recurrent aggregation for prediction.The global CNN produces coarse features used by the recurrent policy to choose the region.Source: Figure 2 · p. 3

3. Method

• AdaFocus targets task-irrelevant regions by localizing informative patches in each frame, reducing spatial redundancy without sacrificing accuracy.The method is presented as distinct from approaches focused on temporal redundancy.Source: §3 · p. 3

3.1. Network Architecture

• A lightweight global CNN supplies coarse frame features to a recurrent policy that selects patches for a high-capacity local CNN, whose features join global features for classification.The global and local networks serve quick localization and discriminative representation, respectively; the classifier aggregates information from processed frames.Source: §3.1 · pp. 3–4

3.2. Training Algorithm

• Training proceeds in three stages: warm up the CNNs and classifier on random patches, train the policy with reinforcement learning, then fine-tune the local CNN and classifier.The policy reward compares ground-truth confidence for its selected patch against the expected confidence from a random patch, yielding zero expected reward and more stable training.Source: §3.2 · pp. 4–5

3.3. Reducing Temporal Redundancy

• AdaFocus+ adds a recurrent policy that decides whether to process each frame’s patch with the local CNN, extending spatial selection with temporal skipping.Its reward weighs the confidence gain from local processing against a computation penalty proportional to patch size squared; at inference, a threshold controls which frames are processed.Source: §3.3 · p. 5

3.4. Offline Video Recognition

• For offline recognition, all frames are available together and the model uses the final-frame prediction; the global and local CNN computations can run in parallel on GPUs.The offline ActivityNet results are presented in Figure 5 for patch sizes 96x96, 128x128, 160x160, and 192x192, comparing AdaFocus with its temporal-skipping extension.Source: §3.4 · p. 6, Figure 5 · p. 6, Table 1 · p. 6

4. Experiment

• The experiments cover ActivityNet, FCVID, Mini-Kinetics, and Something-Something V1&V2, with comparisons to efficient video-recognition methods and an integration with TSM.The paper states five benchmarks, counting Something-Something V1 and V2 separately.Source: §4 · p. 6, §4.2 · p. 7

4.1. Comparisons with State-of-the-art Efficient Video Recognition Methods

• 2.1% higher mAP (83.4% v.s. 81.3%) with 1.3x less computation (26.6 GFLOPs v.s. 33.5 GFLOPs) on FCVID than AR-Net.The authors identify AR-Net as the strongest baseline in this comparison.Source: §4.1 · p. 7
• 2.1-3.2x less GFLOPs per video than AR-Net at a given mAP is reported for AdaFocus+; on Mini-Kinetics, it uses 1.6x less computation to reach 71.7% accuracy.The Mini-Kinetics comparison is 20.3 GFLOPs versus 32.0 GFLOPs; the authors also report a better mAP-FLOPs trade-off for AdaFocus across varied patch sizes.Source: Table 2 · p. 6, §4.1 · p. 7

4.2. Building on Top of Efficient CNNs

• For short Something-Something videos, AdaFocus uses one patch location per video because task-relevant regions change little across frames and adjacent inputs need similar visual angles.The policy-network architecture and training algorithm remain unchanged.Source: §4.2 · p. 8
• AdaFocus achieves the same performance as TSM+ with 1.5x less GFLOPs on Something-Something V1.The authors also report significant practical speedup on an Intel i7 CPU and NVIDIA 2080Ti GPU, with a slight drop from theoretical results that they tentatively attribute to inadequate hardware-oriented optimization.Source: Table 3 · p. 7, §4.2 · p. 8

4.3. Analytical Results

• In eight sampled ActivityNet frames, selected regions include task-relevant objects such as a sailboard, bicycle, and flute.Figure 7 uses green boxes to mark the selected patch locations.Source: §4.3 · p. 8, Figure 7 · p. 8
• Using the global feature for localization only degrades mAP by 1–1.5% compared with also reusing it for recognition.Table 4 reports the effects of reusing the global feature for recognition.Source: §4.3 · p. 8, Table 4 · p. 8
• Learned patch-selection policies perform considerably better than fixed alternatives, and the proposed reward function outperforms the tested confidence and increments rewards.Random sampling appears stronger than centre cropping, which the authors suggest may reflect spatial similarity between adjacent central patches.Source: §4.3 · pp. 8–9, Table 5 · p. 9

• AdaFocus applies a high-capacity network only to small, informative patches adaptively localized with reinforcement learning, reducing computational costs.The method can also be extended to dynamically skip less valuable frames.Source: §5 · p. 9
• Experiments report that AdaFocus outperforms existing works in both theoretical computational efficiency and actual inference speed.Source: §5 · p. 9

• MultiAgent selects important frames using multi-agent reinforcement learning.Source: A. Introduction of Baselines · p. 11
• SCSampler uses a lightweight framework to identify salient temporal clips in long videos.Source: A. Introduction of Baselines · p. 11
• LiteEval allocates computation adaptively according to frame importance using coarse and fine LSTMs.Source: A. Introduction of Baselines · p. 11
• AdaFrame uses reinforcement learning to select informative frames dynamically and perform adaptive inference.Source: A. Introduction of Baselines · p. 11
• Listen-to-look selects key video clips by fusing image and audio information; the comparison uses its image-based version.Source: A. Introduction of Baselines · p. 11
• AR-Net estimates frame importance and processes frames at different resolutions.Source: A. Introduction of Baselines · p. 11

• The global and local CNNs are initialized by fine-tuning ImageNet-pretrained MobileNet-V2 and ResNet-50 on full inputs for 15 epochs.The initial learning rate is 0.01.Source: §B.1 · p. 11
• Stage I trains the local CNN and classifier on randomly sampled patches for 50 epochs, without training the global CNN.The authors say training the global CNN did not significantly improve performance but increased training time.Source: §B.1 · p. 11
• Stage II trains the policy networks with Adam for 50/10 epochs, and Stage III fine-tunes only the classifier with the learned policy for 10 epochs.Further fine-tuning the local CNN brought trivial improvements while prolonging training time.Source: §B.1 · p. 11
• The implementation uses SGD with cosine learning-rate annealing, Nesterov momentum 0.9, batch size 64, and L2 regularization 1e-4.Source: §B.1 · p. 11

B.2. Training Hyper-parameters for Section 4.2

• AdaFocus initializes fG and fL using the same configuration as [31], and otherwise follows Section 4.1 except for specified changes.Source: §B.2 · p. 11
• Stage I trains fL and fC for 10 epochs, using initial learning rates of 1e-5 and 0.01, respectively.Source: §B.2 · p. 12
• Stage III uses an initial learning rate of 5e-4 for fC.Source: §B.2 · p. 12
• TSM+ follows the same training procedure as AdaFocus but does not train policy network π because it uses full frames as inputs.Source: §B.2 · p. 12