PaperlayerView exact revision on arXiv

Source-linked AI summary

FrameExit: Conditional Early Exiting for Efficient Video Recognition

Amir Ghodrati, Babak Ehteshami Bejnordi, Amirhossein Habibian

arXiv:2104.13400v1cs.CVcs.LG

TL;DR

Dense video recognition can impose substantial compute and latency costs, while prior approaches often select frames with learned policies. FrameExit combines deterministic sampling, accumulated feature pooling, and learned gates that stop processing when predictions are sufficiently reliable. It reports improved efficiency and accuracy across three large-scale benchmarks, including a 5× computation-cost reduction on HVU alongside improved recognition accuracy.

  • Problem

    Dense video models demand substantial data, computation, and latency, while frame-selection approaches may rely on assumptions or reward design and add sampler overhead.

    Source: §1 · p. 1

  • Method

    FrameExit combines deterministic frame sampling and accumulated feature pooling with learned gates trained using on-the-fly pseudo-labels to decide when to exit.

    Source: §1 · p. 2

  • Results

    5× reduction in computation costs on HVU while improving recognition accuracy over state-of-the-art methods; the method reports state-of-the-art performance on three large-scale datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FrameExit is model-agnostic and can be used with various network architectures.

    Source: §5 · p. 8

Abstract

from arXiv · show

In this paper, we propose a conditional early exiting framework for efficient video recognition. While existing works focus on selecting a subset of salient frames to reduce the computation costs, we propose to use a simple sampling strategy combined with conditional early exiting to enable efficient recognition. Our model automatically learns to process fewer frames for simpler videos and more frames for complex ones. To achieve this, we employ a cascade of gating modules to automatically determine the earliest point in processing where an inference is sufficiently reliable. We generate on-the-fly supervision signals to the gates to provide a dynamic trade-off between accuracy and computational cost. Our proposed model outperforms competing methods on three large-scale video benchmarks. In particular, on ActivityNet1.3 and mini-kinetics, we outperform the state-of-the-art efficient video recognition methods with 1.3$\times$ and 2.1$\times$ less GFLOPs, respectively. Additionally, our method sets a new state of the art for efficient video understanding on the HVU benchmark.

1. Introduction

Video recognition models can be costly, while static computation and learned frame-selection policies have limitations. FrameExit combines deterministic sampling with learned early-exit gates, adjusting computation to video difficulty; it reports improved efficiency and accuracy across three benchmarks.

1. Introduction

  • Dense 2D or 3D models deliver strong action recognition, but their data, compute, and latency demands limit use on resource-constrained devices.Source: §1 · p. 1
  • Prior frame-selection methods may assume sampler and recognition salience align, require carefully selected rewards, or add sampler-network overhead.Source: §1 · p. 1
  • FrameExit uses recognition loss as a proxy to generate on-the-fly pseudo-labels that train gates to stop when inference is sufficiently reliable.Source: §1 · p. 2
  • A simple, deterministic sampling strategy paired with accumulated feature pooling yields accurate action recognition without complex sampling policies.Source: §1 · p. 2
  • A cascade of gates adapts computation to video difficulty, reducing costs by stopping processing when appropriate.Source: §1 · p. 2
  • 5× reduction in computation costs on HVU while improving recognition accuracy over state-of-the-art methods; results show state-of-the-art performance on three large-scale datasets.Source: §1 · p. 2

2. Related work

Prior efficient-video work includes lightweight architectures, learned frame or clip selection, and conditional computation. FrameExit instead applies early exiting over video’s temporal dimension, using deterministic sampling and gating rather than intermediate classifiers over network features.

2. Related work

  • Efficient video architectures reduce compute through factorized filters [47] [57], efficient 3D extensions [46], shifting [35], or neural architecture search [39] [11]; FrameExit is architecture-agnostic and complementary.Source: §2 · p. 2
  • Frame-selection approaches use policies to choose frames or processing actions, but may add overhead, require joint training, or depend on carefully designed rewards.Source: §2 · p. 2
  • Image-classification early-exit methods use intermediate classifiers so simpler examples can exit early while harder ones proceed for more accurate predictions [34] [43].Source: §2 · p. 2
  • FrameExit shifts early exiting from intermediate network features to the temporal dimension of video recognition.Source: §2 · p. 3

3. FrameExit

FrameExit incrementally builds a video representation and uses gates to stop processing at the earliest sufficiently reliable prediction. Its deterministic frame sampling and accumulated feature pooling support this per-video allocation of computation.

3. FrameExit

  • Each sampled frame is encoded, aggregated into a clip-level representation, and passed to its classifier and associated gate.Processing starts with a single-frame clip and adds temporal detail until a gate halts it.Source: §3 · p. 3
  • Each gate uses aggregated representations from adjacent steps to decide whether to halt, with the first gate receiving only z1.Considering agreement between temporal features can mitigate a confident but incorrect prediction based only on the current representation.Source: §3 · p. 3
  • The model selects the earliest frame meeting the gating condition; if no gate halts, the last classifier predicts the video label.Source: §3 · p. 3
  • A deterministic, parameter-free sampling policy selects frames coarse-to-fine, starting at the video’s middle, beginning, and end before sampling halves.The policy is presented as an alternative to learned sampling policies.Source: §3 · pp. 3–4
  • Accumulated feature pooling incrementally updates the clip representation using the newly sampled frame and prior features; Ψ may use average/max pooling, LSTM, or self-attention.Source: §3 · p. 4
  • Gates are trained with classification-loss-derived pseudo-labels, exiting when a classifier’s loss is sufficiently low and otherwise continuing.The threshold ϵt increases at later stages; β controls the accuracy–computation trade-off, with higher β yielding more computational savings.Source: §3 · p. 4

4. Experiments

FrameExit is evaluated on three video datasets spanning action recognition and holistic video understanding, with accuracy and per-video GFLOPs used to assess performance and cost. It compares favorably with efficient recognition methods, and its ablations examine sampling, feature pooling, frame count, and adaptive exiting.

4.1. Experimental setup

  • The experiments cover ActivityNet-v1.3, Mini-Kinetics, and HVU, measuring mAP or top-1 accuracy as appropriate and computational cost in per-video GFLOPs.The implementation uses ResNet-50, EfficientNet-b3, or X3D-S backbones and ten input frames by default.Source: §4.1 · p. 5

4.2. Results

  • Higher β encourages more videos to exit early; later classifiers handle the harder remaining examples, and premature exits lose accuracy while later exits add little.The matrix compares the actual exit classifier against other classifiers on the same videos.Source: Figure 4 · p. 5, §4.2 · p. 6
  • On ActivityNet, categories with strong object or scene cues exit early with high accuracy, while complex actions require more processed frames.The category comparison plots average precision against the average number of frames needed to exit confidently.Source: Figure 5 · p. 6, §4.2 · p. 6
  • FrameExit with ResNet improves accuracy while using 1.3×–5× less GFLOPs; with EfficientNet, it beats AR-Net using 1.3× and 2.1× less GFLOPs on ActivityNet and Mini-Kinetics.With X3D-S, it reaches 86.0% at similar computational cost, 6.0% higher than the best 2D model.Source: Table 1 · p. 6, §4.2 · pp. 6–7
  • FrameExit sets a new state of the art on HVU, including 45.7% mAP at 8.6 GFLOPs and 49.2% mAP at 18.7 GFLOPs with ResNet.The EfficientNet variant uses 5.7 GFLOPs and achieves 46.1 mAP.Source: Table 2 · p. 7, §4.2 · p. 7

4.3. Ablation study

  • The deterministic sampling policy performs best among the tested policies; random sampling improves over sequential sampling, which has lower mAP and higher GFLOPs.The authors attribute the benefit to sampling that jumps forward and backward through the video.Source: §4.3 · p. 8, Table 3 · p. 8
  • Pooling over features is more effective than pooling over frame-level predictions to produce video-level predictions.The comparison changes whether pooling occurs in feature space or prediction space.Source: §4.3 · p. 8, Table 4 · p. 8
  • Performance rises with input frames only to a limit, motivating T = 10; conditional exiting consistently outperforms fixed-frame budgets.The frame-count study attributes the limit in part to 2D convolutional networks' capacity to leverage temporal information.Source: §4.3 · p. 8, Table 5 · p. 8, Table 6 · p. 8

5. Conclusions

FrameExit uses gates to determine an early exit based on inferred video complexity, with accumulated feature pooling supporting reliable decisions. The authors report that the method balances accuracy and efficiency across action recognition and holistic video understanding.

5. Conclusions

  • FrameExit uses gating modules to select an exit point based on the inferred complexity of each input video.Accumulated feature pooling provides the video representation used to support reliable gate decisions.Source: §5 · p. 8
  • The authors report that simple deterministic sampling combined with early exiting obviates the need for complex sampling policies.Source: §5 · p. 8
  • The method is model-agnostic and is reported to balance accuracy and efficiency across action recognition and holistic video understanding tasks.Source: §5 · p. 8

A. Algorithmic description of the policy

The policy samples frames in a coarse-to-fine sequence, beginning at widely spaced temporal positions and progressively adding detail.

A. Algorithmic description of the policy

B. Wall-clock time inference

FrameExit's wall-clock speedup is reported to track FLOPs because the backbone's frame count drives most computation. With a ResNet50 on an Nvidia GeForce GTX 1080Ti, the reported inference times decrease at lower computational costs.

B. Wall-clock time inference

C. Qualitative results

FrameExit can exit after observing only a few discriminative frames, while videos with greater scene complexity require more temporal information for a confident prediction. Figure 7 illustrates the sampled frames and the method's observed frames across video clips.

C. Qualitative results

  • A few frames can suffice when the observed frames are discriminative enough for classification.Source: C. Qualitative results · p. 11
  • As scene complexity increases, FrameExit needs more temporal information to reach a confident prediction.The text gives obscured objects and cluttered backgrounds as examples of greater complexity.Source: C. Qualitative results · p. 11
  • Figure 7 rows contain 10 sampled video frames, and the green box marks the frame observed by FrameExit for recognition.Source: Figure 7 · p. 11

Related papers

Loading 2104.13400v1…