PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition

Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, Larry S. Davis

arXiv:1912.01601v1cs.CV

TL;DR

Video samples vary in how much computation they need, while compact models typically spend the same amount on each sample. LiteEval uses coarse features and a conditional gate to allocate finer computation when needed. It reports similar or better offline accuracy than uniform sampling with substantially less computation, and the best accuracy-cost trade-off online on both datasets.

  • Problem

    Compact models require the same computation for every sample, although video clips vary in how much evidence is needed for recognition.

    Source: §1 · pp. 1–2

  • Method

    LiteEval combines coarse and fine LSTMs with a conditional gate that uses coarse features and history to decide whether to compute fine features for each frame.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    On FCVID and ActivityNet, LiteEval matches or exceeds uniform-sampling accuracy with 51.8% (94.3 vs. 195.5) and 51.3% (95.1 vs. 195.5) fewer GFLOPs, respectively, and offers the best accuracy-cost trade-off online.

    Source: §3.2 · pp. 5–6

  • Takeaways & Limitations

    Learned fine-feature use varies with prediction difficulty and can focus on salient frames while ignoring redundant ones.

    Source: §1 · p. 3, §3.2 · p. 7

Abstract

from arXiv · show

This paper presents LiteEval, a simple yet effective coarse-to-fine framework for resource efficient video recognition, suitable for both online and offline scenarios. Exploiting decent yet computationally efficient features derived at a coarse scale with a lightweight CNN model, LiteEval dynamically decides on-the-fly whether to compute more powerful features for incoming video frames at a finer scale to obtain more details. This is achieved by a coarse LSTM and a fine LSTM operating cooperatively, as well as a conditional gating module to learn when to allocate more computation. Extensive experiments are conducted on two large-scale video benchmarks, FCVID and ActivityNet, and the results demonstrate LiteEval requires substantially less computation while offering excellent classification accuracy for both online and offline predictions.

1 Introduction

Fixed-compute models spend the same resources on every sample, although video clips vary in how much evidence they need for recognition. LiteEval addresses this by adaptively allocating coarse or fine computation, and its reported offline results reduce computation relative to uniform sampling while maintaining similar or better accuracy.

1 Introduction

  • Compact models reduce computation but apply the same amount to every sample, while harder corner cases may require more expensive models or ensembles.Input resolution also affects CNN computation; the passage gives a ResNet-101 example in which half-resolution inputs save 74% of floating-point operations while retaining reasonable accuracy.Source: §1 · p. 1
  • Video recognition has variable computational demands: static scenes may be recognized from one frame, while distinguishing similar activities can require more frames.The frames needed also vary within a class, including with camera motion and recording style.Source: §1 · p. 2
  • LiteEval processes each incoming frame with coarse features and historical information, using a gate to request fine features when needed or synchronize the two LSTMs otherwise.The coarse LSTM uses downsampled-frame features from a lightweight CNN; fine features from a more powerful CNN update the fine LSTM when further analysis is needed.Source: §1 · p. 2
  • Offline, LiteEval achieves similar or better accuracy than uniform sampling with 51.8% and 51.3% less computation on FCVID and ActivityNet, respectively.The experiments also report that LiteEval outperforms efficient video-recognition approaches in recent literature.Source: §1 · pp. 2–3

2 Approach

LiteEval allocates video-recognition computation coarse-to-fine: a coarse LSTM processes economical features, while a gate selects frames for finer, more expensive features. Its optimization combines video-level classification loss with a target on fine-feature use and uses Gumbel-Softmax to learn the binary gate differentiably.

2 Approach

  • LiteEval pairs coarse- and fine-scale recurrent processing with a conditional gate that decides when incoming frames receive more computation.The coarse LSTM uses lightweight-CNN features from downsampled frames; the fine LSTM uses finer-scale features from a more powerful CNN.Source: §2 · p. 3, §2.1 · p. 3

2.1 A Coarse-to-Fine Framework

  • The gate samples a binary decision from its predicted probabilities, using the prior fine-LSTM state as part of its decision context.This selects whether to compute finer-scale features for the current frame.Source: §2.1 · p. 3
  • When selected, the fine LSTM combines fine and coarse features with its prior states; otherwise, previous hidden states are reused.The coarse and fine LSTMs are synchronized by copying state when fine features are skipped.Source: §2.1 · pp. 3–4

2.2 Optimization

  • Training uses final-time-step predictions and a loss that balances classification accuracy against fine-feature use.The objective includes cross-entropy and limits fine-feature usage toward a predefined target; λ balances the accuracy–computation trade-off.Source: §2.2 · p. 4
  • Gumbel-Softmax makes the sampled binary gating decisions differentiable, with temperature annealed from high to a smaller positive value.The paper uses this approach instead of policy gradients for learning the gate.Source: §2.2 · p. 4

3 Experiments

Across FCVID and ActivityNet, LiteEval is evaluated for offline and online recognition, with substantial offline computation savings and strong online cost–accuracy trade-offs. The experiments also examine its learned fine-feature policies and the effects of gating and LSTM design choices.

3.1 Experimental Setup

  • The evaluation uses FCVID and ActivityNet, reporting mean average precision for offline recognition, top-1 accuracy online, and computational cost in GFLOPs.ActivityNet results are reported on its validation set because test labels are withheld.Source: §3.1 · p. 5

3.2 Main Results

  • 51.8% (94.3 vs. 195.5) and 51.3% (95.1 vs. 195.5) computational savings in GFLOPs versus uniform sampling accompany similar or better accuracies on FCVID and ActivityNet, respectively.The offline comparison covers alternative methods on both datasets.Source: §3.2 · p. 5, Table 1 · p. 6
  • LiteEval offers the best trade-off between computational cost and recognition accuracy for online prediction on both FCVID and ActivityNet.It also outperforms the frame-by-frame SEQ-K strategy; uniform-K is described as impractical online because the number of frames yet to arrive is unknown.Source: Figure 2 · p. 6, §3.2 · p. 6
  • Fine-feature usage varies across categories and within a class, consistent with differing computation requirements across video samples.Fine-feature use is proportional to overall computation required.Source: Figure 3 · p. 7, §3.2 · p. 7
  • Selected fine-feature frames provide salient information for recognizing a class, while redundant frames without additional information are ignored.Figure 4 marks selected frames with green borders.Source: Figure 4 · p. 7, §3.2 · p. 7

3.3 Ablation Studies

  • γ = 0.05 provides the best trade-off between computational cost and accuracy; very small γ, such as 0.01, can overlook important information.Relatively small γ values of 0.1 or less produce decent results, which the authors interpret as evidence of video-frame redundancy.Source: §3.3 · p. 7, Table 3 · p. 8
  • Without updating the fine LSTM hidden states from the coarse LSTM, performance falls to 65.7%, supporting the importance of synchronization.The authors say the transferred information keeps the fine LSTM aware of useful information seen so far.Source: §3.3 · p. 7, Table 2 · p. 8
  • Smaller coarse LSTMs degrade performance, while a fine LSTM with 2,048 hidden units gives the best results in the reported size experiments.LSTMs require only 0.06% of the GFLOPs needed to extract ResNet-101 features.Source: Table 4 · p. 8, §3.3 · p. 8

4 Related Work

Prior conditional-computation work adapts inference through early exits, layer selection, or region selection. LiteEval applies dynamic feature-use decisions to video, distinguishing itself through differentiable operation, online suitability, and complementarity with 3D CNNs.

4 Related Work

  • Conditional computation has been explored through cascaded classifiers, CNN exits, RNN halting, layer selection, and image-region attention.These approaches aim to reduce computation while maintaining recognition accuracy.Source: §4 · p. 8
  • LiteEval learns video feature-use decisions over time; unlike policy-search frame-selection methods, it is fully differentiable and does not require future frames, enabling online as well as offline use.Source: §4 · p. 8
  • LiteEval is complementary to lightweight 3D CNNs: fine-LSTM inputs can come from 3D CNN features, with their computation dynamically selected for incoming snippets.Source: §4 · p. 8

5 Conclusion

LiteEval combines coarse and fine LSTMs with a gating module to allocate video-feature computation dynamically. Its FCVID and ActivityNet experiments demonstrate the effectiveness of the approach for online and offline video prediction.

5 Conclusion

  • LiteEval uses coarse-scale features and a gating module to decide dynamically whether incoming frames need more powerful fine-scale features.A coarse LSTM and a hierarchically organized fine LSTM cooperate in the framework.Source: §5 · p. 8
  • The fine LSTM is synchronized to retain all information seen so far, making it available for predictions.Source: §5 · p. 8
  • Experiments on FCVID and ActivityNet demonstrate the framework's effectiveness for resource-efficient video prediction in both online and offline settings.Source: §5 · p. 8

Related papers

Loading 1912.01601v1…