PaperlayerView exact revision on arXiv

Source-linked AI summary

MoViNets: Mobile Video Networks for Efficient Video Recognition

Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, Boqing Gong

arXiv:2103.11511v2cs.CVcs.AIcs.LG

TL;DR

Video recognition needs models that combine the accuracy of 3D CNNs with the low resource use and online operation required on mobile devices. MoViNets combine neural architecture search, stream buffers, and temporal ensembling to achieve efficient streaming recognition, with strong cross-dataset performance and author-reported hardware and architecture limitations.

  • Problem

    3D CNNs are accurate but require substantial computation and memory, while 2D CNNs are more resource-efficient and support online inference at lower accuracy.

  • Method

    MoViNets use neural architecture search, constant-memory Stream Buffers for consecutive subclips, and temporal ensembles of streaming models.

  • Results

    MoViNets achieve high accuracy and efficiency across Kinetics, Moments in Time, Charades, and other video-recognition datasets; MoViNet-A6 reaches 83.5% accuracy with 60% fewer FLOPs than X3D-XL.

  • Takeaways & Limitations

    MoViNets support online inference while reducing memory and computation costs for video recognition.

  • Takeaways & Limitations

    Stream Buffers provide less memory reduction for ResNet3D-50, and FLOPs-based NAS can yield slower GPU execution than MobileNetV3.

Abstract

from arXiv · show

We present Mobile Video Networks (MoViNets), a family of computation and memory efficient video networks that can operate on streaming video for online inference. 3D convolutional neural networks (CNNs) are accurate at video recognition but require large computation and memory budgets and do not support online inference, making them difficult to work on mobile devices. We propose a three-step approach to improve computational efficiency while substantially reducing the peak memory usage of 3D CNNs. First, we design a video network search space and employ neural architecture search to generate efficient and diverse 3D CNN architectures. Second, we introduce the Stream Buffer technique that decouples memory from video clip duration, allowing 3D CNNs to embed arbitrary-length streaming video sequences for both training and inference with a small constant memory footprint. Third, we propose a simple ensembling technique to improve accuracy further without sacrificing efficiency. These three progressive techniques allow MoViNets to achieve state-of-the-art accuracy and efficiency on the Kinetics, Moments in Time, and Charades video action recognition datasets. For instance, MoViNet-A5-Stream achieves the same accuracy as X3D-XL on Kinetics 600 while requiring 80% fewer FLOPs and 65% less memory. Code will be made available at https://github.com/tensorflow/models/tree/master/official/vision.

1. Introduction

MoViNets address the accuracy–efficiency gap in video recognition by combining neural architecture search, stream buffers, and temporal ensembles. Together, these techniques produce accurate, computationally efficient 3D CNNs with reduced memory usage and online inference support.

  • 3D CNNs provide high accuracy but consume substantial memory and computation, while 2D CNNs use fewer resources and support online inference at lower accuracy.
  • MoViNets use neural architecture search over spatial, temporal, and spatiotemporal operations to trade off feature representation, accuracy, and efficiency.
  • 83.5% accuracy is achieved by MoViNet-A6, 1.6% higher than X3D-XL while requiring 60% fewer FLOPs.
  • A Stream Buffer processes consecutive subclips with constant memory while preserving temporal dependencies, enabling online inference and reducing MoViNet-A5 memory usage by 90%.It caches feature maps at subclip boundaries and supports temporally unidirectional causal operations.
  • Temporal ensembles average logits from two independently trained streaming MoViNets with the same total FLOPs as one model, recovering stream-buffer accuracy loss.
  • The resulting MoViNets combine high accuracy, low memory usage, efficient computation, and online inference across multiple video-recognition datasets.

2. Related Work

Prior video-recognition work extends image models with temporal processing, while efficiency efforts use 2D segmentation, temporal shifting, and causal modeling approaches.

  • 3D convolution is a prominent way to extend 2D image models with a temporal dimension for video understanding.
  • Efficiency-oriented approaches process videos in smaller 2D segments with late fusion or shift channels along the temporal axis for online inference.

3. Mobile Video Networks (MoViNets)

MoViNets use neural architecture search, stream buffers, and temporal ensembling to build efficient 3D video networks that retain long-range temporal modeling while supporting online inference. The design reduces memory growth with video length and targets mobile-friendly computation and accuracy.

  • Progressive approach: The MoViNet design process combines search-space design, stream buffers, and temporal ensembling to improve efficiency, memory usage, and accuracy.The approach searches architectures, carries temporal information across subclips, and uses ensembling to recover accuracy.
  • Searching for MoViNets: NAS explores MobileNetV3-based 3D architectures by varying widths, depths, kernel sizes, and spatial, temporal, or spatiotemporal operations.The search space expands 2D mobile-network blocks for video and searches kernel and bottleneck configurations without temporal downsampling.
  • The Stream Buffer with Causal Operations: Stream buffers process consecutive non-overlapping subclips while caching boundary features, preserving temporal dependencies without recomputing overlapping activations.Causal operations prevent future-frame dependencies and support frame-wise online inference.
  • The Stream Buffer with Causal Operations: Stream-buffer memory depends on O(b + T_clip), remaining constant as total video length or the number of subclips increases.The buffer retains only the last b frames of concatenated features while processing each subclip.
  • Temporal Ensembling: Stream buffers reduce memory footprints by up to an order of magnitude at about a 1% Kinetics 600 accuracy cost, which temporal ensembling can restore.The ensemble trains two independent MoViNets with the same architecture and is presented as an accuracy-recovery strategy.

4. Experiments on Video Classification

Experiments show that MoViNets combine competitive video-recognition accuracy with reduced computation and memory, while supporting streaming inference across multiple datasets. Stream buffers keep memory nearly constant with input duration, and temporal ensembling restores accuracy at equivalent FLOPs.

  • Kinetics 600: MoViNet-A2 achieves slightly higher accuracy than 30-clip X3D-XS or X3D-M using almost half their GFLOPs.The comparison evaluates total computation across the video rather than per subclip.
  • Kinetics 600: 84.8% accuracy is achieved by MoViNet-A6 with AutoAugment without pretraining, while using 40% of TimeSformer-HR’s FLOPs and exceeding its accuracy by 1%.The reported comparison is on Kinetics 600.
  • Streaming memory: Stream buffers reduce memory from linear to constant in input-frame count, producing an order-of-magnitude peak-memory reduction for larger MoViNets A3–A6.The stream buffer processes consecutive subclips while retaining temporal information across boundaries.
  • Streaming accuracy: Temporal ensembling restores the approximately 1% accuracy lost to stream buffers without additional inference cost by combining two half-frame-rate models at equal GFLOPs.Accuracy improvements are reported across streaming architectures, particularly as model size increases.
  • Other datasets: Across other datasets, MoViNets improve over prior models: MoViNet-A5 reaches 39.9% on Moments in Time and 63.2% on Charades without optical-flow or object-segmentation inputs.On Charades, the results are presented as evidence of temporal understanding for longer-duration clips.

5. Conclusion

MoViNets provide efficient models that transfer across video recognition datasets. Combined with stream buffers, they reduce training and inference memory costs while supporting online inference on streaming video.

  • MoViNets provide a highly efficient set of models that transfer well across different video recognition datasets.
  • Coupled with stream buffers, MoViNets reduce training and inference memory cost while supporting online inference on streaming video.

Appendices

The appendices provide supplementary details on the search space, MoViNet architectures, and additional experiments. They also report ablations on the datasets studied in the main text.

  • Appendix A details the search space, its scaling technique, and the search algorithm.
  • Appendix B documents the neural architectures of MoViNets A0–A7.
  • Appendix C reports additional results and ablation studies on the datasets studied in the main text.

A. MoViNet Architecture Search

The architecture search scales a MoViNet search space across model dimensions and uses one-shot neural architecture search to optimize accuracy and FLOPs. Stream buffers reduce memory during search by processing inputs in subclips.

  • Search-space scaling: A single scaling parameter φ progressively expands the search space across width, depth, input resolution, and frame rate.
  • Search-space scaling: The coefficients satisfy αβ^2γ^2δ ≈ 4, so increasing φ by 1 multiplies the average model size by 4.
  • Search-space scaling: The search-space parameters include a frame-rate scaling rule τ = 5f, with parameters rounded to the nearest multiple of 8.
  • Coefficient selection: The scaling coefficients are selected through random search over candidate values, using short training runs and averaged accuracy to choose the final coefficients.
  • Neural architecture search: TuNAS trains a one-shot hypernetwork, while REINFORCE learns a policy that samples architectures using accuracy and FLOPs as reward components.
  • Stream-buffer memory: Stream buffers process inputs in subclips to reduce memory during NAS, enabling larger batch sizes and improving held-out NAS accuracy without significantly increasing per-device GPU memory.
  • Stream-buffer memory: For a temporal kernel width k = 5, each layer caches only the last 4 frames, minimizing discarded frames and memory consumption.

B. Architectures of MoViNets

The appendix provides architecture definitions for MoViNet A0–A5 and describes MoViNet-A6 as an ensemble of A4 and A5.

  • Tables 17–22 define the architectures of MoViNet A0–A5.
  • MoViNet-A6 ensembles architectures A4 and A5 by independently training both models and averaging their logits during inference.
  • All layers use squeeze-and-excitation layers, so this search hyperparameter is omitted from the architecture tables.

B.1 More Details of the Architectures and Training

The architectures combine residual-training refinements, configurable frame rates, and causal convolutions to support efficient video processing. Causal padding restricts each output to current and past frames.

  • Architecture and training: ReZero, residual skip connections, and 1x1x1 convolutions are used to improve convergence, feature representations, channel changes, or downsampling.ReZero applies zero-initialized learnable scalar weights before residual summation; skip connections add a 1x1x1 convolution in each block’s first layer.
  • Architecture and training: Polyak averaging, Hard Swish, RMSProp, synchronized batch normalization, and 240 training epochs define the main training configuration.The optimizer uses momentum 0.9, a base learning rate of 1.8, and batch size 1024; Hard Swish is selected for quantization and CPU inference.
  • Training objectives: Charades uses sigmoid cross-entropy because videos can contain multiple action labels, with frame aggregation before temporal global average pooling.Other datasets use softmax cross-entropy with label smoothing 0.1.
  • Frame-rate choices: Most models are efficient at 50 frames and 5fps or 80 frames and 8fps, while MoViNet-A4 benefits from 12fps.Charades uses 64 frames at 6fps for both training and inference.
  • Causal convolution: Causal convolution uses one-sided temporal padding so each voxel’s receptive field spans only frames from the beginning through time t.This padding shift forces output causality for a temporal kernel size of 3.

C.2 Additional Details of Datasets

The evaluation uses shrinking, currently available Kinetics datasets and reports results across multiple benchmark tables. Dataset availability complicates comparisons with older work.

  • Dataset availability: Kinetics comparisons are difficult because videos taken offline have progressively reduced dataset sizes.The paper reports the most recently available videos and notes that each dataset loses about 4% of examples per year.
  • Reported benchmarks: Results are reported for Kinetics 400, Kinetics 600, Kinetics 700, Moments in Time, Charades, Something-Something V2, and Epic Kitchens 100.The corresponding results appear in Tables 9–15.

C.3 Single-Clip vs. Multi-Clip Evaluation

Single-clip evaluation reports computation per video more directly than multi-clip evaluation, while longer coverage and training clips can improve accuracy. Stream buffers make longer training clips feasible with limited memory.

  • Reported benchmarks: The evaluation tables cover Kinetics 700, Moments in Time, Charades, Something-Something V2, and Epic Kitchens 100 in addition to the Kinetics 600 comparisons.The table captions specify evaluation frames for Moments in Time, Something-Something V2, and Epic Kitchens 100.
  • Evaluation protocol: Single-clip evaluation covers the video directly, whereas multi-clip evaluation can increase accuracy by adding clips and obscure total computation per video.The authors evaluate their results on a single view without multi-clip evaluation and report evaluation frames and frame rates.
  • Multi-clip evaluation: Multi-clip accuracy improves when clips fill the video duration, including five clips for 8-frame training and three clips for 16-frame training.Adding more clips after covering the full duration produces only slight further improvement.
  • Single-clip evaluation: Evaluating a MoViNet-A2 trained on 16 frames with 80 frames covering the full 10-second video outperforms using the same number of frames in multi-clip evaluation.The comparison uses the same MoViNet-A2 model with frame stride 3.
  • Memory-efficient training: Training with 48 frames can raise accuracy further, while stream buffers limit training memory to that required for embedding 16 frames at a time.The recurrent stream-buffer approach enables longer effective training clips without retaining all frames’ activations.

C.4 Streaming vs. Non-Streaming Evaluation

Streaming MoViNets use causal convolutions and stream buffers to process long videos with constant memory, though their per-frame accuracy dynamics differ from non-streaming models. Longer evaluation clips generally improve accuracy.

  • Accuracy dynamics: Non-streaming MobileNetV3 and MoViNet-A2 accuracy peaks near the center of videos, with MoViNet-A2 achieving higher accuracy.Both exhibit an average Laplace-shaped accuracy distribution, consistent with salient action content near video midpoints.
  • Accuracy dynamics: Streaming MoViNet-A2 with causal convolutions has more fluctuating accuracy because its receptive field is time-unidirectional rather than balanced across frames.The feature aggregation therefore differs from non-streaming architectures that can see all frames together.
  • Accuracy comparison: The overall accuracy difference across videos is about 1%, and streaming architectures nearly match non-streaming performance in top-5 accuracy.This comparison concerns the apparent information loss from causal convolutions.
  • Clip-duration generalization: MoViNets generalize beyond their training clip duration, with accuracy improving for more evaluation frames but declining when evaluated on shorter clips than training clips.Longer training clips translate to better accuracy on longer evaluations.
  • Streaming efficiency: Stream buffers enable frame-by-frame operation with constant memory as input length and temporal receptive field increase, while remaining more efficient than X3D on long sequences.The paper describes a memory advantage over multi-clip approaches despite an accuracy reduction in streaming models.
  • Alternative operations: Replacing 3D depthwise convolutions with 2D spatial followed by 1D temporal convolutions lowers FLOPs while retaining similar accuracy.MoViNet-A2b is a searched model similar to MoViNet-A2 using this (2+1)D option.
Loading 2103.11511v2…