PaperlayerView exact revision on arXiv

Source-linked AI summary

Tiny Video Networks

AJ Piergiovanni, Anelia Angelova, Michael S. Ryoo

arXiv:1910.06961v3cs.CV

TL;DR

Video understanding must integrate spatio-temporal information, while prior video models were computationally intensive and often unsuitable for real-time processing. This paper uses evolutionary architecture search to design Tiny Video Networks, which perform competitively or better across video datasets at real-time or faster speeds. The authors identify mobile deployment and lower-cost exploration of video architectures as potential impacts.

  • Problem

    Video understanding requires integrating spatio-temporal information, while previous computationally intensive models were not suitable for real-time video processing.

    Source: §1 · p. 1

  • Method

    Evolutionary architecture search automatically designs Tiny Video Networks from low-level primitives across the video spatio-temporal domain.

    Source: §1 · p. 1

  • Results

    Across four video datasets, Tiny Video Networks outperform or rival strong state-of-the-art methods that are tens to hundreds of times slower.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors identify mobile and robotics deployment and lower-cost exploration of video architectures as potential impacts of Tiny Video Networks.

    Source: §7 · p. 8

    The results are disadvantaged by MiT pretraining because Kinetics pretraining provides better performance, and the authors were unable to use Kinetics.

    Source: §4.3 · p. 6

Abstract

from arXiv · show

Video understanding is a challenging problem with great impact on the abilities of autonomous agents working in the real-world. Yet, solutions so far have been computationally intensive, with the fastest algorithms running for more than half a second per video snippet on powerful GPUs. We propose a novel idea on video architecture learning - Tiny Video Networks - which automatically designs highly efficient models for video understanding. The tiny video models run with competitive performance for as low as 37 milliseconds per video on a CPU and 10 milliseconds on a standard GPU.

1. Introduction

Video understanding requires modeling information across frames, while prior video models remained computationally intensive and unsuitable for real-time use. Tiny Video Networks use evolutionary architecture search to produce efficient video models that perform strongly across datasets and run at real-time or faster speeds.

1. Introduction

  • Evolutionary architecture search designs a family of video networks from low-level primitives to achieve strong recognition performance at a fraction of the computational cost.The search spans the full spatio-temporal domain despite video examples containing far more data, such as 30 times the frames.Source: §1 · p. 1
  • On Charades, Tiny Video Networks surpass X3D and SlowFast in both GFlops and accuracy, while the evaluated models outperform or rival leading methods that are tens to hundreds of times slower.The evaluation covers Moments-in-Time, HMDB, Charades, and MLB.Source: Figure 1 · p. 1, §1 · p. 2
  • TVNs run at real-time or faster speeds, including within 20 ms per video clip on a GPU, while occupying a high-accuracy, low-runtime region of the accuracy-runtime curve.The cited clip is approximately one second long and contains 30 frames.Source: §1 · p. 1
  • The search space combines efficient layers that interleave spatial- and temporal-domain processing, and the resulting mobile TVNs significantly outperform video MobileNet models in both accuracy and runtime.The paper also presents the architectures as simple and interpretable examples of discovered video networks.Source: §1 · pp. 1–2
  • TVNs are proposed for real-time video tasks on mobile phones and robots, as well as computationally heavy or energy-sensitive processing and lower-cost video-architecture research.The authors state that faster model training and exploration could accelerate video-understanding research.Source: §1 · p. 2

2. Related Work

Prior work addresses efficient networks, neural architecture search, and video-model computation, but video architecture search has been scarce and has produced relatively expensive models. This work instead constructs efficient video architectures from first principles and describes its approach as complementary to methods that reuse computations across frames.

2. Related Work

  • Earlier work optimized networks for specific hardware or mobile devices, while neural architecture search demonstrated accuracy gains and time-constrained model design.Some time-constrained NAS approaches use hardware in the loop.Source: §2 · p. 2
  • Efficient video models have been developed by expanding fast 2D models, while video architecture search has been scarce and had produced relatively expensive networks.The authors position their first-principles architecture construction as competitive with the best architecture-searched models.Source: §2 · p. 2
  • Prior efficiency work reduces optical-flow computation or recognizes compressed videos, but relies on heavy CNNs; this work targets compact CNN architectures that can also run on compressed video.The authors describe their approach as complementary to online methods that reuse computations across frames.Source: §2 · p. 2

3. Tiny Video Networks

Tiny Video Networks are designed through evolutionary search over efficient spatial and temporal building blocks, with runtime and optional size or memory constraints. The search explores a broad video-specific design space, evaluates validation accuracy, and discards architectures that violate constraints.

3. Tiny Video Networks

  • The search combines spatial and temporal layers from basic primitives, allowing evolution to select sequences that learn spatiotemporal features efficiently.The design excludes expensive video-specific elements such as 3D convolutions and can choose input resolution, frame count, and sampling rate.Source: §3 · pp. 2–3

3.1. Tiny Video Architecture Search

  • Evolutionary search optimizes validation accuracy under runtime and optional parameter-size or memory constraints, discarding architectures that fail those constraints.Fitness is the sum of top-1 and top-5 accuracy; runtime is measured on a desktop CPU.Source: §3.1 · pp. 3–4
  • Architectures begin as 200 random networks; tournament selection and discrete mutations iteratively produce diverse candidates from high-performing parents.Mutations can change resolution, block count, or a layer within a block.Source: §3.1 · p. 3

3.2. Search space

  • The search space spans input resolution, 1–32 frames, sampling rate, repeated blocks, layer types, and layer parameters such as kernel size and filter count.Blocks may include spatial or temporal convolutions, pooling, non-local, context-gating, and squeeze-and-excitation layers; residual connections are optional.Source: §3.2 · p. 4
  • With about 10^13 possible architectures, exhaustive search is prohibitive; evolutionary search rejects unstable or over-time models and focuses exploration on viable candidates.The authors report that the best architectures perform similarly across different evolution runs when population size and rounds are sufficient.Source: §3.2 · p. 4
  • The default search optimizes CPU runtime, while the search space and constraints can be adapted for other devices, FLOPs, parameter counts, or memory footprints.Networks with many parameters can still run efficiently when they use wide parallel layers rather than deep sequential ones.Source: §3.2 · p. 4

4. Experiments

The experiments evaluate Tiny Video Networks (TVNs) on four video datasets and compare their accuracy and efficiency with prior methods. Across the reported comparisons, TVNs are competitive or better while running substantially faster; the search also produces architectures with different runtimes and capacities.

4.2. Experimental setup

  • The evaluation covers MiT, Charades, HMDB, and MLB, and cross-evaluates models evolved on one dataset against the others.Evaluation reports runtime, FLOPs, and accuracy or mAP; TVNs typically use fewer input frames than the baselines.Source: §4 · p. 4, §4.2 · p. 6

4. Experiments

  • TVNs use simple, deeper architectures of fast layers; non-local layers are rarely selected, while cheaper context-gating layers are common.The authors suggest that the search favors spending computation on deeper or wider networks.Source: §4.1 · p. 5
  • TVN-1 runs at 37ms on CPU and 10ms on GPU; the search also produces models under different runtime, parameter, and memory constraints.For example, TVN-6 uses 32 input frames and runs at 142ms on CPU and 18ms on GPU.Source: §4.1 · p. 5, Table 1 · p. 5

4.3. Comparison to the state-of-the-art

  • On Charades, many TVNs without pretraining outperform Kinetics-pretrained state-of-the-art models and rival flow-equipped AssembleNet variants while running orders of magnitude faster.TVN-4, TVN-5, and TVN-6 are reported to outperform the best state-of-the-art models that use Kinetics pretraining; the authors pre-trained their models on MiT.Source: Table 2 · p. 5, §4.3 · p. 6
  • On MiT, TVN-1 and TVN-2 outperform ResNet-18 by 57x and 33x faster runtimes, and TVN-6 outperforms ResNet-101 while being 26x faster.TVN-1 and TVN-2 match ResNet-34 performance while being 61x and 35x faster, respectively.Source: §4.3 · p. 6, Table 3 · p. 6
  • The MLB results table reports TVN-6 at 56.4 mAP with 142ms CPU runtime, while prior-work runtimes are unavailable.The table caption marks the authors' runtime measurement with an asterisk.Source: §4.3 · p. 7, Table 4 · p. 7
  • TVN-5 and TVN-6 outperform I3D on Charades while being 25 and 13 times faster, respectively.The results also report that TVN-6 outperforms while being at least 13 times faster.Source: Table 2 · p. 5, §4.3 · p. 7
  • On HMDB, TVNs are comparable to and outperform ECO and TSM; TVN-1 reaches 72.1 with 2 frames, while TVN-4 reaches 74.7 with 8.The comparison reports ECO at 68.5 with 16 frames and 61.7 with 4 frames.Source: §4.3 · p. 7, Table 5 · p. 7

5. Tiny Video Models for Mobile Deployment

The paper adds mobile-friendly components to the TVN search space and compares the resulting models with MobileNet-based video models. The selected mobile TVNs reduce computation and parameters relative to TVN-1, while TVNs are reported to outperform MobileNet adaptations in speed and accuracy.

5.1. Mobile-friendly Tiny Video Networks

  • TVN-M-1 and TVN-M-2 achieve 23% fewer Flops and almost twice fewer parameters than TVN-1, with a small reduction in accuracy.Replacing TVN-1's ReLU activations with hard-swish improves accuracy by 1.7% with a 2ms runtime loss.Source: §5.1 · p. 7, Table 7 · p. 8

5.2. Comparison to MobileNet models.

  • TVNs outperform MobileNetV3-based video models in both accuracy and speed, with the advantage especially notable at larger frame counts.The comparison includes TVNs using 2 and 8 frames and a single-frame TVN evolved on MiT.Source: §5.2 · p. 8, Table 7 · p. 8

6. Ablation results

The ablations show that simply increasing TVN-1's frame count on MiT adds runtime without significant performance gains, while evolved or scaled models offer stronger efficiency-performance trade-offs. TVNs also compare favorably with (2+1)D ResNet-50 in runtime and accuracy.

Exploring a range of number of frames

  • For scene-based MiT activities, a single frame can often suffice to discriminate the activity, and the 1-frame TVN performs reasonably on MiT.The passage gives swimming and baseball as examples.Source: §6 · p. 8
  • TVN-1's accuracy on MiT does not improve significantly when its input grows from 2 to 8 or 16 frames, although runtime rises substantially.The table caption likewise notes that simply adding frames is not greatly beneficial.Source: §6 · p. 8, Table 8 · p. 8
  • TVN-5, also using 16 frames, is 2.3 times faster and 6.3% more accurate on MiT than TVN-1.The comparison accompanies the finding that adding frames to TVN-1 does not significantly improve performance while increasing runtime.Source: §6 · p. 8

Scaling Up the TVNs

  • EfficientNet-scaled TVN-1 outperforms the other scaling versions and ResNet-50 while running 10x faster, but evolved TVN-5 and TVN-6 are both more accurate and faster than scaled models.The scaling experiments vary resolution, width, and depth.Source: §6 · p. 8, Table 9 · p. 8

Comparison to (2+1)D ResNet-50

  • Even with one input frame, (2+1)D ResNet-50 runs at 185ms on CPU, far slower than a TVN; TVN-1 outperforms ResNet-50 in accuracy until ResNet-50 uses 16 frames.Source: §6 · p. 8

7. Conclusion

Tiny Video Networks are automatically designed efficient video architectures that achieve strong results or outperform state of the art on four video datasets. The authors identify mobile deployment and lower-cost exploration of video architectures as potential impacts.

7. Conclusion

  • TVNs achieve strong results or outperform state of the art on four video datasets while running at unprecedented speeds.The conclusion identifies mobile-friendly models that meet runtime and parameter constraints for mobile deployment.Source: §7 · p. 8
  • A one-frame (2+1)D ResNet-50 runs at 185ms on CPU, far slower than a TVN, and more accurate ResNet-50 models run at more than 2000ms on CPU.The figure compares (2+1)D ResNet-50 models using different frame counts.Source: Figure 6 · p. 9
  • The authors prepared code and models for open-sourcing.Source: §7 · p. 9

Related papers

Loading 1910.06961v3…