PaperlayerView exact revision on arXiv

Source-linked AI summary

Resource Efficient 3D Convolutional Neural Networks

Okan Köpüklü, Neslihan Kose, Ahmet Gunduz, Gerhard Rigoll

arXiv:1904.02422v5cs.CV

TL;DR

Resource-efficient 3D CNN architectures are needed because video applications benefit from 3D kernels but existing efficient CNN work has focused mainly on 2D models. The paper converts five established efficient 2D architectures to 3D, evaluates them across three video benchmarks and two hardware platforms, and finds considerable performance with real-time execution while warning against overly narrow designs.

  • Problem

    Real-world video applications need resource-efficient 3D CNNs, but efficient CNN architectures have largely focused on 2D kernels despite 3D CNNs’ stronger video-recognition performance.

  • Method

    The paper converts SqueezeNet, MobileNet, ShuffleNet, MobileNetV2, and ShuffleNetV2 to 3D and evaluates accuracy across three benchmarks, complexity levels, FLOPs, and runtime on Titan XP and Jetson TX2.

  • Results

    Across four complexity levels, the architectures provide considerable classification performance, nearly all run in real time on both platforms, and depthwise convolutions capture motion well on Jester.

  • Takeaways & Limitations

    Width multipliers flexibly adjust model capacity, while pretrained models support transfer-learning use in other real-world applications.

Abstract

from arXiv · show

Recently, convolutional neural networks with 3D kernels (3D CNNs) have been very popular in computer vision community as a result of their superior ability of extracting spatio-temporal features within video frames compared to 2D CNNs. Although there has been great advances recently to build resource efficient 2D CNN architectures considering memory and power budget, there is hardly any similar resource efficient architectures for 3D CNNs. In this paper, we have converted various well-known resource efficient 2D CNNs to 3D CNNs and evaluated their performance on three major benchmarks in terms of classification accuracy for different complexity levels. We have experimented on (1) Kinetics-600 dataset to inspect their capacity to learn, (2) Jester dataset to inspect their ability to capture motion patterns, and (3) UCF-101 to inspect the applicability of transfer learning. We have evaluated the run-time performance of each model on a single Titan XP GPU and a Jetson TX2 embedded system. The results of this study show that these models can be utilized for different types of real-world applications since they provide real-time performance with considerable accuracies and memory usage. Our analysis on different complexity levels shows that the resource efficient 3D CNNs should not be designed too shallow or narrow in order to save complexity. The codes and pretrained models used in this work are publicly available.

1. Introduction

Resource-efficient CNN research has largely focused on 2D kernels, while video applications need efficient 3D CNNs that preserve spatio-temporal modeling under runtime, memory, and power constraints. This work converts established efficient 2D architectures to 3D and evaluates them across learning capacity, motion capture, transfer learning, and runtime complexity.

  • Real-world vision applications require CNNs to operate on computationally limited platforms under runtime constraints.Examples include face recognition, robot navigation, and augmented reality.
  • Resource-efficient CNN development has risen recently but remains limited mainly to 2D kernels.
  • 3D CNNs offer stronger video-recognition performance than 2D CNNs, yet efficient 3D architectures must also account for runtime, memory, and power budgets.
  • The study creates 3D versions of SqueezeNet, MobileNet, ShuffleNet, MobileNetV2, and ShuffleNetV2.
  • The models are evaluated on Kinetics-600 for learning capacity, Jester for motion-pattern capture, and UCF-101 for transfer-learning applicability.
  • Complexity is measured with FLOPs, while runtime is tested on an Nvidia Titan XP GPU and Jetson TX2 embedded system.The paper notes that FLOPs are indirect and do not directly indicate speed or latency.

2. Related Work

Prior efficient CNN architectures were designed for static-image benchmarks with 2D kernels, whereas 3D video CNNs have generally remained computationally heavy. This work targets resource-efficient 3D models below 1 GFLOPs by adapting established lightweight architectures for video benchmarks.

  • Efficient CNN research commonly uses pruning, quantization, factorization, or direct small-network design; this work focuses on directly designing efficient 3D CNNs.
  • Existing lightweight architectures such as SqueezeNet, MobileNet, ShuffleNet, MobileNetV2, and ShuffleNetV2 rely on parameter, convolution, or grouping strategies developed for 2D CNNs.
  • Prior resource-efficient architectures operate on static images and are benchmarked on ImageNet, leaving efficient 3D video architectures comparatively underexplored.
  • 3D CNNs require substantially more parameters and computation than 2D counterparts, although large video datasets have supported their training and evaluation.
  • Earlier 3D approaches include inflated filters, spatial-temporal convolution decomposition, and evaluations of 3D versions of major ImageNet architectures.
  • The study implements 3D versions of five efficient architectures at four complexity levels and evaluates models with less than 1 GFLOPs on three video benchmarks.

3. Resource Efficient 3D CNN Architectures

The paper introduces and evaluates resource-efficient 3D counterparts of well-known 2D CNN architectures, then compares their structural design choices and training details.

  • The proposed architectures are 3D versions of well-known resource-efficient 2D CNNs.The section presents their building blocks and network structures.
  • The architectures are compared by number of layers, nonlinearities, and skip connections.
  • The section concludes by describing model training details.

3.1. 3D Versions of Well-known Architectures

The paper converts five resource-efficient 2D CNN families into 3D architectures, adapting their building blocks and complexity controls for video inputs. The designs use different convolutional, shortcut, channel, and downsampling strategies, then compare architectural properties.

  • 3D-SqueezeNet: 3D-SqueezeNet uses Fire blocks and parameter-reduction strategies including 1×1 filters, reduced 3×3 input channels, and late downsampling.It is evaluated only in its default configuration because it lacks a width-multiplier control for generating different complexities.
  • MobileNet architectures: MobileNet factorizes convolution into depthwise and pointwise operations, while MobileNetV2 adds linear bottlenecks and shortcut connections.The MobileNetV2 design is intended to reduce inference memory usage and support deeper models through shortcut connections.
  • ShuffleNet architectures: ShuffleNet combines pointwise group convolution with channel shuffle, whereas ShuffleNetV2 splits channels into identity and convolution branches before concatenation and shuffling.ShuffleNet uses group number g=3, while ShuffleNetV2 varies stage channel counts across complexity levels.
  • Comparative analysis: The architectures are compared by convolutional and linear layers, ReLU nonlinearities, and addition or concatenation skip-connections.These criteria are treated as design factors relevant to architecture performance.

3.2. Training Details

The experiments train the architectures with SGD and benchmark-specific regularization and augmentation procedures. UCF-101 uses pretrained Kinetics-600 models with only the final layer fine-tuned, and inference aggregates clip scores per video.

  • Learning: Training uses SGD with categorical crossentropy, largest fitting mini-batches, momentum 0.9, dampening 0.9, and weight decay 1×10^-3.From-scratch training starts at learning rate 0.1 and reduces it by 10^-1 when validation loss converges.
  • Learning: UCF-101 training initializes from Kinetics-600 pretrained models, freezes network parameters, and fine-tunes only the final layer.Fine-tuning starts at learning rate 0.01, with two reductions after the 30th and 45th epochs.
  • Regularization: Regularization applies weight decay to all parameters and dropout ratios of 0.2 for Kinetics-600 and Jester versus 0.9 for UCF-101.The stronger UCF-101 dropout reflects its stated need for intensive regularization.
  • Augmentation: Temporal and spatial augmentation randomly selects clip and crop positions, applies multi-scale cropping, and flips Kinetics-600 and UCF-101 clips with 50% probability.Augmented network inputs contain 3 channels, 16 frames, and 112×112 spatial pixels.
  • Recognition: For Kinetics-600 and UCF-101 recognition, non-overlapping 16-frame clips are center-cropped, scored individually, and averaged per video.The highest-scoring class becomes the video label.
  • Implementation: The architectures are implemented in PyTorch and trained using a single Titan Xp GPU.

4. Experiments

The experiments compare resource-efficient 3D CNNs across benchmarks, complexity levels, and hardware, showing that depth, width, convolution type, and platform affect accuracy and runtime.

  • Datasets: Kinetics-600 evaluates network capacity, Jester emphasizes motion-pattern capture, and UCF-101 examines transfer-learning applicability.Kinetics-600 contains 600 action classes, while Jester largely controls spatial content to emphasize hand movement.
  • Accuracy: Deeper 3D-ShuffleNet, 3D-ShuffleNetV2, and 3D-MobileNetV2 outperform shallower 3D-SqueezeNet and 3D-MobileNetV1 architectures.The results argue against making resource-efficient 3D CNNs too shallow solely to reduce complexity.
  • Accuracy: Depthwise convolutions better capture motion, enabling 3D-MobileNetV2 to achieve the best Jester performance despite weaker Kinetics-600 and UCF-101 results.Its inverted residual block expands channels before depthwise convolution, providing more depthwise filters than the other architectures.
  • Complexity level: 5-9% worse Kinetics-600 performance occurs for 3D-MobileNetV2 0.2x and 3D-ShuffleNetV2 0.25x than for 3D-ShuffleNetV1 0.5x and 3D-MobileNetV1 0.5x.Capacity degrades severely below a width multiplier of 0.5, with the same pattern observed across all three benchmarks.
  • Complexity level: 3D-SqueezeNet minimizes parameters at its highest complexity level but uses around 300 million more FLOPs than other architectures.Its design prioritizes parameter savings rather than computational savings and omits depthwise convolutions.
  • Runtime performance: Similar FLOPs do not guarantee similar speed because memory access cost and parallelism affect runtime beyond FLOPs.3D-SqueezeNet has the highest FLOPs yet the highest runtime performance, while GPU and Jetson TX2 rankings can differ between ShuffleNet variants.
  • State-of-the-art comparison: 3D-ShuffleNetV1 achieves performance similar to ResNet-18 while requiring 7 times fewer parameters and 15 times fewer FLOPs.The comparison indicates that network design can substantially affect the accuracy-efficiency trade-off.

5. Conclusion

The paper addresses the lack of resource-efficient 3D CNN architectures by implementing 3D versions of established efficient 2D networks and evaluating them broadly. The resulting models provide considerable classification performance, flexible complexity scaling, and near-real-time execution on tested platforms.

  • Motivation: The work targets the shortage of resource-efficient 3D CNN architectures for action recognition under runtime, memory, and power constraints.Prior resource-efficient architecture development had focused on 2D convolutional kernels.
  • Approach: The authors implement 3D SqueezeNet, MobileNet, MobileNetV2, ShuffleNet, and ShuffleNetV2 at four complexity levels.The architectures are evaluated on benchmarks selected for capacity, motion capture, and transfer-learning analysis.
  • Findings: The architectures deliver considerable classification performance, and width multipliers allow their capacity to be modified flexibly.The conclusion summarizes results across the four analyzed complexity levels.
  • Findings: Depthwise convolutions perform well for capturing motion patterns on the Jester benchmark.This conclusion is consistent with the reported advantage of 3D-MobileNetV2 on motion-focused data.
  • Findings: Nearly all models run in real time on both Titan XP and Jetson TX2, and transfer learning is applicable through pretrained models.The paper states that these properties support use in other real-world applications.
Loading 1904.02422v5…