PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvNet Architecture Search for Spatiotemporal Feature Learning

Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, Manohar Paluri

arXiv:1708.05038v1cs.CV

TL;DR

Video representation learning needs architectures that capture motion beyond appearance, but video architecture search is hindered by computational cost, benchmark overfitting, and many sensitive design choices. The paper performs a capacity-controlled architecture search and develops Res3D, a deep 3D Residual ConvNet. Res3D outperforms C3D across five benchmarks while being faster, smaller, and more compact.

  • Problem

    Video frames require dedicated spatiotemporal representations because appearance-based image models cannot capture motion patterns, while video architecture search is costly and sensitive to many design choices.

  • Method

    The paper conducts a capacity-controlled ConvNet architecture search on UCF101 and uses its observations to design Res3D for training on Sports-1M.

  • Results

    Res3D outperforms C3D across Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN, while being 2 times faster, 2 times smaller, and more compact.

  • Takeaways & Limitations

    Careful architecture search remains valuable despite imperfect video benchmarks and machine capacity, and 3D convolutions are more suitable for spatiotemporal feature learning than 2D convolutions.

  • Takeaways & Limitations

    The architecture-search findings are empirical results from the small UCF101 benchmark, and replicating the full search at larger scale is prohibitive.

Abstract

from arXiv · show

Learning image representations with ConvNets by pre-training on ImageNet has proven useful across many visual understanding tasks including object detection, semantic segmentation, and image captioning. Although any image representation can be applied to video frames, a dedicated spatiotemporal representation is still vital in order to incorporate motion patterns that cannot be captured by appearance based models alone. This paper presents an empirical ConvNet architecture search for spatiotemporal feature learning, culminating in a deep 3-dimensional (3D) Residual ConvNet. Our proposed architecture outperforms C3D by a good margin on Sports-1M, UCF101, HMDB51, THUMOS14, and ASLAN while being 2 times faster at inference time, 2 times smaller in model size, and having a more compact representation.

1. Introduction

Video architecture progress is slowed by high computational costs, benchmark overfitting, and sensitivity to design choices. The paper searches architectures under controlled capacity constraints and develops Res3D, which outperforms C3D across multiple benchmarks while reducing computational cost and model size.

  • Motivation: Video ConvNets face higher computation and memory costs than image models, making large-scale architecture search difficult.Training a 3D ConvNet reportedly takes 3–4 days on UCF101 and about two months on Sports-1M.
  • Motivation: UCF101 models can overfit because its frames are highly correlated and its setting is tightly controlled.ConvNets trained from scratch achieve 41–44%, whereas Sports1M pretraining can raise UCF101 accuracy to 82%.
  • Approach: The search constrains networks to similar parameter counts so performance differences can be attributed more confidently to architecture rather than capacity.The resulting observations guide an efficient deep 3D Residual ConvNet, Res3D, trained on Sports-1M.
  • Approach: The paper conducts a ConvNet architecture search across multiple design dimensions on UCF101, measuring sensitivity to each dimension.The search varies input processing, convolution types, depth, and temporal modeling choices.
  • Results: Res3D achieves the best reported performance across Sports-1M, UCF101, and HMDB51 under the stated evaluation settings, with competitive results on THUMOS14 and ASLAN.The comparison is against methods without long-term modeling on Sports-1M and RGB-only methods on UCF101 and HMDB51.
  • Results: Res3D is 2 times faster and 2 times smaller than current deep video features, with a more compact representation.The paper specifically compares Res3D against C3D on speed, model size, and representation compactness.

2. Related Work

Related work spans hand-crafted spatiotemporal representations and deep ConvNet approaches for video understanding. This paper builds primarily on C3D and ResNet while broadening architecture search beyond temporal kernel length and controlling model capacity.

  • Hand-crafted representations: Hand-crafted video representations include STIPs, SIFT-3D, HOG3D, Cuboids, ActionBank, and improved Dense Trajectories.These methods use feature encoding schemes such as histograms or pyramids.
  • Image ConvNets: Image ConvNet progress includes VGG, GoogleNet, and ResNet, with residual connections enabling deeper networks with less overfitting.These architectures established influential design patterns for visual representation learning.
  • Deep video models: Deep video methods include 3D ConvNets, 3D generative models, fusion methods, and two-stream networks for action recognition.Later work enhanced two-stream networks with ResNet architectures and additional stream connections.
  • Positioning: The paper uses 3D ConvNets like C3D but searches more architecture dimensions while constraining model capacity and adopts ResNet-style architectures.C3D’s prior search was limited to the temporal kernel length.

3. Architecture Search

The paper searches video ConvNet architectures under matched parameter capacity, using UCF101 for empirical comparisons before evaluating a selected 3D residual architecture more broadly. The search identifies practical choices for input sampling, resolution, convolution type, and depth.

  • Search setup: The search uses UCF101 because training deep networks is computationally expensive, then trains the final architecture on Sports-1M.The authors use deep residual networks as the search space and later assess learned spatiotemporal features across video-understanding tasks.
  • Search setup: Matched-capacity comparisons reduce the extent to which performance differences can be attributed to network size rather than architecture.The search networks use approximately 33M parameters, while the authors caution that the findings remain empirical results from a small benchmark.
  • Baseline architecture: 10.3 billion FLOPs reduces the 3D-Resnet18 complexity from 19.3 billion FLOPs while maintaining UCF101 accuracy within 0.96% random chance.The reduced-input model is called SR18 and serves as the baseline for subsequent architecture-search experiments.
  • Input sampling: Sampling one frame every 2-4 frames with clip lengths between 0.25s and 0.75s yields good video-classification accuracy.Temporal strides beyond 4 perform significantly worse, while stride 1 is somewhat less accurate because the clips are too short and lack context.
  • Input resolution: An input resolution of 128, using a 112 × 112 crop, balances computational complexity and video-classification accuracy under GPU memory constraints.At resolution 64, SR18 accuracy drops 9.5%; resolution 128 gives higher accuracy and less computation than 256.
  • Convolution and depth: Fully 3D convolutions improve video-classification performance over the tested convolution mixtures when compared at similar parameter counts.The search also finds that 18 layers provide a good accuracy, computational-complexity, and memory trade-off; D18 is comparable in accuracy to D16 while being less complex than D16 and less memory-intensive than D26.

4. Spatiotemporal feature learning with 3D Resnets

The paper applies architecture-search observations to build Res3D, a deep 3D residual network trained on Sports-1M for spatiotemporal feature learning. Res3D improves over C3D while reducing model size and computation.

  • Architecture and training: Res3D applies architecture-search observations to spatiotemporal feature learning on the large-scale Sports-1M dataset.The network is selected as a 3D-ResNet-18 architecture and compared with C3D features.
  • Architecture and training: Res3D is trained on Sports-1M using five randomly extracted 2-second clips per training video and an 8 × 112 × 112 input.Training uses spatiotemporal jittering and SGD; a 2D-ResNet-18 baseline follows the same procedure.
  • Sports-1M results: 2.7%, 4.5%, and 2.6%: Res3D exceeds C3D on Sports-1M top-1 clip, top-1 video, and top-5 video accuracy, respectively.The comparison uses a single center-cropped clip for clip accuracy and ten center-cropped clips for video accuracy.
  • Efficiency and representation: All Res3D 3D filters change across time, indicating that they encode spatiotemporal information rather than only spatial patterns.The corresponding 2D filters mainly have similar appearance patterns within the 3D filters.
  • Efficiency and representation: 33.2 million parameters and 19.3 billion FLOPs make Res3D about 2 times smaller and 2 times faster than C3D.C3D has 72.9 million parameters and 38.5 billion FLOPs.
  • Cross-benchmark results: Res3D outperforms C3D by 3.5% on UCF101 and 3.3% on HMDB51.These results are reported for action recognition using the learned spatiotemporal representation.

5. Res3D as Spatiotemporal Features

The paper evaluates Sports-1M-pretrained Res3D features across action recognition, action similarity labeling, and temporal localization. Res3D generally outperforms C3D and 2D representations, including at substantially lower feature dimensions.

  • Action recognition: 3.5% and 3.3%: Res3D outperforms C3D on UCF101 and HMDB51 action recognition, respectively.It achieves the best accuracy among methods using only RGB input in the reported comparison.
  • Action recognition: 84.5% and 49.8%: the RGB-only I3D results on UCF101 and HMDB51 are 1.3% and 5.1% below Res3D, respectively, though the results use only split 1.The paper notes that the comparisons are not directly comparable because of the evaluation protocol.
  • Feature compactness: Res3D outperforms C3D by 4 to 5% after PCA projection to low dimensions, indicating a more compact feature representation.The comparison is made on UCF101 using low-dimensional projected features.
  • Feature compactness: 56.7 versus 52.8: at 10 dimensions, Res3D outperforms C3D by 3.9%; at 300 dimensions, Res3D reaches 82.9% and exceeds C3D using 4,096 dimensions.Features are classified with a linear SVM after PCA projection.
  • Action similarity labeling: Res3D gives a small improvement over C3D and large improvements over 2D baselines for ASLAN action similarity labeling.The reported table identifies Res3D as achieving the best ASLAN accuracy.
  • Action localization: The THUMOS’14 experiment replaces C3D with Res3D in the S-CNN action-localization framework to test the effect of the feature representation.The benchmark evaluates temporal localization mAP across IoU thresholds from 0.3 to 0.7.
  • Action localization: Some concurrent THUMOS’14 methods achieve better performance while using C3D as their base network.The authors suggest that replacing C3D with Res3D could provide a similar gain in those systems.

6. Discussion

The discussion presents Res3D as a broadly effective and efficient spatiotemporal representation, while noting that architecture search remains constrained by benchmark and hardware limitations.

  • Discussion: Res3D outperforms C3D across Sports-1M, UCF101, HMDB51, ASLAN, and THUMOS14.The paper also reports that Res3D is more compact than C3D on low-dimensional UCF101 features.
  • Discussion: 2 times faster in run-time and 2 times smaller in model size: Res3D improves efficiency relative to C3D while retaining stronger benchmark performance.The paper gives low-dimensional UCF101 performance as an example of its more compact representation.
  • Discussion: The authors report that careful experimental settings can make architecture search valuable despite imperfect video benchmarks and limited machine capacity.The stated constraints include GPU memory and current video-benchmark limitations.
  • Discussion: The paper provides empirical evidence that 3D convolutions are more suitable than 2D convolutions for spatiotemporal feature learning.It reports improvements across action recognition, action similarity labeling, and action detection applications.

Appendix A: Architechture Details

The appendix provides additional architecture-search details for mixed convolutions and networks of varying depth.

  • Architecture details: Tables 13 and 14 detail architectures used in the search with mixed convolutions and varying network depth.

Appendix B: Filter Visualizations

Res3D’s 3D filters capture both motion patterns and appearance information, whereas 2D filters detect appearance alone. Most 2D filters correspond to temporal slices of 3D filters.

  • 3D filters capture different motion patterns alongside appearance information.The visualization contrasts these filters with 2D filters, which detect appearance information only.
  • Most 2D filters can be matched to a temporal slice of a 3D filter.

Appendix C: Architecture Search Validation on HMDB51

Experiments on HMDB51 validate architecture-search patterns observed on UCF101 under adjusted training and evaluation settings. Mixed convolutions generally underperform 3D, depth rankings remain consistent, and small-benchmark searches retain relevance when carefully designed.

  • Experimental settings: HMDB51 experiments reuse the UCF101 architectures while shortening the training schedule because HMDB51 is about 2 times smaller.The learning rate remains 0.01, is divided by 10 every 10K iterations, and training stops at 45K iterations.
  • Experimental settings: Higher sampling rates are excluded because HMDB51’s short videos would remove about 20%, 45%, and 82% of examples at 32, 64, and 128 frames.
  • Architecture comparisons: Mixed-convolution architectures consistently perform worse than 3D on HMDB51 and UCF101, while C2.5D is comparable to 3D for SR18.
  • Depth comparison: Relative architecture rankings are consistent across HMDB51 and UCF101, supporting depth 18 as a good trade-off for 3D ResNets.
  • Search validation: Consistent FLOPs-versus-accuracy distributions across both datasets further support architecture search on small benchmarks when carefully designed.
Loading 1708.05038v1…