PaperlayerView exact revision on arXiv

Source-linked AI summary

More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation

Quanfu Fan, Chun-Fu Chen, Hilde Kuehne, Marco Pistoia, David Cox

arXiv:1912.00869v1cs.CV

TL;DR

Video action recognition models often rely on costly 3D convolutions, while 2D models have limited temporal expressiveness. The paper combines a deep low-resolution branch, a compact high-resolution branch and temporal aggregation; it reports competitive results across three benchmarks and state-of-the-art performance on Something-Something.

  • Problem

    3D ConvNets are costly to train, while resource-friendlier 2D models have limited temporal expressiveness and cannot fully benefit from richer frame input.

    Source: §1 · p. 1, §3.3 · p. 4

  • Method

    The approach combines a deep low-resolution subnet with a compact high-resolution subnet and adds TAM, which exchanges temporal information through weighted channel-wise aggregation.

    Source: §1 · p. 2

  • Results

    The models achieve competitive results on Kinetics-400, Something-Something and Moments-in-time, with state-of-the-art results on Something-Something; bLResNet-101 reaches 53.1% top-1 accuracy on SS-V1 at 32×2 frames.

    Source: §1 · p. 2, §4.2 · p. 6, §5 · p. 9

  • Takeaways & Limitations

    The architecture allows twice as many input frames as the baseline while using less computation and memory, supporting deeper models with richer input under the same GPU resources.

    Source: §5 · p. 9

    The approach is described with 2D convolutions; extending it to architectures based on 3D convolutions is presented only as a possibility.

    Source: §3.2 · p. 3

Abstract

from arXiv · show

Current state-of-the-art models for video action recognition are mostly based on expensive 3D ConvNets. This results in a need for large GPU clusters to train and evaluate such architectures. To address this problem, we present a lightweight and memory-friendly architecture for action recognition that performs on par with or better than current architectures by using only a fraction of resources. The proposed architecture is based on a combination of a deep subnet operating on low-resolution frames with a compact subnet operating on high-resolution frames, allowing for high efficiency and accuracy at the same time. We demonstrate that our approach achieves a reduction by $3\sim4$ times in FLOPs and $\sim2$ times in memory usage compared to the baseline. This enables training deeper models with more input frames under the same computational budget. To further obviate the need for large-scale 3D convolutions, a temporal aggregation module is proposed to model temporal dependencies in a video at very small additional computational costs. Our models achieve strong performance on several action recognition benchmarks including Kinetics, Something-Something and Moments-in-time. The code and models are available at https://github.com/IBM/bLVNet-TAM.

1 Introduction

The introduction motivates resource-efficient video recognition that can use longer frame sequences, then presents bLVNet's dual-resolution branches and TAM for temporal modeling. It reports competitive results across three benchmarks and new state-of-the-art results on Something-Something and Moments-in-time.

  • Longer input sequences generally improve recognition but increase GPU memory and time needs; 2D models are cheaper than 3D models yet lack temporal expressiveness.Source: §1 · p. 1
  • bLVNet pairs a very deep subnet processing low-resolution frames with a compact subnet processing high-resolution frames, merging the branches at each network layer.Source: §1 · p. 2
  • TAM models temporal relations through learnable weighted channel-wise aggregation across frames, using 1×1 depthwise convolution with negligible additional computation and parameters.Source: §1 · p. 2
  • The paper's two contributions are a lightweight dual-path video architecture and temporal aggregation that avoids computationally expensive 3D convolutions.Source: §1 · p. 2
  • bLVNet-TAM achieves competitive results on Kinetics-400, Something-Something and Moments-in-time, setting new state-of-the-art results on the latter two.Source: §1 · p. 2

2 Related Work

Related work spans frame-stack models using 3D convolutions, methods that fuse temporal segments or shift channels across frames, and approaches with dual pathways or temporal convolutions.

  • Frame-stack video models commonly use 3D convolutions, including I3D [1], S3D [15] and ResNet3D [2], which lead to large parameter spaces.Source: §2 · p. 2
  • TSN [5] and TRN [3] fuse video segments at higher layers, while TSM [4] shifts channels across time; TAM instead uses depthwise 1×1 convolutions for temporal dependencies.Source: §2 · p. 2
  • SlowFast [17] uses slow and fast pathways, whereas bLVNet also reduces computation through low image resolution; Timeception [18] uses 10% of model parameters versus TAM's 0.1%.Source: §2 · pp. 2–3

3 Our Approach

The approach adapts the Big-Little dual-path network for video, then adds temporal aggregation to capture dependencies beyond neighboring frames. It combines frame processing at different resolutions with local and global temporal fusion.

3.1 Recap of Big-Little Network

  • bLNet combines parallel high-complexity low-resolution and low-cost high-resolution subnets, merging their branches layerwise to fuse complementary image information.The paper adopts bLResNet-50 and bLResNet-101 as backbones.Source: §3.1 · p. 3

3.2 Big-Little Video Network as Video Representation

  • bLVNet sends odd and even frames to separate branches at different resolutions, using local fusion to capture temporal relations between adjacent frames.It processes twice as many frames as TSN without increasing computation and has 1.5–2.0× fewer FLOPs than TSN.Source: Figure 1 · p. 3, §3.2 · p. 4

3.3 Temporal Aggregation Module

  • TAM aggregates weighted, channel-wise activations from frames within a temporal range, using depthwise convolutions, temporal shifts, and summation.Inserted after bLVNet's local fusion, it adds context that can grow with network depth, potentially capturing long-range temporal dependencies.Source: §3.3 · p. 5, Figure 2 · p. 5
  • TAM is independent of spatial convolutions and uses depthwise convolution, with fewer parameters and less computation than S3D and R(2+1)D.It can also be integrated into 3D convolutional models such as C3D and I3D.Source: §3.3 · p. 5

4 Experiments

The experiments evaluate bLVNet-TAM on Something-Something, Kinetics-400, and Moments-in-time, reporting strong benchmark results and studies of frame count, memory use, and temporal aggregation. The experiments also describe dataset settings and training procedures.

4.1 Experimental Setup

  • The experiments cover Something-Something versions 1 and 2, Kinetics-400, and Moments-in-time, with dataset scales and video characteristics differing across benchmarks.Something-Something emphasizes human-object interactions and temporal relationships; Moments clips are three seconds long.Source: §4.1 · p. 6
  • Training uses progressive finetuning from an ImageNet-finetuned base model, while inference uses single-crop, single-clip evaluation on Something-Something and Moments.Kinetics-400 uses multi-crop and multi-clip evaluation; models were trained on an eight-GPU server, with batch size adjusted to available memory.Source: §4.1 · p. 6

4.2 Main Results

  • On Something-Something, bLResNet-50 outperforms 3D models on SS-V1 with 8×2 frames, while bLResNet-101 reaches 53.1% top-1 accuracy on SS-V1 and 65.2% on SS-V2.The authors report these 32×2-frame results as state of the art; test-set results are also consistently better than compared approaches, including two-stream models.Source: §4.2 · pp. 6–7, Table 1 · p. 7, Table 2 · p. 7
  • On Kinetics-400, bLVNet-TAM mainly performs on par with or better than current large-scale architectures, without the clearer gains reported on Something-Something.The authors relate the smaller separation to temporal relations being less essential for understanding Kinetics videos.Source: §4.2 · p. 7, Table 3 · p. 7
  • On Moments-in-time, the approach outperforms the single-stream and ensemble models reported in the Moments paper.The authors report only a slight top-1 accuracy improvement for 16×2 over 8×2 frames and speculate that three-second clips limit the impact of finer temporal granularity.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Ablation Studies

  • TAM outperforms TSM by over 2% under both TSN and bLVNet, and adds another 13% on top of either architecture in the reported comparison.The bLVNet baseline with local fusion improves accuracy from 17.4% to 33.6% over TSN, and TAM is reported as complementary to local fusion.Source: §4.3 · p. 8
  • As input frames increase, bLVNet-TAM improves on Something-Something while TSN stays nearly flat; bLVNet-TAM also uses about half the memory at equal frame counts.The authors report a similar frame-count trend on Kinetics-400 and say the memory saving permits larger training batches under the same computational budget.Source: §4.3 · pp. 8–9, Figure 3 · p. 8

5 Conclusion

The architecture allows twice as many input frames as the baseline while using less computation and memory, and combines this efficiency with temporal aggregation. The models achieve strong benchmark performance, including a state-of-the-art result on Something-Something.

5 Conclusion

  • Twice as many input frames as the baseline, with less computation and memory, support deeper models and richer input under the same GPU resources.The paper also reports strong performance across several action-recognition benchmarks and a state-of-the-art result on Something-Something.Source: §5 · p. 9
  • Temporal aggregation captures dependencies across frames without relying on computationally expensive 3D convolutions.Source: §5 · p. 9
  • The models achieve strong performance on several action-recognition benchmarks and establish a state-of-the-art result on Something-Something.Source: §5 · p. 9

A Network Architecture

The bLVNet-TAM-50 configuration places TAM before branching into Big-Net and Little-Net, with temporal aggregation on the residual block’s non-shortcut path. The configuration uses distinct branch operations and specifies convolution stride and channel-alignment details.

A Network Architecture

  • TAM is placed before branching into Big-Net and Little-Net, and the last shared residual block; in ResBlockTAM, it runs on the non-shortcut path.Source: A Network Architecture · p. 11
  • The bLVNet-TAM-50 configuration uses α = 2 and β = 4, following the notation of bLNet [8].The network configuration is shown in Table 6.Source: A Network Architecture · p. 11, Table 6 · p. 11
  • ResBlockB uses a stride-2 3 × 3 convolution followed by bilinear upsampling, while ResBlockL ends with a 1 × 1 convolution for channel alignment.Source: A Network Architecture · p. 11
  • ResBlockTAM embeds temporal aggregation with r = 3; s2 denotes stride 2 for the 3 × 3 convolution in ResBlock.Source: A Network Architecture · p. 11

B Data Preprocessing

Preprocessing resizes images differently across Something-Something, Kinetics, and Moments, with dataset-specific shorter-side or square-image dimensions.

B Data Preprocessing

Related papers

Loading 1912.00869v1…