PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Interlacing Network

Hao Shao, Shengju Qian, Yu Liu

arXiv:2001.06499v1cs.CV

TL;DR

Video understanding must model temporal dynamics alongside spatial representations, while existing pipelines can consume substantial computation. TIN interlaces spatial representations across time; it is reported to gain 4% more accuracy with 6x less latency and to achieve state-of-the-art performance.

Abstract

from arXiv · show

For a long time, the vision community tries to learn the spatio-temporal representation by combining convolutional neural network together with various temporal models, such as the families of Markov chain, optical flow, RNN and temporal convolution. However, these pipelines consume enormous computing resources due to the alternately learning process for spatial and temporal information. One natural question is whether we can embed the temporal information into the spatial one so the information in the two domains can be jointly learned once-only. In this work, we answer this question by presenting a simple yet powerful operator -- temporal interlacing network (TIN). Instead of learning the temporal features, TIN fuses the two kinds of information by interlacing spatial representations from the past to the future, and vice versa. A differentiable interlacing target can be learned to control the interlacing process. In this way, a heavy temporal model is replaced by a simple interlacing operator. We theoretically prove that with a learnable interlacing target, TIN performs equivalently to the regularized temporal convolution network (r-TCN), but gains 4% more accuracy with 6x less latency on 6 challenging benchmarks. These results push the state-of-the-art performances of video understanding by a considerable margin. Not surprising, the ensemble model of the proposed TIN won the $1^{st}$ place in the ICCV19 - Multi Moments in Time challenge. Code is made available to facilitate further research at https://github.com/deepcs233/TIN

Introduction

Video understanding must model temporal dynamics alongside spatial representations, but existing approaches face substantial complexity or computational overhead. TIN interlaces spatial representations across time and is reported to improve accuracy while reducing latency.

Introduction

  • Video behavior spans 2D image space and 3D space-time, increasing recognition complexity and making temporal dynamics difficult to embed in spatial representations..Source: Introduction · p. 1
  • The introduction contrasts framewise hand-crafted features and temporal feature-extraction networks with TIN's interlacing of spatial representations across time.The figure depicts these as three video-understanding pipelines; the text notes that existing deep temporal models can require many parameters and FLOPs.Source: Figure 1 · p. 1, Introduction · p. 1
  • TIN splits channels into groups, learns neighboring-frame offsets and weights, shifts and interpolates features, then concatenates and aggregates them.The grouped offsets act like convolution kernels of different sizes and support long-range temporal relationships and varied sampling rates.Source: Introduction · p. 2
  • TIN is reported to gain 4% more accuracy with 6x less latency and to achieve state-of-the-art performance.The introduction also states that TIN is theoretically equivalent to the regularized temporal convolution network (r-TCN).Source: Introduction · p. 2

Related Work

Related work ranges from hand-crafted cues and 2D CNNs to temporal modeling and self-attention. The discussion highlights limitations in temporal modeling that motivate adaptive shifting.

Related Work

  • Improved Dense Trajectories (iDT) is widely exploited in video understanding, providing an efficient video representation; the cited passage does not specify which models’ performance it boosts.Source: Related Work · p. 2
  • TSN fuses segment class scores only after frame processing, performing well on K600, UCF101, and HMDB51 but poorly on temporally dependent datasets such as Something-Something and Jester.Basic 2D CNNs can confuse actions distinguished by opposite motion directions.Source: Related Work · p. 2
  • TSM shifts feature maps by a fixed displacement, whereas the proposed changeable shift adapts to specific datasets and extracted-frame distributions.The changeable shift is inspired by deformable convolution.Source: Related Work · p. 2
  • Self-attention models channel interdependencies for image classification (Hu, Shen, and Sun 2018) and long-range dependencies in vision tasks (Wang et al. 2018).Source: Related Work · p. 2

Temporal Interlacing Network (TIN)

TIN embeds temporal information into spatial feature representations using deformable shifts and differentiable temporal-wise frame sampling. Its grouped offsets provide dynamic temporal receptive fields, and its formulation is equivalent to a constrained temporal convolution.

Intuition

  • Distinct channel groups receive different temporal offsets, enabling dynamic receptive fields while reducing model complexity relative to separate channel offsets.The paper motivates grouping as a way to separate temporal resolutions and stabilize training across heavy backbones.Source: Intuition · p. 3

Deformable Shift Module

  • TIN generates group-specific offsets and weights through separate paths, using pooled temporal-channel features to produce the sampling parameters.OffsetNet aggregates channel information with a 1D convolution and temporal information with fully connected layers; WeightNet uses a 1D convolution and sigmoid.Source: Deformable Shift Module · p. 3

Differentiable Temporal-wise Frame Sampling

  • Fractional offsets linearly interpolate neighboring temporal samples, and shifted and unshifted channels are concatenated into a feature map with the input size.A temporal buffer handles shifts into the intervals just outside the input range; shifts beyond those intervals are clamped to zero.Source: Differentiable Temporal-wise Frame… · p. 4
  • Four shifted channel groups use symmetric offsets, while 3/4 of channels remain unshifted and 1/4 are shifted.The reported best configuration uses two offsets and their reversals across the four groups.Source: Differentiable Temporal-wise Frame… · p. 4

Temporal Interlacing Network (TIN)

  • TIN interlaces information in both temporal directions using reverse offsets, a strategy the paper says accelerates convergence and elevates performance.The module is inserted before each convolutional layer in residual blocks and preserves the input feature-map size.Source: Deformable Shift Module · p. 3, Analysis · p. 4

Analysis

  • TIN converts to a constrained convolution kernel of size 2, with different offsets across groups and shared offsets within each group.Different grouping strategies yield different numbers of equivalent kernels.Source: Analysis · pp. 4–5

Experiments

Experiments evaluate TIN on six video-recognition datasets, compare it with 2D baselines and state-of-the-art methods, and examine learned offsets, weights, and design ablations. The reported results cover both temporal-sensitive and temporal-insensitive datasets.

Setup and Implementation Details

  • Experiments use six video-recognition datasets, with ResNet-50 and 8- or 16-frame inputs for fair comparisons; Something and Jester emphasize temporal relationships.The setup also uses spatial jittering, horizontal flipping, dropout, weight decay, and pretrained models with frozen batch normalization.Source: Setup and Implementation Details · p. 5

Comparison with 2D CNN baseline and TSM

  • On Something-Something V1, TIN performs 26.6% better than 2D TSN and 1.5% better than TSM; on Kinetics-600, it performs 0.9% better than TSM and 1.7% better than 2D baseline.TIN also outperforms the 2D baseline and TSM on the other reported temporal-sensitive and temporal-insensitive datasets.Source: Comparison with 2D CNN baseline and TSM · p. 5

Comparison with State-of-the-arts

  • On Something-Something V1, TIN improves 2.4% over TSM, while its 16-frame model improves 2.6% over Non-local I3D with 5× fewer FLOPs.The {8, 16}-frame ensemble improves 2.8% over TSM with almost the same FLOPs and parameters; on V2, TIN reaches state-of-the-art performance, with a reported 0.7% gap to the previous state of the art.Source: Comparison with State-of-the-arts · pp. 6–7

Visualization of learned offsets and weights

  • Learned offsets are nearly zero in the first seven layers, then increase; reducing input from 16 to 8 frames also increases the learned offsets.The authors interpret this pattern as evidence that temporal fusion at higher levels can bring more improvement.Source: Visualization of learned offsets and… · p. 7
  • TIN assigns lower weights to the first and last frames, which lose partial information when shifted out.Its weight module adjusts those endpoint weights to reduce their impact on network training.Source: Visualization of learned offsets and… · p. 7

Ablation Experiments

  • Two shifted channel groups give the best ablation performance; increasing or decreasing the group count lowers performance.The authors attribute too few groups to insufficient temporal-stage information and too many to excess context and more brittle training.Source: Ablation Experiments · p. 7
  • Two groups plus two reverse groups improve performance by 0.3% over four groups without reverse offsets.The reverse offsets are intended to provide symmetric temporal interlacing.Source: Ablation Experiments · p. 7

Conclusion

TIN is presented as a temporal-modeling operator whose learned shifting module jointly interlaces spatial and temporal information. The conclusion describes its learned features as accurate and efficient, and Figure 5 visualizes learned offsets and temporal weights.

Conclusion

  • TIN is introduced as a simple operator for temporal modeling.The conclusion names it the temporal interlacing network.Source: Conclusion · p. 7
  • Figure 5 shows learned offsets and temporal weights for the 8- and 16-frame models.Its offset plots show cumulative offsets across layers of different depths.Source: Figure 5 · p. 8
  • The learnable shifting module interlaces temporal and spatial information jointly with little computation overhead.The conclusion says TIN provides learned features with accurate and efficient representations for video classification.Source: Conclusion · p. 8

Related papers

Loading 2001.06499v1…