PaperlayerView exact revision on arXiv

Source-linked AI summary

Motion Feature Network: Fixed Motion Filter for Action Recognition

Myunggi Lee, Seungeui Lee, Sungjoon Son, Gyutae Park, Nojun Kwak

arXiv:1807.10037v2cs.CV

TL;DR

Action recognition needs temporal cues, but optical-flow methods and 3D convolutions carry computational costs. MFNet uses motion blocks with RGB frames in a unified network, and reports comparable state-of-the-art test results on Jester and Something-Something.

  • Problem

    Action recognition needs temporal as well as spatial cues, while optical flow is costly to generate and 3D convolution methods demand substantial computation and memory.

    Source: §1 · pp. 1–2

  • Method

    MFNet uses motion blocks to encode spatio-temporal relationships between adjacent RGB frames in a unified network that can be trained end to end.

    Source: §1 · p. 2, Approximation of Optical Flow · p. 6, §3.3 · p. 7

  • Results

    MFNet reports comparable state-of-the-art results, with 96.22% and 37.48% top-1 accuracies on the Jester and Something-Something test sets, respectively.

    Source: Comparisons with I3D · p. 12

  • Takeaways & Limitations

    The motion block can be attached to existing CNN-based action-recognition networks with small additional cost.

    Source: §1 · p. 2, §5 · p. 15

    The stated goal is to derive temporal features from optical flow for action classification, rather than to find the optical flow itself.

    Source: Approximation of Optical Flow · p. 5

Abstract

from arXiv · show

Spatio-temporal representations in frame sequences play an important role in the task of action recognition. Previously, a method of using optical flow as a temporal information in combination with a set of RGB images that contain spatial information has shown great performance enhancement in the action recognition tasks. However, it has an expensive computational cost and requires two-stream (RGB and optical flow) framework. In this paper, we propose MFNet (Motion Feature Network) containing motion blocks which make it possible to encode spatio-temporal information between adjacent frames in a unified network that can be trained end-to-end. The motion block can be attached to any existing CNN-based action recognition frameworks with only a small additional cost. We evaluated our network on two of the action recognition datasets (Jester and Something-Something) and achieved competitive performances for both datasets by training the networks from scratch.

1 Introduction

MFNet uses motion blocks to represent spatial and temporal information from RGB frames in a unified network. The paper reports that it outperforms previous methods on datasets where action labels depend on sequential relationships among frames.

1 Introduction

  • Prior two-stream methods use optical flow, while 3D convolutions incur substantial computational and memory costs.Optical flow also takes substantial time to generate.Source: §1 · p. 1
  • Jester and Something-Something emphasize detailed physical actions whose labels depend on temporal order, unlike UCF101 examples recognizable from a single image.The figure contrasts temporally ambiguous action pairs in the former datasets with UCF101 examples recognizable from spatial context.Source: §1 · p. 2, Figure 1 · p. 2
  • MFNet combines appearance and motion blocks to represent spatial and temporal information from RGB frames in a unified model.It avoids optical-flow precomputation and, as a 2D CNN architecture, has fewer parameters than 3D counterparts.Source: §1 · p. 2
  • MFNet trained using only RGB frames significantly outperforms previous methods on publicly available datasets with sequentially related video labels.The reported results concern action recognition on Jester and Something-Something.Source: §1 · p. 3

2 Related Works

Related work extracts spatio-temporal features using 3D convolutions or two-stream RGB and optical-flow inputs. MFNet is positioned as a unified, end-to-end alternative to approaches that separate spatial and temporal features.

2 Related Works

  • Action recognition in video uses both spatial appearance and temporal information because consecutive frames convey sequential context.CNN-based visual features have been increasingly applied to video action-recognition datasets.Source: §2 · p. 3
  • 3D convolution methods [29] encode spatio-temporal information but require substantial computation and parameters, making training difficult on small datasets.Kinetics [14] facilitated training 3D models, and Carreira et al. proposed inflating 3D filters from 2D filters.Source: §2 · p. 3
  • Two-stream methods combine RGB with externally computed optical flow, and TSN samples both modalities across video segments but still relies on costly flow.TSN trains the RGB and optical-flow streams independently and accumulates their predictions at inference.Source: §2 · pp. 3–4
  • Other approaches feed flow-like inputs or use optical flow as training supervision [20] [38], while Sun et al. [26] separately extract temporal and spatial features.MFNet instead feeds spatial and temporal features forward in a unified network that can be trained end to end.Source: §2 · p. 4

3 Model

MFNet combines appearance blocks with fixed motion filters to encode spatial and temporal information, then propagates both through motion blocks. Its filters shift and difference adjacent-frame features, while motion blocks aggregate them with appearance features.

3 Model

  • The model section introduces MFNet’s architecture and describes motion filters and motion blocks as its core components.Source: §3 · p. 4

3.1 Motion Feature Network

  • Appearance blocks encode spatial information, while motion blocks use consecutive feature maps from the same hierarchy to extract temporal information and pass both types onward.The architecture is based on TSN’s sequence of K snippets, with ResNet used for appearance blocks in the experiments.Source: §3.1 · p. 4

3.2 Motion Representation

  • Fixed displacement directions define residual features by subtracting a feature map at time t from a shifted map at t + ∆t, approximating feature-level motion rather than solving optical flow.The implementation restricts ∆x and ∆y to 0 or ±1, with higher CNN hierarchies able to capture larger movements through larger receptive fields.Source: Approximation of Optical Flow · pp. 5–6
  • Concatenated directional residuals encode spatio-temporal information and are intended to represent directional relationships between feature maps.The motion filter is aimed at encoding directional information via learnable parameters, unlike the cited subtractive correlation layer, which is used for correspondence finding.Source: Figure 3 · p. 6, Motion Filter · p. 7

3.3 Motion Block

  • Motion blocks propagate spatio-temporal information between adjacent appearance blocks by aggregating motion features with appearance features.Source: §3.3 · p. 7
  • Element-wise sum compresses concatenated directional motion features with 1×1 filters before adding them to appearance features; concatenation instead combines motion and appearance features before 1×1 encoding.The sum scheme matches the appearance-feature dimension before addition, while the concatenation scheme reduces channel dimension after combining the features.Source: Element-wise Sum · pp. 7–8, Concatenation · p. 8

4 Experiments

Experiments evaluate MFNet on Jester and Something-Something using a TSN-based setup, then examine segment count, backbone depth, comparisons with other methods, and model behavior. MFNet achieves competitive leaderboard results using RGB inputs and training from scratch.

4.1 Experiment Setup

4.2 Experimental Results

  • 96.1% vs. 90.4% on Jester and 37.3% vs. 17.4% on Something-Something: MFNet-C with 7 training segments outperforms the same model with 3.MFNets also improve significantly from 3 to 5 segments, unlike the TSN baseline; increasing K reduces sampled-frame intervals and allows richer information extraction.Source: Table 1 · p. 10, §4.2 · p. 11, Evaluation on The Number of Segments · p. 11
  • MFNet-C with ResNet-18 achieves performance comparable to a 101-layer ResNet using 76% fewer parameters (11.68M vs. 50.23M).The paper attributes the limited effect of network depth on Jester to its videos' similar human appearances and weak relation between static entities and action classes.Source: Table 2 · p. 11, Comparisons of Network Depths · pp. 11–12
  • MFNet-C achieves competitive performance to I3D on Jester and Something-Something; validation compares 10 sparsely sampled MFNet snippets with I3D predictions using all frames.I3D was trained from scratch for the Something-Something comparison, with 16- and 32-frame snippets.Source: Comparisons with I3D · p. 12, Table 3 · p. 12
  • 96.22% and 37.48% top-1 accuracy on Jester and Something-Something test sets, respectively, make MFNet competitive with leaderboard state of the art using RGB inputs alone.The reported models use ResNet-101 with K = 10, trained from scratch, without other modalities, ensembles, or large-scale pretraining.Source: Comparisons with I3D · p. 12, Table 5 · p. 13

4.3 Analysis on The Behavior of MFNet

5 Conclusions

MFNet combines appearance and motion blocks, using a fixed motion filter to represent spatio-temporal information. The authors report outperforming results on Jester and Something-Something after end-to-end training from scratch.

5 Conclusions

  • The motion filter shifts predefined directional filters and subtracts the resulting feature maps from those of the preceding frame to produce motion features.Source: §5 · p. 14
  • MFNet combines appearance and motion blocks and can be attached to existing CNN-based networks with small additional cost; the authors also report comprehensive ablation studies.Source: §5 · pp. 14–15
  • Figure 6 plots MFNet-C50 validation accuracy against inference-time segment count for models trained with different segment counts.Source: Figure 6 · p. 15
  • MFNet achieves outperforming results on Jester and Something-Something by training end-to-end from scratch; the authors plan to validate it on a large-scale dataset.Source: §5 · p. 15
Loading 1807.10037v2…