Efficient Two-Stream Motion and Appearance 3D CNNs for Video Classification

1 Introduction

• Optical-flow networks discriminate action classes better, but require time-consuming flow computation; C3D handles temporal information without flow but has lower accuracy than two-stream methods.The introduction attributes the remaining temporal-learning difficulty partly to limited video training data.Source: §1 · p. 1
• The proposed two-stream 3D network uses 3D convolutions in spatial and temporal streams, combining abstract features from optical-flow estimation with appearance features.The motion feature comes from the last convolutional layer of the flow-estimation network.Source: §1 · p. 1
• The paper frames its goal as combining two-stream temporal modeling with C3D's speed and spatial-data handling.Figure 1 depicts the initial idea of combining mid-level motion features with regular C3D features.Source: §1 · p. 1, Figure 1 · p. 1

2 Related works

• Two-stream methods use optical flow for temporal data; later work explored deeper networks, action detection, and fusion at multiple layers.Feichtenhofer et al. reported state-of-the-art results without combining with IDT [10].Source: §2 · p. 2
• Donahue et al. applied an LSTM to extracted frame features, but the approach was not end-to-end.The passage says its results did not improve as much as expected.Source: §2 · p. 2
• Improved trajectories can improve CNN-based results when combined with convolutional features, and Wang et al. encoded local trajectory features with Fisher vectors.The reviewed work also extracted CNN features around trajectories before encoding them.Source: §2 · p. 2
• The paper characterizes optical flow and IDT as time-consuming, motivating a raw-image network that handles temporal information without those inputs.The passage contrasts those approaches with a network trained on ImageNet.Source: §2 · p. 2

3 Learning Appearance and Motion CNNs

• The proposed two-stream 3D-CNNs learn spatio-temporal video representations from RGB frames, embedding learned optical-flow information without requiring optical flow as input.The paper presents mid-level motion features as a lower-computational-cost replacement for optical flow in video classification.Source: §3 · p. 2

3.1 Initial Method

• The initial method estimates optical flow with a 3D Conv-Deconv network, then combines its mid-level motion features with RGB 3D-Convnet features for classification.The paper describes this approach as faster than methods that take optical flow as input, with comparable results, but says it still needs improvement.Source: §3.1 · p. 2

3.2 Combined Network

• The combined network shares a 3D-Convnet between action classification and flow estimation, using a softmax loss and a voxel-wise loss, respectively.The method is described as optimizing appearance and motion together and as fast as C3D at test time.Source: §3.2 · pp. 2–3

3.3 3D Conv-Deconv Two-Stream Net

• The two-stream network concatenates the final convolutional features of an RGB appearance stream and a motion-estimation stream before applying the softmax loss.The motion stream uses a 3D-Convnet followed by a 3D-Deconvnet; the paper reports accuracy similar to single-frame two-stream nets at 20 times their frames-per-second rate.Source: §3.3 · p. 3

3.4 The Networks Architecture

• The 3D-Convnet has five convolutional layers and two fully connected layers plus a class layer, with convolution filters sized 3×3×3.The convolutional layer filter counts are 64, 128, 256, 256, and 256; the fully connected layers have 2048 units.Source: §3.4 · p. 3

4 Experiments

• The evaluation uses UCF101; the authors say they will try other datasets in future.Source: §4 · p. 3

4.1 Dataset and Experiments

• Evaluation uses UCF101's 13,320 clips and averages accuracy across its three train, validation, and test sets.Source: §4.1 · p. 3

4.2 Results

• The unshared two-stream network outperforms the earlier proposed variants in mean accuracy.Source: Table 1 · p. 4, §4.2 · p. 4
• About 3%: the three proposed networks improved mean accuracy, while shared-weight end-to-end training added a reported 2% improvement.Source: §4.2 · p. 4
• Separately trained appearance/action and motion/flow streams outperform the single-network C3D appearance baseline.Source: §4.2 · p. 4
• Without separate optical-flow extraction, the proposed methods show a considerable speed difference from other methods; shared weights are fastest among the three.Source: §4.2 · p. 4, Table 2 · p. 4

• The proposed convolutional networks embed both appearance and motion in human-action videos.Source: §5 · p. 4
• The two-stream 3D network is presented as an efficient scheme for applying 3D-ConvNets to video classification.Source: §5 · p. 4
• The authors report accurate results at speeds much faster than real time.Source: §5 · p. 4