PaperlayerView exact revision on arXiv

Source-linked AI summary

Two-Stream RNN/CNN for Action Recognition in 3D Videos

Rui Zhao, Haider Ali, Patrick van der Smagt

arXiv:1703.09783v2cs.CVcs.LG

TL;DR

Methods for processing 3D video action data remain an area of research, despite applications such as health monitoring and assisted living. This paper combines skeleton-based recurrent networks and RGB-video 3D-CNNs, merging their features in an SVM. The resulting system improves on the best published result in the literature by 14%.

  • Problem

    Methods for processing 3D video for action recognition remain under research despite applications including health monitoring, assisted living, and surveillance.

    Source: §I · p. 1

  • Method

    The system combines a skeleton-based RNN with an RGB-video 3D-CNN and uses a linear SVM to fuse their features for classification.

    Source: §I · p. 1, C. Experimental Results and Analysis · p. 5

  • Results

    14% higher than the best published result in the literature, the two-stream RNN/CNN method also improves 13% over the proposed RNN alone.

    Source: §IV · p. 6

  • Takeaways & Limitations

    The RNN and CNN features are highly complementary: the RNN learns long-term motion patterns, while the CNN captures video information about objects.

    Source: C. Experimental Results and Analysis · p. 5

Abstract

from arXiv · show

The recognition of actions from video sequences has many applications in health monitoring, assisted living, surveillance, and smart homes. Despite advances in sensing, in particular related to 3D video, the methodologies to process the data are still subject to research. We demonstrate superior results by a system which combines recurrent neural networks with convolutional neural networks in a voting approach. The gated-recurrent-unit-based neural networks are particularly well-suited to distinguish actions based on long-term information from optical tracking data; the 3D-CNNs focus more on detailed, recent information from video data. The resulting features are merged in an SVM which then classifies the movement. In this architecture, our method improves recognition rates of state-of-the-art methods by 14% on standard data sets.

I. INTRODUCTION

The paper develops deep-learning methods for 3D action recognition, combining skeleton-based RNNs with RGB-video 3D-CNNs. Its proposed RNN emphasizes easier training, while feature fusion combines temporal and spatial information and outperforms the cited state-of-the-art method.

I. INTRODUCTION

  • The paper situates 3D action recognition in applications including health monitoring, assisted living, and robot perception and cognition.Compared with 2D video, 3D video provides more spatial information and could be more informative.Source: §I · p. 1
  • The paper evaluates its methods on the NTU RGB+D Dataset, using its 3D skeletal data and RGB videos.The dataset contains more than 56k action samples across 60 action classes and four modalities.Source: §I · p. 1
  • The proposed skeleton-based RNN trains with 13 times fewer iterations and uses 20% less computational time per iteration than a normal LSTM-based RNN.Its contribution focuses on making the network easier to train, less inclined to overfitting, and deep enough to represent the data.Source: §I · p. 1
  • For RGB video, the method uses a 3D-CNN and compares and fuses its results with those of the proposed RNN models.The 3D-CNN is applied to RGB videos from the NTU RGB+D dataset.Source: §I · p. 1
  • The proposed two-stream feature fusion combines RNN temporal features with CNN spatial features and outperforms the current state-of-the-art method [4] by more than 14% on cross-subject and cross-view settings.The paper also proposes decision fusion, which selects based on classifier confidence; feature fusion has superior performance.Source: §I · p. 1

II. METHODOLOGY

  • The methodology section introduces a deep bidirectional gated recurrent neural network with batch normalization and dropout, alongside the 3D-CNN and two fusion architectures.The fusion architectures are decision fusion and feature fusion.Source: §II · p. 2

1) Vanilla Recurrent Neural Network:

A vanilla RNN processes sequences by updating a hidden state from successive inputs. It can be stacked, but it does not retain information over long spans because of the vanishing-gradient problem.

1) Vanilla Recurrent Neural Network:

2) Long Short-Term Memory:

LSTM uses gated cells to retain information over long sequences, while GRU simplifies the gating structure and combines the cell and hidden states into one state.

2) Long Short-Term Memory:

  • LSTM stores information in gated cells, allowing errors to be backpropagated through hundreds or thousands of time steps.Source: 2) Long Short-Term Memory · p. 2
  • LSTM's input, forget, and output gates control information flow, and candidate values may be added to the cell state.Element-wise multiplication is used in the described operations.Source: 2) Long Short-Term Memory · p. 2

3) Gated Recurrent Unit:

  • GRU has a simpler structure than LSTM and can be computed faster, combining LSTM's three gates into update and reset gates.GRU also combines the cell state and hidden state into a single state.Source: 3) Gated Recurrent Unit · p. 2
  • The GRU equations include a new candidate state, denoted by ˆh_t.Source: 3) Gated Recurrent Unit · p. 2

4) Bidirectional Recurrent Neural Network:

A bidirectional RNN processes input in both temporal directions, and its output can feed a softmax classifier. Batch normalization standardizes activations using mini-batch statistics, with learned scale and shift parameters.

4) Bidirectional Recurrent Neural Network:

5) Batch Normalization:

  • Batch normalization approximates whitening by standardizing layer activations using estimated mini-batch means and standard deviations.The paper motivates it as a way to reduce internal covariate shift, which slows deep-network training.Source: 5) Batch Normalization · p. 2
  • Batch normalization uses learned scale and shift parameters that are fixed during inference.The parameters γ and β are defined for activations x, and ε is added for numerical stability.Source: 5) Batch Normalization · p. 2

6) Proposed RNN Structure:

The proposed skeleton RNN uses bidirectional GRU layers with batch normalization and dropout, while the RGB stream uses a 3D-CNN. The section also describes the CNN’s video-volume processing and pretrained-model setup.

6) Proposed RNN Structure:

  • The skeleton RNN uses two bidirectional layers of 300 GRU cells, followed by batch normalization, dropout, a 600-unit ReLU layer, and a softmax over 60 action classes.Training randomly drops 25% of neurons between batch normalization and the fully connected layer.Source: 6) Proposed RNN Structure · pp. 2–3
  • The RNN comparisons progress from one-layer models to a two-layer bidirectional GRU-BN-DP model with an additional fully connected layer before softmax.The paper names the final variant “2 layer BI-GRU-BN-DP-H.”Source: 6) Proposed RNN Structure · p. 3

B. Convolutional Neural Network

  • The paper selects a 3D-CNN for RGB video, motivated by its promising performance on 2D action-recognition tasks and the suitability of 3D convolution for video features.The 3D-convolution operation is illustrated in Fig. 2.Source: B. Convolutional Neural Network · p. 3, Figure 2 · p. 3
  • 3D convolution processes frame clips across space and time into feature volumes, and stacked convolution and max-pooling layers build video features.The 3D-CNN uses eight convolutional layers, five max-pooling layers, two fully connected layers, and a softmax output.Source: B. Convolutional Neural Network · p. 3, Figure 3 · p. 3
  • The selected 3D-CNN is pretrained on Sports-1M and fine-tuned on the current dataset to reduce overfitting and training time.The model is based on the 3D-CNN described in.Source: B. Convolutional Neural Network · p. 3

C. Two-stream RNN/CNN

The paper proposes decision and feature fusion to combine its RNN and CNN models. Decision fusion selects between their predictions using validation-tuned confidence weights.

C. Two-stream RNN/CNN

  • The paper proposes decision fusion and feature fusion as two ways to combine the RNN and CNN models.Decision fusion uses classifier confidence, while feature fusion combines model features.Source: C. Two-stream RNN/CNN · p. 3

1) Decision Fusion:

  • With two classifiers, decision fusion cannot use majority voting, so it selects the prediction with the greater weighted confidence.The weights are assigned to the RNN and CNN classifiers.Source: 1) Decision Fusion · p. 3
  • The validation set is used to tune the RNN and CNN trust weights, initialized equally at 1.00, by comparing each classifier’s highest softmax probability.Only two trust-weight parameters require tuning; the fused prediction is the RNN or CNN prediction, according to the weighted-confidence comparison.Source: 1) Decision Fusion · p. 3

2) Feature Fusion:

Feature fusion extracts features from the trained skeleton RNN and RGB-video CNN, concatenates and L2-normalizes them, then classifies them with a linear SVM.

2) Feature Fusion:

  • The RNN extracts temporal features from 3D skeleton data, while the CNN extracts spatiotemporal features from RGB video.Both streams use features from their first fully connected layer.Source: 2) Feature Fusion · pp. 3–4
  • The two streams’ features are concatenated, L2-normalized, and passed to a linear SVM for action classification.Fig. 4 depicts the two-stream feature-fusion structure.Source: Figure 4 · p. 4, 2) Feature Fusion · p. 4
  • The SVM parameter C is selected using validation data, and the resulting model is evaluated on the test set.The SVM receives the combined RNN and CNN features.Source: 2) Feature Fusion · p. 4

III. EXPERIMENTS

The experiments evaluate the proposed models on NTU RGB+D using skeleton and RGB data under standard cross-subject and cross-view criteria. The section also specifies the training splits and RNN implementation settings.

A. NTU RGB+D Dataset [2]

  • NTU RGB+D contains over 56,000 videos across 60 action classes, and the experiments use its skeleton and RGB modalities.The dataset has 4 million frames, collected by three Kinect V2 cameras from 40 subjects.Source: A. NTU RGB+D Dataset [2] · p. 4
  • The standard evaluations are cross-subject, with subjects split between training and testing, and cross-view, with one viewpoint held out for evaluation.The dataset’s large intra-class and viewpoint variation makes it challenging, while its sample volume suits data-driven methods.Source: A. NTU RGB+D Dataset [2] · p. 4

B. Implementation details

  • The implementation uses a 10% validation subset of training subjects; RNN inputs contain two skeletons padded to 300 time steps, totaling 150 coordinates per step.RNN training uses RMSprop with learning rate 0.001 and dropout keep probability 75%; batch sizes are 1,000 for one-layer and 650 for two-layer models.Source: B. Implementation details · p. 4

1) RNN Implementation:

The paper evaluates a 3D-CNN on RGB video clips and combines its predictions or features with those of an RNN. Feature fusion gives the best testing results.

2) CNN Implementation:

  • RGB videos are split into non-overlapping 16-frame clips, resized to 3×16×128×171 pixels, and jittered by random cropping during training.The CNN is pretrained on Sport1M and fine-tuned on NTU RGB+D.Source: 2) CNN Implementation · p. 4
  • Video predictions are averaged across clips, and the corresponding fc-6 features are averaged into one 4,096-dimensional vector per video..Source: 2) CNN Implementation · pp. 4–5

3) Fusion Implementation:

  • Decision fusion combines classifier softmax outputs using validation-selected weights: RNN/CNN weights are 1.00/2.88 for cross-subject and 1.00/3.02 for cross-view.Source: 3) Fusion Implementation · p. 5
  • Feature fusion gives the best testing results, combining 600-dimensional RNN features with 4,096-dimensional CNN features for a linear SVM.The concatenated 4,696-dimensional feature array is L2-normalized; C = 8.0 gives the best validation accuracy in both setups.Source: 3) Fusion Implementation · p. 5

C. Experimental Results and Analysis

On NTU RGB+D, feature fusion improves accuracy over both the best RNN and literature methods. The proposed RNN also outperforms other methods on both evaluation protocols, while the CNN’s temporal context is limited by GPU memory.

D. Discussion

The two-stream RNN/CNN reports gains over the proposed RNN and the best published result, while the discussion identifies confusing action pairs and limits in the input streams. The authors propose exploring depth maps and IR sequences in future work.

D. Discussion

  • Reading vs. writing, putting on vs. taking off a shoe, and rubbing hands vs. clapping are relatively confusing action pairs.Source: D. Discussion · p. 6, Figure 7 · p. 6
  • Possible confusion sources include inherent action similarity, noisy skeleton data, short CNN clips, and the RNN’s lack of surrounding-object information.The CNN is trained on 16-time-step clips and focuses on short-term features; RGB frames are more stable but lack 3D information. The RNN lacks surrounding-object appearance and movement cues that could help classification.Source: D. Discussion · p. 6

IV. CONCLUSION AND FUTURE WORK

  • 13% higher than using the proposed RNN alone and 14% higher than the best published result, the two-stream method reports the paper’s final gains.Source: §IV · p. 6
  • Future work will consider depth maps and IR sequences and seek an effective architecture for fusing these modalities.Source: §IV · p. 6

Related papers

Loading 1703.09783v2…