PaperlayerView exact revision on arXiv

Source-linked AI summary

3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks

Keze Wang, Xiaolong Wang, Liang Lin, Meng Wang, Wangmeng Zuo

arXiv:1501.06262v3cs.CV

TL;DR

Complex RGB-D activity recognition must account for temporal variation in how activities decompose into actions. This paper uses a reconfigurable deep model to learn those compositions from raw video, and reports better performance than state-of-the-art approaches in challenging scenarios.

  • Problem

    Complex activities vary in their temporal compositions, with action durations differing across subjects.

    Source: §1 · p. 1

  • Method

    A deep structured model learns action compositions from raw segmented RGB-D videos and partially adjusts its network configuration during inference.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Across challenging scenarios the model outperforms state-of-the-art methods; on Figure 7 it achieves 81.2% average accuracy versus 59.7% [25], 80.6% [9], 62.3%, and 63.1%.

    Source: §1 · p. 2, §6.2 · p. 8

  • Takeaways & Limitations

    The work presents a new RGB-D database containing 1,180 human activities from 20 categories.

    Source: §1 · p. 2

    The number of sub-actions is a tuning choice: small values may limit expressiveness, while large values may overfit due to model complexity.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Human activity understanding with 3D/depth sensors has received increasing attention in multimedia processing and interactions. This work targets on developing a novel deep model for automatic activity recognition from RGB-D videos. We represent each human activity as an ensemble of cubic-like video segments, and learn to discover the temporal structures for a category of activities, i.e. how the activities to be decomposed in terms of classification. Our model can be regarded as a structured deep architecture, as it extends the convolutional neural networks (CNNs) by incorporating structure alternatives. Specifically, we build the network consisting of 3D convolutions and max-pooling operators over the video segments, and introduce the latent variables in each convolutional layer manipulating the activation of neurons. Our model thus advances existing approaches in two aspects: (i) it acts directly on the raw inputs (grayscale-depth data) to conduct recognition instead of relying on hand-crafted features, and (ii) the model structure can be dynamically adjusted accounting for the temporal variations of human activities, i.e. the network configuration is allowed to be partially activated during inference. For model training, we propose an EM-type optimization method that iteratively (i) discovers the latent structure by determining the decomposed actions for each training example, and (ii) learns the network parameters by using the back-propagation algorithm. Our approach is validated in challenging scenarios, and outperforms state-of-the-art methods. A large human activity database of RGB-D videos is presented in addition.

1. INTRODUCTION

The introduction frames complex RGB-D activity recognition around noisy appearance information and activities whose action sequences vary across subjects. It proposes a deep structured model that learns temporal compositions from raw segmented videos and can adjust its network configuration.

  • Activities can have large temporal variation because their action decompositions and action durations differ across subjects.Fixed-length temporal segments may struggle with complex activities composed of actions of diverse durations.Source: §1 · pp. 1–2
  • The deep structured model represents an activity as separated actions associated with cubic-like video segments of unfixed length and learns their temporal structure for classification.Source: §1 · p. 2
  • The network acts directly on raw grayscale-depth video segments, using 3D convolutions for spatial and temporal features before higher-level 2D convolutions and activity classification.Segment-level convolutional results are merged into two fully connected layers.Source: §1 · p. 2
  • Latent variables allow partial network activation during inference, including turning neurons off to adapt to different temporal durations of separated actions.Source: §1 · p. 2
  • Latent Structural Back Propagation alternates between discovering each training example’s action decomposition and learning network parameters with back-propagation.Source: §1 · p. 2
  • Because RGB-D data collection is relatively expensive, the authors pre-train on common RGB video data and transfer the learned parameters as model initializations.Source: §1 · p. 2
  • The model reports superior performance over state-of-the-art approaches in several challenging scenarios and introduces an RGB-D database of 1,180 activities across 20 categories.Source: §1 · p. 2

2. RELATED WORK

Prior work on 3D activity recognition includes feature-engineering methods, compositional models, reconfigurable models, and deep networks. The paper positions its model as a deep-network extension that can reconfigure during learning and inference.

  • Hand-crafted feature methods for 3D activities often overlooked detailed spatiotemporal structure and were limited to periodic activities.Source: §2 · p. 2
  • Compositional approaches model complex activities with parts and context, while other methods discover temporal structure by localizing sequential actions.Examples include latent structural SVMs, actionlets, temporal warping, variable-duration HMMs, and CRFs.Source: §2 · pp. 2–3
  • Reconfigurable And-Or graph models discover alternative compositions during learning and inference, but the cited approaches were built on hand-crafted features.Source: §2 · p. 3
  • The deep structured model extends existing deep architectures by making the network reconfigurable during learning and inference.The related-work discussion also describes deep networks extracting features from raw data and video architectures using spatial and temporal dimensions.Source: §2 · p. 3

3. STRUCTURED DEEP MODEL

The model processes activity videos as action-specific segments with a spatio-temporal CNN, then uses latent structure to adapt each clique to variable segment lengths. Its cliques combine convolutional features for activity classification, while partial neuron inactivation handles differing temporal compositions.

3.1 Spatio-temporal CNNs

  • Each of M cliques extracts features from one decomposed video segment, and their outputs are concatenated for activity classification.Each clique applies two 3D convolutional layers and a 2D convolutional layer to grayscale and depth input; two fully connected layers map the merged features to activity labels.Source: §3.1 · p. 3
  • 3D kernels span spatial and temporal dimensions to capture appearance and motion across adjacent frames.After each 3D convolution, max-pooling subsamples feature maps; 2D convolution then extracts higher-level features.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 Reconfigurable Latent Structure

  • Latent variables set each clique’s starting frame and input length, allowing video segments to vary rather than being evenly divided.When a clique receives fewer than m frames, some neurons are inactivated; the sub-actions need not correspond directly to semantic actions.Source: §3.2 · p. 5
  • The number of cliques M trades temporal expressiveness against overfitting from model complexity.The authors report using M = 4 on the databases and suggest estimating M through parameter tuning such as cross-validation.Source: §3.2 · p. 5

4. LEARNING

The learning procedure alternates between estimating video-segment latent variables and updating network parameters. It also pre-trains convolutional layers on labeled 2D activity videos before optimizing on RGB-D data.

4.1 Latent Structural Back Propagation

  • LSBP alternates between selecting latent video segments and updating network parameters by back-propagation until the training cost converges.With fixed parameters, it chooses latent variables that maximize each sample’s correct-class probability; with segments fixed, it uses stochastic gradient descent to update parameters.Source: §4.1 · pp. 5–6
  • Training minimizes a logistic-regression cost over labeled videos, model parameters, and latent variables, with a regularization term.The objective’s first two terms are the opposite of the log-likelihood.Source: §4.1 · p. 5

4.2 Model Pre-training

  • To address insufficient RGB-D training data, convolutional parameters are initialized from supervised training on labeled 2D activity videos.The 2D data are evenly segmented by clique and trained with back-propagation; convolutional layers transfer to RGB-D, while fully connected layers are randomly initialized.Source: §4.2 · p. 6

5. INFERENCE

Inference jointly selects an activity category and its latent temporal configuration by maximizing the video score. The search respects minimum segment lengths and non-overlap constraints, and can be parallelized across configurations.

5. INFERENCE

  • Inference searches activity labels and latent configurations, selecting the label-configuration pair with the highest video score.The configuration specifies segment starts and durations; candidate segments must satisfy minimum-length and non-overlap constraints.Source: §5 · p. 6
  • The search enumerates configurations with segment lengths from τ to m and no overlap between adjacent segments.The experiments set τ = 5; GPU parallelism accelerates independent forward propagations for different configurations.Source: §5 · p. 6

6. EXPERIMENT

The experiments evaluate the model on CAD-120 and the newly collected OA dataset, which covers single- and two-person office activities. The model reports higher average accuracy than competing methods, while analyses examine pre-training and reconfigurable structure.

6. EXPERIMENT

  • CAD-120 contains 120 sequences across 10 categories, while OA contains 1,180 sequences split into single-subject OA1 and two-subject OA2 activities.CAD-120 uses four-fold cross-validation by subject; OA uses five-fold cross-validation.Source: §6 · p. 6

6.1 Implementation

  • Videos were normalized to 120 frames, with 30 anchor frames used as input; each of four model segments accepts at most nine frames.Frames were scaled to 80 × 60; one inference took around 0.4 seconds, while one training fold took 2 hours on CAD-120 and 5 hours on each OA subset.Source: §6.1 · p. 7

6.2 Results and Comparisons

  • 81.2% average accuracy on CAD-120 exceeded four competing methods; OA1 and OA2 averaged 60.1% and 45.0%, respectively.The OA comparison reports the highest average accuracy on OA1 and better results in six OA2 classes than the two competing methods.Source: Figure 7 · p. 7, §6.2 · p. 8, Table 2 · p. 8
  • Depth data boosted average accuracy, especially on OA1, where grayscale appearance varied more across tested subjects.The authors attribute the observed depth benefit to its smaller appearance variance and its help in capturing motion information.Source: §6.2 · p. 8, Table 4 · p. 8

6.3 Empirical Analysis

  • On OA1, pre-training reduced test error by 8% and reached convergence after 25 iterations rather than 140 without pre-training.The comparison removes parameters trained on 2D videos for the model without pre-training.Source: §6.3 · p. 8
  • On OA1, the structured model’s test error decreased faster and reached a lower result than the non-structured model from the same initialization.The non-structured variant fixes latent variables and evenly segments the input frames.Source: §6.3 · p. 8

7. CONCLUSIONS

The conclusion presents model reconfigurability in layered CNNs as a way to recognize 3D activities from raw RGB-D data. It identifies semantic information and cloud deployment as future directions.

7. CONCLUSIONS

  • The proposed model incorporates reconfigurability into layered CNNs and recognizes activities directly from raw RGB-D data rather than hand-crafted features.Future work proposes adding high-level semantics for events with underlying intentions and deploying the model on cloud platforms for thin-client access.Source: §7 · p. 9

Related papers

Loading 1501.06262v3…