PaperlayerView exact revision on arXiv

Source-linked AI summary

View-invariant Deep Architecture for Human Action Recognition using late fusion

Chhavi Dhiman, Dinesh Kumar Vishwakarma

arXiv:1912.03632v1cs.CV

TL;DR

Human action recognition across unknown views is challenging. This paper combines motion Dynamic Images with view-invariant depth-pose shape dynamics in two late-fused streams. It reports superior results to existing methods, including higher accuracy than the nearest competitor on UWA3D-II and NUCLA for view 1.

  • Problem

    Human action recognition is challenging when viewpoint variation and similar poses across action classes make recognition difficult.

    Source: §I · p. 1

  • Method

    The framework combines RGB Dynamic Images processed with fine-tuned InceptionV3 and SSIM-selected depth-pose HPM features learned as temporal dynamics, then applies late fusion.

    Source: Self-Similarity Index Matrix (SSIM)… · p. 3, §V · p. 9

  • Results

    For view 1, average recognition accuracy is 91.3% on UWA3D-II and 83.6% on NUCLA, about 9% and 10.86% higher, respectively, than HPM_TM+DT.

    Source: D. NTU RGB-D Human Activity Dataset · p. 7

  • Takeaways & Limitations

    Across cross-view and cross-subject evaluations, late fusion produced notable results, with small accuracy variation across views reported as evidence of view invariance.

    Source: D. NTU RGB-D Human Activity Dataset · p. 7

    The authors report lower accuracy on NTU RGB-D than on the other datasets, attributing the challenge to sample variety and complexity.

    Source: D. NTU RGB-D Human Activity Dataset · p. 7

Abstract

from arXiv · show

Human action Recognition for unknown views is a challenging task. We propose a view-invariant deep human action recognition framework, which is a novel integration of two important action cues: motion and shape temporal dynamics (STD). The motion stream encapsulates the motion content of action as RGB Dynamic Images (RGB-DIs) which are processed by the fine-tuned InceptionV3 model. The STD stream learns long-term view-invariant shape dynamics of action using human pose model (HPM) based view-invariant features mined from structural similarity index matrix (SSIM) based key depth human pose frames. To predict the score of the test sample, three types of late fusion (maximum, average and product) techniques are applied on individual stream scores. To validate the performance of the proposed novel framework the experiments are performed using both cross subject and cross-view validation schemes on three publically available benchmarks- NUCLA multi-view dataset, UWA3D-II Activity dataset and NTU RGB-D Activity dataset. Our algorithm outperforms with existing state-of-the-arts significantly that is reported in terms of accuracy, receiver operating characteristic (ROC) curve and area under the curve (AUC).

I. INTRODUCTION

Viewpoint changes and similar poses across actions make human action recognition difficult. The paper proposes a two-stream framework combining motion and view-invariant shape dynamics, evaluated across three public datasets.

I. INTRODUCTION

  • The framework combines a motion stream of Dynamic Images with an STD stream of view-invariant HPM features learned over depth sequences.The authors identify motion and temporal information as important action cues, while noting that common motion representations are not tailored to viewpoint invariance.Source: §I · p. 1
  • The STD stream learns shape dynamics from key frames using Bi-LSTM and LSTM models rather than processing the entire video sequence.The authors state that this design enhances the framework's learning ability.Source: §I · p. 1
  • The motion stream represents action videos as compact Dynamic Images and projects them into a high-dimensional feature space using fine-tuned InceptionV3.Source: §I · p. 1
  • Three late-fusion strategies—maximum, average, and product—combine the streams; experiments use cross-subject and cross-view validation on NUCLA, UWA3D-II, and NTU RGB-D.The introduction reports superior recognition accuracy compared with similar state-of-the-art methods.Source: §I · p. 2

II. RELATED WORK

Related work describes efforts to capture motion compactly and model temporal dynamics, while identifying viewpoint sensitivity and missing shape information as limitations in prior approaches.

II. RELATED WORK

  • Multi-stream architectures combine temporal, motion, and spatial cues, while optical-flow extraction can be slow and temporal pooling methods capture dynamics over short intervals.Source: §II · p. 2
  • Short-window 3D filters capture local motion but miss longer-term patterns, while static motion-history representations lose information and can poorly compact complex dynamics.Source: §II · p. 2
  • Common-viewpoint frameworks can perform poorly on different viewpoints, and view-invariant methods have included geometric descriptors, virtual paths, and knowledge transfer.The cited approaches lack spatial or shape/appearance details; HPM features provide view coverage learned from synthetic poses across 180 viewpoints [34].Source: §II · p. 2

III. PROPOSED METHODOLOGY

The proposed framework processes motion and shape information in separate streams, then combines their scores through late fusion to predict an action.

III. PROPOSED METHODOLOGY

  • The motion stream encodes action motion as Dynamic Images and uses transfer learning with InceptionV3 to analyze RGB videos.Source: §III · p. 2
  • The STD stream extracts view-invariant HPM features describing action shape and learns their sequence with one Bi-LSTM and one LSTM layer.Source: §III · p. 2
  • The two independently learned streams are combined by late fusion to predict the action.Source: §III · p. 2

A. Depth-Human Pose Model (HPM) based action descriptor

The HPM-based STD stream represents view-invariant depth-pose shape over time. It selects key depth frames using SSIM and learns their HPM features with recurrent layers.

A. Depth-Human Pose Model (HPM) based action descriptor

  • HPM features encode depth-pose details independently of viewpoint, and an LSTM sequence model preserves action temporal structure.The HPM model was trained on synthetic multi-viewpoint action data from 180 viewpoints.Source: A. Depth-Human Pose Model (HPM) based… · p. 2
  • SSIM compares consecutive depth frames: high values indicate small pose changes, while low values indicate distinct poses.Morphological preprocessing first produces depth-human silhouettes to reduce background noise.Source: Self-Similarity Index Matrix (SSIM)… · p. 3
  • The method selects the first ten frames after sorting consecutive-frame SSIM values in ascending order, then resizes their salient regions to 227 × 227 for HPM feature extraction.It sets the luminance and contrast exponents to 0.5 and the structural-component exponent to 1 because depth images are not sensitive to luminance and contrast.Source: Self-Similarity Index Matrix (SSIM)… · p. 3
  • The STD architecture connects the HPM CNN's fc7 layer to Bi-LSTM and LSTM layers before fully connected and softmax classification.The final fully connected layer has 10, 30, or 60 outputs for NUCLA, UWA3D, or NTU RGB-D, respectively.Source: Self-Similarity Index Matrix (SSIM)… · p. 3

B. RGB-Dynamic Image (DI) based action descriptor

Dynamic images compress video appearance and temporal motion into an RGB representation, with approximate rank pooling providing an efficient weighted-sum construction. The resulting images are processed by the motion stream’s InceptionV3-based architecture.

B. RGB-Dynamic Image (DI) based action descriptor

  • Dynamic images emphasize salient objects and their motion while averaging away background pixels and preserving long-term action dynamics.Approximate rank pooling also emphasizes frame occurrence order when extracting long-term dynamics.Source: B. RGB-Dynamic Image (DI) based action… · p. 4
  • Rank pooling learns a vector that scores frames in temporal order, using a sequence-level optimization objective with a regularization term and hinge loss.The hinge loss penalizes incorrectly ranked frame pairs that fail to meet a unit margin.Source: B. RGB-Dynamic Image (DI) based action… · p. 4
  • Approximate rank pooling accelerates dynamic-image construction through simple linear operations at the pixel level.It approximates rank pooling using a gradient-based optimization step.Source: B. RGB-Dynamic Image (DI) based action… · p. 4
  • The approximate rank-pooling representation is a weighted sum of frame features, with weights computed for a fixed video length and accounting for frame order.The resulting RGB dynamic image sums the corresponding R, G, and B channels separately across video frames.Source: B. RGB-Dynamic Image (DI) based action… · pp. 4–5, Figure 4 · p. 5
  • The motion stream processes dynamic images using InceptionV3 convolutional features followed by global average pooling, batch normalization, dropout, and dense classification layers.The convolutional layers are trained end to end on multiview datasets.Source: Figure 5 · p. 5, B. RGB-Dynamic Image (DI) based action… · p. 6

IV. EXPERIMENTAL RESULTS

The framework is evaluated for view-invariant human action recognition using three publicly available datasets.

IV. EXPERIMENTAL RESULTS

  • The evaluation uses three public datasets to validate the proposed framework for view-invariant human action recognition.The passage names NUCLA multi-view, UWA3D Depth, and NTU RGB-D activity datasets.Source: §IV · p. 6
  • The selected benchmarks include the NUCLA multi-view action 3D dataset and the UWA3D Depth dataset.Both are part of the three-dataset evaluation set.Source: §IV · p. 6
  • The NTU RGB-D activity dataset is also included in the evaluation.The passage identifies it as the third benchmark used.Source: §IV · p. 6

B. NUCLA multi-view action 3D Dataset

The NUCLA multi-view dataset contains ten action categories recorded from three viewpoints, with cross-view and cross-subject validation settings. Similar walking patterns and viewpoint-sensitive action pairs make recognition challenging.

B. NUCLA multi-view action 3D Dataset

  • NUCLA contains ten action categories performed by ten subjects and captured simultaneously from three viewpoints using Kinect v1.For cross-view validation, two views train the model and the third is used for testing; cross-subject test samples may come from any viewpoint.Source: B. NUCLA multi-view action 3D Dataset · p. 6
  • Many actions share walking patterns before and after the main action, while one-hand and two-hand pick-up actions are difficult to distinguish across viewpoints.The described approach selects ten depth key frames using SSIM and processes them as 3D-HPM shape features.Source: B. NUCLA multi-view action 3D Dataset · p. 6
  • The dataset’s cross-view protocol trains on two viewpoints and tests on the third, while cross-subject test samples are selected irrespective of viewpoint.The dataset was captured simultaneously from three different viewpoints.Source: B. NUCLA multi-view action 3D Dataset · p. 6

C. UWA3D Multi view Activity-II Dataset

UWA3D Multi-view Activity-II includes 30 actions performed by ten subjects and recorded from four viewpoints at different times. Its challenges include unsynchronized action recordings, self-occlusion, and human-object interactions.

C. UWA3D Multi view Activity-II Dataset

D. NTU RGB-D Human Activity Dataset

The framework outperformed recent methods on NUCLA and UWA3D, while NTU RGB-D accuracy was lower and another method performed better there. Across the evaluated datasets, late fusion improved ROC/AUC results and recognition accuracy varied little across views.

D. NTU RGB-D Human Activity Dataset

  • NTU RGB-D contains 56,880 samples across 60 action types from 40 subjects, with cross-subject and cross-view protocols specified for evaluation.For cross-subject evaluation, 20 subjects train and 20 test; for cross-view evaluation, views 2 and 3 train and view 1 tests.Source: D. NTU RGB-D Human Activity Dataset · p. 6
  • Accuracy varied little across views under cross-view and cross-subject validation, while cited state-of-the-arts varied by about 10% across views [34].Source: D. NTU RGB-D Human Activity Dataset · p. 7
  • 91.3% on UWA3D II and 83.6% on NUCLA average recognition accuracy were about 9% and 10.86% above HPM_TM+DT, respectively, for view 1 testing.The reported NTU RGB-D accuracy was lower than on the other datasets, and [41] performed comparatively better using skeleton-joint features.Source: D. NTU RGB-D Human Activity Dataset · p. 7, Table VI · p. 8
  • Late fusion improved ROC/AUC performance over either individual stream across all three multi-view datasets and their individual test views.The ROC curves and AUC comparisons are shown in Fig. 7.Source: D. NTU RGB-D Human Activity Dataset · p. 7, Figure 7 · p. 8
  • The authors report lower computation cost because the motion stream represents a video with one dynamic image and the STD stream processes key pose depth frames rather than all frames.Experiments used a single 8GB NVIDIA GeForce GTX 1080 GPU system.Source: D. NTU RGB-D Human Activity Dataset · p. 8

V. CONCLUSION

The paper presents a two-stream RGB-D framework that combines motion and view-invariant shape dynamics for action recognition across viewpoints. It reports improved AUC for late fusion over either stream alone and consistent accuracy across views, while identifying skeleton details as future work for greater robustness to intra-class variation.

V. CONCLUSION

  • The framework processes RGB motion and depth-based shape temporal dynamics independently, then fuses their scores to predict test-sample labels.Motion uses RGB dynamic images with fine-tuned InceptionV3; the STD stream uses HPM-based key poses and Bi-LSTM/LSTM layers to learn long-term shape dynamics.Source: §V · p. 9
  • Late fusion improved AUC over the individual motion and STD streams, and recognition accuracy remained consistent across views.Source: §V · p. 9
  • Future work will add action skeleton details alongside depth details across viewpoints to improve robustness to intra-class variation in larger sample sets while maintaining time efficiency.Source: §V · p. 9
Loading 1912.03632v1…