PaperlayerView exact revision on arXiv

Source-linked AI summary

View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition

Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng

arXiv:1804.07453v3cs.CV

TL;DR

Viewpoint variation makes skeleton-based action recognition difficult, especially when test viewpoints differ from training data. The paper learns virtual viewpoint transformations jointly with RNN- and CNN-based classifiers. Across five challenging datasets, the framework consistently improves recognition performance and achieves state-of-the-art performance.

  • Problem

    Large viewpoint diversity makes skeleton representations differ substantially and can degrade recognition when testing viewpoints are absent from training data.

  • Method

    The paper learns virtual observation viewpoints within end-to-end VA-RNN and VA-CNN networks, transforming skeletons before classification.

  • Results

    The framework consistently improves recognition performance on five challenging benchmark datasets and achieves state-of-the-art performance.

  • Takeaways & Limitations

    The view-adaptive models transform skeletons from diverse viewpoints into consistent viewpoints that ease training and reduce viewpoint influence.

  • Takeaways & Limitations

    The input sequence is translated so the first frame's body center becomes the new global-coordinate origin.

Abstract

from arXiv · show

Skeleton-based human action recognition has recently attracted increasing attention thanks to the accessibility and the popularity of 3D skeleton data. One of the key challenges in skeleton-based action recognition lies in the large view variations when capturing data. In order to alleviate the effects of view variations, this paper introduces a novel view adaptation scheme, which automatically determines the virtual observation viewpoints in a learning based data driven manner. We design two view adaptive neural networks, i.e., VA-RNN based on RNN, and VA-CNN based on CNN. For each network, a novel view adaptation module learns and determines the most suitable observation viewpoints, and transforms the skeletons to those viewpoints for the end-to-end recognition with a main classification network. Ablation studies find that the proposed view adaptive models are capable of transforming the skeletons of various viewpoints to much more consistent virtual viewpoints which largely eliminates the viewpoint influence. In addition, we design a two-stream scheme (referred to as VA-fusion) that fuses the scores of the two networks to provide the fused prediction. Extensive experimental evaluations on five challenging benchmarks demonstrate that the effectiveness of the proposed view-adaptive networks and superior performance over state-of-the-art approaches. The source code is available at https://github.com/microsoft/View-Adaptive-Neural-Networks-for-Skeleton-based-Human-Action-Recognition.

1 INTRODUCTION

View variation makes skeleton-based action recognition difficult because the same posture can appear very differently across camera viewpoints. The paper addresses this by learning viewpoint transformations jointly with classification through view-adaptive RNN and CNN models.

  • Motivation: 3D skeletons are useful high-level representations, but viewpoint diversity causes large representation differences and can degrade recognition on unseen test viewpoints.The challenge also increases model size and overfitting risks when diverse viewpoints must be handled directly.
  • Motivation: Existing preprocessing strategies use human-defined coordinate transformations to reduce variation, but they can only partially alleviate the problem.These transformations include centering the body and aligning its orientation.
  • Method: The proposed view adaptation scheme automatically determines virtual observation viewpoints within the network and transforms skeletons before classification.The view adaptation and classification subnetworks are trained end to end to optimize recognition performance.
  • Method: VA-RNN integrates an RNN-based view adaptation module with an LSTM classifier, while VA-CNN integrates a CNN-based module with a CNN classifier.The view adaptation module selects suitable viewpoints during recognition in each stream.
  • Evaluation: Data augmentation through random skeleton rotations improves the robustness of the view adaptation models.The paper also analyzes the subnetworks, visualizes transformed skeletons, and discusses failure cases.
  • Evaluation: The two-stream VA-fusion scheme combines the scores of VA-RNN and VA-CNN, and experiments on five challenging datasets show consistent gains.The system regulates skeletons toward more consistent viewpoints while maintaining action continuity.

2 RELATED WORK

Prior skeleton-based methods commonly preprocess or map skeleton sequences without fully addressing viewpoint variation. This work instead uses RNNs and ConvNets to learn suitable virtual viewpoints for recognition.

  • Viewpoint handling: View-invariant preprocessing commonly centers skeletons and aligns body orientation, but frame-level processing can partially lose relative motion information.The related-work discussion identifies viewpoint influence as under-explored in skeleton-based recognition.
  • Viewpoint handling: The paper uses content-dependent view adaptation to learn suitable viewpoints and transform each sequence into representations under those views.This avoids relying on complex, human-defined criteria for non-rigid bodies.
  • RNN approaches: Earlier RNN-based methods mainly learn features and temporal dynamics from skeleton inputs, whereas this work also uses RNNs to determine virtual viewpoints.The distinction is the viewpoint adaptation role assigned to the recurrent network.
  • Proposed architecture: The proposed architecture combines a view adaptation subnetwork with a main classifier, using LSTM or ConvNet processing depending on the stream.The two streams can fuse their classification scores into VA-fusion.
  • CNN approaches: CNN-based methods often convert skeleton sequences into images or projection maps, but many ignore viewpoint variation.The proposed VA-CNN instead learns virtual viewpoints before generating skeleton maps for CNN classification.

3 VIEW ADAPTATION MODELING

View adaptation models re-observe skeleton sequences from learned virtual viewpoints before classification, addressing large viewpoint-induced representation differences. The formulation uses rigid, possibly time-varying transformations shared by all joints in each frame.

  • 3 VIEW ADAPTATION MODELING: Viewpoint variation can make skeleton representations of the same action substantially different, complicating skeleton-based action recognition.The variation arises from differing camera viewpoints and actors’ changing orientations.
  • 3 VIEW ADAPTATION MODELING: The network transforms input skeletons to new virtual viewpoints before the main classification network performs recognition.The view adaptation subnetwork determines transformation parameters, and the full network is trained end-to-end for classification.
  • 3.1 Problem Formulation: The formulation first translates the global coordinate system to the first-frame body center, making the input skeleton sequence insensitive to its initial position.The resulting skeleton input is expressed under this translated global coordinate system.
  • 3.1 Problem Formulation: Each frame’s virtual viewpoint is specified by translation d_t and rotations α_t, β_t, γ_t around the global X-, Y-, and Z-axes.The transformation-parameter set is T_t = {α_t, β_t, γ_t, d_t}.
  • 3.1 Problem Formulation: All joints in a frame share the same transformation parameters because viewpoint changes are modeled as rigid motion.The transformed joint set is represented under the new observation coordinate system, and viewpoints may vary across frames.

4 VIEW ADAPTIVE NEURAL NETWORKS

The paper presents VA-RNN and VA-CNN, which learn virtual observation viewpoints and transform skeleton data before end-to-end action classification. Their adaptation subnetworks use recurrent or convolutional models, while VA-fusion combines both streams.

  • VA-RNN: VA-RNN combines an RNN-based view adaptation subnetwork with a main LSTM network for action recognition from transformed skeletons.The adaptation subnetwork learns rotation and translation parameters, producing view-adapted skeleton representations for the classifier.
  • VA-RNN: The VA-RNN adaptation module uses LSTM and fully connected branches to estimate rotation parameters and the translation vector.Rotation parameters α_t, β_t, γ_t define the rotation matrix, while d_t defines translation.
  • VA-RNN: The main LSTM stacks three LSTM layers followed by a fully connected SoftMax classifier, and the entire network is trained end to end with cross-entropy loss.Gradients flow from the classification network back into the view adaptation subnetwork.
  • VA-CNN: VA-CNN maps skeleton sequences to image maps, learns sequence-level viewpoint parameters with convolutional layers, and classifies transformed maps using a ConvNet.Image-map columns represent frames, rows represent joints, and X, Y, and Z coordinates form the three channels.
  • VA-CNN: Sequence-level viewpoint parameters outperform frame-level parameters for the ConvNet design, despite frame-level parameters being more flexible in theory.The compared designs use 6 × T parameters for frame-level transformation and 6 parameters for sequence-level transformation.
  • Two Stream Fusion (VA-fusion): VA-fusion combines VA-RNN and VA-CNN scores with weights 1 and 4, respectively, while users may select an individual stream based on practical requirements.Learning the fusion weights did not outperform the simple weighted fusion approach.

5 EXPERIMENTAL RESULTS

The proposed view-adaptive frameworks are evaluated on five benchmark datasets, and the reported comparisons show consistent best performance across all five datasets.

  • Evaluation: The frameworks are evaluated on five benchmark datasets: NTU RGB+D, SYSU, UWA3D, Northwestern-UCLA, and SBU Kinect Interaction.The experiments include ablation studies, comparisons with view-invariant strategies, parameter analyses, visualizations, failure-case analysis, and state-of-the-art comparisons.
  • Performance Comparisons: The reported state-of-the-art comparisons show that the proposed scheme consistently achieves the best performance on all five datasets.The experimental section also presents comparative analyses of VA-RNN and VA-CNN.
  • Analysis: The experiments analyze the effectiveness of view adaptation through ablations, comparisons with view-invariant strategies, parameter studies, visualizations, and failure cases.These analyses are presented alongside benchmark performance comparisons.

5.1 Datasets and Experimental Settings

The study evaluates view adaptation across five skeleton-action benchmarks with differing sizes, viewpoints, subjects, and interaction settings. Training uses dataset-dependent batch sizes, optimization, dropout, gradient clipping, image resizing, and view-enriching rotations.

  • Datasets: NTU RGB+D contains 56880 samples, 60 action classes, 25 joints per subject, and diverse camera views and subject orientations.The dataset provides two standard evaluations, although their details are not included in the passage.
  • Datasets: SYSU contains 480 sequences of 12 actions performed by 40 subjects, with 20 joints per subject and high activity similarity.
  • Datasets: UWA3D contains 1075 sequences across 30 actions and four views, and is challenging because of viewpoint diversity, self-occlusion, and activity similarity.The four views are front, left side, right side, and top.
  • Datasets: Northwestern-UCLA contains 1494 videos of 10 actions performed by 10 subjects across three views, while SBU contains 282 two-person interaction sequences across eight classes.Northwestern-UCLA subjects have 20 joints, and SBU subjects have 15 joints.
  • Datasets: SYSU and SBU use a single primary camera viewpoint, but subjects vary in location, camera distance, and orientation.
  • Experimental Settings: Training uses dataset-dependent batch sizes and common optimization settings, with dropout for overfitting and gradient clipping for VA-RNN.VA-CNN uses Adam with an initial learning rate of 0.0001 and resizes skeleton maps to 224×224; some datasets receive random axis rotations during training.

5.2 Ablation Study

The ablation studies show that view adaptation improves recognition beyond human-defined preprocessing and ordinary increases in network capacity. Data augmentation strengthens view adaptation under viewpoint mismatch, while the learned transformations produce more consistent skeleton viewpoints.

  • Comparison with Other Pre-processing Strategies: 3.4% and 5.3%: view adaptation improves accuracy over S-trans+RNN in the CS and CV settings, respectively.Rotation-only adaptation is more effective than translation-only adaptation because most actions occur without shifting position.
  • Comparison with Other Pre-processing Strategies: 5.3% and 3.7%: the final scheme improves accuracy over F-trans&F-rota+RNN in the CS and CV settings, respectively.Against S-trans&S-rota+RNN, the improvements are 3.0% and 2.2% in CS and CV.
  • Influence of Data Augmentation: 9.8-11.8%: viewpoint augmentation brings gains on UWA3D and N-UCLA, where training and testing viewpoints differ substantially.Increasing training-view diversity addresses the mismatch between training and testing viewpoints.
  • Influence of Data Augmentation: 8.1% and 4.9%: augmentation improves VA-CNN on UWA3D and N-UCLA, respectively, by exposing the model to more viewpoints during training.Without augmentation, the view-adaptation models may fail to transform strongly mismatched test sequences to suitable learned views.
  • Influence of Network Parameters: 3.5% and 4.5%: VA-RNN(aug.) exceeds baselines with 5 or 6 LSTM layers in the CS and CV settings, respectively.The comparison indicates that simply stacking layers does not match the efficiency of the view adaptation module.
  • Influence of Network Parameters: 0.5% and 0.9%: the 53-layer proposed scheme outperforms the 152-layer baseline in the CS and CV settings, respectively.Increasing parameters through additional layers or neurons is less efficient than adding the proposed view adaptation module.

5.3 Comparisons to Other State-of-the-Art Approaches

Across five benchmark datasets, the view-adaptive models generally outperform competing skeleton-based action-recognition approaches under varied viewpoint protocols. VA-fusion(aug.) achieves especially strong results on NTU, SYSU, and N-UCLA, while the method also performs strongly on UWA3D and SBU.

  • NTU Dataset: VA-fusion(aug.) outperforms the best state-of-the-art results by 9.4% on NTU CS and 7.8% on NTU CV.The comparisons include RNN- and CNN-based methods using normalization or advanced techniques such as attention.
  • SYSU Dataset: 11.2% and 9.3% higher accuracy than is achieved on SYSU setting-1 and setting-2, respectively, while setting-1 is 10.2% higher than [28].The results use averaged performance from 30-fold cross validation.
  • UWA3D Dataset: Across UWA3D’s 12 partitions, view-adaptive models are evaluated against unseen-view conditions created by training on two viewpoints and testing on the third.The dataset has four substantially different views, making unseen-view recognition difficult.
  • N-UCLA Dataset: VA-fusion(aug.) achieves 95.3% accuracy on the N-UCLA V3 setting.The V3 setting uses samples from views 2 and 3 for training and view 1 for testing.
  • SBU Dataset: The method achieves 98.3% accuracy on SBU and outperforms the compared approaches significantly.Although SBU has no large view change, the model transforms slight view differences; VA-RNN performs better than VA-CNN on this small dataset.

5.4 Comparative Analysis of VA-RNN and VA-CNN

Both view adaptation modules improve their corresponding baselines, but VA-CNN is generally more powerful whereas VA-RNN is substantially smaller. VA-CNN also offers much higher testing speed on well-trimmed sequences, while its larger parameter count can cause overfitting on small datasets.

  • Baseline Comparison: Both VA-RNN and VA-CNN improve their respective baseline networks when view adaptation modules are introduced.The comparison is reported in Table 3.
  • Network Performance: VA-CNN(aug.) is generally more powerful than VA-RNN(aug.) because CNNs can model local and global spatial-temporal joint relationships after the sequence is converted into an image.The paper attributes RNN’s lower performance to its limited history memory.
  • Efficiency: VA-RNN(aug.) has only 2% of the model size of VA-CNN(aug.), whereas VA-CNN(aug.) reaches 83.3 sequences per second on well-trimmed sequences.The speed comparison assumes sequence length 300 frames and batch size 1.

6 CONCLUSION

The paper presents VA-RNN and VA-CNN as end-to-end networks that adapt skeleton viewpoints during recognition instead of relying on human-defined repositioning criteria. Across five challenging benchmarks, the framework consistently improves recognition and reaches state-of-the-art performance.

  • Conclusion: VA-RNN and VA-CNN adapt skeleton observation viewpoints within end-to-end recognition networks.Their optimization target is recognition performance rather than a human-predefined repositioning criterion.
  • Conclusion: The recurrent and convolutional view-adaptation models automatically transform skeletons to consistent viewpoints, reducing viewpoint diversity’s influence and easing training.The conclusion describes this effect as occurring across both model types.
  • Conclusion: The proposed framework consistently improves recognition performance on five challenging benchmark datasets and achieves state-of-the-art performance.
Loading 1804.07453v3…