Source-linked AI summary
View Adaptive Recurrent Neural Networks for High Performance Human Action Recognition from Skeleton Data
Pengfei Zhang, Cuiling Lan, Junliang Xing, Wenjun Zeng, Jianru Xue, Nanning Zheng
TL;DR
Skeleton-based action recognition is challenged by viewpoint variation, which makes the same action appear through substantially different skeleton representations. The paper introduces an end-to-end view adaptive LSTM that selects suitable viewpoints during recognition, improving performance on three benchmark datasets while preserving action continuity.
Problem
Large viewpoint variations make skeleton-based action recognition difficult, while existing human-defined preprocessing can lose motion information and changing body orientation.
Method
An end-to-end view adaptive RNN with LSTM architecture automatically regulates observation viewpoints at each frame and feeds the transformed skeletons to a main LSTM for recognition.
Results
The model significantly improves recognition performance and achieves state-of-the-art results on three benchmark datasets.
Takeaways & Limitations
The learned adaptation produces more consistent viewpoints while maintaining action continuity instead of forcing every frame into the same position and orientation.
Takeaways & Limitations
Human-defined preprocessing can lose moving trajectories, body-center speed, and changing body-orientation dynamics, motivating the learned adaptation approach.
Abstract
from arXiv · showhide
Skeleton-based human action recognition has recently attracted increasing attention due to the popularity of 3D skeleton data. One main challenge lies in the large view variations in captured human actions. We propose a novel view adaptation scheme to automatically regulate observation viewpoints during the occurrence of an action. Rather than re-positioning the skeletons based on a human defined prior criterion, we design a view adaptive recurrent neural network (RNN) with LSTM architecture, which enables the network itself to adapt to the most suitable observation viewpoints from end to end. Extensive experiment analyses show that the proposed view adaptive RNN model strives to (1) transform the skeletons of various views to much more consistent viewpoints and (2) maintain the continuity of the action rather than transforming every frame to the same position with the same body orientation. Our model achieves significant improvement over the state-of-the-art approaches on three benchmark datasets.
1. Introduction
Skeleton-based action recognition must handle large viewpoint differences that make identical postures appear dissimilar. The proposed end-to-end view adaptive LSTM dynamically selects observation viewpoints to improve recognition while preserving action continuity.
- Viewpoint changes in camera position, angle, and subject orientation produce substantially different skeleton representations of the same posture.
- Existing human-defined preprocessing partially removes viewpoint variation but can discard body-center motion, speed, and changing body orientation.Walking may become walking in place, while rotating dancing may become dancing toward a fixed orientation.
- The view adaptive RNN learns suitable viewpoints from input skeletons and regulates them dynamically at each frame instead of applying fixed human-defined transformations.Its View Adaptation Subnetwork determines viewpoints, while the Main LSTM Network uses the transformed skeletons for action classification.
- The model automatically produces more consistent viewpoints while maintaining action continuity rather than forcing every frame into one position and orientation.
- Extensive evaluations report state-of-the-art performance on three benchmark datasets and strong ability to overcome viewpoint variation.
2. Related Work
Prior skeleton-based methods commonly rely on preprocessing or alignment criteria that can lose motion information or misalign non-rigid poses. This work instead learns content-dependent viewpoints frame by frame within an RNN-based recognition model.
- Viewpoint variation remains under-explored for skeleton-based recognition, while many existing approaches were developed for color video and cannot be directly extended.
- Common preprocessing centers and rotates skeletons using human-defined criteria, but this can lose partial relative motion information.
- Sequence-based preprocessing preserves motion by applying one transformation across frames, yet its alignment parameters may be unsuitable because the human body is not rigid.
- The architecture combines a View Adaptation Subnetwork with a Main LSTM Network that classifies actions from newly represented skeletons.
- The proposed content-dependent view adaptation model automatically learns suitable viewpoints for each frame rather than relying on fixed alignment rules.
3. RNN and LSTM Overview
RNNs model sequential data by carrying information across time, while LSTM augments this design with a cell state and gates to regulate retained and added information.
- RNN: RNN outputs at time t depend on the current input x_t and the previous hidden output h_{t−1}.This recurrence supports sequential modeling and feature extraction by allowing previous information to persist.
- RNN: Standard RNNs can suffer vanishing gradients, limiting their ability to model long-term dependencies.
- LSTM: LSTM mitigates this limitation through a cell state c_t that carries information across time.The cell state is regulated through linear interactions rather than relying only on the standard recurrent pathway.
- LSTM: The forget gate f_t and input gate i_t regulate removal of previous information and addition of current information to the cell state.
4. View Adaptation Model using LSTM
The view adaptation model uses an end-to-end LSTM subnetwork to regulate each frame’s observation viewpoint before a main LSTM performs action recognition. It learns frame-specific translations and rotations while preserving temporal continuity through jointly trained subnetworks.
- Architecture: The proposed network combines a View Adaptation Subnetwork with a Main LSTM Network for end-to-end skeleton-based action recognition.The adaptation component determines observation viewpoints, while the main network models temporal dynamics and extracts recognition features.
- Viewpoint regulation: Each frame can be re-observed through a movable virtual camera, transforming its skeleton into a new observation coordinate system.The transformation uses a translation d_t and rotations α_t, β_t, γ_t around the global coordinate axes.
- Viewpoint regulation: All joints in frame t share the same rigid transform parameters, while viewpoints may vary across frames.This applies one translation and rotation configuration to the complete skeleton at each time step.
- View Adaptation Subnetwork: Separate LSTM branches learn rotation parameters and the translation vector from skeleton inputs.The rotation branch uses an LSTM and fully connected layer to produce α_t, β_t, γ_t, while the translation branch produces d_t.
- Main LSTM Network: The Main LSTM Network stacks three LSTM layers followed by a fully connected SoftMax classifier whose output size matches the action classes.
- Training: Gradients flow from the Main LSTM Network back into the View Adaptation Subnetwork, and cross-entropy trains the entire model end to end.The adaptation is optimized for recognition accuracy and is guided to select suitable observation viewpoints.
5. Experiment Results
Experiments on three benchmark datasets evaluate VA-LSTM against state-of-the-art methods, preprocessing strategies, and view-adaptation variants. The model improves accuracy, regulates skeletons toward consistent viewpoints, and preserves temporal motion.
- Experimental setup: VA-LSTM is evaluated on NTU, SBU, and SYSU benchmark datasets, with additional in-depth analyses and skeleton visualizations on NTU.The experiments include comparisons with state-of-the-art approaches, preprocessing methods, and view-adaptation models.
- Comparisons to other state-of-the-art: About 6%, 4%, and 1% accuracy improvements are reported over state-of-the-art approaches on NTU, SBU, and SYSU, respectively.The comparisons use Tables 1–3 for the three datasets.
- View adaptation variants: VA-LSTM combines translation and rotation branches to automatically regulate observation viewpoints within the network.The translation-only and rotation-only variants disable one corresponding branch, while Strans+LSTM disables view adaptation.
- View adaptation variants: 3.4% and 5.3% accuracy gains over Strans+LSTM are reported for NTU Cross-Subject and Cross-View settings, respectively.The improvement is attributed to introducing the view adaptation module.
- Preprocessing comparisons: VA-LSTM improves over human-defined preprocessing, while sequence-level preprocessing outperforms frame-level preprocessing because frame-wise processing loses motion across frames.Against F-trans&F-rota+LSTM, gains are 5.3% for Cross-Subject and 3.7% for Cross-View; against S-trans&S-rota+LSTM, gains are 3.0% and 2.2%.
- Visualization of learned views: Visualizations show that learned transformations produce more consistent viewpoints while retaining continuous, natural motion across frames.Examples include preserved position changes in “staggering,” unlike frame-level preprocessing.
6. Conclusion
The paper presents an end-to-end view adaptation model that automatically regulates skeleton observation viewpoints instead of relying on human-defined repositioning criteria. Experiments show improved recognition performance on three benchmark datasets, achieving state-of-the-art results.
- The model automatically regulates observation viewpoints to suitable ones while optimizing recognition performance.
- It uses a content-dependent recurrent network to exploit optimal viewpoints rather than human-defined preprocessing criteria.
- The proposed model significantly improves recognition performance on three benchmark datasets and achieves state-of-the-art results.