Source-linked AI summary
Skeleton-Based Online Action Prediction Using Scale Selection Network
Jun Liu, Amir Shahroudy, Gang Wang, Ling-Yu Duan, Alex C. Kot
TL;DR
Online action prediction must classify an ongoing action from partial observations in continuous 3D skeleton streams, where temporal scale varies and previous actions can interfere. The paper proposes SSNet with dynamic scale selection, temporal dilated convolutions, activation sharing, and structured skeleton representations; it reports superior performance across evaluated benchmark datasets. Its scope includes limitations associated with ending-stage frames and a maximum temporal scale of 255 frames.
Problem
Online action prediction must recognize the current action from partial observations in untrimmed continuous 3D skeleton streams containing multiple action instances.
Method
SSNet uses temporal dilated convolutions, dynamically selected window scales, activation sharing, and hierarchical dilated tree convolutions for structured skeleton representations.
Results
SSNet yields superior performance on all evaluated benchmark datasets.
Takeaways & Limitations
The framework provides an end-to-end approach for efficient online action prediction in untrimmed skeleton sequences and could be extended to streaming temporal action detection.
Takeaways & Limitations
Prediction accuracy decreases at ending stages for some actions, where postures and motions may be less relevant to the action class; the top temporal scale is 255 frames.
Abstract
from arXiv · showhide
Action prediction is to recognize the class label of an ongoing activity when only a part of it is observed. In this paper, we focus on online action prediction in streaming 3D skeleton sequences. A dilated convolutional network is introduced to model the motion dynamics in temporal dimension via a sliding window over the temporal axis. Since there are significant temporal scale variations in the observed part of the ongoing action at different time steps, a novel window scale selection method is proposed to make our network focus on the performed part of the ongoing action and try to suppress the possible incoming interference from the previous actions at each step. An activation sharing scheme is also proposed to handle the overlapping computations among the adjacent time steps, which enables our framework to run more efficiently. Moreover, to enhance the performance of our framework for action prediction with the skeletal input data, a hierarchy of dilated tree convolutions are also designed to learn the multi-level structured semantic representations over the skeleton joints at each frame. Our proposed approach is evaluated on four challenging datasets. The extensive experiments demonstrate the effectiveness of our method for skeleton-based online action prediction.
1. Introduction
The paper addresses real-time online prediction of ongoing actions in continuous 3D skeleton streams, where partial observations and multiple untrimmed action instances make temporal scale selection difficult. SSNet dynamically selects temporal scales, shares computation across steps, and learns structured skeleton representations, achieving superior performance on four challenging datasets.
- Online action prediction recognizes the current action from only its observed portion in an untrimmed stream containing multiple action instances.
- Temporal scale selection is difficult because the useful window changes as more of the current action is observed and larger windows may include previous actions.
- SSNet dynamically learns a temporal window scale at each step by regressing the distance to the current action’s beginning and using it for subsequent prediction.
- A hierarchy of dilated tree convolutions learns multi-level structured semantic representations over skeleton joints to enhance prediction from 3D skeleton data.
- The framework improves efficiency through activation sharing and avoids expensive multi-stage, multi-network processing with an end-to-end trainable design.
- SSNet achieves superior performance on four challenging datasets for 3D skeleton-based activity analysis.
2. Related Work
Prior work includes skeleton-based recognition, early action prediction, and temporal action analysis, but commonly assumes segmented sequences or addresses offline detection. SSNet instead selects the proper temporal window at every step for online prediction in untrimmed continuous skeleton streams while sharing redundant computations.
- Skeleton-Based Action Recognition: Existing skeleton action-recognition methods generally use fully observed segmented videos, whereas this paper targets continuous streams with partial actions.
- Action Prediction: Earlier action-prediction work includes sparse coding, dynamic bag-of-words, predictive accumulative functions, and real-time skeleton prediction for well-segmented sequences.
- Action Analysis with Untrimmed Sequences: Action detection addresses spatio-temporal segmentation, while action prediction estimates the class of the current ongoing action from its observed portion.
- Action Analysis with Untrimmed Sequences: Unlike fixed-scale or multi-scale multi-round approaches, SSNet chooses the proper temporal window at each step and shares redundant computations across steps.
- Skeleton-Based Action Recognition: This manuscript extends prior work by replacing simple coordinate concatenation with hierarchical dilated tree convolutions that learn multi-level structured skeleton representations.
3. The Proposed Method
SSNet performs online action prediction with temporal dilated convolutions, dynamically selects prediction scales, shares activations across time, and learns structured skeleton representations with tree convolutions.
- 3.1. Temporal Modeling with Convolutional Layers: SSNet uses 1-D causal dilated convolutions over temporal windows to model motion dynamics without using future frames.Dilation expands the receptive field while skipping intermediate time steps; stacked layers represent different temporal scales.
- 3.2. Scale Selection: At each step, SSNet jointly predicts the ongoing action class and regresses the current action’s start-point distance for scale selection.The previous step’s regression guides the current prediction by selecting the corresponding convolutional layers and window scale.
- 3.4. Activation Sharing Scheme: Activation sharing reuses convolutional activations from adjacent time steps, requiring computation only for the current temporal column and enabling real-time operation.The hierarchical network contains 14 temporal scales, while shared activations avoid recomputing overlapping operations.
- 3.5. Multi-level Structured Skeleton Representations: The model processes each frame’s skeleton as a spatial tree rather than concatenated coordinates to capture local, mid-level, and holistic pose structure.The hierarchy of dilated tree convolutions learns multi-level structured representations and is attached as an end-to-end trainable input module.
- 3.5. Multi-level Structured Skeleton Representations: Dilated tree convolutions slide triangular filters over skeleton subtrees and use progressively dilated connections to encode structured spatial semantics.Only boundary nodes within each dilated triangular region contribute to the activation.
4. Experiments
The experiments compare SSNet with fixed-scale, multi-scale, ground-truth-scale, recurrent, and attention-based methods for online skeleton action prediction. They evaluate whether dynamic scale selection and structured skeleton representations improve performance on continuous streams.
- Evaluation setting: The evaluation uses four challenging datasets containing multiple action instances in long videos, with an added blank class for background frames.The experiments target continuous online prediction rather than isolated, single-action clips.
- Compared methods: SSNet is compared against fixed-scale FSNets, exhaustive multi-scale FSNet-MultiNet, an ideal SSNet-GT baseline, and established ST-LSTM, JCR-RNN, and Attention Net methods.All listed architectures use multi-level structured skeleton representations for fair comparisons.
- Experimental rationale: SSNet tests dynamic temporal scale selection against fixed and exhaustive multi-scale alternatives while also comparing with recurrent and attention-based skeleton models.This design isolates the contribution of scale selection within a broader state-of-the-art comparison.
4.1. Implementation Details
The implementation evaluates SSNet’s structured skeleton representation, parameter efficiency, and online prediction accuracy on OAD. Results show strong performance across observation ratios and comparable accuracy to the ideal ground-truth-scale baseline.
- Implementation details: The structured skeleton representation increases SSNet’s parameter count only slightly, with dilated tree convolutions accounting for 13% of the whole model.Table 3 compares parameter counts for alternative skeleton representations.
- OAD results: SSNet with regressed scales achieves results comparable to the ideal SSNet-GT baseline across progress levels.SSNet-GT uses ground-truth scales, whereas SSNet selects scales from its regression output.
- OAD results: Apart from the ideal SSNet-GT baseline, SSNet achieves the best prediction results at every OAD observation ratio, including about 66% accuracy after observing 10% of an action.Table 4 reports the OAD accuracies, while Figure 7 provides the broader progress-level results.
- OAD results: The fixed-scale baselines are weaker than SSNet, and even exhaustive multi-scale FSNet-MultiNet underperforms the single SSNet at all progress levels.This comparison supports the effectiveness of dynamically selecting the temporal scale.
- OAD results: SSNet significantly outperforms JCR-RNN, ST-LSTM, and Attention Net on continuous streaming skeleton sequences.The paper attributes early-stage gains to focusing on the current action and later-stage gains to retaining long-term temporal context.
- OAD results: Average prediction accuracy decreases at ending stages, possibly because some late frames contain motions and postures less relevant to the current action label.This is reported as an observed behavior with a proposed explanation rather than a quantified failure boundary.
4.3. Experiments on the ChaLearn Gesture Dataset
The ChaLearn Gesture experiments evaluate online prediction in a large, challenging dataset with multiple similar action instances per video. SSNet outperforms the other methods at every observation ratio.
- Dataset: The dataset contains 23 hours of Kinect video, 20 action classes, and 27 subjects, with many classes having highly similar body motions.These characteristics make the dataset challenging for online action prediction.
- Dataset: Each video contains 8–20 action instances, making the dataset suitable for online action prediction in continuous streams.Starting and ending frames are annotated for 11,116 action instances.
- Results: SSNet outperforms the other methods at all observation ratios on the ChaLearn Gesture dataset.The results are reported in Table 5 and Figure 8.
4.4. Experiments on the PKUMMD Dataset
The PKUMMD experiments use a cross-subject evaluation of challenging interaction actions in continuous sequences. SSNet achieves the best results at every progress level, including when only 10% of the action is observed.
- Dataset: The experiment uses a cross-subject protocol with 57 subjects for training and 9 for testing, focusing on challenging interaction actions.Videos are subsampled by taking one frame from every four frames.
- Results: SSNet achieves the best prediction results at all PKUMMD progress levels and significantly outperforms other methods at the 10% observation ratio.The results are presented in Table 6 and Figure 9.
- Results: The early-stage advantage is linked to SSNet’s explicit scale selection, which focuses prediction on the current action when only a small portion is observed.Methods without explicit scale selection perform worse in this setting.
- Results: Fixed-scale FSNet performance is highly sensitive to the selected scale, unlike SSNet’s dynamic scale choice.Different fixed scales produce very different results on this dataset.
4.5. Experiments on the G3D Dataset
On G3D, SSNet is evaluated for framewise online action prediction across observation ratios and reports superior performance.
- SSNet achieves superior action-prediction performance on the G3D dataset.
4.6. Evaluation of Skeleton Representations
The experiments show that structured skeleton representations improve SSNet and that even coordinate concatenation outperforms prior skeleton-based methods.
- Table 8 compares SSNet using multi-level structured representations against coordinate concatenation for action prediction.
- Hierarchical dilated tree convolutions significantly improve SSNet action-prediction performance by learning multi-level structured skeleton representations.
- Even with coordinate concatenation, SSNet outperforms JCR-RNN, ST-LSTM, and Attention Net on all four datasets.
4.7. Evaluation of Distance Regression
SSNet evaluates start-point distance regression, network-scale configurations, γ, skip connections, and frame-level classification, with structured representations improving regression.
- SSNet provides the best start-point regression performance, and its structured representation outperforms coordinate concatenation.
- At 5% observation, the average regression error is 6 frames, and regression becomes more reliable as more frames are observed.
- A 14-layer SSNet with a maximum perception window of 255 frames is suitable because 16 layers perform almost the same.
- SSNet achieves its best performance when γ is set to 0.01.
- Skip connections improve action-prediction accuracy by about 1.5%, whereas adding batch normalization shows no obvious improvement.
4.9. Frame-level Classification Accuracies
Frame-level classification accuracies are evaluated across all video frames, and SSNet outperforms the compared approaches.
- SSNet shows superior average frame-level classification accuracy over the compared approaches.
5. Conclusion
SSNet addresses online action prediction in untrimmed skeleton sequences by modeling temporal dynamics, selecting an appropriate window scale, and learning structured skeletal representations. It achieves superior performance across all evaluated benchmark datasets and could be extended to temporal action detection.
- SSNet models temporal dynamics and dependencies in untrimmed skeleton sequences for online action prediction.
- Its scale selection scheme chooses the layer corresponding to the most appropriate temporal window scale at each time step.
- Dilated tree convolutions learn multi-level structured representations of skeletal data to improve network performance.
- The proposed method yields superior performance on all evaluated benchmark datasets.
- The network could be extended to temporal action detection in streaming skeleton sequences, which remains future work.