Source-linked AI summary
Action-Agnostic Human Pose Forecasting
Hsu-kuang Chiu, Ehsan Adeli, Borui Wang, De-An Huang, Juan Carlos Niebles
TL;DR
Human pose forecasting is limited by reliance on action labels and separate short- or long-term settings, despite its relevance to human-machine interaction. The paper proposes TP-RNN, an action-agnostic hierarchical multi-scale recurrent model that forecasts pose velocities across both horizons. Experiments on Human 3.6M and Penn Action report superior results to baseline and state-of-the-art methods, while difficult actions remain close to the zero-velocity baseline.
Problem
Prior pose forecasters often specialize in short- or long-term prediction and require action labels at test time, which are unavailable for many testing videos.
Method
TP-RNN hierarchically encodes pose velocities with recurrent sequences operating at different temporal scales, without requiring external action labels.
Results
The method outperforms baseline and state-of-the-art methods on Human 3.6M MAE and Penn Action PCK for both short- and long-term predictions.
Takeaways & Limitations
Action-agnostic hierarchical modeling supports simultaneous short- and long-term pose forecasting without sacrificing the accuracy of either horizon.
Takeaways & Limitations
TP-RNN has difficulty forecasting actions such as Smoking and Discussion, where results remain close to the zero-velocity baseline.
Abstract
from arXiv · showhide
Predicting and forecasting human dynamics is a very interesting but challenging task with several prospective applications in robotics, health-care, etc. Recently, several methods have been developed for human pose forecasting; however, they often introduce a number of limitations in their settings. For instance, previous work either focused only on short-term or long-term predictions, while sacrificing one or the other. Furthermore, they included the activity labels as part of the training process, and require them at testing time. These limitations confine the usage of pose forecasting models for real-world applications, as often there are no activity-related annotations for testing scenarios. In this paper, we propose a new action-agnostic method for short- and long-term human pose forecasting. To this end, we propose a new recurrent neural network for modeling the hierarchical and multi-scale characteristics of the human dynamics, denoted by triangular-prism RNN (TP-RNN). Our model captures the latent hierarchical structure embedded in temporal human pose sequences by encoding the temporal dependencies with different time-scales. For evaluation, we run an extensive set of experiments on Human 3.6M and Penn Action datasets and show that our method outperforms baseline and state-of-the-art methods quantitatively and qualitatively. Codes are available at https://github.com/eddyhkchiu/pose_forecast_wacv/
1. Introduction
Human pose forecasting supports applications requiring machines to anticipate human dynamics, but prior methods often depend on action labels or specialize in only short- or long-term prediction. The paper introduces TP-RNN, an action-agnostic model intended to forecast both horizons using hierarchical multi-scale temporal dependencies.
- Motivation: Human pose forecasting aims to predict how structured body joints change over time, supporting machines that interact with people.
- Limitations of prior work: Prior methods often produce unrealistic short-term poses and may fail to outperform the zero-velocity predictor.
- Limitations of prior work: Action-specific models or action-label inputs are unsuitable when testing videos lack activity annotations.
- Proposed approach: TP-RNN encodes temporal dependencies at different time-scales through a hierarchical interconnected sequence of RNN cells.
- Proposed approach: The model forecasts pose velocities rather than poses, using velocities as both inputs and outputs because poses change slightly across narrow time-steps.
- Evaluation: On Human 3.6M and Penn Action, the action-agnostic method outperforms baselines and state-of-the-art methods on MAE and PCK for short- and long-term prediction.
- Contributions: The paper contributes an action-agnostic forecaster, TP-RNN, and velocity-space modeling without requiring external action labels during training.
2. Related Works
Related work has addressed future human dynamics at the pixel, trajectory, and pose levels using CNNs, RNNs, GANs, and variational methods. These approaches establish pose forecasting as a distinct direction within broader motion prediction research.
- Motion and human dynamics: Recent motion-representation work has predicted future video pixels using GANs and RNNs, while other methods forecast dense trajectories with CNNs.
- Pose forecasting: Pose forecasting methods have predicted human dynamics from static images, 2D pose locations, or pose sequences.
- Pose forecasting: Existing approaches include 3D-PFNet, 3D-position prediction, probabilistic sequence-to-sequence modeling, and variational or adversarial prediction.
3. Triangular-Prism RNN (TP-RNN)
TP-RNN models human pose velocities with a hierarchical, multi-phase RNN whose levels represent different temporal granularities. Shared weights and phase-shifted sequences support short- and long-term dependency modeling while controlling parameter growth.
- Hierarchical multi-scale dynamics: Human pose sequences exhibit hierarchical dependencies because body parts influence one another while following distinct temporal scales.
- TP-RNN architecture: The first hierarchy level processes velocity at every time-step, while higher levels use K phase-shifted LSTM sequences sampled at modulo-K time-steps.
- TP-RNN architecture: LSTM sequences within each hierarchy level share weights, so phase shifting augments data without increasing the number of physically distinct sequences.
- Prediction stage: A two-layer fully connected network predicts the next velocity from the current velocity and hidden units across all hierarchy levels.
- Temporal modeling: TP-RNN assigns short-term dependencies to lower levels and long-term dependencies to higher levels, unlike the single-layer or stacked RNN alternatives discussed.
- Architecture illustration: The illustrated architecture uses K = 2 and M = 2, with a three-dimensional view and a two-dimensional projection.
4. Experiments
Experiments on Human 3.6M and Penn Action compare TP-RNN with baselines and state-of-the-art methods for short- and long-term pose forecasting. TP-RNN generally improves accuracy while remaining action-agnostic and avoiding a short-/long-term trade-off.
- Datasets and evaluation: Experiments use Human 3.6M and Penn Action, with MAE and PCK@0.05 as evaluation metrics.Human 3.6M contains 15 activity categories; Penn Action evaluation uses PCK@0.05.
- Ablation studies: Velocity-space LSTM inputs and outputs are compared against pose-input models to test whether velocity modeling better captures human motion dynamics.The ablation contrasts Single Layer (Pose) with Single Layer (Vel.).
- Limitations: For difficult-to-predict actions such as Smoking and Discussion, TP-RNN remains superior but produces results close to the zero-velocity baseline.This limitation is reported despite the model’s overall improvements.
- Human 3.6M results: TP-RNN also outperforms the compared methods in long-term forecasting in most cases while avoiding the short-term accuracy sacrifice observed when Residual is adapted for long-term prediction.Dropout-AutoEncoder remains better for 1000ms Discussion prediction.
- Penn Action results: On Penn Action, TP-RNN performs significantly better than 3D-PFNet under PCK@0.05, with a reported p-value of 0.0419.The comparison uses the authors’ model and modified open-source code for the competing method under the same setting.
5. Conclusion
The paper introduces TP-RNN to encode hierarchical human dynamics across time-scales, while evaluating action-agnostic forecasting of both short- and long-term sequences.
- 5. Conclusion: TP-RNN uses RNN sequences at multiple hierarchical time-scales to capture fine- and coarse-grained motion dynamics.Within each hierarchy, RNNs share learnable weights across phases, reducing parameters while using more data to train higher-level RNNs.
- 5. Conclusion: The evaluation adopts action-agnostic forecasting without activity labels and predicts short- and long-term sequences simultaneously.These settings are presented as more rigorous and realistic than prior approaches limited by action labels or a single prediction horizon.
- 5. Conclusion: The proposed framework is designed to encode different hierarchies in human dynamics at different time-scales.The model is inspired by hierarchical multiscale RNN frameworks used in natural language processing.