Action-Agnostic Human Pose Forecasting

1. Introduction

• Prior methods often targeted only short-term [18] [23] or long-term forecasting [8] [16] [36], and some failed to outperform repeating the last observed pose [23].The introduction also motivates pose forecasting as important for machines to interact seamlessly with people.Source: §1 · p. 1
• TP-RNN forecasts regardless of action class, implicitly encoding short- and long-term dependencies without requiring action labels as inputs at test time.Prior methods used action labels as extra inputs or trained separate forecasters by action class, although such labels are unavailable for testing videos.Source: §1 · p. 2
• TP-RNN uses interconnected recurrent cells at different time-scales to model hierarchical dependencies among body parts and joints with distinct motion patterns.For example, during walking, arms and legs move more frequently than the torso.Source: §1 · p. 2
• The model uses pose velocities as both inputs and outputs, a design the authors argue improves forecasting because poses change slightly between nearby frames.Pose velocities are differences between consecutive poses, without dividing by the constant time between frames.Source: §1 · p. 2
• The method outperforms baseline and state-of-the-art algorithms in mean angle error on Human 3.6M and PCK on Penn Action for both short- and long-term forecasting.The comparison includes methods designed specifically for either prediction horizon and methods using action labels as inputs.Source: §1 · p. 2

Predicting Motion and Human Dynamics

• Motion-prediction work has targeted pixels [25] [22], dense trajectories, semantic or activity labels [21] [2], and 2D or 3D poses [35].Human-motion modeling also includes probabilistic and state-transition methods and deep-learning approaches such as RNNs and LSTMs [16] [18] [23].Source: §2 · p. 2

Human Pose Forecasting

• Pose forecasting is comparatively recent despite applications such as predictive surveillance and patient monitoring [8] [11] [18] [23] [24] [36], and real-world inference may lack activity labels [23] [9].Some existing methods use action labels as high-level supervision, which the paper notes may be unavailable at inference time.Source: §2 · p. 3
• Prior pose forecasters include static-image 3D prediction [11], probabilistic pose prediction [8], and recurrent or sequence-based approaches [15] [23] [24] [9].These approaches include predicting future 3D poses from 2D locations [24], LSTM-based architectures [15], and models of short-term motion [23].Source: §2 · p. 3

Hierarchical Multi-Scale RNNs

• TP-RNN is inspired by HM-RNN [12] and multi-scale RNNs [19], which model slowly changing high-level and rapidly changing low-level abstractions, but human motion lacks clear-cut temporal boundaries.The cited hierarchical language models learn representations at levels such as words, phrases, and sentences [7, 14].Source: §2 · p. 3

3. Triangular-Prism RNN (TP-RNN)

• Unlike single-layer RNN/LSTMs [18] and stacked LSTMs [13], TP-RNN uses hierarchical, multi-phase LSTM sequences to model longer-term dependencies across temporal granularities.The design reflects that body-part movements depend on other parts and can follow distinct motion patterns and time-scales.Source: §3 · p. 3
• Given pose velocities, TP-RNN recursively creates K phase-shifted LSTM sequences per level from congruent-modulo-K inputs, yielding K^(M-1) sequences at level M.The first level processes every time-step; each higher level samples phase-specific outputs from the level below.Source: §3 · p. 3
• LSTM sequences within each level share weights, while a two-layer fully connected head predicts velocity from current velocity and hidden states across all hierarchies.The phase shifts augment training data without increasing the number of physically distinct LSTM sequences.Source: §3 · p. 3
• Unlike HM-RNN [12], TP-RNN uses outputs from all hierarchy levels because human dynamics lack natural sequence boundaries, providing up-to-date motion information at different scales during prediction.Lower levels model short-term dependencies and finer-grained motion, while higher levels capture longer-term dynamics and different motion phases.Source: §3 · pp. 3–4

4.1. Datasets

• Human 3.6M experiments used 50 observed frames to forecast 25 and angle-space MAE; Penn Action used one pose to forecast 16 frames, evaluated by PCK@0.05.For Human 3.6M, Subject 5 was held out for testing; Penn Action used 1258 training and 1068 test sequences.Source: §4.1 · pp. 4–5

4.2. Results

• TP-RNN beat all baselines in short-term forecasts, while long-term results trailed Dropout-AutoEncoder [16] only for Discussion at 1000ms; Smoking and Discussion remained difficult.For those difficult actions, results were close to the zero-velocity baseline, though still superior to other methods.Source: §4.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
• TP-RNN outperformed Residual [23] on all-15-action average MAE for both horizons, improving long-term forecasting by 6.56% without sacrificing short-term accuracy.The paper reports that Residual [23] compromises between short- and long-term accuracy, whereas TP-RNN performs better, particularly at 1000ms.Source: Table 4 · p. 7, §4.2 · p. 7
• Ablations found that stacking two velocity-input LSTMs did not improve overall performance, multi-scale extensions did, and TP-RNN achieved the best quantitative results.The HM-RNN-derived double-scale variant did not improve long-term performance; TP-RNN’s higher-level multiple LSTM phases capture larger-scale temporal information.Source: §4.2 · pp. 5, 7, Table 4 · p. 7
• Increasing TP-RNN hierarchy levels generally improved long-term and slightly improved short-term MAE, but performance declined at four short-term or five long-term levels.The authors attribute the decline to sparse updates at the fifth level and increased overfitting risk from more trainable parameters.Source: Figure 3 · p. 7, §4.2 · p. 7
• On Penn Action, TP-RNN significantly outperformed 3D-PFNet [11] (p = 0.0419); adding estimated initial velocity further improved its forecasting results.The comparison without initial velocity set that input to zero to match 3D-PFNet’s setting.Source: Table 5 · p. 8, §4.2 · p. 8
• Visualizations show TP-RNN forecasts closer to ground truth than Residual [23] for Walking and Smoking, including more accurate hand and leg movements.For Walking, TP-RNN’s MAE was 0.25 at 80ms versus 0.32 for [23], and 0.75 at 560ms and 0.77 at 1000ms versus 0.86 and 0.96 for [23].Source: §4.2 · pp. 7–8, Figure 4 · p. 8

• TP-RNN outperforms baseline and state-of-the-art methods on Human 3.6M and Penn Action in quantitative evaluations and qualitative visualizations.Source: §5 · p. 8
• TP-RNN uses RNN sequences at different hierarchical time-scales, with shared weights within each level to reduce parameters and train higher-level RNNs with more data.Source: §5 · p. 8
• The forecasting setup uses no activity labels and predicts short- and long-term sequences simultaneously.Source: §5 · p. 8