Source-linked AI summary
Space-Time-Separable Graph Convolutional Network for Pose Forecasting
Theodoros Sofianos, Alessio Sampieri, Luca Franco, Fabio Galasso
TL;DR
Human pose forecasting has often separated temporal modelling from spatial joint interactions, limiting understanding of their combined dynamics. STS-GCN models both within a single graph, factorizing space-time connectivity into learned spatial and temporal matrices. Across three benchmarks, it surpasses the current best technique by over 32% on average for difficult long-term predictions while using 1.7% of its parameters.
Problem
Prior research decoupled temporal evolution from spatial joint interaction, limiting understanding of complex human body dynamics.
Method
STS-GCN uses a single GCN to encode spatial and temporal correlations, factorizing adjacency into learned spatial and temporal matrices while bottlenecking their cross-talk.
Results
Over Human3.6M, AMASS, and 3DPW, STS-GCN outperforms the current best technique by over 32% on average at difficult long-term predictions while using 1.7% of its parameters.
Takeaways & Limitations
The results support using GCNs to model human pose dynamics with learned joint-joint and time-time interactions.
Takeaways & Limitations
The proposed STS-GCN is classified as a spatial GCN because its non-symmetric signed Laplacian lacks an orthogonal eigendecomposition and is not readily interpretable through spectral-domain constructions.
Abstract
from arXiv · showhide
Human pose forecasting is a complex structured-data sequence-modelling task, which has received increasing attention, also due to numerous potential applications. Research has mainly addressed the temporal dimension as time series and the interaction of human body joints with a kinematic tree or by a graph. This has decoupled the two aspects and leveraged progress from the relevant fields, but it has also limited the understanding of the complex structural joint spatio-temporal dynamics of the human pose. Here we propose a novel Space-Time-Separable Graph Convolutional Network (STS-GCN) for pose forecasting. For the first time, STS-GCN models the human pose dynamics only with a graph convolutional network (GCN), including the temporal evolution and the spatial joint interaction within a single-graph framework, which allows the cross-talk of motion and spatial correlations. Concurrently, STS-GCN is the first space-time-separable GCN: the space-time graph connectivity is factored into space and time affinity matrices, which bottlenecks the space-time cross-talk, while enabling full joint-joint and time-time correlations. Both affinity matrices are learnt end-to-end, which results in connections substantially deviating from the standard kinematic tree and the linear-time time series. In experimental evaluation on three complex, recent and large-scale benchmarks, Human3.6M [Ionescu et al. TPAMI'14], AMASS [Mahmood et al. ICCV'19] and 3DPW [Von Marcard et al. ECCV'18], STS-GCN outperforms the state-of-the-art, surpassing the current best technique [Mao et al. ECCV'20] by over 32% in average at the most difficult long-term predictions, while only requiring 1.7% of its parameters. We explain the results qualitatively and illustrate the graph interactions by the factored joint-joint and time-time learnt graph connections. Our source code is available at: https://github.com/FraLuca/STSGCN
1. Introduction
Human pose forecasting requires modelling complex joint spatio-temporal dynamics, but prior work largely separated temporal modelling from spatial joint interactions. STS-GCN instead uses a learned, factored space-time graph and achieves stronger long-term forecasting with far fewer parameters.
- Motivation: Human pose forecasting models complex structured sequences of joint spatio-temporal dynamics with applications including autonomous driving, healthcare, teleoperations, and collaborative robots.Anticipating human motion can help avoid crashes and support robot planning.
- Architecture: The encoder uses STS-GCN to represent observed body dynamics, and a Temporal Convolutional Network forecasts future poses from that representation.The TCN provides the decoder in the encoder-decoder design.
- Limitations of prior work: Prior research generally modelled time with convolutions, recurrent networks, or Transformers, while GCNs modelled spatial joint interactions, often along a kinematic tree.These separate frameworks leveraged progress in temporal and spatial modelling but did not jointly model both dimensions.
- Proposed approach: STS-GCN encodes spatial joint-joint and temporal time-time correlations within a single GCN framework, allowing cross-talk between body interactions and motion patterns.The model is designed to bottleneck space-time cross-talk while retaining joint-joint and time-time interactions.
- Proposed approach: The first space-time-separable GCN factorizes the graph adjacency matrix into spatial and temporal matrices rather than using depthwise separation.Both factored interactions are learned end-to-end, reducing parameters while modelling space-time structure.
- Results: Over Human3.6M, AMASS, and 3DPW, STS-GCN outperforms the current best technique by over 32% on average for difficult long-term predictions while using 1.7% of its parameters.The result is reported across all three datasets.
2. Related Work
Related work has commonly separated temporal sequence modelling from spatial joint representations. STS-GCN adopts a TCN for future-frame prediction while encoding space-time structure with a GCN.
- Scope: Human pose forecasting is a long-standing problem studied through temporal sequence models and spatial representations.The related-work discussion distinguishes temporal modelling, spatial representations, and separable convolutional networks.
- Temporal modelling: RNNs, GRUs, and LSTMs are flexible temporal models but can suffer from inefficient training and poor long-term memory.Research has also explored generative adversarial networks, imitation learning, and attention to address temporal forecasting challenges.
- Temporal modelling: Temporal Convolutional Networks have achieved state-of-the-art performance in temporal modelling, so the paper uses a TCN for future-frame prediction because of its performance and robustness.The paper separately encodes space-time structure with its proposed STS-GCN.
- Separable modelling: The paper hypothesizes that space-time cross-talk is limited and that decoupling space and time is more effective and efficient.This hypothesis motivates the separable design evaluated by the paper.
3. STS-GCN
STS-GCN encodes observed joint coordinates across time with a single graph convolutional framework, factoring spatial and temporal adjacency to bottleneck their cross-talk. It uses trainable full joint-joint and time-time relations and forecasts future poses through graph encoding followed by temporal convolutions.
- 3.3. Space-Time Separable GCN: The encoder processes observed joint coordinates, and a temporal convolutional network decodes the future joint coordinates.The graph encoder uses four GCN layers with residual connections and PReLU activations.
- 3.1. Problem Formalization: STS-GCN encodes all body joints across observed frames in one spatio-temporal graph.The graph has TV nodes and an adjacency matrix relating joint interactions across times.
- 3.3. Space-Time Separable GCN: The model factors the spatio-temporal adjacency matrix as Ast = AsAt to bottleneck joint-time cross-talk while retaining joint-joint and time-time interactions.The spatial and temporal factors are separately trainable and represent full relations for each time or joint, respectively.
- 3.3. Space-Time Separable GCN: STS-GCN uses only graph convolutional layers for graph encoding, unlike competing approaches that commonly model time with recurrent or temporal convolutional mechanisms.The paper identifies exclusive GCN encoding as a key element of parameter efficiency.
- 3.4. Discussion on the STS-GCN: On Human3.6M, the improvement margin is smaller for very-short-term periodic actions and reaches up to 40% for 10-frame Posing predictions.The table reports MPJPE error in millimetres for short-term 3D joint-position prediction.
4. Experimental evaluation
STS-GCN is evaluated on three challenging pose-forecasting benchmarks, across short- and long-term horizons, against state-of-the-art methods and ablations. It achieves substantial accuracy gains, uses very few parameters, and learns interpretable spatial and temporal graph connections.
- Evaluation setup: The evaluation covers Human3.6M, AMASS, and 3DPW, with short-term and long-term prediction tests, ablations, qualitative results, and learned graph visualizations.Human3.6M contains 3.6 million 3D poses from 7 actors performing 15 actions; 22 joints are used for MPJPE and 16 for MAE.
- Comparison to the state-of-the-art: 34% average improvement over the current best occurs for Human3.6M long-term predictions from 14 to 25 frames, with gains reaching 40% for Posing.At 560 msec the improvement is 27 mm, while at 1 sec it reaches 37 mm; periodic Walking improves by 17% at 10 frames, versus 40% for aperiodic Posing.
- Comparison to the state-of-the-art: Across 120 experiments and nearly all horizons, STS-GCN outperforms competing methods, with only three exceptions involving 2-frame predictions.The comparison includes ConvSeq2Seq, LTD-X-Y, BC-WGAIL-div, and DCT-RNN-GCN, the current best performer.
- Cross-dataset evaluation: STS-GCN improves over the state-of-the-art by 32% on 25-frame AMASS predictions and by at least 43% on 4–25-frame 3DPW predictions after training on AMASS.The 3DPW experiment tests generalizability across datasets.
- Efficiency and qualitative results: The model uses only 1.7% of the current best method’s parameters, while qualitative predictions remain accurate for Walking and Discussion but are less accurate for aperiodic Posing.At 25 frames, average errors are 5.2 cm for Walking, 7.9 cm for Discussion, and 10.6 cm for Posing.
- Ablation study: Ablations show that distinct or full space-time graphs worsen performance, especially at short horizons, while a shared separable graph saves 37% of parameters with larger short-term errors.Distinct graphs produce nearly 3× larger short-term errors; the full graph requires nearly 4× more parameters, and shared adjacency matrices increase short-term errors by 12%.
- Learned graph interactions: Learned spatial edges mostly follow the kinematic tree but also connect distant motion-related joints, while temporal edges emphasize information flow from earlier frames toward the latest observed frames.The learned adjacency matrices are directed and signed; temporal coefficients have a smaller range than spatial coefficients, privileging spatial information during forecasting.
5. Conclusions
STS-GCN combines space and time in one graph while factorizing their interaction, enabling trainable joint-joint and time-time relationships with fewer parameters.
- STS-GCN uses a single graph to model space-time pose dynamics and favor cross-talk between joint interactions and temporal motion patterns.
- Factorizing space-time interactions enables fully trainable joint-joint and time-time relationships while bottlenecking their cross-talk.
- The model considerably improves state-of-the-art performance while requiring only a fraction of competing methods’ parameters.