Source-linked AI summary
QuaterNet: A Quaternion-based Recurrent Model for Human Motion
Dario Pavllo, David Grangier, Michael Auli
TL;DR
Human-motion prediction must handle competing rotation and position representations, long-horizon uncertainty, and constraints such as valid skeleton geometry. QuaterNet uses quaternion recurrent modeling with a differentiable forward-kinematics position loss, improving short-term prediction and matching recent graphics methods qualitatively for long-term locomotion generation. The approach also supports real-time generation and greater control over timing and trajectory constraints, within the paper’s stated scope.
Problem
Rotation regression can suffer from parameterization discontinuities and unequal joint-error weighting, while position regression requires reprojection to enforce skeleton constraints; long-term generation is further complicated by stochastic uncertainty.
Method
QuaterNet is a quaternion-based recurrent network trained with a differentiable loss that applies forward kinematics to predicted rotations and penalizes absolute joint-position errors.
Results
QuaterNet improves short-term angle prediction on Human3.6M and matches previous computer-graphics work in judged long-term locomotion quality.
Takeaways & Limitations
The approach supports both short-term prediction and long-term generation, with real-time generation and better control over timing and trajectory constraints for locomotion.
Takeaways & Limitations
Future work is needed to extend QuaterNet to other motion tasks and to provide richer action conditioning and artistic control.
Abstract
from arXiv · showhide
Deep learning for predicting or generating 3D human pose sequences is an active research area. Previous work regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angle or exponential map parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. Our recurrent network, QuaterNet, represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. On short-term predictions, QuaterNet improves the state-of-the-art quantitatively. For long-term generation, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature.
1 Introduction
QuaterNet addresses short-term prediction and long-term generation of human motion by combining quaternion-based recurrent modeling with a differentiable forward-kinematics position loss. It improves short-term angle prediction on Human3.6M and matches prior locomotion-generation quality while adding real-time and control benefits.
- Motivation: Human motion modeling supports applications including action recognition, action detection, and computer graphics, but long-horizon evaluation is difficult because future motions are intrinsically uncertain.Short-term prediction is commonly assessed against reference recordings, whereas long-term generation relies more heavily on human evaluation.
- Objective: The work targets both short-term prediction and long-term generation against state-of-the-art methods in computer vision and computer graphics, respectively.
- Contributions: QuaterNet uses quaternions in recurrent neural networks to avoid discontinuities and singularities associated with Euler angles and incompletely resolved by exponential maps.The method also introduces a differentiable forward-kinematics loss that combines rotation prediction with position-based supervision.
- Results: QuaterNet improves state-of-the-art angle prediction errors for short-term Human3.6M prediction.
- Results: For long-term locomotion generation, human judgment finds QuaterNet comparable in quality to previous computer-graphics work, while enabling online generation and greater control over timing and trajectory constraints.
- Resources: The paper releases QuaterNet code and pre-trained models publicly.The release is provided through the project’s GitHub repository.
2 Related Work
Prior human-motion research spans multiple representations, architectures, and strategies for handling uncertainty, but rotation and position prediction each retain important trade-offs. Long-term generation is especially difficult because uncertainty grows with duration and autoregressive models can suffer from exposure bias.
- Existing approaches: Motion-capture data underpins human-motion generation, with prior approaches ranging from concatenative methods and probabilistic models to neural networks and physics-inspired models.
- Representations: Human poses can be represented by joint positions or rotations integrated through forward kinematics, and each choice offers different benefits and trade-offs.
- Joint rotations versus positions: Rotation prediction preserves parameterized-skeleton constraints, but equal joint weighting can underrepresent the greater positional impact of trunk and root errors.
- Joint rotations versus positions: Position prediction avoids unequal joint weighting but requires reprojection to prevent bone stretching and invalid configurations, which can be resource intensive and introduce temporal discontinuities.
- Velocity prediction: Velocity prediction can aid statistical learning through a narrower value range, yet long-term accumulation errors and noisy invalid poses can make models unstable.
- Stochastic motion: Long-term generation is challenging because human motion is stochastic, so distant recorded frames represent only a small fraction of the future probability mass.
- Autoregressive learning: Autoregressive training with teacher forcing creates exposure bias by hiding the model’s own errors, while always feeding predictions can make long-sequence training slow.
- Hierarchical generation: Hierarchical locomotion methods simplify generation by separating trajectory, footsteps, and pose synthesis, thereby reducing the uncertainty modeled by the final network.
3 QuaterNet
QuaterNet uses quaternion-based recurrent modeling and differentiable forward-kinematics loss for short-term prediction and controllable long-term locomotion generation.
- Model Architecture & Training Algorithm: The model uses a two-layer autoregressive GRU that predicts future skeleton states from quaternion-encoded joint rotations and optional controls.Each GRU layer has 1,000 hidden units; the architecture supports both short-term prediction and long-term generation.
- Model Architecture & Training Algorithm: A curriculum progressively exposes the network to its own predictions, decaying teacher forcing with β = 0.995 per epoch.This schedule is reported to improve long-term error and model stability.
- Rotation Parameterization and Forward Kinematics Loss: Quaternions avoid the discontinuities and singularities associated with Euler angles and exponential maps, although q and −q remain equivalent representations.The paper identifies quaternion-based human motion prediction as a novel contribution.
- Rotation Parameterization and Forward Kinematics Loss: The loss applies differentiable forward kinematics to predicted rotations and penalizes Euclidean joint-position errors rather than directly averaging rotation errors.This accounts for how errors at different joints affect absolute body positions while retaining skeleton structure.
- Experiments: Short-term prediction improves state-of-the-art angle errors on Human3.6m, while long-term locomotion matches the judged quality of recent graphics methods and supports online generation.The long-term system also provides greater control over timing and artist-imposed trajectory constraints.
- Long-Term Generation: For long-term generation, the system defines trajectory parameters before predicting poses, supporting artist control over speed, facing direction, footsteps, timing, and trajectory constraints.A pace network can fit trajectory features automatically, while separate online and offline variants address real-time or future-aware operation.
4 Experiments
The experiments evaluate QuaterNet on short-term Human3.6m prediction and long-term locomotion generation, including model representations, training strategies, losses, and human judgments. QuaterNet achieves the best reported short-term results, while long-term generation matches prior quality and provides online trajectory and timing control.
- Short-Term Prediction: The short-term evaluation uses Human3.6m motion capture from seven actors performing 15 actions, with 32-joint skeletons recorded at 50 Hz and evaluated by Euler-angle distance.The model observes 50 frames and predicts 10 frames, corresponding to 2 seconds of context and a 400 ms horizon.
- Short-Term Prediction: QuaterNet achieves the best results for all compared actions and time horizons, with velocity modeling generally outperforming absolute rotations for short-term prediction.The comparison includes walking, eating, smoking, and discussions, alongside previous neural approaches and simple baselines.
- Longer-Horizon Prediction: For longer horizons, integrating velocities accumulates errors, whereas absolute rotations are advantageous; scheduled sampling makes the velocity model more stable.With regular feedback, the velocity model exhibits catastrophic drift, while motion becomes mostly stochastic after approximately one second.
- Long-Term Generation: In long-term locomotion training, positional loss achieves lower position errors and faster convergence than angle loss, while angle loss can produce exploding gradients.The long-term experiment conditions on 60 frames and predicts 30 frames; subsequent experiments therefore use positional loss.
- Long-Term Generation: A human study compares eight approximately 15-second walking clips against a phase-functioned neural network, with judgments from 20 assessors per clip.The study uses side-by-side comparisons with randomly ordered methods and workers holding master status.
- Long-Term Generation: QuaterNet generates locomotion online, follows the specified trajectory, and permits fine control over timing at waypoints, unlike methods requiring offline post-processing or lacking timing and space constraints.The examples include walking and running, while the pace network handles curvature and target speed.
Conclusions
QuaterNet combines quaternion-based recurrent modeling with a forward-kinematics position loss, improving short-term prediction and matching prior locomotion-generation quality while adding real-time and constraint control.
- Conclusions: QuaterNet uses quaternions for recurrent-network rotation parameterization and a forward-kinematics position loss on a constrained skeleton.The loss preserves skeleton constraints while weighting joint position errors appropriately.
- Conclusions: The model improves short-term prediction over the Human3.6M state of the art.The experiments report improved angle prediction errors on the Human3.6m benchmark.
- Conclusions: For long-term locomotion generation, QuaterNet qualitatively matches recent computer-graphics work.The system also supports on-line generation and better control over artist-imposed timing and trajectory constraints.
- Conclusions: QuaterNet generation runs in real time and provides improved control over time and space constraints.These controls include the timings and trajectory constraints imposed by the artist.
- Conclusions: The authors recommend reporting position errors or representation-independent dot-product angle distances for clearer comparisons.They also recommend reporting errors after reprojecting directly regressed joint positions onto a valid skeleton.