PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Human Motion with Quaternion-based Neural Networks

Dario Pavllo, Christoph Feichtenhofer, Michael Auli, David Grangier

arXiv:1901.07677v2cs.CVcs.AIcs.RO

TL;DR

Human-motion prediction must handle constrained skeletons, unstable rotation representations, and uncertainty over long horizons. The paper introduces quaternion-based QuaterNet with differentiable forward-kinematics position loss and recurrent or convolutional architectures. It reports strong short- and long-term performance, finds that short context is sufficient, and identifies a more stable Human3.6M evaluation protocol.

  • Problem

    Existing methods trade off skeleton constraints against position-based error weighting, while Euler angles and exponential maps can be discontinuous and long-term motion is uncertain.

  • Method

    QuaterNet represents joint rotations with quaternions and trains them using differentiable forward kinematics to penalize absolute joint-position errors.

  • Results

    QuaterNet improves short-term prediction on Human3.6M and qualitatively matches recent computer-graphics methods for long-term locomotion generation.

  • Takeaways & Limitations

    A relatively short context is sufficient for reliable future predictions, and quaternion parameterization supports a unified approach to short-term prediction and long-term generation.

  • Takeaways & Limitations

    The study uses step-wise regression and scheduled sampling; combining quaternion-parameterized kinematics with adversarial training remains future work.

Abstract

from arXiv · show

Previous work on predicting or generating 3D human pose sequences regresses either joint rotations or joint positions. The former strategy is prone to error accumulation along the kinematic chain, as well as discontinuities when using Euler angles or exponential maps as parameterizations. The latter requires re-projection onto skeleton constraints to avoid bone stretching and invalid configurations. This work addresses both limitations. QuaterNet represents rotations with quaternions and our loss function performs forward kinematics on a skeleton to penalize absolute position errors instead of angle errors. We investigate both recurrent and convolutional architectures and evaluate on short-term prediction and long-term generation. For the latter, our approach is qualitatively judged as realistic as recent neural strategies from the graphics literature. Our experiments compare quaternions to Euler angles as well as exponential maps and show that only a very short context is required to make reliable future predictions. Finally, we show that the standard evaluation protocol for Human3.6M produces high variance results and we propose a simple solution.

1 Introduction

The paper presents QuaterNet as a unified approach to short-term human-motion prediction and long-term generation, addressing rotation parameterization, position-based training, and evaluation instability.

  • Tasks and evaluation: The method addresses both short-term prediction and long-term generation, evaluated on Human3.6M and a locomotion dataset, respectively.The paper compares its goals with state-of-the-art methods in computer vision and computer graphics.
  • Contributions: QuaterNet uses quaternions in recurrent and convolutional neural networks to avoid discontinuities and singularities associated with Euler angles and exponential maps.These parameterization issues can cause exploding gradients and training difficulty.
  • Contributions: A differentiable forward-kinematics loss penalizes absolute joint-position errors while retaining the constraints of a parameterized skeleton.This combines joint-orientation prediction with position-based weighting of errors.
  • Results: Long-term generation qualitatively matches recent computer-graphics methods while supporting online generation and artist control over timing and trajectory constraints.Short-term performance is slightly outperformed by very recent adversarial-training work.
  • Findings and evaluation: A relatively short context performs as well as longer context for reliable future predictions.The paper also identifies high variance in the standard Human3.6M evaluation protocol and proposes a stabilizing adjustment.

2 Related work

Related work frames human-motion modeling as a choice among representations, architectures, training strategies, and conditioning mechanisms under substantial future uncertainty.

  • Neural architectures: RNNs, convolutional networks, feed-forward networks, and hierarchical locomotion systems have all been applied to short-term prediction or long-term generation.Architectural choice, task framing, input-output variables, loss functions, and conditioning controls are all treated as consequential design factors.
  • Quaternion methods: Quaternion neural-network work has used hyper-complex latent variables for long-term dependencies, whereas this paper uses quaternions to represent rotations along a kinematic chain.The latter motivation follows a classical computer-graphics formulation.
  • Representations: Human motion can be represented through joint positions or rotations integrated with forward kinematics, each offering different benefits and trade-offs.Rotation representations are typical for animating skinned meshes in games and movies.
  • Joint rotations versus positions: Rotation prediction preserves skeleton constraints, whereas position prediction avoids unequal joint weighting but requires reprojection to prevent bone stretching.Errors at trunk and root joints can affect pose quality more than errors at limb endpoints.
  • Prediction targets: Velocity prediction narrows the value range for statistical learning but can become unstable over long horizons through accumulated errors.Noise and invalid training poses can also create large velocity variations.
  • Uncertainty and training: Human motion is highly uncertain, so long-term generation is difficult because a recorded future represents only a small part of the possible probability mass.Autoregressive teacher forcing can create exposure bias, while always feeding predictions can make training slow on long sequences.

3 QuaterNet

QuaterNet models human-motion sequences with quaternion-encoded joint rotations using a shared pose network that supports both short-term prediction and long-term generation.

  • Recurrent architecture: The original QuaterNet uses a two-layer GRU autoregressive model that predicts each pose from the previous recurrent state and previous-pose features.Each GRU layer has 1,000 hidden units, and the initial states are learned from data.
  • Recurrent architecture: The paper found no benefit from LSTMs over GRUs and found an empirical advantage for a second recurrent layer but not a third.GRUs were selected for simplicity and efficiency.
  • Pose network: The architecture takes joint rotations encoded as unit quaternions, plus optional inputs, and predicts future skeleton states across k time steps from n initialization frames.The values of k and n depend on the task.

3.2 Convolutional architecture

The convolutional QuaterNet replaces the recurrent backbone with causal temporal convolutions, using dilation and skip connections to model motion sequences efficiently.

  • Motivation: Convolutions offer parallelization across batch and time dimensions and a constant input-output path length compared with recurrent models.These properties simplify training and reduce dependence on sequential processing.
  • Architecture: The convolutional architecture adapts the RNN-based model by replacing its GRU and linear backbone with a sequence of convolutional layers.The comparison tests whether convolutional architectures benefit human-motion modeling.
  • Architecture: The model uses causal filters with width W = 2, exponentially increasing dilation D = 2^k across five layers, and skip connections between layers.The dilation strategy forms a tree-like information path in which inputs are processed once per layer.
  • Ablation: Dilated convolutions consistently outperform standard dense convolutions in the reported ablation.The paper suggests their sparsity may improve generalization.

3.3 Training details

QuaterNet uses scheduled sampling to expose models progressively to their own predictions for long-term generation, while comparing recurrent and convolutional backbones.

  • Training procedure: Scheduled sampling progressively replaces ground-truth inputs with the model’s own predictions during long-term motion generation.Training begins with teacher forcing, p = 1, and decays p exponentially by β = 0.995 per epoch.
  • Training procedure: For recurrent models, self-generated inputs produce gradients that account for both current-target accuracy and effects on future predictions.The convolutional architecture receives gradient only through the current prediction term.
  • Architectures: Figure 1’s QMul option makes the model output rotational velocities, whereas bypassing QMul makes it emit absolute rotations.The center block is the recurrent backbone.
  • Architectures: The convolutional backbone uses skip-connections and, in the depicted model, a receptive field of 16 frames.The figure specifies W = 1 in the last layer.

3.4 Parameterization of forward kinematics

The paper represents rotations with normalized quaternions and trains using differentiable forward kinematics to penalize joint-position errors. It resolves temporal sign ambiguity by selecting the representation closest to the preceding frame, while noting that forcing a unique sign reintroduces discontinuities.

  • Rotation representations: Euler angles have non-uniqueness, discontinuities, and singularities such as gimbal lock.These issues motivate using quaternions for joint-rotation representation.
  • Quaternion parameterization: Quaternions must have unit length to represent valid rotations, so the network includes explicit normalization and a norm-penalty regularizer.The penalty is λ(w^2 + x^2 + y^2 + z^2 −1)^2, with λ = 0.01 used in training.
  • Quaternion parameterization: The model chooses the quaternion representation with the lowest Euclidean distance from the previous frame to reduce temporal discontinuities.This leaves two sign-inverted representations, q and −q, which the authors report do not hinder autoregressive models.
  • Contribution: The work introduces quaternion-based human-motion prediction while using quaternions to represent joint rotations along a kinematic chain.The authors distinguish this use from prior quaternion applications such as pose clustering, joint-limit estimation, and motion retargeting.
  • Forward-kinematics loss: The proposed positional loss applies forward kinematics to predicted joint rotations and computes Euclidean distances between predicted and reference joint positions.Because forward kinematics is differentiable with respect to rotations, the resulting position error can train the network.

3.5 Parameterization of rotations

The paper compares 3D rotation parameterizations and favors unit quaternions for avoiding singularities and discontinuities while supporting direct rotation composition.

  • Euler angles: Euler angles use axis rotations but have multiple equivalent representations, discontinuities at 2π, and singularities including gimbal lock.Their ordering convention must also be fixed when composing rotations.
  • Axis-angle representation: Exponential maps reduce the likelihood of Euler-angle problems but retain fundamental singularities, discontinuities, and multiple representations.They also lack direct rotation composition, which is fundamental for forward kinematics.
  • Unit quaternions: Unit quaternions encode rotations with four normalized parameters and avoid singularities and discontinuities in the representation space.They support smooth regression and interpolation without periodic functions.
  • Unit quaternions: Quaternions can be composed directly through quaternion multiplication, unlike exponential maps, making them suitable for forward kinematics.Quaternion slerp provides continuous interpolation with constant velocity and minimal torque.
  • Unit quaternions: Quaternions still have antipodal representations: q and −q encode the same 3D orientation.Forcing quaternions into one hemisphere removes the dual representation but introduces discontinuity, so the paper instead addresses it in the data and autoregressive model.
  • Empirical angle distributions: 7% of H3.6M local angles lie in the potentially problematic range outside the safe interval −π/2 to π/2.The figure marks the safe range in orange and the potentially problematic range in blue.

3.6 Short-term prediction

For short-term prediction, the quaternion network can predict relative rotation deltas or absolute rotations, and the experiments replicate an Euler-angle evaluation protocol while correcting angle periodicity.

  • Prediction targets: The quaternion network predicts either relative rotation deltas or absolute rotations for short-term pose forecasting.Relative deltas are applied to input quaternions through quaternion multiplication, analogous to residual angle prediction.
  • Evaluation: The replicated evaluation protocol maps quaternions to Euler angles and computes L1 distance to reference angles using the best match modulo 2π.This handles angle periodicity explicitly for comparability with prior work.

3.7 Long-term generation

Long-term generation is framed as producing locomotion poses from a target trajectory and average speed, using trajectory features followed by an autoregressive quaternion pose network.

  • Task definition: The locomotion task generates a pose sequence conditioned on average speed and a ground trajectory to follow.This task is common in computer graphics.
  • Pipeline: The pipeline first defines trajectory parameters, then predicts the pose sequence with an autoregressive quaternion network.Trajectory parameters include facing direction, local speed, and footstep frequency.
  • Trajectory conditioning: A pace network can automatically fit trajectory features, while artists can alternatively define them manually.The pace network predicts facing direction relative to the spline tangent, footstep frequency, and local speed.
  • Trajectory conditioning: The pace network has bidirectional offline and delayed one-directional real-time variants.The real-time variant avoids observing the trajectory far into the future.
  • Pose network: The pose network adds translation and control inputs, including spline tangent, facing direction, longitudinal speed, and walk-cycle features.Speed and walk cycle are represented as A[cos(θ), sin(θ)] to encode a cyclic signal.
  • Pose network: High-frequency movement details are modeled by comparing spline position with the position obtained from integrating average speed, helping realism and reducing foot sliding.The feature is derived from low-pass- and high-pass-filtered trajectory speed.

4 Experiments

The experiments evaluate QuaterNet on short-term Human3.6M prediction and long-term generation, while exposing weaknesses in teacher forcing and the standard four-sample evaluation protocol.

  • Short-term evaluation uses Human3.6M motion capture from seven actors performing 15 actions, with 32-joint skeletons down-sampled to 25 Hz.
  • Teacher forcing slightly lowers recurrent error at 80 ms, but scheduled sampling performs better over longer horizons by exposing the model to its own predictions.Scheduled sampling is much less effective for the convolutional model.
  • For long-term prediction, integrating velocities accumulates error, whereas absolute rotations are advantageous; scheduled sampling stabilizes recurrent velocity models.Regular feedback causes catastrophic drift over time, while absolute rotations show small first-frame discontinuities.
  • The standard protocol samples only four random chunks per test sequence, making test results highly variable despite improving computational efficiency and cross-paper comparability.The proposed protocol increases sampling to 128 chunks per sequence.
  • The proposed evaluation compares errors across all 15 actions and their average using 128 samples per sequence, rather than the four samples used in the standard protocol.

1 Reference

Long-term generation experiments show that forward-kinematics positional loss, quaternion rotations, and online trajectory conditioning improve practical motion quality and stability across the tested comparisons.

  • 4.3 Long-term generation: Optimizing forward-kinematics positional loss achieves lower position errors and faster convergence than angle loss while avoiding the latter’s exploding gradients.The model is conditioned on 60 frames and predicts 30 frames for this comparison.
  • 4.3 Long-term generation: A human study found QuaterNet performed similarly to Holden et al. (2017) on walking clips, without postprocessing.The study used eight approximately 15-second clips, 20 assessors per clip, and randomized method ordering.
  • 4.3 Long-term generation: Online generation follows the instructed trajectory while providing fine control over passage time at waypoints, unlike the offline constraints of Holden et al. (2016).Holden et al. (2017) is online but does not support time or space constraints.
  • 4.4 Ablations: For conditioning length, prediction error saturates after 10–20 frames, or 400–800 ms, for both recurrent and convolutional models.Large receptive fields are not necessarily best for CNNs with absolute rotations.
  • 4.4.2 Parameterizations: Quaternions achieve the lowest error and fastest convergence, while also producing smoother predictions than axis-angle and Euler-angle alternatives.Euler-angle performance depends on rotation order, with yzx producing discontinuities and xyz closer to axis-angle on position error but worse on velocity error.
  • 4.4.3 Positions versus rotations: Quaternion outputs and direct joint-position outputs have similar position loss, but re-projection increases velocity error by introducing discontinuities.The quaternion model is slightly worse after 40 frames, likely because its loss is more complex.

5 Conclusion and future work

The paper concludes that QuaterNet unifies quaternion-based rotation modeling with forward-kinematics positional loss for short-term prediction and long-term locomotion generation, while identifying evaluation and training extensions for future work.

  • QuaterNet combines quaternion rotation parameterization with a forward-kinematics position loss in recurrent and convolutional neural networks.
  • Experiments report improved short-term prediction on Human3.6M and locomotion quality comparable to recent computer-graphics methods.
  • The generated motion is real-time and supports finer control of time and space constraints.
  • The standard Human3.6M evaluation protocol produces high-variance results, motivating the proposed higher-sample solution.
  • Future work includes extending QuaterNet to action recognition, video-based pose estimation, quaternion-domain neural networks, and adversarial training.
Loading 1901.07677v2…