PaperlayerView exact revision on arXiv

Source-linked AI summary

Robust Motion In-betweening

Félix G. Harvey, Mike Yurick, Derek Nowrouzezahrai, Christopher Pal

arXiv:2102.04942v1cs.CVcs.GRcs.LG

TL;DR

The paper addresses the difficulty of generating robust, variable-length transitions between sparse animation keyframes using conventional target-conditioned motion predictors. It introduces adversarial recurrent architecture modifiers and demonstrates production use through a MotionBuilder plugin, while proposing benchmarks and the LaFAN1 dataset for transition generation.

  • Problem

    Target-conditioned motion predictors handle transitions of different lengths poorly and remain deterministic, limiting their use as robust transition generators.

  • Method

    The method adds time-to-arrival embeddings and scheduled target noise to latent recurrent representations within an adversarial motion-generation architecture.

  • Results

    The resulting system generates smooth, on-time motions of variable length between sparse keyframes and supports robust automatic in-betweening in animation software.

  • Takeaways & Limitations

    The system lets animators quickly generate quality motion between sparse keyframes and introduces LaFAN1 for transition-generation benchmarking.

  • Takeaways & Limitations

    Longer sequences can change the initial time-to-arrival embedding substantially, potentially destabilizing recurrent hidden states and hurting early-transition performance.

Abstract

from arXiv · show

In this work we present a novel, robust transition generation technique that can serve as a new tool for 3D animators, based on adversarial recurrent neural networks. The system synthesizes high-quality motions that use temporally-sparse keyframes as animation constraints. This is reminiscent of the job of in-betweening in traditional animation pipelines, in which an animator draws motion frames between provided keyframes. We first show that a state-of-the-art motion prediction model cannot be easily converted into a robust transition generator when only adding conditioning information about future keyframes. To solve this problem, we then propose two novel additive embedding modifiers that are applied at each timestep to latent representations encoded inside the network's architecture. One modifier is a time-to-arrival embedding that allows variations of the transition length with a single model. The other is a scheduled target noise vector that allows the system to be robust to target distortions and to sample different transitions given fixed keyframes. To qualitatively evaluate our method, we present a custom MotionBuilder plugin that uses our trained model to perform in-betweening in production scenarios. To quantitatively evaluate performance on transitions and generalizations to longer time horizons, we present well-defined in-betweening benchmarks on a subset of the widely used Human3.6M dataset and on LaFAN1, a novel high quality motion capture dataset that is more appropriate for transition generation. We are releasing this new dataset along with this work, with accompanying code for reproducing our baseline results.

1 INTRODUCTION

The paper develops robust in-betweening for sparse animation keyframes because conventional motion prediction and manual authoring struggle with long, stochastic human motion. It introduces temporally aware and stochastic recurrent generation, evaluates it in production and benchmarks, and contributes LaFAN1 with reproducible code.

  • Human motion is complex and stochastic over long horizons, while manually authoring many animation sequences or using motion capture is time-consuming.
  • Simply conditioning a state-of-the-art motion predictor on a future keyframe poorly handles varying transition lengths and produces deterministic outputs.
  • A time-to-arrival embedding and scheduled target-noise vector modify latent representations to support variable-length, robust, and stochastic transitions.The temporal embedding represents remaining timesteps, while scheduled noise distorts target embeddings early and decreases during synthesis.
  • The resulting adversarial architecture generates smooth cyclic or acyclic motion of variable length between sparse keyframes without smoothing post-processing.It uses 10 frames of past context and one target keyframe, filling gaps under a soft upper-bound on duration.
  • The system is evaluated through a MotionBuilder plugin and in-betweening benchmarks on Human3.6M and LaFAN1, including global displacement and longer-horizon generalization.The work releases LaFAN1 and accompanying code for reproducing baseline results.

2 RELATED WORK

Prior work addresses animation control, constrained synthesis, and motion prediction through search, optimization, neural networks, and reinforcement learning. This paper positions transition generation as sparse-keyframe control and uses recurrent motion prediction to fill unconstrained gaps.

  • 2.1 Motion Control: Motion-control methods use motion graphs, search, motion matching, or runtime learning to generate animation from control signals and motion databases.These approaches map character states or motion segments to traversable structures, or match poses and trajectories to stored animation.
  • 2.2 Constrained Motion Synthesis: MAP-based methods combine motion priors with constraints such as keyframes, using statistical dynamics, deformable models, or other probabilistic motion models.
  • 2.1 Motion Control: Deep neural networks address runtime constraints by training on large heterogeneous datasets while maintaining a fixed computation budget during use.Related systems support controls such as root trajectories, end-effectors, phase, modes, or actions.
  • 2.1 Motion Control: Reinforcement-learning approaches formulate motion control as Markov decision processes, but several require motion data at runtime or produce limited human-like movement.
  • 2.3 Transition generation: The proposed recurrent architecture builds on long-term motion prediction and feed-forward encoders to apply embedding modifiers to distinct input components.
  • 2.3 Transition generation: The paper defines transition generation as filling large motion gaps under temporally sparse spatial constraints, extending interpolation to whole motion cycles.Earlier probabilistic approaches often use models specific to particular actions and actors, which can make combined actions appear scripted.

3 METHODS

The transition generator uses recurrent motion synthesis with separate encoders, latent time-to-arrival and target-noise modifiers, and kinematic processing to produce constrained character motion. Time-to-arrival embeddings provide awareness of variable transition lengths, while the architecture uses quaternion normalization and Forward Kinematics to recover motion outputs.

  • Transition Generator: The recurrent generator augments encoded representations with ztta and ztarget before feeding them to an LSTM and decoder.The decoder predicts local quaternions and root velocities, with optional contact predictions for LaFAN1.
  • Transition Generator: Predicted quaternions are normalized and combined with root position and bone offsets through Forward Kinematics to retrieve global positions.Global positions are also used for positional losses and transition evaluation.
  • Time-to-arrival embeddings: Time-to-arrival embeddings encode the number of timesteps remaining before the target, giving the recurrent layer continuous temporal awareness for variable-length transitions.The embeddings are added to latent input representations and use sinusoidal functions based on tta.
  • Time-to-arrival embeddings: The time-to-arrival embedding is bounded, smooth, and dense, while its additive form makes it harder for the network to ignore than concatenated conditioning.For longer transitions, the embedding is held constant beyond a defined maximum duration to reduce instability at the transition start.
  • Time-to-arrival embeddings: Simpler scalar time-to-arrival inputs are evaluated against the temporally unaware model and the proposed ztta embedding on Human 3.6M walking transitions.The figure reports that TG-Q + ztta yields the best results, especially near transition ends.

3.4 Scheduled target noise

The scheduled target-noise modifier improves robustness to target modifications and produces diverse transitions for fixed keyframes. Noise is sampled once per sequence, scaled by a controllable standard deviation, and scheduled to diminish near the target for smooth arrival.

  • Scheduled target noise: The method adds target noise to encoded target and offset representations to improve robustness to keyframe modifications and enforce diversity for fixed context.The noise vector is sampled once per sequence from a zero-centered spherical Gaussian distribution.
  • Scheduled target noise: The target-noise standard deviation σtarget controls the level of accepted target distortion.The noise is added after concatenating the encoded offset and target keyframe representations.
  • Scheduled target noise: A target-noise multiplier scales ztarget down as the number of remaining timesteps decreases, supporting smooth transitions to the target.Beyond 30 frames before the target, the target noise remains constant with λtarget = 1.
  • Scheduled target noise: Because the modifier also distorts time-to-arrival information, it can modify the pace of generated motion while preserving plausible transitions.The authors report greater transition diversity than a common concatenated-noise approach.
  • Scheduled target noise: In a 100-frame transition resampled 10 times, ztarget produces noticeable variations with controllable scale, whereas zconcat is learned to be ignored.The comparison is made midway through the transition.

3.5 Motion Discriminators

The method combines reconstruction, adversarial, and curriculum-learning components to stabilize training and improve motion realism across variable transition lengths. Long- and short-term critics evaluate sliding motion windows, while training progressively expands the sampled transition horizon.

  • Motion Discriminators: Two discriminators provide complementary adversarial supervision: C1 evaluates 10-frame windows and C2 evaluates 2-frame windows.Both critics use three fully connected layers, with 512 and 256 units in their hidden layers.
  • Losses: The loss design combines multiple soft constraints to stabilize training and obtain realistic motion outputs.Reconstruction losses are computed with the L1 norm.
  • Reconstruction Losses: The main reconstruction terms are local-quaternion and root-position losses, supplemented by global joint-position loss computed through Forward Kinematics.A contact loss is also defined for contact predictions.
  • Adversarial Losses: The LSGAN formulation trains the generator to make discriminator scores approach 1 while critics distinguish generated from real motion.The discriminator losses are averaged over time because critics operate on sliding windows.
  • Training: Curriculum learning begins with short transitions and progressively increases the current maximum transition length during training.Each minibatch samples transition lengths uniformly between the minimum and current maximum.
  • Training: At evaluation time, the transition length is fixed to the desired length, while curriculum training accelerates the beginning of training and empirically outperforms gradient clipping.The authors report results similar to teacher forcing.

4 EXPERIMENTS AND RESULTS

Experiments establish a competitive unconstrained motion predictor, then evaluate transition generation on Human3.6M and LaFAN1. The proposed modifiers improve longer-horizon robustness and performance, while a MotionBuilder plugin supports practical animation authoring.

  • 4.1 Motion prediction: ERD-QV is competitive with state-of-the-art angular errors and improves NPSS performance on all Human3.6M actions.The model provides the motion-prediction architecture used as the basis for transition generation.
  • 4.2 Walking in-betweens on Human 3.6M: Transition benchmarks report L2Q and L2P distances, with normalized global positions used to assess visual transition quality.The Human3.6M benchmark evaluates models beyond the 50-frame training horizon, up to 100 frames.
  • 4.2 Walking in-betweens on Human 3.6M: The time-to-arrival modifier provides the most significant improvement, while scheduled target noise generally improves longer transitions and enables varied transitions from fixed keyframes.Target noise is disabled for quantitative evaluation but is used qualitatively to control variation.
  • 4.3 Scaling up with the LaFAN1 dataset: LaFAN1 evaluates models trained up to 30 frames on 5, 15, 30, and 45-frame transitions using high-quality motion capture with varied actions and global displacements.The dataset contains 496,672 frames from five subjects, with Subject 5 reserved for testing.
  • 4.3 Scaling up with the LaFAN1 dataset: On LaFAN1, the complete model improves all quantitative measurements and surpasses interpolation even on very short transitions.This differs from the Human3.6M walking subset, where short-transition superiority over interpolation was not observed.
  • 4.4 Practical use inside an animation software: The MotionBuilder plugin applies generated transitions to user-defined keyframes or existing clips, including contact-based IK postprocessing.The plugin uses 170MB of memory and generates transition candidates at nearly no cost compared with manual authoring or database search.

5 DISCUSSION

The discussion finds the additive modifiers effective for time robustness and transition sampling, while dataset composition and architectural choices affect results. It also identifies limitations in positional loss and reports three variation types for transitions.

  • Additive modifiers: The time-to-arrival and scheduled target-noise modifiers were effective for robustness to time variations and enabling sampling capabilities.Concatenation-based alternatives performed worse, possibly because additive shifts in latent space are harder for the network to bypass.
  • Dataset ablations: The full LaFAN1 dataset was necessary to generate style, while dance-only data achieved similar results to the larger dataset without dance sequences.The authors suggest that combining generic and specialized data is important.
  • Positional loss: Lpos can produce correct joint positions without guaranteeing bone orientations, causing noticeable artifacts at the ends of kinematic chains.The positional loss uses forward kinematics but does not fully define character configurations.
  • Architecture: Empirical comparisons favored LSTMs over GRUs, despite GRUs having fewer parameters.The motion predictor used velocity-based inputs, following the representation suggested for TP-RNN.

6 LIMITATIONS AND FUTURE WORK

The method remains constrained by computationally expensive forward-kinematics alternatives and by conditions outside the training distribution. Scheduled target noise provides variation but not controllable style.

  • Representation limits: Using character-local or global positional offsets would require forward kinematics during generation, but its training backward pass is prohibitively slow.The system therefore uses root offset and rotational offsets only.
  • Data-distribution limits: The method struggles with transitions whose conditions are unrealistic or outside the range covered by the training set.This is identified as a limitation of the data-driven approach.
  • Style control: Scheduled target noise changes how a character reaches its target but does not provide control over those motion variations.Style control for a fixed context is left out of scope.

7 CONCLUSION

The conclusion presents robust transition generation from sparse keyframes through two additive modifiers, demonstrates animator use, and introduces LaFAN1 for benchmarking.

  • Contributions: The method combines time-to-arrival and scheduled target-noise modifiers to address transition-length robustness, target-keyframe variation, and sampling.The conclusion frames these modifiers as converting state-of-the-art motion predictors into robust transition generators.
  • Applications: A MotionBuilder plugin allows animators to generate quality motion between sparse keyframes inside animation software.The system was shown in production-oriented animation scenarios.
  • Dataset: LaFAN1 is introduced as a high-quality dataset suited to transition-generation benchmarking.The dataset is presented as part of the paper's benchmarking contribution.

A.1 Sliding critics

The sliding-critics design evaluates transitions at multiple temporal positions, using conditional context at the boundaries and averaging timestep scores.

  • Critic layout: Two timescales critics inspect transition sequences, with blue frames denoting context and green frames denoting generated transitions.The first and last critic positions are shown without transparency.
  • Boundary conditioning: The critics receive ground-truth context conditionally at the beginning and end of transitions.This boundary conditioning is part of the critic setup.
  • Scoring: Scalar scores from each timestep are averaged to obtain the final score.The figure summarizes how per-timestep critic outputs are aggregated.

A.2 Time-to-arrival embedding visualization

The visualization shows time-to-arrival embeddings, with T_max (z_tta) improving generalization to longer transitions by avoiding novel LSTM hidden-state initializations.

  • T_max (z_tta) improves generalization to longer transitions by preventing the LSTM hidden state from being initialized with novel embeddings.
Loading 2102.04942v1…