A Survey on Deep Learning for Skeleton-Based Human Animation

1. Introduction

• Human motion varies with mood, intentions, activity, and individual characteristics, while incomplete biomechanical understanding and production trade-offs complicate animation.Animators balance production costs, realism, manual work, and required expertise.Source: §1 · p. 1
• The survey reviews deep learning and deep reinforcement learning methods for humanoid skeletal character animation, including best practices for motion data and approaches to major animation problems.It focuses on state-of-the-art methods in the field.Source: §1 · pp. 1–2
• Deep learning suits compact motion models for synthesis and editing, whereas deep reinforcement learning suits character control by treating actuation parameters as agent actions.The review describes reinforcement learning as selecting actions in a simulated environment to maximize cumulative reward.Source: §1 · p. 2
• Earlier reviews did not cover deep learning or deep reinforcement learning methods, while a later machine-learning review included only some deep-learning advances.The survey positions itself as a review of the state of the art in deep learning and deep reinforcement learning for skeletal human character animation.Source: §1 · p. 2
• The review covers pose representations, motion datasets, and methods for learning spatial and temporal features in motion data.These topics are addressed before the survey turns to animation applications.Source: §1 · p. 2
• Motion synthesis is organized into short-term prediction, long-term prediction, and generative methods, while character control is grouped into kinematic, physical, and biomechanical approaches.Motion editing covers cleaning, retargeting, and style transfer; cleaning is distinguished from prediction and in-betweening.Source: §1 · p. 2

2. Human Motion Representation, Data and Modelling

• Pose representation shapes neural-network input and output spaces, affecting learning effectiveness and which aspects of human motion are retained.The section reviews common pose representations and their strengths and weaknesses.Source: §2 · p. 2
• The section considers commonly used human-motion datasets alongside pose representations as foundational choices for deep animation.It also examines neural-network architectures structured around spatial and temporal aspects of motion.Source: §2 · p. 2
• The modelling overview addresses neural architectures for learning spatial and temporal domains of motion data.These are treated as distinct aspects of the computational workflow.Source: §2 · p. 2

2.1. Pose Representation

• A skeletal pose sequence represents linked bones and joints over time; angular representations keep bone lengths fixed, while positional representations encode joint coordinates without that constraint.The choice affects the information available to a model and how motion can be optimized or used for skinning and rigging.Source: §2.1 · p. 3

2.1.1. Positional Pose Representations

• Cartesian joint positions support interpolation and optimization, but do not encode bone orientation and require additional constraints to preserve bone lengths.Positional representations can separate limb motion relative to the body from global body movement relative to the environment.Source: §2.1.1 · p. 3

2.1.2. Angular Pose Representations

• Euler angles can lose a degree of freedom at gimbal lock, while exponential-map coordinates are singular at rotations of 2kπ and can be stressed by large rotations.Quaternions avoid singularities but require unit-length constraints; continuous 6D rotation representations avoid discontinuities, though projection-based reduction can make learning harder.Source: §2.1.2 · pp. 3–4
• Angular representations preserve bone lengths and hierarchical joint control, but small orientation errors can accumulate into large joint-position errors along the chain.This makes end-effectors difficult to handle accurately, especially in fast or ample motions.Source: §2.1.2 · pp. 3–4, Figure 2 · p. 4
• Forward kinematics lets networks predict angular poses while training against joint-position errors; an alternative weights joint-angle errors by their downstream effect.The survey notes that the weighting approach was not compared with the FK-based method and that its weight choice was not assessed.Source: §2.1.2 · pp. 4–5

2.1.3. Hybrid Representations

• Hybrid representations combine static positional bone offsets with time-varying joint rotations, processing morphology and motion in separate branches.Some approaches penalize both positional errors after forward kinematics and quaternion errors; other models also add positions or velocities as redundant pose information.Source: §2.1.3 · p. 5

2.2. Human Motion Datasets

• Only a small number of human-motion datasets are repeatedly used, and fewer still serve as standard benchmarks.The survey summarizes datasets most commonly used in the reviewed works.Source: §2.2 · p. 5
• CMU and Human3.6M are the two most widely used large-scale datasets; they are comparable in size, while Human3.6M includes synchronized RGB+D video and pose sequences.That additional modality can make Human3.6M more suitable for computer-vision-oriented tasks such as motion prediction; both datasets are also used together in some evaluations.Source: §2.2 · p. 5
• Dataset coverage varies: HDM05 contains 100 specific sequences repeated by five actors, while NTU RGB+D 120 has 120 action classes but represents motion only through joint positions.NTU RGB+D includes synchronized RGB, infrared, depth, and motion sequences; the position-only representation limits its use in skeleton-based human animation.Source: §2.2 · pp. 5–6
• Most motion-capture datasets have few subjects with minor morphological differences, limiting their diversity for retargeting across characters.Mixamo is often used for retargeting because it provides animation sequences performed by 3D character models with varied morphologies.Source: §2.2 · p. 6
• AMASS unifies 15 optical marker-based motion-capture datasets, and its sequences use SMPL parameters to compute skeletons, joint orientations, and body meshes.AMASS initially contained around 42 hours of data and is still increasing in size.Source: §2.2 · p. 6

2.3.1. Spatially-Structured Architectures

• Spatially structured networks divide skeletons into body parts and process them in parallel branches or hierarchically to encode skeletal characteristics.Parallel branches may specialize by body part, while hierarchical methods recursively merge or predict parts from the root toward the end effectors.Source: §2.3.1 · pp. 6–7

2.3.2. Convolutional Neural Networks

• Flattening skeletons for CNNs complicates spatiotemporal learning because joint relationships do not fit a single dimension; enlarged or deformable kernels address spatial correlations.Graph convolutions offer an alternative for irregular skeletal structures.Source: §2.3.2 · p. 7

2.3.3. Graph Convolutional Networks

• GCNs apply spatial neighborhood convolutions or spectral operations based on the graph Laplacian to model skeletal data [BBL*17; SNF*13].Spatial GCNs can be topology-limited; Aberman et al. [ALL*20] use skeletal pooling and unpooling to process topologies homeomorphic to a common primal skeleton.Source: §2.3.3 · p. 7
• Spectral GCN approaches learn adjacency weights, including fully connected, separated connected-and-global, or multiscale graph structures [MLSL19; CSY20; LCZ*20; LCC*21].The multiscale methods either process scales in parallel or fuse them within each graph convolutional block.Source: §2.3.3 · p. 7

2.4.1. Recurrent Neural Networks

• RNNs process sequences timestep by timestep while maintaining temporal context, and human-motion applications commonly use LSTMs or GRUs.LSTMs use a memory cell and gates to regulate information flow and address vanishing or exploding gradients; GRUs use fewer gates and parameters, while LSTMs can better handle longer sequences.Source: §2.4.1 · pp. 7–8
• Bidirectional LSTMs process past and future context, and have been used for motion embeddings, sequence construction, and motion refinement [WMR*17; XXN*20; LZZ*19; LZZL20].Other extensions and LSTM-GRU combinations have also been used to model human dynamics [TMLZ18; WAC*19].Source: §2.4.1 · p. 8

2.4.2. Temporal Convolutions

• Temporal CNNs capture nearby and distant frame dependencies efficiently in parallel, but fixed-length architectural elements can limit variable-length input.They are comparatively rare in synthesis, prediction, and control, but more common in editing tasks that suit fixed-length sequences.Source: §2.4.2 · p. 8

2.4.3. Miscellaneous

• Other temporal methods include phase-conditioned or expert-blended network weights and frequency-domain learning using DCT and inverse DCT transforms [HKS17; ZSKS18; SZKS19; SZKZ20; LZCvdP20; MLSL19; MLS20; CHW*20].The phase-based approaches described here concern kinematic character control, while the DCT approaches concern motion prediction.Source: §2.4.3 · p. 8

3. Motion Synthesis

• Short-term prediction deterministically extrapolates motion for up to one or two seconds from a past clip, whereas long-term prediction seeks plausible, diverse continuations beyond that horizon.Ground-truth reproduction becomes infeasible over longer horizons because human motion is stochastic [FLFM15; PGA18; ZLX*18].Source: §3 · p. 8
• Motion generation synthesizes pose sequences from non-historical parameters that match a reference dataset's distribution while capturing its variation.The survey excludes application-specific synthesis such as speech-driven gestures and music-driven dance.Source: §3 · p. 8
• The ERD architecture uses an RNN to capture motion dynamics in a latent space, with feedforward encoder and decoder networks mapping skeletal poses to and from that representation [FLFM15].The architecture has been used in short- and long-term motion prediction in original and modified forms.Source: Figure 5 · p. 9

3.1. Short-term Prediction

• Short-term prediction forecasts future poses from an observed temporal window, and 3D end-effector positions can provide richer information than trajectories.The survey identifies applications including robotic navigation in crowds, human-robot interaction, video surveillance, virtual reality, and cloud gaming.Source: §3.1 · p. 9

3.1.1. Classes of methods

• Other prediction frameworks include direct feedforward networks, GCNs, and GANs; GCNs encode skeletal graphs and can learn spatial and temporal joint dependencies [MLSL19; MLS20; LCZ*20; LCC*21; CSY20].Direct feedforward methods compute predictions from the observed poses, while some approaches combine body-part and temporal networks [BBKK17] [GC19].Source: §3.1.1 · pp. 9–10
• Seq2Seq methods commonly train an RNN in a learned latent space, but can converge to mean poses and must manage recovery from their own prediction errors.The ERD architecture trains an LSTM in an autoencoder's hidden layer [FLFM15]; later methods incorporate skeletal structure, residual connections, adversarial losses, or alternative pose representations [MBR17] [PGA18] [PFAG20].Source: §3.1.1 · pp. 9–10

3.1.2. Spatial and Temporal Dependencies

• Methods capture spatial and temporal dependencies using body-structure graphs, multiscale temporal encodings, long- and short-range representations, attention, GCNs, and transformers [JZSS16; BBKK17; LZLL18; CSK*21; MLSL19; CSY20].The survey notes that modeling motion at multiple scales can help short-term prediction.Source: §3.1.2 · p. 11

3.1.3. Plausibility and Realism

• Prediction methods address biomechanical plausibility and realism with fidelity and continuity discriminators, pose-consistency losses, bone-length preservation, and curriculum training [PGA18].The reported improvement from discriminating real sequences is small; progressive switching from teacher-forcing to self-predictions improves error and model stability.Source: §3.1.3 · p. 11

3.1.4. Context, Environment and Interactions

• Scene and social context matter for realistic prediction, but limited databases and varied situations make these settings especially challenging.Proposed approaches incorporate object-human graph interactions or extract scene and social context from monocular video [CPAM20; AAR*20].Source: §3.1.4 · p. 11

3.2. Long-term Prediction

• Long-term prediction targets diverse plausible continuations of a past clip or interpolations between temporally and spatially distant poses.Unlike short-term prediction, its synthesized motion need not reproduce ground truth.Source: §3.2 · p. 11

3.2.1. Forecasting

• ERD forecasting can accumulate recurrent output errors, degrade with prediction distance, converge to a static pose, or diverge from human motion.The review attributes static-pose regression in part to the ill-posedness of long-term prediction, motivating external control signals.Source: §3.2.1 · p. 12
• Forecasting mitigations include GAN discriminators and stochastic-code critics, external action or environment constraints, and training strategies that reduce exposure bias.Other approaches add temporal attention or hierarchical recurrent context to capture longer dependencies.Source: §3.2.1 · p. 12
• Forecasting models also improve motion representation and dynamics with quaternions, se(3) transforms, temporal pose derivatives, and probabilistic GMM transitions.Separate spatial and temporal subnetworks, including a backward predictor, are used to model motion diversity and dependencies at different horizons.Source: §3.2.1 · p. 12

3.2.2. In-betweening

• In-betweening treats motion between distant keyframes as long-term prediction conditioned on past and future contexts.Source: §3.2.2 · pp. 12–13
• In-betweening methods range from efficient end-effector trajectory interpolation to learned long-gap transitions with temporal constraints and artifact mitigation.Approaches include denoising autoencoders with curriculum learning, hierarchical subsegments with style transfer, and an ERD using pose deltas and time-to-arrival to address stalling or teleportation.Source: §3.2.2 · p. 13

3.3. Generative Synthesis

• Generative synthesis processes a random seed through a deep generative model, optionally conditioned on semantic cues such as trajectory or motion style.Holden et al. are an exception, using deterministic high-level cues.Source: §3.3 · p. 13

3.3.1. Restricted Boltzmann Machines

• RBMs model a data distribution with visible and hidden units, then generate samples through alternating conditional sampling initialized from training data.Conditional and factored variants add temporal conditioning or style controls, while hierarchical FCRBMs mix separate style models to address overfitting across sparse style combinations.Source: §3.3.1 · pp. 13–14, Figure 6 · p. 13

3.3.2. Variational Autoencoders

• VAEs constrain encoder latent codes to a prior distribution, enabling stochastic motion generation by decoding prior samples.Temporal models combine VAEs with Markov or recurrent structure; other approaches condition primitive-specific VAEs on motion style or perturb past features with latent components to retain stochastic input.Source: §3.3.2 · p. 14, Figure 7 · p. 14

3.3.3. Generative Adversarial Networks

• GANs train a generator and discriminator jointly so generated motion samples become difficult to distinguish from training data.GAN-based approaches are used to improve plausibility, with generator designs including adversarial autoencoders, spatio-temporal graph convolutions, and separate spatial and temporal networks.Source: §3.3.3 · p. 15, Figure 8 · p. 15

3.3.4. Normalising Flows

• Normalising flows generate motion through invertible transforms, allowing sample densities to be computed in closed form and training by maximum likelihood.Henter et al. [HAB20] condition transform layers on past root-joint trajectory and use an LSTM for temporal dependencies.Source: §3.3.4 · p. 15

3.3.5. Miscellaneous

• Miscellaneous approaches include deterministic locomotion generation, learned sampling for diverse outputs, and mixture-of-experts motion prediction.The sampling strategy balances diversity against fidelity through repulsion and prior-consistency losses; the mixture-of-experts scheme blends specialized networks using frame features.Source: §3.3.5 · p. 15, Figure 9 · p. 17

4.1. Kinematics-based

• Kinematic controllers generate joint-angle motions from examples and high-level controls, but autoregressive models may accumulate errors or respond slowly to interaction.Phase-functioned networks and mixtures of experts address control timing and complex movements; local phases allow asynchronous body-part motion, while related work explores few-shot style adaptation and character interaction.Source: §4.1 · pp. 17–18

4.2. Physics-based

• Physics-based control uses multibody simulation and learned policies, commonly trained by reinforcement learning with state observations, task and physical rewards, and torque or target-angle actions.Training practices include motion-capture imitation, hierarchical controllers, symmetry, curricula, state initialization, and early termination; optimization-based methods use deep learning to accelerate demanding steps.Source: §4.2 · p. 18, Figure 10 · p. 18

4.2.1. Character Locomotion

• Locomotion work combines hierarchical skill controllers, curricula, symmetry objectives, and adaptive body-shape sampling to train policies across tasks and morphologies.A search-based approach also uses simulated action sequences and GAN-guided tree expansion; its generated reference motions and plausible states support DRL training, with authors reporting reduced training time.Source: §4.2.1 · pp. 18–19

4.2.2. Other Motions

• Other-motion controllers learn acrobatics, martial arts, video-derived skills, and styles using imitation or bounded deviations from reference trajectories.Some methods use one controller per skill, while others learn a single controller to execute many skills or reproduce motions from an input controller.Source: §4.2.2 · pp. 19–20

4.2.3. Interactions

• Interaction methods use DRL for object and character tasks, including control fragments, corrected target poses, object manipulation, dressing, and multiagent crowd animation.The crowd approach shares a low-level controller across characters while conditioning each character's high-level controller on nearby agents and objects.Source: §4.2.3 · p. 20

4.3. Biomechanics-based

• Biomechanical controllers use neural networks to learn muscle activations from offline biomechanical simulations, reducing runtime control costs.Full-body models and DRL have been explored, but musculoskeletal realism generally entails complex modeling and costly simulations; a joint-actuation formulation approximates metabolic energy and state-dependent torque limits with DNNs.Source: §4.3 · p. 20

5. Motion Editing

• The survey organizes motion editing into cleaning, retargeting, and style transfer, and summarizes the methods in Table 4.Its miscellaneous-data category includes hand-crafted, synthetic, proprietary, unspecified, or other public datasets.Source: §5 · p. 21, Table 4 · p. 23

5.1. Cleaning

• Cleaning methods use learned motion manifolds to denoise, repair corruption, and fill missing data, with constraints or foot-contact estimates addressing specific artifacts.Constraints can enforce bone lengths and smoothness; predicted foot contacts are often used in inverse kinematics to reduce footskate.Source: §5.1 · pp. 21–22

5.2. Retargeting

• Retargeting transfers a source character’s motion to a different morphology, usually using unpaired data because matched source-target sequences are difficult to obtain.One approach handles different, homeomorphic skeleton topologies by mapping source motion through a common primal skeleton to the target character.Source: §5.2 · p. 22

5.3. Style Transfer

• Style transfer changes a motion’s stylistic qualities while retaining its content, using either learned style measures or annotated style information.Approaches include Gram-matrix measures, registered poses across styles, and separate content and style encodings; adult-child translation addresses limited child-motion data.Source: §5.3 · pp. 23–24

Representations

• No simple pose representation meets demands for precision, optimization, skinning, and rigging; the survey calls for frameworks combining hierarchical orientations with absolute positions.Joint-position losses appear helpful, while the spectral domain remains almost unexplored.Source: §6 · p. 24

Motion prediction

• Autoregressive short-term predictors can diverge or converge to a mean pose, with deterministic prediction becoming more ambiguous over longer horizons.The discussion points to contextual information or sampling from a distribution of possible futures as ways to mitigate ill-posedness.Source: §6 · p. 24

Motion synthesis

• Generative models still do not fully represent the human-motion distribution, and general-purpose models can lack flexibility for hard constraints and animator control.The discussion identifies scarce in-the-wild data and low-resolution skeletons as constraints, and suggests markerless capture, self-supervised learning, and higher-resolution data as directions.Source: §6 · p. 24

Character control

• Physics-based controllers are often physically correct but may lack naturalness, whereas kinematics-based controllers can appear more natural but lack physical correctness.The survey expects approaches combining data-driven generative models with biomechanical or physical models, alongside greater runtime flexibility and interactivity.Source: §6 · pp. 24–25

Motion editing

• Motion editing remains relatively unexplored, and the discussion identifies interpretability, controllability, and reliable automated footskate removal as unresolved challenges.The survey notes that no method has achieved both interpretability and controllability.Source: §6 · p. 25

Evaluation

• Most surveyed studies lack perceptual and user studies, while statistical metrics may not evaluate perceived motion plausibility or naturalness.The discussion calls for more perceptual studies and metrics involving perceptual cues.Source: §6 · p. 25

• DL means Deep Learning, DRL means Deep Reinforcement Learning, and DNN means Deep Neural Network.Source: Acronyms · p. 25
• CNN means Convolutional Neural Network, GCN means Graph Convolutional Network, and RNN means Recurrent Neural Network.Source: Acronyms · p. 25
• GAN means Generative Adversarial Network, VAE means Variational Autoencoder, and IK means Inverse Kinematics.Source: Acronyms · p. 25