PaperlayerView exact revision on arXiv

Source-linked AI summary

Single Motion Diffusion

Sigal Raab, Inbal Leibovitch, Guy Tevet, Moab Arar, Amit H. Bermano, Daniel Cohen-Or

arXiv:2302.05905v2cs.CVcs.AIcs.GR

TL;DR

Motion data is scarce, particularly for animals and exotic creatures with unique skeletons. SinMDM learns motion motifs from a single sequence using diffusion and a local-attention network; the authors report better quality and time-space performance than current work, plus motion manipulation at inference time.

  • Problem

    Motion datasets are limited, especially for non-human skeletons and motion patterns that differ from available humanoid data.

    Source: §1 · p. 1

  • Method

    SinMDM uses a diffusion model with a shallow UNet and local QnA attention to learn motifs from a single motion sequence with arbitrary topology.

    Source: §1 · p. 1, §3 · p. 3, §4 · p. 4

  • Results

    SinMDM significantly outperforms current work in quality and time-space performance and supports long motions and several manipulation tasks.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Motion composition, in-betweening, expansion, harmonization, style transfer, and crowd animation are available at inference time without application-specific retraining.

    Source: §1 · p. 2

    Single-instance learning limits out-of-distribution synthesis, and iterative diffusion entails relatively long inference time.

    Source: §7 · p. 10

Abstract

from arXiv · show

Synthesizing realistic animations of humans, animals, and even imaginary creatures, has long been a goal for artists and computer graphics professionals. Compared to the imaging domain, which is rich with large available datasets, the number of data instances for the motion domain is limited, particularly for the animation of animals and exotic creatures (e.g., dragons), which have unique skeletons and motion patterns. In this work, we present a Single Motion Diffusion Model, dubbed SinMDM, a model designed to learn the internal motifs of a single motion sequence with arbitrary topology and synthesize motions of arbitrary length that are faithful to them. We harness the power of diffusion models and present a denoising network explicitly designed for the task of learning from a single input motion. SinMDM is designed to be a lightweight architecture, which avoids overfitting by using a shallow network with local attention layers that narrow the receptive field and encourage motion diversity. SinMDM can be applied in various contexts, including spatial and temporal in-betweening, motion expansion, style transfer, and crowd animation. Our results show that SinMDM outperforms existing methods both in quality and time-space efficiency. Moreover, while current approaches require additional training for different applications, our work facilitates these applications at inference time. Our code and trained models are available at https://sinmdm.github.io/SinMDM-page.

1 INTRODUCTION

SinMDM targets motion synthesis from a single sequence, especially where motion data and non-human examples are scarce. It uses diffusion with a narrow-receptive-field architecture and supports multiple motion-manipulation tasks at inference time.

1 INTRODUCTION

  • Motion data is scarce, and available datasets largely contain humanoids with fixed topology and bone ratios, limiting common data-driven methods for customized skeletons.The text identifies animal and magical-creature characters as examples of customized skeletons.Source: §1 · p. 1
  • SinMDM learns from a single motion input and synthesizes variable-length motions that retain its core elements across arbitrary skeletal topologies.The paper gives a flying dragon and hopping ostrich as examples of single-sequence inputs.Source: §1 · p. 1
  • The authors argue diffusion suits single-input learning because gradual denoising provides descriptive ability in a lightweight model with fewer parameters and shorter training time than prior art.They also state that diffusion can work with limited data.Source: §1 · p. 1
  • SinMDM uses a shallow motion-adapted UNet with QnA local attention instead of global attention, addressing the overfitting concern posed by whole-motion receptive fields.The paper says narrow receptive fields promote diversity and reduce overfitting.Source: §1 · p. 1
  • At inference time, SinMDM groups motion composition, in-betweening, expansion, harmonization, style transfer, long-sequence generation, and crowd animation without application-specific retraining.The benchmarks are presented as two complementary evaluation settings for single-motion synthesis.Source: §1 · p. 2
  • SinMDM outperforms current works on both proposed benchmarks and is reported to improve quality and time-space performance over prior art.The benchmarks use MIXAMO metrics without an additional feature-extracting model and HumanML3D metrics that include latent features such as single-FID.Source: §1 · p. 2

2 RELATED WORK

Related work situates SinMDM among single-instance learning, diffusion methods, and broader motion-synthesis models. It identifies Ganimator as the immediate single-motion comparison and distinguishes SinMDM's local-attention design from global-attention motion diffusion models.

2.1 Single-Instance Learning

  • Ganimator is the only prior single-motion learning work identified, and SinMDM is reported to outperform it while avoiding its temporal and skeletal hierarchies.The paper describes Ganimator as combining a temporal hierarchy of motions with a skeletal hierarchy of joints.Source: §2.1 · p. 2

2.2 Diffusion Models

  • Motion-domain diffusion work includes MDM, which uses a lightweight transformer and predicts motion; SinMDM instead uses a QnA-based UNet because transformer receptive fields span the full motion.The section also notes that image inpainting can be adapted to motion composition by holding body parts or temporal intervals constant while denoising the rest.Source: §2.2 · p. 2

2.3 Motion Synthesis Models

  • Motion-synthesis models include conditioned approaches using action, text, motion parts, or music, while fewer models are fully unconditioned and learn the motion manifold unsupervised.The section classifies architectures including recurrent, autoencoder, GAN, normalizing-flow, neural-field, and diffusion models.Source: §2.3 · p. 3

3 PRELIMINARY

SinMDM learns internal motion motifs from one sequence using a lightweight diffusion model. The section outlines its denoising process and represents motion as dynamic features synthesized for a fixed skeleton.

3 PRELIMINARY

  • SinMDM uses a lightweight DDPM with a UNet whose attention layers are replaced by QnA layers to learn motifs from a single motion of arbitrary topology.The framework synthesizes varied motions that retain the input's core elements.Source: §3 · p. 3

3.1 Denoising Diffusion Probabilistic Models (DDPM)

  • DDPM synthesis starts from noise and iteratively predicts a clean motion, re-noises it, and repeats until timestep 0.The forward process adds Gaussian noise over T steps; the reverse process predicts the input motion rather than the noise.Source: §3.1 · p. 3

3.2 Motion representation

  • Motion is split into time-varying dynamic features and fixed static features; SinMDM predicts dynamics while leaving the fixed skeleton intact.The paper uses “motion synthesis” to mean generating dynamic features only.Source: §3.2 · p. 3
  • The motion tensor has N frames and F features per frame, with representations varying by dataset, including root motion, joint features, and contact labels.HumanML3D includes root motion and joint positions, velocities, and rotations; other datasets use root location, joint angles, and contact labels.Source: §3.2 · pp. 3–4
  • Contact labels may come from the dataset or be computed from joint velocities obtained through forward kinematics.The labels cover joints that contact the ground.Source: §3.2 · p. 4

4 GENERATIVE NETWORK

The generative network is designed to learn internal motion motifs without overfitting to a single sequence. It limits receptive field size with a shallow UNet and overlapping local attention, and the paper validates these design choices experimentally.

4 GENERATIVE NETWORK

  • SinMDM requires no motion pyramid and limits receptive-field width by reducing UNet depth.Reduced depth alone is insufficient because standard UNets still use global attention.Source: §4 · p. 4
  • A UNet trained on one input can overfit because its large receptive field makes generated sequences closely resemble the input.The paper's central design insight is that limited receptive fields help learn internal motifs more effectively.Source: §4 · p. 4
  • QnA provides overlapping local attention with shared learned queries, avoiding the restricted cross-window interaction of non-overlapping windows.In the motion-domain UNet, QnA replaces global attention and offers lower time and space costs.Source: §4 · p. 4
  • The experiments assess narrow receptive fields, QnA layers, and the choice of a UNet over a transformer.The supplied section states the evaluation targets but gives no outcomes.Source: §4 · p. 4

5 APPLICATIONS

SinMDM groups diverse motion applications into motion composition, harmonization, and straightforward generation, applying them at inference time without retraining. Composition combines reference and synthesized motion, while harmonization adjusts unseen motion to match learned motifs.

5.1 Motion Composition

  • Temporal composition fills selected frames: in-betweening synthesizes the interior, while expansion generates beyond the reference sequence.A linearly interpolated ROI mask can soften transitions between reference and synthesized segments when the reference differs from the learned motion.Source: §5.1 · pp. 4–5, Figure 4 · p. 5, Figure 5 · p. 6
  • Spatial composition assigns selected joints to the reference motion and synthesizes the remaining joints with learned motifs.The illustrated example combines a reference warm-up in the upper body with a learned curved walk in the lower body.Source: §5.1 · p. 5, Figure 6 · p. 6

5.2 Motion Harmonization

  • Harmonization combines an unseen motion with learned motifs, while style transfer uses the unseen motion as content and a learned motion as style.The method adds synthesized high-frequency details to the input motion’s low frequencies; the resulting style transfer preserves the content’s step rhythm and size.Source: §5.2 · pp. 5–7

5.3 Straight-forward Applications

  • One-minute motions and varied crowds can be generated from a single trained sequence without additional training.Crowd members are sampled using different Gaussian noise inputs.Source: §5.3 · p. 7

6 EXPERIMENTS

Experiments compare SinMDM across motion benchmarks and show strong quantitative, qualitative, and user-rated results. Ablations examine receptive-field width, attention choice, and architecture.

6.1 Benchmarks

  • Experiments use HumanML3D and Mixamo benchmarks, whose skeletons and motion characteristics differ, alongside Truebones Zoo and artist-created animation data.Mixamo includes 70 characters with distinct bone lengths and some distinct topologies; its motions are more diverse and dynamic than HumanML3D motions.Source: §6.1 · p. 7, §6 · p. 7

6.3 Quantitative Results

  • SinMDM leads Ganimator on all Mixamo metrics but one and has a significant advantage in the Harmonic Mean metric.The comparison uses the Mixamo benchmark.Source: Table 1 · p. 8, §6.3 · p. 8

6.2 Metrics

  • HumanML3D uses deep-feature metrics, while Mixamo adds motion-level inter- and intra-diversity and efficiency measures; Harmonic Mean balances mixed metric scores.The Mixamo metrics include network parameters, iteration count, per-iteration time, and total running time.Source: §6.2 · pp. 7–8

6.3 Quantitative Results

  • On HumanML3D deep-feature metrics, SinMDM balances good scores across metrics, unlike vanilla MDM overfit and crop-trained MDM divergence.Vanilla MDM has good SiFID and intra-diversity but poor inter-diversity; MDM trained on crops shows the opposite pattern.Source: Table 3 · p. 8, §6.3 · p. 8
  • On the Ganimator-selected Gangnam-style motion, SinMDM leads in two metrics and is comparable in the third.The comparison also includes MotionTexture and acRNN.Source: §6.3 · p. 8, Table 2 · p. 8

6.5 Ablation

  • Ablations associate wider receptive fields with overfitting; QnA-based UNet scores better than vanilla attention or no attention, while the QnA transformer has poor SiFID.The transformer variant uses QnA layers and relative temporal positional embeddings.Source: Table 4 · p. 9, §6.5 · p. 9

6.3 Quantitative Results

  • Users significantly preferred SinMDM over Ganimator and crop-trained MDM when judging diversity, fidelity, and quality.Each pair was assessed over eight motions by ten users per comparison.Source: §6.3 · pp. 8–9, Figure 11 · p. 9

6.4 Qualitative results

  • Qualitative comparisons show mode collapse in Ganimator, overfitting in MDM, and jitter in crop-trained MDM, while SinMDM exhibits none of these issues.The examples include Mixamo punch-to-elbow and HumanML3D standing-to-jumping motions.Source: §6.4 · p. 9, Figure 12 · p. 9, Figure 13 · p. 9

7 CONCLUSIONS

The conclusion presents SinMDM as a lightweight diffusion approach for single-motion synthesis, with reported quality and efficiency gains and a stated out-of-distribution limitation.

7 CONCLUSIONS

  • The work applies diffusion models to single-motion synthesis, particularly for scarce animal and imaginary-creature motion data with distinctive skeletons and motifs.The authors say such motion cannot readily be captured or learned from available human motion data.Source: §7 · p. 10
  • Across several datasets, the method reportedly outperforms current work in quality and time-space performance and supports long-motion synthesis and multiple manipulation tasks.Tasks include spatial and temporal in-betweening, motion expansion, harmonization, style transfer, and crowd animation.Source: §7 · p. 10
  • Single-instance learning limits out-of-distribution synthesis, and the diffusion approach has relatively long inference time because of its iterative nature.The authors identify learning from motion data of creatures with compatible skeletons as a possible future direction.Source: §7 · p. 10

A HYPERPARAMETERS AND TRAINING DETAILS

The appendix reports the hardware used for training the models.

A HYPERPARAMETERS AND TRAINING DETAILS

B QNA RECAP

The recap describes QnA as a local, shift-invariant attention layer and notes its accuracy-efficiency trade-off.

B QNA RECAP

  • QnA is an efficient attention-based layer that operates in a shift-invariant manner over k-size windows.Its window outputs use a self-attention mechanism.Source: B QNA RECAP · p. 12
  • QnA learns a query for the training data and applies attention locally to each k-size window, avoiding the query-key operation described as costly.The output at frame i uses its k-neighbourhood.Source: B QNA RECAP · p. 12
  • The paper characterizes QnA as having a state-of-the-art accuracy-efficiency trade-off.This claim is referenced to Fig. 15.Source: B QNA RECAP · p. 12

C USER STUDY – SCREENSHOTS

The user study asks viewers to compare video clips for quality, fidelity, or diversity. Figure 16 shows representative screenshots for the three attributes.

C USER STUDY – SCREENSHOTS

  • Users select the clip they find more suitable for quality, fidelity, or diversity from several clips displayed on each screen.The study presents several video clips per screen.Source: C USER STUDY – SCREENSHOTS · p. 12
  • Figure 16 shows a representative screenshot for each attribute, and each human figure shown is played as a video.The screenshots correspond to the study’s quality, fidelity, and diversity comparisons.Source: C USER STUDY – SCREENSHOTS · p. 12, Figure 16 · p. 14
Loading 2302.05905v2…