PaperlayerView exact revision on arXiv

Source-linked AI summary

Flexible Motion Generation from Language and Style References

Kai Weixian Lan, Bodie Criswell, Briana Fedkiw, Zhan Zhang, Joseph Teran, Daniel Holden

arXiv:2609.08032v1cs.CVcs.GRcs.LG

TL;DR

Text prompts convey action semantics but can leave fine-grained style details underspecified. FlexMoGen combines text with style-reference clips, using a label-free variational encoder and temporal control. It reports a balance between content fidelity and style reflection, while style transfer is less reliable for references outside its shared training-data origin.

  • Problem

    Text describes semantic motion content but can be ambiguous or insufficient for specifying nuanced style, including rhythm, energy, and joint-level articulation.

    Source: §1 · p. 2

  • Method

    FlexMoGen combines a variational style encoder with a pretrained text-to-motion diffusion model, using style adaptation and relative position encoding for temporal control.

    Source: §3.2 · p. 5, §5 · p. 14

  • Results

    FlexMoGen reports the best balance between content fidelity and style reflection, and leads on nearly every metric in the time-varying setting.

    Source: §1 · p. 2, §5 · p. 14

  • Takeaways & Limitations

    The framework supports time-varying style inputs and long-sequence generation, including sequences longer than prior methods can handle.

    Source: §5 · p. 14

    Transfer is markedly less reliable for references outside 100STYLE that feature largely upper-body and prop-handling motions, although content prompts continue to be followed.

    Source: §4.3 · p. 11

Abstract

from arXiv · show

We introduce FlexMoGen, a novel framework for flexible human motion synthesis conditioned on both natural language descriptions and motion style references. Text prompts are effective at defining semantic content, but they are often limited in capturing fine-grained style details such as timing, limb articulation, and expressive dynamics. A style example clip supplements the text by conveying these nuanced motion characteristics directly, enabling the model to preserve high-level intent while reproducing the desired stylistic traits. Given a text prompt and a style example clip, FlexMoGen generates high-quality motions that preserve semantic content while faithfully reflecting the target style, offering users greater control over the animation generation process. Unlike prior methods that rely on discrete style labels and do not generalize to long or multi-style generation, FlexMoGen learns a variational style encoder without style supervision and supports long, time-varying, multi-style synthesis. Our framework jointly pre-trains the style encoder and a text-to-motion latent diffusion model within a unified architecture, modulating motion style through a lightweight adaptation module. It integrates an efficient relative positional encoding scheme and is trained on both stylized and non-stylized datasets, enabling strong generalization to unseen text-style combinations. Experiments show that FlexMoGen achieves the best balance between content fidelity and style reflection.

1 Introduction

FlexMoGen combines text-defined motion content with example-clip style references to preserve semantic intent while conveying stylistic details that text alone may not specify. It addresses limits in prior style-conditioned methods with label-free style learning, temporal control, and a reported balance between content fidelity and style reflection.

1 Introduction

  • Text prompts specify action semantics, while style clips convey expressive traits such as rhythm, energy, and joint-level articulation.The authors describe those stylistic nuances as ambiguous or difficult to specify precisely with text alone.Source: §1 · p. 2
  • Prior methods are limited to short sequences and offer limited control; most require style labels, while a label-free exception still targets short, single-style sequences.The proposed approach supports separate controls and style examples for different animation segments.Source: §1 · p. 2
  • FlexMoGen reports the best balance between content fidelity and style reflection while advancing flexibility and versatility.The paper also provides supplementary videos and materials to substantiate its approach.Source: §1 · p. 2
  • A variational style encoder learns motion-style representations from example clips without explicit style-class labels.Source: §1 · p. 2
  • A lightweight adaptation module injects learnable bias vectors into attention key and value projections to modulate motion style.Source: §1 · p. 2
  • Temporal attention provides fine-grained control over changing semantics and styles and supports motions longer than those seen during training.Source: §1 · p. 2

2 Related work

Prior work spans text-conditioned motion generation, long-sequence control, and motion stylization. FlexMoGen situates its contribution in scheduling style over time and across body parts, alongside content control.

2.1 Text-conditioned human motion generation

  • Text-to-motion work progressed from VAE approaches to diffusion, latent and discrete motion representations, and models supporting multiple motion-language tasks.Long-sequence methods schedule changing content over time, including parallel tracks for different body parts.Source: §2.1 · p. 3
  • Prior long-sequence methods schedule content along timelines; FlexMoGen also schedules style over time and per body part.Source: §2.1 · p. 3

2.2 Human motion stylization

  • Motion stylization includes feature-statistic, adversarial, and pretrained-diffusion approaches; recent text-guided methods add reference-based style control to pretrained text-to-motion models.StyleMotif uses a reference without a time index, whereas FlexMoGen resolves style along time and per body part.Source: §2.2 · p. 3

3 Method

FlexMoGen combines text-defined motion content with style examples through a pretrained text-to-motion diffusion model and a lightweight style adaptation module. Its temporal encodings and variational style encoder support time-varying control over content and style.

3 Method

  • The framework pretrains a text-to-motion model and variational style encoder, then uses SAM to inject style while combining positional encodings for temporal control.SAM adds style as attention key and value biases, while relative and global positional encodings support fine-grained temporal control.Source: §3 · pp. 3–4

3.1 Framework overview

  • SAM injects example-derived style embeddings as key and value biases into a pretrained text-to-motion model, preserving content semantics while reflecting the reference style.The module conditions the backbone without duplicating it; only SAM is trained during style finetuning.Source: §3.1 · p. 4, §3 · p. 4

3.2 Pretraining text-to-motion model

  • The text-to-motion diffusion model operates on a temporally preserved latent sequence, with combined relative and global positional encodings supporting local control and global coherence.The latent sequence is four times shorter than the motion, corresponding to one token per four frames, or 0.13 seconds at 30 fps.Source: §3.2 · p. 5

3.3 Pretraining style encoder

  • The variational style encoder learns style without explicit labels by reconstructing motion from its root trajectory and latent style code, using 63 non-action styles from 100STYLE.It produces a 384-dimensional code with separate embeddings for five body parts and a root component.Source: §3.3 · p. 6

3.4 Further implementation details

  • Style finetuning uses separate classifier-free guidance weights for content and style, alongside reconstruction and style-preservation losses.The style-preservation loss compares the generated motion’s style embedding with the reference motion’s; λsty is 0.01 in the experiments.Source: §3.4 · p. 7

4 Experiments

Experiments compare FlexMoGen with baselines across long-sequence and time-varying style generation, generalization, temporal localization, body-part mixing, ablations, and user preferences. The results favor evaluating content and style together, while revealing limits for external references and rapid limb dynamics.

4.1 Long sequence generation

  • FlexMoGen achieves the best balance among content fidelity, stylistic accuracy, and motion quality across methods in long-sequence generation.The comparison evaluates sequences up to 400 frames; Figure 6 shows only FlexMoGen performing the leftward hop while retaining the Zombie style.Source: Table 2 · p. 9, Figure 6 · p. 9, §4.1 · p. 10

4.2 Temporally-varying style generation

  • In time-varying generation, FlexMoGen leads on Multimodal Distance, R-Precision, CLIP Score, and SRA, with FID the exception.Figure 7 shows accurate action transitions and temporal ordering of styles; the baselines misorder actions or styles.Source: Table 3 · p. 9, Figure 7 · p. 9, §4.2 · p. 10

4.3 Style generalization test

  • 0.6634 (out-of-domain top-1 SRA) versus 0.6882 (in-domain) for FlexMoGen in long-sequence generation, compared with SMooDi’s 0.5467 versus 0.6721.In time-varying generation, FlexMoGen’s out-of-domain top-1 SRA is 0.4979, more than twice SMooDi’s in-domain 0.2306.Source: Table 4 · p. 11, Table 5 · p. 11, §4.3 · p. 11

4.4 Evaluations on temporal localization and long sequences

  • 0.1064 leakage and 90.6% switch detection within 1.5 frames show localized style changes at 200 frames; localization remains unaffected at longer lengths.At 800 frames, each style interval is twice the maximum training length; the reported cost is a mild loss of style fidelity.Source: §4.4 · pp. 11–12

4.5 Body-part style mixing

  • Per-part style embeddings can be assembled from different references to create mixed or asymmetric body-part styles without model changes or retraining.The demonstrated compositions are evaluated qualitatively because mixed styles have no ground-truth label in the 100STYLE taxonomy.Source: §4.5 · p. 12

4.6 Ablation study

  • Ablations support relative positional encoding, motion VAE pretraining, and SAM as design choices for motion quality and content–style flexibility, especially with time-varying styles.The value bias carries style, while key bias alone does not; SAM is the only tested injection mechanism whose style accuracy does not collapse under time-varying input.Source: §4.6 · pp. 12–14

4.7 User study

  • In 330 pairwise comparisons by 22 participants, FlexMoGen is preferred on nearly every criterion for content preservation, style reflection, and overall motion quality.The two criteria won by a baseline are the ones its metrics already predict.Source: §4.7 · p. 14

4.8 Failure cases

  • Transfer is less predictable for references outside the encoder’s pretraining distribution and for styles defined by rapid limb motion, whose high-frequency dynamics can be over-smoothed.For external references, content may remain while stylistic character is muted; the authors limit their unseen-style claim to the encoder’s training distribution.Source: §4.3 · p. 11, §4.8 · p. 14

5 Conclusion

FlexMoGen combines a label-free variational style encoder, lightweight style adaptation, and relative position encoding to support long motions with time-varying styles. The supplementary examples illustrate single-style, changing-style, and multi-action generation, while the conclusion notes remaining limits in fast-motion fidelity and external-style transfer.

5 Conclusion

  • FlexMoGen combines a label-free variational style encoder, a lightweight style adaptation module, and relative position encoding for long-sequence and time-varying style generation.The paper reports the best balance between style fidelity and content preservation across its evaluation settings.Source: §5 · p. 14
  • Current limitations include over-smoothed rapid limb movements, reduced fidelity for fast or abrupt styles, unreliable transfer beyond the style encoder’s pretraining distribution, and slow diffusion sampling.The paper suggests frequency-domain supervision for rapid dynamics and flow-matching models for faster sampling.Source: §5 · p. 14

A.1 Single style

  • Across single-style examples, FlexMoGen follows the requested action while reflecting style, including backpedaling without erratic movement and spinning without unintended translation.The standing long-jump example also shows a clean jump without visible ground penetration or unnecessary extra steps.Source: Figure 8 · p. 19, Figure 9 · p. 19, Figure 10 · p. 20

A.2 Time-varying style

  • FlexMoGen performs three actions in temporal order while reflecting successive Chicken, Penguin, and unseen BeatChest styles.The supplementary examples group visualizations by single-style, time-varying, external-style, and body-part mixing settings, with SASI and a rapid-limb failure case shown separately.Source: A More visualizations · p. 19, Figure 11 · p. 20

A.3 Styles outside 100STYLE

Style references outside 100STYLE transfer unevenly: some stylistic traits appear while others are missed, even as the requested content is preserved. Body-part mixing combines style codes from multiple references into one motion, including asymmetric arm styles.

A.3 Styles outside 100STYLE

  • References outside 100STYLE transfer hit or miss: some styles appear clearly while others are only partly or barely reflected, although the content prompt remains followed.The paper attributes this boundary to the style encoder’s pretraining distribution and limits its generalization claim to styles within that distribution; this setting has no quantitative score.Source: §A.3 · pp. 20–22

A.4 Body-part style mixing

  • Body-part style mixing assembles style codes from different clips, producing combined traits and even asymmetric left- and right-arm styles that no single reference provides.The examples compose three references across legs, arms, and spine, or four references with separate clips for the two arms.Source: §A.4 · p. 22, Figure 13 · p. 22, Figure 14 · p. 22

A.5 Style injection with SASI

The SASI injection variant can transfer a single style but does not reliably confine styles to their assigned time intervals. A separate failure case shows that rapid or abrupt style dynamics can be represented but emerge over-smoothed.

A.5 Style injection with SASI

  • SASI’s requested style switch is undetectable in 70.3% of time-varying sequences, despite transferring some single styles faithfully.Its single-style top-1 SRA is 0.6719; in the illustrated time-varying example, it performs the actions but remains near-neutral in style.Source: §A.5 · p. 22, Figure 15 · p. 23

A.6 Rapid or abrupt style references

  • For BeatChest, the model represents the style but suppresses its speed, producing gentler chest-striking motions than the reference.This is distinct from external-reference failures, which concern references outside the style encoder’s pretraining distribution.Source: §A.6 · p. 23

B Temporal style transfer evaluation

The temporal style-transfer evaluation tests whether FlexMoGen switches styles at the requested boundary while preserving motion continuity. It outperforms baselines on style localization, though a fast in-domain style exposes over-smoothing of motion pace and amplitude.

B Temporal style transfer evaluation

  • The protocol evaluates two style references assigned to separate sequence intervals and measures segment scores, leakage, switch timing or failure, and boundary motion peaks.Leakage measures neighboring-style probability within a segment; boundary peaks are normalized by each sequence's median and interpreted against the Real row.Source: B Temporal style transfer evaluation · pp. 23–24
  • For the fast in-domain BeatChest style, FlexMoGen follows the reference arm trajectory but loses its striking pace and amplitude, while retaining the requested walking content.The reference alternates sharply between hands at the chest and hands near the face; the generated movement is gentler.Source: Figure 16 · p. 24
  • FlexMoGen's in-domain interval scores are balanced (0.5380 against 0.5352), while both baselines lose 0.17 to 0.34 between intervals after the requested switch.The baselines' leakage is ≈0.49, compared with 0.1064 for FlexMoGen; its switch is detected in 90.6% of sequences and lands within 1.5 frames.Source: B Temporal style transfer evaluation · p. 24
  • For out-of-domain styles, FlexMoGen's leakage is 0.1017 and delay is −1.45, with a 6.8% failure rate; SMooDi keeps both intervals near chance.T2M+MP has first-interval SRA of 0.4788 against FlexMoGen's 0.4620 ± 0.0577, but its second interval falls to 0.3528 and leakage is 0.4916.Source: B Temporal style transfer evaluation · p. 24
  • Boundary peaks for all three methods stay between 0.84 and 1.10 times the Real values, while FlexMoGen's are 0.96 to 1.03 times Real.In-domain boundary foot skating is 82.8% for T2M+MP versus 12.2% for FlexMoGen; SMooDi's low peaks reflect absent transitions, not smoother ones.Source: B Temporal style transfer evaluation · p. 24

C Long sequences with time-varying style

FlexMoGen maintains time-varying style localization as output length grows from 200 to 800 frames, without degradation in sequence-level FID or foot skating. At 800 frames, top-1 style accuracy declines slightly, and the evaluation covers only one transition per sequence.

C Long sequences with time-varying style

  • The evaluation repeats the two-style, midpoint-switch protocol at 200, 400, and 800 frames, corresponding to 6.7, 13.3, and 26.7 seconds at 30 fps.The 800-frame setting makes each style interval 400 frames, twice the entire training length; only FlexMoGen is evaluated because baselines do not localize style at 200 frames.Source: C Long sequences with time-varying style · pp. 24–25
  • FID is 2.06, 2.40, and 2.02 across the three lengths, while foot-skating ratio stays between 0.084 and 0.089.MM Dist and CLIP score improve slightly with length, but the Real row improves as well, so the reported gap between rows stays constant.Source: Table 9 · p. 25, C Long sequences with time-varying style · p. 25
  • Across 200, 400, and 800 frames, FlexMoGen's two interval scores remain alike, leakage stays between 0.1155 and 0.1238, and failure rate remains 0.102 to 0.105.The switch lands within 1.2 frames of the requested boundary at all three lengths, including when each style interval is 400 frames.Source: C Long sequences with time-varying style · pp. 25–26, Table 10 · p. 26
  • At 800 frames, top-1 SRA falls from 0.5466 at 400 frames to 0.5053, while top-3 SRA remains flat at 0.7309, 0.7312, and 0.7178.The two segment-wise top-1 values also fall, but the style remains localized to its assigned interval.Source: C Long sequences with time-varying style · p. 26
  • The number of transitions remains one per sequence, so sequences with several short style intervals are outside this evaluation's scope.The varied interval duration ranges from 100 to 400 frames.Source: C Long sequences with time-varying style · p. 26

D User study

In a pairwise user study, FlexMoGen is preferred over baselines across nearly all criteria, supporting its balance of content fidelity, style reflection, and motion quality. The section also provides pseudocode for training and inference.

E Pseudo code

  • The supplementary pseudocode covers both the training and inference procedures of FlexMoGen.The study used 20 motion samples, 330 total responses, and 22 participants.Source: E Pseudo code · p. 26, D User study · p. 26

D User study

  • FlexMoGen is preferred over SMooDi on content preservation (61.1%), style reflection (67.7%), and motion quality (71.3%); against the other baselines, preferences vary by criterion.It leads T2M+MP on content (82.1%) and quality (94.0%), and LoRA-MDM on style reflection (83.7%); overall, the study corroborates the reported balance across the three criteria.Source: D User study · p. 26, Figure 17 · p. 27

Algorithm 1 FlexMoGen training

  • Training encodes content and style, adds noise to the content latent, predicts its clean form, and updates only the Style Adaptation Module parameters using reconstruction and style-preservation losses.The pretrained motion encoder, decoder, text-to-motion backbone, and style VAE are frozen.Source: Algorithm 1 FlexMoGen training · p. 27

Algorithm 2 FlexMoGen inference (stylized text-to-motion generation)

  • Inference encodes the text prompt and style example, initializes a noisy latent, iteratively denoises with DDIM, and decodes the final latent into stylized motion.The style code is set deterministically to the style encoder's mean.Source: Algorithm 2 FlexMoGen inference… · p. 28

F Implementation details

The implementation uses a frozen pretrained backbone with a trainable Style Adaptation Module, which is smaller than the cited adapter alternatives. FlexMoGen also has the shortest reported inference time among the compared models.

F Implementation details

  • The text-to-motion model and style encoder each require about 10 hours of training, while SAM takes around 8 hours on one NVIDIA RTX 6000 Ada Generation GPU.The respective iteration counts are 60,000, 40,000, and 20,000; SAM uses bfloat16 precision.Source: F Implementation details · p. 28
  • The diffusion model is trained for 1000 steps and sampled with 100 DDIM steps.Source: F Implementation details · p. 28

F.1 Parameter count of SAM

  • SAM learns separate style projections at each of seven layers, using a 384-dimensional style code and a 512-wide frozen backbone; zero-initialized gates leave the pretrained model unchanged at initialization.The projections map the style code to backbone width, followed by gates of the same width.Source: §F.1 · p. 28
  • SAM has 6.44M trainable parameters, 31% of the deployed model's 20.77M parameters, and is its only trained component.A ControlNet-style branch trains 16.1M parameters, about 2.5 times as many as SAM; SASI trains 14.25M.Source: §F.1 · p. 28

G Inference time

  • FlexMoGen achieves the shortest inference time among the compared models.The supplied table caption identifies the comparison as inference time across the model and baselines.Source: G Inference time · p. 28, Table 11 · p. 29

H Dataset

The dataset material lists training styles and classifies styles for training and evaluation. The following subsection describes a five-part skeleton division and learned masking of independent body-part interactions.

H Dataset

I Skeleton and body-part division

I.1 Effect on the style latent space

For 20 styles, including three out of distribution, both style-code variants form one cluster per style. Per-part codes produce tighter, more separated clusters, which the paper connects to style fidelity and recombination across references.

I.1 Effect on the style latent space

  • For 20 styles, including three out of distribution, both variants form one cluster per style, but per-part codes yield tighter clusters and wider margins.Source: §I.1 · p. 30, Figure 19 · p. 30
  • Because SAM injects this code at every layer, tighter separation reduces style ambiguity at injection and is the stated mechanism behind higher style fidelity.The paper notes the higher fidelity in supplementary video.Source: §I.1 · p. 30
  • Distinct part components remain sufficiently distinct to be recomposed across references.Source: §I.1 · p. 30

Related papers

Loading 2609.08032v1…