Source-linked AI summary
Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases
Ian Mason, Sebastian Starke, Taku Komura
TL;DR
Existing style-transfer systems require a reference content clip, which does not suit real-time animation with changing user input. This paper combines local-phase motion synthesis and FiLM-based style modulation, and reports meaningful style-space interpolations and a wider variety of modelled motions. Its training data does not densely sample the style space enough for robust generalisation to unseen styles.
Problem
Existing style-transfer methods require a reference content clip, whereas real-time systems need to handle future motion content that changes with user input.
Source: §1 · p. 1Method
An animation-synthesis network uses local phases, while a style-modulation network applies FiLM transformations to its hidden layers for real-time style control.
Source: §1 · p. 2Results
The learned compact style representation supports meaningful linear interpolations, and improved animation synthesis enables modelling a wider variety of motions.
Source: §7 · p. 8Takeaways & Limitations
From a single stylised motion clip, the system can generate arbitrary-length user-controlled motions in a variety of styles.
Source: §6 · p. 8The 100STYLE training data does not densely sample the style space enough for robust generalisation to unseen styles.
Source: §5.3.2 · p. 8
Abstract
from arXiv · showhide
Controlling the manner in which a character moves in a real-time animation system is a challenging task with useful applications. Existing style transfer systems require access to a reference content motion clip, however, in real-time systems the future motion content is unknown and liable to change with user input. In this work we present a style modelling system that uses an animation synthesis network to model motion content based on local motion phases. An additional style modulation network uses feature-wise transformations to modulate style in real-time. To evaluate our method, we create and release a new style modelling dataset, 100STYLE, containing over 4 million frames of stylised locomotion data in 100 different styles that present a number of challenges for existing systems. To model these styles, we extend the local phase calculation with a contact-free formulation. In comparison to other methods for real-time style modelling, we show our system is more robust and efficient in its style representation while improving motion quality.
1 INTRODUCTION
Existing clip-based style transfer depends on content motion specified in advance, while real-time systems need to respond to changing user input. The paper proposes a real-time style-modelling framework and introduces 100STYLE to evaluate it.
1 INTRODUCTION
- Existing style-transfer methods map a reference motion clip’s style onto different content, but depend on content specified upfront.This limits their suitability for real-time systems where future timing and contacts are unknown and may change with user input.Source: §1 · p. 1
- The system combines an animation-synthesis network with a style-modulation network that uses FiLM to transform hidden layers.It conditions motion on learned style representations without requiring paired or aligned data.Source: §1 · p. 2
- A contact-free local-phase function extends the local phase network to stylised locomotion lacking the contact information it requires.The framework is designed to retain interactive control while learning compact representations for many styles.Source: §1 · p. 2
- 100STYLE provides 100 performative locomotion styles and over 4 million motion-capture frames for evaluating style modelling.The dataset and framework together model varied locomotion modes and continuous transitions between styles.Source: §1 · p. 2
- The paper’s stated contributions include improved style modelling, the 100STYLE release, contact-free local phases, and a low-cost style representation for modelling more styles.The introduction also describes style modelling as combining user-conditioned motion synthesis with on-the-fly style changes.Source: §1 · pp. 1–2
2 RELATED WORK
Prior work spans data-driven motion modelling, style transfer, and controllable style modelling. The paper builds on local-phase animation synthesis and hidden-feature transformations while addressing limits in controllability and style representation.
2.1 Data-Driven Motion Modelling
- Autoregressive neural networks predict future motion from input frames, while reinforcement-learning approaches model motion in physics-based systems.The paper notes that animation quality in physics-based systems still trails recent kinematic approaches.Source: §2.1 · p. 2
- Local motion phases model asynchronous limb movements, while earlier locomotion systems use global phase to condition network weights.The paper builds on local phases because many motion styles lack a coherent global phase; these systems still lack general motion features for unseen movements without retraining.Source: §2.1 · p. 2
2.2 Style Transfer and Modelling
- Image stylisation methods transform convolutional hidden units, and style transfer is also related to domain adaptation.Source: §2.2 · p. 2
- Most learned animation style-transfer methods require content motion to be specified upfront, limiting their suitability for interactive applications with high-level user control.Prior controllable approaches include one-hot style labels, motion matching without an explicit style representation, and residual adapters without a general style-parameterisation function.Source: §2.2 · p. 2
3 STYLISED LOCOMOTION DATA
The paper introduces 100STYLE, a large locomotion dataset designed to cover varied styles and provide data for style-modelling research. It processes the motion into input frames, output frames, and style clips for the models.
3.1 The 100STYLE Dataset
- 100STYLE combines 100 locomotion styles with many motion-capture frames per style, joining the variety of broad datasets with the depth of datasets containing fewer styles.The dataset was designed for data-hungry methods; earlier data-driven evaluations often used few globally cyclic styles.Source: §3.1 · p. 3
- The corpus includes styles with stochastic motion, asymmetry, or no clear global phase, alongside variations on a neutral walk.Figure 2 illustrates six styles, including Drunk, Drag Right Leg, and Wild Arms.Source: §3.1 · p. 3, Figure 2 · p. 3
- A single actor was captured at 60 fps with a 28-joint skeleton; each style includes idling, walking, and running.The capture used Xsens motion capture technology in a 4.5 m x 4.5 m area.Source: §3.1 · p. 3
3.2 Data Processing
- Processing removes T-poses and non-stylised locomotion, mirrors symmetric styles, and extracts input frames, output frames, and style clips.The input includes local-coordinate trajectory and joint features plus an 8-dimensional local phase representation; outputs include future trajectory and joint features, contact labels, and local phases and updates.Source: §3.2 · p. 3
- Style clips contain 240 frames of relative joint positions, velocities, and rotations, excluding trajectory and global offset.Source: §3.2 · p. 3
4 STYLE MODELLING METHODOLOGY
The system combines motion synthesis with style modulation to generate animation from local phases and style-conditioned network features.
4 STYLE MODELLING METHODOLOGY
- The animation-synthesis network predicts motion using local phases and FiLM parameters supplied by a style-modulation network.The modulation network extracts style information from motion clips and transforms the synthesis network’s hidden layers.Source: §4 · p. 3, Figure 3 · p. 4
4.1 Animation Synthesis Network
The synthesis network blends expert weights using local signals, while local phases capture asynchronous limb motion that global foot-contact cycles cannot represent. Contact-based or contact-free source functions are fitted into phase features for end-effectors, with velocity scaling accommodating limited movement.
4.1 Animation Synthesis Network
- A gating network uses local signals to blend expert weights, forming the parameters of the motion synthesis network.Each expert weight set represents one possible configuration of network weights.Source: §4.1 · p. 4
4.1.1 A Hybrid Approach to Local Phases.
- Local phases model asynchronous limb timings that a global phase defined by foot contacts may not capture in stylised motion.The gating network blends expert weights according to individually specified joints’ phase cycles.Source: §4.1.1 · p. 4
4.1.2 Contact-Based Local Phases.
- Contact-based source functions label foot contact when the foot is near the floor and moving below a velocity threshold.The defaults are d_max=0.01 and v_max=0.15, adjustable when detection is too sensitive or insufficiently sensitive.Source: §4.1.2 · p. 4
4.1.3 Contact-Free Local Phases.
- For contact-free phases, bone positions are centered by style and gait, then projected onto the first principal component to capture local cyclic movement.The principal component is directed using the character’s forward axis so the source function can distinguish in-sync from half-cycle-offset motion.Source: §4.1.3 · pp. 4–5
4.1.4 Processing and Calculating.
- Fitted sinusoidal source functions yield per-frame phase features, scaled by bone velocity over a one-second window to account for limited movement.The source functions are normalized within a one-second window and smoothed with a Butterworth filter; phases are calculated for hand and foot end effectors, using contacts when available.Source: §4.1.4 · p. 5
4.2 Style Modulation Network
A FiLM generator learns style representations from stylised motion clips and uses them to modulate hidden layers of the motion synthesis network. Layer normalization is applied before this feature-wise affine modulation.
4.2 Style Modulation Network
- The FiLM generator learns style representations from stylised motion clips and produces parameters that modulate the motion synthesis network’s hidden layers.The parameters are applied through an affine transformation.Source: §4.2 · p. 5
- For each hidden layer, FiLM parameter vectors element-wise scale and shift its hidden units.Source: §4.2 · p. 5
- Layer normalization precedes FiLM modulation, which is applied only to the hidden layers.Figure 4 illustrates local phase labelling across Pendulum Hands, Hands In Pockets, and Left Hopping styles.Source: §4.2 · p. 5, Figure 4 · p. 5
4.3 Combining Animation Synthesis and Style
The system combines phase-conditioned animation synthesis with style modulation to predict motion frames. It is trained with mean squared error and a bone-length loss, using balanced minibatches across training styles.
4.3 Combining Animation Synthesis and Style
- The model combines phase-conditioned motion synthesis and style modulation to predict each output frame.The synthesis network’s expert parameters are gated by local phase features, and its hidden representations are modulated by the style network.Source: §4.3 · p. 5
- Training uses 95 styles from 100STYLE, reserving the remaining styles for fine-tuning performance tests on new styles.The FiLM generator uses two temporal convolutional layers followed by two fully connected layers.Source: §4.3 · p. 5
- The objective combines mean squared error across predicted features with a bone-length loss to help reduce bone-stretching artifacts.Each minibatch contains one style; styles are cycled in a fixed order to balance training, for 90 epochs using Adam at a learning rate of 1 × 10^-4.Source: §4.3 · p. 6
4.4 Creating the Real-Time Demo
The real-time demo maps keyboard input to a trajectory blended with the predicted future trajectory. Inverse kinematics addresses foot sliding and inaccurate rotations during style interpolation; style FiLM parameters can also be precomputed and blended online.
4.4 Creating the Real-Time Demo
- The demo converts keyboard input into a trajectory and blends it with the network’s predicted future trajectory.Source: §4.4 · p. 6
- Foot-contact inverse kinematics post-processes the output to reduce foot sliding, while arm inverse kinematics aligns bones to predicted positions when interpolated rotations deform the character.The rotation issue is attributed to linear interpolation not always corresponding to correct rotation interpolation.Source: §4.4 · p. 6
- Precomputed FiLM parameters can be stored as a single vector per style and blended online using barycentric coordinates among three selected training styles.Source: §4.4 · p. 6
5 EVALUATION
The evaluation compares style-modulation and animation-synthesis variants, assessing qualitative failure cases, runtime and storage, test error, and adaptation to new styles. It finds advantages for local phases and FiLM in several settings, alongside runtime and generalization constraints.
5.1.1 Motions Without a Global Phase.
- Local phases better handle motions without a coherent global phase, and contact-free local phases accommodate styles with varying environmental contacts.The contact-free example has only one foot contacting the ground.Source: §5.1.1 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
5.1.2 Dealing With Large Numbers of Styles.
- FiLM scales and shifts all hidden layers, whereas one-hot and residual-adapter modulation add a style-dependent shift to one layer; the latter methods show artifacts and one-hot interpolation is not smooth.For one-hot interpolation, the first style tends to persist until interpolation is complete.Source: §5.1.2 · p. 7, Figure 7 · p. 7, Figure 8 · p. 7
5.1.3 No Explicit Phase Features.
- MANN is competitive but, like PFNN, can struggle when limbs have distinct phase cycles; explicit phase features may help LPN separate such styles.The comparison concerns velocity-gated MANN and the explicit phase features used by LPN.Source: §5.1.3 · p. 7, Figure 9 · p. 7
5.2.1 Computational Costs.
- All models remain able to run at 60 fps, while LPN and MANN incur extra runtime; LPN FiLM uses half as many parameters per style as LPN One-hot.The authors suggest PFNN may be preferable for small numbers of globally cyclic movements.Source: §5.2.1 · p. 7, Table 2 · p. 7
5.2.2 Test Error.
- All FiLM models have the lowest test errors, but MSE correlates weakly with perceived quality, so test error alone can misrepresent motion quality [37].The reported test errors use seen styles and unseen motion frames.Source: §5.2.2 · p. 7, Figure 10 · p. 8
5.3.1 Theoretical Analysis.
- FiLM and residual adaptation can be expressed as parameter changes in the MTDS framework [6], while sharing style-agnostic parameters keeps the style representation more compact than methods [6].The FiLM formulation uses style-specific scaling and shift parameters.Source: §5.3.1 · p. 8
5.3.2 Modelling New Styles.
- Because the dataset does not densely sample the style space, unseen-style generalization is not robust; fine-tuning the FiLM generator while freezing LPN parameters yields clearer separation for new styles.The approach assumes training data for each new style.Source: §5.3.2 · p. 8, Figure 11 · p. 8
6 DISCUSSION
The system generates arbitrary-length, user-controlled motions from a single stylised clip, while its account of style depends on a content-invariance assumption. Its source function handles stochasticity poorly, and style interpolation can produce physically implausible poses.
6 DISCUSSION
- From a single stylised motion clip, the system generates arbitrary-length user-controlled motions in a variety of styles.Source: §6 · p. 8
- The shared LPN parameters define content as what stays the same across dataset styles and style as what differs, an assumption the authors note is not the only possible definition.They describe style as a subjective and creative concept.Source: §6 · p. 8
- The source function does not handle stochasticity well, motivating further exploration of contact-free techniques for new situations.Source: §6 · p. 8
- Although interpolation between styles is smooth, it often takes a direct path that can produce physically implausible poses, including clipping.Source: §6 · p. 8
7 CONCLUSION
The work explores humanoid style modelling through the 100style dataset, contact-free local phases, and higher-capacity feature-wise linear transformations. Its compact style representation supports meaningful linear interpolations, while improved animation synthesis enables modelling a wider variety of motions.
7 CONCLUSION
- The 100style dataset, contact-free local phases, and higher-capacity feature-wise linear transformations support style modelling of a wider variety of motions.The authors attribute the broader motion modelling to improved animation synthesis and report meaningful linear interpolations in the compact style space.Source: §7 · p. 8