PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionBricks: Scalable Real-Time Motions with Modular Latent Generative Model and Smart Primitives

Tingwu Wang, Olivier Dionne, Michael De Ruyter, David Minor, Davis Rempe, Kaifeng Zhao, Mathis Petrovich, Ye Yuan, Chenran Li, Zhengyi Luo, Brian Robison, Xavier Blackwell, Bernardo Antoniazzi, Xue Bin Peng, Yuke Zhu, Simon Yuen

arXiv:2604.24833v1cs.ROcs.AIcs.GRcs.LG

TL;DR

Real-time interactive motion control needs both scalable generation and fine-grained control, needs that existing approaches do not fully meet. MotionBricks combines a modular latent generative backbone with smart primitives for navigation and object interaction. The paper reports state-of-the-art motion quality, 15,000 FPS throughput, and 2ms latency, and demonstrates the framework in animation and on a humanoid robot.

  • Problem

    Existing approaches do not fully meet demands for real-time generation across many motion skills and fine-grained control through velocity commands, styles, and precise keyframes.

    Source: §1 · p. 2

  • Method

    MotionBricks combines a modular latent generative backbone with smart primitives that provide a unified keyframe interface for navigation and object interaction.

    Source: §1 · pp. 2–3

  • Results

    The backbone achieves state-of-the-art motion quality with 2ms latency and 15,000 FPS throughput, and demonstrations cover navigation, interaction, and Unitree G1 robot control.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A unified framework can address motion control across virtual characters and physical robots without fine-tuning or tagging.

    Source: §8 · p. 15

    The 350,000-motion dataset has limited coverage of rare motion types, object interactions are not explicitly modeled, and real-world deployment lacks privileged object-pose and terrain information for the smart object primitive.

    Source: §8.1 · p. 15

Abstract

from arXiv · show

Despite transformative advances in generative motion synthesis, real-time interactive motion control remains dominated by traditional techniques. In this work, we identify two key challenges in bridging research and production: 1) Real-time scalability: Industry applications demand real-time generation of a vast repertoire of motion skills, while generative methods exhibit significant degradation in quality and scalability under real-time computation constraints, and 2) Integration: Industry applications demand fine-grained multi-modal control involving velocity commands, style selection, and precise keyframes, a need largely unmet by existing text- or tag-driven models. To overcome these limitations, we introduce MotionBricks: a large-scale, real-time generative framework with a two-fold solution. First, we propose a large-scale modular latent generative backbone tailored for robust real-time motion generation, effectively modeling a dataset of over 350,000 motion clips with a single model. Second, we introduce smart primitives that provide a unified, robust, and intuitive interface for authoring both navigation and object interaction. Applications can be designed in a plug-and-play manner like assembling bricks without expert animation knowledge. Quantitatively, we show that MotionBricks produces state-of-the-art motion quality on open-source and proprietary datasets of various scales, while also achieving a real-time throughput of 15,000 FPS with 2ms latency. We demonstrate the flexibility and robustness of MotionBricks in a complete production-level animation demo, covering navigation and object-scene interaction across various styles with a unified model. To showcase our framework's application beyond animation, we deploy MotionBricks on the Unitree G1 humanoid robot to demonstrate its flexibility and generalization for real-time robotic control.

1 Introduction

Runtime motion control must balance scalable real-time generation with fine-grained control, while existing approaches leave important production needs unmet. MotionBricks combines a latent neural backbone with smart primitives and reports high-quality, real-time generation across animation and robotics applications.

1 Introduction

  • Animation graphs can require thousands of animations and states, making high-quality runtime behavior systems difficult to maintain and extend.The introduction describes such systems as labor-intensive and largely the domain of well-resourced studios.Source: §1 · p. 2
  • Command-driven models require manual tags or predefined controls, while text-driven models lack the fine-grained control needed for industrial applications.The paper identifies balancing scalability, quality, and speed as a challenge for text-driven approaches.Source: §1 · p. 2
  • MotionBricks combines a low-level latent neural backbone with high-level smart primitives to provide fine-grained, scalable runtime control.The primitives generate keyframes for the backbone, and the applications reuse a fixed pretrained model without fine-tuning or additional tagging.Source: §1 · pp. 2–3
  • The modular backbone and smart primitives support navigation and object interaction through a unified interface, including a Unitree G1 robot deployment.The paper reports state-of-the-art motion quality, 2ms latency, and 15,000 FPS throughput.Source: §1 · p. 3

2 Related Work

Prior runtime animation methods offer control or motion quality but face limitations in scalability, specialization, or inference speed. MotionBricks is positioned against graph-based, deterministic, in-betweening, and generative approaches to address real-time interactive use.

2 Related Work

  • Animation graphs organize clips into states and transitions, but become fragile and labor-intensive as the number of states and transitions grows.Motion matching enables more flexible frame-level retrieval, yet is typically used as a navigation subcomponent within broader graphs.Source: §2 · p. 3
  • Deterministic neural motion models can suffer from foot sliding, object penetration, and excessive smoothing, and are often specialized to few tasks.The paper says these issues restrict their applicability in production.Source: §2 · p. 3
  • In-betweening methods use starting and ending keyframes but are generally narrow in skill scope and predominantly used offline.The passage notes target keyframes are usually unavailable during runtime animation.Source: §2 · p. 3
  • Motion diffusion models offer scalability and motion quality, but their inference speed and quality-speed trade-offs remain challenging for real-time pipelines.The passage says standard diffusion may take seconds or minutes, while faster variants barely achieve real-time speed.Source: §2 · p. 3
  • MotionBricks’s inference pipeline turns commands or game events into keyframes, predicts root timing and trajectory, models pose tokens, and decodes continuous motion.The caption describes the four stages in sequence.Source: Figure 2 · p. 4

3 Overview

MotionBricks uses smart primitives to convert commands into keyframes for a modular generative backbone. The backbone progressively predicts timing, root trajectories, and pose tokens before decoding continuous motion, while accepting flexible constraints.

3 Overview

  • A conditional structured tokenizer compresses motion into discrete tokens, after which root and pose modules generate motion from in-betweening constraints.The root module predicts timing and initial trajectories; the pose module predicts pose-token distributions conditioned on constraints and root predictions.Source: §3 · p. 4
  • Smart primitives translate user commands and game events into target keyframes that, with context frames, condition the generative backbone.Smart locomotion supports navigation and smart object supports scene interactions; the backbone requires no fine-tuning after training.Source: §3 · p. 4
  • The system accepts arbitrary subsets of local-root, global-root, and pose constraints, using learnable mask embeddings for unspecified constraints.Navigation typically uses one or two keyframes, while object interaction may need consecutive hand positions; supported segments span 12–64 frames at 30 FPS.Source: §3 · pp. 4–5

4 Structured Multi-headed Tokenizer

MotionBricks represents motion with structured latent tokens and separates root from pose information. Its decoder incorporates variable keyframe constraints, and its training samples varying constraint counts to support flexible use.

4 Structured Multi-headed Tokenizer

  • The tokenizer maps motion into structured latent representations and separates root from pose while supporting flexible conditioning.The motion state includes global and local root values, joint positions and rotations, joint velocities, and contact labels.Source: §4 · p. 5

4.1 Encoder

  • The encoder uses multiple discrete codebooks to decompose continuous latent features without manually prescribing motion partitions.The paper says this encourages compositionality and disentanglement and can make the latent manifold more robust to mispredicted tokens.Source: §4.1 · p. 5

4.2 Decoder

  • The decoder progressively upsamples pose tokens and injects root trajectories and variable-length joint keyframe constraints through skip connections.It can also refine the predicted root trajectory to produce cleaner footsteps and smoother transitions.Source: §4.2 · pp. 5–6

4.3 Quantizer and Training Í𝐾

  • Training samples 0–10 keyframe constraints at arbitrary temporal positions, while inference uses context frames and one to four target keyframes.The tokenizer uses a VQ-VAE quantizer loss and a running-mean codebook update for more stable training.Source: §4.3 · p. 6

5 Neural Backbone

MotionBricks uses a progressive, modular backbone to generate in-betweening motion in two stages: a root module predicts timing and trajectory, then a pose module generates poses conditioned on roots and keyframes.

5 Neural Backbone

  • The modular backbone generates in-betweening motion in a coarse-to-fine progression, with intermediate results available for inspection and refinement.The root module predicts timing and an initial trajectory before the pose module generates poses conditioned on roots and keyframes.Source: §5 · p. 6

5.1 Root Module

  • The root module uses a two-step design: it predicts the number of in-between frames, then estimates an initial root trajectory conditioned on the timing prediction and first-step hidden states.At inference, the distribution over in-between frame counts is sampled or argmaxed at 4-frame resolution; when the ground-truth frame count is provided, only that count is allowed.Source: §5.1 · p. 6
  • The root module outputs global root values that support direct trajectory control, then converts them to local values for finer-grained motion generation.Each final embedding decodes into four consecutive frames of root trajectory.Source: §5.1 · p. 7

5.2 Pose Module

  • The pose module models pose-token distributions conditioned on root trajectories and keyframe constraints, using masked-token prediction to generate poses.A single inference pass typically produces high-quality motion; cosine scheduling serves as a training curriculum.Source: §5.2 · p. 7

6 Smart Primitives

MotionBricks combines smart locomotion and smart-object primitives on a shared neural backbone, using runtime keyframes and commands to author navigation and interactions. The section describes how trajectory refinement and object bindings support varied styles, scene interactions, and motion variations.

6 Smart Primitives

  • Smart locomotion and smart-object primitives provide a unified framework for navigation and interaction, with task specifications set at runtime rather than supervised during training.The framework is designed for broad applicability without retraining.Source: §6 · p. 7

6.1 Smart Locomotion

  • Smart locomotion addresses ambiguous or infeasible navigation commands and style transitions through progressive root refinement and styled keyframe placement with repetitive planning.A spring-smoothed initial trajectory is computed from commanded velocity and heading over a 1.0-second horizon.Source: §6.1 · pp. 7–8
  • Replanning and a generative backbone support natural motions and smooth style transitions even with a single keyframe and artificially inserted velocity or heading values.The method also supports keyframe generation via 2D blendspaces.Source: §6.1 · p. 8
  • Neural root refinement adjusts trajectories to account for context and keyframes, while the decoder coordinates root values with full-body details and supports root-trajectory adjustments.The section reports diverse locomotion styles in UE5 and on the Unitree G1 robot.Source: §6.1 · pp. 8–9

6.2 Smart Object

  • Smart objects use intent keyframes and object bindings instead of pre-canned clips, with keyframes serving as soft guidance or hard constraints depending on the interaction phase.Keyframe sets may be obtained from a motion library or authored manually; drop-frame settings allow early transitions from soft-guidance poses.Source: §6.2 · pp. 9–10
  • Object bindings manage interaction detection, placement, and keyframe anchoring, allowing identical keyframe definitions to support different approach angles and object positions.The section describes motions adapting to varied heights, angles, velocities, and styles without additional authoring.Source: §6.2 · pp. 10–11, Figure 8 · p. 10

7 Experiments

The experiments evaluate MotionBricks on four datasets spanning different scales, skeletons, and animation and robotics domains. They compare benchmark performance and analyze tokenizer capacity, dataset scaling, root interpolation, and deployments in UE5 and on a humanoid robot.

7.1 Dataset

  • The four datasets span a 350,000-clip proprietary collection, HumanML3D, LaFAN1-G1, and a 62,000-clip proprietary subset, with varied scales and skeletons.The 350k dataset contains approximately 700 hours, 9,300 skills, and 36 categories; all methods are retrained at 30 FPS for fair comparison.Source: §7.1 · p. 11

7.2 Quantitative Evaluation

  • Evaluation compares six in-betweening methods using speed, distribution, human ratings, diversity, smoothness, physical plausibility, and keyframe-precision metrics.The evaluation uses randomly sampled context and target keyframes; reaching success requires reaching the target root within 5 cm and heading within 15 degrees.Source: §7.2 · pp. 11–12
  • MotionBricks outperforms baselines across all datasets, with better distribution matching, lower jitter, stronger keyframe precision, near-perfect reaching success, and the highest human-evaluation score and win rate.The authors report 2 ms latency and 15,000 FPS throughput; diversity advantages are strongest on larger datasets, while smaller-dataset baselines are comparable or marginally better.Source: §7.2 · p. 12, Table 3 · p. 12, Table 4 · p. 13

7.3 Ablation Studies and Analysis

  • As codebook size grows, the multi-head tokenizer continues improving while the single-head baseline plateaus; MotionBricks balances FID and keyframe precision around 10^6 to 10^7 tokens.At extreme scales, keyframe errors rise slightly; the baseline shows no meaningful improvement in FID or keyframe error as capacity increases.Source: Figure 10 · p. 12, §7.3 · p. 13
  • With total capacity fixed, token perturbations reveal a head-size trade-off: too many tokens per head hurt NPSS, while too few hurt FID and keyframe precision.The authors recommend 128 to 256 tokens per head for best overall performance.Source: §7.3 · p. 13, Figure 11 · p. 13
  • As training data grows, keyframe precision improves for MotionBricks but baseline keyframe errors rise; FID worsens for both methods as datasets get larger.The authors hypothesize that lower FID on smaller datasets may reflect overfitting and less diverse outputs.Source: §7.3 · pp. 13–14, Figure 13 · p. 14
  • Root trajectory interpolation preserves FID and reaching success, keeps foot skate low, and benefits from pose keyframe constraints that improve FID and reaching success.The decoder keeps predicted pose tokens fixed while overriding root values with interpolated trajectories.Source: §7.3 · p. 14, Figure 14 · p. 14

7.4 Interactive Animation, Robotics Applications and Engineering Details

  • The same model architecture and training settings support UE5 animation and Unitree G1 deployment; G1 inference latency rises to 5 ms on Jetson Orin.The UE5 demo uses ONNX and TensorRT; G1 motions are retargeted and supplied as targets to a physical tracking controller, with replanning at 10 Hz or command changes.Source: §7.4 · p. 15

8 Discussion

MotionBricks combines a scalable latent backbone with smart primitives for motion generation across virtual characters and robots. The discussion identifies limits in dataset coverage, physical awareness, retargeting, and training compute, while reporting that fewer GPUs remain viable without significant visual-quality degradation.

8 Discussion

  • MotionBricks combines a scalable latent neural backbone and smart primitive interfaces for task-agnostic motion generation across virtual characters and physical robots.The framework is demonstrated on datasets of varying scales and in UE5 and on the Unitree G1, without fine-tuning or tagging.Source: §8 · p. 15

8.1 Limitations and Future Work

  • The 350,000-motion dataset has limited coverage of rare motions, and object interactions are not explicitly modeled.The authors identify dataset expansion and video foundation models as possible future directions.Source: §8.1 · p. 15
  • Kinematic planners may produce self-collisions or motions beyond hardware constraints, such as dangerous flips.The authors suggest co-training the planner and tracking policy as a possible way to address this.Source: §8.1 · p. 15
  • Retargeting across morphologies remains challenging: runtime retargeting is fast but low-quality, while offline retargeting is accurate but requires months of iteration.The UE5 demo uses runtime retargeting and the G1 uses offline retargeting.Source: §8.1 · p. 15

A Scaling Behavior with Number of GPUs

  • Fewer GPUs remain viable without significant degradation in visual quality, although more GPUs generally yield faster convergence and modestly better asymptotic performance.Throughput scales near-linearly with GPU count; numerical gaps primarily affect tokenizer quality and translate to only minor perceptual changes.Source: A Scaling Behavior with Number of GPUs · p. 18

B Comparison between FSQ and VQ-VAE

With matched token capacity, FSQ and VQ-VAE perform similarly across the reported metrics, but VQ-VAE has lower cross-entropy loss and better FID. MotionBricks therefore uses VQ-VAE as its default tokenizer.

B Comparison between FSQ and VQ-VAE

  • For a matched-capacity comparison, the study treats FSQ latent dimensions as heads and levels per dimension as codebook size per head.The pose module predicts the corresponding token indices from the FSQ tokenizer.Source: B Comparison between FSQ and VQ-VAE · p. 18
  • VQ-VAE consistently achieves lower cross-entropy loss and better FID than FSQ, and the authors observe similar improvements in visual quality and robotics deployment.Given the same number of tokens and heads, the two methods otherwise yield similar performance in FID, reconstruction loss, and cross-entropy loss.Source: B Comparison between FSQ and VQ-VAE · p. 18
  • The authors use VQ-VAE as MotionBricks’ default tokenizer because it produces a latent space that is slightly easier for the generative model to model.This choice follows the reported loss and FID comparisons.Source: B Comparison between FSQ and VQ-VAE · p. 18

C Replanning Frequency

MotionBricks replans based on motion-buffer state and control changes, with experiments supporting a practical interval of 3 to 9 frames. Its keyframe-level blending offers finer control than clip-level blendspaces without requiring a fully authored grid.

C Replanning Frequency

  • MotionBricks replans when fewer than τ frames remain in the motion buffer and also when control commands change.Command changes allow an instant control reaction.Source: C Replanning Frequency · p. 18
  • 3 to 9 frames is the recommended default replanning interval, with instant replanning upon command changes.Across intervals from 1 to 8 frames, discrete latent methods such as MotionBricks and MMM show slightly worse FID, keyframe error, and joint jitter at higher replanning frequencies; continuous diffusion methods show improved FID and joint jitter.Source: C Replanning Frequency · p. 18

D 2D Blendspace and Keyframe Blending in MotionBricks

  • Traditional 2D blendspaces interpolate neighboring clips using two input axes, typically speed and direction, and require careful sample coverage to avoid artifacts.MotionBricks instead interpolates reference-pose keyframes in joint space.Source: D 2D Blendspace and Keyframe Blending… · p. 18
  • Keyframe-level blending interpolates individual joint-space keyframes and feeds them to the neural backbone, avoiding the need to populate a full blendspace grid.The model handles artifacts associated with naive interpolation, and drop-frame replanning ensures blended keyframes are not rigidly enforced during generation.Source: D 2D Blendspace and Keyframe Blending… · p. 18

E Dataset Statistics

The full dataset contains 350,000 clips across varied categories, activities, content types, and performers; its open-source subset includes distinct test sets for generalization. Coverage is strongest for locomotion and sparse for manipulation, terrain, and rare height variations.

E Dataset Statistics

  • The full dataset contains 350,000 clips spanning 36 categories, 203 activities, 9,285 content types, and 163 performers.Categories, activities, and content types represent progressively finer levels of motion categorization; Figure 18 shows overall statistics and diversity by categories, activities, and contents.Source: E Dataset Statistics · p. 18, Figure 18 · p. 19
  • The 143,792-motion BONES-SEED subset is split into 129,661 training clips, 7,166 Test-Unseen clips, and 6,965 Test-Heldout clips.The open-source subset retains roughly 40% of motions and all 163 performers.Source: E Dataset Statistics · p. 18
  • Test-Unseen has no content-type overlap with training and includes five unseen activities, while Test-Heldout shares category, activity, content, and actor levels but contains novel actor-content combinations.The two test sets are designed to evaluate different aspects of generalization.Source: E Dataset Statistics · p. 20
  • Locomotion is the most represented category, while object manipulation and terrain interaction are the least covered.The dataset lacks finger and object motion data, and has no continuous terrain information, limiting realistic motion generation on uneven surfaces.Source: E Dataset Statistics · p. 20
  • Vaulting over a 1m obstacle has only one or two captured clips, posing challenges for learning and generalization.The dataset includes height variations, but their coverage remains sparse.Source: E Dataset Statistics · p. 20

F In-betweening Visual Comparisons

The in-betweening comparisons show MotionBricks outperforming the baselines across three pose transitions. The baselines exhibit implausible poses and poor keyframe adherence.

F In-betweening Visual Comparisons

G UE5 Demo Implementation

The UE5 demo uses task actors to supply target keyframes to MotionBricks, with state management and inference scheduling supporting interactive locomotion and object interactions. The contributor list identifies the project’s research, animation, robotics, and advising teams.

G UE5 Demo Implementation

  • Task actors encapsulate runtime logic, assets, and state, supplying target keyframes sampled from user clips for object interactions or parameterized locomotion.At runtime, the active actor provides the keyframes consumed by MotionBricks.Source: G UE5 Demo Implementation · p. 21
  • Locomotion actors remain active on the character, while scene-placed object actors provide interaction keyframes and can orient sequences to the character’s approach angle.The demo draws target keyframes from selected clips.Source: G UE5 Demo Implementation · p. 21
  • New keyframes interrupt current motion and trigger generation; deterministic replanning supports these overrides without introducing artifacts.Inference runs only when new target keyframes arrive; otherwise, the pose generator samples the existing motion buffer.Source: G UE5 Demo Implementation · pp. 21–22
  • The StateTree switches among locomotion, interaction, and falling; for falls, the system simulates a projectile to determine the landing position and keyframe.Locomotion keyframes depend on velocity, orientation, and replanning frequency; control returns to locomotion after the fall sequence.Source: G UE5 Demo Implementation · p. 22

H Contributors

  • The contributor list assigns project leadership to Tingwu Wang, robotics development to Tingwu Wang, Chenran Li, Zhengyi Luo, and Ye Yuan, and advising to Simon Yuen, Xue Bin Peng, and Yuke Zhu.Source: H Contributors · p. 22

Related papers

Loading 2604.24833v1…