PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Space

Qing Yu, Kent Fujiwara

arXiv:2609.34190v1cs.CV

TL;DR

Most text-to-motion methods generate in compressed latent spaces that can limit fidelity and direct frame- and joint-level control. MSFLOW predicts motion directly in continuous space with representation-aware flow and attention, achieving state-of-the-art results and exact inference-time control without control-conditioned training.

  • Problem

    Many text-to-motion methods use reconstruction-focused, temporally downsampled latent spaces that can limit generation quality and obscure direct frame- and joint-level manipulation.

    Source: §1 · p. 1

  • Method

    MSFLOW predicts clean motion directly in continuous space, scales its flow path for representation anisotropy, and adapts joint language-motion attention to the motion representation.

    Source: §1 · p. 2

  • Results

    MSFLOW achieves state-of-the-art text-to-motion performance across HumanML3D and SnapMoGen, while its XYZ variant provides exact any-joint, any-frame projection control without control-conditioned training.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Direct XYZ motion supports training-free projection control with exact constraints while retaining frame- and joint-level access.

    Source: §1 · p. 2, §6 · p. 9

    Experiments cover only HumanML3D and SnapMoGen, so whether the observed interactions generalize across other datasets, skeletons, and motion domains remains unevaluated.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent advances in diffusion and flow models have substantially improved text-driven human motion generation. Yet most methods generate in low-dimensional, temporally downsampled latent spaces learned primarily for reconstruction, a bottleneck that can limit generation quality and preclude direct manipulation of individual frames and joints. We introduce MotionSpaceFlow (MSFlow), a representation-aware flow-matching framework that predicts clean motion directly in continuous motion space without a learned encoder or decoder. To account for the anisotropic structure of direct motion representations, we propose representation-aware noise scaling and show how the initial Gaussian source scale governs the covariance of intermediate probability-path marginals. We further introduce a Representation-Aware Multimodal Diffusion Transformer (RA-MMDiT), which jointly updates token-level language and full-resolution motion features through joint attention while adapting temporal information flow to the motion representation: causal attention for incremental features defined by frame-to-frame changes, and bidirectional attention for global features such as absolute joint coordinates. Across different datasets and motion representations, MSFlow achieves state-of-the-art text-to-motion performance. Its global representation variant additionally enables zero-shot, inference-time control over any joint or frame through projection sampling without control-conditioned training, delivering leading motion quality with exact constraint satisfaction.

1 INTRODUCTION

MSFLOW generates full-resolution motion directly, addressing reconstruction and control limits of compressed latent representations. It combines representation-aware flow paths and temporal attention, with state-of-the-art text-to-motion results and training-free spatial control.

1 INTRODUCTION

  • Latent motion methods can be limited by autoencoder capacity and reconstruction fidelity, while latent tokens lose direct correspondence to frames and joints.Direct motion-space generation must still handle correlated joint, feature, and frame variables and align short prompts with full-resolution motion.Source: §1 · p. 1
  • MSFLOW predicts clean motion in continuous motion space and uses the Gaussian source scale to shape signal emergence and intermediate path anisotropy.It converts clean-endpoint predictions into a velocity field for deterministic ODE sampling.Source: §1 · p. 2
  • RA-MMDiT jointly updates language and full-resolution motion features, using causal attention for incremental features and bidirectional attention for absolute coordinates.The latter provides two-sided context for constraints around a controlled keyframe.Source: §1 · p. 2
  • MSFLOW achieves state-of-the-art text-to-motion performance on HumanML3D and supports exact, training-free any-joint, any-frame control with its XYZ variant.The control uses projection sampling without control-conditioned training.Source: §1 · p. 2

2 RELATED WORK

Prior work shortens motion sequences with discrete or autoencoded representations, while autoregressive and bidirectional models use different temporal dependencies. MSFLOW instead transports continuous motion directly, projects constraints at sampling, and jointly refines language and motion features.

2.1 MOTION REPRESENTATIONS AND TEMPORAL MODELING FOR MOTION GENERATION

  • Discrete and autoencoded representations shorten sequences, but temporal downsampling limits information capacity and decoded fidelity remains bounded by reconstruction quality.MSFLOW instead transports continuous motion while preserving frame- and joint-level access.Source: §2.1 · p. 3

2.2 FLOW MATCHING AND SPATIAL CONTROL

  • Prior motion-control methods use conditioned training, guidance, test-time optimization, or large-scale controllable models; ProjFlow projects clean endpoints during sampling [Watanabe et al., 2026].MSFLOW applies endpoint projection directly to XYZ motion for arbitrary frame–joint–axis constraints.Source: §2.2 · p. 3

2.3 MOTION–LANGUAGE CONDITIONING

  • Unlike generators that condition on frozen language features, MSFLOW refines token-level DistilBERT features over flow time and couples them to motion tokens through multimodal attention.Direct motion-space generation lacks a temporally downsampled latent bridge, making textual semantics more challenging to preserve.Source: §2.3 · p. 3

3 METHOD

MSFLOW transports full-resolution motion directly from scaled Gaussian noise and uses RA-MMDiT to predict clean motion endpoints. Its source scale and temporal attention are adapted to the representation, and global XYZ endpoints can be projected to impose spatial constraints at inference.

3.1 PROBLEM SETTING AND DIRECT MOTION REPRESENTATIONS

  • Direct motion-space generation removes the decoder-induced fidelity floor while retaining ordinary generative error; the bound can be zero if the decoder range contains the data support.The method also preserves frame- and joint-level access for spatial control.Source: §3.1 · p. 4

3.2 CLEAN-MOTION PREDICTION AND FLOW SAMPLING

  • Training interpolates a scaled Gaussian source with clean motion, predicts the clean endpoint, and converts it to velocity; sampling integrates all frames jointly with Heun and a final Euler update.The same interpolation time is used across valid frames, keeping the noisy state a coherent sequence.Source: §3.2 · pp. 4–5

3.3 REPRESENTATION-AWARE NOISE SCALING

  • The fixed source scale controls signal emergence and intermediate covariance conditioning; the authors use s = 5 for both 263D and XYZ models.Larger s delays signal emergence and improves conditioning by reducing relative variance disparity, without changing the absolute eigenvalue gap.Source: §3.3 · p. 5

3.4 REPRESENTATION-AWARE MULTIMODAL TRANSFORMER

  • RA-MMDiT jointly attends over text and motion: causal prefix masks suit integrated 263D velocities, while bidirectional context supports globally coupled XYZ trajectories.A time-aware Token Refiner adapts frozen DistilBERT features to the flow time.Source: §3.4 · p. 6

3.5 INFERENCE-TIME ANY-JOINT, ANY-FRAME CONTROL

  • At inference, XYZ endpoint projection imposes selected frame–joint–axis targets without constraint-conditioned training, recomposes a path-consistent state, and applies a final hard projection.The sampler refreshes the source estimate with noise at the training path’s source scale.Source: §3.5 · p. 6

4 EXPERIMENTS

MSFLOW is evaluated across motion representations and datasets, with experiments testing representation-aware design choices and inference-time spatial control.

4.1 EXPERIMENTAL SETUP

  • Experiments use HumanML3D sequences of up to 192 frames at 20 fps, plus 272D MotionStreamer and native 296D SnapMoGen representations.The primary model generates normalized 263D HumanML3D features; the XYZ variant generates normalized positions for 22 joints, evaluated through the 67D root- and rotation-invariant evaluator.Source: §4.1 · p. 7

4.2 QUANTITATIVE TEXT-TO-MOTION RESULTS

  • 0.038±0.004 FID is the global XYZ variant’s best HumanML3D result; the incremental variant has the strongest R-Precision, MM-Dist, and CLIP point estimates.Against CMDM, the global variant reduces FID from 0.078 to 0.038 (51.3%), while the incremental variant improves Top-1/2/3 R-Precision by 0.008/0.005/0.004, reduces MM-Dist by 0.030, and increases CLIP by 0.001.Source: §4.2 · p. 7
  • Top-1/3 R-Precision on SnapMoGen improves from .831/.958 with CMDM to .910/.984 with MSFLOW, alongside 16.342 FID and 12.538 multimodality.Table 2 presents text-to-motion results on SnapMoGen.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 ANALYSIS OF REPRESENTATION-AWARE DESIGN CHOICES

  • Causal attention lowers 263D FID from 0.067 to 0.046, while bidirectional attention lowers XYZ FID from 1.563 to 0.038.The authors interpret this reversal as a finite-model filtering–smoothing inductive bias, not a universal advantage of restricted context.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • Increasing source scale from 1 to 5 lowers causal 263D FID from 0.111 to 0.046 and substantially improves every reported bidirectional XYZ metric.The authors explain that larger scale improves path isotropy and delays signal emergence; source scale should be selected for the representation.Source: §4.3 · p. 8
  • Clean-motion prediction beats direct v-prediction on every reported metric in all four matched comparisons, lowering FID from 0.061 to 0.046 for causal 263D and 0.249 to 0.038 for bidirectional XYZ.The reported result is improved distributional fidelity in high-dimensional motion spaces.Source: §4.3 · p. 8

4.4 INFERENCE-TIME SPATIAL CONTROL

  • Without control-conditioned training, MSFLOW exactly satisfies evaluated spatial constraints; in the all-joints setting it improves FID from 0.097 to 0.061 and R-Precision@3 from 0.779 to 0.818 over ProjFlow.The protocol averages metrics across five control densities, from 1 to 196 keyframes.Source: §4.4 · p. 8, Table 4 · p. 9

4.5 QUALITATIVE RESULTS

  • Qualitative examples cover local limb placement, forward locomotion, circular motion, and cartwheels, with controlled joints meeting constraints while unconstrained motion remains coherent and prompt-aligned.Figure 3 uses colored markers to indicate constraints on selected joints and frames.Source: §4.5 · p. 8, Figure 3 · p. 9

5 LIMITATIONS

MSFLOW’s evaluation and control scope has dataset, efficiency, and constraint boundaries.

5 LIMITATIONS

  • Experiments cover only HumanML3D and SnapMoGen, leaving generality across other datasets, skeletons, and motion domains unevaluated.Whether the observed interplay among representation, source scale, and temporal attention holds in those settings remains to be evaluated.Source: §5 · p. 9
  • Full-resolution motion requires longer token sequences than temporally compressed latent models, which may limit efficiency for very long motions.Hierarchical or streaming direct-space models are proposed as possible ways to improve scalability while retaining fine-grained control.Source: §5 · p. 9
  • Projection sampling enforces linear equality constraints on joint coordinates but does not explicitly guarantee physical feasibility or support nonlinear constraints such as collision avoidance, contact, and joint limits.The authors identify physical priors and more expressive constraint solvers as future directions.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents MSFLOW as direct continuous-space flow matching with representation-aware noise scaling and temporal modeling.

6 CONCLUSION

  • MSFLOW predicts clean motion without a learned encoder or decoder and uses representation-aware noise scaling for anisotropic direct-space probability paths.Its RA-MMDiT adapts temporal information flow to the motion representation.Source: §6 · p. 9
  • Experiments on HumanML3D and SnapMoGen demonstrate state-of-the-art text-to-motion performance, with causal attention favored for incremental 263D features and bidirectional attention for absolute XYZ coordinates.Source: §6 · p. 9
  • The XYZ variant enables training-free any-joint, any-frame projection with exact constraint satisfaction.Source: §6 · p. 9

SUPPLEMENTARY MATERIAL

The supplement adds qualitative comparisons, design ablations, and evaluations under additional protocols, alongside implementation and theoretical material. Its results cover both model choices and performance under the standard 263D HumanML3D evaluator.

SUPPLEMENTARY MATERIAL

  • The supplement provides additional text-to-motion and spatial-control results, reproducibility details, sample code, and theoretical and empirical analyses.The section index lists further protocols, SnapMoGen, compute efficiency, detailed spatial control, and covariance and attention analyses.Source: SUPPLEMENTARY MATERIAL · p. 14, SECTION INDEX · p. 14

A.1 QUALITATIVE RESULTS

  • Both MSFLOW variants more consistently preserve highlighted prompt details than ACMDM and SALAD across six qualitative prompts.The prompts probe action count, limb specificity, directed reaching, trajectory reversal, lateral direction, and ground-contact transitions.Source: §A.1 · p. 14, Figure 4 · p. 15

A.2 MODEL CAPACITY AND TEXT–MOTION ATTENTION

  • The eight-block, width-512 model is the capacity–quality trade-off: four blocks worsen FID, while 12 blocks offer no consistent gains.Four-block FID rises from .046 to .206 for 263D and from .038 to .098 for XYZ; 12-block 263D worsens FID, R-Precision, and MM-Dist.Source: §A.2 · p. 14, Table 5 · p. 15
  • Without the Token Refiner, cross-attention improves retrieval point estimates but degrades FID to .099 for 263D and .104 for XYZ.With the Refiner retained, cross-attention is competitive for causal 263D but raises XYZ FID from .038 to .079; multimodal attention with flow-time-aware refinement is used in the primary models.Source: §A.2 · pp. 14–15

A.3 COMPLETE REPRESENTATION-AWARE DESIGN ANALYSIS

  • Table 6 consolidates attention, source-scale, and prediction-target configurations, with metric point estimates and 95% confidence intervals.The reported configurations use source scale s = 5 and clean-motion prediction unless stated otherwise.Source: Table 6 · p. 16, §A.3 · p. 16

A.4 EVALUATION WITH THE STANDARD 263D PROTOCOL

  • Under the standard 263D evaluator, the incremental variant leads Top-1 and Top-3 R-Precision at 0.591 and 0.863 and MM-Dist at 2.565.It ties CMDM for best Top-2 R-Precision at 0.778; the global variant has the second-best FID, 0.050, and MM-Dist, 2.607.Source: §A.4 · p. 16

A.5 TRAINING AND EVALUATION WITH THE 272D MOTIONSTREAMER PROTOCOL

The authors train MSFLOW from scratch on HumanML3D’s 272D representation and evaluate it with the MotionStreamer protocol. The model improves reported retrieval and distributional metrics against the protocol’s baselines.

A.5 TRAINING AND EVALUATION WITH THE 272D MOTIONSTREAMER PROTOCOL

  • The 272D model is trained from scratch with source scale s = 5 and causal attention, then evaluated with the official 272D evaluator.The authors choose causal attention because this representation, like 263D, encodes incremental motion.Source: §A.5 · p. 16
  • Compared with 263D, 272D replaces scalar yaw velocity with a 6D rotation increment and includes root positions and rotations.Its source SMPL rotations retain twist information and allow direct SMPL animation without the SMPLify post-processing needed when using only standard-representation positions.Source: §A.5 · pp. 16–17
  • FID improves to 6.540 from 11.790 for MotionStreamer and 9.460 for UMO-Unified; MSFLOW reaches Top-1/2/3 R-Precision 0.792/0.919/0.955 and MM-Dist 13.864.It outperforms both UMO variants on every reported metric; UMO evaluates converted 201D HY-Motion outputs, unlike the models trained on 272D motion.Source: §A.5 · p. 16, Table 8 · p. 17

A.6 TRAINING AND EVALUATION ON SNAPMOGEN

On SnapMoGen, MSFLOW trains on the native 296D representation and combines bidirectional attention with strong text–motion alignment. The supplement also reports compute efficiency for 196-frame generation, where MSFLOW pairs competitive quality with relatively low model size and latency.

A.6 TRAINING AND EVALUATION ON SNAPMOGEN

  • The SnapMoGen model is trained from scratch on native 296D motion with source scale s = 10 and evaluated using the prior-work protocol.The hybrid representation and longer sequences (312 versus 192 frames) motivate bidirectional attention; the authors frame this as an interaction among representation, sequence length, and source scale, not a universal advantage.Source: §A.6 · pp. 17–18
  • MSFLOW’s Top-1/2/3 R-Precision reaches 0.910/0.969/0.984, exceeding CMDM’s 0.831/0.926/0.958.FID is 16.342, higher than CMDM (14.451) and MoMask++ (15.061) but slightly lower than ScaleMoGen (16.350); multimodality is second-best at 12.538, behind MDM (13.412).Source: Table 9 · p. 18, §A.6 · p. 18
  • SnapMoGen evaluation embeds the first 148 root-and-rotation channels, although MSFLOW trains on and generates all 296 channels.The remaining channels provide redundant learning signals; the protocol differs from HumanML3D’s 67D evaluator.Source: §A.6 · p. 18

A.7 COMPUTE EFFICIENCY

  • Compute comparisons use each method’s official guidance and sampling schedule, native representation, and complete sampling trajectory on one NVIDIA A100-SXM4-80GB GPU.Timing is the median of five runs after two warm-ups, with batch size one and a 196-frame output.Source: §A.7 · pp. 18–19
  • For 196-frame generation, MSFLOW is second-smallest at 68.77M parameters and second-fastest at 1.742 seconds, with 1.502 TFLOPs the third-lowest.It ties FloodDiffusion for best FID at 0.057 and has the highest R-Top3 at 0.863 under the standard 263D evaluation.Source: Table 10 · p. 19, §A.7 · p. 19

B DETAILED SPATIAL-CONTROL RESULTS

The detailed spatial-control results report exact satisfaction of constrained coordinates across tested joints and strong metrics for upper-body editing. They also qualify what these measures establish about unconstrained motion.

B DETAILED SPATIAL-CONTROL RESULTS

  • Zero trajectory, location, and average errors hold across pelvis, feet, head, and wrists.The authors caution that exact projected-coordinate satisfaction and one physical metric do not establish unconstrained-motion quality.Source: B DETAILED SPATIAL-CONTROL RESULTS · p. 19
  • Across all joints, FID improves from 0.097 to 0.061 and R-Precision@3 rises from 0.779 to 0.818 versus ProjFlow (Watanabe et al., 2026).Average foot-skating ratio decreases from 0.0603 to 0.0578.Source: B DETAILED SPATIAL-CONTROL RESULTS · p. 19
  • For upper-body editing, FID falls from 0.066 to 0.034 relative to the strongest baseline.MSFLOW reports R-Precision 0.511, 0.701, and 0.794 at ranks 1–3, matching distance 3.214, and diversity 10.714.Source: B DETAILED SPATIAL-CONTROL RESULTS · p. 19

C IMPLEMENTATION AND REPRODUCIBILITY DETAILS

This section records the configurations and procedures used for MSFLOW’s text-to-motion generators and inference-time spatial control. It also identifies a checkpoint reproducibility caveat for the primary causal 263D result.

C IMPLEMENTATION AND REPRODUCIBILITY DETAILS

  • Training uses a 68M-parameter transformer with eight RA-MMDiT blocks, width 512, four attention heads, and feed-forward width 1024.The default training setup runs for 500 epochs with batch size 64, AdamW, and learning rate 2 × 10^-4.Source: C IMPLEMENTATION AND REPRODUCIBILITY… · p. 19
  • Table 11 covers text-conditioned spatial control and upper-body editing, including pelvis controls, per-joint controls, and an editing format following ProjFlow.Source: Table 11 · p. 20

C.1 CAUSAL 263D TEXT-TO-MOTION CONFIGURATION

  • The causal 263D model’s evaluated EMA checkpoint was not retained with an immutable run and epoch identifier.Table 12 gives its configuration.Source: §C.1 · p. 20, Table 12 · p. 21

C.2 BIDIRECTIONAL XYZ TEXT-TO-MOTION CONFIGURATION

C.3 TRAINING AND SAMPLING ALGORITHMS

  • Algorithms 1 and 2 specify the shared training and text-to-motion sampling procedures.Source: §C.3 · p. 21

C.4 BIDIRECTIONAL XYZ ENDPOINT-PROJECTION CONFIGURATION

  • Spatial control reuses the trained bidirectional XYZ generator’s source scale and attention graph, with arbitrary binary masks and target coordinates supplied only at inference.Table 14 summarizes this inference-only configuration, and Algorithm 3 details endpoint projection.Source: §C.4 · p. 21

D SAMPLE CODE

The appendices provide training and sampling algorithms, theoretical explanations for direct-space generation and representation-dependent attention, and empirical analyses of motion covariance and source scale. Together, these details clarify the scope and mechanisms behind the proposed design choices.

D SAMPLE CODE

  • The training algorithm samples a scaled Gaussian source and time, forms an interpolated motion, predicts the clean endpoint, and trains on stabilized residual fields; sampling updates all frames with Heun steps and a final Euler step.Source: D SAMPLE CODE · p. 22

E.1 PROOFS OF MAIN PROPOSITIONS

  • Proposition 1 formalizes an irreducible fidelity error when generated motions are restricted to a fixed decoder’s range, while Proposition 2 links source scale to signal emergence and intermediate-distribution conditioning.Source: §E.1 · p. 22

E.2 EMPIRICAL SPATIOTEMPORAL COVARIANCE AND SOURCE SCALE

  • At 64 frames, normalized motion remains anisotropic: 263D has Coff = .148, Aspec = 2116.0, and reff = .007, while XYZ has .444, 1808.7, and .001, respectively.At T0 = 64, the matched Gaussian has .007, 1.0, and 1.000 for 263D’s corresponding diagnostics.Source: §E.2 · p. 24, Table 15 · p. 24
  • SALAD is most isotropic by Coff = .076 and reff = .238, but covariance diagnostics characterize geometry rather than ranking generators by sample quality.CMDM has Aspec = 118.9 and reff = .077, compared with 263D’s 2116.0 and .007.Source: §E.2 · p. 24, Table 16 · p. 24
  • Increasing source scale from 1 to 5 lowers log10 κt for both representations at t = 0.1, 0.5, and 0.9; matched clean-prediction FID falls by 58.6% for causal 263D and 73.6% for bidirectional XYZ.The reported association does not establish that improved conditioning alone causes the FID gains or that s = 5 is universally optimal.Source: §E.2 · p. 25, Table 17 · p. 25

E.3 WHY ATTENTION DEPENDS ON THE REPRESENTATION

  • In the idealized model, future observations do not reduce Bayes risk for increments, but they strictly reduce it for absolute positions when prefix posterior variance is nonzero.HumanML3D mixes several feature types and is not a sequence of independent increments, so the proposition is a mechanism rather than a literal model of the full representation.Source: §E.3 · p. 26
  • Bidirectional context cannot increase Bayes-optimal squared error, but a context pattern that eases framewise regression need not yield the best integrated generative dynamics.Source: §E.3 · p. 26

E.4 EMPIRICAL ATTENTION ROUTING

Across selected checkpoints, bidirectional XYZ attention routes over a broader temporal span than 263D, while causal masks remove future motion access. The causal field also guarantees prefix autonomy under the stated regularity assumption.

E.4 EMPIRICAL ATTENTION ROUTING

  • At t = 0.5, bidirectional XYZ has a longer normalized span than 263D at both scales: averaged spans are 19.2% versus 11.2%, and the s = 5 values are 20.9% versus 12.7%.Across t = 0.2, 0.5, and 0.8, the span ordering remains: 22.7% versus 14.4%, 19.2% versus 11.2%, and 16.4% versus 8.0%.Source: Table 18 · p. 27, §E.4 · p. 27
  • All four selected causal checkpoints assign zero weight to future motion and prevent text queries from reading motion, whereas bidirectional XYZ uses broader two-sided context.Source: §E.4 · p. 27
  • Attention weights describe routing, not causal attribution, because values, output projections, residual paths, and subsequent layers also determine influence on generated motion.Source: §E.4 · p. 28
  • Under a locally Lipschitz causal field, each generated prefix depends only on the source prefix and text; changing the source suffix cannot alter it.Where differentiable, the motion Jacobian is block lower triangular.Source: §E.4 · p. 28
  • Causal attention updates all frames at each ODE step while preventing suffix perturbations from feeding back into an earlier prefix; bidirectional attention instead supports two-sided propagation of spatial constraints.These structural properties do not by themselves establish smaller numerical error or better samples.Source: §E.4 · p. 28

Related papers

Loading 2609.34190v1…