PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Number-Free Text-to-Motion Generation Via Flow Matching

Guanhe Huang, Oya Celiktutan

arXiv:2603.27040v1cs.CV

TL;DR

Text-to-motion models struggle to generate realistic motion for arbitrary numbers of agents because interaction data are scarce and autoregressive methods can be inefficient and accumulate errors. UMF unifies heterogeneous motion data in a multi-token latent space and combines P-Flow for motion priors with S-Flow for reactions. It achieves state-of-the-art multi-person generation and robust zero-shot generalization, while remaining constrained to medium-sized groups centered on a primary agent.

  • Problem

    Existing text-to-motion methods struggle to generalize to arbitrary agent counts because interaction datasets are smaller and less diverse than single-person datasets.

  • Method

    UMF unifies heterogeneous motion datasets in a multi-token latent space and uses P-Flow for hierarchical motion-prior generation and S-Flow for probabilistic reaction synthesis.

  • Results

    UMF achieves state-of-the-art multi-person generation, including FID 4.772 on the InterHuman benchmark, and demonstrates zero-shot generalization to unseen group scenarios.

  • Takeaways & Limitations

    UMF provides a generalist framework for number-free text-to-motion generation across heterogeneous single-person and multi-person motion data.

  • Takeaways & Limitations

    UMF is mainly trained and evaluated on dual-agent scenarios because SMPL-based datasets with three or more interacting agents are scarce.

Abstract

from arXiv · show

Generative models excel at motion synthesis for a fixed number of agents but struggle to generalize with variable agents. Based on limited, domain-specific data, existing methods employ autoregressive models to generate motion recursively, which suffer from inefficiency and error accumulation. We propose Unified Motion Flow (UMF), which consists of Pyramid Motion Flow (P-Flow) and Semi-Noise Motion Flow (S-Flow). UMF decomposes the number-free motion generation into a single-pass motion prior generation stage and multi-pass reaction generation stages. Specifically, UMF utilizes a unified latent space to bridge the distribution gap between heterogeneous motion datasets, enabling effective unified training. For motion prior generation, P-Flow operates on hierarchical resolutions conditioned on different noise levels, thereby mitigating computational overheads. For reaction generation, S-Flow learns a joint probabilistic path that adaptively performs reaction transformation and context reconstruction, alleviating error accumulation. Extensive results and user studies demonstrate UMF' s effectiveness as a generalist model for multi-person motion generation from text. Project page: https://githubhgh.github.io/umf/.

1. Introduction

Number-free text-to-motion generation remains difficult because existing approaches are constrained by fixed agent counts, scarce heterogeneous interaction data, and autoregressive error accumulation. UMF addresses these issues with a unified latent space, hierarchical motion-prior generation, and semi-noise reaction flow.

  • Motivation: Number-free text-to-motion generation must handle arbitrary agent counts, but existing methods struggle with unseen crowded scenes and limited motion data.These limitations affect applications requiring transitions between independent and collaborative tasks.
  • Motivation: Interaction datasets are smaller and less diverse than single-person datasets, although overlapping basic movements allow single-person data to serve as a heterogeneous interaction prior.This overlap motivates unified use of heterogeneous motion data.
  • Unified Motion Flow: UMF unifies heterogeneous single-person and multi-person datasets in a multi-token latent space for number-free, text-conditional generation.The latent space is intended to stabilize flow matching across HumanML3D and InterHuman data.
  • Unified Motion Flow: UMF decomposes generation into single-pass motion-prior initialization with P-Flow and subsequent multi-pass reaction transformation with S-Flow.This separates prior generation from reaction generation while mapping text, motion prior, and reaction.
  • Pyramid Motion Flow: P-Flow uses hierarchical resolutions conditioned on timestep noise levels to reduce multi-token computational overhead while maintaining high-fidelity generation.It preserves original resolution later and downsamples earlier, noisier stages.
  • Semi-Noise Motion Flow: S-Flow learns a joint probabilistic path between generated context and subsequent reactions, alleviating error accumulation in autoregressive interaction synthesis.The method balances reaction transformation with context reconstruction.

2. Related Work

Related work progresses from single- and dual-agent motion synthesis toward generalist models, but existing number-free approaches remain limited by agent count, inefficiency, or error accumulation.

  • Motion Synthesis: Most human motion generation methods focus on single-agent or dual-agent scenarios, while newer work explores controllability and dynamic interaction modeling.The related methods include masked motion generation and interleaved learning strategies.
  • Unified Generation: Unified motion generation research spans motion understanding, diverse input modalities, and variable-agent handling, but dualFlow remains limited to dual-agent scenarios.These directions reflect broader efforts toward generalist motion models.
  • Number-Free Generation: FreeMotion enables number-free motion generation through decoupled generation and interaction modules but suffers from inefficiency and error accumulation in multi-person scenarios.FreeDance also addresses number-free generation, but targets music-to-motion rather than text-to-motion.

3. Preliminaries

Flow matching learns a velocity field that transports a source distribution to a target distribution through an ordinary differential equation. Its simulation-free objective regresses this field toward a conditional vector field along a chosen probability path, enabling motion-prior and reaction generation.

  • Flow Matching: Flow generative models learn a velocity field v_t that maps source samples x_0 from p to target samples x_1 from q through an ODE.The ODE describes the continuous transport between source and target distributions.
  • Flow Matching: Flow matching trains by directly regressing the model velocity field v_t(x_t) toward a conditional vector field u_t(x_t|x_1).This provides a simulation-free training objective.
  • Probability Paths: The conditional vector field uniquely determines a conditional probability path p_t(·|x_1) toward the data sample x_1.Choosing the probability path specifies how distributions are interpolated during training.
  • Probability Paths: Linear interpolation uses x_t = t x_1 + (1-t)x_0 to connect data and noise over time.This is presented as an effective conditional probability path.
  • Probability Paths: The interpolation distribution is modeled as x_t ∼ N(t x_1, (1-t)^2 I), with conditional velocity u(x_t|x_1) = x_1 − x_0.Flow matching can also interpolate between non-Gaussian distributions, supporting both motion-prior and reaction generation.

4. Proposed Method

UMF builds number-free text-to-motion generation on a unified multi-token latent space, then separates motion-prior generation from autoregressive reaction generation. P-Flow uses hierarchical resolutions for efficient prior synthesis, while S-Flow jointly models reaction transformation and context reconstruction.

  • Unified Latent Space: UMF converts heterogeneous individual and interaction motions into a unified non-canonical representation and regularized multi-token latent space.The tokenizer and VAE are designed to bridge heterogeneous data formats and stabilize flow matching across single-agent and multi-agent datasets.
  • Unified Latent Space: The VAE uses multiple latent tokens and geometric, reconstruction, and KL losses to preserve motion details and interaction relationships.The latent adapter decouples internal token representation from the final latent dimension, while geometric loss improves physical plausibility.
  • Unified Motion Flow: UMF decouples number-free generation into single-pass motion-prior initialization with P-Flow and subsequent multi-pass reaction transformation with S-Flow.P-Flow generates an individual motion prior, after which S-Flow generates reactions for subsequent agents.
  • Pyramid Motion Flow: P-Flow divides flow matching into hierarchical resolution stages, beginning with downsampled latents and progressively producing finer-grained endpoints from noise to data.Each time window interpolates between successive resolutions, with coupled endpoint noise encouraging a straighter flow trajectory.
  • Pyramid Motion Flow: P-Flow concentrates computation at lower resolutions and theoretically reduces cost by a factor of ≈1/K while retaining original resolution at later stages.Euler ODE sampling advances through discretized pyramid stages, with rescaling and renoising used to maintain continuity between resolutions.
  • Semi-Noise Motion Flow: S-Flow uses a context adapter and jointly optimizes reaction transformation and context reconstruction to adapt reaction generation to previously generated motions.The auxiliary reconstruction path helps model context globally and is intended to alleviate error accumulation in autoregressive generation.

5. Experiments

UMF outperforms generalist and specialist baselines on multi-person benchmarks, generates more plausible unseen multi-agent interactions, and is preferred in user studies. Ablations support heterogeneous priors, latent adapters, Pyramid Flow efficiency, and Semi-Noise Flow design choices.

  • Quantitative Results: UMF improves Top3 R-Precision by 28% and reduces FID by 29% versus FreeMotion on InterHuman.It also outperforms InterMask by 7% in FID and achieves competitive R-Precision and MM-Distance.
  • Quantitative Results: UMF improves Top3 R-Precision by over 30% and reduces MM-Distance by 27% versus ReGenNet on InterHuman-AS.The results indicate improved reactive motion quality on the action-reaction benchmark.
  • Qualitative Results & User Study: UMF produces plausible kicking, strolling, and fighting interactions, while FreeMotion shows incoherent motion, interpenetration, or static-agent artifacts.These examples test dual-agent, three-agent, and N > 3 zero-shot scenarios.
  • Ablation Studies: HumanML3D priors improve text adherence and motion fidelity, while the Latent Adapter is crucial for multi-token flow matching and single-token capacity is insufficient for number-free generation.The ablation attributes the modest prior improvement to the complexity gap between single-agent and multi-agent targets.
  • Ablation Studies: With 60 inference steps, UMF uses fewer FLOPs and is nearly 5× faster than FreeMotion, whereas reducing P-Flow steps nearly halves FLOPs but degrades performance.Sharing the transformer backbone between S-Flow and P-Flow, removing reconstruction loss, or replacing the Context Adapter with ControlNet harms fidelity.

6. Conclusion

UMF combines P-Flow and S-Flow in a unified heterogeneous latent space for number-free text-conditioned motion generation. It achieves state-of-the-art multi-person performance and robust zero-shot generalization, but remains limited to medium-sized groups of approximately 10 agents centered on one primary agent.

  • Conclusion: UMF uses P-Flow to mitigate computational overheads and S-Flow to alleviate error accumulation during number-free motion generation.Both modules operate on a unified heterogeneous latent space.
  • Conclusion: UMF achieves state-of-the-art performance for multi-person generation and robust zero-shot generalization to challenging group scenarios.The conclusion reports these outcomes without specifying an additional metric.
  • Conclusion: UMF remains constrained to medium-sized group interactions of approximately 10 agents centered on a primary agent.The paper identifies scaling to dense crowd dynamics of approximately 100 agents as future work.
Loading 2603.27040v1…