PaperlayerView exact revision on arXiv

Source-linked AI summary

BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion

Wanjiang Weng, Yongliang Wu, Xiaofeng Tan, Xingyu Zhu, Wenbo Zhu, Hongsong Wang

arXiv:2609.35407v2cs.CV

TL;DR

Autoregressive unified motion-text models impose a fixed generation order, limiting their fit to bidirectional language-motion dependencies. BiMoGen combines masked diffusion with decoupled training and generation-aware self-correction, and reports competitive results across both tasks and datasets.

  • Problem

    Autoregressive unified models impose a fixed generation order, leaving early errors in context and risking degraded temporal coherence and motion-text consistency.

    Source: §1 · pp. 1–2

  • Method

    BiMoGen uses shared masked prediction, paired-sequence pretraining followed by supervised fine-tuning, and self-correction from generated context during training and early sampling.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    BiMoGen reports competitive performance in both directions on HumanML3D and KIT-ML; on HumanML3D T2M, it leads unified models at R@1 0.555 and R@3 0.841.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    The experiments support masked bidirectional modeling as a competitive shared formulation for motion generation and captioning.

    Source: §5 · p. 10

    Generation quality remains partly bounded by VQ-VAE reconstruction fidelity and codebook expressiveness, and the sequence-level formulation lacks fine-grained control over individual body parts.

    Source: §5 · p. 10

Abstract

from arXiv · show

Text-to-motion generation and motion-to-text captioning are two fundamental tasks in human motion modeling, both grounded in the same underlying motion-text correspondence. Existing unified approaches mostly rely on autoregressive modeling, which imposes a fixed generation order and is therefore poorly suited to the bidirectional dependencies between language and motion, allowing early prediction errors to persist as fixed context and degrade both temporal coherence and cross-modal consistency. Masked discrete diffusion, which models sequences through iterative bidirectional prediction, allows prediction from bidirectional context without a fixed generation order. We therefore propose BiMoGen (Bidirectional Motion-text Generation), a unified masked discrete diffusion framework for bidirectional motion-text modeling. To stabilize training, we design Decoupled Uni- and Cross-Modal Training, in which masked pretraining first establishes cross-modal correspondence on paired motion-text sequences, after which supervised fine-tuning specializes the model for bidirectional generation. Masked diffusion nonetheless faces a train-test mismatch, as the model is trained on clean ground-truth context yet encounters self-generated and potentially erroneous context at inference, with errors committed under heavily masked states propagating through subsequent steps. We further introduce Generation-Aware Self-Correction that exposes the model to its own predictions during training and uses correction passes at early sampling steps for further refinement of previously unmasked tokens. Extensive experiments on HumanML3D and KIT-ML demonstrate competitive performance on both tasks, validating the effectiveness of the proposed two-stage training and self-correction designs. The project page is available at https://wengwanjiang.github.io/BiMoGen-Page.

1 Introduction

Unified motion-text modeling seeks to use shared representations for text-to-motion generation and motion-to-text captioning, but autoregressive decoding imposes a fixed order that prevents revision of early errors. BiMoGen uses masked discrete diffusion and separates correspondence learning from generation training, while adding self-correction for predictions made from generated context.

1 Introduction

  • Autoregressive unified models impose a fixed left-to-right order, so early errors remain in context and can degrade temporal coherence and motion-text consistency.Masked discrete diffusion instead predicts masked tokens from bidirectional context without a fixed generation order.Source: §1 · pp. 1–2
  • Unified masked diffusion faces two challenges: disentangling correspondence learning from conditional generation and handling self-generated context at inference.The mismatch is most severe under high mask ratios early in sampling, when committed errors can cascade into later predictions.Source: §1 · p. 2
  • BiMoGen uses masked pretraining on paired motion-text sequences before supervised fine-tuning specializes the model for bidirectional generation.Generation-Aware Self-Correction trains recovery from the model’s own predicted context and adds correction passes early in sampling.Source: §1 · p. 2

2 Related Works

Prior work spans continuous and token-based motion generation, unified motion-text models, and masked discrete diffusion. BiMoGen’s related-work distinction is its focus on train-test misalignment and error accumulation in bidirectional motion-text modeling.

Text-to-Motion Generation

  • Text-to-motion methods include continuous diffusion, such as MDM [45] and MLD [6], and token-based approaches including T2M-GPT and MoMask.BAMM [37] combines autoregressive generation with bidirectional refinement, while BiPO uses partial occlusion and selective token regeneration.Source: §2 · pp. 2–3
  • These methods are tailored to text-to-motion synthesis and do not provide motion-to-text captioning or a shared formulation for both directions.The cited bidirectional predictors train on ground-truth context rather than self-generated context.Source: §2 · p. 3

Unified Motion-Text Modeling

  • Unified work includes TM2T’s separate translators [14], MotionGPT’s autoregressive Transformer [21], MotionGPT3’s diffusion head [73], and Unimotion’s joint diffusion approach [25].DiMo [68] also uses masked discrete diffusion, with an RVQ predictor and reinforcement-learning fine-tuning.Source: §2 · p. 3
  • BiMoGen focuses on masked-diffusion train-test misalignment and error accumulation, using generation-aware self-correction during training and sampling.The related-work passage describes its design as a single Transformer over a single-layer VQ-VAE.Source: §2 · p. 3

3 Method

BiMoGen represents motion and text in a shared token sequence and uses masked bidirectional prediction for both generation directions. Its training separates correspondence learning from conditional generation, while self-correction trains on model predictions and revises committed tokens during sampling.

3.1 Unified Masked Discrete Diffusion Model

  • Motion and text tokens share one sequence, which a bidirectional Transformer trains to predict at masked positions and iteratively completes from visible context.A pretrained VQ-VAE tokenizes motion; the motion vocabulary is appended to the text vocabulary, and the two are concatenated for masked diffusion.Source: §3.1 · pp. 3–4

3.2 Decoupled Uni- and Cross-Modal Training

  • DUCMT first pretrains on symmetrically corrupted paired sequences to learn correspondence, then fine-tunes with the condition visible and only the target corrupted for T2M or M2T.Paired modalities are concatenated in either order with equal probability; an auxiliary M2M objective adds dense intra-motion supervision.Source: §3.2 · pp. 4–5

3.3 Generation-Aware Self-Correction

  • GASC replaces tokens in the generation region with detached model predictions, then trains the model to recover ground truth across that region, including already committed positions.The correction loss covers the entire generation region because committed tokens can also be incorrect and should remain revisable.Source: §3.3 · p. 5
  • At sampling, the model progressively unmasks high-confidence target tokens and periodically predicts a complete candidate, corrects it, and writes revisions only to already unmasked positions.The target begins fully masked; T2M uses a given target length, while M2T generates up to a maximum length and truncates at the first [EOS].Source: §3.3 · pp. 5–6

4 Experiments

Experiments evaluate BiMoGen on HumanML3D and KIT-ML using established motion representations and task-specific retrieval, motion-quality, and caption-quality metrics. Results and ablations examine bidirectional performance, training stages, self-correction, sampling cost, tokenizers, and user preferences.

4.2 Main Results

  • Among unified models, BiMoGen achieves T2M R@1 of 0.555 and R@3 of 0.841, and M2T CIDEr of 60.2 and BERTScore of 38.5 on HumanML3D.T2M FID is 0.069 and MM Dist is 2.733; DiMo has lower FID, but differences preclude attributing that gap solely to tokenization.Source: §4.2 · p. 7

4.3 Ablation and Analysis

  • PT alone learns cross-modal correspondence but is insufficient for bidirectional generation; PT before SFT improves MM Dist from 3.259 to 3.017, FID from 0.325 to 0.102, and M2T R@1 from 0.474 to 0.547.SFT is necessary to produce valid target sequences.Source: Table 3 · p. 7, §4.3 · p. 8
  • With sampling SC disabled, training SC raises R@1 from 0.514 to 0.551, lowers FID from 0.102 to 0.070, and improves CIDEr from 44.0 to 58.8.Sampling SC alone degrades these metrics; after training with SC, sampling correction further raises R@1 to 0.555 and CIDEr to 60.2.Source: §4.3 · p. 8, Table 5 · p. 9
  • Applying sampling SC at T/4 and T/2 raises T2M R@1 and R@3 to 0.555 and 0.841 and lowers MM Dist to 2.733, with latency increasing from 0.55s to 0.67s.It also improves M2T R@1, BLEU@4, and CIDEr over Disabled; applying SC at every step degrades T2M retrieval.Source: §4.3 · pp. 8–9, Table 6 · p. 9
  • SC modification rates fall from 31.82% at T/4 to 22.38% at T, while revised-token stability stays between 82.73% and 84.70%.The authors report more frequent early corrections and a larger fraction of revised tokens moving toward ground truth.Source: §4.3 · p. 9, Table 7 · p. 10
  • Increasing sampling steps from 5 to 20 lowers T2M FID from 0.118 to 0.069 and raises M2T BERTScore from 28.9 to 38.5; 20 steps use 1.08T FLOPs.This setting gives the best overall balance reported; MotionGPT and DiMo require 7.45T and 2.56T FLOPs, respectively, and 30 steps bring no consistent gain.Source: Table 4 · p. 7, §4.3 · p. 9
  • Replacing the whole-body tokenizer with Part VQ-VAE worsens performance, while flattened RVQ yields an FID of 0.487 under the same single-Transformer design.The authors attribute the drop to added learning difficulty from part-wise or longer token sequences and support whole-body tokenization for this evaluated framework.Source: §4.3 · pp. 9–10, Table 8 · p. 10
  • BiMoGen has the highest preference rates across all three user-study criteria, and adding SC further improves preference.The study used 30 HumanML3D test prompts; the authors associate the preference improvement with text alignment, physical plausibility, and temporal consistency.Source: Figure 5 · p. 9, §4.3 · p. 10

5 Conclusion

BiMoGen uses shared iterative masked prediction for text-to-motion and motion-to-text, with DUCMT establishing correspondence before conditional fine-tuning and GASC addressing self-generated context. Experiments show gains in both directions, while motion-tokenizer fidelity and lack of part-level control constrain the current formulation.

5 Conclusion

  • Experiments on HumanML3D and KIT-ML show consistent gains in both directions, with competitive performance among unified models.BiMoGen uses a shared motion-text token space with iterative masked prediction; DUCMT establishes correspondence before conditional fine-tuning, and GASC trains recovery from self-generated context and supports sampling refinement.Source: §5 · p. 10
  • Motion quality remains partly bounded by VQ-VAE reconstruction fidelity and codebook expressiveness, especially for subtle or highly detailed motions.The current sequence-level formulation also lacks explicit fine-grained control over individual body parts; stronger part-aware tokenizers and controllable representations are proposed as future directions.Source: §5 · p. 10

Supplementary Material

The supplement collects additional visualizations, experimental details, ablations, user-study protocol, and sampling pseudo-code. Its visualizations cover text-to-motion generation and motion in-betweening, including comparisons of self-correction and transition quality.

Supplementary Material

  • The supplement also includes further experimental details, ablations on sampling and model choices, a user-study protocol, and self-correction sampling pseudo-code.It additionally presents qualitative results for text-to-motion generation and motion in-betweening.Source: Supplementary Material · p. 16, A More Visualization · p. 16

A More Visualization

  • Fig. S1 compares text-to-motion outputs on HumanML3D; red text marks prompt descriptions not correctly reflected in the generated motion.BiMoGen with self-correction produces motions more faithful to descriptions than BiMoGen without it, especially for fine-grained semantics and directional cues.Source: A More Visualization · p. 16, Figure S1 · p. 16
  • Fig. S2 shows in-betweening with fixed start and end segments, and BiMoGen produces smooth, semantically consistent transitions without an architectural change.The visualization identifies the input human in blue and generated human in orange.Source: A More Visualization · p. 16, Figure S2 · p. 17

B.1 More Experiment Details

The evaluation details define measures for text-motion alignment, motion realism and diversity, and caption quality. They use feature-based evaluation for T2M and standard NLP metrics plus retrieval-based R Precision for M2T.

B.1 More Experiment Details

  • R@k measures the fraction of samples whose paired item ranks among the top k retrieval candidates, while MM Dist averages paired text-motion embedding distances.Candidates from the other modality are ranked by evaluator distance for each query.Source: §B.1 · p. 17
  • For T2M, Fréchet Inception Distance measures the distributional distance between generated and ground-truth motions, while Diversity measures variation among generated motions in evaluator feature space.The protocol also evaluates text-motion alignment; feature-based metrics use the official HumanML3D evaluator unless otherwise noted.Source: §B.1 · p. 17
  • For M2T, BLEU and ROUGE-L measure lexical overlap, CIDEr reference-caption consensus, and BERTScore embedding-based semantic similarity.These metrics are used together because a single overlap metric does not fully capture caption quality; R Precision also measures alignment of generated texts with corresponding motions.Source: §B.1 · p. 17

B.2 Additional Ablation Study

The supplement reports additional ablation studies on BiMoGen design choices, but this passage does not specify their findings.

B.2 Additional Ablation Study

  • Additional ablation studies examine BiMoGen design choices, with no specific choices or outcomes given here.Source: §B.2 · p. 17

B.2.1 Training Mixture

The training-mixture comparison indicates a trade-off between T2M-only training and unified training, while changing the mixture prevents isolating M2M’s contribution.

B.2.1 Training Mixture

  • Removing M2M in the 8:2:0 mixture changes R@1 from 0.555 to 0.522, CIDEr from 60.2 to 58.8, and FID from 0.069 to 0.133.The table compares T2M:M2T:M2M ratios; the M2T proportion also changes in this comparison.Source: §B.2.1 · pp. 17–18, Table S1 · p. 18
  • T2M-only training at 10:0:0 yields stronger T2M results than unified training at 8:1:1.The authors describe this as a trade-off in supporting both generation directions.Source: §B.2.1 · p. 17, Table S1 · p. 18
  • Because the M2T proportion also changes, the 8:2:0 comparison does not isolate M2M’s effect.Source: §B.2.1 · pp. 17–18

B.2.2 Backbone and Sampling Strategy

The appendix compares backbone and sampling choices, describes the user-study protocol and self-correction procedure, and notes potential applications and deployment considerations.

B.2.2 Backbone and Sampling Strategy

  • BiMoGen, trained from scratch, is on par with BERT-Large and outperforms BERT-Base on most T2M metrics, with best R@1, R@3, Div, and MM Dist.The BERT variants use pretrained checkpoints.Source: §B.2.2 · p. 18

B.3 User Study Protocol

  • The user study sampled 30 HumanML3D test descriptions, recruited 15 users, and presented generated motions in random order for comparison.The protocol covers baseline preferences and a self-correction ablation, with a “Same” option for the latter.Source: §B.3 · p. 18

B.2.2 Backbone and Sampling Strategy

  • Increasing CFG scale from 1.0 to 4.0 improves R@1/R@3 from 0.505/0.795 to 0.551/0.838, lowers FID from 0.323 to 0.070, and lowers MM Dist from 3.305 to 2.745.Further increasing the scale to 5.0 slightly degrades these metrics; the authors use w = 4.0 as the default.Source: §B.2.2 · p. 18, Table S3 · p. 19
  • Parallel sampling performs best across metrics; among sequential block-wise variants, smaller blocks perform better than larger ones.The comparison contrasts parallel masked decoding with blocks denoised in parallel but generated sequentially.Source: §B.2.2 · p. 18, Table S4 · p. 19

C Sampling with Self-Correction

  • At selected correction steps, self-correction temporarily completes the target, revises visible tokens, and leaves remaining masked tokens for later steps.Remasking targets low-confidence predictions only at positions masked at the start of the current step; earlier unmasked tokens are preserved.Source: C Sampling with Self-Correction · p. 19

D Broader Impact

  • The authors identify motion content creation, animation, virtual agents, and retrieval as potential benefits, while urging responsible deployment and disclosure of synthetic content.They note that generated motions may enter downstream avatar or animation systems.Source: D Broader Impact · p. 19

NeurIPS Paper Checklist

The checklist reports that the paper’s claims and experimental descriptions are supported, while identifying omissions in compute reporting, asset licensing, and participant instructions.

NeurIPS Paper Checklist

  • The abstract and introduction accurately reflect the contributions, which the authors say are supported by experiments on HumanML3D and KIT-ML.Source: NeurIPS Paper Checklist · p. 20
  • The paper discusses limitations including VQ-VAE tokenization dependence and lack of explicit fine-grained body-part control.Source: NeurIPS Paper Checklist · p. 20
  • The paper reports datasets, protocols, model and training settings, sampling details, and a single NVIDIA H100, but does not provide full reproduction scripts.The checklist marks the experimental-setting details as specified and open access to code and data as incomplete.Source: NeurIPS Paper Checklist · pp. 21–23
  • T2M and M2T results are averaged over 20 evaluation runs rather than reported as single-run estimates.Source: NeurIPS Paper Checklist · p. 22
  • The checklist marks compute-resource reporting insufficient, although the paper reports model parameters, FLOPs, latency, and use of one NVIDIA H100.Source: NeurIPS Paper Checklist · p. 23
  • The checklist notes that asset licenses, versions, and terms of use are not explicitly listed, and that full user-study instructions are omitted.Appendix B.3 summarizes the user-study protocol, and participants received no monetary compensation.Source: NeurIPS Paper Checklist · pp. 24–25

Related papers

Loading 2609.35407v2…