PaperlayerView exact revision on arXiv

Source-linked AI summary

TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis

Mathis Petrovich, Michael J. Black, Gül Varol

arXiv:2305.00976v2cs.CVcs.CL

TL;DR

Text-to-motion retrieval is less studied than related cross-modal retrieval tasks, despite offering a way to search realistic existing motions. TMR extends TEMOS with joint synthesis and contrastive retrieval training, using filtered negatives, and reports substantial improvements over prior work while demonstrating zero-shot temporal localization.

  • Problem

    Text-to-motion retrieval is a relatively little-studied cross-modal task, although searching existing motion collections can provide realistic motions and support applications such as indexing.

  • Method

    TMR extends TEMOS with joint text-to-motion synthesis and retrieval training, adding contrastive loss and filtering semantically similar negatives.

  • Results

    TMR significantly improves over prior text-motion retrieval work and supports zero-shot temporal localization on long motion sequences.

  • Takeaways & Limitations

    The results support treating text-to-motion retrieval as a standalone task and indicate that synthesis and retrieval components should be analyzed jointly.

  • Takeaways & Limitations

    The small motion-text training datasets limit generalization to in-the-wild motions, and database retrieval requires storing the full encoded training set in memory.

Abstract

from arXiv · show

In this paper, we present TMR, a simple yet effective approach for text to 3D human motion retrieval. While previous work has only treated retrieval as a proxy evaluation metric, we tackle it as a standalone task. Our method extends the state-of-the-art text-to-motion synthesis model TEMOS, and incorporates a contrastive loss to better structure the cross-modal latent space. We show that maintaining the motion generation loss, along with the contrastive training, is crucial to obtain good performance. We introduce a benchmark for evaluation and provide an in-depth analysis by reporting results on several protocols. Our extensive experiments on the KIT-ML and HumanML3D datasets show that TMR outperforms the prior work by a significant margin, for example reducing the median rank from 54 to 19. Finally, we showcase the potential of our approach on moment retrieval. Our code and models are publicly available at https://mathis.petrovich.fr/tmr.

1. Introduction

TMR treats text-to-motion retrieval as a standalone cross-modal search task, motivated by realistic reuse of existing motion data and applications such as indexing. It extends synthesis with contrastive training and addresses similar descriptions through filtered negatives, while also demonstrating zero-shot temporal localization.

  • Motivation: Text-to-motion retrieval ranks existing motions by correspondence to a natural-language query, rather than generating new motion sequences.Retrieved motions are guaranteed to be realistic because they come from a motion-capture collection.
  • Motivation: The task is relatively understudied compared with text-to-image retrieval and can support motion indexing, text labeling, and symmetric motion-to-text retrieval.A shared cross-modal embedding enables both text-to-motion and motion-to-text searches.
  • Approach: TMR extends TEMOS with a cross-modal InfoNCE loss to jointly train text-to-motion synthesis and retrieval.The synthesis branch is retained while contrastive training structures the joint latent space using negatives.
  • Approach: Similar motion descriptions make naive in-batch contrastive training problematic because treating every nonmatching pair as negative can separate semantically related samples.The paper addresses this challenge by filtering negatives using text-text similarity.
  • Additional use case: TMR also demonstrates zero-shot temporal localization by retrieving corresponding moments from long BABEL motion sequences without localization training or BABEL labels.The authors present this as a potential future research avenue.

2. Related work

Prior work has explored text-conditioned motion synthesis and motion-to-motion retrieval, but text-to-motion retrieval remains a more challenging and less developed cross-modal task. TMR builds on TEMOS and contrasts with earlier retrieval models by combining synthesis, contrastive learning, and evaluation focused on retrieval.

  • Text and human motion: Text-conditioned motion modeling adds semantic control to human-motion generation, whereas motion retrieval has received comparatively less attention.Existing retrieval examples include motion-to-motion methods such as motion matching.
  • Motion retrieval: Text-to-motion retrieval requires nearest-neighbor search across text and motion modalities, making it more challenging than motion-to-motion retrieval.The user query is textual, while the retrieved item is a motion.
  • Motion retrieval: Guo et al. train a joint embedding primarily as an evaluation measure for motion synthesis and report low performance, especially with fine-grained galleries.Their setup queries generated motions against a gallery of descriptions and uses a margin-based contrastive loss.
  • Relation to TMR: TMR builds on TEMOS by adding contrastive training with negatives, improving retrieval capability from large galleries of fine-grained motions.TEMOS aligns positive text-motion pairs but does not train with negatives.
  • Cross-modal retrieval: TMR follows cross-modal retrieval work that uses contrastive learning and draws on synthesis branches introduced in image-text models such as BLIP and CoCa.The paper uses InfoNCE for text-motion retrieval and synthesis jointly.

3. Text-to-motion retrieval

The paper defines text-to-motion retrieval as ranking motion galleries by semantic correspondence and implements TMR with dual encoders, a motion decoder, and joint synthesis-retrieval training. Its contrastive objective uses in-batch negatives while filtering semantically similar descriptions, and the total loss combines TEMOS objectives with InfoNCE.

  • Definitions: Text-to-motion retrieval ranks motions in a database by semantic correspondence to a natural-language query, with motion-to-text retrieval as its symmetric counterpart.The framework treats retrieval as a search engine for indexing motions and captions.
  • Definitions: A 3D human motion is a sequence of human poses, while descriptions may be action names, full sentences, or fine-grained multi-action text.The method is demonstrated with SMPL but is applicable to other skeleton topologies.
  • Architecture: TMR uses independent motion and text encoders plus a decoder that generates motion, extending TEMOS with retrieval-oriented losses.The encoders map both modalities into a joint latent space, and the decoder supports variable-duration non-autoregressive generation.
  • Contrastive training: The contrastive objective increases similarity for matched text-motion pairs and decreases similarity for in-batch mismatches, while excluding pairs with highly similar descriptions.The similarity matrix contains positive diagonal entries and filtered negative off-diagonal entries.
  • Contrastive training: InfoNCE uses a temperature hyperparameter τ and evaluates both text-to-motion and motion-to-text directions within the batch.The formulation averages the two directional log-ratio terms over batch items.
  • Joint objective: The total training loss is LTEMOS + λNCE LNCE, combining synthesis losses with the weighted contrastive loss.The TEMOS component includes reconstruction, KL-divergence, and cross-modal embedding similarity terms.
  • Negative filtering: The method filters potentially incorrect negatives rather than converting them into positives because sentence-similarity estimates can also be wrong.MPNet sentence embeddings provide the similarity scores used for thresholding.

4. Experiments

Experiments establish benchmarks on HumanML3D and KIT-ML, compare TMR with prior methods, and analyze losses, negative filtering, qualitative retrieval, and moment-retrieval potential.

  • Benchmark and comparison: TMR substantially outperforms Guo et al. and TEMOS across four evaluation protocols on HumanML3D and KIT-ML.The protocols vary gallery construction and difficulty, with retrieval evaluated on unseen test motions.
  • Ablation study: 41.93 vs 36.87 R@3 shows that retaining the motion synthesis branch improves contrastive retrieval.The synthesis branch may encourage latent vectors to capture the full text content rather than a subset of words.
  • Ablation study: 41.93 vs 34.46 R@1 shows that InfoNCE outperforms the margin loss when both use the same negative filtering.The controlled comparison isolates the contrastive-loss formulation while keeping negative filtering fixed.
  • Ablation study: A text-similarity threshold of 0.8 performs best overall by removing wrong negatives while retaining sufficient negatives.Without filtering, performance is 36.02 R@3 compared with 41.93.
  • Qualitative results: Qualitative retrieval finds visually matching motions, including unseen-gallery examples and swimming motions for a free-form query absent from the gallery.For the swimming example, the model retrieves swimming-like motions among the top three, while incorrect motions have similarity below 0.6.
  • Moment retrieval: A model trained for retrieval shows qualitative potential for moment retrieval by scoring sliding 20-frame windows over long BABEL motion sequences.The evaluation uses an H3D-pretrained model despite a domain gap between BABEL labels and H3D training data.
  • Limitations: Generalization to in-the-wild motions is limited because the motion-text training data is much smaller than image-text foundation-model datasets.The paper cites approximately 23K motions in HumanML3D as an example of this data-scale limitation.

5. Conclusion

The paper frames TMR as a framework for jointly training text-to-motion retrieval and synthesis, emphasizing fine-grained negative selection. It reports improvements over prior work and identifies language synthesis as future work.

  • TMR jointly trains text-to-motion retrieval and text-to-motion synthesis.
  • The framework defines negatives with special attention to fine-grained similarities in motion-language databases.
  • The experiments report significant improvement over prior work and analyze the importance of each component.
  • Future work may add a language synthesis branch alongside the motion synthesis branch.

APPENDIX

The appendix contains statistical analyses, additional experiments, and further qualitative results, alongside a supplementary video demonstrating retrieval and moment retrieval applications.

  • The appendix includes statistical analysis, additional experimental results, and further qualitative results.
  • The supplementary video demonstrates text-to-motion retrieval on KIT and H3D and moment retrieval on BABEL.

A. Statistics

The appendix quantifies how often similar text descriptions occur in the test sets under the evaluation protocol’s similarity threshold.

  • Protocol (b) accepts a retrieved motion as correct when its text is similar to the query above a 0.95 threshold.

B. Additional experimental results

Additional experiments examine synthesis, negative filtering, latent dimensionality, and zero-shot moment retrieval. They report benefits from negative filtering and the synthesis branch, while showing localization potential despite longer-sequence noise.

  • Negative filtering: Filtering negatives at a 0.8 text-similarity threshold performs well overall, while filtered-pair percentages decrease with higher thresholds.
  • Motion synthesis results: The synthesis appendix compares Guo Syn., TEMOS, TMR’s synthesis branch, and real motions across KIT-ML and H3D.
  • Latent dimensionality: A latent dimensionality of d=128 performs overall best, although the main experiments use d=256 as in TEMOS.
  • Contrastive-only baseline: Negative filtering improves results for both the contrastive-only model and TMR with its synthesis branch.
  • Contrastive-only baseline: The added synthesis branch empirically improves results consistently, paralleling findings from text-to-image multimodal models.
  • Moment retrieval: Zero-shot moment retrieval localizes queries by comparing text with temporal motion crops, despite no training on this task or BABEL text.
  • Moment retrieval: On very long sequences, similarity plots become noisier as the search space grows, but maximum similarity still occurs at the ground-truth location.

C. Additional qualitative results

Additional qualitative results evaluate text-to-motion retrieval on H3D across four protocols, from the full gallery to smaller or easier candidate sets. The examples show semantically coherent retrieval, while observed ranks vary with protocol difficulty and correctness criteria.

  • Protocol design: Four protocols evaluate retrieval on H3D, including full-gallery, similarity-threshold, dissimilar-text, and random-batch settings.Protocols (a) and (b) use all 4,380 motions; (c) uses 100 dissimilar texts; (d) randomly samples batches of 32.
  • Protocol design: Protocol (b) can treat a motion as correct when its text similarity exceeds 0.95, unlike protocol (a)'s stricter correctness criterion.This distinction explains why a semantically coherent retrieval can receive different ranks under the two full-gallery protocols.
  • Qualitative retrieval: The model retrieves motions semantically similar to the text descriptions across the qualitative examples.The examples are randomly chosen and therefore presented as representative of their corresponding protocols.
  • Protocol comparison: Performance improves from harder to easier protocols, with smaller or more dissimilar galleries producing better observed retrieval behavior.Random batches of 32 make the gallery very small, while protocol (c) uses only the most dissimilar 100 texts.
  • Full-gallery results: In full-gallery examples, correct motions are ranked at 31, 10, and 138, despite other retrieved motions remaining reasonable or coherent.The figures also include an example where the model generalizes to “rocking a baby” without seeing that exact training text.
Loading 2305.00976v2…