Source-linked AI summary
MotionCLIP: Exposing Human Motion Generation to CLIP Space
Guy Tevet, Brian Gordon, Amir Hertz, Amit H. Bermano, Daniel Cohen-Or
TL;DR
MotionCLIP addresses the limited semantic coverage of labeled human-motion data and its weak support for rich natural-language descriptions. It trains a transformer motion auto-encoder aligned with CLIP text and rendered-image representations, producing a semantic, disentangled latent space. The resulting system supports out-of-domain text-to-motion, compositionality, interpolation, editing, and recognition, while remaining limited on directions, some styles, and certain cultural references.
Problem
Labeled motion data does not generalize well to the full richness of the human motion manifold or the natural language describing it.
Method
MotionCLIP trains a transformer-based motion auto-encoder to reconstruct motion while aligning its latent representation with CLIP text labels and rendered images.
Results
MotionCLIP provides semantic text-to-motion generation, including out-of-domain and abstract descriptions, while its latent space supports compositionality, interpolation, editing, and recognition.
Takeaways & Limitations
Aligning motion with CLIP transfers semantic knowledge and disentanglement into the motion latent space despite CLIP never seeing motion data.
Takeaways & Limitations
MotionCLIP struggles with directions, some motion styles, and consistency for certain out-of-domain cultural references.
Abstract
from arXiv · showhide
We introduce MotionCLIP, a 3D human motion auto-encoder featuring a latent embedding that is disentangled, well behaved, and supports highly semantic textual descriptions. MotionCLIP gains its unique power by aligning its latent space with that of the Contrastive Language-Image Pre-training (CLIP) model. Aligning the human motion manifold to CLIP space implicitly infuses the extremely rich semantic knowledge of CLIP into the manifold. In particular, it helps continuity by placing semantically similar motions close to one another, and disentanglement, which is inherited from the CLIP-space structure. MotionCLIP comprises a transformer-based motion auto-encoder, trained to reconstruct motion while being aligned to its text label's position in CLIP-space. We further leverage CLIP's unique visual understanding and inject an even stronger signal through aligning motion to rendered frames in a self-supervised manner. We show that although CLIP has never seen the motion domain, MotionCLIP offers unprecedented text-to-motion abilities, allowing out-of-domain actions, disentangled editing, and abstract language specification. For example, the text prompt "couch" is decoded into a sitting down motion, due to lingual similarity, and the prompt "Spiderman" results in a web-swinging-like solution that is far from seen during training. In addition, we show how the introduced latent space can be leveraged for motion interpolation, editing and recognition.
1 INTRODUCTION
MotionCLIP addresses the limited semantic coverage of labeled motion data by aligning a motion auto-encoder with CLIP’s visual-textual latent space. This alignment supports semantic generation, compositionality, interpolation, editing, and recognition, including descriptions beyond the training domain.
- Motivation: Labeled motion datasets do not generalize well to the full richness of human motion or to natural-language descriptions.Existing data may cover substantial motion but lacks sufficient semantic and descriptive breadth.
- Approach: MotionCLIP aligns a transformer-based 3D motion auto-encoder with CLIP text and visual representations.The model reconstructs motion while placing its latent representation in CLIP space, with rendered frames providing an additional self-supervised alignment signal.
- Semantic control: CLIP alignment benefits semantic description and supports text-to-motion and motion style transfer.The method leverages CLIP’s semantic structure despite CLIP not being trained on the motion domain.
- Latent-space capabilities: The learned latent space demonstrates compositionality, semantic interpolation, and natural linear editing of motion.These properties are attributed to semantic knowledge and inherited disentanglement in CLIP space.
- Applications: MotionCLIP accepts arbitrary natural-language, abstract intent, and pop-culture descriptions, producing unseen motions such as web-swinging for “Spiderman.”Other examples include flapping motion for “wings” and a tennis serve for “Williams sisters.”
3 METHOD
MotionCLIP learns a reconstructive motion representation in CLIP space using transformer encoding and decoding, text and rendered-image alignment, and motion reconstruction losses. The shared latent space supports text-to-motion generation, style transfer, and action recognition.
- Architecture: MotionCLIP learns both encoding from motion sequences into a semantic latent representation and decoding back into explicit motion.The auto-encoder is designed as a basis for generation and editing tasks.
- Training objectives: Training combines motion reconstruction with cosine-distance alignment to CLIP embeddings of text labels and rendered images.Text-motion pairs come from labeled datasets, while images are produced by rendering a pose from a motion sequence.
- Applications: At inference, CLIP text embeddings can be decoded directly into motion, while latent arithmetic supports style transfer and encoding supports action classification.Style transfer adds a style vector to a content representation before decoding.
- Architecture: A transformer encoder maps a motion sequence to a latent vector, while a transformer decoder reconstructs the sequence from that vector.The decoder attends to the latent representation and predicts the motion sequence frame by frame.
- Training objectives: The reconstruction objective uses L2 losses on joint orientations, joint velocities, and vertex locations.These terms preserve both pose and motion dynamics during auto-encoding.
4 RESULTS
MotionCLIP uses CLIP-aligned motion representations to support text-to-motion generation, style generation, abstract language conditioning, and structured latent-space applications. Evaluations show strong text-generation preferences, semantic interpolation and editing, and action-recognition performance close to a dedicated method.
- Text-to-Motion: MotionCLIP converts CLIP text embeddings into motion with its decoder, without further training, handling explicit, nuanced, and abstract language.The text-to-motion pipeline uses the pretrained CLIP text encoder and MotionCLIP decoder.
- Text-to-Motion: Users clearly preferred MotionCLIP for both seen and unseen action generation in a comparative study.The unseen set contained Olympic sports absent from both models’ training labels.
- Text-to-Motion: Abstract prompts such as “couch,” “Spiderman,” Usain Bolt, and Swan Lake produced corresponding or signature motions, including motions not seen during training.The examples rely on linguistic or cultural associations rather than explicit motion descriptions.
- Motion Manifold Applications: Linear interpolation between latent codes produced semantic transitions in time and space, while latent arithmetic enabled compositional motion and style-based editing.The editing experiments decomposed upper- and lower-body actions and added style through a style-name embedding vector.
- Text-to-Motion: MotionCLIP generated style from textual action-and-style names and won twice while breaking even once against Aberman et al. [2020].Aberman et al. [2020] receives separate content and style motion sequences, whereas MotionCLIP uses text inputs.
- Motion Manifold Applications: Combining the MotionCLIP encoder with the CLIP text encoder yielded action recognition performance marginally close to 2s-AGCN on BABEL-60.Classification used cosine distances between motion embeddings and 60 class-name encodings.
5 CONCLUSIONS
MotionCLIP improves motion-manifold semantics and enables intuitive generation, editing, and interpolation by aligning an auto-encoder with CLIP’s knowledge-rich latent space. The authors note limitations in directional understanding, style capture, and consistency for some cultural references.
- 5 CONCLUSIONS: Aligning the auto-encoder with CLIP’s latent space yields better understanding of the motion manifold and its semantics using the same data.The authors attribute this improvement to alignment with a well-behaved, knowledge-rich latent space.
- 5 CONCLUSIONS: CLIP alignment transfers semantics and disentanglement despite CLIP never having seen motion or temporal signals.The alignment encourages semantically similar samples to lie closer together and induces CLIP-like disentanglement.
- 5 CONCLUSIONS: MotionCLIP generates signature motions for cultural figures and phrases, extending text-conditioned generation beyond direct motion descriptions.The paper presents abstract-language and cultural-reference examples as demonstrations of this capability.
- 5 CONCLUSIONS: MotionCLIP enables semantic interpolation between motions and latent-space editing, including upper- and lower-body composition and style transfer.These capabilities are illustrated in the latent-space interpolation and editing applications.
- 5 CONCLUSIONS: MotionCLIP struggles with directional terms, some styles, and consistency for out-of-domain cultural references.Reported failures include Cristiano Ronaldo’s goal celebration and Superman’s signature pose.
- 5 CONCLUSIONS: Future work includes larger latent spaces, more elaborate generation architectures, domain adaptation, and further downstream-task investigation.The authors identify these directions as opportunities opened by MotionCLIP’s limitations and backbone.