PaperlayerView exact revision on arXiv

Source-linked AI summary

Fg-T2M++: LLMs-Augmented Fine-Grained Text Driven Human Motion Generation

Yin Wang, Mu Li, Jiapeng Liu, Zhiying Leng, Frederick W. B. Li, Ziyao Zhang, Xiaohui Liang

arXiv:2502.05534v1cs.CV

TL;DR

Fine-grained text-driven motion generation requires detailed body-part parsing and modeling of linguistic relationships. Fg-T2M++ combines LLM semantic parsing, hyperbolic text representation, and hierarchical multimodal diffusion fusion. It outperforms existing methods on HumanML3D and KIT-ML and generates motions that better align with complex text.

  • Problem

    Existing methods inadequately parse body-part semantics and linguistic structures, producing imprecise motion for detailed text.

  • Method

    Fg-T2M++ combines LLMs Semantic Parsing, hyperbolic dependency-graph representation, and hierarchical multi-modal fusion in conditional diffusion.

  • Results

    Fg-T2M++ outperforms existing SOTA methods on HumanML3D and KIT-ML, generating high-quality fine-grained motions aligned with complex text.

  • Takeaways & Limitations

    The framework improves fine-grained text-to-motion generation by combining detailed semantic parsing with hierarchical linguistic and motion-feature modeling.

  • Takeaways & Limitations

    The method depends on pretrained language-model capabilities, may require detailed input formats, and currently supports sequences up to 196 frames.

Abstract

from arXiv · show

We address the challenging problem of fine-grained text-driven human motion generation. Existing works generate imprecise motions that fail to accurately capture relationships specified in text due to: (1) lack of effective text parsing for detailed semantic cues regarding body parts, (2) not fully modeling linguistic structures between words to comprehend text comprehensively. To tackle these limitations, we propose a novel fine-grained framework Fg-T2M++ that consists of: (1) an LLMs semantic parsing module to extract body part descriptions and semantics from text, (2) a hyperbolic text representation module to encode relational information between text units by embedding the syntactic dependency graph into hyperbolic space, and (3) a multi-modal fusion module to hierarchically fuse text and motion features. Extensive experiments on HumanML3D and KIT-ML datasets demonstrate that Fg-T2M++ outperforms SOTA methods, validating its ability to accurately generate motions adhering to comprehensive text semantics.

1 Introduction

Fg-T2M++ targets fine-grained text-driven motion generation by parsing body-part semantics, modeling linguistic structure, and hierarchically fusing text with motion. It is evaluated on HumanML3D and KIT-ML, where results demonstrate stronger alignment with detailed text.

  • Motivation: Existing methods struggle to capture intricate spatiotemporal relationships between body parts and detailed linguistic cues in text prompts.Coarse descriptions and compact sentence representations can cause generated motions to deviate from specified meanings.
  • Motivation: Fg-T2M++ decomposes whole-body generation into combinable part-level motions while analyzing syntactic roles and relationships between words.The framework uses part-level and word-level descriptions to model fine-grained motions beyond the training distribution.
  • Method: LLMs Semantic Parsing extracts body-part motions and detailed word semantics, mapping individual textual elements to joint movements.The module analyzes semantic roles involving nouns, adjectives, adverbs, and motions.
  • Method: Hyperbolic Text Representation embeds dependency graphs in hyperbolic space, while Multi-Modal Fusion hierarchically combines global and local text-motion relationships during diffusion denoising.These modules address syntactic hierarchy and multi-level conditioning within a conditional diffusion framework.
  • Results: Fg-T2M++ demonstrates SOTA performance on HumanML3D and KIT-ML, with qualitative results showing finer motions that match text.The reported evaluation uses quantitative metrics and qualitative comparisons.

2 Related Work

Prior text-to-motion methods use aligned embeddings, autoregressive tokens, or diffusion, but remain limited in preserving fine-grained semantics and complex text-motion relationships. Fg-T2M++ responds with detailed linguistic annotations and structural modeling.

  • Diffusion-based methods: Diffusion-based methods improve probabilistic text-motion modeling but may still lack fidelity for complex multimodal relationships.The cited methods include MotionDiffuse, MDM, FLAME, MLD, and ReMoDiffuse.
  • Limitations: Existing methods perform better on coarse text than on prompts combining multiple body-part actions and complex syntax-kinematic associations.Latent alignment can lose feature details, while autoregressive prediction may miss future context.
  • Research gap: Fine-grained generation is constrained by sparse datasets, insufficient linguistic cues, and limited part-specific text-motion correspondences.Detailed annotations of body-part actions and word explanations are introduced to address these gaps.
  • LLM-assisted generation: Fg-T2M++ uses LLMs to parse body-part descriptions and word-level semantics, exploiting linguistic structures for more intricate motion understanding.The approach supplies detailed explanations for nouns, adjectives, and adverbs in complex texts.

3 Preliminaries

Fg-T2M++ uses conditional diffusion to progressively denoise motion from noise while conditioning each step on natural-language text. Its hyperbolic representation models syntactic trees with low distortion for hierarchical linguistic structure.

  • Diffusion preliminaries: The diffusion framework corrupts clean motion with Gaussian noise and reverses this process conditionally on natural-language text.The reverse process progressively removes noise from x1 through xT to generate the target motion.
  • Diffusion preliminaries: Fg-T2M++ conditions each denoising step on motion context and linguistic input to align generated motions with detailed descriptions.This formulation supports rich language-motion mappings through progressive conditional generation.
  • Hyperbolic representation: Hyperbolic space provides a non-Euclidean domain suited to representing hierarchical syntactic structures with less distortion than Euclidean space.The Poincaré ball supports hierarchical embeddings with a large branching factor.
  • Hyperbolic graph convolution: Hyperbolic Graph Convolution projects word features into the Poincaré ball, aggregates neighboring nodes, and preserves tree structure during representation learning.Its operations alternate between hyperbolic and Euclidean spaces while modeling syntactic dependencies.

4 Methodology

Fg-T2M++ is a diffusion framework for fine-grained text-driven motion generation that parses body-part semantics, models linguistic structure, and fuses text with motion hierarchically.

  • 4 Methodology: Fg-T2M++ generates human motion sequences with a diffusion model conditioned on natural-language text.The model denoises random inputs backward from X_T to X_0 while predicting clean motion states with an L2 objective.
  • 4 Methodology: The framework uses parsed part-level descriptions, syntactic text representations, and motion features as conditioning signals during denoising.These components are combined through the model’s multimodal reasoning process to obtain clean motion embeddings.
  • 4 Methodology: The method addresses coarse annotations and shallow text representations that omit coordinated, part-specific motion details.Datasets provide brief descriptions that may overlook movements such as hand waving, while sentence-level encodings can deviate from detailed text meaning.
  • 4 Methodology: LLM semantic parsing decomposes prompts into body-part motions and analyzes nouns, adjectives, adverbs, quantifiers, and conjunctions.The approach performs 15 sub-analyses to capture relationships between linguistic elements and joint movements.
  • 4 Methodology: Hyperbolic Text Representation constructs a dependency-based word tree and models its hierarchical structure in hyperbolic space.Dependency parsing supplies nodes and edges, while hyperbolic graph convolution captures syntactic relationships before features are projected back to Euclidean space.

Hyperbolic Graph Convolution.

HTP represents text as a dependency graph and processes it in hyperbolic space to capture hierarchical syntax, then combines structural and parsed-text encodings through cross-perception.

  • Hyperbolic Graph Convolution: Hyperbolic graph convolution updates dependency-graph features through Möbius operations and hyperbolic nonlinear activations.The representation is projected into a Poincaré-ball model and later mapped back to Euclidean space.
  • Hyperbolic Graph Convolution: HTP produces text encodings informed by both dependency relations and syntactic context.Hyperbolic geometry is used because it preserves tree-like local structure with low distortion.
  • Hyperbolic Graph Convolution: The cross-perception module relates LLM-parsed descriptions to full-prompt structural encodings through multi-stage attention.It computes global context with key-value attention and then applies cross-attention to enrich the representations.
  • Hyperbolic Graph Convolution: MMF hierarchically fuses sentence-level overall semantics with word-level text-motion interactions to guide fine-grained diffusion.Sentence-level fusion supplies global context, while word-level fusion iteratively refines features against a reference sequence.

Multi-Modal Sentence-Level Feature Fusion

Multi-modal fusion first aligns motion with overall parsed and prompt-level semantics, then iteratively refines word-level text-motion relationships through hybrid attention.

  • Multi-Modal Sentence-Level Feature Fusion: Sentence-level fusion computes attention between motion features and parsed-content and prompt sentence features.The resulting cross-modal features highlight motion channels relevant to both semantic sources.
  • Multi-Modal Sentence-Level Feature Fusion: Word-level fusion refines text-motion representations through hybrid self-attention and cross-attention to a shared reference sequence.At each timestep, encodings provide queries while the reference provides keys and values.
  • Multi-Modal Sentence-Level Feature Fusion: Sentence-level features encode overall semantics, and trainable projections convert them into reference representations for subsequent fusion.The word features are concatenated with opposite-modal references before unified keys and values are computed.
  • Multi-Modal Sentence-Level Feature Fusion: Global templates extracted with softmax attention support iterative refinement of text-motion representations.A motion-derived query attends to the template, which contains dependencies inferred from hybrid attention across both modalities.
  • Multi-Modal Sentence-Level Feature Fusion: Repeated refinement grounds subtle motion details in word-level semantics and improves textual-content comprehension through implied motion concepts.The claimed benefit follows from incrementally fusing hierarchical semantics between modalities.

5 Experiments

Fg-T2M++ consistently improves text–motion alignment and fine-grained generation across benchmark, rarity, user-study, ablation, and long-text evaluations. Its LLM semantic parsing, hyperbolic text representation, and multimodal fusion each contribute, while diversity and GT consistency remain practical limitations.

  • 5.1 Quantitative Comparison: Fg-T2M++ achieves higher R-TOP, FID, and MM-Dist than competing methods across HumanML3D and KIT-ML, including without additional ground-truth motion priors.The reported gains indicate stronger alignment between generated motions and textual meanings across both datasets.
  • 5.2 Fine-Grained Evaluation: Under increasingly fine-grained POS conditions, Fg-T2M++ remains robust, and a 30-participant user study gives it the best voting ranking and preference.The fine-grained evaluation ranks samples by counts of adjectives, adverbs, conjunctions, and quantifiers across four difficulty ranges.
  • 5.4 Component Analysis and Discussion: LLM semantic parsing improves generation quality, with action-body parsing contributing more strongly than word-semantic parsing in the component analysis.The analysis evaluates 15 parsing sub-components and finds body-level action information especially important.
  • 5.3 Rareness Evaluation: For rare text conditions, Fg-T2M++ better matches the ground-truth distribution, while removing LLM semantic parsing substantially degrades FID on the rarest split.The LLM module provides prior knowledge that improves generalization when text is rare.
  • 5.4 Component Analysis and Discussion: Replacing standard graph convolution with hyperbolic GCN improves R-TOP, FID, and MM-Dist by better capturing tree structure, while cross-perception refines fine-grained feature assimilation.The learned representations distinguish related concepts such as left leg and right leg more clearly than ablated variants.
  • 5.4 Component Analysis and Discussion: The multimodal fusion module shows the least degradation on long, complex prompts, whereas Fg-T2M and the no-MMF variant degrade significantly.Prompts are divided into four length ranges from fewer than 6 words to more than 10 words.

6 Limitations, Future Work and Conclusion

Fg-T2M++ achieves fine-grained text-driven motion generation while remaining limited by language-model dependence, sequence length, and difficult long sentences. The authors propose longer sequences, smoother action transitions, and human-environment interaction as future directions.

  • Limitations and Future Work: Pretrained language-model capabilities constrain Fg-T2M++ because applications may require detailed, specific input formats.This dependence limits the range of application scenarios.
  • Limitations and Future Work: The current model generates sequences of at most 196 frames, restricting its use for longer motions.Future work includes extending sequence length and smoothing transitions between actions.
  • Limitations and Future Work: Future research should model interactions between humans and environments, including other people and scenes.
  • Conclusion: Fg-T2M++ outperforms existing SOTA methods on text-driven motion generation, producing fine-grained motions aligned with complex text prompts.The conclusion reports quantitative and qualitative superiority across the evaluated tasks.
  • Limitations and Future Work: Lengthy sentences can cause Fg-T2M++ to omit specific actions, producing suboptimal long-sequence results.The authors suggest splitting long sentences into independent tasks to preserve fine-grained characteristics.

C More Analysis of Text Features

The text-feature analysis shows that Fg-T2M++ better distinguishes sequential actions, preserves dependency hierarchy in hyperbolic space, and aligns text and motion feature spaces more closely than ReMoDiffuse.

  • Text Feature Visualization: Fg-T2M++ distinguishes successive actions in “A person sits down, stands up, and walks forward,” whereas ReMoDiffuse compresses them and misses later actions.The comparison attributes the difference to how the methods represent text features.
  • Hyperbolic Text Representation: Fg-T2M++ better preserves the hierarchical structure of text dependencies than ReMoDiffuse in the geodesic-distance analysis.Table 5 compares average distances from the first, second, and third layer nodes to the root.
  • Text-Motion Feature Alignment: Fg-T2M++ achieves closer alignment between text-feature and motion-feature spaces than ReMoDiffuse.The analysis presents this as improved cross-modal feature alignment.
  • Hyperbolic Text Representation: Hyperbolic representations are motivated as a way to preserve hierarchical relationships in large-scale settings.The passage cites MERU as related evidence for this property.

D Visual Comparison against Different Methods

Visual comparisons show that Fg-T2M++ handles multi-action and fine-grained prompts more comprehensively than latent-space, autoregressive, and diffusion-model baselines.

  • Visual Comparison: The comparisons cover Temporal VAE, TM2T, and ReMoDiffuse on prompts involving stumbling, arm swinging, and jumping while clapping.
  • Visual Comparison: Fg-T2M++ comprehensively generates motions matching prompts that require multiple simultaneous or sequential actions, unlike the compared baselines.ReMoDiffuse performs only one action for jumping and clapping, while another baseline performs only a single hand swing.
  • Qualitative Samples: Additional qualitative examples include jogging while looking down, cartwheeling, dancing, picking something up, lunges, and running-jumping-turning sequences.These examples are presented as evidence of understanding complex motion descriptions and generating high-quality motion.

F Dependency Analysis of Fg-T2M++ on LLMs

The dependency analysis examines cases where LLM-generated descriptions omit numerical details and finds that Fg-T2M++ can still produce motions consistent with the original prompts. The model uses these descriptions as reference information rather than strict constraints.

  • Coarse-Grained LLM Descriptions: Fg-T2M++ still generates walking and kicking motions when LLM parsing omits “three” steps or “twice” kicking.The examples test prompts whose coarse-grained parsed descriptions lose numerical details.
  • Dependency on LLMs: Fg-T2M++ generation is not significantly affected when LLM descriptions are poor because those descriptions serve as reference information, not strong constraints.The authors report that the model remains aligned with the semantics of the full text prompt.
  • Qualitative Analysis: The qualitative samples include fine-grained prompts involving stumbling, cartwheeling, dancing, picking up objects, lunges, and turning after jumping.

G Performance differences between GPT-3.5 and GPT-4.

GPT-4 improves motion-generation evaluation metrics over GPT-3.5, while qualitative comparisons show more accurate parsing of fine-grained body-part details. The difference is attributed to GPT-4’s more detailed and comprehensive text analysis.

  • G Performance differences between GPT-3.5 and GPT-4.: GPT-4 improves R-TOP, FID, and MultiModal Dist after replacing GPT-3.5 in the LLM parsing module.The comparison used 100 randomly sampled HumanML3D examples and evaluated generated motion quality quantitatively.
  • G Performance differences between GPT-3.5 and GPT-4.: GPT-3.5 misparses the pivot prompt by selecting the right foot, whereas GPT-4 identifies the left foot as the pivot-supporting root.The prompt describes two forward strides followed by a pivot on the left foot and walking in the opposite direction.
  • G Performance differences between GPT-3.5 and GPT-4.: The qualitative evaluation visualizes motion differences generated from prompts processed by GPT-3.5 and GPT-4.The comparison focuses on parsing specific details such as “left leg.”

H LLM-parsed Fine-grained Descriptions

The section presents LLM-generated fine-grained descriptions that decompose original motion text into actions associated with individual joint parts, alongside corresponding visualizations.

  • LLMs parse original sentences into fine-grained actions for each joint part.
  • The examples pair text prompts with LLM-parsed descriptions and corresponding motion visualizations.
  • Figures 23 and 24 show the correspondence between body joints, visual motion, and LLM-parsed fine-grained descriptions.
Loading 2502.05534v1…