PaperlayerView exact revision on arXiv

Source-linked AI summary

DrawMotion: Generating 3D Human Motions by Freehand Drawing

Tao Wang, Lei Jin, Zhihua Wu, Qiaozhi He, Jiaming Chu, Yu Cheng, Junliang Xing, Jian Zhao, Shuicheng Yan, Li Wang

arXiv:2605.20955v1cs.CV

TL;DR

Text descriptions may not capture users’ detailed intended motions, especially when semantic and spatial constraints conflict. DrawMotion combines text and hand-drawn conditions in a diffusion framework, and user studies find its outputs more aligned with users’ imagination.

  • Problem

    Text descriptions may not fully capture users’ detailed motion intentions, while semantic and spatial conditions can conflict during generation.

  • Method

    DrawMotion combines text and hand-drawn trajectory and pose conditions through a Multi-Condition Module and training-free guidance in diffusion generation.

  • Results

    User studies indicate that DrawMotion’s generated motions are more aligned with users’ imagination.

  • Takeaways & Limitations

    DrawMotion offers a motion-generation interface with greater creative freedom through user-specified trajectories and character poses.

  • Takeaways & Limitations

    When user inputs conflict with text or basic human-motion principles, generated motions may deviate from those inputs and have reduced fidelity.

Abstract

from arXiv · show

Text-to-motion generation, which translates textual descriptions into human motions, faces the challenge that users often struggle to precisely convey their intended motions through text alone. To address this issue, this paper introduces DrawMotion, an efficient diffusion-based framework designed for multi-condition scenarios. DrawMotion generates motions based on both a conventional text condition and a novel hand-drawing condition, which provide semantic and spatial control over the generated motions, respectively. Specifically, we tackle the fine-grained motion generation task from three perspectives: 1) freehand drawing condition. To accurately capture users' intended motions without requiring tedious textual input, we develop an algorithm to automatically generate hand-drawn stickman sketches across different dataset formats; 2) multi-condition fusion. We propose a Multi-Condition Module (MCM) that is integrated into the diffusion process, enabling the model to exploit all possible condition combinations while reducing computational complexity compared to conventional approaches; and 3) training-free guidance. Notably, the MCM in DrawMotion ensures that its intermediate features lie in a continuous space, allowing classifier-guidance gradients to update the features and thereby aligning the generated motions with user intentions while preserving fidelity. Quantitative experiments and user studies demonstrate that the freehand drawing approach reduces user time by approximately 46.7% when generating motions aligned with their imagination. The code, demos, and relevant data are publicly available at https://github.com/InvertedForest/DrawMotion.

I. INTRODUCTION · II. RELATED WORK

DrawMotion addresses the limits of text-only human-motion generation by adding hand-drawn spatial controls alongside text semantics. It introduces automatic stickman generation, multi-condition diffusion fusion, and combined training-based and training-free guidance to support precise, natural motion synthesis.

  • I. INTRODUCTION: Text-to-motion generates natural human-motion sequences from language descriptions, reducing the need for manual 3D character key-framing.Human-motion generation supports applications including film and television, virtual reality, and gaming.
  • I. INTRODUCTION: Simple textual descriptions may fail to capture users’ detailed imagination of complex motions, motivating additional spatial controls.The paper illustrates this limitation with the description “A high kick forward.”
  • I. INTRODUCTION: DrawMotion introduces hand-drawn trajectories and stickman figures, allowing multiple poses at arbitrary trajectory positions for more precise motion control.This extends StickMotion, which could specify only three frames at dynamically placed positions.
  • I. INTRODUCTION: The Stickman Generation Algorithm automatically produces stickman sketches in diverse styles, addressing the time and stylistic limitations of collecting hand-drawn annotations.This is one of three challenges identified for DrawMotion: data generation, multi-condition fusion, and guidance.
  • I. INTRODUCTION: DrawMotion is presented as the first motion-generation approach to use hand-drawn representations for precise control without extensive textual descriptions.The paper also proposes a Multi-Condition Module that reduces computational complexity while improving performance compared with standard self-attention.
  • II. RELATED WORK: Diffusion models learn to progressively remove noise during inference so generated outputs approximate the distribution of original data.The related-work discussion traces this direction to denoising diffusion probabilistic models and subsequent advances.
  • II. RELATED WORK: Human-motion generation includes prediction, action-to-motion, and sound-to-motion tasks, differing in inputs and control granularity.Action-to-motion provides more direct but coarse-grained control, while sound-to-motion includes music-to-dance and speech-to-gesture.
  • II. RELATED WORK: Hybrid training-based and training-free methods balance constraint alignment with motion naturalness and achieve superior Traj.Err. and FID over purely training-free methods.The related-work discussion identifies OmniControl and DrawMotion as examples of this hybrid strategy.

III. TRAINING-BASED GUIDANCE · A. Hand-Drawing Representation

DrawMotion supports text and hand-drawn sketches in any combination, using a representation built from a 2D trajectory and optional stickman figures. The trajectory controls pelvis path, motion trajectory, and speed, while automatically generated stickmen and candidate-based supervision address ambiguity and noise.

  • III. TRAINING-BASED GUIDANCE: DrawMotion accepts text and hand-drawn sketches through four condition combinations: C(text, draw), C(text, ∅), C(∅, draw), and C(∅, ∅).This multi-modal design allows either or both modalities to condition generation.
  • A. Hand-Drawing Representation: A hand-drawing representation contains one trajectory and any number of stickman figures placed along it.The representation must therefore support trajectory generation, encoding, and application.
  • A. Hand-Drawing Representation: The user’s 2D coordinate sequence is processed into a trajectory that DrawMotion uses as the target 2D pelvis path.The processed path provides fine-grained control over motion trajectory and speed.
  • A. Hand-Drawing Representation: At training time, trajectories extracted from dataset motion sequences are directly input into DrawMotion with additional supervision.The method motivates this choice by noting inertia in both hand-drawn trajectories and real human-motion trajectories.
  • A. Hand-Drawing Representation: DrawMotion automatically generates hand-drawn stickmen from human joints in existing motion datasets.The generation process considers hand-drawing characteristics including stroke smoothness and inevitable misplacement.
  • A. Hand-Drawing Representation: The stickman representation is challenging because nearby limbs can obscure left-right identity and pose estimation introduces noise.These issues affect both algorithmically generated stickmen and user sketches.
  • A. Hand-Drawing Representation: Candidate loss addresses ambiguous stickman poses by avoiding the requirement that the decoder predict one exact pose.The passage states that this alleviates latent information loss and ambiguous outputs while improving motion prediction accuracy.
  • A. Hand-Drawing Representation: The processed trajectory bJt ∈R(T,2) is encoded as ej ∈R(T,E), where T is motion-sequence length and E is encoding channel dimension.This defines the trajectory encoding supplied to DrawMotion.

B. Diffusion-based Motion Generation

DrawMotion uses diffusion models to generate human motions under text and hand-drawing conditions, combining efficient denoising with weighted condition mixtures. Its training objective predicts added noise from motion sequences and both conditioning modalities.

  • Diffusion Process: Diffusion models serve as DrawMotion’s base because they can bias generated motions toward textual descriptions or hand-drawing conditions.The model approximates the data distribution with a learnable model distribution.
  • Training Objective: DrawMotion trains a noise predictor with a denoising objective conditioned on sequence length, drawing, and text inputs.The loss minimizes the squared difference between injected noise and predicted noise.
  • Training Objective: 0.7: Drawing and text conditions are each activated with probability 0.7 during training.The condition probabilities are denoted pc_draw and pc_text.
  • Reverse Process: DDIM provides the reverse process, using efficient and stable deterministic sampling paths that can reduce the steps needed for high-quality motion sequences.Reverse diffusion starts from xT ∼ N(0, I) and gradually removes noise.
  • Condition Mixture: DrawMotion mixes predicted noises from four text-and-drawing condition combinations with weights constrained by w1 + w2 + w3 + w4 = 1.The combinations include both conditions, each condition alone, and neither condition.

C. Architecture of DrawMotion

DrawMotion uses four input encoders and Multi-Condition Modules to generate motions from noisy motion, trajectories, stickman figures, and text. Its architecture combines draw-based spatial control through standard attention with text-based semantic control through efficient attention, while MCMs support multiple condition combinations in latent space.

  • Input: Four input encoders transform noisy motion, trajectories, stickman figures, and textual descriptions into representations for motion generation.The encodings have dimensions [T, E], [T, E], [T, E], and [L, E], respectively.
  • Multi-Condition Module: MCMs generate outputs for the four condition combinations: (text, draw), (text, ∅), (∅, draw), and (∅, ∅).Condition Fusion incorporates hand-drawing and text into motion features in latent space before further fusion by the Latent Encoder.
  • Draw Decoder: The Draw Decoder uses stickman and trajectory encodings to model frame-wise local poses and global spatial positions with dot-product attention.The motion, trajectory, and stickman embeddings are combined into a key-value representation with shape R(2×T,E).
  • Text Decoder: The Text Decoder uses efficient attention to align globally semantic text representations with motion features.Efficient attention reduces computational cost because its complexity scales linearly with the query token length n.
  • Multi-Condition Module: MCMs partition data along the batch dimension into four segments to implement condition combinations efficiently.The four segments are denoted (B1, B2, B3, B4) and represent the condition combinations used by the module.

D. Supervision

DrawMotion uses a unified supervision objective combining trajectory, stickman, and motion reconstruction losses. These terms enforce spatial consistency, pose-level fidelity, and closeness to ground-truth motion while supporting varied stickman-position combinations.

  • Unified supervision objective: The final loss combines trajectory, stickman, and motion reconstruction losses: L_final = L_motion + L_traj + L_stick.The objective uses ground-truth motion x and predicted motion x̂.
  • Loss components: L_traj enforces global trajectory alignment between generated and ground-truth motions, preserving spatial consistency with user-provided trajectories.
  • Loss components: L_stick regularizes pose-level fidelity by comparing predicted and ground-truth 3D poses frame by frame.
  • Mask-based conditioning: A randomly sampled binary mask m_i ∈ {0, 1} enables different combinations of stickman positions, with M = Σ_i m_i as the normalization factor.
  • Loss components: L_motion constrains the reconstructed motion sequence to remain close to the reference ground-truth motion.

IV. TRAINING-FREE GUIDANCE

DrawMotion introduces Intermediate Feature Guidance (IFG), a training-free method built on the Multi-Condition Module to align user-provided trajectories with generated motions. The section presents IFG’s motivation and analyzes intermediate feature spaces in traditional models and DrawMotion’s MCM.

  • IV. TRAINING-FREE GUIDANCE: DrawMotion proposes Intermediate Feature Guidance (IFG), a novel training-free method for motion alignment.IFG aligns the user-provided trajectory with the generated motion without additional training.
  • IV. TRAINING-FREE GUIDANCE: IFG is built on DrawMotion’s Multi-Condition Module (MCM) to align user-provided trajectories with generated motions.The guidance operates through the MCM rather than requiring additional model training.
  • IV. TRAINING-FREE GUIDANCE: The section motivates training-free guidance, surveys current works, and analyzes intermediate feature spaces in traditional models and MCM.These topics are covered in Sections IV-A and IV-B.

A. Motivation · B. Intermediate Feature Space

DrawMotion addresses conflicts between semantic and spatial controls by exploiting a continuous intermediate feature space in its Multi-Condition Module (MCM). This continuity supports direct gradient-based feature updates while preserving the distribution and fidelity of generated motions.

  • A. Motivation: A. Motivation: Conflicting text and trajectory conditions can make generated motion deviate from the user-provided trajectory.Text provides global semantic guidance, whereas trajectory control provides global spatial guidance.
  • A. Motivation: A. Motivation: Existing approaches face a fidelity-efficiency trade-off when applying spatial guidance during diffusion generation.Directly backpropagating to generated motion may move x_{t−1} outside its original distribution, while alternative preservation strategies rely on additional mechanisms or perturb only x_T.
  • B. Intermediate Feature Space: B. Intermediate Feature Space: Ordinary intermediate features form discrete, lower-dimensional manifolds that are vulnerable to perturbations.Slight changes can move features outside their distribution, unlike VAE features regularized toward N(0, I).
  • B. Intermediate Feature Space: B. Intermediate Feature Space: PCA evidence shows ReMoDiffuse features are extremely irregular, whereas DrawMotion’s last MCM encoder features exhibit continuity.Figure 5 uses 80,000 samples at diffusion step 299.
  • B. Intermediate Feature Space: B. Intermediate Feature Space: Multi-condition fusion expands effective dimensionality through the Minkowski sum of condition features.This process combines representations from conditions such as text and drawing into a higher-dimensional, more continuous feature space.
  • B. Intermediate Feature Space: B. Intermediate Feature Space: The multi-condition setting reaches 417 dimensions for 99.9% variance explanation, exceeding the single-condition cases.The comparison covers (text, draw), (text, ∅), (∅, draw), and (∅, ∅), using principal-component counts as intrinsic-dimensionality proxies.
  • B. Intermediate Feature Space: B. Intermediate Feature Space: Shuffled-batch interpolation tests whether intermediate features remain within their distribution under perturbation.The perturbed feature is defined as F̄ = F + λ(F̂ − F), where λ is the perturbation factor.
  • B. Intermediate Feature Space: B. Intermediate Feature Space: MCM maintains stable generation quality across a wide range of interpolation factors and tolerates gradient-based perturbations.This enables direct updates at step t−1 without causing x_{t−1} to leave its distribution or requiring additional modules such as ControlNet.

C. Intermediate Feature Guidance

Intermediate Feature Guidance updates diffusion intermediate features with spatial signals while constraining them to remain near their statistical distribution. Mahalanobis-distance clipping stabilizes guidance, preserves fidelity, and supports a balanced hyperparameter configuration.

  • Intermediate Feature Guidance: The method splits the model around the Nth MCM layer and optimizes its intermediate feature during the DDIM reverse process.The feature is updated using spatial guidance while the model is divided into Model1 and Model2 around the optimization objective.
  • Intermediate Feature Guidance: Spatial guidance updates intermediate features through SGD using a masked reconstruction loss, so unconstrained regions do not contribute.The target c has the same shape as the predicted state, and only spatially guided parts participate in loss calculation.
  • MD Clipping: Mahalanobis-distance clipping constrains updates to the high-probability feature region, preventing out-of-distribution artifacts while stabilizing reverse-diffusion guidance.Clipping activates at the boundary M(F)+ϵMD and accounts for feature variance and correlations.
  • Hyperparameter Tuning: Deeper guidance layers reduce trajectory error but increase FID, motivating selection of Nth = 3 as a computation-conscious balance.The layer-selection analysis compares rows 1–3 of Table II and identifies the third MCM layer as the adopted compromise.
  • Hyperparameter Tuning: Increasing SGD repeat count improves results but increases computation, while fewer iterations require a larger learning rate.The repeat count controls SGD iterations under spatial guidance, with the learning rate adjusted accordingly.

V. EXPERIMENTS · A. Experiment Settings · B. Quantitative Analysis

Experiments evaluate DrawMotion on KIT-ML and HumanML3D using established text-to-motion protocols and additional drawing-trajectory metrics. Quantitative analyses report stronger stickman similarity and motion-editing performance, while explaining trade-offs from condition alignment and training-free guidance.

  • A. Experiment Settings: DrawMotion is evaluated on the KIT-ML and HumanML3D human-motion-generation datasets using the evaluation protocol of Guo et al.The motion representation uses local skeleton poses relative to the root plus global root translations and rotations across frames.
  • A. Experiment Settings: StiSim and 2D Trajectory error are additionally reported to compare generated motions with given trajectories in motion-editing tasks.Training uses four 4090 GPUs, batch size 1024, and 40 dataloader workers for stickman generation through SGA.
  • B. Quantitative Analysis: The quantitative comparison follows text-to-motion protocols on both datasets, while generating evaluation stickmen and trajectories similarly to training.Stickmen are selected at the beginning, middle, and end of motion sequences, matching StickMotion’s selection strategy.
  • B. Quantitative Analysis: DrawMotion achieves higher Stickman Similarity than StickMotion on the HumanML3D and KIT-ML test sets.The paper attributes this to explicitly specified stickman positions and an average of 7 training stickmen, compared with StickMotion’s maximum of 3.
  • B. Quantitative Analysis: DrawMotion achieves the best reported performance and speed among the compared diffusion-based motion-editing methods.The comparison covers Tables V and VI, including motion-editing quality and efficiency on the HumanML3D dataset with batch size 16.
  • B. Quantitative Analysis: Training-free motion-editing methods exhibit poor FID because they cannot effectively handle data deviating from the training distribution.Although DNO should theoretically have better FID, its official implementation uses DDIM-10 because of GPU memory limitations.
  • B. Quantitative Analysis: Purely training-based motion-editing methods achieve only moderate trajectory error because they lack additional constraints.This contrasts with the limitations attributed to training-free methods in FID.

C. Ablation Study

Ablations identify decoder structures and MCM components that support DrawMotion’s multi-condition design, while varying stickman count has limited impact beyond a small number of inputs. User studies further indicate that DrawMotion improves imagination alignment and practical efficiency over comparison workflows.

  • Structure of Condition Decoders: Text conditions favor efficient attention, whereas drawing conditions benefit from dot-product attention because they capture global semantics and local details, respectively.Table VII supports this condition-specific decoder assignment on KIT-ML.
  • Structure of Multi-Condition Module: The MCM ablation replaces Condition Fusion and the Latent Encoder with a traditional mask mechanism and fully connected layer to test both components’ validity.The first two Table VIII rows represent the traditional-mask implementation and the minimum-complexity alternative described in the passage.
  • Number of Stickmen: With increasing stickman count, FID fluctuates slightly while R-Precision and StiSim generally improve, gains become marginal beyond 3 stickmen, and Diversity remains largely unchanged.The best FID value occurs at 7 stickmen on KIT, but the passage does not provide the corresponding numeric score.
  • User Study: DrawMotion’s outputs are more aligned with users’ imagination than ReMoDiffuse and StickMotion in ratings collected on a 0-to-10 scale.ReMoDiffuse receives text and trajectory inputs, while StickMotion and DrawMotion receive text and hand-drawing inputs under the comparison setup.
  • Practical Workflow Efficiency: Manual animation requires about 3 hours per sample and scores 7.4 on average, while animators report that AI-generated motions are richer and more natural and that DrawMotion latency is acceptable.Handmade results follow target trajectories well but have comparatively weaker motion naturalness.

VI. LIMITATION · VII. CONCLUSION

DrawMotion offers flexible hand-drawn trajectory and pose control, but fidelity decreases when user inputs conflict with text or fundamental human-motion principles. The paper concludes that its hand-drawing condition and combined training-based and training-free guidance address detailed motion-generation requirements, supported by qualitative and quantitative analyses.

  • VI. LIMITATION: DrawMotion lets users specify motion trajectories and character poses at designated positions, providing substantial creative freedom.The limitation section characterizes this as the highest level of creative freedom.
  • VI. LIMITATION: Fidelity decreases when user-drawn trajectories or stickman figures conflict with text conditions or fundamental human-motion principles.Under these conflicts, generated motion sequences often deviate from the input.
  • VI. LIMITATION: The limitation analysis reports that mismatched or physically invalid inputs can cause generated sequences to deviate from users’ intended motion.This deviation is identified as the main fidelity problem associated with the system’s flexibility.
  • VII. CONCLUSION: The paper presents a novel hand-drawing condition and the DrawMotion model for generating motion from simple textual descriptions while addressing detailed user requirements.The conclusion frames the method as a response to users’ need for more detailed control than text alone provides.
  • VII. CONCLUSION: DrawMotion combines training-based and training-free guidance to maintain consistency between input conditions and generated motion sequences.Training-based guidance maps relationships between inputs and outputs through the model, while training-free guidance is also used for consistency.
  • VII. CONCLUSION: The authors validate DrawMotion’s effectiveness through both qualitative and quantitative analyses.These analyses are presented as experimental evidence supporting the proposed method.
  • VII. CONCLUSION: The paper positions DrawMotion as a professional and convenient motion-generation method for art creators.The authors also state that it may promote motion-generation research and the relevant community.
Loading 2605.20955v1…