PaperlayerView exact revision on arXiv

Source-linked AI summary

DrawMotion: Generating 3D Human Motions by Freehand Drawing

Tao Wang, Lei Jin, Zhihua Wu, Qiaozhi He, Jiaming Chu, Yu Cheng, Junliang Xing, Jian Zhao, Shuicheng Yan, Li Wang

arXiv:2605.20955v1cs.CV

TL;DR

Text descriptions may not capture users’ detailed motion intentions. DrawMotion combines text and hand-drawn conditions, and its user-study outputs were rated as more aligned with participants’ imagination.

  • Problem

    Text descriptions may fail to capture users’ detailed imagined gestures, while text and trajectory constraints can conflict during generation.

    Source: §I · p. 1, A. Motivation · p. 7

  • Method

    DrawMotion combines text with hand-drawn trajectories and stickman poses, using an MCM and training-free classifier-guidance gradients to align generated motion with inputs.

    Source: §I · p. 2, §VII · p. 14

  • Results

    User ratings indicated that DrawMotion’s generated motions were more aligned with participants’ imagination than the evaluated alternatives.

    Source: C. Ablation Study · p. 14

  • Takeaways & Limitations

    Hand-drawn sketches offer a way to specify motion details without extensive textual descriptions.

    Source: §I · p. 2

    When user trajectories or poses conflict with text or basic human-motion principles, generated motions often deviate, so inputs must be roughly physically reasonable and semantically consistent.

    Source: §VI · p. 14

Abstract

from arXiv · show

Text-to-motion generation, which translates textual descriptions into human motions, faces the challenge that users often struggle to precisely convey their intended motions through text alone. To address this issue, this paper introduces DrawMotion, an efficient diffusion-based framework designed for multi-condition scenarios. DrawMotion generates motions based on both a conventional text condition and a novel hand-drawing condition, which provide semantic and spatial control over the generated motions, respectively. Specifically, we tackle the fine-grained motion generation task from three perspectives: 1) freehand drawing condition. To accurately capture users' intended motions without requiring tedious textual input, we develop an algorithm to automatically generate hand-drawn stickman sketches across different dataset formats; 2) multi-condition fusion. We propose a Multi-Condition Module (MCM) that is integrated into the diffusion process, enabling the model to exploit all possible condition combinations while reducing computational complexity compared to conventional approaches; and 3) training-free guidance. Notably, the MCM in DrawMotion ensures that its intermediate features lie in a continuous space, allowing classifier-guidance gradients to update the features and thereby aligning the generated motions with user intentions while preserving fidelity. Quantitative experiments and user studies demonstrate that the freehand drawing approach reduces user time by approximately 46.7% when generating motions aligned with their imagination. The code, demos, and relevant data are publicly available at https://github.com/InvertedForest/DrawMotion.

I. INTRODUCTION · II. RELATED WORK

DrawMotion adds hand-drawn trajectory and stickman controls to text-to-motion generation, aiming to address the difficulty of conveying detailed motion intentions through text alone. It combines multi-condition diffusion with training-free guidance, while the related work frames it among human-motion generation and diffusion-based motion-editing approaches.

I. INTRODUCTION

  • Hand-drawn trajectories and stickman figures provide spatial control over motion details, addressing the limits of text descriptions that may not capture users’ intended gestures.The paper presents this drawing condition as a way to reduce the need for extensive textual descriptions.Source: §I · p. 1
  • Unlike StickMotion, which places poses at fixed, automatically determined times, DrawMotion lets users place multiple stickman poses anywhere along a hand-drawn trajectory.The added spatial control is intended to enable finer and more precise motion specification, while requiring users to take greater responsibility for final-result fidelity.Source: §I · pp. 1–2
  • DrawMotion addresses data generation, condition fusion, and trajectory alignment with an automatic Stickman Generation Algorithm, an efficient Multi-Condition Module, and training-free Intermediate Feature Guidance.IFG uses the continuity of MCM intermediate features to improve trajectory alignment amid possible conflicts between trajectory constraints and text semantics.Source: §I · p. 2
  • DrawMotion is reported to achieve competitive performance against state-of-the-art text-to-motion methods on KIT-ML and HumanML3D, with superior StiSim and Traj.Err. results.The paper also reports consistent improvement over StickMotion and other state-of-the-art methods in fine-grained, user-controlled motion generation.Source: §I · p. 2

II. RELATED WORK

  • Human-motion generation includes prediction, action-to-motion, sound-to-motion, text-to-motion, and interaction-to-motion tasks, with text-to-motion generating sequences from natural-language descriptions.The related work notes that text alone can make precise control of individual limb positions difficult.Source: §II · pp. 2–3
  • Diffusion-based motion editing uses training-based spatial conditioning or training-free inference constraints, with the latter risking disrupted motion distributions, reduced vividness, or high computational cost.Examples include trajectory-conditioned or auxiliary-module methods, in contrast with inpainting and classifier-guidance approaches that enforce constraints during inference.Source: §II · p. 3
  • Hybrid training-based and training-free methods such as OmniControl and DrawMotion are described as balancing constraint alignment and naturalness, with better Traj.Err. and FID than purely training-free methods.The cited discussion attributes the gains to integrating both paradigms.Source: §II · p. 3

III. TRAINING-BASED GUIDANCE · A. Hand-Drawing Representation

DrawMotion accepts text, hand-drawn sketches, or any combination of the two. This section describes how it represents and encodes drawing conditions for motion generation.

III. TRAINING-BASED GUIDANCE

  • Section III supports text and drawing conditions in any combination, including either alone or neither, and introduces the drawing representation before the guidance and fusion modules.The section overview places classifier-free guidance in III-B, the Multi-Condition Module in III-C, and training-free guidance in Section IV.Source: §III · p. 3

A. Hand-Drawing Representation

  • Users’ 2D trajectories are resampled to the motion-frame count using uniform or density-based sampling, transformed as desired, and supplied as target pelvis paths.Density-based sampling preserves drawing speed, while uniform sampling ignores it.Source: A. Hand-Drawing Representation · p. 3
  • During training, dataset trajectories are used directly because density-based sampling aligns hand and body inertia, and the model can refine imperfect paths for fidelity.The paper says this supports consistency between generated motion and the trajectory condition.Source: A. Hand-Drawing Representation · p. 3
  • The Stickman Generation Algorithm uses dataset joint coordinates and accounts for stroke smoothness, placement errors, and scale differences, with poses viewed from the front.The specified front view has a line of sight approximately perpendicular to the pose’s pelvic plane.Source: A. Hand-Drawing Representation · pp. 3–4
  • Instead of encoding at least 200 points and their pairwise interactions, users draw six one-stroke lines that a transformer encoder aggregates into a stickman embedding.The six lines represent the head, torso, and four limbs; the paper says this compact encoding reduces computational complexity and improves recognition accuracy.Source: A. Hand-Drawing Representation · p. 4
  • A pretrained, frozen stickman encoder is trained with a feature-to-pose decoder; its candidate loss addresses left-right ambiguity and noisy pose estimates to improve motion prediction accuracy.The encoder maps stickmen to embeddings, and the decoder reconstructs poses from them.Source: A. Hand-Drawing Representation · p. 4
  • Unlike the stickman encoder, the trajectory encoder is trained jointly with DrawMotion and uses six Conv1d layers to map trajectories from R^(T,2) to R^(T,E).T is the motion-sequence length and E is the encoding channel dimension.Source: A. Hand-Drawing Representation · p. 4

B. Diffusion-based Motion Generation

DrawMotion uses conditional diffusion to generate motions from text and hand-drawn sketches. It trains a noise predictor and uses staged condition mixtures during reverse denoising.

B. Diffusion-based Motion Generation

  • The forward process gradually adds Gaussian noise to clean motion, and the noised state can be sampled directly from the original motion rather than generated through intermediate steps.At t = T, the state follows a standard Gaussian distribution.Source: B. Diffusion-based Motion Generation · p. 4
  • Training minimizes noise-prediction loss conditioned on sequence length, drawing, and text; Figure 3 shows frozen text and stickman encoders feeding trainable MCMs.Both drawing and text conditions are activated with probabilities set to 0.7.Source: B. Diffusion-based Motion Generation · p. 4, Figure 3 · p. 5
  • DrawMotion adopts DDIM for reverse sampling because its deterministic paths improve efficiency and stability and can reduce the steps needed for high-quality sequences.The reverse process starts from Gaussian noise and gradually removes noise to recover motion.Source: B. Diffusion-based Motion Generation · pp. 4–5
  • The condition mixture uses four text/drawing combinations early in denoising, then applies both conditions together in the final stage to refine the motion.Early weights let the drawing preference vary through ˆw; the final stage sets w1 = 1 and w2,3,4 = 0.Source: B. Diffusion-based Motion Generation · p. 5

C. Architecture of DrawMotion

DrawMotion uses four encoders and Multi-Condition Modules (MCMs) to combine noisy motion with text and drawing conditions. Its decoders use attention mechanisms suited to the spatial, frame-wise drawing inputs and global text semantics.

Input

  • Noisy motion uses a linear layer, trajectories a 1D CNN, text CLIP ViT-B/32, and stickman figures a standard transformer encoder.The encoders map these inputs to token representations for the condition modules.Source: C. Architecture of DrawMotion · p. 6

Draw Decoder

  • The Draw Decoder applies dot-product attention to drawing and trajectory features, enabling frame-specific spatial alignment and cross-frame temporal consistency.Stickman and trajectory inputs encode local poses and global spatial coordinates, respectively.Source: C. Architecture of DrawMotion · p. 6

Text Decoder

  • The Text Decoder uses efficient attention for global text semantics, with computational cost scaling linearly with query token length.The attention maps global semantic information sequentially to each query token.Source: C. Architecture of DrawMotion · p. 6

Multi-Condition Module

  • MCM partitions batches into four text/drawing condition combinations and uses two condition decoders to produce offsets, fuse motion features, and re-encode them.Unlike masked self-attention, MCM reduces computational complexity and supports training-free guidance.Source: C. Architecture of DrawMotion · p. 6

D. Supervision

DrawMotion uses a unified supervision objective combining trajectory, stickman, and motion reconstruction losses. Together, these losses guide global spatial alignment, pose-level fidelity, and reconstruction toward the ground-truth motion.

D. Supervision

  • The final loss sums trajectory, stickman, and motion reconstruction losses to train DrawMotion under multiple condition settings.The objective is L_final = L_motion + L_traj + L_stick.Source: D. Supervision · pp. 6–7
  • Trajectory loss aligns generated and ground-truth global trajectories, while stickman loss compares predicted and ground-truth 3D poses frame by frame.A randomly sampled binary mask over frames enables different combinations of stickman positions, with the number of masked frames serving as the normalization factor.Source: D. Supervision · p. 7
  • Motion reconstruction loss constrains the reconstructed motion sequence to remain close to the ground-truth reference sequence.The loss sums squared differences between predicted and reference motion across frames.Source: D. Supervision · p. 7

IV. TRAINING-FREE GUIDANCE

DrawMotion introduces Intermediate Feature Guidance (IFG), a training-free method built on the Multi-Condition Module (MCM) to align user-provided trajectories with generated motion while preserving motion fidelity.

IV. TRAINING-FREE GUIDANCE

  • IFG uses the MCM to align user-provided trajectories with generated motion without additional training.The section describes IFG as a novel training-free guidance method.Source: §IV · p. 7
  • The section analyzes why MCM intermediate features are amenable to gradients from spatial loss.It contrasts intermediate feature spaces in traditional models, MCM, and generative models.Source: §IV · p. 7
  • IFG regularizes its update process to ensure generated-motion fidelity remains unaffected.This regularization is presented as part of the proposed guidance method.Source: §IV · p. 7

A. Motivation · B. Intermediate Feature Space

Text and trajectory conditions can conflict, causing generated motion to deviate from the user’s trajectory. DrawMotion’s MCM provides intermediate features that support gradient-based updates, and experiments indicate these features form a relatively continuous space.

A. Motivation

  • Text supplies global semantic guidance while trajectory control supplies global spatial guidance; their conflict can make generated motion deviate from the user-provided trajectory.The authors turn to motion-editing approaches to refine generation while preserving semantic alignment and fidelity.Source: A. Motivation · p. 7
  • OmniControl guides reverse-step motion directly but risks fidelity, while DNO perturbs prior noise at high computational cost; MCM’s broad features enable efficient intermediate updates.OmniControl uses ControlNet to guide motion back toward its distribution; MCM aims to combine OmniControl’s efficiency with DNO’s vividness without extra training or computational cost.Source: A. Motivation · p. 7

B. Intermediate Feature Space

  • PCA shows irregular ReMoDiffuse features, clustered DrawMotion Latent Encoder outputs, and continuous, dense MCM fusion features.These patterns support using MCM intermediate features for backpropagated reverse gradients; cited work describes feature collapse in classification and regression.Source: B. Intermediate Feature Space · p. 8, Figure 5 · p. 8
  • The Condition Fusion module’s Minkowski sum expands the dimensionality of encoded conditions, producing a higher-dimensional, more continuous feature space.Individual text or drawing features may lie on low-dimensional nonlinear manifolds.Source: B. Intermediate Feature Space · p. 8
  • With text and drawing conditions, MCM features require 417 dimensions to explain 99.9% of variance and appear progressively denser than single-condition and unconditional features.The PCA visualizations suggest single-condition and unconditional distributions are lower-dimensional projections of the multi-condition space.Source: B. Intermediate Feature Space · p. 8, Figure 6 · p. 8
  • To test feature continuity without biasing toward distributions with larger means, the authors interpolate shuffled batches of intermediate features using a perturbation factor λ.The perturbed features are defined as F + λ( F̂ − F).Source: B. Intermediate Feature Space · p. 9
  • MCM maintains stable generation quality across interpolation factors, unlike ReMoDiffuse, where interpolation degrades performance, supporting a relatively continuous latent space.The interpolation test uses FID; Table I compares FID under different perturbation factors λ.Source: B. Intermediate Feature Space · p. 9, Table I · p. 9
  • MCM features tolerate gradient-based perturbations at step t −1, allowing direct updates without causing xt−1 to deviate from its distribution or requiring ControlNet.This supports training-free guidance through updates to intermediate features.Source: B. Intermediate Feature Space · p. 9

C. Intermediate Feature Guidance

Intermediate Feature Guidance (IFG) updates an MCM intermediate feature with spatial guidance, while Mahalanobis-distance clipping constrains updates to a plausible feature-distribution region. The paper reports that this stabilizes guidance, improves DrawMotion performance, and shortens inference time.

C. Intermediate Feature Guidance

  • IFG uses SGD to optimize an intermediate MCM feature toward a spatial target, then continues reverse diffusion from the updated feature.Unconstrained regions are excluded from the reconstruction loss with a mask, allowing the resulting motion to retain fidelity while following spatial signals.Source: C. Intermediate Feature Guidance · p. 9
  • Mahalanobis-distance clipping constrains feature updates to a high-probability region, accounting for feature correlations and variance to prevent out-of-distribution artifacts.The method estimates the feature mean and covariance during evaluation and clips gradients when distance exceeds the original feature’s distance plus a threshold.Source: C. Intermediate Feature Guidance · p. 9
  • MD clipping stabilizes spatial guidance, improves DrawMotion performance, and shortens inference time.The paper reports these empirical effects in its analyses of Tables II and V.Source: C. Intermediate Feature Guidance · p. 9
  • The selected layer balances trajectory error, FID, and computation, while MD threshold and clip scale determine whether guidance updates destabilize generation.Deeper layers reduce trajectory error but increase FID; the paper selects Nth = 3. Threshold ϵMD = 1 improves both metrics, whereas smaller or larger thresholds harm trajectory error or both metrics. Retaining around 0.01 of the update gradient beyond the MD boundary is recommended; row 16 offers a good trajectory-error/FID trade-off, and more repeats improve results at higher computational cost.Source: C. Intermediate Feature Guidance · p. 10, Table II · p. 10

V. EXPERIMENTS · A. Experiment Settings · B. Quantitative Analysis

Experiments compare DrawMotion on KIT-ML and HumanML3D with text-to-motion and motion-editing methods, reporting favorable results alongside alignment choices, evaluation metrics, and implementation settings.

A. Experiment Settings

  • Evaluation uses KIT-ML and HumanML3D with the protocol of Guo et al., measuring R Precision, FID, MM Dist, Diversity, and Multimodality.The metrics assess text-motion retrieval, generated-versus-ground-truth feature distributions, motion-text embedding distance, sequence variability, and variation under a specified text.Source: A. Experiment Settings · p. 10
  • Experiments also report Stickman Similarity (StiSim) and 2D Trajectory error (Traj. err.) for comparison with motion-editing tasks.Input trajectories are obtained from the corresponding motion sequences, and stickmen are selected at the beginning, middle, and end as in StickMotion.Source: A. Experiment Settings · p. 10, B. Quantitative Analysis · pp. 10–11
  • DrawMotion is trained on 4 4090 GPUs with batch size 1024, uses T = 1000 diffusion steps, and has 4 KIT-ML or 6 HumanML3D MCM layers.The corresponding model sizes are 208M parameters for KIT-ML and 227M for HumanML3D, including condition encoders; 40 dataloader workers generate stickmen through SGA.Source: A. Experiment Settings · p. 10

B. Quantitative Analysis

  • DrawMotion shows excellent performance against prior text-to-motion methods on KIT-ML and HumanML3D, with reverse-process settings biased toward hand-drawing alignment.Because text and drawing conditions can conflict, the authors set p(ŵ = w) = 20% and (w1 = 1, w2 = 0, w3 = 0, w4 = 0).Source: B. Quantitative Analysis · p. 11, Table III · p. 11, Table IV · p. 11
  • DrawMotion has higher Stickman Similarity (StiSim) than StickMotion, supported by explicit stickman positioning, more training stickmen, and cross-attention.DrawMotion uses an average of 7 stickmen during training, while StickMotion uses at most 3 per sequence; the paper also attributes the difference to reduced positional ambiguity and its cross-attention structure.Source: B. Quantitative Analysis · p. 11
  • DrawMotion achieves the best reported performance and speed among compared diffusion-based motion-editing methods.The comparisons cover T2M and KIT datasets, and efficiency on HumanML3D at batch size 16.Source: B. Quantitative Analysis · p. 11, Table V · p. 11, Table VI · p. 11
  • Training-free motion-editing methods [60], have poor FID, while purely training-based methods, achieve only moderate trajectory error.The authors note that DNO theoretically should have better FID, but GPU memory limitations led its official implementation to use DDIM-10.Source: B. Quantitative Analysis · p. 11

C. Ablation Study

The ablations assess condition-decoder and MCM structures, condition mixtures, and stickman count; further evaluations report user alignment and practical trade-offs in guidance and animation production.

Analysis on the Structure of Condition Decoders

  • The results support efficient attention for text conditions and dot-product attention for drawing conditions, matching their global and local-detail roles, respectively.The study used the KIT-ML dataset; the supplied table caption identifies its scope but does not provide cell values.Source: C. Ablation Study · p. 12, Table VII · p. 12

Analysis on the Structure of Multi-Condition Module

  • An additional latent encoder enhanced MCM performance, while the traditional mask implementation and a fully connected layer tested the two components’ validity.The traditional-mask implementation is the first Table VIII row, and the second row has the minimum computational demand.Source: C. Ablation Study · p. 12, Table VIII · p. 12

Analysis on Condition Mixture

  • Results across condition-mixture configurations were relatively close, but p(ŵ = 80%) showed stronger drawing dependence; the third-row configuration was adopted for fair comparison and to ease sketching demands.The mixture combines outputs from four drawing/text-condition configurations, with coarse-stage bias set by p(ŵ = w) and final-stage refinement using weights (w1, w2, w3, w4).Source: C. Ablation Study · p. 12, Table IX · p. 12

Analysis on the Number of Stickmen

  • With more stickmen, R-Precision and StiSim generally improved but gains became marginal beyond 3; FID fluctuated slightly and was best at 7, while Diversity stayed largely unchanged.Stickmen provide semantic cues and affect only a small subset of frames; multiple stickmen can serve as mutual references for minor motion variations.Source: C. Ablation Study · p. 12, Table X · p. 12

Analysis on Intermediate Feature Guidance

  • Intermediate Feature Guidance parameter experiments informed configuration choice, and repeat-time analysis showed that improved performance came at the cost of increased computation time.The repeat-time analysis was conducted on HumanML3D with batch size 16.Source: C. Ablation Study · p. 12, Table XI · p. 14

Visualization of DrawMotion

  • For simple trajectory constraints, DrawMotion closely matched the input; for more complex trajectories, it sought vivid outputs without deviating too much.The passage identifies cases (a), (d), and (e) as simple and (b) and (c) as complex.Source: C. Ablation Study · p. 12, Figure 7 · p. 13

User Study

  • DrawMotion’s outputs were more aligned with users’ imagination, and the method saved users time while achieving the highest satisfaction.The user study involved 20 volunteers comparing DrawMotion and StickMotion with ReMoDiffuse; the passage does not provide the final time or user-score values.Source: C. Ablation Study · p. 14
  • Manual animation took about 3 hours per sample and scored 7.4 on average; animators found its trajectory adherence good but its motion naturalness weaker than AI-generated motion.The animators also reported that DrawMotion’s generation latency was acceptable in practice.Source: C. Ablation Study · p. 14, Table XII · p. 14

VI. LIMITATION · VII. CONCLUSION

DrawMotion offers flexible trajectory and pose control, but conflicting or physically unreasonable inputs can reduce fidelity. The paper concludes that its training-based and training-free guidance mechanisms support condition alignment while preserving generated-motion fidelity.

VI. LIMITATION

  • DrawMotion lets users specify a motion trajectory and character poses at designated positions along it.Source: §VI · p. 14
  • When user drawings conflict with text or basic human-motion principles, generated sequences often deviate from the input and lose fidelity.Source: §VI · p. 14
  • Users must ensure their inputs are roughly physically reasonable and semantically consistent to benefit from DrawMotion’s flexibility.Source: §VI · p. 14
  • The final guidance loss can be returned to users to indicate input conflict and help tune configurations.Source: §VI · p. 14

VII. CONCLUSION

  • DrawMotion combines a hand-drawing condition with an efficient motion-generation model to address detailed motion requirements beyond simple textual descriptions.Source: §VII · p. 14
  • Its training-based guidance uses stickman-based self-supervised encoding and an efficient Multi-Condition Module to map input conditions to generated motions.Source: §VII · p. 14
  • Training-free classifier guidance uses MCM’s continuous intermediate feature space to improve condition-generation alignment while preserving generation fidelity.Source: §VII · p. 14
  • Qualitative and quantitative experiments validate DrawMotion’s effectiveness, and the authors expect it to serve art creators and advance motion-generation research.Source: §VII · p. 15

Related papers

Loading 2605.20955v1…