PaperlayerView exact revision on arXiv

Source-linked AI summary

HandX: Scaling Bimanual Motion and Interaction Generation

Zimu Zhang, Yucheng Zhang, Xiyan Xu, Ziyin Wang, Sirui Xu, Kai Zhou, Bing Zhou, Chuan Guo, Jian Wang, Yu-Xiong Wang, Liang-Yan Gui

arXiv:2603.28766v1cs.CV

TL;DR

Fine-grained bimanual hand synthesis lacks suitable data and evaluation for finger articulation, contact timing, and inter-hand coordination. HandX unifies standardized and newly captured motion data with scalable feature-based LLM annotation, then benchmarks diffusion and autoregressive models. The benchmark finds consistent gains in text-motion alignment and contact accuracy as model capacity and training data increase, while larger scaling can eventually saturate.

  • Problem

    Existing motion resources and methods often lack fine-grained hand detail, bimanual interaction richness, suitable annotations, and hand-focused evaluation.

  • Method

    HandX consolidates and standardizes motion data, adds high-fidelity motion-capture interactions, generates event-aligned descriptions through feature extraction and LLM reasoning, and benchmarks diffusion and autoregressive models.

  • Results

    HandX experiments show that increasing training data and model capacity generally improves text-motion alignment and contact quality across diffusion and autoregressive models.

  • Takeaways & Limitations

    HandX provides a unified dataset and benchmark foundation for expressive bimanual motion and interaction synthesis.

  • Takeaways & Limitations

    The dataset remains finite in volume and diversity and cannot exhaustively cover human dexterity or every real-world interaction scenario.

Abstract

from arXiv · show

Synthesizing human motion has advanced rapidly, yet realistic hand motion and bimanual interaction remain underexplored. Whole-body models often miss the fine-grained cues that drive dexterous behavior, finger articulation, contact timing, and inter-hand coordination, and existing resources lack high-fidelity bimanual sequences that capture nuanced finger dynamics and collaboration. To fill this gap, we present HandX, a unified foundation spanning data, annotation, and evaluation. We consolidate and filter existing datasets for quality, and collect a new motion-capture dataset targeting underrepresented bimanual interactions with detailed finger dynamics. For scalable annotation, we introduce a decoupled strategy that extracts representative motion features, e.g., contact events and finger flexion, and then leverages reasoning from large language models to produce fine-grained, semantically rich descriptions aligned with these features. Building on the resulting data and annotations, we benchmark diffusion and autoregressive models with versatile conditioning modes. Experiments demonstrate high-quality dexterous motion generation, supported by our newly proposed hand-focused metrics. We further observe clear scaling trends: larger models trained on larger, higher-quality datasets produce more semantically coherent bimanual motion. Our dataset is released to support future research.

HandX Dataset

HandX presents high-fidelity bimanual and dexterous hand motions with fine-grained descriptions, including detailed finger articulation, contact, and inter-hand movement.

  • Finger articulation includes continuous bending, posture opening, and coordinated extension across individual fingers.
  • Bimanual examples capture sustained fingertip–palm contact, rapid release, and changing relative hand positions.
  • HandX pairs bimanual and dexterous motions with fine-grained textual descriptions.

1. Introduction

Hand motion synthesis is limited by insufficiently detailed data and evaluation, motivating HandX’s unified data, annotation, benchmarking, and scaling framework.

  • Existing motion methods often miss precise finger articulation, contact timing, and smooth bimanual coordination under semantic intent.
  • Available datasets provide limited hand detail, narrow interaction coverage, coarse annotations, or incompatible formats, while metrics rarely assess hand fidelity or bimanual coordination.
  • HandX standardizes consolidated datasets, adds motion-captured dexterous two-hand interactions, and generates event-aligned descriptions through feature extraction and LLM reasoning.
  • The benchmark compares diffusion and autoregressive models with masked conditioning for reaction generation, in-betweening, and keyframe-guided synthesis.
  • HandX supports scaling analysis, with increasing model capacity and training data improving text alignment and contact accuracy.

2. Related Work

Prior motion-generation methods span multiple paradigms, but text-driven fine-grained bimanual hand synthesis remains underserved by existing datasets and annotations.

  • Human Motion Generation: Human motion generation has progressed from latent-variable and recurrent models to autoregressive and diffusion approaches.
  • Human Motion Generation: Existing methods are generally not designed to generate hand motion directly from free-form natural language.
  • Hand Motion Datasets: Full-body datasets emphasize whole-body annotations, whereas hand-centric datasets may lack language supervision or fine-grained coverage.
  • Hand Motion Datasets: Existing resources lack the precision, diversity, and rich inter-hand contact needed for fine-grained bimanual motion from text.

3. Dataset

HandX addresses the lack of suitable fine-grained bimanual text-to-motion data by constructing a large-scale benchmark from standardized high-quality sources.

  • HandX aggregates high-quality open-source bimanual motion data into a unified skeletal representation and coordinate system.

4. Bimanual Motion Captioning

HandX uses a two-stage captioning framework that extracts structured kinematic features and applies LLM reasoning to generate fine-grained descriptions of bimanual motion. Its dataset comparison emphasizes multi-level language descriptions and contact-rich motions.

  • HandX annotation first extracts structured kinematic features from raw hand motion, then uses an LLM to generate coherent textual descriptions.
  • HandX provides fine-grained, multi-level language descriptions and contact-rich bimanual motions in the dataset comparison.
  • The prompting strategy separately describes the left hand, right hand, and inter-hand relationships to cover local articulation and global coordination.
  • The prompts require critical events such as contact, separation, and hyperextension, while incorporating temporal context to preserve event progression.

5. Bimanual Motion Generation

HandX benchmarks diffusion and autoregressive approaches for generating two-hand motion from separate left-hand, right-hand, and inter-hand descriptions. The models use distinct representations and conditioning mechanisms, with the diffusion model supporting versatile inference-time control.

  • The task generates a two-hand motion sequence consistent with text describing the left hand, right hand, and inter-hand motion.
  • Diffusion Model: The diffusion model separately cross-attends left-hand, right-hand, and bimanual text embeddings with noisy motion embeddings before residual fusion and denoising.
  • Diffusion Model: Diffusion motion representations concatenate 3D joint coordinates with one compact rotation scalar per joint at each frame.
  • Versatile Bimanual Motion Generation: Inference-time partial denoising blends known input constraints with the current sample, enabling diverse spatiotemporal and conditional generation tasks.
  • Autoregressive Model: The autoregressive model uses a local motion representation, FSQ tokenization, and a text prefix formed from the three prompt types.
  • Autoregressive Model: Autoregressive generation predicts each motion token from preceding motion latents and the concatenated text prefix using causal motion attention.

6. Experiments

Experiments evaluate scaling across data, model capacity, and generation paradigms using text-alignment, realism, diversity, and contact-focused metrics. Results show positive but regime-dependent scaling, with matched data and capacity improving quality while excessive or mismatched scaling can hurt.

  • Experimental Setup: The benchmark varies training data from 5%, 20%, and 100% and evaluates diffusion and autoregressive models across multiple capacity configurations.Diffusion models use 4, 8, 12, and 16 decoder layers; autoregressive models vary Transformer layers and FSQ codebook size.
  • Metrics: The evaluation measures realism, diversity, text alignment, and bimanual contact accuracy using FID, Diversity, R-Precision, MM Dist, Cprec, Crec, and CF1.Contact events are matched at corresponding frames using a 2 cm threshold.
  • Scaling Results: Both diffusion and autoregressive models show generally positive scaling trends in text-motion alignment and hand-contact quality, although improvements are not strictly monotonic for every metric.The trend is observed as both training data and model capacity increase.
  • Diffusion Models: Diffusion scaling improves primary metrics, especially R-Precision and contact scores, with the 12-layer model achieving the best overall contact performance.An ultra-large model with 6.7× more parameters than the 12-layer model drops across all metrics, indicating saturation beyond an appropriate regime.
  • Autoregressive Models: Autoregressive performance is strongest when FSQ codebook size and model capacity increase jointly, whereas increasing codebook size alone is unreliable.Finer discrete representations benefit from sufficient autoregressive capacity.
  • Qualitative Evaluation: Qualitative visualizations demonstrate fine-grained finger articulation, realistic inter-hand coordination, and complex contact events specified by text prompts.The benchmark supports a wide range of generation tasks.

7. Conclusion

HandX unifies data, annotation, evaluation, and generative modeling for realistic text-conditioned bimanual hand motion. Its benchmark supports versatile generation and reveals that jointly scaling data and model capacity improves alignment and contact accuracy.

  • Contributions: HandX consolidates diverse motion sources and adds high-fidelity, contact-rich bimanual interactions to form a unified dataset.The dataset targets dexterous two-hand interactions and supports realistic bimanual motion generation.
  • Contributions: A decoupled annotation strategy combines kinematic feature extraction with LLM reasoning to produce fine-grained, multi-level textual descriptions.The descriptions are aligned with motion events and dynamics.
  • Contributions: The benchmark covers diffusion and autoregressive models, masked conditioning, versatile generation tasks, and contact-focused evaluation.Supported tasks include hand reaction generation, motion in-betweening, and keyframe-guided synthesis.
  • Findings: Jointly increasing dataset size and model capacity consistently improves text-motion alignment and contact accuracy.The results provide empirical scaling trends for dexterous hand motion synthesis.

Supplementary Material

The supplementary material documents the motion-capture setup, HandX construction, captioning pipeline, motion representations, generation controls, and additional evaluation materials.

  • Supplementary Organization: Section A describes the motion-capture system and construction of HandX from public data, including quality-control procedures.The capture configuration uses a 36-camera OptiTrack studio with 25 markers on each hand.
  • Supplementary Organization: Sections B–E cover bimanual motion captioning, motion representations, versatile generation, and additional metrics, evaluation results, and the user study.The supplementary material provides implementation and evaluation details for the main paper.

A. HandX Dataset

HandX standardizes heterogeneous hand-motion data, reconstructs anatomically consistent skeletons, and filters clips to retain valid, informative bimanual dynamics. It also extracts kinematic descriptors and evaluates interaction quality using contact and motion-intensity metrics.

  • Motion Capture and Reconstruction: OptiTrack captures 25 surface markers spanning the wrist, palm, finger joints, and fingertips for detailed hand articulation.
  • Motion Capture and Reconstruction: Skeleton joints are recovered by offsetting marker positions along anatomical normals using depth values scaled to calibrated hand size.
  • Motion Capture and Reconstruction: Wrist optimization assumes constant MCP-to-wrist bone lengths and minimizes per-frame distance errors to suppress soft-tissue artifacts.
  • Clip Processing: Long sequences are segmented into nonoverlapping 60-frame clips, with defective frames removed before partitioning valid contiguous intervals.
  • Metrics and Features: Dataset quality is characterized by Contact Ratio, Contact Duration, Contact Freq, and Motion Intensity, while extracted descriptors encode finger and spatial dynamics.

D.2. Task-Specific Mask Construction

HandX uses joint-specific masks to support multiple controlled generation tasks and evaluates interaction fidelity with intra-hand and inter-hand contact metrics. Its annotation prompts convert structured motion features into varied textual descriptions.

  • Task-Specific Mask Construction: Joint-specific center-frame sets configure motion in-betweening, keyframe-based generation, wrist control, hand reactions, and long-horizon synthesis.
  • Motion-Specific Masks: Motion in-betweening constrains the first and last five frames for every joint while generating the sequence interior.
  • Motion-Specific Masks: Wrist-trajectory generation constrains wrist joints throughout the sequence while leaving finger joints unconstrained.
  • Interaction Evaluation: Contact evaluation separates intra-hand fingertip contacts from inter-hand closest-point contacts and reports standard metrics from generated and ground-truth labels.
  • Annotation Prompting: The annotation prompt asks an LLM to produce five JSON annotations describing left-hand, right-hand, and two-hand physical motion without interpreting gesture intent.
  • Quality Evaluation: A user-study framework evaluates annotation correctness through finger bending, intra-hand contact, and inter-hand interaction questions, alongside motion quality.

E.3. User Study on Data Quality

The data-quality user study compares HandX annotations with direct video captioning and its motion with existing bimanual datasets. A separate scaling study measures perceptual quality across training-data sizes.

  • Study Design: Twenty participants rated samples on a 1–5 scale, with at least three independent ratings per sequence.
  • Data Quality Results: HandX significantly surpasses direct motion captioning in annotation quality and existing bimanual datasets in motion quality.
  • Scaling Study: The scaling study samples 10 test prompts and compares diffusion models trained on subsets containing 5%, 20%, and 100% of the training data.
  • Scaling Study: 48% of votes favored the model trained on 100% of the data, versus 33% for 5% and 19% for 20%.
  • Limitations: HandX remains finite in volume and diversity, cannot cover every dexterity or interaction scenario, and inherits some residual quality issues from public source datasets.
Loading 2603.28766v1…