HandX: Scaling Bimanual Motion and Interaction Generation

• Prior methods [52],, and [48] often miss fine-grained hand cues, while datasets and [19] [26] [38] and existing metrics leave gaps in hand detail and coordination evaluation.The cited datasets emphasize locomotion or narrow object interaction, and differences in skeletons, frame rates, and annotation protocols hinder unification.Source: §1 · p. 2
• HandX consolidates egocentric and human-object interaction datasets into a standardized, quality-controlled corpus and adds motion-capture sequences of dexterous two-hand interactions.The consolidated sequences are converted to a shared representation, with implausible or inactive segments filtered out.Source: §1 · p. 2
• A two-stage annotation strategy extracts structured motion events such as touch, slide, and release, then uses LLM reasoning to generate aligned, fine-grained descriptions.The approach enables scalable, consistent annotation with minimal manual effort.Source: §1 · p. 2
• In the core text-to-motion benchmark, larger model capacity and training data consistently improve text alignment and contact accuracy, and dexterous skills transfer to a humanoid platform.The humanoid platform is equipped with dexterous robot hands.Source: §1 · p. 2
• HandX benchmarks diffusion and autoregressive token-based models, using masked conditioning for reaction generation, in-betweening, and keyframe-guided synthesis, alongside contact-focused metrics.Source: §1 · p. 2

Human Motion Generation

• Text-to-motion progressed from latent/recurrent [2, 3, 21, 41] to autoregressive [25] [37] [58] and diffusion [47] [52] [56] [66] [74] models, yet often omits articulated hands.Widely used datasets treat hands as rigid SMPL end-effectors, while human-object interaction work [56] emphasizes object manipulation over bimanual coordination.Source: §2 · p. 2

Hand Motion Generation

• Text-driven hand synthesis remains underexplored: object-interaction methods [9] [11], communicative gestures [4] [6] [16], and CLUTCH [53] leave gaps in dexterous bimanual contact.CLUTCH’s action-level input limits motion granularity, while existing object-centric and gesture-focused approaches lack broad inter-hand coordination and contact dynamics.Source: §2 · p. 3

Hand Motion Datasets

• Existing datasets lack the precision, diversity, and rich inter-hand contact needed for fine-grained bimanual motion from text; HandX is proposed to bridge this gap.Motion-X and InterAct [28] mainly annotate whole-body motion; hand-centric datasets may lack language [38], while BOTH2Hands offers 8.31 hours of finger-level text annotations. BOBSL3DT [6] is sign-language-specific, and CLUTCH [53] uses coarse descriptions and monocular reconstruction.Source: §2 · p. 3

3. Dataset

• HandX combines high-quality open-source bimanual datasets, unified into a common skeleton and coordinate system, with newly captured motion rich in dexterity and inter-hand contact.Low-quality source sequences are filtered during aggregation.Source: §3 · p. 3
• A 36-camera OptiTrack system and 25 reflective markers per actor capture wrist, palm, finger, and fingertip motion during complex bimanual interactions.The hand skeleton is reconstructed from marker trajectories by estimating joint centers, enforcing anatomical bone-length constraints, and refining each frame for kinematic consistency.Source: §3 · p. 3
• Sequences are segmented into clips and filtered by joint angular velocity to remove static or near-static segments and retain meaningful interactions.The filter targets segments that could cause generative models to freeze.Source: §3 · p. 3

• HandX automates annotation in two stages: extracting structured kinematic features from motion and using an LLM to reason over them and generate descriptions.The framework is designed to produce large-scale, multi-level, fine-grained annotations.Source: §4 · pp. 3–4

Kinematic Feature Extraction

• Feature extraction encodes finger flexion, finger-palm distances, and inter-hand relationships, then segments descriptor changes and stable intervals into events organized as JSON for LLM interpretation.The event representation captures both dynamic transitions and steady states over time.Source: §4 · p. 4

Translating Kinematic Features into Natural Language

• Unlike template-based labeling [9] and concurrent work [6] [53], HandX produces diverse, motion-grounded descriptions with fine-grained bimanual relations at five progressively richer detail levels.The levels range from concise salient-movement summaries to comprehensive accounts of major events, subtle changes, and speed variations.Source: §4 · p. 4
• The LLM prompt requires descriptions of each hand and their relationships, critical events such as contact or separation, and temporal context that preserves event progression.Source: §4 · p. 4

5.1. Diffusion Model

• The diffusion model represents each joint with 3D coordinates and a rotation scalar, and separately encodes the three prompt types before cross-attending them with motion embeddings.Residual connections fuse the attended embeddings to predict denoised motion.Source: §5.1 · p. 5, Figure 2 · p. 5
• Inference-time partial denoising supports motion in-betweening, keyframe-based generation, wrist-trajectory generation, hand-reaction synthesis, and autoregressive long-horizon generation.At each denoising step, the method blends known input conditions with the current sample.Source: §5.1 · p. 5

5.2. Autoregressive Model

• The autoregressive model uses a local motion representation and an FSQ tokenizer, then predicts motion tokens from a text prefix and preceding motion tokens.The text-prefix model uses bidirectional attention among text tokens and causal attention in the motion branch.Source: §5.2 · pp. 5–6, Figure 2 · p. 5

6.1. Implementation Details

• Experiments vary training data across 5%, 20%, and 100%, diffusion depth across 4–16 layers, and autoregressive depth and FSQ codebook size; all models use the same validation split.The autoregressive tokenizer uses 1D convolutional encoder and decoder blocks with temporal downsampling factor 2, and inference selects the highest-probability token at each step.Source: §6.1 · p. 6

6.2. Metrics

• Evaluation measures realism and diversity with FID and Diversity, text alignment with R-Precision and MM Dist, and hand-contact accuracy with Cprec, Crec, and CF1.Contact labels come from ground-truth interaction annotations, with a 2 cm contact threshold.Source: §6.2 · pp. 6–7

6.3. Quantitative Evaluation

• Under a fixed 5% data budget, Rprec = 0.4391 × log10(FLOPS) −3.8707, reflecting the reported log-linear relationship between R-Precision and FLOPS.Figure 4 reports a correlation coefficient of 0.96 and evaluates R-Precision with batch size 16.Source: Figure 4 · p. 7, §6.3 · p. 7
• For autoregressive models, increasing FSQ codebook size alone does not reliably improve performance; jointly scaling codebook size and model capacity yields the strongest results.The table caption notes that FID is best when both are scaled up, while scaling only one can degrade performance; R-Precision is highest when model size is comparable to codebook size.Source: §6.3 · p. 7, Table 3 · p. 8

6.4. Qualitative Evaluation

• Qualitative visualizations show fine-grained finger articulation, realistic inter-hand coordination, and complex prompt-specified contact events across a range of generation tasks.Comparisons illustrate the visual effects of training-data scale and decoder-layer count on text alignment, contact, and generation fidelity.Source: §6.4 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 7

6.3. Quantitative Evaluation

• Diffusion scaling improves R-Precision and contact scores with more data or depth, but 6.7× more parameters than the 12-layer model lowers all metrics; 12 layers has the best overall contact performance.The table caption reports that Intra-hand CF1 is strongest with 12 decoder layers and all training data; scaling benefits are not strictly monotonic across every metric.Source: §6.3 · p. 7, Table 2 · p. 8

• HandX introduces a large-scale unified dataset that consolidates diverse motion sources and captures new high-fidelity, contact-rich bimanual interactions.Source: §7 · p. 8
• Its automatic captioning decouples kinematic feature extraction from LLM-based semantic reasoning to produce fine-grained, multi-level textual annotations.Source: §7 · p. 8
• The benchmark evaluates diffusion and autoregressive models across generation tasks including motion inbetweening and hand-reaction synthesis.Source: §7 · p. 8
• Jointly increasing dataset size and model capacity consistently improves text-motion alignment and contact accuracy.Source: §7 · p. 8

Supplementary Material

• The motion-capture configuration uses a 36-camera OptiTrack studio and 25 markers on each hand.Source: Supplementary Material · p. 12
• The supplement is organized into sections on motion capture and public-data construction with quality control, bimanual captioning, motion representations, versatile generation, and metrics, evaluation, and the user study.These topics appear in Sections A through E, respectively.Source: Supplementary Material · p. 12

A.1. Motion Capture

• Motion capture uses 36 infrared cameras and 25 markers per hand; inward-normal joint reconstruction and wrist optimization using calibrated MCP-to-wrist lengths address surface-marker offsets and soft-tissue artifacts.The wrist optimization keeps its position consistent with the relatively rigid palm.Source: §A.1 · p. 12

A.2. Data Consolidation and Unification

• HandX maps aggregated data to a unified 21-joint topology and canonicalizes sequence coordinates in a right-handed frame.In the first frame, x points left wrist to right wrist, y toward the fingertips, and z upward.Source: §A.2 · p. 13

A.3. Clip Extraction

• Sequences are divided into nonoverlapping 60-frame clips; defective frames are removed first, and clips are extracted only from valid contiguous intervals.At 30 FPS, each clip spans 2 seconds.Source: §A.3 · p. 13

A.4. Clip Filtering

• To counter low-activity imbalance, filtering retains clips where both hands move strongly, using joint angular velocity and thresholds τhand = 25 and τavg = 30.The intensity metric is independent of global hand motion and emphasizes proximal joints when computing hand-level intensity.Source: §A.4 · pp. 13–14

A.5. Dataset Metrics

• Dataset interaction quality is measured by Contact Ratio, Contact Duration, Contact Freq, and Motion Intensity.These quantify contact-frame fraction, average continuous-contact span, contact events per minute, and fine-grained finger dynamics, respectively.Source: §A.5 · p. 14

B. Additional Details on Motion Captioning

• Captioning extracts six descriptors as events: Finger Flexing, Finger Spacing, Finger-finger Distance, Palm-palm Relation, Finger-palm Distance, and Wrist Trajectory.JSON-formatted features are supplied to an LLM to generate five annotations at varying granularity; constant feature values can also constitute events.Source: B. Additional Details on Motion… · pp. 14–15

C. Hand Motion Representation

• The motion representation concatenates each joint’s 3D coordinates with a rotation scalar derived by projecting its finger-chain angle onto the plane perpendicular to the little-to-index MCP vector.The resulting sequence representation has shape F × 2J × 4.Source: C. Hand Motion Representation · p. 15

D.1. Masked Partial Denoising for Skeleton Control

• Masked partial denoising conditions selected motion degrees of freedom while generating the rest, with constraints softly interpolated from keyframes to nearby frames.Constraint weights decay linearly with temporal distance and are strongest at center frames; the modified clean prediction is then re-noised.Source: §D.1 · p. 15

D.2. Task-Specific Mask Construction

• In-betweening constrains all joints at the sequence’s first and last five frames, while keyframe generation enforces sparse poses on the full skeleton.For in-betweening, Kinbet = 5.Source: §D.2 · p. 15
• Wrist-trajectory generation constrains wrist joints throughout the sequence while leaving fingers free, whereas hand-reaction synthesis fully constrains one hand.The other joints have γt,j = 0 in wrist-trajectory generation.Source: §D.2 · p. 16
• Long-horizon generation proceeds autoregressively, constraining the current sequence’s start with the preceding sequence’s last Khor + ktrans frames.The temporal smoothing radius is ktrans.Source: §D.2 · p. 16

E.1. Contact Metrics

• Intra-hand contact measures whether the thumb fingertip comes within a threshold distance of each other fingertip at any frame.The metric covers the thumb’s contact with the other four fingertips within a single hand.Source: §E.1 · p. 16
• Inter-hand contact is detected when the minimum distance between the closest point pair across the two hands falls below a threshold.Source: §E.1 · p. 16
• The evaluation extracts intra-hand and inter-hand contact labels from ground-truth and generated motion, computes TP, TN, and FP, and reports standard metrics.Intra-hand evaluation is presented in Table 2, and inter-hand precision in Table D.Source: §E.1 · p. 16

E.2. Evaluator Details

• Annotations are evaluated for finger bending, intra-hand contact, inter-hand contact and spatial relations, and completeness against the motion video.The prompt requests five JSON annotations describing left and right hands and their relation, restricted to physical motion rather than gesture meaning or intent.Source: §E.2 · pp. 17–18
• Motion quality is rated for smoothness, coherence, naturalness, expressiveness, and whether the motion conveys clear purpose rather than appearing random.Source: §E.2 · p. 18
• The evaluator jointly trains text and hand-motion encoders with sequence-level InfoNCE contrastive learning, using global hand-joint positions and T5-encoded prompts.The prompts combine left-hand, right-hand, and inter-hand-relation text, separated by tokenizer-defined tokens.Source: §E.2 · pp. 16, 18

E.3. User Study on Data Quality

• HandX significantly surpasses direct video captioning with Gemini 3 Pro [51] in annotation quality and GigaHands [19] and HoloAssist [55] in motion quality.Participants rated samples from 1 to 5, with higher scores indicating better quality.Source: §E.3 · p. 18
• The data-quality study recruited 20 participants, randomly distributed sequences, and obtained ratings from at least three independent participants per sequence.The study evaluated annotation and motion quality as separate dimensions.Source: §E.3 · p. 18

E.4. User Study on Scaling Trend

• 48% of votes favored motion trained on 100% of the data, versus 33% for 5% and 19% for 20%.Ten participants compared generated videos for 10 prompts, considering motion naturalness and semantic alignment; motions were rendered as 3D meshes with optimized camera angles.Source: §E.4 · pp. 18–19

F. License

• Users must follow each source dataset’s license; HandX includes assets with differing usage terms.GigaHands [19] uses CC BY-NC 4.0; HOT3D [5] its dataset agreement; ARCTIC [14] a non-commercial research license; H2O [26] academic/non-commercial terms; and HoloAssist [55] CDLA v2 Permissive.Source: F. License · p. 19

G. Discussion

• HandX remains finite in volume and diversity, and filtering cannot eliminate all quality issues in aggregated public data.The remaining issues can include minor jitter or kinematic implausibility, and the dataset cannot cover every form of dexterity or real-world interaction scenario.Source: G. Discussion · p. 19
• Text-conditioned hand-motion generation risks deepfake misuse; the authors restrict code and data to research and non-commercial use.They report obtaining participant consent and releasing only skeletal motion representations without personally identifiable features.Source: G. Discussion · p. 19