Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interaction

1. Introduction

• Scarce and unrepresentative labeled data limits diverse, physically and semantically plausible text-guided interaction modeling.Existing text-guided human motion work often omits object-interaction context, limiting generated-motion semantics.Source: §1 · pp. 1–2

3.1. Contact map prediction

• The contact predictor combines local object-surface geometry, text, and object scale to generate category-agnostic contact probabilities for general objects.Its scale conditioning allows predicted contact regions to vary with object size.Source: §1 · p. 2

3. Method

• Text2HOI’s Transformer-based diffusion generator uses contact maps and geometric features as guidance, while a hand refiner adjusts contacts and suppresses penetration artifacts.The framework uses frame-wise and agent-wise positional embeddings to encode motion sequences and their left hand, right hand, and object agents.Source: §1 · p. 2, Figure 2 · p. 3, §3.2.1 · p. 4

1. Introduction

• Across H2O, GRAB, and ARCTIC, Text2HOI outperforms baselines in accuracy, diversity, and physical realism, and applies to objects unseen during training.The conclusion also reports strong physical plausibility and accuracy against three baselines.Source: §1 · p. 2

2. Related Work

• Prior hand-object methods focus on stationary-object grasping or require object-motion inputs, while existing interaction methods do not use text.The related-work discussion also notes that one full-body interaction method requires an optimization stage.Source: §2 · pp. 2–3

3.2.2 Forward process

• The diffusion model represents each frame with left-hand, right-hand, and object motion, then adds Gaussian noise to the original motion at each timestep.Its forward process uses a timestep-dependent noise schedule.Source: §3.2 · p. 3, §3.2.1 · p. 3, §3.2.2 · p. 4

3.2.3 Backward process

• In the backward process, Text2HOI estimates the original motion from noisy motion conditioned on text, contact maps, object features, and scale.Distance-map and relative-orientation losses support geometric accuracy; the distance-map loss activates for hand-object distances below an empirical 2 cm threshold.Source: §3.2.3 · pp. 4–5

3.3. Hand refinement network

• The hand refiner receives generated hand motion, joints, contact maps, object points, and distance-based attention, and outputs refined hand motion.Its training loss combines a simple motion loss with penetration and contact losses.Source: §3.3 · p. 5

4.1. Implementation details

• Implementation selects left-, right-, or both-hand conditioning by CLIP similarity and predicts motion length from text features and Gaussian noise, with a 150-frame maximum.The diffusion model uses 1,000 noising steps and sinusoidal frame-wise and agent-wise positional encodings.Source: §4.1 · p. 6

4.2. Dataset

• The experiments use H2O, GRAB, and ARCTIC; prompts are generated from action labels for H2O and GRAB and manually labeled for ARCTIC.Further dataset characteristics and annotation details are deferred to the supplemental material.Source: §4.2 · p. 6

4.3. Evaluation metrics and baselines

• The baselines are Text2Motion, MDM, and IMOS, retrained on hand-object motion data for comparison with text-guided hand-object generation.Figure 4 compares generated motions from these baselines with Text2HOI.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4. Experimental results

• Across all three datasets, Text2HOI outperforms baselines on multiple measures, with the highest physical-realism scores and strong diversity, multimodality, and accuracy results.Qualitative comparisons report realistic, text-aligned interactions, including with objects unseen during training.Source: §4.4 · p. 7

4.5. Ablation study

• The ablations report that frame-wise and agent-wise positional encodings and distance-map and relative-orientation losses improve hand-object motion realism and 3D relationship modeling.The study evaluates these components on GRAB.Source: §4.5 · p. 7
• Removing contact-map and object-scale conditions reduces the appropriateness of generated hand poses to the object.The study compares removing both conditions or either one.Source: §4.5 · p. 8
• The refiner improves physical realism, while adding its contact and penetration losses consistently improves performance.Figure 5 presents generated motions and predicted contact maps for objects seen and unseen during training.Source: §4.5 · p. 8, Figure 5 · p. 8

5. Conclusion

• The method estimates text-guided, scale-variant contact maps, generates hand-object motion with Transformer-based diffusion, and refines contacts and penetration.The input is a text prompt and canonical object mesh.Source: §5 · p. 8
• In comparisons with three baselines, the method is reported to improve physical plausibility and accuracy.Source: §5 · p. 8

H2O

• H2O prompts are generated automatically from action and object labels, with interacting hand type inferred using a 2 cm proximity threshold; the process yields 272 sentences.H2O contains 660 interactions involving two hands and eight objects.Source: §6 · p. 10

GRAB

• GRAB prompts are generated automatically from action and object labels, with hand type inferred by proximity, yielding 1,104 sentences for 1,335 motions.The dataset includes 51 objects and 29 action labels.Source: §6 · p. 10

ARCTIC

• ARCTIC receives manual video-based annotations of action, hand type, and object category, producing 644 sentences for 4,597 motions.The annotations use 11 defined action labels.Source: §6 · p. 10

Text prompt annotation

• Prompts specify action, object category or name, and interacting hand type, with additional passive, subject-modified, and gerund forms used as augmentations.One basic format is “{action} {object category} with {hand type}.”Source: §6 · p. 10

7. Qualitative results for ablation studies

• The geometry-loss and conditioning comparisons, followed by positional-encoding comparisons, are presented in Figure S1.The evaluated factors include distance-map and relative-orientation losses, contact-map and scale conditions, and frame-wise and agent-wise positional encodings.Source: §7 · p. 10, Figure S1 · p. 11
• Figures S2 and S3 illustrate hand refinement and the effects of removing penetration and contact losses, respectively.The reported prompt for these qualitative examples is “Use headphones with both hands.”Source: §7 · pp. 10–11, Figure S2 · p. 11, Figure S3 · p. 11
• Predicted contact maps reflect text prompts, while generated motions vary with both the maps and prompts.The prompt-dependent examples are shown in Figure S4.Source: §7 · p. 11, Figure S4 · p. 11
• Object scale changes predicted contact maps, influencing the number of fingers involved and the manner of grasping.Figure S5 illustrates variation in contact maps and generated motion across object scales.Source: §7 · p. 11, Figure S5 · p. 11

8. Refiner design

• Table S1 compares physical realism scores across different refiner designs.Source: Table S1 · p. 12
• Figure S6 compares physical realism scores for ground-truth, Gaussian-noised, and refined motion parameters.The noisy parameters are produced by adding Gaussian noise to ground-truth motion parameters.Source: Figure S6 · p. 12
• The proposed refiner fref does not use diffusion and demonstrates superior results to the diffusion-based refinement design in Table S1.The diffusion-based approach adds noise to already generated motions and then denoises them.Source: §8 · p. 12

9. Effect of refiner

• The hand refinement network highly increases the physical realism score.Source: §9 · p. 12
• The refined motion’s physical realism score outperforms that of ground-truth motion.Source: §9 · p. 12

10. Inference speed

• The method generates 150 frames faster than the diffusion-based method MDM [27].Source: §10 · p. 12
• Inference time is measured for generating 150 frames of hand-object interaction using an RTX 4090.Source: §10 · p. 12

11. Notations

• Tab. S3 summarizes the notations used in the main paper and supplementary material.Source: §11 · p. 12

12. Network

• The contact predictor combines object geometry, text, scale, and a latent code to predict a surface contact map.It extracts normalized point-cloud features and CLIP text features; the latent code is encoded from contact data during training and sampled from a Gaussian at inference.Source: §12 · pp. 12–13
• The interaction generator takes noised motion and conditions including text, object features, contact map, and object scale to produce denoised motion.Its architecture includes input and condition embeddings, a Transformer encoder, and output embeddings.Source: §12 · p. 13
• The hand refiner uses generated hand motion, joints, contact estimates, object points, and distance-based attention to produce refined hand motions.The inputs are concatenated and passed through a Transformer encoder.Source: §12 · p. 14

• For a left- or right-hand prompt, the opposite hand’s inputs, outputs, and losses are masked; both-hand prompts use no masking.Source: §13 · pp. 14–15
• The hand-type indicator maps left, right, and both-hand prompts to the corresponding left/right activation pair.Source: §13 · p. 15
• Input masking inhibits attention, output masking blocks visualization, and loss masking restricts backpropagation for the masked losses.Source: §13 · p. 15

• Baseline model architectures, training schemes, and inference processes were maintained, with output dimensions adjusted for two hands and the object.Source: §14 · p. 15
• The hand-refinement network architecture is listed in Table S10.Source: Table S10 · p. 15
• Baselines use pre-estimated motion lengths when available and otherwise use their predefined fixed lengths.Source: §14 · p. 15

15. How to use the articulation parameter

• Articulation parameters are generated for all objects, but applied only when the dataset marks an object as articulated.ARCTIC uses the parameter; for rigid objects in H2O and GRAB it is a placeholder and is not applied.Source: §15 · p. 15

16. More qualitative results

• The supplementary figures show varied interactions from one laptop prompt, hand-pose changes in canonical coordinates, and results for different hand types.Frames are ordered left to right, then top to bottom, to show temporal progression.Source: §16 · p. 15, Figure S8 · p. 17, Figure S9 · p. 18

17. Limitation.

• The generated interactions model relative hand-object location and contact but omit forces, which could provide better physical understanding.Source: §17 · p. 16