InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation

1. Introduction

• The benchmark consolidates carefully curated interactions from existing resources and enriches them with detailed textual annotations.The authors position this as a response to inconsistent datasets and coarse, incomplete annotations.Source: §1 · p. 2
• A unified optimization framework corrects whole-body penetration and floating artifacts before refining hand-object interactions.The authors also identify inaccurate hand poses and motion jitter as prevalent dataset artifacts.Source: §1 · p. 2
• Contact-invariance augmentation varies human motion while maintaining object contacts, expanding InterAct into InterAct-X with approximately 9 additional hours of data.The passage also reports substantially improved generative-model performance after augmentation.Source: §1 · p. 2
• InterAct benchmarks six HOI generation tasks and achieves state-of-the-art performance with a unified modeling and representation approach.The tasks include text-to-interaction, action-to-interaction, object-to-human, human-to-object, interaction prediction, and interaction imitation.Source: §1 · p. 2

2. Related Work

• InterAct targets whole-body interactions with dynamic objects, from approaching and manipulation to interactions involving multiple body parts.The related-work discussion distinguishes this scope from datasets focused on hand-object interactions.Source: §2 · pp. 2–3
• InterAct emphasizes motion quality, fine-grained textual annotations, detailed hand gestures, and comprehensive annotation modalities compared with existing datasets.The authors state that Table 1 quantitatively compares these dataset aspects.Source: §2 · p. 3
• Prior dynamic HOI generation work includes hand-object, static, and full-body approaches, while some full-body methods have narrow action repertoires and rely on static objects.The passage also notes physical inaccuracies and a shortage of high-quality, large-scale datasets in current methods.Source: §2 · p. 3

3. InterAct Dataset

• The basic InterAct version consolidates seven datasets into 21.81 hours of annotated 3D whole-body interactions with semantic descriptions.An advanced version, InterAct-X, extends the basic dataset with synthetic data; annotation quality is checked through manual review.Source: §3 · p. 3

3.1. Data Collection, Annotation, and Unification

• Human annotators and GPT-4 create, rephrase, simplify, and classify interaction descriptions, which are manually reviewed for quality and alignment.Sequences are split into clips averaging 300 frames and no longer than 400 frames; descriptions identify actions and interacting body parts.Source: §3.1 · p. 3
• Human markers provide a unified representation across SMPL-H and SMPL-X, with marker correspondences showing maximum approximation error below 1 cm.Markers represent motion and interactions and support contact inference; generative models use this representation, while correction and augmentation retain the original body models.Source: §3.1 · p. 4

3.2. Interaction Correction and Augmentation

• Correction proceeds through full-body correction, hand correction, and interaction augmentation, with hand correction separated to balance its objectives against whole-body objectives.Hand correction promotes contact only where ground truth indicates hand-object interaction, while applying penetration, smoothness, and finger-range constraints; full-body correction also uses reconstruction loss.Source: §3.2 · pp. 4–5
• Augmentation displaces an object trajectory, realigns human motion to preserve contact consistency and reconstruction, then filters out low-quality results.Filtering removes unreasonable displacements, significant human-object or self-penetration, and alignment failures such as excessive jitter.Source: §3.2 · p. 5
• The correction objective preserves initially close vertex-pair distances more strongly than farther-pair distances, while additional terms encourage naturalness and stability in non-interactive pose regions.The framework takes human motions and object geometries as input and iteratively adjusts poses using gradient-based optimization.Source: §3.2 · pp. 4–5

4. Tasks and Methods

• Across five kinematic tasks, sequences pair human and object motion with action and text annotations; human features include marker coordinates, velocities, signed distance vectors, and foot-ground contact labels.Object motion includes rotations and translations, and object geometry is represented with Basis Point Set.Source: §4 · p. 5
• The six benchmark tasks are text-conditioned generation, action-conditioned generation, object-conditioned human generation, human-conditioned object generation, interaction prediction, and interaction imitation.The first five are kinematic generative tasks; imitation instead learns physics-based control policies in a simulator.Source: §4 · p. 5
• The unified multi-task framework adds human-object relationship vectors to the outputs of the first five tasks, prompting the model to learn spatial relationships within interactions.Interaction imitation maps reference interactions to joint proportional-derivative target action sequences.Source: §4 · p. 5

5. Experiments

• The experiments evaluate correction and augmentation before benchmarking methods, standardizing metrics and including ablation studies.Source: §5 · p. 5

5.1. Correction and Augmentation

• Correction improves contact and reduces penetration; augmented data is comparable to corrected data and exceeds raw data in quality.Over 39% of judges ranked corrected data highest, while augmented data received ratings comparable to corrected data.Source: Table 2 · p. 6, §5.1 · p. 6

5.2. Language Conditioned HOI Generation

• Contact modeling and BPS encoding substantially improve FID, while interaction-aware encoders improve interaction quality and text alignment.Classifier guidance provides a slight overall performance improvement.Source: Table 3 · p. 7, §5.2 · p. 7
• Marker-based representation produces interactions with fewer artifacts than other representations without changing contact modeling.Source: §5.2 · p. 7

5.4. Interaction Prediction

• Larger-scale training on InterAct supports improved interaction prediction, whereas training on limited data leads to overfitting.Source: Table 8 · p. 8, §5.5 · p. 8

5.5. Interaction Imitation

• 90.7% success with corrected data exceeds 84.4% with raw data in interaction imitation.The rates average four sequences over 2,048 environments.Source: §5.5 · p. 8

• InterAct combines correction and augmentation in a unified optimization framework, scaling the dataset to 30.70 hours and 48,630 textual descriptions.Source: §6 · p. 8

• The planned release includes consolidated and synthetic MoCap data and models for benchmarking tasks.Supplementary materials include demo videos, further process and implementation details, licensing information, and discussion of limitations and potential societal impacts.Source: Generation · p. 14

A.1. Webpage

• The website demonstrates generated results across six interaction-generation tasks.The demos complement the paper’s quantitative results.Source: §A.1 · p. 14
• The demos compare raw data with corrected and augmented versions.Source: §A.1 · p. 14
• Marker-based representations are reported as better suited to interaction modeling than joint-position and rotation-based representations.The stated reason is that contact occurs on surfaces rather than joints.Source: §A.1 · p. 14

B. Data Collection, Annotation, and Unification

• GPT-4 is used to rephrase and simplify collected interaction annotations.The simplification prompt asks that descriptions retain their meaning and the object’s name.Source: §B.1 · p. 14
• GPT-4 assigns each annotation a single best-fitting label from 15 predefined action categories.The listed categories include Carry, Sit, Hold, Lift, and Manipulate.Source: §B.1 · p. 14
• Manually annotated action labels are included in prompts to improve in-context learning and response accuracy.Source: §B.1 · p. 14

B.2. Processing of Each Sub-dataset

• Source datasets are aligned to a unified global coordinate system and ground heights, with frame rates and human annotations standardized as needed.Processing includes downsampling or interpolation and dataset-specific choices such as selecting rigid-object sequences for Chairs.Source: §B.2 · pp. 14–15

B.3. Interaction Correction

• Interaction correction combines penetration, smoothness, and reconstruction objectives to refine human and object motion and address body-object interpenetration.The hand-correction objectives also include a prior that maintains natural poses and prevents joints from exceeding GRAB-derived range-of-motion limits.Source: §B.3 · p. 15

B.4. Interaction Augmentation

• Augmentation aligns the human body to maintain object contact while regularizing non-contact body parts and promoting temporal smoothness.The mask applies to joints not involved in interaction; beta is set to 5 to emphasize these vertices, and the pose is updated for 300 iterations.Source: §B.4 · pp. 15–16

C.1. Marker-Based Representation with Shape Variance

• Generated human shapes varied from 1.71m to 1.81m in height across 50 batches, despite the task not targeting specific shape control.The model uses a marker-based representation that couples pose and shape.Source: §C.1 · p. 16

C.3. Action-Conditioned Interaction Generation

• Action-conditioned generation rephrases labels as “A person [action] the [object name]” and uses the text-conditioned task’s data split, motion representation, and loss functions.Experiments ran on one NVIDIA A40 GPU for eight days.Source: §C.3 · p. 16

C.4. Object-Conditioned Human Generation

• Object-conditioned human generation uses disjoint object-category splits with 168 training objects and 29 unseen test objects.The model uses four self-attention blocks with four heads each and trains for 300k iterations.Source: §C.4 · p. 16

C.5. Human-Conditioned Object Generation

• Human-conditioned object generation splits the data at a 10:1 training-to-testing ratio and weights additional feature reconstruction loss at 0.5 times the human-marker reconstruction loss.Training uses batch size 64 for 260k iterations.Source: §C.5 · pp. 16–17

C.6. Interaction Prediction

• Interaction prediction replaces InterDiff’s SMPL representation with marker positions and adjusts the loss accordingly.Inputs and outputs include marker positions, object angles, and translations, with object geometry as a diffusion condition; marker MSE receives ten times the SMPL MSE weight.Source: §C.6 · p. 17

D. License

• The shared data are licensed CC BY-NC-SA, and users must also review the original licenses for each sub-dataset.The listed source licenses include noncommercial scientific-research terms for several datasets.Source: D. License · p. 17

E. Discussion

• The dataset covers 217 objects but not the full diversity of in-the-wild categories, so broader unseen-object generalization requires expansion.The authors report generalization to out-of-distribution objects within the dataset’s scope, while noting broader unseen-object robustness remains future work.Source: E. Discussion · p. 18
• Severe noise in original data can exceed the correction method’s capacity, and large human-object distances may be classified as no contact and remain uncorrected.A more aggressive contact threshold can resolve specific floating-object issues, but the authors retain unified hyperparameters across data for simplicity.Source: E. Discussion · p. 18