PaperlayerView exact revision on arXiv

Source-linked AI summary

COSMI: COmpositional Synthesis of Multi-object Interactions

Daniel Eskandar, Ilya A. Petrov, Gerard Pons-Moll

arXiv:2610.03252v1cs.CV

TL;DR

Most human-object interaction data captures one object at a time, limiting the scale of multi-object training data. COSMI composes single-object clips into a large dataset and trains a variable-object diffusion transformer; on held-out combinations, COSMI outperforms baselines in text alignment and contact accuracy, especially for an unseen object.

  • Problem

    Most captured human-object interaction datasets record one object at a time, while capturing multiple objects is combinatorially expensive.

    Source: §1 · p. 2

  • Method

    COSMI composes contact-consistent single-object clips with semantic and geometric checks, then generates a variable number of objects using shared slots in a text-conditioned diffusion transformer.

    Source: §1 · p. 2, §4 · pp. 6–7

  • Results

    COSMI outperforms HIMO and multi-object adaptations of MDM and PriorMDM in text alignment and contact accuracy, with the largest margin on the unseen object.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The composed dataset contains 222k sequences and 275 hours with up to five objects, and grows with added clips rather than recording time.

    Source: §1 · p. 2

    The composition covers sustained single interactions at once or in sequence, while text does not specify the person's path or the timing of sequential actions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative models of human-object interaction are bounded by the data that exists: everyday activities involve several objects, but most captured datasets record one at a time, as multi-object capture is combinatorially expensive. Our observation is that interactions are local, so single-object captures already contain the parts of multi-object activities. We compose them: contact-consistent clips of single interactions, mirrored to balance the hands, transfer between bodies, and a language model and geometric checks admit only the pairings that are plausible, semantically and physically. Therefore, the dataset grows combinatorially with the clips rather than recording time. The COSMI dataset holds 222k sequences and 275 hours with up to five objects, nearly thirty times the largest multi-object capture, and can be extended by adding datasets or even hand-object recordings. On this data we train the COSMI method, a text-to-interaction diffusion transformer that follows how the data is built: weight-shared object slots generate a variable number of objects, predicted relative to the body parts that move them. On a benchmark with an unseen object and unseen interaction combinations, models trained on the dataset generalize to the unseen combinations. COSMI outperforms baselines in text alignment and contact accuracy, where its margin is largest on the unseen object. Code, models, and the dataset pipeline will be released on the project page: https://ptrvilya.github.io/cosmi.

1 INTRODUCTION

COSMI addresses the lack of large-scale multi-object interaction data and generators that handle variable object counts. It composes single-object captures into a large dataset and trains a text-conditioned model that generalizes to held-out combinations.

1 INTRODUCTION

  • Multi-object activities are common, but datasets mostly capture one person with one object and multi-object capture is costly, leaving no multi-object data at single-object scale.Existing generators also tend to support one object or fixed object counts and may leave implicit which body part moves each object.Source: §1 · p. 2
  • 222k sequences and 275 hours of composed interactions make COSMI nearly thirty times the size of the largest captured multi-object dataset.The dataset supports up to five objects and can be extended by adding datasets or hand-object recordings.Source: §1 · p. 2
  • Contact-consistent single-object clips are transferred between bodies, with language-model and geometric checks filtering for semantic plausibility, contact, and lack of penetration.Mirroring balances left- and right-sided interactions.Source: §1 · p. 2
  • COSMI outperforms HIMO and multi-object adaptations of MDM and PriorMDM in text alignment and contact accuracy, with its largest margin on the unseen object.On a benchmark holding out an unseen object and five unseen combinations, COSMI and baselines trained on COSMI data generalize to the unseen combinations.Source: §1 · p. 2
  • COSMI uses a text-conditioned diffusion transformer with shared object slots to generate a variable number of objects relative to the body joints that move them.The method includes part assignment, auxiliary losses, and contact-consistent guidance.Source: §1 · p. 2

2 RELATED WORK

Prior work covers single-object capture, body-part composition, and text-driven interaction generation, while multi-object datasets and generators remain limited. COSMI composes object interactions from single-object captures and uses semantic and physical checks to select combinations.

Human-object interaction datasets

  • Most full-body interaction datasets capture one object, while recent multi-object datasets remain small; hand-object datasets offer more recordings and can supply source clips after fitting a body.COSMI composes compatible clips into sequences that can themselves serve as targets, so sequence count grows combinatorially with clips rather than linearly with recording time.Source: §2 · p. 3

Compositionality in motion and interaction modeling

  • Motion work composes clips in time or across body parts, while COSMI extends body-part composition to object interactions with cross-body transfer, semantic gating, and physical checks.SINC uses GPT-3 part labels but leaves semantic compatibility to future work.Source: §2 · p. 3

Text-driven interaction synthesis

  • Multi-object generators use separate models, fixed or zero-padded states, or multiple stages; COSMI instead uses one text-conditioned model for one to five objects in a single stage.COSMI grounds objects to the body parts that move them.Source: §2 · p. 3

3 THE COSMI DATASET

COSMI builds multi-object sequences by recombining contact-consistent single-object clips, using semantic annotations and physical checks to filter combinations. The dataset also supports held-out tests of generalization to unseen objects and interaction combinations.

Contact-based extraction

  • Single-interaction clips require sustained object contact, are extracted with 13 body-part and pose rules, and total 1,491 clips.Contact is defined as proximity below 5 cm to the object surface; contiguous windows must last at least 3 seconds.Source: §3 · p. 4

Relative poses and mirroring

  • Objects are expressed relative to the joints that drive their interactions, allowing their poses to transfer with the limbs to mirrored or composed bodies.Mirroring fits a proper rotation and translation to the original mesh so contact is preserved despite the body's asymmetry.Source: §3 · pp. 4–5

Mirroring

  • Mirroring one-handed and one-footed clips swaps their interaction rules and raises the clip count to 2,663, balancing captured right-hand bias.The mesh is not simply reflected: a proper rotation and translation preserve contact with the mirrored hand.Source: §3 · pp. 4–5

Composition

  • Composition pairs compatible actions on free limbs, using language-model annotations and geometric checks to preserve contacts and reject body self-penetration or object collisions.Actions requiring simultaneous gaze or head involvement are sequenced rather than composed simultaneously.Source: §3 · pp. 4–5, Figure 2 · p. 4

Statistics and quality

  • 221,677 sequences and 275 hours cover 59 subjects, 36 objects, and up to five objects per sequence, with left/right hands balanced at 49/51.The authors report that foot sliding is unchanged, while jerk and penetration shift less than the differences among captured datasets.Source: §3 · p. 5

Extensibility and benchmark

  • The unified contact, pose, and reference-joint rules admit additional sources, including fitted hand-object recordings, and support tests on five unseen combinations and an unseen object.A pairing such as walking with an apple can be held out while walking and eating an apple remain in training.Source: §3 · p. 5

4 THE COSMI METHOD

COSMI uses a text-conditioned diffusion transformer to generate human motion and a variable set of object motions. Shared object slots and joint-relative prediction support generation across different object counts and interaction types.

Model overview

  • One text-conditioned model generates human motion and one to five object motions in a single denoising pass, including articulated and worn objects.It takes a prompt, initial human pose, and object meshes, names, and initial poses as input.Source: §4 · p. 6

Object representation

  • Object representations include absolute pose, articulation angle, candidate joint-relative poses, and soft contact indicators for 15 body regions.Joint-relative poses encode how an object moves in candidate driving-joint frames.Source: §4 · p. 6

Conditioning

  • CLIP text tokens and pooled text embeddings condition generation alongside initial poses, object names, and BPS geometry embeddings.Text tokens link object names in the prompt to their corresponding objects.Source: §4 · p. 6

Object slots

  • Object slots share input and output projections without slot-specific embeddings, making the network permutation-equivariant and independent of object count.Masks exclude empty slots from attention and per-object losses; the number of slots is set at runtime.Source: §4 · p. 7

Part assignment

  • A part-assignment head predicts which joints drive each object, selecting joint-relative decoding for driven objects and absolute-pose decoding for undriven ones.A chair supporting a seated person is undriven, unlike an object held, worn, or kicked.Source: §4 · p. 7

Objective

  • Training combines clean-sample diffusion reconstruction with losses on motion differences, contacts, body plausibility, and consistency between absolute and relative object poses.Body-plausibility penalties include foot skating and forward kinematics.Source: §4 · p. 7

Contact-consistent guidance

  • During sampling, gradient guidance penalizes predicted contact-distance and foot-speed violations, helping close finger-object gaps and stop planted feet.The guidance is applied to predicted clean samples in the final sampling steps.Source: §4 · p. 7

5 EXPERIMENTS

Experiments compare COSMI with baselines on one-to-five-object sequences, HIMO’s captured-data benchmark, and held-out objects and interaction combinations. COSMI leads on several text-alignment and contact metrics, while articulated-object motion and the unseen chair expose limitations.

Data split and protocol

  • The split holds out the black chair and five pairings of familiar parts, yielding 35k test sequences versus 187k for training; one model is evaluated across tiers with three generations per test sequence.The all-objects tier includes one to five objects, articulated and worn objects included; the two- and three-object tiers use rigid interacted objects only, with no worn objects.Source: §5 · pp. 7–8

Qualitative examples

  • HOT3D hand-object recordings become full-body source clips that can be paired with captured sitting clips; Figure 5 shows COSMI seated with plausible contacts for an unseen-chair prompt.Composed sequences also animate clothed scans registered as SMPL-X+D.Source: §5 · p. 7, Figure 4 · p. 8, Figure 5 · p. 9

Comparison with baselines

  • Two thirds of ground truth is COSMI’s hinge sweep; it leads the all-objects tier on FID, R-precision, MM-Dist, and foot sliding, with ground-truth-level penetration.COSMI also has the lowest hinge-angle error; the two- and three-object tiers largely agree, with HIMO the exception on three-object top-3 R-precision.Source: Table 2 · p. 8, Table 3 · p. 8, §5 · p. 9

Comparison on the HIMO dataset

  • On HIMO’s captured-data benchmark, COSMI has the highest top-3 R-precision in both tiers and the best FID with three objects.Source: §5 · p. 9

Ablations

  • Ablations find that auxiliary losses matter most, part assignment improves text alignment and contact accuracy, and contact-consistent guidance reduces foot sliding and hand-object gaps without harming other metrics.Source: §5 · p. 9

Generalization

  • On every unseen-combination subset COSMI is most accurate; on the black chair, baselines lose four to eight points of temporal contact accuracy while COSMI stays within one point of its whole-test value.The test holds out the black chair and five pairings whose parts are common in training but never occur together.Source: §5 · pp. 8–9, Table 4 · p. 9

6 CONCLUSIONS

COSMI composes local, contact-consistent interactions into a large multi-object dataset and generates variable-object interactions; the paper reports generalization to unseen combinations and objects. Its scope and fidelity remain bounded by sustained interactions and imperfect motion and penetration quality.

6 CONCLUSIONS

  • COSMI composes contact-consistent interactions into 222k sequences and 275 hours with up to five objects, and its method generates a variable number of objects.The conclusion also reports generalization to unseen combinations and an unseen object.Source: §6 · p. 10
  • The composition covers sustained single interactions at once or in sequence, while text prompts do not specify the person’s path or sequential-action timing.Finger motion also falls short of the precision of dedicated grasp synthesis.Source: §6 · p. 10

APPENDIX

  • The appendix reports sequence-quality checks, model ablations, HIMO comparisons, qualitative results and failures, and extensions to hand-object recordings and clothed scans.It also describes the learned evaluator, physical metrics, dataset construction, and the COSMI method.Source: APPENDIX · p. 18

A.1 QUALITY OF COMPOSED SEQUENCES

  • Foot sliding is unchanged, while jerk and penetration change less than captured datasets differ; InterCap body penetration rises from 0.17 to 0.40 cm.Table 5 compares composed sequences against their parent recordings using HUMOTO physical-plausibility metrics.Source: §A.1 · p. 19, Table 5 · p. 19

A.2 ABLATIONS

The ablations show that guidance improves contact consistency, auxiliary losses matter most among unguided variants, and model components contribute differently to text alignment and contact accuracy.

A.2 ABLATIONS

  • Guidance reduces foot sliding by a fifth and lowers predicted contacts violating their bound by more than 5 mm from 18.9% to 0.9%.Text alignment and contact accuracy stay unchanged; guidance enforces predicted contacts, at the cost of slightly more object jerk and penetration.Source: §A.2 · p. 19
  • Without auxiliary losses, foot sliding doubles and temporal contact accuracy drops by 17 points among unguided variants.The auxiliary interaction and plausibility terms matter most in the ablation comparison.Source: §A.2 · p. 19
  • Part assignment improves text alignment and contacts, while text tokens and object names improve text alignment by two and one top-1 R-precision points.Source: §A.2 · p. 19
  • Chunk size four balances single-frame object tokens, which lower temporal contact accuracy by two points, against chunks of eight, which lower top-1 R-precision by two points.Replacing the hierarchical geometry encoder with an MLP changes little, with slightly worse top-3 R-precision and multimodal distance but better FID and object jerk.Source: §A.2 · p. 19

A.3 COMPARISON ON THE HIMO DATASET

On HIMO, COSMI leads in top-3 R-precision in both object tiers, while HIMO has better two-object FID and MM-Dist; qualitative comparisons also show COSMI retaining hand-object attachment on held-out prompts.

A.3 COMPARISON ON THE HIMO DATASET

  • COSMI has the highest top-3 R-precision in both tiers: 0.730 and 0.582 versus HIMO’s 0.637 and 0.535.For three objects, COSMI also has the lowest FID and MM-Dist and diversity closest to the data; HIMO has lower two-object FID, 1.481 versus 4.896, and lower MM-Dist.Source: §A.3 · p. 19, Table 7 · p. 20

A.4 HELD-OUT SUBSETS

  • The held-out test subsets include a black chair, walking with an apple, sitting with a flip phone, a foot interaction while drinking, standing with a water bottle, and a camera with a mug.The remaining 186,559 sequences form the training set.Source: §A.4 · pp. 19–20

A.5 QUALITATIVE COMPARISON

  • On unseen-object or unseen-combination prompts, COSMI keeps objects in their driving hands, while baselines can drift or float beside the body as arms move.HIMO appears only in the three-object tier because it supports a fixed number of objects.Source: §A.5 · p. 20

A.6 FAILURE CASES

The appendix documents failure cases, hand-object recording integration, clothed-scan rendering, and the learned evaluator used for motion metrics.

A.6 FAILURE CASES

  • COSMI’s flip-phone hinge moves less than ground truth, while an unseen black chair can be misplaced relative to the body and intersect its armrest.The training seats cited lack armrests, so the model has no example of keeping arms clear of them.Source: §A.6 · pp. 20, 22, Figure 7 · p. 22

A.7 HAND-OBJECT RECORDINGS

  • HOT3D clips with a stable head and one lifted hand-contacting object are fitted to a full-body template, then processed like other source clips for composition.This lets HOT3D objects be composed onto walking, sitting, and large-object targets.Source: §A.7 · p. 22

A.8 CLOTHED SCANS

  • Clothed scans use per-vertex displacements animated with SMPL-X skinning; forearm displacements are attenuated and hand displacements removed to preserve recorded finger motion.Objects are reattached through joint-relative transforms so a held object follows the scan’s hand.Source: §A.8 · p. 22

B.1 EVALUATOR

  • The learned evaluator embeds sequences from joint positions and object features, then scores text alignment, distribution distance, and diversity; metrics are compared only under the same evaluator.Multimodality is reported only on HIMO.Source: §B.1 · pp. 22–23

B.2 PHYSICAL METRICS

The section specifies operational definitions for physical interaction metrics, including foot sliding, object jerk, penetration, contact accuracy, and articulated-object motion.

B.2 PHYSICAL METRICS

  • Foot sliding measures horizontal ankle and toe displacement below height thresholds, while object jerk is mean third temporal difference of object position.Foot thresholds are 8 cm for ankles and 4 cm for toes; the measures are reported in mm per frame and cm per frame, respectively.Source: §B.2 · p. 23
  • Penetration is the fraction of frames with a body vertex more than 2 cm inside an object, but Table 5 uses HUMOTO’s deepest-intrusion measure.Table 5 averages the deepest body or hand intrusion over frames.Source: §B.2 · p. 23
  • T-Acc evaluates per-frame contact states and B-Acc per-marker states, with contact defined as a body marker within 2 cm of the object surface.Hinge-angle error is wrapped to [−π, π], while sweep ratio compares total generated hinge motion with ground truth.Source: §B.2 · p. 23

C DATASET DETAILS

The dataset pipeline standardizes source sequences, labels interactions, and composes compatible body and object clips under contact and feasibility checks. It also specifies representations for the resulting human, object, and geometry data.

C.1 PREPROCESSING; C.3 ANNOTATION PROMPTS

  • Source datasets are resampled to 30 fps and standardized to a Y-up frame, neutral body shapes, and template-mesh object poses; annotation prompts select an interaction verb and form a parsed sentence.The prompts also determine whether each verb-object pair requires gaze or head involvement.Source: §C.1 · p. 26, §C.3 · p. 26

C.2 INTERACTION RULES

  • Thirteen interaction rules define contact groups, pose constraints, local references, and source datasets; a rule holds when contact parts are within 5 cm and its other constraints are met.For upright-torso rules, pelvis-to-head direction must fall within a rule-specific 20–30 degrees of vertical.Source: §C.2 · p. 26

C.4 JOINT-RELATIVE OBJECTS

  • Objects are represented relative to the joint that drives them, and objects referenced to multiple joints recover their global pose from each joint before averaging.Rotations are averaged as unit quaternions and renormalized.Source: §C.4 · p. 26

C.5 MIRRORING

  • Mirroring the joint-relative pose preserves contact but reflects the object mesh; COSMI instead aligns the reflected mesh to the original template and retains the original mesh.Across 26 mirrored templates, the alignment residual averages 0.9 mm and stays below 0.7% of the bounding-box diagonal.Source: §C.5 · p. 29

C.6 COMPOSITION GRAPH

  • The composition graph allows whole-body targets to accept every source, while one-hand targets accept compatible other-hand actions or holds and exclude overlapping limbs.The remaining pairs undergo semantic and physical checks.Source: §C.6 · p. 29, Figure 13 · p. 29

C.7 MERGE AND FEASIBILITY; C.8 SEQUENTIAL COMPOSITION AND LEVELS

  • Composition replaces target joints with source-action joints and keeps sequences only when rule contacts persist in at least half the frames and specified collision or self-intersection checks pass.Sequential compositions place left- and right-hand actions in turn, with a 15-frame overlap; the pipeline enumerates one-, two-, three-object, and sequential levels.Source: §C.7 · p. 29, §C.8 · pp. 29–30, Figure 14 · p. 31

D.1 HUMAN REPRESENTATION; D.2 OBJECT REPRESENTATION

  • Human frames encode body-joint positions and rotations, hand coefficients, and foot-joint positions, while object frames include absolute pose, joint-relative pose, and articulation information.An articulated object occupies one slot with its base-part pose and hinge angle.Source: §D.1 · p. 33, §D.2 · p. 33

D.3 GEOMETRY ENCODER

  • The geometry encoder pools nearest-surface offsets from 1,024 fixed basis points through eight horizontal slabs, preserving an object’s vertical layout in its geometry embedding.Each slab partitions the ball into 16 cells.Source: §D.3 · p. 33

D.4 ARCHITECTURE

The model uses a shared-slot transformer for variable object counts, a part-assignment head to select driving joints, and losses that supervise motion, contacts, and consistency.

D.4 ARCHITECTURE

  • Eight transformer blocks denoise human frames and four-frame object chunks, reducing a five-object window to 270 motion tokens rather than 720.The denoiser, part-assignment head, geometry encoder, and CLIP projection total 30.4M parameters.Source: §D.4 · p. 33

D.5 PART-ASSIGNMENT HEAD

  • The part-assignment head predicts which candidate joint drives each object, and decoding selects the highest-scoring joint once per window or falls back to absolute pose.A driving joint is labeled when it contacts the object for at least ten frames and travels at least half the joint’s distance; scores below 0.5 trigger the fallback.Source: §D.5 · pp. 33–34

D.6 LOSS TERMS

  • Training combines human and object pose errors, contact and foot-skating terms, forward-kinematics supervision, cross-stream consistency, and part-assignment classification.The consistency term compares 100 object surface points and points on moving parts against ground truth.Source: §D.6 · p. 34

D.7 TRAINING DETAILS

The section reports training and runtime settings, long-sequence window blending, and test-time contact-consistent guidance based on predicted contact scores.

D.7 TRAINING DETAILS

  • Training runs for 80k steps with batches of 256 windows, Adam at 5·10−4, and an exponential moving average for sampling.Training takes eight hours on one RTX 5090.Source: §D.7 · p. 34

D.8 RUNTIME

  • Contact-consistent guidance costs 7.2 times single-sequence generation time but 1.55 times batched test-set time.Runtime is measured on one RTX 5090; the median test sequence has 119 frames and fits one window.Source: §D.8 · p. 34, Table 9 · p. 34

D.9 LONG SEQUENCES

  • Consecutive 120-frame windows overlap by 60 frames, and their predictions are blended in the overlap with rotations projected back onto SO(3).Hinge angles are averaged as (cos, sin) pairs.Source: §D.9 · p. 34

D.10 CONTACT-CONSISTENT GUIDANCE

  • Guidance minimizes contact and sliding losses over the final ten sampling steps, using Adam edits to the predicted clean sample and temporal blurring to reduce jitter.It uses five Adam iterations per step and 100 at the final step; predicted contact scores define bounds on finger-object distance and foot displacement.Source: §D.10 · p. 35

Related papers

Loading 2610.03252v1…