PaperlayerView exact revision on arXiv

Source-linked AI summary

Harnessing Coupled Stream Completion For Human-Object Interaction Modeling

Dawei Guan, Di Yang, Jiangtao Wang

arXiv:2609.32551v2cs.CVcs.AI

TL;DR

Text-conditioned HOI generation must coordinate distinct body, object, and hand streams, yet shared representations can limit stream structure and independent updates cannot respond to other streams’ current states. TRACE keeps separate latent states, couples stream updates, and adds decoded geometric constraints while supporting completion and language-model-based understanding. Experiments report generation and completion performance across three datasets and understanding gains over raw-motion encoding.

  • Problem

    HOI generation must coordinate body, object, and hand streams with distinct structures while aligning contact, relative pose, and interaction timing.

    Source: §1 · p. 2

  • Method

    TRACE encodes body, hand, and object motion into separate latents, predicts each stream velocity from the complete current state, and applies decoded geometric supervision.

    Source: §1 · p. 2

  • Results

    Across three datasets, TRACE demonstrates generation performance, stream-completion capability, and consistent HOI-understanding gains over raw-motion encoding.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The reported results show structured cross-stream coupling provides a unified representation for generation, stream completion, and HOI understanding.

    Source: §5 · p. 9

Abstract

from arXiv · show

Text-conditioned human-object interaction (HOI) generation requires body motion, object trajectories & rotations, and hand articulation to remain coordinated. These components differ in scale and dynamics, but must agree on contact, relative pose, and timing. A shared representation may limit the distinct structure of each stream, while independent generation prevents each stream from responding to changes in the others. Latent supervision alone also does not directly constrain contact after decoding. We propose TRACE, a continuous latent framework that keeps stream states separate and couples their updates. TRACE encodes body, object, and hand motion into separate latents and predicts each stream velocity from the complete current interaction state. Geometric losses on decoded motion further constrain contact and object-relative motion over time. The same model supports completion of any single absent stream from the other two. Frozen flow features also serve as input to a language model for HOI understanding. Experiments on InterAct, OMOMO, and BEHAVE show that joint completion training improves generation and that frozen flow features improve understanding over raw-motion encoding. On InterAct, TRACE achieves the highest contact precision, recall, and F1 among the compared methods.

1 INTRODUCTION

HOI generation must coordinate body, object, and hand streams with distinct structures and dynamics, but shared representations and independent updates each limit coordination in different ways. TRACE separates stream states while coupling their updates, adding decoded geometric supervision and reusing the model for completion and understanding.

1 INTRODUCTION

  • HOI generation must align body, object, and hand motion in contact, relative pose, and timing, while shared representations may limit stream structure and independent dynamics cannot respond to other streams’ current states.Latent regression also leaves contact, object-frame relations, and interaction timing unconstrained after decoding.Source: §1 · p. 2
  • The three-stream decomposition defines completion of any one missing stream from the other two, and features for this task provide a semantic interface for understanding.The contribution reuses the coupled representation for completion and bridges frozen flow features to a language model.Source: §1 · pp. 2–3
  • TRACE preserves separate body, hand, and object latents while predicting each stream’s velocity from the complete evolving HOI state.Decoded clean-motion estimates receive supervision for contact, object-frame anchoring, and interaction timing.Source: §1 · p. 2
  • The paper evaluates generation, stream completion, and HOI understanding on three datasets, with ablations assessing stream coupling, decoded geometric supervision, and latent representation.The stated ablation targets include contact consistency and the effect of latent representation.Source: §1 · p. 3

2 RELATED WORK

Related work spans text-driven interaction generation, motion representations, and motion-language understanding. TRACE is positioned as coupling temporal body, object, and hand streams through continuous latent flow while reusing generation features for language understanding.

Text-driven HOI Generation

  • Text-driven HOI generation has expanded from hand-object synthesis to dynamic whole-body interaction, but maintaining spatial and temporal coherence across text-aligned body, object, and hand motion remains challenging.Recent work addresses controllability and interaction realism through conditioning, correction, guidance, relation-aware generation, and geometric-fidelity methods.Source: §2 · p. 3

Motion Representation Learning

  • Motion representations range from full-sequence and factorized forms to interaction-aware, latent, and conditional models; TRACE instead couples temporal streams through continuous latent flow.The passage contrasts this with discrete representations and asynchronous denoising approaches.Source: §2 · p. 3

Motion-Language Understanding

  • Motion-language work includes token-based, alignment-based, and interaction-centered models, while whether frozen human-object generation features retain reusable interaction semantics remains underexplored.The proposed interface uses the HOI representation for language-model fine-tuning.Source: §2 · p. 3, Figure 2 · p. 4

3 METHODOLOGY

TRACE represents body, hand, and object motion in separate, temporally aligned latents, then jointly evolves their states. Decoded geometric constraints, shared completion training, and frozen flow features extend the framework to interaction consistency, missing-stream recovery, and HOI understanding.

HOI Representation

  • TRACE uses separate encoders and decoders for body, hand, and object streams, preserving their distinct motion properties on a temporally aligned latent grid.The split VAE is frozen after training; its regularization includes object-pose reconstruction, temporal velocity consistency, body kinematic consistency, and weak KL regularization.Source: §3 · pp. 4–5

Latent Flow Modeling

  • The flow learns conditional velocities from Gaussian sources toward the three streams’ latent targets, using a shared flow time and a joint state.Each stream’s velocity is predicted from the complete current state, allowing cross-stream responses while keeping separate stream outputs and latent coordinates.Source: §3 · p. 5

Decoded Interaction Geometry

  • Decoded-motion losses add contact, object anchoring, and temporal-consistency supervision beyond latent flow matching.The contact objective uses BCE and Dice losses; anchoring and episode terms constrain where interaction occurs in object coordinates and how the relation evolves over time.Source: §3 · p. 5

Stream Completion

  • Joint training combines HOI generation with completion of any one missing stream from the other two, sharing the VAE, coupled flow, and sampling head.During completion, observed latents remain fixed while joint attention infers the missing stream; the passage also reports that completion training improves HOI generation.Source: §3 · p. 6

HOI Understanding

  • For HOI understanding, frozen coupled-flow hidden states are compressed into memory tokens and projected as a continuous prefix for a language model.The flow is read at τ = 1 under zero conditioning; the language branch uses LLaDA with LoRA adaptation and does not alter generation or completion.Source: §3 · p. 6

4 EXPERIMENTS

TRACE is evaluated on InterAct, with ablations on OMOMO and BEHAVE, across generation, stream completion, and HOI understanding. Results favor coupled stream modeling, continuous VAE latents, and decoded geometric constraints.

Evaluation Setup

  • Evaluation uses InterAct's official train-test split and reports realism, diversity, text alignment, physical plausibility, and understanding metrics with 20 repeated evaluations and 95% confidence intervals.The understanding metrics are Bleu@1/4, Rouge, Cider, and BertScore, alongside R-Precision and MM Dist.Source: §4 · p. 7

Baselines

  • Generation baselines share preprocessing, split, and evaluator, while adapted methods retain their original objectives and schedules; understanding comparisons include TRACE Raw, which replaces flow features with raw-motion encoding.The understanding baselines are retrained on InterAct, and TRACE Raw shares the language configuration and schedule to isolate the encoder comparison.Source: §4 · pp. 7–8

Quantitative Evaluation

  • TRACE achieves the strongest interaction accuracy on InterAct, with the highest contact precision, recall, and F1, and performs best on selected generation and understanding metrics.The reported understanding metrics are R@1, MM Dist, BLEU@4, and CIDEr; coupled-flow features outperform raw-motion encoding on all understanding metrics.Source: §4 · p. 8, Table 2 · p. 8

Qualitative Evaluation

  • TRACE better follows descriptions while maintaining coordinated interactions and stable contact; baselines often omit actions or show drift, penetration, or floating artifacts.Figure 3 presents the qualitative generation comparison.Source: §4 · p. 8

Stream Completion

  • TRACE completes a missing body, object, or hand stream from the other two, preserving observed motion and coordinating the generated stream with it.Body completion follows fixed hand-object trajectories; object completion recovers a rigid trajectory consistent with human motion; hand completion aligns fine-grained movements with body and object.Source: §4 · p. 8, Figure 4 · p. 8

Impact of Stream Modeling

  • Coupled stream modeling performs best in R-Precision, FID, MM Dist, FSR, Contact, and all three interaction metrics, unlike unified or independent modeling.Qualitative comparisons show object drift or hand-object misalignment in the alternatives, while coupled modeling maintains coherent, coordinated trajectories.Source: §4 · p. 8, Figure 5 · p. 9, Table 3 · p. 9

Impact of Latent Representation

  • Continuous VAE latents achieve the best R-Precision, FID, MM Dist, FSR, penetration, contact, and interaction metrics among the compared representations.Raw motion also outperforms VQ on most metrics; the paper attributes the VAE's advantage to preserving continuous stream-specific information while reducing temporal redundancy.Source: §4 · p. 9, Table 4 · p. 9

Impact of Geometric Constraints

  • Adding Lc consistently improves R@3, FID, FSR, penetration, and contact accuracy, while adding LA/E achieves the best result on every reported metric.The paper describes contact supervision as improving local alignment and object-local anchoring and temporal consistency as further stabilizing interaction geometry over time.Source: §4 · p. 9, Table 5 · p. 9

5 CONCLUSION

TRACE uses separate body, object, and hand states with coupled stream updates and decoded-space geometric constraints. Across three datasets, it demonstrates generation and completion capability and understanding gains over raw-motion encoding.

Conclusion

  • TRACE combines separate stream states, full-state-conditioned velocities, and decoded geometric constraints in one framework for generation, completion, and HOI understanding.The conclusion describes the constraints as grounding generation in contact locations, object-local relations, and interaction timing.Source: §5 · p. 9

APPENDIX

The appendix provides additional training, completion, and language-adaptation details, followed by further experimental results.

Appendix

  • The appendix adds details about training objectives, completion training, and language adaptation, then presents additional experimental results.Source: APPENDIX · p. 13

LLM Usage

  • The authors report using LLMs to polish grammar, word choice, and fluency, not to formulate the methodology or run experiments.Source: APPENDIX · p. 13

A ADDITIONAL METHOD DETAILS

This section introduces additional method details for the generation and completion model and the language-adaptation branch.

Additional Method Details

A.1 GENERATION AND COMPLETION TRAINING

TRACE trains text-to-HOI generation and single-stream completion with masked-stream flow modeling. The training also applies VAE regularizers and decoded-motion constraints on contact and object-relative geometry.

A.1 GENERATION AND COMPLETION TRAINING

  • Generation-only and joint generation-completion TRACE variants are trained for 1000 epochs under the same optimization budget.Training uses bf16 on NVIDIA A800 GPUs; inference integrates the coupled flow with 32 Euler steps.Source: §A.1 · pp. 13–14
  • The split VAE combines masked reconstruction with regularizers for object pose, temporal velocity, body kinematics, and latent distribution.The reconstruction includes complete-motion and object-stream errors; regularizers apply Smooth L1 object translation, rotation, velocity, and bone-vector supervision.Source: §A.1 · p. 13
  • Decoded interaction constraints supervise contact and human anchors’ object-frame locations and temporal evolution.The contact label uses a joint-to-object-surface distance threshold; positive contacts receive weight 3, with γ = 0.05 m and σc = 0.01 m.Source: §A.1 · p. 13
  • Text-to-HOI generation and three single-stream completion tasks share the flow: observed streams retain clean latents while the missing stream follows the generation flow.Observed streams are restored after transformer blocks and copied unchanged into the decoded output.Source: §A.1 · p. 14

A.2 HOI UNDERSTANDING AND LANGUAGE ADAPTATION

For HOI understanding, TRACE extracts motion-only states from its frozen VAE and coupled flow, then compresses them into a continuous language-model prefix. The language bridge and LoRA parameters are trained to recover interaction descriptions.

A.2 HOI UNDERSTANDING AND LANGUAGE ADAPTATION

  • Frozen VAE latents pass through the coupled flow with text and geometry conditions zeroed, so extracted states depend only on interaction motion.The flow transformer’s pre-velocity-head states encode each stream and its relations to the complete interaction through joint attention.Source: §A.2 · p. 14
  • A temporal resampler compresses duration-proportional flow tokens into 32 learned slots before projection into the language model’s embedding space.The resampler uses two blocks and eight attention heads; the projected continuous prefix is passed to the language model.Source: §A.2 · p. 14
  • The language model receives an object-category prompt and is trained with a masked diffusion objective computed only over answer tokens.The VAE, coupled flow transformer, and LLaDA backbone are frozen while the HOI language bridge and LoRA parameters are optimized; answers are fully masked with probability 0.8.Source: §A.2 · p. 15
  • The bridge is trained alone for 5 epochs before LoRA is enabled, followed by 100 epochs of joint adaptation.At inference, LLaDA converts the continuous HOI memory into a textual description through 48 denoising steps.Source: §A.2 · p. 15

B ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments find that coupled stream modeling remains effective on OMOMO and BEHAVE, while observed streams provide the main signal for completion. Qualitative understanding benefits from complete interaction evidence and coupled-flow features, and TRACE has lower inference latency than the compared methods.

B ADDITIONAL EXPERIMENTAL RESULTS

  • Across datasets, observed streams provide the main completion signal; full conditioning yields the lowest object reconstruction errors and highest contact scores for object completion.For hand and body completion, Streams-only remains comparable and sometimes gives lower paired reconstruction errors.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 17
  • Complete three-stream evidence recovers more object movement and temporal progression in qualitative descriptions, while coupled-flow features capture a floor-lamp orientation adjustment omitted by raw-motion encoding.The qualitative comparisons include plastic-box, tripod, and floor-lamp interactions.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 17
  • Coupled stream modeling achieves the strongest overall generation and interaction results on both OMOMO and BEHAVE.The OMOMO and BEHAVE ablations compare independent, unified, and coupled strategies; coupled is reported strongest overall, consistent with InterAct.Source: B ADDITIONAL EXPERIMENTAL RESULTS · pp. 17–18
  • TRACE requires 3.44 seconds per batch and is 10.2×, 12.5×, and 56.0× faster than HOI-Diff, InterDiff, and LIGHT, respectively.Timing uses an NVIDIA A800 with BF16 for batches of 64 interactions of 300 frames; runtime includes sampling and decoding but excludes data loading, rendering, and file I/O.Source: B ADDITIONAL EXPERIMENTAL RESULTS · p. 18

Related papers

Loading 2609.32551v2…