PaperlayerView exact revision on arXiv

Source-linked AI summary

OneHOI: Unifying Human-Object Interaction Generation and Editing

Jiun Tian Hoe, Weipeng Hu, Xudong Jiang, Yap-Peng Tan, Chee Seng Chan

arXiv:2604.14062v1cs.CVcs.MM

TL;DR

Existing HOI generation and editing methods are disjoint and have limited support for mixed conditions, pose-contact decoupling, and multiple interactions. OneHOI unifies them through structured interaction-conditioned diffusion with an R-DiT backbone, and reports state-of-the-art results across generation and editing benchmarks. The method supports diverse controls and multi-HOI editing, although no competing baseline performs layout-guided multi-HOI editing.

  • Problem

    Existing HOI generation and editing methods are disjoint, with generation struggling on mixed conditions and arbitrary layouts while editing struggles with pose-contact decoupling, multiple interactions, and explicit structure.

  • Method

    OneHOI uses a unified conditional denoising framework with structured interaction representations, implemented through R-DiT, joint modality-dropout training, and the HOI-Edit-44K dataset.

  • Results

    State-of-the-art performance is reported across controllable HOI generation, layout-free editing, and layout-guided single- and multi-HOI editing benchmarks.

  • Takeaways & Limitations

    OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control while enabling multi-HOI editing within one pipeline.

  • Takeaways & Limitations

    No other baseline performs layout-guided multi-HOI editing, so the paper reports only OneHOI for that task.

Abstract

from arXiv · show

Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.

1. Introduction

Existing HOI generation and editing methods address complementary controls but remain disjoint and lack explicit relational modelling. OneHOI unifies both tasks through structured interaction conditioning, relational attention, and joint training.

  • HOI generation synthesises layout-conditioned scenes but struggles with mixed HOI and object-only inputs or arbitrary-shape layouts.
  • HOI editing struggles to decouple pose from physical contact, scale beyond one interaction, and provide fine spatial control.
  • DiTs provide high-fidelity generation and spatial control but treat scenes as independent objects without explicit interaction modelling.
  • OneHOI reframes generation and editing as a single conditional denoising process driven by structured interaction representations.
  • R-DiT combines role- and instance-aware HOI encoding, Structured HOI Attention, and HOI RoPE to model interaction topology and separate multi-HOI instances.
  • Joint training on HOI-Edit-44K and established HOI and object-level datasets supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control.

2. Related Works

Related work develops controllable image generation and HOI-specific synthesis, while OneHOI’s design addresses relational structure within a DiT-based framework.

  • Fine-grained control and spatial-conditioning methods place individual entities but do not specify how those entities relate.
  • OneHOI’s pipeline unifies HOI editing and generation on a DiT backbone using interaction-aware encoding, topology attention, and instance-separating positional indices.
  • DiTs replaced convolutional U-Nets with pure transformers, offering improved scaling properties and becoming a standard backbone for high-quality image generation.

3. Methodology

OneHOI unifies HOI generation and editing through a diffusion-transformer pipeline conditioned on structured interaction tokens and layouts. Its R-DiT backbone grounds actions spatially, enforces verb-mediated topology, preserves role and instance identity, and separates multiple interactions.

  • Unified Pipeline: The unified pipeline accepts a global prompt and structured interaction triplets, producing images that realise all specified targets for generation or editing.Generation denoises sampled latent noise, while editing concatenates source-image latents with noise and conditions the same denoiser on new interaction targets.
  • Action Grounding: Action Grounding adds semantic action tokens and spatial action regions to complement subject and object layout grounding.The spatial action region uses the union of subject and object regions rather than the prior between operator, supporting overlapping and disjoint pairs.
  • HOI Encoder: The HOI Encoder injects role, instance, and box signals into HOI tokens to reduce role confusion and interaction blending in multi-HOI scenes.These signals are combined through a gated residual before entering the DiT backbone.
  • Structured HOI Attention: Structured HOI Attention blocks direct subject-object links and cross-instance links, forcing relational information to flow through action tokens.With layouts, subject, object, and action tokens attend to their corresponding spatial regions; without layouts, these grounding constraints are removed.
  • HOI RoPE: HOI RoPE assigns each interaction instance a distinct positional slot, reducing inter-instance interference when multiple HOIs are processed together.All HOI tokens in an instance share an index distinct from the image grid and other instances.

4. Experiments

Experiments evaluate OneHOI across layout-free and layout-guided HOI editing, HOI generation, qualitative control settings, and component ablations. Results show strong editing and generation performance, flexible mixed-condition control, and complementary contributions from its relational modules.

  • Experimental Setup: Joint training uses modality dropout with the HOI-Edit-44K dataset and established HOI and object-level datasets.The implementation also adapts the Flux.1 Kontext MM-DiT backbone and trains with LoRA for 10K steps.
  • Quantitative Results: Layout-free editing achieves Editability–Identity 0.638 and HOI Editability 0.596, improving over the strongest priors by +10.0% and +16.0%.It also attains the best HPS, ImageReward, and PickScore among the compared methods.
  • Quantitative Results: Layout-guided editing reaches Spatial 0.822 and HOI Editability 0.570 for single-HOI edits, while multi-HOI edits retain Spatial 0.675 and maintained quality scores.The study evaluates both single- and multi-HOI editing under layout guidance.
  • Quantitative Results: HOI generation surpasses the strongest prior on PickScore 21.41 (+0.7%), HPS 0.2617 (+4.8%), and ImageReward 0.5524 (+33.2%).OneHOI also slightly surpasses [13] on Spatial and HOI accuracy.
  • Qualitative Results: Qualitative results show identity-preserving interaction edits, semantically faithful multi-HOI scenes, arbitrary-shape masks, and mixed HOI/object-only control.Layout-aware HOI attention and modality-dropout training support flexible combinations of layouts, masks, and object-only entities.
  • Ablation Studies: Ablations show progressive gains from Action Grounding, HOI Encoder, Structured HOI Attention, and HOI RoPE, with attention improving correctness and RoPE separating instances.The multi-action example evolves from a single pet action to correctly depicting both holding and petting after all components are added.
  • Ablation Studies: The unified model outperforms task-specific models through a reported synergy effect between generative priors and editing robustness.The paper states that generative priors enhance editing robustness and vice versa.

5. Conclusion

OneHOI unifies HOI generation and editing in a single DiT-based framework by explicitly modelling interaction structure. It supports flexible control and multi-HOI editing while achieving state-of-the-art controllability and perceptual quality.

  • OneHOI unifies HOI generation and editing through explicit interaction-structure modelling in a single DiT-based framework.
  • The HOI Encoder, Structured HOI Attention, and HOI RoPE provide role and instance identity, layout-constrained relational topology, and instance separation.
  • OneHOI bridges layout-guided generation and layout-free editing, supports flexible control, and enables multi-HOI editing.
  • OneHOI achieves state-of-the-art controllability and perceptual quality across editing and generation benchmarks.

A. Implementation Details

The implementation adapts large DiT backbones with parameter-efficient LoRA fine-tuning and adds a trained HOI Encoder. It budgets and batches role-sequence tokens while using fixed inference settings and established baselines.

  • The model adapts Flux.1 Kontext, Eligen, and Flux.1 Dev backbones while freezing text-encoder weights and applying rank-64 LoRA fine-tuning.The adaptation uses 0.3 billion trainable parameters, or 2.5% of the frozen 12B base model; the HOI Encoder is trained from scratch.
  • Inference uses 28 sampling steps with classifier-free guidance scale 3.5, while Nano Banana is evaluated using one trial per prompt without exposed seed control.
  • HOI interactions yield subject, object, and action role-token sequences, while object-only entities contribute only object tokens.
  • Each active role sequence receives a common length L selected under the token budget, then is padded or truncated for batching.

B. Dataset Details

The paper constructs paired HOI-editing data and benchmarks to support unified training and evaluate layout-free and multi-HOI editing. Dataset curation combines synthetic and real sources with statistical layout proposals and manual filtering.

  • For IEBench layout-free edits, the authors build a 5-dimensional Gaussian geometry bank over HOI classes to model object geometry relative to subjects.
  • Target layouts use object-size heuristics: large or stable objects keep their box while subject geometry is sampled, whereas small or movable objects use the converse strategy.
  • MultiHOIEdit evaluates a new multi-HOI editing task using synthetic two- or three-HOI images and spans 54 object categories, 40 source actions, and 74 target actions.The benchmark covers transitions across 112 source HOI-object pairs and 252 target HOI-object pairs, and its proposals were manually inspected for implausible layouts.
  • HOI-Edit-44K contains 44,117 paired editing examples with source images, target triplets, edited images, and corresponding HOI layouts.
  • HOI-Edit-44K covers 79 object categories, 92 target actions, and 372 unique HOI triplets for joint-model supervision.
  • The dataset combines Flux.1 generations with real HICO-DET photos and retains pairs using HOI correctness and identity-consistency criteria.The curation applies PViC and DINOv2 identity consistency of at least 0.75, rejecting approximately 90% of candidates.

C. Evaluation

The evaluation measures whether edits achieve target interactions while preserving subject and object identity. Its metrics combine detector-based editability with identity consistency, including a harmonic-mean score for their trade-off.

  • The evaluation protocol is designed to quantify the trade-off between intended interaction-transformation correctness and identity preservation.
  • HOI Editability measures the mean rate at which edited images contain the target interaction detected by PViC.Each sample receives one when the target interaction is detected and zero otherwise.
  • Editability-Identity Score combines HOI Editability and Identity Consistency through their harmonic mean.The formulation penalizes low performance in either dimension, analogous to an F1 score.
  • Identity Consistency assesses preservation of subject and object identities using GroundingDINO, SAM, and DINOv2 feature cosine similarity.Source and edited subject and object embeddings are compared and aggregated over images and seeds.

C.2. Human Evaluation Study

The human evaluation used blind randomized side-by-side comparisons to assess OneHOI against baselines on HOI editing quality. Results favored OneHOI in physical plausibility and overall quality against QwenImageEdit.

  • Study Design: 26 respondents completed 450 blind, randomized side-by-side trials comparing anonymized model outputs from source images and edit instructions.Each trial used a 5-point Likert scale with an Equal option.
  • Evaluation Criteria: The study evaluated HOI Correctness, Physical Plausibility, Identity Preservation, and Overall Quality.
  • Results: 58.2% of QwenImageEdit comparisons preferred OneHOI for HOI Physics Plausibility, versus 8.2% favoring the baseline.
  • Results: 72.0% of QwenImageEdit comparisons were OneHOI wins or ties for Overall Quality, comprising a 50.4% outright win rate and 21.6% tie rate.

D. Additional Qualitative Results

Additional qualitative results illustrate OneHOI’s handling of specific interaction edits, spatial action grounding, and unified multi-step generation-editing workflows. The examples emphasize improved action correctness and interaction-focused attention.

  • HOI Editing: Figure 23 compares layout-free edits including jump-to-sit, kick-to-hold, and hold-to-jump interactions across multiple baselines and OneHOI.
  • Unified Workflow: Figure 14 depicts a workflow combining mixed-condition generation, multi-HOI editing, and single-HOI editing within one model.
  • Action Grounding: For throwing a frisbee, action-token attention focuses on the thrower and frisbee, and the Union region better matches this footprint than the Between band.

E. Ablation on Unification vs. Task-specific

The ablation compares unified and task-specific training while accompanying figures document the dataset and qualitative evaluation settings. Under matched computation, unified training improves both generation accuracy and layout-free editing ability.

  • Unification Ablation: 26.4% higher HOI Accuracy in generation and 21.1% higher HOI Editability in layout-free editing were achieved by the unified model over task-specific models.The comparison used matched computation of 1k training steps.
  • Unification Ablation: Unified training supports mixed conditioning, including text-only, partial layouts, and multi-HOI inputs, whereas task-specific training produces brittle task-specific priors.
  • Qualitative Comparisons: Qualitative figures compare object-level and HOI-level methods for generation and editing, including examples from HOI generation and MultiHOIEdit.
Loading 2604.14062v1…