InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models

Jiun Tian Hoe; Xudong Jiang; Chee Seng Chan; Yap-Peng Tan; Weipeng Hu

Source-linked AI summary

InteractDiffusion: Interaction Control in Text-to-Image Diffusion Models

Jiun Tian Hoe, Xudong Jiang, Chee Seng Chan, Yap-Peng Tan, Weipeng Hu

arXiv:2312.05849v2Dec 10, 2023cs.CVcs.GRcs.MM

TL;DR

Existing T2I diffusion models lack precise control over interactions between objects, despite supporting other forms of image conditioning. InteractDiffusion adds a pluggable interaction-control module that tokenizes HOI triplets and bounding boxes, models their relationships, and injects them into visual tokens. It improves interaction controllability and outperforms existing methods while maintaining image-generation quality, though finer details remain limited by object-focused pretrained models.

Problem
Existing T2I diffusion models struggle to control interactions between objects, even though they support controls such as localization, posture, and contours.
Method
InteractDiffusion is a pluggable module that tokenizes HOI triplets and bounding boxes, embeds interaction relationships, and injects interaction conditioning into existing T2I diffusion models.
Results
InteractDiffusion significantly outperforms existing methods in interaction controllability while maintaining high-quality image generation across the reported metrics.
Takeaways & Limitations
The framework enables interaction and location control in existing T2I diffusion models using interaction information as an additional condition.
Takeaways & Limitations
Generated interactions still differ from realistic interactions in finer details, and object-focused pretrained models hinder interaction controllability.

Abstract

from arXiv · show

Large-scale text-to-image (T2I) diffusion models have showcased incredible capabilities in generating coherent images based on textual descriptions, enabling vast applications in content generation. While recent advancements have introduced control over factors such as object localization, posture, and image contours, a crucial gap remains in our ability to control the interactions between objects in the generated content. Well-controlling interactions in generated images could yield meaningful applications, such as creating realistic scenes with interacting characters. In this work, we study the problems of conditioning T2I diffusion models with Human-Object Interaction (HOI) information, consisting of a triplet label (person, action, object) and corresponding bounding boxes. We propose a pluggable interaction control model, called InteractDiffusion that extends existing pre-trained T2I diffusion models to enable them being better conditioned on interactions. Specifically, we tokenize the HOI information and learn their relationships via interaction embeddings. A conditioning self-attention layer is trained to map HOI tokens to visual tokens, thereby conditioning the visual tokens better in existing T2I diffusion models. Our model attains the ability to control the interaction and location on existing T2I diffusion models, which outperforms existing baselines by a large margin in HOI detection score, as well as fidelity in FID and KID. Project page: https://jiuntian.github.io/interactdiffusion.

1. Introduction

Existing T2I diffusion models control content and layout but struggle to represent and generate interactions between objects. InteractDiffusion addresses this gap with a pluggable interaction-conditioning framework that tokenizes HOI information, models relationships, and integrates the resulting control into existing models.

Motivation: Interaction control remains difficult because existing diffusion models can specify object locations but not the relationships between interacting objects.The paper frames interaction-conditioned image generation as a problem involving representation, intricate relationships, and integration into existing models.
Approach: InteractDiffusion treats each interaction as a human-object triplet and converts its localization and category information into three role-specific tokens.The tokens represent the subject, action, and object, and are grouped to encode their roles within an interaction.
Approach: An interaction embedding framework captures relationships among tokens from multiple interaction instances and their different roles.This design targets the challenge of representing intricate relationships across interacting entities.
Approach: A new Interaction Self-Attention layer is inserted between the existing self-attention and cross-attention layers to inject interaction information while preserving the original model.The added layer maps interaction conditioning into the visual-token processing of the pretrained T2I model.
Results: InteractDiffusion significantly outperforms baseline methods in HOI detection scores while maintaining generation quality with slight improvements in FID and KID.The framework is presented as a pluggable extension for existing T2I diffusion models.

2. Related Work

Prior HOI work primarily detects interactions or synthesizes images using pose, references, or layout proposals, while InteractDiffusion directly controls interactions in existing T2I diffusion models. Its framework uses a pluggable module composed of tokenization, interaction embedding, and interaction self-attention.

Human-Object Interactions: HOI research has mainly focused on detecting human-object pairs using bounding boxes and triplet categories, while HOI image synthesis remains relatively underexplored.Detection methods still face data scarcity for rare interactions.
Human-Object Interactions: InteractGAN generates HOI images from human pose and reference images, whereas a related layout-proposal method generates object-placement proposals from HOI triplets.These approaches require pose or reference inputs, or focus on layout rather than direct interaction control.
InteractDiffusion: The proposed framework uses an Interaction Tokenizer, Interaction Embedding, and Interaction Self-Attention to incorporate interaction information into visual tokens.The module is designed as a pluggable extension to an existing T2I diffusion model.
Human-Object Interactions: InteractDiffusion controls interactions in existing T2I diffusion models end to end using bounding boxes and interaction relations without human pose or reference images.The paper distinguishes this capability from methods that provide only object placement or require additional visual inputs.

3. Method

InteractDiffusion adds a pluggable interaction module to existing text-to-image diffusion models, representing each human-object interaction as structured tokens and injecting them into visual processing. The method combines interaction tokenization, relationship embeddings, and gated interaction self-attention while preserving the pretrained model’s original attention layers.

3.5. Interaction-conditional Diffusion Model: The resulting InToken, InBedding, and InFormer components form a pluggable module that incorporates interaction conditions into existing pretrained T2I diffusion models.The design targets minimal-cost integration while preserving knowledge from large-scale pretrained models.
3.1. Preliminary: InteractDiffusion represents each interaction as a triplet of subject, action, and object labels with corresponding subject, action, and object bounding boxes.The action bounding box is obtained from the spatial relationship between the subject and object boxes through a “between” operation.
3.2. Interaction Tokenizer (InToken): The interaction tokenizer encodes labels with CLIP and bounding boxes with Fourier embeddings, then uses separate MLPs to produce subject, object, and action tokens.Subject and object tokens are fused with ObjectMLP, while ActionMLP separately processes the semantically distinct action token.
3.3. Interaction Embedding (InBedding): Learnable instance and role embeddings group tokens by interaction and distinguish subject, action, and object roles across multiple interaction instances.The instance embedding is shared within a triplet, while role embeddings identify the subject, action, or object position; the paper reports improved generation especially for multiple interactions.
3.4. Interaction Transformer (InFormer): The interaction transformer adds a gated Interaction Self-Attention layer between the visual self-attention and caption cross-attention layers.It attends over concatenated visual and interaction tokens, slices out the visual-token outputs, and uses a zero-initialized scale to gradually control interaction conditioning.

4. Experiments

Experiments evaluate interaction controllability and image quality on HICO-DET using HOI detection, FID, KID, and qualitative comparisons. InteractDiffusion achieves the strongest reported benchmark performance while preserving generation quality and improving interaction rendering.

Qualitative results: The method renders interaction relationships and multiple interacting pairs more accurately than existing methods, which often mismatch actions or interaction details.Qualitative comparisons report improved rendering of detailed interactions and correct rendering of all pairs in multi-interaction scenes.
Ablation studies: InToken translates interaction conditions into tokens and InBedding models complex interaction relationships, providing the key components behind the observed control improvements.The interaction conditions include bounding boxes, object labels, action labels, and relationships.
Quantitative results: InteractDiffusion achieves the best result among existing baselines on FID, KID, and HOI Detection Score.Table 1 compares generated image quality and interaction controllability; the reported comparison identifies the proposed method as best overall.
Quantitative results: InteractDiffusion produces slightly higher image-generation quality than baselines, indicating that added interaction-control parameters do not reduce FID and KID performance.GLIGEN* is fine-tuned on HICO-DET, while the proposed method remains marginally better in image-generation quality.
Quantitative results: Interaction control improves substantially over methods using only localization, because the proposed method encodes interaction information together with object locations and sizes.StableDiffusion lacks object localization, while GLIGEN and GLIGEN* model location and size without the full interaction condition.
Quantitative results: The gap between generated and real-image detection scores widens with a larger detector, leaving room to improve fine interaction details.With FGAHOI and a Swin-Tiny backbone, generated interactions are nearly indistinguishable from real interactions to the detector.
Ablation studies: InToken raises detection score from 25.23 to 28.73, while adding InBedding raises it further to 29.53.These ablation results isolate the contributions of tokenizing interaction conditions and encoding their complex relationships.

5. Conclusion

The paper presents InteractDiffusion as a pluggable module for controlling interactions in generated images beyond existing text, image, and layout conditions. Quantitative and qualitative evaluations show effectiveness and significant gains over state-of-the-art approaches.

5. Conclusion: InteractDiffusion addresses the challenge of controlling interactions in generated images through a pluggable module integrated into existing T2I models.The proposed module extends conditioning beyond existing text, image, and layout controls.
5. Conclusion: Quantitative and qualitative evaluations demonstrate effective interaction control and significant improvement over state-of-the-art approaches.The conclusion summarizes the reported effectiveness without specifying a single benchmark value.

Supplementary Material

Figure 10 reports HOI detection scores for different ω values using FGAHOI with a Swin-Tiny backbone.

Supplementary Material: HOI detection score is evaluated across different ω values with FGAHOI using a Swin-Tiny backbone.The supplied figure passage identifies the metric, varying parameter, and detector configuration but does not report plotted values.

6. Implementation Details.

Implementation uses a Stable Diffusion-based architecture with an added Interaction Module, while model complexity remains close to GLIGEN. InteractDiffusion has about 210 million trainable parameters, only 1 million more than GLIGEN.

6. Implementation Details.: InteractDiffusion has about 210 million trainable parameters, only 1 million more than GLIGEN, while adding interaction controllability.The parameter comparison is reported in Table 3.
6. Implementation Details.: All methods use Stable Diffusion v1.4 as the base model, with the U-Net transformer block adapted to include the Interaction Module.The remaining network architecture is maintained.

7. Additional Ablation Studies

Additional studies examine scheduled sampling, transfer to personalized Stable Diffusion models, zero-shot interaction combinations, and qualitative controllability.

7.1. Scheduled Sampling: ω = 1.0 produces the best HOI detection scores and the lowest FID and KID distances, while ω = 0.8 balances caption and interaction adherence.Interaction controllability improves with ω and converges around ω = 0.6; at ω = 0.0, the model reduces to Stable Diffusion.
7.2. Model Transferability: The interaction module maintains personalized Stable Diffusion models’ stylistic attributes while improving interaction controllability.Evaluations cover CuteYu, ToonYou, Lyriel, DarkSushiMix, RealisticVision, and ChilloutMix.

8. More Qualitative Results

Qualitative results show that InteractDiffusion varies objects while holding the action fixed and varies actions while holding the object fixed, preserving coherent interactions.

The model generates different objects for the same action and different actions for the same object while maintaining coherent and natural interactions.

9. Limitations

The generated interactions still differ from realistic images, particularly in fine details, and object-focused pretraining limits interaction understanding.

Generated interactions remain different from realistic interactions, especially in finer details that larger HOI detectors can assess.
Object-focused pretraining in CLIP and Stable Diffusion limits interaction understanding and hinders interaction control.The authors expect more diverse pretraining that includes both objects and interactions to improve controllability.