PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamFuse: Adaptive Image Fusion with Diffusion Transformer

Junjia Huang, Pengxiang Yan, Jiyang Liu, Jie Wu, Zhao Wang, Yitong Wang, Liang Lin, Guanbin Li

arXiv:2504.08291v1cs.CV

TL;DR

Most fusion methods directly place foregrounds, while interactive scenarios require foregrounds to adapt to background context. DreamFuse builds diverse fusion data and uses a DiT framework with shared attention, positional affine, and localized preference optimization. The paper reports stronger performance than existing approaches across multiple benchmarks.

  • Problem

    Many fusion methods focus on direct placement, while practical scenes also involve occlusion and interactions such as holding or wearing, and suitable datasets for these cases are lacking.

    Source: §1 · p. 2

  • Method

    An iterative human-in-the-loop pipeline creates diverse fused data, and DreamFuse uses a DiT with shared attention, positional affine, and localized direct preference optimization.

    Source: §1 · p. 2

  • Results

    DreamFuse outperforms existing approaches on multiple benchmarks.

    Source: §4.5 · p. 9

  • Takeaways & Limitations

    DreamFuse responds to text prompts for attribute modifications in fused scenes, including turning a car gold.

    Source: §1 · p. 2

    Foreground identity consistency remains insufficient in demanding cases such as text or faces, which may require an IP adapter or post-processing.

    Source: §B.4 · p. 14

Abstract

from arXiv · show

Image fusion seeks to seamlessly integrate foreground objects with background scenes, producing realistic and harmonious fused images. Unlike existing methods that directly insert objects into the background, adaptive and interactive fusion remains a challenging yet appealing task. It requires the foreground to adjust or interact with the background context, enabling more coherent integration. To address this, we propose an iterative human-in-the-loop data generation pipeline, which leverages limited initial data with diverse textual prompts to generate fusion datasets across various scenarios and interactions, including placement, holding, wearing, and style transfer. Building on this, we introduce DreamFuse, a novel approach based on the Diffusion Transformer (DiT) model, to generate consistent and harmonious fused images with both foreground and background information. DreamFuse employs a Positional Affine mechanism to inject the size and position of the foreground into the background, enabling effective foreground-background interaction through shared attention. Furthermore, we apply Localized Direct Preference Optimization guided by human feedback to refine DreamFuse, enhancing background consistency and foreground harmony. DreamFuse achieves harmonious fusion while generalizing to text-driven attribute editing of the fused results. Experimental results demonstrate that our method outperforms state-of-the-art approaches across multiple metrics.

1. Introduction

Existing fusion methods largely place foreground objects directly, while practical scenes often require occlusion or interaction. DreamFuse pairs an iterative data-generation pipeline with a DiT framework to support more adaptive fusion.

1. Introduction

  • Most existing methods focus on direct foreground placement, leaving interactive cases such as partial occlusion, holding, and wearing less addressed.These scenarios require the foreground to adjust or interact with the background context.Source: §1 · p. 2
  • The iterative human-in-the-loop pipeline generates fused foregrounds, backgrounds, and images, then curates samples to address shortages in suitable fusion data.It uses diverse prompts and scenarios including placement, handheld interactions, wearable items, logo printing, and style transfer.Source: §1 · p. 2
  • The resulting dataset contains 80,000 high-quality, multi-scene, multi-scale fused image samples.The pipeline filters low-quality samples such as mismatched foregrounds or degraded images.Source: §1 · p. 2
  • DreamFuse combines shared attention, positional affine, and localized preference optimization to integrate foregrounds more naturally while balancing background consistency and foreground harmony.The framework also supports text-prompted attribute modifications in fused scenes, such as turning a car gold.Source: §1 · p. 2
  • The paper reports that DreamFuse outperforms state-of-the-art methods across benchmarks and remains effective in real-world and out-of-distribution scenarios.Text prompts can also modify attributes in fused scenes.Source: §1 · p. 2

2. Related Work

Prior work addresses customized generation, visual fusion, and preference learning through distinct approaches. DreamFuse extends these lines toward interactive fusion scenarios and region-specific preference optimization.

2. Related Work

  • Customized-image methods incorporate reference concepts into prompts, encode reference images as visual prompts, or concatenate reference and target images.DreamFuse fine-tunes diffusion models to generate customized foregrounds, backgrounds, and fused images from corresponding prompts.Source: §2 · p. 2
  • Figure 2 presents the data-generation design alongside position matching and data format, with prompts and style LoRAs used to diversify fused data.Source: Figure 2 · p. 3
  • Fusion approaches adjust lighting, shadows, colors, perspective, pose, or style, but are often restricted to object placement.DreamFuse also targets handheld, wearable, and style-transformation scenarios.Source: §2 · p. 3
  • Human-feedback methods use reward models or comparison data, while DreamFuse applies localized direct preference optimization to background consistency and foreground harmony.Source: §2 · p. 3

3. Methodology

DreamFuse builds fusion data through iterative human-in-the-loop generation, then trains a DiT framework that conditions on foreground and background images. Positional affine mapping and localized preference optimization address placement, background consistency, and foreground harmony.

3.1. Iterative Human-in-the-Loop Data Generation

  • The iterative data pipeline starts with 86 curated foreground–fused-image pairs, manually removes foregrounds and their effects from backgrounds, and generates descriptions with GPT-4o.The initial data is used to train a Flux-based generation model.Source: §3.1 · p. 3
  • Two data-generation models—with and without positional offsets—balance diverse fused scenes against multi-resolution performance.Offset training improves scene diversity but performs poorly across resolutions; without offsets, results are overly consistent but adapt well to multi-scale data.Source: §3.1 · p. 4
  • The pipeline expands prompts and styles iteratively, manually curating generated samples and using RoMA and SAM2 to match and segment foregrounds for position and size estimates.Prompts cover varied objects, indoor and outdoor scenes, placement and interaction scenarios, and style transfer; style LoRAs further diversify the data.Source: §3.1 · p. 4

3.2. Adaptive Image Fusion Framework

  • The framework treats the foreground and background as conditions and the fused image as the denoising target, with joint attention integrating their information.The DiT is trained with a Flow Matching objective.Source: §3.2 · pp. 4–5
  • Positional affine maps the foreground’s position and size into the background’s target region without scaling or compressing the foreground.The alternatives either compress foreground information through resizing or rely on a tokenizer to encode positional masks.Source: §3.2 · p. 5, Figure 4 · p. 5
  • Localized Direct Preference Optimization uses human-preference pairs and a dilated foreground bounding-box region to improve background consistency and foreground harmony.The method localizes preference optimization because some parts of rejected images, such as copy-paste backgrounds, may still align with human preferences.Source: §3.2 · pp. 5–6

4. Experiments

DreamFuse is evaluated on its curated dataset and TF-ICON, with comparisons, user-study results, and ablations examining its design choices. The experiments also report the scale and composition of the generated dataset.

4.1. Dataset Analysis

  • The pipeline generates approximately 400k samples before filtering to 80k high-quality fused images; over half have outdoor backgrounds, and about 23k show handheld scenarios.Filtering includes GPT-4o screening and gradient comparison.Source: §4.1 · p. 6

4.2. Implementation Details

  • DreamFuse training uses two stages, followed by evaluation on 500 unseen DreamFuse samples and 332 TF-ICON samples using CLIP, AES, ImageReward, and VisionReward.The first stage trains on the 80k dataset; the second uses 15k manually selected samples for LDPO training.Source: §4.2 · pp. 6–7

4.3. Comparisons with Existing Methods

  • DreamFuse improves the TF-ICON VR score by 0.9 over the second-best method and surpasses the second-best method by 1.4 on the DreamFuse test set.The TF-ICON result is reported in Table 1, and the DreamFuse test-set result in Table 2.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7
  • The user study favors DreamFuse across consistency, harmony, and overall quality, with 64.6% overall; qualitative examples include adaptive mirror reflections.The study randomly selected 200 samples and included 17 participants.Source: §4.3 · p. 7, Figure 7 · p. 7, Figure 6 · p. 7

4.4. Ablation Study

  • Positional affine achieves the highest first-stage CLIP and VR scores, 35.215 and 5.278, respectively, among the three tested positional strategies.The comparison is on the DreamFuse test set after first-stage training.Source: §4.4 · p. 7, Table 3 · p. 7
  • LDPO raises IR scores by approximately 0.11 over first-stage results; copy-paste negatives help capture perspective and affine adjustments while supporting background consistency.The paper reports comparable performance when additionally incorporating Lnoise loss.Source: §4.4 · p. 8, Figure 8 · p. 8, Table 4 · p. 8
  • DreamFuse uses α = 1.5 for LDPO training, where α sets the local harmonious region around the foreground, and responds to prompts changing foreground attributes or the fusion scene without additional training.Source: §4.4 · p. 8

4.5. Conclusion

  • The conclusion describes an 80k-sample dataset spanning placement, handheld, wearable, and style-transfer scenarios, and reports outperformance on multiple benchmarks.DreamFuse combines positional affine transformations, shared attention, and localized preference optimization.Source: §4.5 · pp. 8–9

A. Details about the Data Generation

The data-generation process uses text prompts to describe foregrounds, backgrounds, and fused images, then feeds those descriptions to a generation model.

A. Details about the Data Generation

  • Foreground prompt subjects include animals, plants, humans, pets, logos, and products, while background prompts are drawn from website images and GPT-4o descriptions.The prompts are intended to cover varied real-world scenarios.Source: §A.1 · p. 11
  • GPT-4o classifies randomly sampled foreground and background examples into foreground, background, and fused-image text descriptions.These descriptions are then passed to the data-generation model.Source: §A.1 · p. 11
  • The prompt-generation process separates foreground and background content before producing descriptions for fused images.The resulting descriptions are used to generate fused data.Source: §A.1 · p. 11

A.2. Training Details about the Data Generation Model

The supplementary data-generation details describe model training, offset choices, and the use of LoRAs to diversify generated fusion data.

A.2. Training Details about the Data Generation Model

  • The data-generation model uses Flux-Dev, trains for 10k iterations on 8 A100 GPUs, and compares offset ∆ = (0, w) with ∆ = (0, 0).Input images are randomly scaled to 512, 768, or 1024 resolution.Source: §A.2 · p. 11

A.3. Effectiveness of the Offset ∆

  • ∆ = (0, w) generalizes better to scenarios absent from the initial small dataset, while ∆ = (0, 0) improves background alignment and is used for multi-scale fused images.The offset condition ∆ = (0, w) can produce background misalignment, especially for multi-scale images.Source: §A.3 · pp. 11–12

A.4. Effectiveness of the Existing LoRA.

  • AntiBlur, Realism, and Asian Ethnicity LoRAs diversify generated data, and the generative LoRA also works with FLUX-based models such as PixelWave.The tested base models include Flux-DEV and PixelWave.Source: §A.4 · p. 12

A.5. Data Filtering

The data-filtering procedure screens generated fusion samples for mismatched objects, foreground remnants, image-quality issues, and offset artifacts.

A.5. Data Filtering

  • Table 5 reports the number of fused images across scenarios.Source: Table 5 · p. 12
  • GPT-4o filters samples with mismatched foreground objects, foreground remnants in the background, or significant quality or aesthetic issues.Figure 15 illustrates the three GPT-4o filtering categories.Source: §A.5 · p. 12, Figure 15 · p. 12
  • Offset-affected samples are filtered using Dice scores between background and fused-image gradient maps within the outer 100-pixel boundary.A low score indicates edge mismatch associated with an offset artifact.Source: §A.5 · p. 12

A.6. Data Analysis

The dataset contains 84k high-quality fused samples spanning varied scenarios and image resolutions. The generation model also generalizes to producing multi-foreground fusion data.

A.6. Data Analysis

  • The dataset contains 84k high-quality fused samples, classified by scenario, indoor or outdoor setting, and scene complexity.Table 5 breaks down sample counts by scenario and classifies settings as indoor or outdoor and scenes as simple or complex.Source: §A.6 · p. 12
  • Image resolutions range from 600 to 1400 pixels rather than being restricted to a fixed resolution.Figure 17 visualizes the resolution distribution.Source: §A.6 · p. 12

A.7. Multi-Foreground Generation

  • The trained data-generation model can generate multi-foreground scenes from prompts specifying two foregrounds, supporting data production for occlusion or nesting relationships.The authors plan to explore multi-foreground fusion data further.Source: §A.7 · p. 13, Figure 16 · p. 13

A.8. Data Visualization

The dataset spans varied foreground objects and fusion scenarios, while the evaluation and optimization details describe a multi-dimensional quality score and human-preference-guided refinement.

A.8. Data Visualization

  • The dataset includes products, people, animals, plants, vehicles, and natural objects, with examples from multiple fusion scenarios.The passage names style transfer among the scenario examples.Source: §A.8 · p. 13
  • Figure 17 visualizes the distribution of image resolutions in the dataset.The caption does not specify the plotted axes or values.Source: Figure 17 · p. 13

B.1. Details about the Vision Reward (VR) Score in Evaluation

  • The Vision Reward score uses a vision-language model to assess images through eight questions, scoring satisfactory answers +1 and unsatisfactory answers -1.The questions cover coordination, clarity, detail, realism, recognizability, emotional response, emptiness, and harmlessness.Source: §B.1 · p. 13

B.2. The Pseudo-code for LDPO.

  • LDPO optimizes DreamFuse at each denoising step using human preferences and copy-pasted data as negative samples to improve background consistency and foreground harmony.The method directly optimizes DreamFuse based on human preferences.Source: §B.2 · p. 13

B.3. Performance of DreamFuse in Real-World Scenarios

On the real-image FOSCom dataset, DreamFuse outperforms the second-best method in VR score, and qualitative results show realistic effects such as reflections and shadows.

B.3. Performance of DreamFuse in Real-World Scenarios

  • On FOSCom, qualitative results show foregrounds integrated harmoniously into backgrounds, with realistic reflections and shadows.Figure 18 presents the qualitative results.Source: §B.3 · p. 14
  • 1.76 VR score points separate DreamFuse from the second-best method on FOSCom.FOSCom contains real images; Table 6 reports the quantitative evaluation.Source: Table 6 · p. 14, §B.3 · p. 14

B.4. Limitations

DreamFuse has difficulty preserving foreground identity in cases requiring strong consistency, particularly text and faces.

B.4. Limitations

  • Foreground identity may remain insufficiently consistent, with text and faces not fully aligned to the original foreground.The authors say such strong-consistency cases require an IP adapter or post-processing strategies.Source: §B.4 · p. 14
  • Figure 18 presents qualitative comparisons on FOSCom, without a stated outcome in the supplied caption.The dataset visualizations also compare background-to-fused-image gradients with direct copy-paste examples across foregrounds and scenarios.Source: Figure 18 · p. 15, Figure 19 · p. 16, Figure 20 · p. 17
Loading 2504.08291v1…