Does FLUX Already Know How to Perform Physically Plausible Image Composition?

Shilin Lu; Zhuming Lian; Zihan Zhou; Shaocong Zhang; Chen Zhao; Adams Wai-Kin Kong

Source-linked AI summary

Does FLUX Already Know How to Perform Physically Plausible Image Composition?

Shilin Lu, Zhuming Lian, Zihan Zhou, Shaocong Zhang, Chen Zhao, Adams Wai-Kin Kong

arXiv:2509.21278v4Sep 25, 2025cs.CVcs.AIcs.LG

TL;DR

Image composition requires realistic insertion across difficult lighting and resolution conditions, but existing methods struggle with these settings and with pose constraints from inversion. SHINE is a training-free framework that combines latent steering, degradation suppression, and adaptive blending, and it achieves state-of-the-art results on ComplexCompo and DreamEditBench. The paper also introduces ComplexCompo for evaluating diverse resolutions and challenging scenes.

Problem
Existing image-composition models struggle with complex lighting, high-resolution inputs, and contextually appropriate object placement despite relevant priors in base text-to-image models.
Method
SHINE uses MSA loss with pretrained customization adapters, DSG during sampling, and ABB to preserve subject and background quality while removing visible seams.
Results
SHINE achieves state-of-the-art performance on both ComplexCompo and DreamEditBench across standard and human-aligned evaluation metrics.
Takeaways & Limitations
ComplexCompo enables evaluation of image composition across varied resolutions and challenging conditions including low lighting, intricate shadows, and water reflections.
Takeaways & Limitations
Incorrect inpainting colors tend to persist, and subject similarity depends on the quality of the customization adapter.

Abstract

from arXiv · show

Image composition aims to seamlessly insert a user-specified object into a new scene, but existing models struggle with complex lighting (e.g., accurate shadows, water reflections) and diverse, high-resolution inputs. Modern text-to-image diffusion models (e.g., SD3.5, FLUX) already encode essential physical and resolution priors, yet lack a framework to unleash them without resorting to latent inversion, which often locks object poses into contextually inappropriate orientations, or brittle attention surgery. We propose SHINE, a training-free framework for Seamless, High-fidelity Insertion with Neutralized Errors. SHINE introduces manifold-steered anchor loss, leveraging pretrained customization adapters (e.g., IP-Adapter) to guide latents for faithful subject representation while preserving background integrity. Degradation-suppression guidance and adaptive background blending are proposed to further eliminate low-quality outputs and visible seams. To address the lack of rigorous benchmarks, we introduce ComplexCompo, featuring diverse resolutions and challenging conditions such as low lighting, strong illumination, intricate shadows, and reflective surfaces. Experiments on ComplexCompo and DreamEditBench show state-of-the-art performance on standard metrics (e.g., DINOv2) and human-aligned scores (e.g., DreamSim, ImageReward, VisionReward). Code is available at https://github.com/ZhumingLian/SHINE.

1 INTRODUCTION

Image composition remains difficult because current models struggle with realistic lighting, varied resolutions, and contextually appropriate object placement. SHINE addresses these gaps with a training-free framework and is evaluated on challenging composition benchmarks.

Motivation: Multimodal foundation models still exhibit imprecise placement, inconsistent lighting, and subject identity drift in image composition.Figure 2 presents challenging cases involving backlighting, shadows, and water surfaces.
Motivation: Existing composition models struggle with accurate shadows, water reflections, and varied high-resolution backgrounds despite these priors being present in their base models.Fine-tuned variants are limited by fixed resolutions and synthetic-data defects inherited from inpainting models.
Motivation: Training-free prior methods often rely on inversion that locks inserted objects to reference-image poses, producing contextually inappropriate orientations.The introduction identifies inversion bottlenecks as a central limitation of prior approaches.
SHINE: SHINE introduces MSA loss, DSG, and ABB to preserve subject identity and background structure, suppress low-quality outputs, and remove mask-boundary seams.MSA uses pretrained customization adapters to steer noisy latents toward the reference subject; DSG and ABB address degradation and visible seams.
Evaluation: SHINE achieves state-of-the-art performance on ComplexCompo and DreamEditBench across standard and human-aligned metrics.The reported metrics include DINOv2, DreamSim, ImageReward, and VisionReward.

2 RELATED WORK

Prior image-composition methods decompose the task into specialized subproblems, while diffusion models have motivated more unified frameworks.

Classical and diffusion-based composition: Classical image composition separates object placement, blending, harmonization, and shadow generation into distinct subtasks often handled by separate models.The related-work discussion contrasts this decomposition with the field’s shift toward unified diffusion-model frameworks.

3 METHOD

SHINE is a model-agnostic, training-free composition framework that prepares non-inverted latents and combines MSA, DSG, and ABB to preserve subjects, suppress degradation, and reduce seams.

3 METHOD: The framework requires only standard model features—customization adapters, self-attention maps, and text-image cross-attention—without architectural changes.These assumptions support integration into existing generative-model pipelines.
3.1 NON-INVERSION LATENT PREPARATION: SHINE replaces inversion-based latent initialization with one-step forward diffusion from a VLM-captioned inpainting result, avoiding reference-pose constraints.Existing copy-paste inversion can force contextually inappropriate object orientations, while the new initialization starts from an inpainted scene before adding noise.
3.2 MANIFOLD-STEERED ANCHOR LOSS: MSA steers noisy latents toward faithful subject representations while aligning adapter-augmented predictions with the base model to preserve background structure.It uses pretrained customization adapters and an anchor prediction from the original noisy latent.
3.3 DEGRADATION-SUPPRESSION GUIDANCE: DSG constructs a negative velocity by blurring FLUX image queries, steering denoising away from low-quality regions while retaining semantic content and spatial layout.Blurring Qimg produces pronounced degradation while preserving structure, making it the selected degradation signal.
3.3 DEGRADATION-SUPPRESSION GUIDANCE: FLUX’s internal attention analysis finds that blurring Qimg is the most effective degradation lever, whereas blurring text features has negligible impact and blurring Vimg makes outputs unintelligible.Blurring Kimg moderately affects quality while images remain visually acceptable.
3.4 ADAPTIVE BACKGROUND BLENDING: ABB replaces rigid user-mask blending with an attention-derived mask during early denoising, producing smoother transitions for irregular objects and preserving surrounding scenes.The attention mask is obtained from subject-token cross-attention, then dilated and restricted to its largest connected component.

4 EXPERIMENTS

Experiments evaluate SHINE on DreamEditBench and the diverse, high-resolution ComplexCompo benchmark using identity, human-aligned, and image-quality metrics. SHINE consistently outperforms baselines, while ablations attribute gains to its three components.

Experimental Setup: The evaluation combines subject-identity metrics with human-aligned retrieval and quality metrics to capture both identity consistency and perceptual quality.The benchmark suite includes CLIP-I, DINOv2, IRF, DreamSim, ImageReward, and VisionReward.
Experimental Setup: The study compares SHINE against six training-based and five training-free composition methods.The baselines include UniCombine, AnyDoor, PBE, ObjectStitch, MADD, DreamCom, EEdit, TIGIC, DreamEdit, TF-ICON, and TALE.
Experimental Results: SHINE consistently remains the top performer on challenging ComplexCompo scenes, while most competing methods suffer notable performance drops.ComplexCompo includes non-square resolutions, low-light settings, water surfaces, and scenes with complex shadows.
Experimental Results: On DreamEditBench, both SHINE variants surpass all baselines on DreamSim, ImageReward, and VisionReward.These metrics are described as the most critical indicators of quality because they align with human preferences.
Ablation Study: Ablations show that MSA improves subject identity, DSG raises ImageReward and VisionReward, and ABB suppresses visible mask-boundary seams.The seam improvement is clearer visually than in LPIPS or SSIM, which mainly measure structural similarity.

5 CONCLUSION

The paper concludes that SHINE provides training-free, high-fidelity composition across diverse resolutions and lighting conditions, supported by the new ComplexCompo benchmark. It also identifies incorrect prompt colors and customization-adapter quality as important limitations.

5 CONCLUSION: SHINE achieves state-of-the-art results on both ComplexCompo and DreamEditBench.The framework combines Manifold-Steered Anchor Loss, Degradation-Suppression Guidance, and Adaptive Background Blending.
5 CONCLUSION: Incorrect inpainting colors can persist in the final composition even when MSA converges to the correct subject identity.The limitation is illustrated by the failure case in Figure 8(b).
5 CONCLUSION: Inserted-object similarity depends on the customization adapter, with test-time LoRA producing higher identity consistency than pretrained open-domain adapters.The authors expect improvements as open-domain customization adapters advance.

Results Our

The reported failure cases show that composition can inherit an erroneous color from the inpainting prompt. The limitation is presented as a specific vulnerability of the composition process.

Results Our: An incorrect inpainting color is inherited and preserved by the final composition.The paper identifies this as a failure case rather than a failure to recover subject identity.

ETHICS STATEMENT

The framework broadens access to image composition but raises misuse and bias concerns. The paper therefore calls for cautious and responsible use.

ETHICS STATEMENT: SHINE may be misused for harassment or misinformation, while its pretrained large-scale T2I models may introduce social and cultural biases.The authors state that these risks require careful consideration and responsible use.

REPRODUCIBILITY STATEMENT

The supplied material reviews composition methods and introduces query blurring as an efficient alternative to directly smoothing attention weights. It also describes how SHINE’s components preserve structure, suppress degradation, and blend backgrounds without architectural changes.

Related Work: The reviewed literature distinguishes training-based composition from training-free inference modification and subject-customization approaches that require either test-time tuning or zero-shot pretraining.Classical composition also separates placement, blending, harmonization, and shadow generation into distinct subtasks.
Method: SHINE combines MSA loss, DSG, and ABB to preserve subject identity and background structure while reducing low-quality outputs and visible seams.MSA uses personalization adapters, DSG steers sampling away from low-quality distributions, and ABB performs adaptive background blending.
C.1 BLURRING THE QUERY MATRIX: Query blurring is mathematically equivalent to Gaussian smoothing of self-attention weights, providing a computationally efficient proxy for attention-matrix manipulation.The equivalence follows from convolution linearity and matrix multiplication properties.
C.1 BLURRING THE QUERY MATRIX: Blurring the key or value matrices does not produce the same equivalence because convolution does not commute with their roles in attention computation.For keys, the mismatch follows from the general Toeplitz matrix property; values enter after attention weights are computed.

C.3 IMPLEMENTATION DETAILS OF GAUSSIAN BLURRING

The implementation reshapes query embeddings into spatial maps, applies Gaussian smoothing, and restores sequence form before attention computation. An empirical analysis evaluates cross-attention masks across FLUX.1-dev blocks using SAM-derived masks and IoU.

C.3 IMPLEMENTATION DETAILS OF GAUSSIAN BLURRING: The implementation chooses a kernel radius of r = 3σ and the nearest odd kernel size to 6σ, using σ = 10 in all experiments.The stated radius captures approximately 99.7% of Gaussian mass.
C.3 IMPLEMENTATION DETAILS OF GAUSSIAN BLURRING: Query matrices are reshaped into spatial tensors, blurred with a Gaussian kernel, and reshaped back before attention computation.The implementation takes Q ∈ R^B×L×D and outputs smoothed queries Q′ in sequence form.
D EVALUATING CROSS-ATTENTION MAP ACCURACY VIA IOU: The last multi-stream block yields the most accurate cross-attention segmentation masks among 57 evaluated FLUX.1-dev masks.Masks are averaged over 20 denoising steps and evaluated against SAM-segmented outputs over 100 generated images.

E EXPERIMENTS WITH SDXL, SD3.5, AND PIXART

Experiments test SHINE across SDXL, SD3.5, PixArt, and FLUX using benchmark and user-study evaluations. The supplied results indicate strong performance across architectures and favorable human rankings.

E EXPERIMENTS WITH SDXL, SD3.5, AND PIXART: SHINE uses MSA, DSG, and ABB in a model-agnostic design that integrates standard personalization, self-attention, and cross-attention features without architectural changes.The components rely on LoRA or personalization adapters, self-attention maps, and text-image cross-attention maps.
E EXPERIMENTS WITH SDXL, SD3.5, AND PIXART: SHINE’s PixArt-Σ experiments show substantial performance gains, supporting generality across diverse generative architectures.The broader evaluation covers SDXL, SD3.5, and PixArt across DreamEditBench and ComplexCompo.
F USER STUDY: In the user study, the method received the most favorable rankings from the majority of participants across composition results.Fifty participants completed ranking tasks drawn from DreamEditBench and ComplexCompo, where lower ranks indicate better results.

G BENCHMARK DETAILS

ComplexCompo evaluates composition under varied resolutions and placements using subject, background, and bounding-box triplets. The benchmark expands beyond fixed 512 × 512 settings and is paired with a user-study ranking protocol.

G BENCHMARK DETAILS: ComplexCompo contains 300 triplets of subject image, background image, and bounding box, with manually designed placements for contextual plausibility.Backgrounds come from OpenImages and include landscape and portrait resolutions such as 768 × 1088 and 1360 × 768.
G BENCHMARK DETAILS: The composition protocol uploads a subject image, background image, and mask image, then specifies the insertion region and desired identity preservation in the prompt.Resolution and bounding-box coordinates are adjusted for each case.

I SUBJECT IDENTITY METRICS ANALYSIS

The analysis finds that common subject-identity metrics can diverge from human judgments of compositional realism, while DreamSim better reflects appearance changes across difficult conditions. The broader experiments also show DSG is more adaptive and consistently outperforms tested negative prompts on SD3.5.

Metric reliability: AnyDoor can receive higher CLIP-I, DINOv2, and often IRF scores despite visibly less realistic and consistent compositions.This mismatch demonstrates that semantic-similarity metrics may not faithfully capture compositional quality.
Metric reliability: DreamSim consistently recognizes subjects across underwater, shadowed, occluded, low-light, and back-lit scenes, unlike the other evaluated metrics.The passage identifies DreamSim as the only metric exhibiting the desired human-aligned behavior across these conditions.
Evaluation scope: SHINE’s evaluation spans image-quality metrics, varied baselines, and qualitative comparisons, with additional assessments using HPSv3 and UnifiedReward variants.The supplied materials identify Tables 6 and 7 and Figures 13–18 as evaluation resources, but do not provide their unseen numerical cells.
Degradation suppression: Standard negative prompting is highly sensitive to wording, with performance depending on manually engineered prompt choices.The experiments report that one prompt set performs better than another, confirming the fragility of heuristic prompt engineering.
Degradation suppression: DSG consistently outperforms all tested standard negative prompts across every reported metric on SD3.5.DSG constructs an image-specific low-quality direction by blurring attention, avoiding manual prompt tuning.