IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance

1. Introduction

• Earlier finetuning-based alignment methods rely on high-quality preference data that is limited and difficult to scale.Editing-based methods target local regions but may compromise overall image quality, and their detectors can miss critical misalignments.Source: §1 · p. 2
• IMG uses an MLLM to detect prompt–image misalignments, then an Implicit Aligner to refine diffusion conditioning features for re-generation.The aligner is trained with an Iteratively Updated Preference Objective combining direct preference optimization and self-play finetuning.Source: §1 · p. 2
• Unlike editing-based alignment, IMG manipulates diffusion conditioning features and re-generates images without additional editing operations.The paper describes this native-generation approach as maintaining pipeline simplicity and high-quality outputs.Source: Figure 2 · p. 2, §1 · p. 2
• Evaluations across SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing alignment methods.The framework is also described as a plug-and-play adapter for base and finetuned diffusion models.Source: §1 · p. 2

2. Related Work

• Preference-based finetuning and later reinforcement or preference learning use human-labeled data, while LLM-based approaches integrate language models into diffusion systems.Examples include DDPO reward optimization, Diffusion-DPO, layout-grounded generation, and LLM-driven editing.Source: §2 · p. 3
• Existing LLM-driven editing methods can automatically correct misalignment, whereas many MLLM editing and customization studies execute human-provided instructions.The cited discussion distinguishes automatic correction from instruction-following editing.Source: §2 · p. 3
• Diffusion adapters add input conditions beyond text, including structural controls and visual encodings for image prompting and related tasks.ControlNet and T2I-Adapter provide structural conditioning; SeeCoder and IP-Adapter illustrate visual conditioning.Source: §2 · p. 3

3.1. Preliminaries

• Diffusion models use forward and reverse processes; image-prompt diffusion adds encoded image features as a conditioning input during training and inference.The reverse process transforms noise into an image, while image-prompt encoders provide image features as conditions.Source: §3.1 · p. 3

3.2. MLLM-driven Misalignment Analysis

• IMG fine-tunes an MLLM to detect misalignment, training it on image-and-prompt inputs to produce descriptions supervised by edit instructions.The training triplets use an original image, an edited prompt, and an edit instruction from instruction-based image data.Source: §3.2 · pp. 3–4

3.3. Implicit Multimodal Guidance

• In the reported examples, Instruct Pix2Pix edits incorrectly, while SLD degrades overall aesthetics and fails on one case despite adding a woman in another.The Instruct Pix2Pix instructions are generated by the finetuned MLLM.Source: Figure 3 · p. 3, §3.3 · p. 4
• IMG uses an Implicit Aligner to transform initial image features with MLLM guidance, then supplies the re-aligned features as conditions for image re-generation.The aligner is structured as cross-attention layers and takes MLLM hidden states and image-prompt-encoder features as inputs.Source: §3.3 · p. 4

3.4. Iteratively Updated Preference Objective

• The Iteratively Updated Preference Objective combines base feature fitting with DPO- and SPIN-inspired preference terms, trained using human preference datasets.The reference aligner is initialized randomly and updated by copying the current aligner after it outperforms the reference for k consecutive iterations.Source: §3.4 · p. 5

4.2. Comparison with Base Models and Finetuning-based Methods

• IMG achieves an average win rate of 84.6% over SDXL and 80.5% over SDXL-DPO; with FLUX, its average win rate is 67.6%.Approximately 70% of users preferred re-aligned images over originals.Source: Table 1 · p. 6, §4.2 · p. 6
• With SDXL and SDXL-DPO, IMG addresses varied alignment errors and improves cases that challenge both models, while requiring no additional training for SDXL-DPO.Examples include concept comprehension, aesthetic quality, object addition, and object correction.Source: Figure 5 · p. 6, §4.2 · p. 6
• IMG operates zero-shot on T2I-CompBench and achieves leading performance with both SDXL and FLUX.It requires no specialized training on compositional prompts such as CoMat.Source: Table 2 · p. 6, §4.2 · p. 6

4.3. Comparison with Editing-based Methods.

• Compared with SLD, IMG better identifies misalignments using image and prompt inputs; SLD’s detector-and-editing pipeline can miss errors or damage intended content.On HPD, SLD degrades overall image quality and alignment, whereas IMG improves alignment through re-generation.Source: Figure 6 · p. 7, §4.3 · p. 7, Table 3 · p. 7

4.4. Discussion

• Evaluations show consistent alignment improvements across different MLLMs, and repeated IMG rounds further refine details that remain misaligned after the first round.A second round refines examples including a daikon radish’s dog-like shape and a diving action.Source: §4.4 · p. 7, Table 4 · p. 7
• When integrated with FLUX, IMG enhances details in dense-prompt images and helps FLUX stand out among the compared community models.The comparisons include SD3.5, PixArt-σ, and Playground v2.5.Source: §4.4 · pp. 7–8, Figure 8 · p. 8

4.5. Ablation Studies

• Finetuned MLLM features produce the best alignment scores; text embeddings yield only minor gains, while original MLLM features provide significant gains across two base models.The text embeddings lack information about specific misalignments.Source: §4.5 · p. 8
• A loss ratio λ = 1 is a balanced trade-off, and adding the DPO and SPIN components of Lpref progressively improves alignment.The basic Lbase objective is already competitive with SDXL-DPO when paired with SDXL + IMG.Source: §4.5 · p. 8, Table 5 · p. 8

• IMG improves prompt-image alignment without requiring additional finetuning data or explicit editing operations.Source: §5 · p. 9
• An MLLM identifies potential misalignments, and an Implicit Aligner refines diffusion conditioning features to enable re-generation.Source: §5 · p. 9
• The Iteratively Updated Preference Objective trains the aligner, and evaluations on SDXL, SDXL-DPO, and FLUX show IMG outperforming existing alignment methods.Source: §5 · p. 9
• IMG acts as a plug-and-play adapter that can enhance prior finetuning-based alignment methods.Source: §5 · p. 9

• Experiments use SDXL and FLUX, compare against SDXL-DPO and SLD, and also assess compositional-generation methods ELLA and CoMat.Source: §A.1 · p. 12
• At inference, IMG executes the forward pass three times to enhance performance.Source: §A.1 · p. 12
• The Implicit Aligner combines MLLM and image features in four cross-attention layers and two linear layers.The figure’s color cubes represent token dimensions, not token counts.Source: Figure 9 · p. 12, §A.1 · p. 12

A.2. Datasets and Benchmarks.

• The Implicit Aligner projects MLLM features to image-feature dimensions, applies cross-attention with initial image features, then linearly maps the outputs to aligned image features.Source: Figure 10 · p. 12
• Evaluation spans Pick Score, HPS v2, Parti-Prompts, T2I-CompBench, and user studies, using Pick-a-Pic preference pairs to train the Implicit Aligner.The Pick-a-Pic set contains 851K pairs. Evaluation includes 500 prompts for Pick Score, 3,400 HPD v2 prompts across five categories, 1,632 Parti-Prompts, 1,800 T2I-CompBench prompts, and A-B tests with 33 evaluators.Source: §A.2 · p. 12

A.3. MLLM Finetuning.

• LLaVA 1.5-13b is fine-tuned for 1 epoch on Instruct-Pix2Pix triplets of original images, edited prompts, and edit instructions, using the first two as inputs and supervising outputs with instructions to enhance prompt-image misalignment comprehension.Source: §A.3 · p. 12, Figure 11 · p. 13
• The original MLLM mainly describes image generation from the prompt, while the fine-tuned model emphasizes matching the image to the prompt.Source: §A.3 · p. 13

A.4. IMG Training and Evaluation.

• The Implicit Aligner is trained on the Pick-a-Pic training set [36] for 100K iterations using 8 A100 GPUs and batch size 8, with AdamW, constant learning rate 1×10−4, and weight decay 1×10−4.The ratio parameter in Eq. 7 is set to 1.Source: §A.4 · p. 13
• Inference adds about 4% time and requires 15–25 GB of GPU memory; classifier-free guidance and sampling steps differ across the evaluated models.SDXL and SDXL-DPO use guidance 7.5 and 50 sampling steps; FLUX uses guidance 3.5 and 30 steps.Source: §A.4 · p. 13

B.1. Basic Objective

• The basic objective trains fθ to predict preferred outputs from conditions using mean squared error, which corresponds to a Gaussian conditional model.The passage states that the MSE is the negative log-likelihood and that training implicitly approximates pdata(xw|c).Source: §B.1 · p. 13

B.2. Preference Objective

• The preference objective draws on DPO and SPIN, optimizing reward differences involving preferred, non-preferred, and reference outputs while regularizing against a reference distribution.Source: §B.2 · pp. 13–14
• A logistic loss is adopted because a linear loss can make the objective unbounded, whereas the logistic loss helps prevent excessive reward growth and supports stable training.Source: §B.2 · p. 14
• The reference aligner starts randomly initialized and is updated by copying fθ when its prediction is closer to the preferred output for k consecutive iterations.The reference model is a copy of fθ from an earlier iteration.Source: §B.2 · pp. 13–14

B. Objective Derivation

• The final training objective combines Lbase and Lpref with ratio parameter λ.The objective is introduced as the combination of a basic objective and a preference objective.Source: B. Objective Derivation · p. 13, §B.2 · p. 14

C. Additional Quantitative Results

• IMG shows consistent improvements across GenEval and DPGBench.Source: C. Additional Quantitative Results · p. 14, Table 6 · p. 14

D. Additional Qualitative Results

• IMG shows better alignment performance and visual quality than leading MLLM-based image-editing methods.The comparison is shown in Figure 13.Source: D. Additional Qualitative Results · p. 14, Figure 13 · p. 14
• Additional qualitative results show superior prompt adherence and aesthetic quality when IMG is integrated with FLUX.These results are presented in Figure 14.Source: D. Additional Qualitative Results · p. 14, Figure 14 · p. 15
• Additional qualitative results also show IMG integrated with SDXL and SDXL-DPO.These results are presented in Figure 15.Source: Figure 15 · p. 16