PaperlayerView exact revision on arXiv

Source-linked AI summary

Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference

Agata Żywot, Iason Skylitsis, Thijmen Nijdam, Zoe Tzifa-Kratira, Derck Prinzhorn, Konrad Szewczyk, Aritra Bhowmik

arXiv:2605.25191v1cs.CV

TL;DR

Text-to-image diffusion models lack a way to incorporate visual references at inference without retraining, while existing alternatives can be costly or semantically misaligned. VCF aligns image features with the text space and fuses both modalities, producing outputs that retain prompt content while resembling reference style, composition, and colour; its scope is limited by uncontrolled feature influence and incomplete baseline comparisons.

  • Problem

    Existing visual-guidance methods require fine-tuning or can risk semantic misalignment, leaving inference-time dual conditioning without concept-specific training unavailable.

  • Method

    VCF aligns image tokens to Stable Diffusion’s text embedding space, fuses image and text tokens, and optionally refines conditioning and noise through PNO.

  • Results

    VCF transfers reference style, composition, and colour palette while capturing textual prompt contents, with stronger reference correspondence and lowest LPIPS among the compared fusion methods.

  • Takeaways & Limitations

    VCF is particularly effective for abstract or vague prompts, improving generated-image quality, detail, and semantic coherence through reference conditioning.

  • Takeaways & Limitations

    VCF cannot control which reference features are incorporated, and the study lacks direct benchmarking against existing reference-guided baselines such as SDEdit.

Abstract

from arXiv · show

Text-to-image diffusion models like Stable Diffusion generate high-quality images from text, but lack a way to inject visual guidance (e.g. sketches, styles) at inference without retraining. Existing methods either require computationally expensive fine-tuning or rely on style transfer techniques that risk semantic misalignment with textual prompts. We introduce Visual Concept Fusion (VCF), the first method offering dual conditioning on both an image and text prompt at inference time without any concept-specific training. VCF enables visual concept injection into Stable Diffusion by aligning CLIP image features with the text embedding space. VCF consists of three components: (1) a lightweight aligner that maps image tokens to the text embedding manifold using InfoNCE and cross-attention reconstruction losses, (2) a fusion strategy that preserves both textual and visual semantics, and (3) an optional Prompt-Noise Optimization (PNO) module for test-time refinement. Our experiments demonstrate that VCF successfully transfers visual attributes including style, composition, and color palette from reference images while maintaining prompt adherence. Quantitative results show a trade-off between text alignment (CLIP score) and visual correspondence (LPIPS), with VCF outperforming baselines in reference fidelity.

1. Introduction

VCF addresses the difficulty of injecting visual references into text-to-image diffusion models at inference without fine-tuning, while preserving textual content. It aligns and fuses image and text features, with optional test-time optimization, and transfers reference visual attributes in generated images.

  • Motivation: Existing visual-guidance methods often require expensive fine-tuning, additional datasets, or risk semantic misalignment with the text prompt.Jointly conditioned models may also integrate reference cues inconsistently.
  • Contribution: VCF enables simultaneous conditioning on an image and text prompt at inference without concept-specific training or generative-model fine-tuning.The approach addresses the mismatch between textual and image feature distributions.
  • Method: VCF combines modality alignment, text–image fusion, and optional Prompt–Noise Optimisation for test-time refinement.The aligner uses limited image–caption data without training the diffusion model.
  • Results: VCF transfers reference style, composition, or colour palette while capturing the contents of textual prompts.The method also evaluates how components such as the aligner and PNO affect faithfulness and sample quality.

2. Related Work

Prior work spans generative modeling, training-free guidance, fine-tuned adapters, and image-prompt adapters, but existing methods impose training, scalability, or conditioning limitations. VCF is presented as a feature-alignment approach offering dual image–text conditioning without concept-specific training.

  • Diffusion Models: Diffusion and latent diffusion models improve image generation efficiency, with Stable Diffusion using CLIP text embeddings through cross-attention.Latent diffusion performs denoising in a lower-dimensional latent space.
  • Fine-tuning and Adapter-based Conditioning: Fine-tuning methods customize concepts or styles but require separate training, substantial computation, or risk overfitting and limited scalability.Examples include DreamBooth, CustomDiffusion, and StyleDrop.
  • Fine-tuning and Adapter-based Conditioning: Trainable conditioning modules such as T2I-Adapter and ControlNet provide visual steering but require large paired image–condition datasets.These methods inject additional conditioning into a frozen diffusion backbone.
  • Training-free Guidance: Training-free methods such as SDEdit and SkipInject steer generation at inference but have limitations in visual-reference injection, scheduling, or text-guided applicability.SkipInject primarily operates on a single image and requires careful timestep scheduling.
  • Limitations of Existing Approaches and Contribution: Prior methods do not simultaneously provide image-and-text conditioning at inference without concept-specific training.VCF instead aligns feature maps rather than learning discrete tokens, preserving the original text prompt while enabling visual guidance.

3. Method

VCF injects image guidance into Stable Diffusion at inference by aligning image tokens with the text embedding space, fusing both modalities, and optionally refining conditioning and noise. Its aligner uses global and local alignment objectives, while concatenation fusion preserves modality-specific semantics.

  • Pipeline overview: VCF uses an Image Aligner, Text–Image Fusion block, and optional Prompt–Noise Optimisation module to integrate reference images into text-conditioned diffusion.The aligner maps image tokens into the text embedding space; fusion combines aligned image and text features; PNO refines conditioning and initial noise at inference.
  • Image-to-text alignment: Stable Diffusion v2 conditions denoising on pre-projection CLIP text tokens, so directly injecting pre-projection image tokens creates a modality mismatch.The mismatch is attributed to the image and text token distributions remaining unaligned before their projection layers.
  • Image-to-text alignment: VCF trains a lightweight two-layer aligner while keeping the underlying Stable Diffusion model frozen, mapping image tokens into the text-token dimensionality.The aligner uses LayerNorm and ReLU activations and produces aligned image tokens ˆI in R^m×dtext.
  • Image-to-text alignment: The alignment objective combines a global InfoNCE loss with a local cross-attention reconstruction loss to match text-token distributions and preserve token-level structure.The combined loss is Lalign = λ LInfoNCE + Lattn., with λ set to 0.2.
  • Text–image fusion: Naive fusion averages aligned image tokens and adds the same signal to every text token, often suppressing linguistic nuances and producing noisy, semantically inconsistent outputs.The image influence is controlled by α, but uniform perturbation is retained only as a baseline.
  • Text–image fusion: VCF concatenation fusion appends aligned image tokens to the text sequence, preserving individual modality semantics and empirically providing the best balance between prompt fidelity and reference adherence.Cross-attention fusion reduces some naive-fusion artifacts but does not match concatenation fusion in the reported experiments.

4. Results

VCF is evaluated for joint reference-image and text-guided generation using Stable Diffusion v2, with CLIP measuring prompt alignment and LPIPS measuring reference correspondence. Qualitative and quantitative comparisons show stronger visual-reference integration, alongside a trade-off with strict text alignment.

  • Experimental Setup: The evaluation uses Stable Diffusion v2 with DDIM sampling at 768 × 768 resolution, while the aligner is trained on approximately 60,000 COCO image–caption pairs.The training set is a 10% COCO Captions subset with an 80/10/10 train-validation-test split.
  • Qualitative Results: Naive fusion often resembles the text-only baseline and fails to reliably integrate the reference, producing noisier and less realistic outputs.The comparison holds the prompt fixed as “A photo of a cat” while varying the reference image.
  • Qualitative Results: VCF transfers high- and low-level reference features, including artistic style, background objects, colour distribution, shading, and depth, while retaining the prompted content.Reference realism also influences output realism: photorealistic references yield realistic generations, while paintings or prints produce matching stylistic attributes.
  • Evaluation Metrics: CLIP score measures alignment with the text prompt, whereas LPIPS measures perceptual similarity between generated and reference images, with lower LPIPS indicating greater similarity.CLIP uses cosine similarity in CLIP space; LPIPS uses fixed unit weights across pretrained VGG layers.
  • Quantitative Results: VCF achieves the lowest LPIPS score, indicating the greatest perceptual similarity to reference images, while its CLIP score is reduced relative to text-only SDv2.The results expose a trade-off in which increased visual guidance can override strict prompt literalism.

5. Ablations

Ablations show that the aligner’s InfoNCE and cross-attention reconstruction losses contribute different forms of control. Their combination provides a better balance between reference similarity and prompt fidelity than either loss alone.

  • Effect of the Aligner Loss Function: The aligner ablation compares InfoNCE-only, cross-attention-only, and combined training under the same “A photo of a cat” prompt.The combined configuration uses λInfoNCE = 0.2.
  • Effect of the Aligner Loss Function: InfoNCE-only training preserves image quality comparable to SDv2 but produces little or no visual resemblance to the reference.The result suggests that global distribution alignment alone is insufficient for guiding Stable Diffusion’s cross-attention mechanism.
  • Effect of the Aligner Loss Function: Cross-attention-only training closely follows the reference, sometimes sacrificing prompt fidelity by generating the referenced object instead of the prompted one.A dog reference can produce a dog despite a cat prompt, and a floral girl reference yields a girl surrounded by flowers.
  • Effect of the Aligner Loss Function: Combining both losses produces a more desirable balance between prompt semantics and salient reference features.InfoNCE regularises the embedding space globally, while cross-attention reconstruction injects local structure and fine-grained visual cues.

6. Discussion

VCF is reported to be especially useful for ambiguous prompts, where reference conditioning improves the quality, detail, and semantic coherence of generated images. The authors also identify limits in feature control, ablation breadth, and comparative evaluation.

  • Discussion: VCF produces particularly significant improvements for abstract or vague text prompts, where Stable Diffusion otherwise struggles to generate coherent and meaningful content.The discussion highlights ambiguous prompts such as “A charming character emerging from the scene.”
  • Discussion: Reference conditioning transforms imprecise textual descriptions into images reflecting both the prompt and reference attributes such as style, shape, and texture.The reported benefit is framed as valuable for creative workflows when users have a visual concept that is difficult to articulate precisely.
  • Limitations: The method has no mechanism to control which visual features are incorporated, potentially causing unpredictable or overly dominant reference influence.This is identified as the first limitation in the authors’ limitations discussion.
  • Limitations: Aligner ablations are limited to three loss configurations on COCO with one randomly sampled caption per image, and VCF is not directly benchmarked against reference-guided baselines such as SDEdit.The authors attribute the missing baseline comparison to time constraints.

A. Prompt-Noise Optimisation (PNO) Details

Prompt–Noise Optimisation is an optional test-time refinement that jointly adjusts conditioning tokens and initial noise to increase similarity between generated and reference images. It uses CLIP similarity and noise regularisation before deterministic DDIM sampling.

  • PNO Procedure: PNO refines the final conditioning tokens Tfinal and initial diffusion noise xT before reverse sampling begins.It is optional and operates at test time rather than retraining the diffusion model.
  • PNO Objective: PNO jointly optimises Tfinal and xT to maximise CLIP similarity between the generated image x0 and visual reference xguide.The objective is adapted from prior PNO work by replacing toxicity minimisation with visual-reference similarity maximisation.
  • PNO Objective: A regularisation loss on xT prevents degenerate solutions and maintains a plausible initial-noise structure, with λreg set to 0.1 by default.The optimisation uses a small number of gradient steps, typically 10–50, before full DDIM sampling.
  • PNO Assumption: Under deterministic DDIM sampling, optimising the initial noise xT is sufficient because fixed conditioning and model parameters uniquely determine the full denoising trajectory and final image.This explains why PNO need not optimise the entire noise trajectory in the presented implementation.

B. Ablation: Effect of PNO

PNO is evaluated as a test-time refinement module for text-only and cross-attention-fused generations. The qualitative results indicate improved reference alignment, reduced fusion artefacts, and stronger preservation of reference-specific details.

  • PNO setup: PNO jointly refines the conditioning signal and initial noise at test time to enhance alignment with the reference image.The evaluation fixes the number of PNO steps at 50 for the cross-attention setting.
  • Text-only refinement: PNO improves structural alignment and reference fidelity even without explicit fusion.This effect is shown for the text-only Stable Diffusion v2 setting.
  • Fusion refinement: With cross-attention fusion, PNO suppresses visual noise and artefacts introduced during fusion.The qualitative examples specifically show noise reduction in the top row.
  • Reference details: PNO can steer outputs toward reference-specific details, including stronger colour fidelity such as amplified orange cat stripes.The bottom-row example illustrates improved adherence to a characteristic colour pattern.

C. Additional Qualitative Examples of Main Results

Additional examples compare text-only Stable Diffusion, naive fusion, and VCF across portrait, character, and pizza prompts. Reference conditioning is especially valuable for vague or abstract prompts, where VCF improves coherence and semantic alignment.

  • Additional qualitative examples: For the abstract character prompt, text-only Stable Diffusion struggles with coherent and meaningful content, whereas reference conditioning improves quality, detail, and coherence.The examples use the prompt “A charming character emerging from the scene”.
  • Additional qualitative examples: Naive fusion remains poor on the abstract character example, underscoring the difficulty of integrating visual and textual modalities.The comparison is shown alongside text-only generation and VCF.
  • Additional qualitative examples: The additional examples cover portrait, character, and pizza prompts, comparing reference images with text-only, naive-fusion, and VCF outputs.Figures 7–9 provide the qualitative comparisons for these prompts.

D. Cross-Attention Fusion

Cross-attention fusion lets text tokens query aligned image tokens to inject fine-grained visual cues, with α controlling image-signal strength. Qualitatively, it transfers reference features but is noisier and less coherent than concatenation fusion.

  • Fusion mechanism: Cross-attention fusion lets text tokens query aligned image tokens to inject fine-grained visual cues into the conditioning stream.The resulting fused representation is blended with the original text tokens.
  • Fusion mechanism: The parameter α ∈[0, 1] controls the overall weight of the image signal, while γ rescales fused tokens to match the text-token norm.The rescaling is applied at every denoising step.
  • Qualitative comparison: Cross-attention fusion transfers some reference features but produces noisier, less coherent images than concatenation fusion.The comparison is based on qualitative outputs in Figure 10.
  • Qualitative comparison: Cross-attention fusion can introduce artefacts absent from both the prompt and reference, so the paper retains concatenation fusion for its main results.The cross-attention variant is reported for completeness.
Loading 2605.25191v1…