PaperlayerView exact revision on arXiv

Source-linked AI summary

Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation

Haipeng Liu, Yang Wang, Meng Wang

arXiv:2610.07684v1cs.CVcs.AI

TL;DR

Text prompts can misalign with generated backgrounds in customization and with foregrounds in style transfer. Dual-FDM disentangles dual references by substituting selected frequency bands and adapting denoising timesteps. The paper reports superiority over state-of-the-art diffusion models for personalized image generation.

  • Problem

    Image customization and style transfer both face text-prompt misalignment, including with complex prompts and multi-object scenes.

    Source: §1 · pp. 2–3

  • Method

    Dual-FDM uses frequency-band masks to substitute reference content and adapt denoising timesteps across customization style transfer and color style transfer.

    Source: §1 · pp. 3–4

  • Results

    Experiments report that Dual-FDM is superior to state-of-the-art diffusion models for personalized image generation.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Adaptive timestep modulation is intended to balance text alignment with similarity to both images.

    Source: §1 · p. 4

    For customization style transfer, extremely sparse style-reference patterns can have mid-frequency customized cues attenuate high-frequency style details, leaving style information incomplete.

    Source: I Limitations and Broader Impacts · p. 20

Abstract

from arXiv · show

Personalized image generation aims to synthesize text-driven images conditioned on reference images, while mainly casting the generation as image customization for foreground and style transfer for background. Previous arts of diffusion models suffers from the text misalignment with background for image customization and foreground for style transfer during the denoising process. Such facts, as we observed, rooted from the entanglement among hybrid frequency bands during the denoising process. To address such salient limitation, in this paper, we study personalized generation based on dual references - customization and color and style reference - and propose a paradigm to disentangle these Dual image references within Frequency-aware Diffusion Models, dubbed Dual-FDM, to simultaneously tackle two crucial personalized image generation tasks: customization style transfer and color style transfer, by disentangling different frequency bands via mask strategy within frequency domain. For customization style transfer, we replace the mid-frequency band of the background in the style reference with that from the foreground of the customized reference. For color style transfer, we substitute the low-frequency band of the background in the style reference with that from both the foreground and background of the color reference. Both the substituted frequency bands are used as the key and value to reconstruct the query foreground and background of the denoised personalized image.Extensive experiments validate the superiority of Dual-FDM over the state-of-the-art diffusion models for personalized image generation. Our code can be accessed from https://github.com/htyjers/Dual-FDM.

1 Introduction

Personalized image customization and style transfer can misalign generated content with text prompts, a problem the paper links to entangled frequency bands. Dual-FDM uses two references and frequency-band substitutions for both tasks, with adaptive denoising timesteps to balance text alignment and image similarity.

  • Customization substitutes reference foreground semantics while preserving prompt background, whereas style transfer uses reference style while aligning foreground with the prompt.Source: §1 · p. 2
  • Customization methods can misalign complex background prompts, while style-transfer methods can leak background style into foregrounds or struggle with multi-object prompts.The introduction illustrates these issues with IP-Adapter, MS-Diffusion, InstantStyle, and StyleStudio.Source: §1 · pp. 2–3
  • The paper attributes prompt misalignment to entangled frequency bands, with low- and mid-frequency bands dominating early denoising and suppressing text guidance.Low and mid frequencies capture color and global structure; high frequencies capture local texture.Source: §1 · p. 3
  • Dual-FDM uses DCT and masks to disentangle bands, replacing the style reference’s background mid-frequency band for customization or low-frequency band for color transfer.The substituted reference supplies keys and values for reconstructing the denoised image’s foreground and background in decoupled cross-attention.Source: §1 · p. 3
  • For each task, Dual-FDM uses substituted-band information entropy to adapt early- and late-stage denoising timesteps for a balance between text alignment and dual-image similarity.Source: §1 · p. 4
  • Experiments report that Dual-FDM is superior to state-of-the-art diffusion models for personalized image generation.Source: §1 · p. 4

2 Methodology

Dual-FDM separates image-reference information by frequency band: it swaps different bands for customization and color style transfer, then uses information entropy to adapt denoising stages.

2.1 Preliminary

  • Decoupled cross-attention integrates text and image-reference information into each denoising layer, where the latent feature is updated iteratively until the final image is produced.The latent feature is the query, and reference-image embeddings provide keys and values.Source: §2.1 · p. 4

2.2 Popping up the Background of Image Reference

  • SVD and inverted singular values suppress dominant foreground components while enhancing background-related information in the image-reference matrix.Larger singular values tend to correspond to foreground, while smaller values are more likely associated with background.Source: §2.2 · pp. 4–5

2.3 Customization Style Transfer: Replacing the Mid-frequency of Image Reference

  • For customization style transfer, Dual-FDM replaces the style reference background's mid-frequency band with the customized reference foreground's mid-frequency band.The mid-pass mask isolates the band, and the resulting reference features contribute to decoupled cross-attention.Source: §2.3 · pp. 5–6

2.4 Color Style Transfer: Replacing the Low-frequency of Image Reference

  • For color style transfer, Dual-FDM replaces the style reference background's low-frequency band with low-frequency information from both the color reference foreground and background.The low-pass mask selects the band used for the replacement.Source: §2.4 · p. 6

2.5 Adaptively Modulating the Early and Late Denoising Stage

  • Dual-FDMcust consistently achieves the best performance across several datasets, while entropy-based thresholds adaptively allocate denoising timesteps between early and late stages.A larger entropy ratio means more timesteps are assigned to the early stage.Source: §2.5 · pp. 6–7, Table 1 · p. 7

3 Experiments

Experiments compare Dual-FDM with diffusion-based personalization methods and test how its modules, denoising schedule, and frequency masks affect generated images.

3.1 Implementation Details

  • Across experiments, Dual-FDM is evaluated on customization, style transfer, and text-to-image settings using nine alignment and image-quality metrics.Experiments use DreamBench, TI benchmark, DreamBench++, StyleBench, IS-Plus, and T2I-CompBench++ prompts, with IP-Adapter and SDXL.Source: §3.1 · p. 7

3.2 Comparison with State-of-the-arts

  • Dual-FDMcust reports larger CLIP-T, CLIP-TC, CLIP-TS, CLIP-CS, IR, and HPS scores than competitors for customization-style transfer.The comparison includes image-customization methods and image-driven style-transfer methods.Source: §3.2 · pp. 7–8
  • Dual-FDMcost achieves the best performance on T2I-CompBench++ color-style-transfer comparisons, with larger CLIP-TS, CLIP-S, and DINO-S scores than competitors.The evaluation uses complex compositional text prompts.Source: Table 2 · p. 8, §3.2 · p. 8

3.3 Ablation Studies

  • Removing W_im k lowers similarity to the style reference, while using the same mid-frequency masks for early and late stages degrades performance.The authors interpret this as evidence that W_im k enhances the style-reference background and that mid-frequency bands converge faster than high-frequency bands.Source: §3.3 · pp. 8–9, Table 3 · p. 9
  • An early-stage proportion of 0.1 yields the highest style-reference similarity and lowest customized-reference similarity, while 0.5 reverses that pattern.Adaptive timing falls between the 0.4 and 0.5 settings and is reported to balance text alignment and image similarity.Source: §3.3 · p. 9, Figure 9 · p. 9
  • Wider mid-frequency masks favor customized-reference similarity, whereas narrower masks favor style-reference similarity.For low-frequency masks, reducing the proportion from 0.8 to 0.2 improves style-reference similarity but worsens color consistency.Source: §3.3 · p. 9, Table 4 · p. 9

4 Conclusion

Dual-FDM uses frequency-band masks to disentangle dual image references for customization-style transfer and color-style transfer.

4 Conclusion

  • Dual-FDM disentangles dual image references through frequency-band masks for customization-style transfer and color-style transfer, and experiments validate its superiority over state-of-the-art diffusion models.The conclusion identifies customization and color/style references as the dual references.Source: §4 · p. 9

A Frequency Variation During the Denoising Process

The appendix measures frequency-related image statistics over denoising timesteps and reports that low-frequency measures change little while high-frequency measures rise.

A Frequency Variation During the Denoising Process

  • The appendix measures low- and high-frequency changes at different timesteps to test the claim that low and mid bands recover before high-frequency details.Table 5 presents the frequency-band measures across denoising timesteps.Source: A Frequency Variation During the… · p. 13, Table 5 · p. 13
  • Low-frequency metrics remain nearly unchanged during denoising: Mean Gray varies only slightly from 119.5 to 120.9.Mean Gray, Mean HSV-V, and Mean Luma are used to capture global brightness and luminance structure.Source: A Frequency Variation During the… · p. 13
  • High-frequency metrics increase steadily during reverse denoising, with Avg Gradient rising from 59.01 to 233.88.Average Gradient, Variance, and LBP Variance measure edge sharpness, local contrast, and texture complexity.Source: A Frequency Variation During the… · p. 13

B More intuitions for the rationale between W im

The appendix examines how singular-value operations distinguish foreground and background information in cross-attention and tests alternative suppression strategies.

B More intuitions for the rationale between W im

  • The appendix evaluates foreground and background disentanglement using text-prompt similarity and image-level similarity measured with CLIP and DINO.The comparison evaluates disentangled images against prompts and generated images for foreground or background.Source: B More intuitions for the rationale… · p. 14
  • Hard suppression of the top 10–50% of singular values yields no clear performance improvement, and adjustment strategies show no significant difference in mean background extraction scores.The authors attribute limited gains and higher image-to-image variability to the difficulty of choosing a fixed foreground/background partition for each image.Source: B More intuitions for the rationale… · p. 14
  • The reciprocal operation 1/σ weakens dominant foreground components while amplifying smaller singular values associated with background.The authors contrast this with soft suppression, which attenuates foreground components without explicitly enhancing background components.Source: B More intuitions for the rationale… · p. 14
  • Across layers at the same spatial resolution, top singular-value proportions show consistent patterns, supporting the association of larger values with foreground and smaller values with background.The appendix also reports similar cross-layer behavior after inverting singular values.Source: B More intuitions for the rationale… · p. 14

C The algorithm of customization style transfer and color style transfer

Dual-FDM uses frequency masks to disentangle dual references for customization style transfer and color style transfer. The algorithms use task-specific substituted bands to reconstruct the personalized image.

C The algorithm of customization style transfer and color style transfer

D More details about the implementation details

The implementation details specify joint-task datasets, evaluation measures, and a fixed model setup. The customization algorithm also uses early- and late-stage frequency masks and reports efficiency and singular-value ablations.

D More details about the implementation details

  • DreamBench and TI comprise 38 subjects with 25 prompts each, while DreamBench++ includes 65 object, 20 human, and 45 animal classes with nine GPT-4o prompts per image.Source: D More details about the… · p. 15
  • IS-Plus contains 104 style images, StyleBench contains 490 images across 73 styles, and T2I-CompBench++ provides 8,000 compositional prompts.Source: D More details about the… · pp. 15–16
  • The appendix describes singular-value ablations that remove top-ranked values or reweight larger values, and reports reciprocal top-k singular-value summaries across cross-attention layers.It also states that the strategies are assessed for suppressing foreground components and enhancing background components.Source: Table 8 · p. 16, Table 9 · p. 16, D More details about the… · p. 16
  • The joint-task datasets pair DreamBench and TI or DreamBench++ customization references with IS-Plus or StyleBench style references, and pair InstantStyle-Plus and StyleBench color references with T2I-CompBench++ prompts.Source: D More details about the… · p. 16
  • Evaluation covers text/reference alignment, image quality, inference time, and VRAM; CLIP-TC and CLIP-TS assess reference similarity jointly with prompts, while CLIP-CS compares generated images with dual references.Image Reward and HPS V2 assess image quality; inference time and VRAM measure computational efficiency and memory footprint.Source: D More details about the… · pp. 16–17
  • The customization algorithm calculates mid-frequency key and value representations with early- and late-stage masks, then computes a task-specific threshold before denoising.Source: D More details about the… · p. 17
  • The customization and style-transfer algorithms use substituted frequency bands as key and value to reconstruct query foreground and background during decoupled cross-attention.Source: D More details about the… · p. 17
  • Experiments use pre-trained IP-Adapter-Plus with SDXL, 30 sampling steps, seed 42, and recommended method-specific hyperparameters on an RTX 4090.Source: D More details about the… · p. 17

E More Analysis of Computational Efficiency in the Sec.3.2 of the Mainbody

Dual-FDMcust is reported as efficient against representative baselines, with a stated inference time and the lowest VRAM consumption.

E More Analysis of Computational Efficiency in the Sec.3.2 of the Mainbody

F More generation results of different T2I Models

The section compares generation with three text-to-image models whose training data and characteristics differ. The reported examples show different balances between style emphasis and semantic alignment.

F More generation results of different T2I Models

  • RealisticVisionXL balances customized content and style well in the illustrated generation results.It is fine-tuned on photography-oriented datasets emphasizing realism, lighting, and fine-grained textures.Source: F More generation results of different… · p. 18
  • DreamshaperXL emphasizes style more strongly, with visually rich and creative outputs that may sacrifice strict prompt fidelity.Its training includes artistic and stylized datasets.Source: F More generation results of different… · p. 18
  • HunyuanDiT is described as offering stronger generalization across diverse prompts and better semantic alignment, and as more favorable to image customization.The passage attributes this to its broad multi-domain training and realism-oriented training.Source: F More generation results of different… · p. 18

G Additional generation results for the ablation study about the denoising timesteps between the early and late stages

The timestep ablation shows opposite reference-similarity trends at the 0.1 and 0.5 early-stage proportions, while adaptive adjustment falls between settings and balances text alignment with dual-image similarity.

G Additional generation results for the ablation study about the denoising timesteps between the early and late stages

  • At an early-stage proportion of 0.1, generated images are most similar to the style reference and least similar to the customized reference.The adaptive setting lies between the 0.4 and 0.5 proportions.Source: G Additional generation results for… · p. 18
  • At an early-stage proportion of 0.5, generated images show the opposite similarity trend, favoring the customized reference over the style reference.The adaptive setting lies between the 0.4 and 0.5 proportions.Source: G Additional generation results for… · p. 18
  • The adaptive adjustment regulates the early-to-late stage ratio and supports balanced text alignment and dual-image similarity.Its results lie between those for early-stage proportions 0.4 and 0.5.Source: G Additional generation results for… · pp. 18–19

H Additional generation results for the ablation study about the different masks to filter mid-and-low frequency bands

The mask ablation reports trade-offs between reference similarity and color consistency as the mid- and low-frequency proportions change.

H Additional generation results for the ablation study about the different masks to filter mid-and-low frequency bands

I Limitations and Broader Impacts

The section discusses limitations and broader impacts, including a sparse-pattern failure case in customization style transfer and the potential for personalized image generation to enable misleading content, prompting a call for responsible oversight. It also notes that LLMs are not an important, original, or non-standard component of the method.

I Limitations and Broader Impacts

  • Sparse line-based patterns in the style reference can cause mid-frequency customized cues to attenuate high-frequency style details, leaving style information incomplete.The authors identify this as a limitation of customization style transfer and plan to address it in future work.Source: I Limitations and Broader Impacts · p. 20
  • Personalized image generation could be misused to create misleading information, so the authors advocate clear accountability and stronger legal and technical oversight.They urge that the technology be applied responsibly.Source: I Limitations and Broader Impacts · pp. 20–21

J Code

  • Dual-FDM is implemented with Python 3.6, PyTorch 1.7, and CUDA 10.0, and its code is provided in the supplementary material.The code is in the folder named Dual-FDM.Source: J Code · p. 21

NeurIPS Paper Checklist

  • The authors report high experimental reproducibility but provide no statistical significance tests or error bars, noting that prior related work also omitted error bars [47] [1] [11] [49].They say their results are replicable and stable rather than selected as the highest outcomes.Source: NeurIPS Paper Checklist · p. 24
  • The checklist says safeguards for releasing Dual-FDM code will be set up, while noting that the work uses public personalized-image datasets and pretrained Stable Diffusion XL.The safeguard statement appears in the checklist’s response to the model-release risk question.Source: NeurIPS Paper Checklist · p. 26
  • The authors mark the LLM-related checklist item N/A because LLMs are not important, original, or non-standard components of the method.Source: NeurIPS Paper Checklist · p. 28
Loading 2610.07684v1…