PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Concept Distillation for One-Step Diffusion Personalization

Yixiong Yang, Tao Wu, Senmao Li, Shiqi Yang, Yaxing Wang, Joost van de Weijer, Kai Wang

arXiv:2510.20512v3cs.CV

TL;DR

Fast one-step diffusion models are difficult to personalize: conventional approaches fail to learn concepts reliably. OPAD jointly trains a multi-step teacher and one-step student with alignment and adversarial supervision, and experiments report effective, high-quality one-step personalization; the method still faces training-efficiency and one-shot limitations.

  • Problem

    Conventional personalization methods fail to learn new concepts reliably in one-step diffusion models, while personalization of text-to-image models is important for synthesizing user-provided concepts.

    Source: §1 · p. 1

  • Method

    OPAD jointly trains a multi-step teacher and one-step student, using alignment losses and adversarial supervision, then feeds student-generated samples back into training.

    Source: §1 · p. 2, §3.3.2 · p. 5

  • Results

    OPAD achieves precise concept learning and strong prompt alignment, while several baselines fail or struggle in the one-step setting.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    OPAD enables effective adaptation to new visual concepts in one-step diffusion models while retaining one-step generation.

    Source: §1 · p. 2, §5 · p. 8

    Multi-discriminator overhead hinders training efficiency, and reliance on multiple reference samples makes true one-shot personalization challenging.

    Source: §7 · p. 14

Abstract

from arXiv · show

Recent progress in accelerating text-to-image diffusion models enables high-fidelity synthesis within a single denoising step. However, customizing the fast one-step models remains challenging, as existing methods consistently fail to produce acceptable results, underscoring the need for new methodologies to personalize one-step models. Therefore, we propose One-step Personalized Adversarial Distillation (OPAD), a framework that combines teacher-student distillation with adversarial supervision. A multi-step diffusion model serves as the teacher, while a one-step student model is jointly trained with it. The student learns from alignment losses that preserve consistency with the teacher and from adversarial losses that align its output with real image distributions. Beyond one-step personalization, we further observe that the student's efficient generation and adversarially enriched representations provide valuable feedback to improve the teacher model, forming a collaborative learning stage. Extensive experiments demonstrate that OPAD is the first approach to deliver reliable, high-quality personalization for one-step diffusion models; in contrast, prior methods largely fail and produce severe failure cases, while OPAD preserves single-step efficiency.

1. Introduction

The introduction motivates one-step personalization by describing failures of existing approaches and presenting OPAD’s joint teacher–student training, alignment and adversarial supervision, and collaborative learning. It also identifies a comparison of OPAD with existing methods in the 1-SDP setup using SDTurbo, with models unchanged from the initial distillation stage.

1. Introduction

  • On DreamBench, prior methods severely break down in one-step personalization, whereas OPAD adapts reliably while preserving high-quality one-step generation.The introduction identifies OPAD as the first approach to tackle one-step diffusion personalization.Source: §1 · p. 2

3.3.1. Overview

  • OPAD jointly trains the multi-step teacher and one-step student, sharing the teacher’s text encoder and updating only key and value projections to support knowledge transfer and limit overfitting.The teacher follows Custom Diffusion, while the student receives teacher-alignment and real-image adversarial supervision.Source: §1 · p. 2, §3.3.1 · p. 4

3.3.3. Collaborative learning stage

  • In a collaborative stage, student-generated concept samples replace real training examples and feed back into both models, improving their generative performance.The student’s rapid generation supplies additional samples after first-stage distillation, complementing its adversarial alignment to few-shot concept images.Source: §1 · p. 2, §3.3.3 · p. 5

2. Related works

  • Prior personalization methods tune concept tokens or model weights, while encoder-based methods remain poorly suited to few-step models and yield weak concept fidelity and image quality.Word-inversion methods preserve semantic consistency but can limit identity fidelity; weight tuning enables richer learning, while pretrained encoders often require backbone retraining.Source: §2 · pp. 2–3

3.1. Preliminaries

  • OPAD uses SD2.1 as its multi-step teacher and SD-Turbo as its one-step student, which generates an image directly from noise.The student’s one-step generation is the fast setting used for new-concept learning.Source: §3.1 · p. 3

3.2. Core Challenges in 1-SDP

  • Conventional personalization fails because one-step students resist token-only tuning, backbone updates can degrade quality, and teacher-first distillation is slow and may provide unreliable guidance.The teacher-first pipeline can require 25 or 50 NFEs for supervision, and a teacher that fails to learn a concept can mislead the student.Source: §3.2 · pp. 3–4

3.3.2. Distill with Alignment and Adversarial Guidance

  • Alignment combines identity-feature, latent-MSE, and multi-scale sliced-Wasserstein losses, while three frozen-backbone discriminators provide adversarial supervision; adversarial training can improve student visual quality beyond the teacher.The discriminators use DINOv1, DINOv2, and CLIP feature extractors; teacher guidance is downweighted at higher noise levels because its predictions become less reliable.Source: §3.3.2 · pp. 4–5, §3.3.3 · p. 5

3.3.3. Collaborative learning stage

  • Figure 3 compares OPAD with existing methods in the 1-SDP setup using SDTurbo as the one-step backbone, with one representative concept image shown.Source: Figure 3 · p. 6
  • Models remain unchanged from the initial distillation stage; the motivation for this design is discussed in supplementary Section 10.Source: §3.3.3 · p. 6

4. Experiments

Experiments compare OPAD with existing personalization methods and evaluate its components, inference-step behavior, backbones, and collaborative learning stage. Across these tests, OPAD performs strongly in one-step personalization, while added inference steps and student feedback show further effects.

4.1. Experimental setups

  • Evaluation covers 30 DreamBooth concepts, comparing word-inversion, optimization-based, and encoder-based methods using CLIP-T, DINO, and CLIP-I.The compared approaches include Textual Inversion, Cones 2, Custom Diffusion, DreamBooth, DisenBooth, TextBoost, and IP-Adapter.Source: §4.1 · p. 6

4.2. Experimental results

  • OPAD achieves the best CLIP-I and DINO scores, remains competitive on CLIP-T, and surpasses Custom Diffusion (SD2.1) in the first-row comparison.The first row uses Custom Diffusion trained for 1,000 steps and evaluated with 25 inference steps; other methods, including OPAD, use one training and inference step.Source: Table 1 · p. 7, §4.2 · p. 7

4.3. Ablation Study

  • Removing the teacher or discriminators degrades all evaluation metrics; teacher removal loses target-specific detail, while discriminator removal produces noisier outputs.The authors attribute the latter degradation in part to residual noise in the teacher’s single-step x0 predictions being propagated during student training.Source: Table 2 · p. 7, §4.3 · p. 7, Figure 4 · p. 8

4.2. Experimental results

  • Qualitatively, four baselines fail at effective denoising or concept learning, while OPAD produces precise concepts and strong alignment between images and prompts.DisenBooth and Cones2 struggle with precise concepts; IP-Adapter outputs are often blurry, prompt-misaligned, and affected by reference backgrounds.Source: §4.2 · p. 7

4.3. Ablation Study

  • Hyper-SD1.5 produces reasonable outputs but performs worse than the SDTurbo backbone in OPAD.The authors chose SDTurbo partly because its distillation-based training aligns with their framework and it is among few one-step models capable of high-quality images.Source: Table 3 · p. 7, §4.3 · p. 8
  • After collaborative learning, the teacher improves significantly on CLIP-I and DINO but declines slightly on CLIP-T; student feedback helps it reproduce challenging concepts.The comparison also reports that student outputs can enhance both teacher and student performance, particularly in identity and visual similarity.Source: Table 3 · p. 7, §4.3 · p. 8
  • With two or four inference steps, quantitative metric changes remain minimal, while image fidelity improves through richer backgrounds and finer textures.These evaluations use the same OPAD model without retraining.Source: §4.3 · pp. 7–8, Figure 4 · p. 8

5. Conclusions

The conclusion frames one-step diffusion personalization as a new task with three barriers to conventional methods. OPAD combines joint teacher–student training with alignment and adversarial supervision, and experiments on DreamBench report consistent outperformance of existing approaches.

5. Conclusions

  • The paper introduces one-step diffusion personalization as a task bridging fast generative inference and concept-personalized image synthesis.It identifies three major challenges that prevent conventional personalization methods from applying directly to one-step diffusion models.Source: §5 · p. 8
  • OPAD combines joint teacher–student training, alignment- and adversarial-guided distillation, and a collaborative stage using one-step student outputs to enhance training.Source: §5 · p. 8
  • Experiments on DreamBench report that OPAD consistently outperforms existing personalization approaches and enables adaptation to new visual concepts.Source: §5 · p. 8

6. Broader Impacts

The paper describes OPAD as efficient, high-fidelity personalized image generation with potential creative uses. It also identifies misuse risks and calls for safeguards and collaboration on ethical guidance.

6. Broader Impacts

  • OPAD is presented as a tool with potential applications in design, education, and virtual content creation.The passage says it reduces the need for extensive data and computation and may let users with minimal resources create customized visual content.Source: §6 · p. 14
  • Potential risks include unauthorized content generation, impersonation, and misleading or harmful imagery.Source: §6 · p. 14
  • The paper calls for content moderation, usage auditing, user authentication, and joint development of ethical guidelines and technical mitigations.Source: §6 · p. 14

7. Limitations

OPAD has unresolved limits in training efficiency, one-shot personalization, and run-to-run stability.

7. Limitations

  • The multi-discriminator architecture adds computational overhead to training, motivating optimization or redesign.The authors identify this overhead as a constraint on training speed.Source: §7 · p. 14
  • The discriminator relies on multiple reference samples, making faithful one-shot personalization challenging.The authors leave learning faithfully from a single image as an open problem.Source: §7 · p. 14
  • Training may be unstable across runs, particularly for challenging concepts, and may require a few trials to achieve optimal results.The authors identify improved training stability as future work.Source: §7 · p. 14

8. Ethical and LLM Statements

The authors discuss potential ethical risks of generative-model deployment and clarify the limited role of LLMs in this work.

8. Ethical and LLM Statements

  • The authors flag privacy, data misuse, and bias propagation as potential ethical implications of generative models.They also warn of potential misuse of customization for generating misinformation and encourage responsible use.Source: §8 · p. 14
  • All models used in the paper are publicly available, and the authors say they will release modified code to enable reproduction.Source: §8 · p. 14
  • LLMs were used only minimally, specifically to correct grammatical errors.Source: §8 · p. 14

9. Overview of T2I Personalization Methods

The overview compares personalization methods across several practical dimensions. OPAD's algorithm alternates teacher, student, and discriminator updates, then uses student-generated samples in a collaborative stage to train the teacher.

9. Overview of T2I Personalization Methods

  • Table 4 systematically contrasts personalization approaches by subject support, training paradigm, input-image requirements, and inference efficiency.The approaches were re-implemented and adapted to the same base-model configuration for comparison with the highlighted methods.Source: §9 · p. 14, Table 4 · p. 15

10. Algorithmic Description of OPAD Training

  • Each distillation iteration optimizes the teacher and text encoder with noise-prediction loss, trains the student with alignment and adversarial losses, and updates discriminators.Source: §10 · p. 14
  • In the collaborative stage, student-generated one-step samples replace real-image training examples and are used to enhance the teacher.The authors explain that the student learns a distribution partly shaped by the discriminator, with image features not inherently present in the teacher.Source: §10 · pp. 14, 16

11. Additional results on method comparison

Additional comparisons report fast OPAD inference and qualitative advantages over representative methods. In the one-step DreamBooth evaluation, Flux+OminiControl trails OPAD, though the authors note the comparison is not entirely equitable.

11. Additional results on method comparison

  • 0.18 seconds per instance is OPAD's reported inference time, alongside claims of superior visual fidelity in additional comparisons.Figures 6 through 9 compare reference images with outputs from representative methods.Source: §11 · p. 16
  • Table 5 reports training and inference time in seconds and the number of optimization iterations for each method.Source: §11 · p. 16, Table 5 · p. 17
  • On DreamBooth under the one-step setup, Flux+OminiControl outperforms the other reported baselines but remains significantly inferior to OPAD.The authors attribute the gap to Flux's weaker generation capability under one-step inference and caution that OminiControl's large-scale training data makes the evaluation not entirely equitable.Source: §11 · p. 17

12. Discussions for alternative designs

Alternative-design experiments compare joint teacher–student training with teacher-first distillation and other options. They also discuss feed-forward customization and the shared text encoder’s role.

12. Discussions for alternative designs

  • Teacher-first training performs worse than OPAD and is constrained by slow teacher inference, teacher unreliability, limited image diversity, and a teacher-bounded performance ceiling.The comparison is reported for the DreamBooth dataset.Source: §12 · p. 17, Table 6 · p. 18
  • Directly distilling teacher concepts with VSD, SDS, or MSE losses was insufficient to transfer the teacher’s personalization capability to the student.Source: §12 · p. 17
  • Feed-forward customization followed by distillation may miss concept-specific details because few-step models are difficult to align with the teacher’s full denoising trajectory.The discussion cites AYF [74], ADD [77], and LCM [51] as examples of such discrepancies.Source: §12 · p. 18
  • Removing the shared text encoder causes a clear performance drop, while the shared encoder simplifies training and improves learning efficiency.Source: §12 · p. 18

13. User study

A user study compared OPAD with other personalization methods using participant preferences for text-image alignment and object identity consistency. OPAD received substantially more votes than the second-best method.

13. User study

  • OPAD received at least 34% more user votes than the second-best method in a study with 15 participants and 840 preference annotations per method.Participants judged both text-image alignment and object identity consistency.Source: §13 · p. 18

14. Extended Ablation Study

The extended results examine component ablations, one-shot supervision, and generalization on CustomConcept101. They report component-dependent quality and stability, a decline under one-shot training, and strong qualitative generalization across concepts and prompts.

14. Extended Ablation Study

  • Removing ID loss, latent MSE loss, or MSSWD degrades performance, especially DINO scores; removing any discriminator causes broader metric declines and can destabilize or diverge training.The ablation study’s qualitative comparisons illustrate component effects.Source: §14 · p. 18, Table 9 · p. 19, Figure 10 · p. 24

15. 1-shot performance.

  • One-shot training declines across all evaluation metrics versus few-shot training and can produce plausible images that lack fine-grained concept details.The paper attributes the decline to its not being explicitly optimized for one-shot learning and to greater training instability with scarce data.Source: §15 · pp. 18–19

16. Results on the CustomConcept101 Dataset

  • On CustomConcept101, OPAD qualitatively generalizes across diverse concepts and prompt types while consistently generating high-quality outputs.Figures 11 and 12 show the results across the dataset.Source: §16 · p. 19, Figure 11 · p. 25, Figure 12 · p. 26

Related papers

Loading 2510.20512v3…