PaperlayerView exact revision on arXiv

Source-linked AI summary

Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA

James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, Hongxia Jin

arXiv:2304.06027v2cs.CVcs.AIcs.LG

TL;DR

The paper asks how text-to-image models behave when fine-grained concepts are customized sequentially, since existing methods suffer catastrophic forgetting. It proposes C-LoRA, a self-regularized low-rank adaptation of cross-attention layers, and reports reduced forgetting in Continual Diffusion plus state-of-the-art performance in rehearsal-free continual image classification. The authors caution that performance degrades on longer sequences and that multi-concept faces of similar individuals remain difficult.

  • Problem

    Sequentially customizing text-to-image models with new fine-grained concepts from few examples causes strong interference and catastrophic forgetting of prior concepts.

  • Method

    C-LoRA continually self-regularizes low-rank updates in Stable Diffusion cross-attention layers while using object-word-free prompts initialized with random embeddings.

  • Results

    C-LoRA alleviates catastrophic forgetting in Continual Diffusion and achieves state-of-the-art performance when extended to rehearsal-free continual image classification.

  • Takeaways & Limitations

    C-LoRA provides a promising approach for continually customizing text-to-image models while avoiding interference with previously learned concepts.

  • Takeaways & Limitations

    C-LoRA suffers on longer concept sequences such as 50 or 100 faces, and existing methods still struggle with multi-concept faces involving similar individuals.

Abstract

from arXiv · show

Recent works demonstrate a remarkable ability to customize text-to-image diffusion models while only providing a few example images. What happens if you try to customize such models using multiple, fine-grained concepts in a sequential (i.e., continual) manner? In our work, we show that recent state-of-the-art customization of text-to-image models suffer from catastrophic forgetting when new concepts arrive sequentially. Specifically, when adding a new concept, the ability to generate high quality images of past, similar concepts degrade. To circumvent this forgetting, we propose a new method, C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the popular Stable Diffusion model. Furthermore, we use customization prompts which do not include the word of the customized object (i.e., "person" for a human face dataset) and are initialized as completely random embeddings. Importantly, our method induces only marginal additional parameter costs and requires no storage of user data for replay. We show that C-LoRA not only outperforms several baselines for our proposed setting of text-to-image continual customization, which we refer to as Continual Diffusion, but that we achieve a new state-of-the-art in the well-established rehearsal-free continual learning setting for image classification. The high achieving performance of C-LoRA in two separate domains positions it as a compelling solution for a wide range of applications, and we believe it has significant potential for practical impact. Project page: https://jamessealesmith.github.io/continual-diffusion/

1 Introduction

Continual Diffusion studies sequential customization of text-to-image models with fine-grained concepts and few examples, where existing methods suffer catastrophic forgetting. The paper proposes C-LoRA to adapt Stable Diffusion while preserving prior concepts and reports gains in both continual customization and image-classification continual learning.

  • Motivation: Continual Diffusion introduces sequential customization of text-to-image models with previously unseen fine-grained concepts using only a few examples.The setting avoids retraining on past concepts after each update, reducing computational and data-privacy burdens.
  • Motivation: Existing customization methods catastrophically forget earlier fine-grained concepts when new concepts arrive sequentially.The paper analyzes this failure in Stable Diffusion using human faces and landmarks.
  • Method: C-LoRA uses continual self-regularized low-rank adaptation in Stable Diffusion cross-attention layers to learn new concepts while preserving prior knowledge.The method adapts a small number of weights and is designed to support multiple learned concepts together.
  • Method: C-LoRA also removes the customized object word from prompts and initializes custom tokens as random embeddings, outperforming commonly used customization prompts.This strategy is presented as an alternative to arbitrary embeddings of lesser-used words.
  • Motivation: Per-concept adapters and simple learned tokens are inadequate because they impede multi-concept generation or underperform for fine-grained concepts such as specific faces.The paper uses these limitations to motivate a shared model that can generate multiple learned concepts.
  • Results: The method alleviates catastrophic forgetting in Continual Diffusion and achieves state-of-the-art performance when extended to continual image classification.The paper supports these claims with quantitative and qualitative analysis in the two domains.

2 Background and Related Work

The paper builds on diffusion-based conditional image generation, prior single- and multi-concept customization methods, continual-learning strategies, and parameter-efficient fine-tuning. These lines of work motivate adapting a text-conditioned diffusion model while limiting parameter updates and avoiding replay of user data.

  • Conditional Image Generation Models: Diffusion models generate images by iteratively adding and removing noise, with cross-attention injecting text conditions into a transformer-based U-Net.The paper focuses on diffusion-based conditional generation because it accepts free-form text prompts.
  • Customization Methods: DreamBooth fine-tunes all diffusion-model parameters, Textual Inversion learns custom feature embeddings, and Custom Diffusion combines cross-attention fine-tuning, regularization, and weight merging.These methods primarily address customization to one or multiple concepts rather than sequential arrival.
  • Continual Learning: Continual learning trains on a sequence of tasks with different data distributions without forgetting previously learned knowledge.Regularization, rehearsal, and architecture-based methods are common strategies for mitigating catastrophic forgetting.
  • Continual Learning: Rehearsal methods store or generate previous-task samples, which may be infeasible because of privacy or copyright concerns.The paper contrasts this data-buffer requirement with rehearsal-free approaches.
  • Continual Learning: Prior multimodal continual-learning methods include latent replay and adaptation across coarse vision-language tasks, but some require storing compressed training data.These methods provide related context for continual learning beyond unimodal settings.
  • Parameter-Efficient Fine-Tuning: Parameter-efficient fine-tuning reduces the number of updated parameters through adapters, low-rank adapters, prompt learning, or selective fine-tuning.C-LoRA belongs to this broader family of efficient adaptation approaches.

3 Method

C-LoRA continually adapts Stable Diffusion by updating a small number of cross-attention parameters while preserving room for future concepts and reducing interference with past concepts. It combines self-regularized low-rank adaptation with random personalized tokens and prompts that omit object names.

  • C-LoRA: C-LoRA adds task-specific low-rank adapters and uses past LoRA parameters to guide future updates toward less-interfering parameters.The method aims to avoid overfitting new concepts and overwriting information learned from past concepts.
  • C-LoRA: The method updates only the key-value projections in Stable Diffusion’s U-Net cross-attention modules rather than the full model.These projections are selected because prior work identifies U-Net cross-attention parameters as sensitive during customization.
  • Self-regularization: The regularization avoids replay-data storage while retaining the parameter and storage efficiency of low-rank matrices.The paper contrasts this design with replay and parameter-importance approaches that require additional data or stored parameter copies.
  • Self-regularization: C-LoRA self-regularizes future LoRA updates by penalizing changes at locations edited by previous concepts.The penalty uses summed products of past LoRA parameters and adds no parameters beyond the LoRA parameters.
  • Custom tokenization: C-LoRA initializes personalized token embeddings randomly and removes object names from prompts to reduce interference among learned concepts.The authors report that including object names or using those words for initialization produces higher interference and can overwrite earlier concepts.
  • Putting It Together: The final approach combines continual custom tokenization with self-regularized low-rank adapters as a parameter-efficient method for continual customization.The paper evaluates the approach in two continual-learning experiment settings.

4 Continual Text-To-Image Experiments

The experiments evaluate continual customization across sequential celebrity-face and waterfall-landmark tasks using qualitative and quantitative measures of generation quality, forgetting, alignment, and parameter cost. C-LoRA produces recognizable generations across tasks, achieves the best reported Ammd performance, and offers lower storage than Custom Diffusion Merge in the landmark setting.

  • Metrics: The metrics include Ammd for average final-concept MMD, Fmmd for forgetting, CLIP-based image alignment, and the percentages of parameters trained and stored.Lower Ammd and Fmmd are better, and Ammd is identified as the more important metric.
  • Experimental setup: The evaluation compares Textual Inversion, DreamBooth, Custom Diffusion variants, Generative Replay, EWC, and ZipLoRA against C-LoRA.Custom Diffusion is evaluated with both sequential training and constrained merging.
  • Celebrity faces: The celebrity-face benchmark samples 10 sequential customization tasks from Celeb-A HQ and evaluates concepts 1, 6, and 10 after all tasks are trained.The study reports qualitative results for those tasks and quantitative results in Table 1.
  • Celebrity faces: C-LoRA produces recognizable images for all evaluated individuals and has the best Ammd performance on Celeb-A HQ.The passage also states that C-LoRA uses significantly fewer parameters than Custom Diffusion Merge.
  • Waterfall landmarks: The waterfall-landmark benchmark uses a 10-task sequence from Google Landmarks v2, with concepts 1, 5, and 10 shown after training all tasks.Waterfall landmarks are described as visually similar yet individually distinctive, creating a challenging sequence.
  • Waterfall landmarks: Custom Diffusion Merge requires 25× more storage than C-LoRA for the 10-task waterfall sequence, while C-LoRA is reported as the top performer.The comparison concerns on-the-fly merged models that retain KV values for all tasks.

5 On Longer Task Sequences

C-LoRA shows promise for moderate-length concept sequences, but its performance falters as sequences reach 50 concepts and eventually approaches Textual Inversion.

  • C-LoRA’s performance begins to falter as the concept sequence lengthens to 50 concepts across faces and landmarks.The sequence combines Celeb-A HQ and Google Landmarks dataset v2.
  • After 10 sequential Celeb-A HQ tasks, Figure 6 annotates which target data belongs in each generated image using standard quadrant numbering.
  • C-LoRA eventually approaches the performance of Textual Inversion, which only learns custom tokens without altering the backbone.
  • C-LoRA still outperforms Custom Diffusion with EWC regularization in the extended-sequence comparison.

6 Continual Image Classification Experiments

The paper evaluates C-LoRA in rehearsal-free continual image classification using the ImageNet-R benchmark and reports state-of-the-art performance against established baselines.

  • The evaluation uses ImageNet-R, a 200-class benchmark with varied image styles and a distribution substantially distant from ImageNet pre-training data.
  • The experiments reproduce CODA-Prompt’s setting with a ViT-B/16 backbone pretrained on ImageNet-1K and compare against rehearsal-free continual-learning methods.
  • Average accuracy AN measures performance across all past classes, while average forgetting FN measures the average drop in task performance.The paper emphasizes AN as the more important metric because it captures both plasticity and forgetting.
  • C-LoRA achieves state-of-the-art performance in rehearsal-free continual image classification, despite not being designed specifically for that setting.
  • C-LoRA improves clearly over L2P, DualPrompt, and CODA-Prompt, while ablating the forgetting loss substantially reduces performance.The ablation identifies the forgetting loss as the main driving force behind the reported performance.

7 Conclusion

C-LoRA addresses catastrophic forgetting in continual text-to-image customization through low-rank cross-attention adaptation and self-regularization. Its gains also transfer to rehearsal-free image classification, where it reaches state-of-the-art performance.

  • C-LoRA adapts Stable Diffusion’s cross-attention layers in a low-rank manner while preserving prior concepts through self-regularization.
  • The method alleviates catastrophic forgetting in continual text-to-image customization according to quantitative and qualitative analyses.
  • C-LoRA also achieves state-of-the-art performance in the standard continual-learning benchmark for image classification.
  • The authors present C-LoRA as a promising solution for continual customization while avoiding interference and forgetting of previously seen concepts.

8 Limitations & Broader Impact Statement

The authors report scalability limits for long face sequences and persistent difficulty with similar individuals, while highlighting consent, privacy, and misuse risks.

  • C-LoRA succeeds on sequences of up to ten faces but should not be trained on sequences of 50 or 100 faces because performance suffers.
  • All evaluated methods still struggle to generate multi-concept faces involving similar individuals.
  • The benchmark selection targets varied domains, using Celeb-A HQ for fine-grained facial variability and Google Landmarks for cross-domain evaluation.
  • The authors caution against generating faces of non-consenting individuals because realistic blended identities could enable disinformation or reputational harm.
  • Production use is discouraged without stringent ethical oversight and compliance with legal privacy protections.

A Additional Metrics and Results

The appendix defines additional metrics for evaluating customization quality and forgetting, then explains how these metrics complement and sometimes misrepresent continual-concept performance.

  • Metric definitions: Nparam measures the percentage of the U-Net backbone parameters trained during a task, while Ammd and Fmmd measure final distribution mismatch and average forgetting.Ammd and Fmmd use MMD-based comparisons in CLIP semantic space.
  • Metric definitions: Ammd compares original dataset images with generated images after all concept tasks, using CLIP embeddings and Maximum Mean Discrepancy.Lower Ammd is better.
  • Metric definitions: Fmmd measures the average distance by which images change during training, using MMD between images generated after each task and after the final task.The metric is intended to quantify forgetting over training.
  • Additional metrics: Image-alignment, text-alignment, and KID provide additional comparisons with prior offline customization evaluations.Text-alignment compares generated images with the generic prompt “a object.”
  • Metric interpretation: Text-alignment can favor methods with high catastrophic forgetting because it is conditioned on a generic object prompt rather than individual concepts.The authors include it for consistency with offline customization but consider it less insightful for Continual Diffusion.

B Ablation Studies

The ablation study identifies prompt design and the forgetting loss as important components of C-LoRA, with both removals producing clear performance drops.

  • Prompt and loss ablations: Removing the concept-free prompt design causes a clear performance drop relative to the full C-LoRA method.The ablation replaces “a photo of V*” with “a photo of V* person.”
  • Prompt and loss ablations: Removing the forgetting loss causes a clear performance drop relative to the full C-LoRA method.The ablation removes Eq. 3 from the method.
  • Ablation setup: The study separately ablates random token initialization, prompt concept inclusion, and the forgetting loss on Celeb-A HQ.These components are evaluated after sequentially training on ten concepts.

C Additional Implementation Details

Additional analyses examine implementation cost, parameter interference, sequential concept retention, multi-concept generation, and hyperparameter sensitivity across customization experiments.

  • Implementation details: 0.09% of the U-Net backbone parameters are trained by C-LoRA, compared with 2.23% for Custom Diffusion.For ten concepts, storing individual C-LoRA weights costs 0.9%, while the maximum storage cost is bounded by 2.23%.
  • Interference analysis: C-LoRA reduces the magnitude of sequential update interference more than LoRA alone, while also showing the least directional interference among compared methods.The analysis compares Custom Diffusion, Custom Diffusion plus LoRA, and C-LoRA.
  • Sequential concept retention: After ten concepts, Custom Diffusion Merge forgets the identity of concept 1, whereas C-LoRA continues generating good samples of that concept.Custom Diffusion retains identity through six concepts before interference begins after seven.
  • Multi-concept generation: In six-concept multi-concept generation, the identity of the second prompted concept tends to be lost for Custom Diffusion Merge, despite similar single-concept generations.The analysis uses the CustomConcept101 dataset, which lacks closely related fine-grained concepts.
  • Fair comparison: The appendix compares offline first-concept performance to ensure C-LoRA’s continual-learning gains are not attributable simply to using LoRA.Custom Diffusion performs better in this simpler offline comparison.
  • Hyperparameter analysis: C-LoRA has a range of effective λ and rank values, with performance declining outside bounded regions of the hyperparameter analysis.The analysis uses the first five concepts under the Table 1 setting.

I Additional Results for ImageNet-R

On ImageNet-R, C-LoRA generally preserves the reported continual-learning trends across shorter and longer task sequences, but slightly underperforms CODA-Prompt for twenty tasks.

  • Long task sequences: C-LoRA slightly underperforms CODA-Prompt on the twenty-task ImageNet-R sequence.The twenty-task sequence uses ten classes per task, and the comparison reports accuracy and average forgetting.
  • Parameter scaling: Prompting methods such as CODA-Prompt increase total parameters with each task, whereas C-LoRA remains constant-sized after LoRA parameters are folded into the model.The authors suggest a larger LoRA rank might improve C-LoRA on long sequences.
  • Evaluation setup: The appendix evaluates ImageNet-R sequences of 5, 10, and 20 tasks, reporting task-averaged accuracy and average forgetting over three trials.The task configurations use 40, 20, and 10 classes per task, respectively.
  • Image sources: Because of licensing constraints, many displayed images are generated substitutes, while training and evaluation use the original datasets.The appendix identifies which figures contain pseudo figure images and which contain model-produced results.
Loading 2304.06027v2…