Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

Shih-Ying Yeh; Yu-Guan Hsieh; Zhidong Gao; Bernard B W Yang; Giyeong Oh; Yanmin Gong

Source-linked AI summary

Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

Shih-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard B W Yang, Giyeong Oh, Yanmin Gong

arXiv:2309.14859v2Sep 26, 2023cs.CVcs.AIcs.GRcs.LG

TL;DR

The paper addresses limited personalization and the difficulty of systematically comparing Stable Diffusion fine-tuning methods. It introduces LyCORIS and a broad evaluation framework, finding that algorithm suitability depends on task complexity and concept structure. The study also identifies important evaluation challenges and scope limits.

Problem
Pretrained text-to-image models often struggle with highly personalized or novel concepts, while fine-tuning methods lack systematic comparative evaluation.
Method
The paper develops LyCORIS, a library of Stable Diffusion fine-tuning methods, and evaluates algorithms, hyperparameters, prompt types, and multiple performance criteria.
Results
LoHa seems better suited for simple multi-concept fine-tuning, whereas full-dimension LoKr is better for complex single-concept tasks.
Takeaways & Limitations
Fine-tuning algorithm and configuration choices have relative strengths and limitations that depend on the target task and concept structure.
Takeaways & Limitations
The study does not explore generating images with multiple learned concepts because the task is highly prompt-sensitive and harder to evaluate.

Abstract

from arXiv · show

Text-to-image generative models have garnered immense attention for their ability to produce high-fidelity images from text prompts. Among these, Stable Diffusion distinguishes itself as a leading open-source model in this fast-growing field. However, the intricacies of fine-tuning these models pose multiple challenges from new methodology integration to systematic evaluation. Addressing these issues, this paper introduces LyCORIS (Lora beYond Conventional methods, Other Rank adaptation Implementations for Stable diffusion) [https://github.com/KohakuBlueleaf/LyCORIS], an open-source library that offers a wide selection of fine-tuning methodologies for Stable Diffusion. Furthermore, we present a thorough framework for the systematic assessment of varied fine-tuning techniques. This framework employs a diverse suite of metrics and delves into multiple facets of fine-tuning, including hyperparameter adjustments and the evaluation with different prompt types across various concept categories. Through this comprehensive approach, our work provides essential insights into the nuanced effects of fine-tuning parameters, bridging the gap between state-of-the-art research and practical application.

1 Introduction

The paper addresses the difficulty of customizing Stable Diffusion for personalized concepts and evaluating diverse fine-tuning methods systematically. It introduces LyCORIS and studies algorithms, hyperparameters, and evaluation processes through extensive experiments.

Text-to-image models make high-quality image generation broadly accessible through text prompts and user-friendly interfaces.
Customization is motivated by pretrained text-to-image models’ difficulty capturing highly personalized or novel concepts from ordinary prompts.
LyCORIS is an open-source Stable Diffusion fine-tuning library spanning LoRA, LoHa, LoKr, GLoRA, and (IA)3.
The evaluation framework measures concept fidelity, text-image alignment, image diversity, and preservation of the base model’s style.
The experiments compare LyCORIS algorithms and hyperparameters while emphasizing the complexity of systematic model evaluation.

2 Preliminary

The paper builds on Stable Diffusion’s latent diffusion formulation and adapts pretrained models to target concepts with small datasets. It then describes LoRA as a parameter-efficient low-rank customization method.

2.1 Stable Diffusion: Stable Diffusion performs text-conditioned denoising in the latent space of a pretrained variational autoencoder.
2.1 Stable Diffusion: Stable Diffusion is pretrained on the LAION 5-billion image dataset and predicts noise applied to encoded image latents using text conditioning.
2.2 Model Customization With LoRA: Model customization fine-tunes the pretrained loss on a new dataset, potentially using as few as one image per target concept and a concept descriptor.
2.2 Model Customization With LoRA: LoRA freezes the foundational weight W0 and updates low-rank matrices B and A, reducing the number of trainable parameters.
2.2 Model Customization With LoRA: The LoRA forward pass adds γBAh to the pretrained transformation, with γ = α/r controlling the merge ratio.

3 The LyCORIS Library

LyCORIS provides a practical test bed for diverse Stable Diffusion fine-tuning strategies, including extensions that relax conventional low-rank constraints. Its methods trade off parameterization, rank capacity, and configurability.

LyCORIS integrates multiple fine-tuning strategies and supports existing command-line tools, graphical interfaces, and popular image-generation user interfaces.
LoRA (LoCon): LoCon extends LoRA from attention layers to convolutional layers in diffusion models to involve more layers during fine-tuning.
LoHa: LoHa uses a Hadamard product of two low-rank decompositions, allowing the update rank to reach r^2 while matching LoRA’s trainable-parameter count with dimension 2r.
LoKr: LoKr uses Kronecker-product decomposition, whose multiplicative ranks can move beyond low-rank assumptions and whose factor and dimension determine parameter sizes.
LoKr: LoKr can support a wide range of parameter counts and achieve smaller file sizes when configured appropriately.
Others: The library implements additional methods including DyLoRA, GLoRA, (IA)3, OFT, BOFT, and DoRA.

4 Evaluating Fine-Tuned Text-To-Image Models

The evaluation framework separates prompt types from image-quality criteria to assess fine-tuned text-to-image models more systematically. It examines memorization, generalization, leakage, fidelity, controllability, diversity, preservation, and visual quality.

The framework distinguishes training, generalization, and concept-agnostic prompts to expose different aspects of model behavior.
Prompt Types: Training prompts assess how closely generated images align with the training data, whereas generalization prompts test broader contexts and concept combinations.
Prompt Types: Concept-agnostic prompts avoid training triggers and help assess concept leakage, with further refinement based on class-word usage.
Evaluation Criteria: The image criteria cover fidelity to the target concept, text controllability, output diversity, preservation of base-model capabilities, and visual quality.
Together, prompt categories and evaluation criteria support analysis of multiple concepts, mutual interference, and merging separately trained models.

5 Experiments

The experiments compare LyCORIS algorithms and configurations across diverse concepts, prompt types, and evaluation criteria. Results show that training choices create trade-offs among fidelity, alignment, diversity, and base-model preservation, while evaluation itself remains challenging.

Experimental Setup: The study evaluates LoRA, LoHa, LoKr, and native fine-tuning across 26 configurations, three random seeds, and 234 checkpoints.Hyperparameters include learning rate, trained layers, dimension and alpha for LoRA and LoHa, and factor for LoKr.
Evaluation Procedure: Evaluation uses training, trigger, content-alteration, and style-alteration prompts to measure fidelity, controllability, diversity, and base-model preservation.Metrics use DINOv2, CLIP, Vendi, and style loss, respectively.
Evaluation Challenges: The evaluation exposes sensitivity to hyperparameters, concept-dependent performance, dataset effects, conflicting criteria, and unreliable image-quality metrics.The study addresses these issues with many configurations, category-specific analyses, and qualitative inspection, while excluding unsuitable primary quality metrics.
Training Epochs: Increasing training epochs generally improves concept fidelity but reduces text-image alignment, diversity, and base-model preservation, with overtraining especially evident for scenes and stuffed toys.Some cases exhibit overfit-then-generalize behavior, so the trend is not universal.
Learning Rate: Within a reasonable range, increasing learning rate has effects similar to increasing training epochs, but an excessively low rate cannot be fixed by training longer.Without data balancing, models remained undertrained even after 50 epochs.
Algorithm: Native fine-tuning at 5 · 10−6 achieves high training-prompt image similarity and high generalization-prompt text similarity, but lower image similarity on those generalization prompts.The comparison highlights why prompt types must be evaluated independently.
Trained Layers: Restricting fine-tuning to attention layers substantially decreases image similarity while improving other metrics, and omitting feedforward layers can prevent learning details such as a character’s uniform.The effects of convolutional-layer tuning are less discernible under the metrics used.
Capacity: Increasing model capacity can resemble increasing learning rate or training epochs when dimension-to-alpha ratio is fixed, although effects may reverse when alpha equals 1.Capacity is increased by raising dimension for LoRA or LoHa and lowering factor for LoKr.

6 Concluding Remarks

The paper contributes LyCORIS, a broad Stable Diffusion fine-tuning library, and a comprehensive evaluation framework. Its experiments indicate that method suitability depends on task complexity and concept composition, while the study does not cover multi-concept image generation.

Contributions: LyCORIS implements diverse Stable Diffusion fine-tuning methods, while the paper advocates evaluation across multiple criteria and configurations.The experiments characterize relative strengths and limitations rather than presenting a single universally best method.
Findings: LoHa seems better suited to simple multi-concept fine-tuning, whereas full-dimension LoKr seems better for complex single-concept tasks.The reported distinction suggests that matrix-update rank is not always the definitive predictor of efficacy.
Scope: The study does not explore generating images with multiple learned concepts because that setting is highly prompt-sensitive and more difficult to evaluate.The authors identify emerging evaluation frameworks as a way to enrich future comparisons.

A Related Works

Prior work spans customization methods for learning new concepts and evaluation metrics for assessing generated images. This paper focuses on evaluating fine-tuned models with separate prompting and image-assessment dimensions.

Model Customization: Text-to-image customization methods adapt pretrained models to new concepts using limited data, including DreamBooth, Textual Inversion, and concurrent embedding-network fine-tuning.Related approaches also target catastrophic forgetting, embedding balance, one-shot customization, and parameter efficiency.
Model Customization: LoRA-based methods extend across attention and convolutional layers, while LoHa, LoKr, OFT, and related techniques pursue higher rank, parameter efficiency, or pretrained-knowledge preservation.Different methods optimize distinct model components and can sometimes be combined complementarily.
Model Evaluation: Existing evaluation work measures distributional quality, text-image alignment, and broader criteria such as visual reasoning, social bias, and creativity.Common metrics include FID, KID, Precision-Recall, CLIPScore, R-precision, BLEU, and CIDEr.
Model Evaluation: Fine-tuned-model evaluation additionally separates concept fidelity, image fidelity, image quality, aesthetics, base-model preservation, and prompting or usage conditions.The authors note that their current metrics remain relatively rudimentary and could benefit from more advanced measures.
Model Evaluation: Human evaluation can be accurate and reproducible, but comprehensive human assessment is impractical at the paper’s scale of hundreds of checkpoints and millions of images.Most human-evaluation studies also compare few models using relatively few samples and criteria.
Model Evaluation: Evaluation results may reflect dataset characteristics and reward-function limitations, including the risk of overlooking under-represented preferences and reinforcing bias.The cited discussion cautions that evaluation is often more tied to the dataset than the training algorithm itself.

B.1 Effect of Merge Ratio

The merge ratio scales decomposed weight updates, and under stated optimizer conditions its effect can be reproduced by scaling initialization parameters and learning rates. The section also describes convolutional decompositions used by LyCORIS, including LoRA-style, Tucker, LoHa, and LoKr forms.

Effect of Merge Ratio: For homogeneous decompositions, replacing each merge ratio γℓ with 1 is equivalent to scaling layer initialization and learning rates by γℓ^c, with c=2 for SGD and c=1 for Adam, RMSProp, or AdaGrad when ε=0.The equivalence is stated for distinct layer-specific decompositions and merge ratios.
Effect of Merge Ratio: The equivalence follows because scaling decomposed tensors changes the weight update predictably, while layer-wise Jacobian structure permits the result to be established independently per layer.For adaptive optimizers with ε=0, the two procedures produce the same scaled update vector rather than the exact SGD-style relation.
Effect of Merge Ratio: Using merge ratios remains important for numerical precision because directly scaling initialization and learning rates would make stored decomposed parameters much smaller and potentially destabilize training.The theorem’s algebraic equivalence therefore does not imply that merge ratios are practically dispensable.
The Standard Approach: LoRA factorizes a convolutional weight update into reduced-rank matrices that can be reshaped into two consecutive convolutions with kernel sizes k and 1.The low-rank dimension r determines the intermediate channel count, and the construction extends straightforwardly to LoHa and LoKr.
Tucker Decomposition: Tucker decomposition represents a convolutional update with a core tensor and two matrices, reducing parameters from r(c_in k^2+c_out) to r(rk^2+c_in+c_out) when r is sufficiently small.The reduced form is substantially smaller when r ≪ c_in(1−1/k^2), and it can be implemented as three convolutional layers.
LoKr as Consecutive Linear Layers: LoKr uses Kronecker products to represent a weight update as two or three linear layers, with the third layer enabled by an optional low-rank decomposition of the right Kronecker block.The representation relies on the mixed Kronecker matrix-vector product property.

C Challenges in Algorithm Evaluation

Fine-tuning algorithms are difficult to compare because performance varies with hyperparameters, concepts, datasets, and evaluation criteria. The paper therefore argues for broader, more nuanced evaluation rather than single-metric or single-setting comparisons.

Overall Challenge: The paper argues that evaluations limited in metrics, hyperparameters, or concept categories may miss algorithm capabilities and unfairly penalize some methods.Its proposed response is a more comprehensive and systematic evaluation process.
Sensitivity to Hyperparameters: Performance is sensitive to learning rate, dimension, and factor, making comparisons based on one hyperparameter setting an oversimplification.Without a single objective metric, the best hyperparameters for a method remain difficult to identify.
Performance Discrepancy Across Concepts: Results can differ substantially across concepts, especially when concepts differ fundamentally or have different numbers of training images.Discrepancies may persist even after these factors are mitigated.
Influence of Dataset: Dataset composition, including images and captions, significantly influences fine-tuning performance and may require dataset-specific strategies or configurations.The authors emphasize adapting fine-tuning strategies to the particular dataset.
Conflicting Criteria: Higher concept fidelity can coincide with lower controllability, diversity, and base-model preservation, so selecting an optimal balance requires case-by-case analysis.The criteria cannot be reduced reliably to a simple aggregate score.
Unreliability of Evaluation Metrics: Automatic metrics may not align with human judgment or capture nuanced concept details, and a single numerical value can be insufficiently informative.The paper presents metric unreliability as a reason to avoid simplistic evaluation.
Practical Considerations: Practical evaluation must account for scaling weight differences, combining networks, changing the base model, and prompt engineering because these variables affect outcomes.These adjustments add complexity beyond the training algorithm itself.

D.1 Dataset

The study uses a hierarchically structured dataset spanning diverse concept categories, with imbalanced class sizes and tailored captioning. It evaluates multiple fine-tuning configurations on Stable Diffusion 1.5 while examining computational costs and training behavior.

Data Sources: Images come from multiple datasets and sources, including DAF:re, Kaggle, CustomConcept101, WikiArt, Old Book Illustrations, and Studio Ghibli’s website.The sources were selected to cover varied styles and subjects.
Captioning: Captioning combines automatic ConvNeXt V2 tagging with manual adjustments for scenes and stuffed toys, using a 0.35 threshold and filtering concept-bound tags.Unique tokens and class words define concepts and outfit subclasses, producing 32 distinct tokens.
Training Design: The experiments omit prior-preservation loss because it requires a separate regularization dataset and complicates analysis across the broad concept set.The authors also note that regularization prompts require careful design to ensure sample diversity and quality.
Training Configuration: Training compares LoRA, LoHa, and LoKr configurations on Stable Diffusion 1.5 using shared optimization and image-generation settings.LoRA is faster than LoHa and LoKr when convolutional layers are trained because of the chosen implementation strategy.
Training Efficiency: LoRA’s convolutional-layer speed advantage arises because it computes updates without explicitly constructing the full weight matrix, unlike LoHa and LoKr.The advantage is especially pronounced in middle U-Net layers with larger dimensions.

D.4 Evaluation Metrics

The evaluation framework represents images in encoder-based feature spaces and computes metrics for alignment, similarity, diversity, and style preservation. It normalizes and aggregates results across concepts, prompt types, classes, and random seeds.

Metric Framework: The framework evaluates text-image alignment, image similarity, diversity, and style preservation using analytically defined functions over encoder-derived representations.This design uses lower-dimensional feature spaces intended to capture perceptually relevant information.
Image Similarity: Squared centroid distance complements average cosine similarity because average cosine similarity inherently rewards less-diverse image sets.The authors therefore use squared centroid distance when contrasting diversity with image similarity.
Diversity: The Vendi score estimates the effective number of modes from eigenvalues of a kernel matrix built from normalized feature vectors.The study uses a linear kernel, and interprets higher sensitivity to mode count as complementary to intra-set dissimilarity.
Encoders: Image similarity and Vendi scores use DINOv2, CLIP, and ConvNeXt V2 encoders, while text-image alignment uses CLIP because it supports both image and text features.Main experiments use DINOv2 for image similarity and Vendi scores, with padded 224 × 224 inputs.
Normalization and Aggregation: Metric values are rank-normalized within each class or subclass and prompt type, then averaged across subclasses, classes, categories, and random seeds.Scatter-plot error bars represent standard errors across random seeds and classes.
Metric Robustness: Metric correlations remain strong across encoders but are often below 0.5 across different prompt types, indicating that prompt choice can substantially alter model rankings.Resizing methods produce correlations from 0.95 to 1, while encoder changes typically retain correlations of 0.65–0.9.

E.2 Relation between Different Metrics

The correlation analysis shows that metric relationships depend on prompt type and metric construction. It supports evaluating alignment, similarity, diversity, and style preservation as distinct dimensions rather than interchangeable measures.

Prompt-Dependent Correlations: Text and image similarity show a strong negative correlation for generalization prompts, but this relationship weakens considerably for training prompts.The contrast indicates that metric relationships depend on the prompt distribution used for generation.
Metric Distinctions: Vendi score, text similarity, and style loss correlate weakly despite each being negatively correlated with image similarity, supporting their independent evaluation.The analysis treats these metrics as distinct indicators of model performance.
Diversity and Similarity: The correlation between Vendi score and image similarity weakens when similarity uses squared centroid distance, validating that comparison for diversity analysis.Figure 6 reports this relationship using DINOv2-based image similarity and Vendi scores.
Supporting Analysis: The supporting plots include SHAP beeswarm plots and metric scatter plots organized by concept category, with epoch-30 scatter plots omitted because epochs 10 and 50 were considered sufficient.Observed discrepancies are classified as genuine deviations or metric-induced deviations caused by metric limitations or biases.

F.1 Plots for Category “Movie Characters”

Across categories and configurations, the plots reveal that fine-tuning behavior depends strongly on concept category, training duration, and hyperparameters, while evaluation metrics can disagree with visual evidence.

F.1 Plots for Category “Movie Characters”: LoKr shows high image similarity but low text similarity at lower epochs for movie-character generalization prompts, whereas LoRA reaches a similar pattern at higher epochs.
F.2 Plots for Category “Scenes”: Scenes are severely overtrained at 30 and 50 epochs, so SHAP analysis uses epoch-10 checkpoints; LoRA has the best base-model style preservation in this category.The difference between LoRA and LoHa is less pronounced for scenes, and qualitative examples validate LoRA’s style-preservation advantage.
Algorithm performance varies greatly across concept categories because of differences in training-sample counts and intrinsic concept complexity, even for comparable object classes.
Metric-based trends do not always match human perception: image-similarity encoders can misrank outputs, especially for less-common categories such as anime characters.The authors show that a native-fine-tuning configuration appearing worst by image similarity was misleading on inspection.
For castle, higher-capacity models show more mode collapse and overtraining artifacts, while an excessively small learning rate can prevent concept learning despite additional epochs.After 50 epochs of native fine-tuning at learning rate 10^-6, image quality is compromised while the concept remains barely learned.
Concept leakage from other classes becomes more pronounced in models that fit the target concepts more strongly, with words such as “forest” or “peacock” implicated as sources.

G.4.2 Improved Controllability Over Training

Additional qualitative studies show that controllability can improve with training or model capacity in ways that contradict simple overfitting rules, although the effect depends on concept and style.

G.4.2 Improved Controllability Over Training: For anime characters, longer training improves text-image alignment on generalization prompts, with examples showing more accurate horse-riding and spacesuit attributes.The authors suggest these concepts may need to overfit before they generalize, possibly because Stable Diffusion 1.5 struggles with anime-style images.
G.4.2 Improved Controllability Over Training: LoHa with dimension 16 and alpha 8 exhibits overfit-then-generalize behavior: weak prompt response at 10 epochs becomes substantially better after 50 epochs across repeated random seeds.The configuration shifts from an upper-left to a lower-right position when comparing image and text similarity for alter prompts.
G.4.3 Richer Stylistic Variations with Increased Model Capacity: For LoKr, reducing factors and thereby increasing capacity can produce more stylistically rich images with style prompts, contrary to the usual expectation of weaker base-style preservation.Whether the generated images authentically match the requested styles remains open, and mixed styles in the dataset may contribute to the effect.
G.4.3 Richer Stylistic Variations with Increased Model Capacity: Different styles require different amounts of training and model capacity, while increasing LoRA or LoHa dimension and alpha can harm generalization for object-specific style datasets.
H.1 Investigating the Relevance of Image Features: The relevance of image features depends on the similarity task: VGG-19 produces lower diversity ratios for style datasets, supporting its suitability for style similarity evaluation.Resizing method has little influence, while Vendi score generally yields lower diversity ratios and may better distinguish datasets with different diversity levels.

H.2 Image Quality Assessment with Pretrained Models

Pretrained image-quality assessors show substantial limitations on generated-image evaluation, including style bias, weak inter-model agreement, and failure to consistently rate real images higher than generated ones.

H.2 Image Quality Assessment with Pretrained Models: The IQA models’ scores vary with image style: LIQE and MANIQA tend to score anime images higher, whereas the artifact scorer tends to score them lower.The authors attribute part of this bias to training on natural images and to the artifact scorer’s differing target.
H.2 Image Quality Assessment with Pretrained Models: The models exhibit only weak correlation in their predicted scores, indicating limited agreement among the quality assessments.
H.2 Image Quality Assessment with Pretrained Models: None of the three IQA models consistently assign higher quality scores to real images, and MANIQA often scores artifact-containing generated images higher.The models evaluated are LIQE, MANIQA, and an artifact scorer.
H.2 Image Quality Assessment with Pretrained Models: Because these observations undermine confidence in the three IQA models for generated-image evaluation, the authors exclude them from the primary experiments.

H.3 Impact of Captioning Strategies on Model Performance

The study compares no-tags, all-tags, and manually adjusted-tag captions, finding distinct trade-offs among concept fidelity, controllability, diversity, and base-style preservation. Adjusted tags balance these extremes, while qualitative results show that overly simple or exhaustive captions can distort learned concepts.

Impact on Concept Fidelity, Controllability, and Base Style Preservation: No tags improve concept fidelity but reduce controllability and base-style preservation, whereas all tags produce the opposite trade-off.All tags can fragment the target concept across automatically assigned descriptors, limiting what the concept descriptor captures.
Captioning Strategies: Adjusted tags are used in the main experiment because they balance the controllability and concept-integrity failures of no-tags and all-tags captions.The strategy manually adjusts tags after initial tagging to avoid the extremes observed with the alternative captioning strategies.
Impact on Diversity: Both alternative strategies increase image diversity, but no tags broaden concepts inaccurately while all tags can leak training-set background details.The diversity gain from no tags reflects additional elements being captured, whereas all-tag diversity is associated with background information entering the learned concepts.
Impact on Diversity: In the stuffed-toys category, all tags improve both diversity and descriptor-only image similarity by producing more neutral backgrounds.Predicted tags can retain background details, increasing similarity and diversity when tag-based prompts are used.
Impact on Concept Fidelity, Controllability, and Base Style Preservation: LoHa is an exception: simple captions improve base-style preservation, potentially because of the phrasing “A ... of [V]”.This exception qualifies the broader trend linking simple captions with weaker base-model preservation.
Qualitative and Quantitative Evaluation: Figure 53 quantifies captioning trade-offs, while Figure 54 shows that short captions reduce prompt flexibility and all tags can misassociate target concepts with incidental tags.The qualitative examples include failures to respond appropriately to prompts and inaccurate capture of character attributes.