Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation

• Stable Diffusion enables accessible, high-quality text-to-image generation, while extensions such as cross-attention control and ControlNet broaden its utility.Source: §1 · p. 1
• Personalized concept customization remains challenging because fine-tuning varies across datasets, image types, captioning strategies, methods, and hyperparameters, while evaluation lacks systematic coverage.DreamBooth and Textual Inversion demonstrate customization from small representative image sets, but research methods are not always integrated for comprehensive testing or adoption.Source: §1 · p. 2
• LyCORIS is an open-source Stable Diffusion fine-tuning library spanning standard LoRA and emerging methods including LoHa, LoKr, GLoRA, and (IA)3.Source: §1 · p. 2
• The proposed evaluation framework measures concept fidelity, text-image alignment, image diversity, and preservation of the base model’s style.Source: §1 · p. 2
• Extensive experiments compare LyCORIS fine-tuning algorithms and assess how hyperparameters influence results, motivating more comprehensive and systematic evaluation processes.Source: §1 · p. 2

2 Preliminary

• The study’s preliminary components are Stable Diffusion and LoRA for model customization.Source: §2 · p. 2

2.1 Stable Diffusion

• Stable Diffusion (Rombach et al., 2022) performs text-conditioned diffusion in latent space, encoding images, adding noise, and training a U-Net to predict it.The model was pretrained on the LAION 5-billion image dataset (Schuhmann et al., 2022).Source: §2.1 · p. 2

2.2 Model Customization With LoRA

• Customization fine-tunes Stable Diffusion on potentially single-image concepts using descriptors, while LoRA freezes the base weights and updates low-rank matrices.The descriptor contains a neutral trigger word and optional class word; prior-preservation loss is not used, and γ balances pretrained information retention against adaptation.Source: §2.2 · p. 3

3 The LyCORIS Library

• LyCORIS provides an open-source collection of Stable Diffusion fine-tuning methods for experimentation across command-line tools, graphical interfaces, and popular image-generation user interfaces.Its stored parameters generally permit reconstructing ΔW, enabling updates to be scaled, combined with other fine-tuned models, compressed, or integrated with tools such as ControlNet.Source: §3 · p. 3, §3.1 · p. 3

3.2 Implemented Algorithms

• The library centers on LoRA (LoCon), LoHa, and LoKr, with the merge ratio γ = α/r implemented across these methods.LyCORIS also includes DyLoRA, GLoRA, and (IA)3 (Liu et al., 2022), plus later additions including OFT (Qiu et al., 2023), BOFT (Liu et al., 2024), and DoRA (Liu et al., 2024).Source: §3.2 · pp. 3, 5
• LoCon extends LoRA from attention layers to Stable Diffusion’s convolutional layers, aiming to improve generated-image quality and fidelity by involving more layers during fine-tuning.LoRA’s original focus was attention layers in large language models, whereas convolutional layers are important in Stable Diffusion.Source: §3.2 · p. 4
• LoHa uses Hadamard-product decomposition to exceed conventional low-rank limits, reaching maximum rank r^2 versus 2r for an equal-parameter LoRA comparison when r > 2.The method targets stronger fine-tuning capacity without the memory and storage increases associated with simply using a larger rank.Source: §3.2 · p. 4
• LoKr uses Kronecker-product decomposition, adapts the approach to convolutional layers, and optionally applies low-rank decomposition to the larger right block for finer parameter control.Its multiplicative rank structure moves beyond low-rank assumptions, while the factor f and dimension r determine matrix sizes and configurations can produce the smallest file sizes.Source: §3.2 · pp. 4–5

4 Evaluating Fine-Tuned Text-To-Image Models

• The evaluation framework separates prompt types from generated-image criteria, enabling more nuanced assessment of fine-tuned model performance.This distinction addresses the wide range of LyCORIS algorithms and hyperparameters and supports systematic evaluation.Source: §4 · p. 5

4.1 Classification of Prompts for Image Generation

• Prompts are classified as training, generalization, or concept-agnostic, respectively testing concept capture, broader-context transfer and disentanglement, and concept leakage without trigger words.Concept-agnostic prompts may further distinguish whether class words are used.Source: §4.1 · p. 5

4.2 Evaluation Criteria

• Evaluation criteria cover fidelity, controllability, diversity, base-model preservation, and image quality, while excluding dataset-dependent aesthetics from training-method assessment.The framework also evaluates learning multiple distinct concepts without interference and merging multiple independently trained models.Source: §4.2 · p. 6

5.1 Dataset

• The dataset contains 1,706 images spanning anime characters, movie characters, scenes, stuffed toys, and styles, with class sizes ranging from 4–12 to 45–200 images.This broad, hierarchically structured and imbalanced dataset differs from prior single-concept studies using very few images.Source: §5.1 · p. 6

5.2 Algorithm Configuration and Evaluation

• The study evaluates LoRA, LoHa, LoKr, and native fine-tuning across 26 configurations, three random seeds, and three checkpoints, producing 234 checkpoints.Other parameter-efficient methods are excluded because most proposed modifications are considered complementary to the study’s approach.Source: §5.2 · pp. 6–7
• Models are tested with training, trigger, content-alteration, and style-alteration prompts, generating 14,900 images per checkpoint to assess concept reproduction and generalization.Concept-agnostic prompts are not included, and generalization prompts contain only one target concept.Source: §5.2 · p. 7

5.3.2 Analysis of Results and Insights for Fine-Tuning with LyCORIS

• Increasing epochs generally improves concept fidelity while reducing text-image alignment, diversity, and base-model preservation, although sparse concepts can become overtrained and some cases overfit before generalizing.The 30- and 50-epoch checkpoints are almost universally overtrained for scenes and stuffed toys; without balancing, models remained undertrained even after 50 epochs.Source: §5.3.2 · p. 8
• Increasing learning rate within the tested ranges has an effect similar to increasing training epochs, but an excessively low learning rate cannot be fixed simply by training longer.Native fine-tuning uses 5 · 10−7, 10−6, and 5 · 10−6; the other algorithms use 10−4, 5 · 10−4, and 10−3.Source: §5.3.2 · pp. 8–9
• Native fine-tuning at 5 · 10−6 achieves high training-prompt image similarity and generalization-prompt text similarity, but lower image similarity on generalization prompts.Comparisons among LoRA, LoHa, and LoKr use similar parameter counts and otherwise identical hyperparameters; style preservation shows no consistent category-level trend.Source: §5.3.2 · p. 9, Table 1 · p. 10
• Restricting training to attention layers substantially decreases image similarity while improving other metrics, whereas feedforward-layer training can be necessary to learn details such as a character’s uniform.The effect of convolutional-layer training is less discernible, possibly because the selected metrics are insufficiently sensitive.Source: §5.3.2 · p. 9, Figure 3 · p. 9
• Increasing model capacity by raising dimension or lowering factor behaves roughly like increasing learning rate or epochs when the dimension-to-alpha ratio is fixed.The effect can reverse when alpha equals 1, and varying capacity may amplify sensitivity to other hyperparameters.Source: §5.3.2 · pp. 9–10

6 Concluding Remarks

• LyCORIS provides an open-source library implementing diverse methods for Stable Diffusion fine-tuning.This is the paper’s first stated purpose.Source: §6 · p. 10
• The paper advocates comprehensive evaluation that better captures differences among fine-tuning methods.This is the paper’s second stated purpose.Source: §6 · p. 10
• Experiments show that algorithm and configuration choices affect model performance, revealing relative strengths and limitations.The experiments examine how these choices shape fine-tuning outcomes.Source: §6 · p. 10
• LoHa appears better suited to simple, multi-concept fine-tuning, whereas full-dimension LoKr is better for complex, single-concept tasks.These conclusions are based on the paper’s experiments.Source: §6 · p. 10
• The differing method applicability suggests matrix-update rank may not definitively predict efficacy across fine-tuning scenarios.Task complexity and concept structure qualify the role of rank as a performance predictor.Source: §6 · p. 10
• The study does not evaluate generating images with multiple learned concepts because the task is prompt-sensitive and harder to assess.The authors identify this as a scope limitation and point to emerging evaluation frameworks for future comparisons.Source: §6 · p. 10

A Related Works

• DreamBooth [Ruiz et al., 2023] fine-tunes the U-Net with prior-preservation loss, whereas Textual Inversion [Gal et al., 2023] optimizes input text embeddings for concept generation.These represent distinct adaptation targets: network weights versus text embeddings.Source: A Related Works · p. 20
• Custom Diffusion [Kumari et al., 2023] and Perfusion [Tewel et al., 2023] fine-tune K-V cross-attention layers differently, using model merging versus gated key-locking to improve contextual concept generation.Imagic [Kawar et al., 2023] instead optimizes a text embedding before further network fine-tuning for single-image editing.Source: A Related Works · p. 20
• LoKr and (IA)3 provide LyCORIS weight updates with parameter counts comparable to compact methods such as SVDiff [Han et al., 2023] and Cones [Liu et al., 2023].SVDiff optimizes singular values, while Cones targets concept neurons in K-V attention layers; Textual Inversion remains more parameter-efficient but often struggles outside the pretrained domain.Source: A Related Works · p. 20
• Text-to-image evaluation commonly combines image fidelity or quality with text-image alignment metrics, but prior customization studies have largely limited quantitative assessment to these two aspects.Examples include FID, KID, Precision-Recall, CLIPScore, R-precision, BLEU, CIDEr, TIFA, and X-IQE.Source: A Related Works · p. 21
• Fine-tuned-model evaluation must separately assess concept fidelity, preservation of pretrained knowledge, image fidelity, image quality, and aesthetics, while distinguishing prompting skills from image evaluation.The authors note that their current metrics remain relatively rudimentary and could benefit from more advanced methods.Source: A Related Works · p. 21
• Related innovations can be combined into a broader fine-tuning toolkit, including DreamArtist’s positive and negative guidance, XTI’s layer-wise embeddings, and Perfusion’s key-locking with LyCORIS algorithms.Table 2 recapitulates which model components and mechanisms different strategies optimize or employ.Source: A Related Works · p. 21, Table 2 · p. 22
• Human evaluation was omitted because the study scales to hundreds of checkpoints and millions of generated images, making comprehensive assessment prohibitively costly in time and resources.The authors identify multi-armed bandits and Bayesian optimization as possible approaches to scalable human evaluation.Source: A Related Works · p. 22

B Further Discussion on Implemented Algorithms

• The appendix examines finer algorithmic details, focusing on merge-ratio effects on training dynamics, Tucker decomposition for convolutional layers, and LoKr as consecutive linear layers.These topics extend the discussion of the algorithms implemented in the study.Source: B Further Discussion on Implemented… · p. 23

B.1 Effect of Merge Ratio

• For homogeneous decomposition functions, replacing γℓ by 1 is mathematically equivalent to scaling each layer’s initialization parameters and learning rate; c = 2 for SGD and c = 1 for Adam-type optimizers.For Adam, RMSProp, and AdaGrad, the ε parameter must be set to 0.Source: §B.1 · p. 23
• Merge ratios remain important for numerical precision because replacing them with scaled initialization and learning rates can produce smaller stored tensors, instability, or reduced final performance.The theorem explains training equivalence, but the implementation rationale is preserving numerical scale during optimization.Source: §B.1 · p. 24

B.2 Decomposition of Convolutional Layers

• LyCORIS applies matrix decomposition to convolutional weight updates through standard factorization and Tucker decomposition, extending the framework beyond linear layers.The convolutional update is unrolled into a matrix for factorization, while Tucker decomposition directly separates channel dimensions and a core tensor.Source: §B.2 · p. 24, §B.2.1 · p. 25, §B.2.2 · p. 25

B.2.1 The Standard Approach

• The standard convolutional approach reshapes LoRA factors into two consecutive convolutions with kernel sizes k and 1, and adapts straightforwardly to LoHa and LoKr.The low-rank dimension r becomes the output and input channel count of the first and second layers, respectively.Source: §B.2.1 · p. 25

B.2.2 Tucker Decomposition

• Tucker decomposition represents a convolutional update with a core tensor G and channel matrices A and B, reducing parameters from r(c_in k^2 + c_out) to r(rk^2 + c_in + c_out).This reduction is substantially smaller when r ≪ c_in(1−1/k^2), and the factors can be implemented as three convolutional layers.Source: §B.2.2 · p. 25
• The Tucker construction is also implemented for LoHa using two Tucker-decomposed updates combined elementwise, while LoKr uses the same convolutional decomposition methodology.The LoHa factors use separate cores and channel matrices for the two multiplicative components.Source: §B.2.2 · p. 25

B.3 LoKr as Consecutive Linear Layers

• LoKr decomposes ∆W as C ⊗ BA and, via the mixed Kronecker matrix-vector property, represents the update as two or three consecutive linear layers.The three-layer interpretation includes an additional low-rank decomposition and is illustrated in Figure 4 and implemented in Algorithm 1.Source: §B.3 · p. 26, Figure 4 · p. 26

C Challenges in Algorithm Evaluation

• Performance is sensitive to learning rate, dimension, and factor, making single-hyperparameter comparisons an oversimplification when no objective metric identifies the optimum.Figure 2 illustrates this hyperparameter sensitivity.Source: C Challenges in Algorithm Evaluation · p. 26
• Model performance can differ substantially across concepts, especially when concepts differ fundamentally or have different numbers of training images, even after mitigating those factors.The appendix refers to Appendix G.1 for remaining cross-concept discrepancies.Source: C Challenges in Algorithm Evaluation · pp. 26–27
• Dataset composition, including images and captions, significantly affects performance, so fine-tuning strategies and model capacity should match each dataset’s characteristics.Larger datasets or concepts far from pretrained knowledge may benefit from higher-capacity models, while good captions are important.Source: C Challenges in Algorithm Evaluation · p. 27
• Concept fidelity can conflict with controllability, diversity, and base-model preservation, requiring case-by-case balancing rather than simple metric aggregation.Section 5.3 particularly shows that higher concept fidelity often accompanies lower performance on the other criteria.Source: C Challenges in Algorithm Evaluation · p. 27
• Evaluation metrics may diverge from human judgment, miss conceptual nuances, and make a single score ambiguous because low image similarity can indicate underfitting or overfitting.The appendix cites Thrush et al. (2022); Yuksekgonul et al. (2023) and further discusses the issue in Appendix G.2.Source: C Challenges in Algorithm Evaluation · p. 27
• Practical evaluation must account for scaling weight differences ∆W, combining trained networks, changing base models, and prompt engineering, all of which add complexity.These adjustments are described in Section 3.1, while prompt effects introduce another source of variation.Source: C Challenges in Algorithm Evaluation · p. 27
• Fine-tuning evaluations should use broader frameworks because limited metrics, hyperparameters, and concept categories can unfairly penalize methods and miss their capabilities.The authors call for more extensive evaluation frameworks and ecosystems to support robust, meaningful comparisons.Source: C Challenges in Algorithm Evaluation · p. 27

D Experimental Details

• The experimental-details appendix records the datasets, algorithm settings, and procedures used to evaluate the models.It covers experiment information ranging from dataset specifics and algorithm configurations to model-evaluation details.Source: D Experimental Details · p. 28

D.1 Dataset

• The dataset covers anime characters, movie characters, scenes, stuffed toys, and styles, sourced from DAF:re (Rios et al., 2021), Kaggle (Young, 2019), CustomConcept101 (Kumari et al., 2023), WikiArt, and Old Book Illustrations (gigant, 2007).Style-related images also include Studio Ghibli’s official website; its images should be used “with common sense” and are not intended for commercial use.Source: §D.1 · p. 28
• Captions use a ConvNeXt V2 tagger trained on Danbooru at threshold 0.35, with manual scene and toy edits, concept-bound tag filtering, and 32 unique class or outfit tokens.The study omits prior-preservation loss because constructing a broad regularization set and its prompts is labor-intensive, while the setup isolates algorithmic effects and avoids added analysis complexity.Source: §D.1 · pp. 28, 30

D.2 Algorithm Configuration and Hardware

• Fine-tuning uses non-EMA Stable Diffusion 1.5 with 8-bit AdamW, weight decay 0.1, β1 = 0.9, β2 = 0.99, warm-up, gradient clipping, aspect-ratio bucketing, batch size 8, and 5% caption dropping.Images are generated at 512 × 512 using 25 DDIM steps, CFG scale 7, and no negative prompt.Source: §D.2 · p. 30
• Default learning rates are 10^-6 for native fine-tuning and 5 · 10^-4 for LoRA, LoHa, and LoKr, with LoRA and LoHa dimensions and alphas of (8, 4) and (4, 2), respectively.The default LoKr factor is 8, and shared configurations are applied across trained layers; file size, training time, and VRAM vary primarily with trained layers, dimension, and factor.Source: §D.2 · pp. 30–31, Table 5 · p. 32
• LoRA trains faster than LoHa and LoKr with convolutional layers because its separate-computation implementation better exploits middle U-Net layers with large unfolded dimensions.VRAM and efficiency figures are implementation- and hardware-dependent reference values rather than universal measurements.Source: §D.2 · p. 31
• Experiments run on four A6000 GPUs using sd-scripts 0.6.5, stable-diffusion-webui 1.6.0, LyCORIS 1.9.0.dev9, Python 3.10, PyTorch 2.0, transformers 4.26, diffusers 0.10.2, and accelerate 0.15.Fine-tuned images are generated through the stable-diffusion-webui API.Source: §D.2 · p. 31

D.3 Evaluation Prompts

• Evaluation uses natural-language prompts rather than tag-based training captions, with eight 10-prompt content-alteration templates and five shared style-alteration prompts.Style prompts are excluded for style classes, movie-character “others” subclasses are excluded, and each checkpoint produces 14,900 generated images.Source: §D.3 · pp. 31–32

D.4 Evaluation Metrics

• The evaluation framework separates encoders from analytically defined functions in a lower-dimensional representation space, using image similarity, text-image alignment, Vendi diversity, and VGG-based style loss.Average cosine similarity can favor less diverse image sets, so squared centroid distance is used as a complementary measure; Vendi score reflects the effective number of modes.Source: §D.4 · pp. 32–34
• Prompt preprocessing removes or replaces trigger and class words differently for characters, scenes, stuffed toys, styles, and outfit subclasses before text-image alignment evaluation.The procedure retains class words for anime and movie characters, substitutes class names for scenes and stuffed toys, and removes trigger and class words for styles and outfits.Source: §D.4 · p. 34

D.5 Encoders

• DINOv2 is the main encoder for image similarity and Vendi scores, while CLIP is used for text-image alignment and ConvNeXt V2 provides an additional image encoder.Images are padded when non-square and resized to 224 × 224 for DINOv2; all models use size “L” variants.Source: §D.5 · p. 35

D.6 Metric Value Processing

• Metric values are rank-normalized within each class, subclass, prompt type, metric, and checkpoint comparison, then averaged across subclasses, classes, and, for scatter plots, random seeds.The highest-ranked value receives 1 and the lowest 0; error bars represent standard errors across random seeds and classes.Source: §D.6 · p. 35
• SHAP analysis models average normalized category scores using algorithm, trained layers, epoch, capacity, and learning-rate level, with categorical features handled by CatBoostRegressor.Capacity is encoded differently for altered LoRA, LoHa, LoKr, and native fine-tuning configurations, with excluded configurations specified for the analysis.Source: §D.6 · p. 36

E Correlation Analysis

• The correlation analysis computes Pearson correlations of unaggregated normalized scores across four resizing methods: scale, letterbox, center crop, and stretch.Scale preserves aspect ratio and is used for ConvNeXt V2 and VGG19, stretch is used for DINOv2 and CLIP, and letterbox and crop are tested across the three image encoders.Source: E Correlation Analysis · p. 36

E.1 Influence of Encoders, Resizing Methods, and Prompt Types

• Resizing methods correlate at 0.95–1, while encoder-based metrics typically correlate at 0.65–0.9; correlations across prompt types are often below 0.5, with CLIP highest.Figure 5 reports stronger encoder agreement for generalization prompts than training prompts or concept-only captions, while prompt-type differences can change model rankings.Source: Figure 5 · p. 37, §E.1 · p. 37

E.2 Relation between Different Metrics

• The metric relationships are computed with DINOv2 for image similarity and Vendi score, while dataset images use letterbox resizing to reach the target resolution.This setup focuses on the metrics used in Section 5.3.Source: §E.2 · p. 37
• Text and image similarity show a strong negative correlation for generalization prompts, but this relationship weakens substantially for training prompts.These results are reported in Figure 6 using the paper’s primary metric setup.Source: §E.2 · p. 37, Figure 6 · p. 38
• Vendi score, text similarity, and style loss correlate weakly despite each being negatively correlated with image similarity, supporting their independent evaluation.The metrics therefore capture distinct indicators of model performance rather than interchangeable properties.Source: §E.2 · pp. 37–38
• The correlation between Vendi score and image similarity weakens when image similarity uses squared centroid distance, supporting the diversity comparison in Figure 2.This observation is consistent with the prior discussion in Appendix D.4.Source: §E.2 · p. 38, Figure 6 · p. 38

F Supporting Plots

• Supporting plots include SHAP-value beeswarms and metric scatter plots organized by concept category, with epoch 30 scatter plots omitted because epochs 10 and 50 were considered sufficient.The selected epochs were intended to demonstrate sensitivity to training duration.Source: F Supporting Plots · p. 38
• Discrepancies between supporting plots and Section 5.3 guidelines are classified as genuine deviations or metric-induced deviations caused by metric limitations or biases.Genuine deviations receive further qualitative discussion in Appendix G.4.Source: F Supporting Plots · p. 38

F.1 Plots for Category “Movie Characters”; F.2 Plots for Category “Scenes”; F.3 Plots for Category “Stuffed Toys”

• LoRA, LoHa, LoKr, and native fine-tuning exhibit category-dependent behavior, with movie-character plots showing LoKr’s high image and low text similarity earlier, while LoRA reaches similar behavior later.For scenes, epoch-30 and epoch-50 checkpoints are severely overtrained, and LoRA has the best base-model style preservation; for stuffed toys, LoRA’s greater diversity trades off against measured style preservation, while reducing LoKr’s factor improves style preservation.Source: §F.1 · p. 39, §F.2 · p. 41, §F.3 · p. 43

F.4 Plots for Category “Anime Characters”

• Anime-character metrics produce inconsistent conclusions, although qualitative results support the otherwise surprising improvement in text-image alignment for generalization prompts.The considered metrics are less suitable for anime characters, especially for image similarity on training prompts.Source: §F.4 · p. 45

F.5 Plots for Category “Styles”

• Style evaluation is inconclusive because average style loss and image similarity do not always agree, and “style” remains too ambiguous for definitive judgments.The styles plots use average style loss for style fidelity, retain image similarity as DINOv2 cosine similarity, and omit base-model style preservation.Source: §F.5 · p. 47

G Further Qualitative Results; G.1 Discrepancy of Results Across Classes

• Qualitative results show that fine-tuning performance varies greatly across concept categories and even comparable classes, with different models favoring lobster versus panda toys and canal versus garden contexts.The section uses qualitative examples to examine algorithm evaluation, factor trends, and deviations from general principles.Source: G Further Qualitative Results · p. 49, §G.1 · p. 49

G.2 Unreliability of Metrics

• Metric scores can misrepresent visual quality: CLIP may miss compositional relationships, and encoders may rank an inferior anime-character image above a visually better one.The authors therefore caution against over-reliance on metrics and emphasize task-relevant metrics and encoders for specialized applications.Source: §G.2 · p. 49, Figure 24 · p. 51

G.3 Illustrating the Impact of Different Algorithm Components; G.3.1 A Case Study on Sub-Class “Abukuma [Dark Color Uniform]”

• For difficult Abukuma uniforms, LoRA and default LoKr learn similar uniforms, LoHa requires increased dimension and alpha, and attention-only tuning is insufficient for LoRA and LoKr.None of the models perfectly reproduce the outfit, and the comparison uses epoch-50 checkpoints.Source: §G.3.1 · p. 49, Figure 26 · p. 52

G.3.2 A Case Study on Class “Castle”

• Castle customization exposes capacity-dependent failure modes: higher capacity increases mode collapse and overtraining artifacts, while too-small learning rates cause underfitting; LoKr and LoHa span the most- and least-fitted outcomes, respectively.Concept leakage from waterfalls and sculptures is stronger in more-fitted models but less evident for LoHa, LoKr with factor 12, and native fine-tuning.Source: §G.3.2 · p. 53

G.3.3 A Case Study on Sub-Class “Bodhi Rook [Realistic]”

• For Bodhi Rook, most models learn the character, but more-fitted models preserve dataset clothing more strongly while retaining prompt flexibility; LoHa and lower-learning-rate models produce greater diversity, whereas native fine-tuning best preserves base style.The observed style-preservation advantage of native fine-tuning holds at learning rates 10^-6 and 5 · 10^-6.Source: §G.3.3 · pp. 53, 58

G.4.2 Improved Controllability Over Training

• Longer training can improve text-image alignment and generalization, producing an overfit-then-generalize effect for some concepts and configurations.For LoHa with dimension 16 and alpha 8, generalization improves substantially from 10 to 50 epochs across all classes and three random seeds, resembling grokking or double descent.Source: §G.4.2 · p. 58, Figure 40 · p. 66

G.4.3 Richer Stylistic Variations with Increased Model Capacity

• Increasing capacity does not uniformly reduce style preservation: LoKr with smaller factors can produce richer stylistic images, whereas LoRA shows no comparable improvement.The richer outputs may reflect incorporation of multiple styles present in the fine-tuning dataset.Source: §G.4.3 · p. 66, Figure 41 · p. 67

G.5 Example Generations for Category “Styles”

• Style concepts require different training durations and capacities, so increasing LoRA or LoHa dimension and alpha can either benefit or harm generalization depending on the style class.The “Felix Vallotton” class is harmed by these changes, whereas another style class benefits; later training can also oversaturate some styles.Source: §G.5 · p. 66, Figure 43 · p. 68, Figure 45 · p. 69

H Additional Experiments

• The additional experiments validate design choices from the main study and provide further insight into fine-tuning text-to-image models.This appendix includes experiments on model behavior, style variation, and image-feature relevance.Source: H Additional Experiments · p. 71

H.1 Investigating the Relevance of Image Features

• The study examines how four image encoders and four resizing techniques affect image-similarity features, comparing style-loss and semantic-similarity behavior across 10 distinct encoding methods, including ImageNet100 and DAF:re-250.Source: §H.1 · p. 71
• The diversity ratio compares within-class feature diversity with overall dataset diversity, while intra-dissimilarity and variance differ only by feature normalization.The analysis uses Vendi Score, intra-dissimilarity, and variance, with intra-dissimilarity equal to the variance of normalized vectors.Source: §H.1 · p. 71
• Resizing has little influence on diversity-ratio results, while VGG-19 yields the most style-sensitive ratios for intra-dissimilarity and variance.VGG-19 produces lower ratios for style datasets and higher ratios for ImageNet100 and DAF:re-250, supporting its suitability for style similarity evaluation.Source: §H.1 · p. 72, Figure 48 · p. 72
• Vendi Score generally produces lower diversity ratios and better distinguishes datasets with different degrees of diversity, although its behavior does not match the encoder pattern seen with the other metrics.The discrepancy in Vendi-based encoder results remains an open question for further investigation.Source: §H.1 · p. 73, Figure 49 · p. 73

Results

• The models exhibit style-related bias: LIQE and MANIQA score anime-style images higher, whereas the artifact scorer scores them lower.The paper attributes these effects to natural-image training for LIQE and MANIQA and possible biases in the AI Horde ratings data for the artifact scorer.Source: §H.2 · p. 74, Figure 50 · p. 74
• The models show only weak correlation in their predicted scores across the evaluated image sets, indicating limited agreement.Correlations are examined both across all images and after excluding the “anime characters” and “styles” categories.Source: §H.2 · pp. 74–75, Figure 51 · p. 74
• None of the models consistently score real images higher than generated images, and MANIQA often rates artifact-containing generated images higher.This comparison is shown for training-set images and images generated with a 10th-epoch LoRA checkpoint using default hyperparameters.Source: §H.2 · pp. 74–75, Figure 52 · p. 75
• The three IQA models are unreliable for assessing AI-generated image quality, so they are excluded from the primary experiments.The models evaluated are LIQE (Zhang et al., 2023), MANIQA (Yang et al., 2022), and an AI Horde-trained artifact scorer (Wang et al., 2022).Source: §H.2 · pp. 73, 75

H.3 Impact of Captioning Strategies on Model Performance

• The comparison evaluates no tags, all tags, and manually adjusted tags using LoRA, LoHa, and LoKr with default hyperparameters, plus native fine-tuning at 5 × 10−6.The native-fine-tuning rate is selected because the main experiment found it consistently better than 10−6; quantitative and qualitative comparisons appear in Figures 53 and 54.Source: §H.3 · pp. 75–76
• The no-tag trend has a LoHa exception: base style preservation improves with simple captions, potentially because of the phrasing “A ... of [V]”.Source: §H.3 · p. 76
• In the stuffed-toys category, all tags also improve similarity for concept-only prompts, while tag-conditioned prompts can increase similarity and diversity through background leakage.Manually adjusted tags better represent backgrounds, whereas predicted tags can associate background details with the learned concept.Source: §H.3 · p. 76
• Captioning strategies trade off concept fidelity against controllability and base style preservation, with adjusted tags balancing these competing effects.No tags tend to improve concept fidelity while reducing controllability and style preservation; all tags show the opposite pattern, whereas adjusted tags preserve concept integrity with controllability.Source: §H.3 · p. 76, Figure 53 · p. 77
• Both no-tag and all-tag captions improve image diversity over adjusted tags, but for different reasons tied to inaccurate or incomplete concept descriptors.No-tag captions absorb additional training-set elements, while all-tag captions capture only part of the target concept and therefore permit greater generative freedom.Source: §H.3 · p. 76, Figure 53 · p. 77
• All tags can fragment the target concept across automatically predicted descriptors, causing missing or misdirected concept attributes in generated images.Qualitative examples include failure to capture a character’s hairstyle and uniform, omission of a sculpture, and association of the sculpture with a frequently occurring “Christmas tree” tag.Source: §H.3 · p. 76, Figure 54 · p. 78

I Author Contributions

• The contributions of each author are summarized in Table 7.Source: I Author Contributions · p. 79, Table 7 · p. 79