PaperlayerView exact revision on arXiv

Source-linked AI summary

Illustrious: an Open Advanced Illustration Model

Sang Hyun Park, Jun Young Koh, Junha Lee, Joy Song, Dongha Kim, Hoyeon Moon, Hyunju Lee, Min Song

arXiv:2409.19946v1cs.CV

TL;DR

Illustration and animation generation has lagged photorealistic image generation amid a shortage of large-scale fine-tuned open models and dataset constraints. Illustrious is introduced as an anime-generation model using a large dataset and detailed prompt guidance. The paper reports that it surpasses existing models across various aspects and supports high-resolution generation with anatomical integrity.

  • Problem

    Illustration and animation generation has progressed more slowly than photorealistic generation, partly because large-scale fine-tuned open models and suitable datasets are lacking.

    Source: §1 · p. 1

  • Method

    Illustrious combines a large dataset with detailed prompt guidance for anime image generation.

    Source: §1 · p. 1

  • Results

    Illustrious is reported to surpass existing models across various aspects, with high-resolution generation and anatomical integrity.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The reported model can express a wide range of concept combinations through prompt guidance and produce high-resolution images with anatomical integrity.

    Source: §1 · p. 1

    Danbooru’s tag-based metadata makes it difficult to control compositions and positioning of multiple characters or actions, and the detailed descriptions needed are often absent.

    Source: Limitations · p. 11

Abstract

from arXiv · show

In this work, we share the insights for achieving state-of-the-art quality in our text-to-image anime image generative model, called Illustrious. To achieve high resolution, dynamic color range images, and high restoration ability, we focus on three critical approaches for model improvement. First, we delve into the significance of the batch size and dropout control, which enables faster learning of controllable token based concept activations. Second, we increase the training resolution of images, affecting the accurate depiction of character anatomy in much higher resolution, extending its generation capability over 20MP with proper methods. Finally, we propose the refined multi-level captions, covering all tags and various natural language captions as a critical factor for model development. Through extensive analysis and experiments, Illustrious demonstrates state-of-the-art performance in terms of animation style, outperforming widely-used models in illustration domains, propelling easier customization and personalization with nature of open source. We plan to publicly release updated Illustrious model series sequentially as well as sustainable plans for improvements.

1 Introduction

Illustration and animation generation has lagged photorealistic generation, partly because large-scale fine-tuned open models and suitable datasets are lacking. Illustrious is presented as an anime model that addresses this gap with broad concept control and high-resolution generation.

1 Introduction

  • Illustration and animation generation has progressed more slowly than photorealistic generation, partly because large-scale fine-tuned open models and suitable datasets are lacking.The introduction contrasts this with photorealistic generation’s use of large datasets such as ImageNet and OpenImages.Source: §1 · p. 1
  • Illustrious is presented as a state-of-the-art anime model that surpasses existing models across various aspects.The introduction highlights concept combinations, detailed prompt guidance, and high-resolution images with anatomical integrity.Source: §1 · p. 1
  • Figure 1 presents Illustrious samples described as showing vibrant color and contrast across image styles.Figure 2 is labeled as a set of model comparison images, but the supplied caption does not report a comparison outcome.Source: Figure 1 · p. 2, Figure 2 · p. 3

2 Preliminary

The preliminary section situates Illustrious within latent diffusion and the anime-image data landscape, then reviews newer text-to-image models and text encoders. It also frames data ethics around transparent artist attribution and disclosure of model transformations.

2.1 SDXL

  • SDXL uses a U-Net with cross-attention and dual CLIP text encoders, which the paper says improves text-description understanding over earlier Stable Diffusion models.The cited SDXL model is described as having 817M text-encoder parameters and a 2.6B-parameter U-Net.Source: §2.1 · p. 3

2.2 Illustration / Animation Domain

  • Danbooru is a public anime dataset with over 8 million community-contributed images annotated for characters, scenes, copyrights, and artists.The paper notes that other available datasets are often processed Danbooru versions or face datasets used for benchmarking.Source: §2.2 · p. 3

2.3 Next-generation Text-to-Image Generative Models

  • Newer text-to-image models use additional or alternative text encoders to better interpret natural-language input and generate more sophisticated compositions.The paper names Hunyuan-DiT, Kolors, SD3, Flux, and AuraFlow as examples.Source: §2.3 · p. 4

2.3.1 Features of next-generation models

  • The reviewed models vary their architectures and training strategies, including Hunyuan-DiT’s fine-grained English and Chinese understanding and Kolors’ GLM-based caption comprehension.The section also describes SD3’s rectified-flow approach, Flux’s 12B-parameter hybrid architecture, and AuraFlow’s DiT encoder.Source: §2.3.1 · p. 4

2.3.2 Text Encoder

  • CLIP-only models have not shown significant success on complex compositions and glyph generation, and Illustrious may share limitations associated with its unchanged SDXL architecture.The text also says it is unknown whether SDXL is fundamentally limited in those tasks.Source: §2.3.2 · p. 4

2.4 Data Ethics

  • The paper argues for identifying styles with artist names and disclosing training methods and modifications when data or models are transformed.It presents transparency and fair public AI license terms as safeguards against artist exploitation.Source: §2.4 · pp. 4–5

3 Methodology

The methodology addresses dataset imbalance, sparse or ambiguous tags, resolution problems, and limits in the tag vocabulary through dataset refinement and expanded training strategies. It combines higher-resolution data with token, caption, and dropout methods intended to improve controllability and text-image correspondence.

3.1.1 Dataset Bias

  • Danbooru’s gender imbalance, overlapping tag meanings, sparse annotations, and problematic high-resolution images motivated dataset refinement and synthetic data with captions.The authors report that removing unfocused annotations later resolved discrepancies observed in v0.1.Source: §3.1.1 · pp. 5–6

3.1.2 Data Preprocessing

  • Tags were reordered by schema and combined with natural-language prompts, while score ranges were set using a percentile-based moving window.The initial schema orders person count, character names, rating, general tags, artist, score-range rating, and year modifier.Source: §3.1.2 · p. 6
  • Minimal pruning retained diverse samples while images above 4MP received mixed NEAREST/LANCZOS resizing and a minimally resized high-resolution set was used for training.Images below 768 × 768 were pruned, as were some images around 40MP and with aspect ratios above 1:10.Source: §3.1.2 · p. 6

3.1.3 Resolution

  • Training at 2.25MP supported native 2MP generation and generation up to 20MP with suitable img2img pipelines, while v2.0 added 0.15MP images for a wider output-resolution range.The paper associates minimal pruning with generating rare concepts, including MS Paint-like concepts, without harming overall generation quality.Source: §3.1.3 · pp. 6–7, Figure 4 · p. 7

3.1.4 Limited Corpus

  • The Danbooru vocabulary lacked tags for specific actions, and v0.1 struggled with longer natural-language prompts.The authors cite “covering wound with left hand” as an action the model struggled to generate due to insufficient data.Source: §3.1.4 · p. 7

3.2 Training Method

  • No Dropout Token training retained provocative or specific tokens, while weak-probability dropout for character and artist names improved characterwise understanding but could induce character generation without CFG control.The authors report faster character learning with smaller batch sizes under weak-probability dropout.Source: §3.2 · pp. 7–8
  • Quasi-Register Tokens captured outlier concepts missing from metadata, and padding tokens used for sequence matching were also treated as register tokens.The authors also adopted cosine annealing for lower learning rates and reasonably converged checkpoints, and report noise offset as useful for broader color ranges but unsuitable with lower batch sizes in the common training procedure.Source: §3.2 · pp. 7–8
  • Paraphrasing tag sequences diversified text inputs, while four dropout levels trained the model to handle varying caption detail and preserved strict control for no-dropout tokens.The dropout levels use different token-count rules with stated probabilities of 30%, 20%, 10%, and 4%.Source: §3.2 · p. 8

4 Training Setups

The paper reports sequential training setups from v0.1 through v2.0, varying dataset size, resolution, batch size, learning rates, epochs, and tagging strategy.

4 Training Setups

  • Illustrious v0.1 used a 7.5M-image, 1024 × 1024 dataset, batch size 192, and 20 epochs; v1.0 used 10M images at 1536 × 1536, batch size 128, and 8 epochs.The v0.1 U-Net and text-encoder learning rates were 3.5e-5 and 4.5e-6; v1.0 used 1e-5 and 6e-6, respectively.Source: §4 · p. 8, Table 5 · p. 8
  • Illustrious v1.1 used 12M images at 1536 × 1536, batch size 512, and 4 epochs, with natural-language descriptions combined with tags.Its U-Net and text-encoder learning rates were 3e-5 and 4e-6, respectively.Source: §4 · p. 8
  • Illustrious v2.0 used 20M images at 1536 × 1536, batch size 512, and 2 epochs, mainly incorporating the multi-caption method.Its U-Net and text-encoder learning rates were 4e-5 and 3e-6, respectively.Source: §4 · p. 8

5 Evaluation

The evaluation combines human preference ratings with character-similarity measures to assess image models. It describes Elo, CCIP, and TrueSkill, but the supplied passages do not state comparative evaluation outcomes.

5 Evaluation

  • The evaluation uses Elo Rating, TrueSkill 2, and character-wise similarity (CCIP), balancing quantitative measures with human judgment.The authors motivate human evaluation because image metrics may emphasize pixel similarity rather than image meaning.Source: §5 · p. 9, §5.1 · p. 9, §5.3 · p. 10

5.1 User Preference with Elo Rating

  • Elo ratings update from actual versus expected match results, and the study includes fixed-characteristics, free-prompt-duel, and free-for-all comparisons.Fixed-characteristics polls use two images with fixed prompts and allow draws; free-prompt duels use two free-prompt images, while free-for-all matches have four images.Source: §5.1 · p. 9

5.2 CCIP

  • CCIP uses CLIP-based feature extraction to estimate visual differences between images and assess character similarity.It focuses on character features such as facial attributes, clothing, and color schemes.Source: §5.2 · pp. 9–10

5.3 TrueSkill Algorithm

  • TrueSkill updates mean skill estimates while accounting for estimate uncertainty and game-outcome variance, and is described as more stable than Elo in sparse-duel conditions.The paper states that TrueSkill requires fewer trials to estimate expected skill scores.Source: §5.3 · p. 10

6 Limitations and Future Works

Illustrious is described as a generalized anime image generator, but the paper identifies limits in character-detail embeddings and tag-based descriptions. Future directions include improved natural-language data and text rendering, while Illustrious v1.0+ supports generation above 20MP.

Limitations

  • CLIP’s instability with character details can reduce performance in embedding-similarity calculations.The passage notes Flux and Kolors use T5 and GLM instead of CLIP.Source: Limitations · p. 11
  • Danbooru’s tag-based metadata makes multidimensional image descriptions difficult, limiting control over multiple characters’ composition and positioning.The paper says detailed descriptions of characters, positions, backgrounds, and relationships are often missing from such datasets.Source: Limitations · p. 11
  • The paper proposes large-scale refined natural-language datasets to address the identified limitations.It describes the dataset as custom-built and intended for release in future work.Source: Limitations · p. 12

Future Works

  • Generating full sentences or meaningful words in anime images remains challenging, although frequent phrases such as “Merry Christmas” can sometimes render correctly.The paper identifies text rendering as a key challenge for anime image generation.Source: Future Works · p. 12
  • Illustrious v2.0 shows improved glyph generation through synthetic captions, but its capability remains limited.The paper suggests OCR-based datasets and conditioning as possible future improvements.Source: Future Works · p. 12

A.1 Resolution

  • Illustrious v1.0+ generates images natively at 1536x1536, can expand to 2048x2048 without modification, and supports over 20MP generation.The passage says other models fail to follow at higher resolutions.Source: §A.1 · p. 18

A.2 Analysis

The section introduces experimental results from efforts to develop the Illustrious model and says they will be described in detail.

A.2 Analysis

  • The section frames the following material as experimental results from efforts to develop Illustrious.It says the results will be described in detail.Source: §A.2 · p. 18

A.2.1 Limitations regard to aesthetic/biased models

The section warns that aesthetic tuning can overfit human preferences and harm performance on the true data distribution, while Illustrious releases non-fine-tuned bases for broader adaptation. It also reports compatibility of model-derived add-ons with future models.

A.2.1 Limitations regard to aesthetic/biased models

  • Aesthetic tuning to human preferences can degrade performance on the true data distribution and reduce image-generation diversity.The authors say overfitting can make future fine-tuning harder because the model’s knowledge must be realigned with the true distribution.Source: §A.2.1 · p. 18
  • Model-derived LoRAs are reported to remain compatible with future models; Figure 10 describes a LoRA trained on Illustrious v0.1 as widely usable across checkpoints.The paper presents these add-ons as a way to derive models effectively.Source: Figure 10 · p. 18, §A.2.1 · p. 18
  • Illustrious releases non-fine-tuned base models to support broader public usability and adaptation to different tasks and concepts.The passage presents this release choice as a response to the risks of aesthetic tuning.Source: §A.2.1 · p. 18

A.2.2 Multiple-character generation

The analysis describes character separation under strict token control and discusses how detailed or upsampled captions affect generated images. It recommends sophisticated captions to make greater use of prompt and model capabilities.

A.2.2 Multiple-character generation

  • Strict token control yields strong character-feature separation with a limited budget, a behavior reported from Illustrious v0.1 through v2.0.Figure 11 depicts the character-separation behavior.Source: §A.2.2 · p. 18, Figure 11 · p. 19
  • The example prompt for the v0.1 separation result combines two character tags with “side-by-side” and “masterpiece.”The passage names Otonose Kanade and Hatsune Miku.Source: §A.2.2 · p. 19

A.2.3 the Effect of "Long Prompts"

  • Detailed or upsampled captions can produce sophisticated images, and the authors recommend them to use prompt and model capabilities further.Simple prompts are described as exposing the model’s creativity.Source: §A.2.3 · p. 19
  • Figure 12 presents the effect of long prompts; its caption says details in an upsampling prompt can help escape trivial solutions.TIPO is used to show sample differences across models.Source: Figure 12 · p. 19, §A.2.3 · p. 19

A.2.4 Batch Size and Learning Rates

Batch size and learning-rate choices shape concept learning, while later sections describe improvements in inpainting, brightness control, and captioning. Illustrious v2.0 adds multiple natural-language and tag captions per image.

A.2.4 Batch Size and Learning Rates

  • Large batches stabilize learning of sparse tags, while small batches produce more frequent attention binding that benefits broader concepts.The authors suggest adaptive batch sizes with learning-rate scheduling; below a threshold, learning new concepts may give way to stable attention splits.Source: §A.2.4 · p. 20

A.3 Inpainting

  • Illustrious inpainting produces results that harmonize with original content, including color and saturation, after masked areas are restored.The authors partially cropped and corrupted images, masked damaged areas, and applied inpainting.Source: §A.3 · p. 20, Figure 13 · p. 20

A.4 Dynamic Color Range

  • Illustrious improves prompt-based control of color and brightness, retaining colors at very low brightness so subjects can form silhouettes.The authors demonstrate this by increasing the brightness of generated low-brightness images.Source: §A.4 · p. 20, Figure 14 · p. 21

A.5 Multi Level Captions

  • Starting with v2.0, Multi-Level Captions assigns multiple captions to each image, combining natural language and tags to convey context that tags alone may not express.The authors plan to add captions with more detailed descriptions and contextual or narrative elements.Source: §A.5 · p. 21, Figure 15 · p. 21

A.6 Padding token wise analysis

The section discusses padding-token behavior, text-encoder fine-tuning, batch-size effects, safety control, and examples and generation settings across Illustrious versions.

A.6 Padding token wise analysis

  • Trained padding tokens can retain composition knowledge in classifier-free guidance, creating problems with imbalanced token lengths; masked loss is recommended for future training.The authors identify padding-token use in CFG setups as problematic and illustrate the issue with Figure 16.Source: §A.6 · p. 21, Figure 16 · p. 22

A.7 Further finetuning recommendations

  • Text encoders converged stably and were interchangeable, but the authors advise against fine-tuning them unless datasets are large enough to counter possible catastrophic forgetting.The recommendation is qualified by the dataset size needed to address that risk.Source: §A.7 · p. 22
  • Lower batch sizes are associated with fluctuating character learning, while higher batch sizes stabilize forgetting; sparse concepts may require still larger batches.The passage presents these as observed trends and notes the possibility of needing larger batches for sparse concepts.Source: §A.7 · p. 22

A.8 Safety control and Red-Teaming

  • The authors describe strict concept control followed by a LECO-based approach for safety control over provocative generations, and note that rating tokens or prompting “general” can provide simple control.GUIDED variants are cited as a reference for the approach.Source: §A.8 · p. 22

B.1.1 Illustrious v0.1

  • Figure 18 shows high-quality samples from Illustrious v0.1; its caption states that the model can generate creative pictures, supporting its capacity for creative image generation.A separate caption describes a Hatsune Miku image cosplaying Hakurei Reimu in 90s animation style with glowing eyes, generated in Illustrious v2.0 at 840×1216 resolution.Source: Figure 17 · p. 23, Figure 18 · p. 24

B.1.2 Illustrious v1.0

  • Illustrious v1.0 examples include a 2048 × 2048 image generated without upscaling and varied-style samples at 1536 × 1536.The high-resolution sample and style examples are described in separate figures.Source: Figure 19 · p. 25, §B.1.2 · p. 25, Figure 20 · p. 26

B.1.3 Illustrious v1.1

  • Illustrious v1.1 is illustrated with varied-style high-quality samples at 1536 × 1536.The passage identifies these as v1.1 examples.Source: §B.1.3 · p. 26, Figure 21 · p. 27

B.1.4 Illustrious v2.0

Related papers

Loading 2409.19946v1…