ConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints

1 INTRODUCTION

• ConceptLab seeks novel concepts that belong to a broad category while differing from existing members, without explicitly describing their appearance.The paper illustrates this goal with new pet concepts that resemble pets but do not correspond to a specific species.Source: §1 · p. 1
• The method represents the target concept as a learned token and uses positive category constraints alongside negative constraints for existing members.For example, “pet” is positive while “cat” and “dog” are negative constraints.Source: §1 · p. 1
• Optimizing constraints over Diffusion Prior outputs avoids unstable image-space CLIP optimization and does not require reference images of the nonexistent target.The resulting constraints are called “prior constraints.”Source: §1 · pp. 1–2
• A question-answering VLM periodically identifies the current concept’s category member and adds it as a negative constraint, progressively shifting generation toward greater novelty.This expands the negative set during segmented optimization rather than requiring all category members to be specified manually.Source: §1 · p. 2
• Prior constraints also mix generated concepts into hybrids, and quantitative and qualitative evaluations demonstrate their effectiveness for creative generation.The introduction presents concept mixing as an extension of the same constraint mechanism.Source: §1 · p. 2

Text-Guided Synthesis

• Text-guided synthesis commonly conditions diffusion models on embeddings from pretrained text encoders, while this work pairs a Latent Diffusion Model with a Diffusion Prior.The Diffusion Prior is used as part of the paper’s text-to-image synthesis pipeline.Source: §2 · p. 2

Diffusion Prior

• Diffusion Prior models map text embeddings to CLIP image embeddings before decoding, supporting diversity, image variations, and interpolations.Prior-based pipelines have also been adopted for video synthesis and 3D generation.Source: §2 · p. 2

Personalization

• Personalization methods inject user-specific concepts through optimized embeddings, network fine-tuning, or trained encoders, whereas ConceptLab generates concepts without a specified subject.This distinction separates creative generation from teaching a model a known subject or style.Source: §2 · pp. 2–3

Creative Generation

• Prior creative-generation work includes evolutionary shape modeling, deviation from existing artistic styles, and composition of fine-level components.These approaches cover preference-guided diversity, style deviation, and component fusion.Source: §2 · p. 3
• ConceptLab instead optimizes novel concepts to match a given category while differing from existing concepts, generating diverse results without directly describing their look.The approach is illustrated as optimizing one embedding with positive and negative constraints and expanding negatives using a VLM.Source: §2 · p. 3, Figure 3 · p. 3

Diffusion Prior

• ConceptLab is built on Kandinsky 2, which combines a Latent Diffusion Model with a Diffusion Prior to enable constraints over Prior outputs.This architecture supplies the intermediate representation used by the creative-generation method.Source: §3 · pp. 3–4

Latent Diffusion Models

• A Latent Diffusion Model performs diffusion in an autoencoder’s latent space, whose encoder maps images to latent codes and decoder reconstructs them.The denoising diffusion model is trained to produce latent codes in this learned space.Source: §3 · p. 3
• The denoising network removes noise from a latent code using the noisy code, timestep, and conditioning vector.The conditioning vector can be supplied during denoising.Source: §3 · p. 3

Diffusion Prior

• A Diffusion Prior first predicts an image embedding from text, after which a diffusion decoder generates an image conditioned on that embedding.The two models are trained independently for their respective stages.Source: §3 · p. 4
• Combining the stages improves image diversity and exposes an intermediate CLIP image-embedding space for direct constraint insertion.The paper reports this output space as more effective for constraints than a standard diffusion model or direct CLIP text embeddings.Source: §3 · p. 4

4 METHOD

• ConceptLab optimizes a token embedding to resemble a broad category while differing from existing concepts through prior constraints.Positive constraints encourage category similarity, while negative constraints discourage existing members.Source: §4 · p. 4, §4.1 · p. 4

4.1 Diffusion Prior Constraints

• Diffusion-prior constraints compare generated realizations of the learned concept against broad positive and negative textual constraints for consistent generations.The learned concept passes through the Diffusion Prior, while constraints remain broad rather than representing specific images.Source: §4.1 · p. 4
• A maximal-similarity regularizer penalizes the closest negative constraint, while bounded similarity prevents pathological text-in-image solutions.The regularizer addresses collapse toward one constraint when many negatives are present.Source: §4.1 · p. 4

4.2 Adaptive Negatives with VLM-Guidance

• Adaptive negatives query BLIP-2 during optimization, adding detected category members as constraints and producing diverse creations across training seeds.The target category is included in the question, and constraints are added from concepts emerging during optimization.Source: §4.2 · p. 5, Figure 4 · p. 5

4.3 Evolutionary Generation

• Concept mixing learns a new concept from CLIP image embeddings of generated concepts or real images, supporting iterative hierarchical generations.An optional weight controls each source concept’s influence on the generated output.Source: §4.3 · p. 5
• Figure 5 shows creative generation results across categories using adaptive negatives with different training seeds.The visual documents the diversity associated with varying seeds under adaptive negatives.Source: Figure 5 · p. 5

5 IMPLEMENTATION DETAILS

• The experiments use Kandinsky 2.1, an extended multilingual CLIP text encoder, and single-GPU training for up to 2500 steps.Training uses batch size 1, learning rate 0.0001, and λ=1.Source: §5 · p. 5
• BLIP guidance is queried every 250 steps, with optimization manually stopped when BLIP can no longer correctly classify the generated concept.Optimization steps take about 0.2 seconds, while BLIP-guidance steps take about 8 seconds.Source: §5 · p. 5

6.1 Results

• ConceptLab learns diverse novel concepts across categories and places them in new scenes, styles, and contexts using adaptive negatives.The method also discovers artistic styles while remaining faithful to guiding prompts.Source: §6.1 · pp. 5, 8
• ConceptLab forms hybrids by combining traits from multiple concepts, whereas curated prompts fail to preserve key aspects of all desired concepts.Examples combine lobster color and claws with a turtle shell, or snake, zebra, and hippopotamus characteristics.Source: §6.1 · p. 8, Figure 9 · p. 8
• ConceptLab evolves concepts across multiple generations by repeatedly mixing learned traits, producing new creations such as a blue-like rat mammal.The evolutionary process combines properties from parent concepts and continues across generations.Source: §6.1 · p. 8, Figure 8 · p. 8

6.2 Comparisons

• Against Stable Diffusion and Kandinsky with negative prompting, ConceptLab consistently achieves higher positive CLIP similarity across all five domains.The evaluation covers pets, plants, fruits, furniture, and musical instruments using generated images and CLIP-based measurements.Source: §6.2 · p. 9, Figure 11 · p. 9
• ConceptLab exceeds Stable Diffusion on negative distance in every category and exceeds Kandinsky in four of five categories, while participants heavily favor it in user ratings.The user study included 30 respondents and 300 ratings per method.Source: §6.2 · pp. 9–10, Table 1 · p. 10

6.3 Additional Analysis

• Applying constraints in the Diffusion Prior improves consistency: SD-ConceptLab often fails to align, while CLIP-ConceptLab respects constraints but varies across prompts.The comparison supports applying the Diffusion Prior over the learned concept representation.Source: §6.3 · p. 10, Figure 12 · p. 10
• Setting λ=1 balances positive and negative constraints; lower positive weighting loses category alignment, whereas higher weighting causes negative constraints to be ignored.The weighting study identifies a practical balance between resemblance to the target category and distinction from existing concepts.Source: §6.3 · p. 10, Figure 13 · p. 10
• Adaptive negatives preserve similarity to the positive category while reducing similarity to newly added negative constraints during optimization.A new negative constraint is added every 250 iterations based on BLIP-2 predictions.Source: §6.3 · p. 11, Figure 15 · p. 11

• Concept properties are not always preserved when learned concepts are used with different prompts, producing more generic outputs.The paper illustrates these failures in Figure 16.Source: §7 · p. 11, Figure 16 · p. 11
• Some classes, including airplanes and fish, yield concepts that are not sufficiently creative or do not respect the positive constraint.Source: §7 · p. 11
• Highly specific negatives generated by BLIP-2 can weaken optimization constraints and contribute to unsuccessful creative generation.The paper gives particular airplane models as an example of negatives that are too specific to serve as strong constraints.Source: §7 · p. 11

• ConceptLab uses Diffusion Prior models and positive and negative prior constraints to learn novel concepts within broad categories.Source: §8 · p. 11
• A question-answering VLM adaptively encourages uniqueness while keeping generated concepts distinct from existing category members.Source: §8 · p. 11
• Prior constraints also support mixing generated concepts into hybrids, extending the flexibility of creative generation.Source: §8 · p. 11