Cones 2: Customizable Image Synthesis with Multiple Subjects

1 Introduction

• Multi-subject customization remains less explored than single-subject customization, although it better aligns with practical demands.Existing customized-generation techniques fine-tune pretrained models using a few subject-specific images.Source: §1 · p. 1
• Jointly training for subject combinations can require separate models as combinations grow and can cause subjects to disappear or inherit one another’s attributes.These problems are especially evident for semantically similar subjects.Source: §1 · p. 1
• Cones 2 fine-tunes a text encoder per subject and derives a residual token embedding that shifts the raw category toward the customized subject.A text-embedding-preservation loss limits changes to the subject-token embedding, allowing individually represented subjects to be composed without retraining.Source: §1 · p. 2
• Layout guidance uses subject bounding boxes to direct subjects to distinct image locations by strengthening target-subject and weakening irrelevant-subject cross-attention activations.The layout is an abstract, easy-to-obtain prior that users can specify in advance.Source: §1 · p. 2
• Cones 2 achieves competitive text alignment and visual similarity across varied multi-subject settings, including customization with six subjects.The evaluation covers categories such as pets, scenes, and decorations.Source: §1 · p. 2

Large-scale text-conditioned image synthesis

• Text-conditioned synthesis has advanced from domain-specific GAN approaches to open-vocabulary diffusion and autoregressive models, with newer text-to-image GANs offering faster sampling.Examples include GLIDE [3], DALLE2 [4], StableDiffusion, Imagen [6], DALLE, Make-A-Scene, CogView, Parti [20], StyleGAN-T [7], GALIP [21], and GigaGAN [8].Source: §2 · pp. 2–3

Customized image generation

• Few-shot customization includes learning an identifier word in Text Inversion [12], fine-tuning the whole model in DreamBooth [9], joint training combinations in Custom [10], and activating a small neuron cluster in Cones [11].Source: §2 · p. 3

Spatial guidance in diffusion models

• Spatial-guidance approaches add spatial conditions during training, train adapters, edit cross-attention, or constrain attention maps; Cones 2 uses layout to separate subject locations.The related work identifies Composer [22], ControlNet [23], T2I-Adapters [24], Prompt-to-prompt [25], and diffusion-based image translation [26].Source: §2 · p. 3

3 Method

• The method combines subject-specific residual token embeddings with a pretrained diffusion model and uses layout guidance to compose subjects.The framework targets new images containing any combination of the supplied subjects.Source: §3 · p. 3

3.1 Text-conditioned diffusion model

• The pretrained text encoder conditions the diffusion model through cross-attention, and inference iteratively denoises Gaussian noise.The conditional diffusion model is trained with a reconstruction objective for its reverse process.Source: §3.1 · pp. 3–4

3.2 Representing subjects with residual token embedding

• Residual subject embeddings are learned with a preservation loss that keeps non-subject token embeddings close to the original encoder’s outputs.For the dog example, the method samples 1,000 sentences containing “dog” and averages the customized-versus-original embedding shift to obtain the residual.Source: §3.2 · pp. 4–5
• Precomputed residuals can be added to subject-token embeddings for arbitrary combinations without further tuning, with token-dimension operations supporting efficient storage and scaling.The paper says each subject requires only a float32 vector rather than the large parameters used by previous methods.Source: §3.2 · p. 5

3.3 Composing subjects with layout guidance

• Bounding-box layouts guide cross-attention edits: target-subject activations are strengthened in desired regions and irrelevant-subject activations weakened elsewhere during inference.The edited attention outputs are used at every generation step, with η(t) controlling edit intensity over time.Source: §3.3 · pp. 5–6

4.1 Experimental setups

• The setup evaluates 15 subjects across scenes, pets, and objects, using image similarity and prompt-level CLIP textual similarity against three baselines.The metrics include mean image similarity to each target subject and average CLIP similarity between generated images and prompts.Source: §4.1 · p. 6

4.2 Main results

• As subject count increases, Cones 2 outperforms other methods across four evaluation metrics and produces visually accurate images for all subjects without retraining combinations.Table 1's complexity metric counts fine-tuning iterations needed to generate a combination of n subjects.Source: §4.2 · pp. 6–7, Table 1 · p. 8
• Users most preferred Cones 2 for both image and text alignment in multi-subject customization.The user study generated 1,280 images and asked participants to judge customized-subject presence and prompt content.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 Towards challenging cases

• For highly similar subjects, including two customized dogs, Cones exhibits attribute confusion, while Cones 2 performs well in visual and textual similarity.The comparison uses Cones, described as one of the stronger multi-subject baselines.Source: §4.3 · p. 8, Figure 4 · p. 9
• As subject count increases, other methods' image quality decreases significantly, while Cones 2 is less affected and is illustrated with five- and six-subject generations.The paper presents diverse generation examples for both five and six customized subjects.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 Ablation studies

• Strengthening target regions alone can confuse subject attributes, whereas weakening irrelevant regions alone can omit subjects or their attributes.For “mug + teapot,” 30 guidance steps suffice; the more challenging “cat + dog” combination requires 50 steps for better attribute binding.Source: §4.4 · p. 8, Figure 6 · p. 10
• Adding Attend-and-Excite improves Cones, but combining Cones 2's residual token embedding with its layout guidance performs best and addresses attribute confusion.The comparison reports that joint-training baselines do not resolve attribute confusion simply by adding a guidance method.Source: §4.4 · p. 9, Figure 7 · p. 10

• The method combines subject-specific residual token embeddings with a pretrained diffusion model and uses layout priors to guide generation.Source: §5 · p. 10
• Individually learned subject-specific residual token embeddings can be combined for multi-subject customization without retraining.Source: §5 · p. 10
• Qualitative and quantitative experiments report superiority over state-of-the-art methods across varied settings, including six subjects and semantically similar subjects.The paper characterizes these results as evidence of effectiveness and robustness.Source: §5 · p. 10

• All methods use Stable Diffusion v2-1-base1 and 50 DDIM steps with a scale of 7.5; experiments run on one A-100 GPU.Source: A Experimental details · p. 12
• Textual Inversion trains each subject token for 3,000 steps, while DreamBooth, Custom Diffusion, and Cones scale their stated training steps with the number of subjects.Textual Inversion uses batch size 4 and learning rate 0.002; the other methods' batch sizes are 2.Source: A Experimental details · p. 12
• Our method trains each subject-specific residual token embedding for 3,000 steps and uses user-appointed bounding boxes to guide all 50 inference steps.The target-subject signal is strengthened by +2.5 and irrelevant-subject signals are weakened by −1 × 10−5.Source: A Experimental details · p. 13
• The user study scores generated images for image and text alignment using questions assigned to 25 users.For each task and method, each user scores 480 questions.Source: A Experimental details · p. 13

• Our approach is reported to outperform three baselines for semantically similar subjects and tasks involving three or more subjects, while single-subject performance is similar.The cited comparison describes cases such as a dog and a cat.Source: B More comparisons · p. 13
• Table A1 compares learning a residual token embedding with learning a token embedding directly.The supplied table caption identifies the comparison but gives no outcome values.Source: Table A1 · p. 14
• Figure A2 provides additional generated results described as further evidence of the method's robustness.Source: B More comparisons · p. 14

• When customized subjects reach four, other methods' performance significantly deteriorates, while our approach can generate a larger number of subjects.Source: §B.1 · p. 14

C Importance of residual token embedding

• The approach updates the text encoder and averages class-to-subject embedding shifts across text templates.The comparison concerns residuals computed using a certain number of text templates.Source: C Importance of residual token embedding · p. 14
• The residual approach outperforms directly updating class-embedding parameters in a single text embedding.Source: C Importance of residual token embedding · p. 14
• The comparison reports higher textual and visual similarity for the residual approach.Source: C Importance of residual token embedding · p. 14

C.1 Generated results of textual inversion

• Textual Inversion [12] struggles to generate complex single subjects.Source: §C.1 · p. 14
• Textual Inversion [12] also struggles with generation involving multiple subjects.Source: §C.1 · p. 14
• The reported difficulty spans both complex single-subject and multi-subject generation.Source: §C.1 · p. 14

D Social impact and limitations

• Linearly combining personalized single-subject models is presented as a way to generate multi-subject images with computation and storage advantages.Source: D Social impact and limitations · p. 14
• Beyond six subjects, the approach may not consistently produce satisfactory results.Source: D Social impact and limitations · p. 14
• The user-provided layout needs to be roughly consistent with the textual description for the desired generation results.Source: D Social impact and limitations · p. 14
• The appendix includes additional results for the approach, multi-subject generation, and Textual Inversion [12].Source: Figure A2 · p. 15, Figure A3 · p. 16, Figure A4 · p. 17