WithAnyone: Towards Controllable and ID Consistent Image Generation

• Wei Cheng leads the project, and the paper identifies corresponding authors.Source: §1 · p. 1

• Copy-paste artifacts arise when models rigidly reproduce references despite natural variation in pose, expression, makeup, and illumination, limiting controllability and practical usability.The paper names this failure mode copy-paste because the model copies the reference instead of flexibly synthesizing the identity.Source: §1 · p. 2, Figure 2 · p. 2
• Scarce paired images of each identity force many methods into single-person reconstruction training, where reference and target coincide and copying is promoted.Existing datasets generally lack diverse paired images spanning expressions, poses, hairstyles, and viewpoints.Source: §1 · p. 2
• MultiID-2M provides 500k group photos with 1–5 recognizable celebrities, hundreds of paired references per celebrity, and 1.5M additional unpaired group photos.Source: §1 · p. 2
• MultiID-Bench evaluates multi-identity generation beyond identity similarity by measuring copy-paste artifacts and the trade-off between fidelity and variation.Evaluation across 12 state-of-the-art customization models exposes a clear trade-off between ID similarity and copy-paste artifacts.Source: §1 · p. 2
• WithAnyone maintains state-of-the-art identity similarity while substantially reducing copy-paste artifacts, breaking the fidelity–artifact trade-off.The model uses paired training and an ID contrastive loss with a large negative pool.Source: §1 · p. 2
• WithAnyone is a FLUX-based identity customization model designed to generate high-fidelity multi-identity images while mitigating copy-paste artifacts and enhancing visual quality.Source: §1 · pp. 2–3

• Multi-identity preservation remains relatively underexplored, with prior methods targeting spatial control or identity fidelity through different strategies.Source: §2 · p. 3
• DynamicID [18] is excluded from the experiments because its code and pretrained models are unavailable.Source: §2 · p. 3
• General-purpose customization and editing models [30] [36] [37] [53] [54] [55] [56] [61] can synthesize multiple identities, but identity similarity is often compromised for generality.Source: §2 · p. 3
• Paired reference images remain scarce, so reconstruction dominates multi-identity training, while evaluation protocols are underdeveloped and can undermine reproducibility.Some protocols sample identities from CelebA [29], which the paper identifies as a reproducibility problem.Source: §2 · p. 3

• MultiID-2M begins with approximately 1M single-ID reference images across approximately 3k identities, averaging 400 images per identity through ArcFace [11] embedding clustering.Source: §3 · p. 3
• WithAnyone encodes each reference using a face-recognition network and a general image encoder, while restricting face embeddings to corresponding face regions.Source: Figure 4 · p. 4
• Training combines diffusion loss with ID contrastive and ground-truth-aligned ID losses for consistent and accurate identity supervision.Source: Figure 4 · p. 4
• Automated filtering and annotation use Recognize Anything [69], aesthetic scoring, OCR-based watermark and logo removal, and LLM-based caption generation [1].Source: §3 · p. 4
• The final corpus contains approximately 500k identified multi-ID images with matched references plus approximately 1.5M additional unidentified multi-ID images.Source: §3 · p. 4

• MultiID-Bench contains 435 test cases with 1–4 people, each pairing a ground-truth image, corresponding references, and a prompt, using rare identities absent from training.Source: §4 · p. 4
• SimGT measures similarity between the generated image and prompted ground truth, while SimRef measures similarity between the generated image and reference identity.Source: §4 · p. 4
• Using SimGT as the primary metric penalizes excessive copying when pose, expression, or occlusion should vary, while rewarding faithful realization of the prompted scene.Source: §4 · p. 4
• The Copy-Paste metric normalizes generated-image bias toward the reference versus ground truth; 1 denotes perfect copying and −1 denotes full agreement with ground truth.Source: §4 · p. 4
• The benchmark additionally reports identity blending, prompt fidelity through CLIP I/T, and aesthetics.Source: §4 · p. 4

• WithAnyone uses paired-reference supervision and tailored objectives to improve identity fidelity, suppress copy-paste artifacts, and control multi-identity composition.Source: §5 · p. 4, §5.2 · p. 6

Training Objectives

• Ground-truth-aligned ID loss applies supervision across all noise levels with negligible overhead and improves identity preservation through more accurate identity measurement.Source: §5.1 · p. 5
• The ID contrastive loss pulls generated images toward same-identity references and away from different identities, using thousands of negatives enabled by the labeled reference bank.Source: §5.1 · p. 5
• The overall objective combines identity and contrastive losses, with λID and λCL both set to 0.1 across the described training phases.Source: §5.1 · p. 5

Results

• Figure 5 shows that most models follow a face-similarity/copy-paste trade-off, whereas WithAnyone deviates from this curve toward higher similarity with lower copy-paste.Source: Figure 5 · p. 6

Training pipeline

• The four-phase pipeline progresses from fixed-prompt reconstruction and captioned reconstruction to paired tuning and quality tuning, shifting the objective toward controllable identity-preserving synthesis.Source: §5.2 · p. 6

Quantitative Evaluation

• WithAnyone achieves the highest face similarity to ground truth while maintaining markedly lower copy-paste than competing methods.Source: §6.1 · p. 8
• WithAnyone attains the highest score among ID-specific reference models on OmniContext, although general customization and editing models often perform better overall.Source: §6.1 · p. 8

Qualitative Comparison

• Qualitatively, WithAnyone preserves identity while flexibly changing smiles, head pose, and eye gaze, unlike several competing methods that copy or distort reference faces.Source: §6.2 · p. 8

Ablation Study

• Paired-data fine-tuning reduces copy-paste without diminishing ground-truth similarity, whereas training on FFHQ performs significantly worse than training on the curated dataset.Source: §6.3 · p. 8, Table 3 · p. 8
• GT-aligned ID loss lowers denoising error at low noise and produces higher-variance, more informative gradients at high noise, strengthening identity learning.Source: §6.3 · p. 8, Figure 7 · p. 8

User Study

• Ten participants ranked WithAnyone highest across identity similarity, copy-paste absence, prompt adherence, and aesthetics, while the copy-paste metric moderately correlates with human judgments.Source: §6.3 · pp. 8–9, Figure 8 · p. 9

• WithAnyone reduces copy-paste artifacts while maintaining or improving identity similarity, breaking the reported fidelity-versus-copying trade-off.Source: §7 · p. 9
• MultiID-Bench and WithAnyone support standardized assessment and high-fidelity multi-identity generation with reduced copy-paste artifacts.Source: §7 · p. 9, A. Family of WithAnyone · p. 13
• WithAnyone integrates into Kontext for face editing by injecting identity information from reference images into target images.Source: A. Family of WithAnyone · p. 13

• Single-ID references are collected using celebrity-name searches, ArcFace face filtering, and DBSCAN clustering, with human review confirming the resulting identity bank.Source: §B.1 · p. 13
• Multi-ID candidates are retrieved through group-photo queries combining celebrity names, person-count keywords, scene descriptors, and negative keywords, yielding over 20 million images.Source: §B.1 · p. 13
• Faces in group photos are assigned identities by comparing ArcFace embeddings with single-ID cluster centers and applying a similarity threshold of 0.5.Source: §B.1 · p. 13
• Filtering uses recognition, aesthetic, collage, OCR, watermark, and logo checks, after which captions are generated with a large language model.Source: §B.1 · pp. 13–14
• The pipeline produces 1M single-ID reference images, 500k paired multi-ID images with identified persons, and 1M unpaired multi-ID images.Source: §B.1 · p. 14
• Dataset statistics indicate diverse nationalities and captions, with a long-tail distribution in identity appearances.Source: §B.2 · p. 14, Figure 10 · p. 14

• MultiID-Bench standardizes evaluation along identity fidelity and generation quality, including unintended identity blending, semantic faithfulness, and aesthetics.Source: C. Benchmark and Metrics Details · p. 14
• The benchmark contains 435 long-tail test cases with no training-identity overlap, each pairing 1–4 reference identities, a ground-truth image, and a descriptive prompt.Source: C. Benchmark and Metrics Details · p. 14
• Identity fidelity uses diagonal similarity averages, while identity blending uses off-diagonal averages; lower blending indicates less unintended identity mixing.Source: C. Benchmark and Metrics Details · pp. 14–15
• Generation quality is measured with CLIP-I, CLIP-T, and an aesthetic score assessing image-ground-truth similarity, prompt alignment, and visual appeal.Source: C. Benchmark and Metrics Details · p. 15
• WithAnyone injects ArcFace-derived identity tokens through cross-attention, optionally adds SigLIP for mid-level attribute retention, and uses masks for face localization.Source: E. Model Framework Details · p. 15
• Training uses pre-training, paired tuning with half paired and half reconstruction samples, and a final quality/style-tuning stage.Source: §F.1 · p. 19
• The extended contrastive loss treats the target as positive and draws additional negatives from the reference bank beyond in-batch identities.Source: §F.1 · p. 19

• GPT’s unusually high identity-similarity scores, including an identity absent from the references, support the metric’s ability to expose copy-and-paste replication.The 3-and-4-people subset largely contains television-series images that GPT may have encountered during pre-training.Source: §F.2 · p. 19
• A complete reference-image copy reaches SimGT = 0.521, SimRef = 1.0, and copy-paste = 0.999, matching the metric’s theoretical copy-paste limit.Source: §F.2 · p. 19
• WithAnyone achieves state-of-the-art CLIP-I and CLIP-T performance and ranks highest in the user study.Prompt following is measured by CLIP-I and CLIP-T in the benchmark and by VLM evaluation in OmniContext.Source: §F.2 · p. 19

SigLIP signal

• Increasing the SigLIP signal weight amplifies copy-paste while increasing similarity to the reference images by transferring finer appearance cues.The retained cues include hairstyle and accessories.Source: G. Ablation Study Details · p. 19, Figure 16 · p. 20

Training strategy and dataset construction

• Paired-data fine-tuning reduces copy-paste without compromising ground-truth similarity, whereas reconstruction training on FFHQ performs poorly across all metrics.FFHQ contains only 70k face-only portrait images, suggesting limited diversity and size as likely reasons for its weaker performance.Source: G. Ablation Study Details · p. 19

GT-aligned ID-Loss

• GT-aligned ID-Loss lowers denoising error at low noise and produces higher-variance, more informative gradients at high noise to strengthen identity learning.Source: G. Ablation Study Details · pp. 19–20

User study

• The user study ranks generated images on identity similarity, copy-and-paste effect, prompt following, and aesthetics using trained participants and five generated results per trial.The study samples single-person, two-person, and all three-and-four-person cases, with 10 participants recruited.Source: H. User Study Details · p. 20, §I.1 · p. 21

Metric validation

• The proposed copy-paste metric shows a moderate positive correlation with human ratings of copy-paste effect.Table 5 reports Pearson’s r, Spearman’s ρ, and Kendall’s τ with corresponding p-values.Source: §H.1 · p. 20, Table 5 · p. 20