PureCC: Pure Learning for Text-to-Image Concept Customization

Zhichao Liao; Xiaole Xian; Qingyu Li; Wenyu Qin; Meng Wang; Weicheng Xie; Siyang Song; Pingfa Feng; Long Zeng; Liang Pan

Source-linked AI summary

PureCC: Pure Learning for Text-to-Image Concept Customization

Zhichao Liao, Xiaole Xian, Qingyu Li, Wenyu Qin, Meng Wang, Weicheng Xie, Siyang Song, Pingfa Feng, Long Zeng, Liang Pan

arXiv:2603.07561v2Mar 8, 2026cs.CV

TL;DR

Existing concept customization methods achieve high-fidelity personalization but can neglect preservation of the original model’s behavior and capabilities. PureCC decouples target-concept guidance from original conditional prediction through a dual-branch pipeline and adaptive guidance scale λ⋆. Extensive experiments report state-of-the-art preservation while maintaining high-fidelity customization.

Problem
Existing concept customization methods often overlook how learning personalized concepts affects the original model’s behavior and capabilities.
Method
PureCC combines decoupled implicit target-concept guidance with original conditional prediction using a frozen representation extractor, trainable flow model, and adaptive guidance scale λ⋆.
Results
PureCC achieves state-of-the-art preservation of original behavior and capabilities while enabling high-fidelity concept customization.
Takeaways & Limitations
PureCC provides pure personalized-concept learning while balancing customization fidelity with preservation of the original model.
Takeaways & Limitations
PureCC adds a training stage and model branch, increasing training time and GPU memory usage, although single-concept training takes 0.33 A100 hours and memory does not increase significantly in practice.

Abstract

from arXiv · show

Existing concept customization methods have achieved remarkable outcomes in high-fidelity and multi-concept customization. However, they often neglect the influence on the original model's behavior and capabilities when learning new personalized concepts. To address this issue, we propose PureCC. PureCC introduces a novel decoupled learning objective for concept customization, which combines the implicit guidance of the target concept with the original conditional prediction. This separated form enables PureCC to substantially focus on the original model during training. Moreover, based on this objective, PureCC designs a dual-branch training pipeline that includes a frozen extractor providing purified target concept representations as implicit guidance and a trainable flow model producing the original conditional prediction, jointly achieving pure learning for personalized concepts. Furthermore, PureCC introduces a novel adaptive guidance scale $λ^\star$ to dynamically adjust the guidance strength of the target concept, balancing customization fidelity and model preservation. Extensive experiments show that PureCC achieves state-of-the-art performance in preserving the original behavior and capabilities while enabling high-fidelity concept customization. The code is available at https://github.com/lzc-sg/PureCC.

1. Introduction

Concept customization adapts pretrained text-to-image models to learn personalized concepts from a few reference images, but existing approaches can disrupt unrelated original-model behavior. PureCC addresses this through decoupled guidance and dual-branch training that preserve the original model while learning the target concept.

Motivation: Concept customization enables personalized concepts in varied scenes from only 3–5 reference images.Applications include continuous content creation, artistic production, and advertising design.
Motivation: Existing methods adapt pretrained-model distributions toward user-specific concepts using full fine-tuning or parameter-efficient techniques such as LoRA.They associate the target concept with an identifier such as [V] and use prompt injection at inference.
Problem: Limited reference images make existing methods alter unrelated image elements, including background, style, and lighting, instead of only replacing the original concept.Figure 2 reports visualization and KL-divergence evidence of distribution drift when pretrained models are aligned to target distributions.
PureCC: PureCC uses a decoupled objective combining implicit target-concept guidance with the original conditional prediction to minimize changes to original behavior and capability.The formulation lets training focus on the personalized concept while explicitly considering the original model.
PureCC: PureCC’s dual-branch pipeline uses a frozen representation extractor for target-concept guidance and a trainable flow model for original conditional prediction.The broader framework builds on conditional flow matching, where a transformer regresses a conditional velocity field from interpolated samples.

4. Methodology

PureCC decouples target-concept guidance from original conditional prediction and trains them through a dual-branch flow pipeline. An adaptive scale λ⋆ adjusts guidance strength to balance concept fidelity with preservation of the original model.

Learning Objective in PureCC: PureCC defines the target velocity as original conditional prediction plus λ-scaled implicit target-concept guidance.This decoupled objective keeps the original model component explicit while learning the personalized concept.
Representation Extractor: The representation extractor is fine-tuned on the custom set, then frozen to provide target-concept representations during pure learning.The extractor uses a pre-trained flow model and layer-wise tunable concept embeddings to capture target-concept information.
Representation Extractor: Layer-wise tunable concept embeddings replace the [V] embedding at each transformer layer, forming complete text embeddings with base and target components.The embeddings are preserved for replacement during the subsequent learning stage.
Pure Learning Pipeline in PureCC: The dual-branch pipeline combines the frozen extractor’s target guidance with a trainable flow model that predicts the original conditional velocity from Base Text.The trainable model is initialized from another pre-trained flow model.
Adaptive Guidance Scale λ⋆: λ⋆ is obtained by minimizing projection error between the trainable model’s learned concept representation and the frozen model’s target representation.It decreases when the learned direction is inconsistent with guidance and increases when the target direction has been learned relatively well.
Overall Loss: The overall loss combines the PureCC loss with the concept-customization loss, enabling personalized-concept learning while minimizing impact on original behavior and capability.The flow matching component preserves the generative prior, and η controls regularization strength.

5. Experiments

Experiments evaluate PureCC across concept fidelity, original-model preservation, multi-concept customization, style–instance composition, and ablations. PureCC preserves original behavior while achieving high-fidelity and semantically coherent customization.

Experimental Setup: PureCC is evaluated on 14 DreamBooth concepts and 16 additional instance and style concepts, using SD 3.5-M with matched tuning-based baselines.The additional concepts include Pikachu, Yann LeCun, cartoon, and sketch; LoRA-based baselines use rank 4 and learning rate 1.0×10^-4.
Experimental Setup: Evaluation measures concept fidelity with CLIP-I (target), DINO, and CSD, while preservation uses differential text-alignment, quality, aesthetic, and segmentation metrics.Smaller ∆CLIP-T, ∆HPSv2.1, and ∆PickScore indicate better preservation; Seg-Cons measures spatial and structural consistency.
Qualitative Evaluation: PureCC preserves non-target background, lighting, and pose while avoiding the behavioral disruption and composition changes observed in DreamBooth and Mix-of-Show.In multi-concept settings, it also reduces semantic interference and preserves independence between learned concepts.
Qualitative Evaluation: PureCC balances style transfer with object-structure preservation, whereas competing tuning-based methods tend to overfit style or distort object identity.Each style–instance case combines an instance concept with a specific style.
Pure Learning Analysis: PureCC’s prediction evolves from the original prediction toward a target-aware prediction that preserves original content while expressing the personalized concept.The visualization supports additive integration rather than disruption of the model’s generative behavior.
Quantitative Evaluation: 69.37 Seg-Cons demonstrates preserved spatial and structural consistency, while PureCC achieves the smallest reported preservation gaps and competitive or superior concept-responsiveness scores.The results cover ∆CLIP-T(base), ∆HPSv2.1, ∆PickScore, CLIP-I(target), DINO, and CSD on DreamBenchPCC.
Ablation Study: Adding the PureCC loss improves all reported preservation metrics while maintaining strong CLIP-I(target) and DINO responsiveness; joint training instead weakens target-concept fidelity.A fixed λ also exposes a fidelity–preservation trade-off: λ = 1 yields CLIP-I 0.43 and CSD 0.26 because guidance is insufficient.

6. Conclusion

PureCC targets high-fidelity concept customization while preserving the original model’s behavior and capabilities. Its decoupled objective, dual-branch pipeline, and adaptive guidance scale support this balance.

Conclusion: PureCC combines a decoupled learning objective, dual-branch training pipeline, and adaptive guidance scale λ⋆ to preserve the original model during concept customization.The method is reported to outperform existing approaches in maintaining the original model while enabling concept customization.

8. Dataset Details

The dataset details combine established DreamBooth concepts with newly collected instance and style concepts to evaluate broader customization scenarios.

Dataset Details: The qualitative evaluation uses 14 personalized concepts from DreamBooth and 16 newly collected concepts spanning 11 instances and 5 styles.Examples include Pikachu and Yann LeCun for instances, and cartoon and sketch for styles.

9. More Implementation Details

Training uses an NVIDIA A100 with small batches and 400 optimization steps per personalized concept, followed by generation with 28 inference timesteps.

More Implementation Details: Training uses an NVIDIA A100 GPU, batch size 2, and 400 steps for both the representation extractor and flow model per personalized concept.Images are generated using the default inference setting of 28 timesteps.

10. Evaluation Metrics Details

Pure Concept Customization evaluates both personalized-concept fidelity and preservation of the original model’s behavior and capabilities. The evaluation uses fidelity metrics for instance and style concepts, differential preservation metrics, and qualitative datasets and results.

Fidelity of the personalized concept: Instance concepts are evaluated with CLIP-I (target) and DINO, while style concepts are evaluated with CSD for style consistency.These metrics compare generated concepts with reference images or assess consistency for style-level concepts.
Original model preservation: Original-model preservation is measured using differential metrics comparing custom-model outputs on Complete text with original-model outputs on Base text.The differential metric is ΔM = M_custom(I(y_complete)) − M_original(I(y_base)); smaller ΔM indicates better preservation.
Original model preservation: ΔCLIP-T(base), ΔHPSv2.1, and ΔPickScore assess base-text alignment, image quality, and aesthetic quality retention, respectively.Seg-Cons additionally measures segmentation consistency between custom and original model outputs under Complete text and Base text.
Qualitative evaluation: Qualitative evaluation uses personalized instance and style concepts, including DreamBooth samples, additionally collected samples, and DreamBenchPCC style samples.The evaluation materials include prompts containing personalized identifiers such as [V].
Qualitative evaluation: Additional figures report more qualitative evaluation results for personalized concept customization.The supplied evaluation materials include further qualitative result figures and example prompts for instance and style concepts.

12. Computational Cost

PureCC adds a training stage and model branch, increasing training time and GPU memory usage, but the reported training remains efficient and inference adds no overhead over baselines.

Training cost: PureCC’s additional training stage and dual-branch Pure Learning design inevitably increase training time and GPU memory usage.The extra branch is used during Stage-2 Pure Learning.
Training cost: 0.33 A100 hours are required to train one personalized concept with PureCC.The paper characterizes this training time as highly efficient despite the additional stage.
Memory cost: PureCC’s dual-branch design requires additional memory, but the overall GPU memory consumption does not increase significantly in practice.The implementation loads one main DiT network with the relevant LoRA modules.
Inference cost: During inference, PureCC uses only the single model vθ2_t(·), incurring no additional overhead over baselines.The computational-cost comparison is reported on a single NVIDIA A100 GPU.

13. Analysis of Hyperparameter η in LPCC

PureCC’s weighting parameter η controls the balance between pure learning and conventional concept customization. Excessive or insufficient values respectively harm concept fidelity or original-model preservation, motivating a balanced setting.

Sensitivity analysis: A large η over-injects the target concept and harms visual fidelity, whereas a small η lets LCC dominate and degrades original-model behavior and capabilities.The sensitivity is evaluated qualitatively in Fig. 15 and quantitatively in Tab. 6.
Original Conditional Prediction: Using the trainable model for Original Conditional Prediction closely matches using an additional frozen pretrained model, with virtually no loss in generation quality.PureCC adopts the trainable model’s prediction because it is more computationally efficient.
Ablation study: The ablation compares conventional LCC alone with the complete LPCC objective and compares jointly conducted training with PureCC’s two-stage training.These comparisons assess the learning objective and the separation of training stages.

16. User Study

A 42-participant user study compares PureCC with competing approaches across original behavior, base-text alignment, aesthetic preference, and target-attribute fidelity. The reported results favor PureCC for preserving original capabilities while maintaining comparable customization effects.

Study design: 42 participants completed pairwise comparisons, with each session lasting approximately 30 minutes.Participants came from diverse social backgrounds.
Study design: The study evaluated Original Behavior Preservation, Base Text Alignment, Aesthetic Preference, and Target Attribute Fidelity.These dimensions compare preservation, prompt alignment, visual quality, and resemblance to target concepts.
Results: PureCC significantly improves preservation of the original model’s behavior and capabilities compared with existing approaches.This outcome is reported from the user-study results in Table 7.
Results: PureCC achieves customization effects for target concepts comparable to those of existing methods focused on personalized-concept fidelity.The study therefore reports a combination of stronger preservation and comparable customization effects.