PaperlayerView exact revision on arXiv

Source-linked AI summary

UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens

Ruichuan An, Sihan Yang, Renrui Zhang, Zijun Shen, Ming Lu, Gaole Dai, Hao Liang, Ziyu Guo, Shilin Yan, Yulin Luo, Bocheng Zou, Chaoqun Yang, Wentao Zhang

arXiv:2505.14671v3cs.CV

TL;DR

Existing personalized models commonly separate understanding and generation, leaving attribute-reasoning generation insufficiently addressed. UniCTokens uses unified concept tokens, progressive three-stage training, and UnifyBench to integrate and evaluate both tasks. It achieves competitive understanding and generation performance and state-of-the-art personalized attribute-reasoning generation, while remaining constrained by base-model capabilities and domain shift.

  • Problem

    Existing methods use separate concept tokens for understanding and generation, and evaluations do not quantify how understanding facilitates attribute-reasoning generation.

  • Method

    UniCTokens trains unified concept tokens in a unified VLM using progressive stages that transfer information between understanding and generation, evaluated with UnifyBench.

  • Results

    UniCTokens achieves competitive performance in concept understanding and generation and state-of-the-art results in personalized attribute-reasoning generation.

  • Takeaways & Limitations

    The analysis indicates that better understanding enhances generation, while generation can provide information supporting understanding.

  • Takeaways & Limitations

    The approach inherits base-model limitations, including difficulty varying output styles and handling domain-shifted inputs such as specialized medical content.

Abstract

from arXiv · show

Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}.

1 Introduction

UniCTokens addresses the separation of personalized understanding and generation by using unified concept tokens and progressive cross-task training. UnifyBench evaluates these capabilities jointly, with competitive understanding and generation results and state-of-the-art attribute-reasoning generation.

  • Motivation: Existing methods treat personalized understanding and generation as independent tasks, limiting their use of complementary semantics and unified personalization.This separation can hinder generation involving attributes not explicitly described in the prompt.
  • Method: UniCTokens personalizes a unified VLM by training unified concept tokens instead of separate tokens for understanding and generation.The shared tokens are designed to support cross-task information transfer.
  • Method: Its progressive training strategy uses three stages to transfer information between personalized understanding and generation.The stages warm up understanding, bootstrap generation from understanding, and deepen understanding from generation.
  • Evaluation: UnifyBench evaluates concept understanding, concept generation, and personalized attribute-reasoning generation in unified personalized models.The benchmark measures whether understanding and generation capabilities can be assessed together.
  • Results: UniCTokens achieves competitive performance in concept understanding and generation, with state-of-the-art results in attribute-reasoning generation.The reported experiments compare UniCTokens with leading personalization methods on UnifyBench.

2 Related Work

Prior personalized understanding and generation methods use varied architectures and training strategies, but unified personalization remains limited by separate task training. UniCTokens instead trains shared concept tokens to improve information transfer between understanding and generation.

  • Existing personalization methods: Personalized generation methods recontextualize text conditions, while language and vision-language models personalize outputs through prompt tuning, weight modification, adapters, retrieval, or fine-tuning.These approaches integrate concept-related information into model outputs through different training strategies.
  • Unified personalization: Yo’Chameleon introduced unified personalized modeling, but its separate training strategy limits information sharing between understanding and generation.UniCTokens addresses this limitation by training unified concept tokens.

3 UniCTokens

UniCTokens personalizes a unified VLM with shared concept representations and progressively transfers information between understanding and generation. Its stages use understanding to bootstrap generation, then generation-derived visual details to strengthen understanding.

  • 3.2 Stage-1: Personalized Understanding Warm-up: The three-stage strategy warms up understanding, bootstraps generation from understanding, and deepens understanding from generation.Warm-up uses recognition, random, conversational, and text-only QA samples to optimize the initial tokens.
  • 3.1 Unified VLM Personalization with Unified Concept Tokens: The framework defines concept understanding, concept generation, and attribute-reasoning generation as three personalized capabilities.The tasks respectively produce textual responses, conditional images, and images integrating additional textual attributes.
  • 3.1 Unified VLM Personalization with Unified Concept Tokens: UniCTokens uses unified concept tokens rather than separate tokens for understanding and generation.The personalized prompt includes a unique concept identifier and shared tokens encoding concept-specific semantic attributes.
  • 3.3 Stage-2: Bootstrap Generative Learning from Understanding: Training solely on understanding does not directly equip the unified model to generate images containing the target concept.UniCTokens therefore initializes additional generation tokens while retaining priors from understanding; direct shared-token generation can diminish understanding performance.
  • 3.3 Stage-2: Bootstrap Generative Learning from Understanding: UniCTokens generates images that retain concept features with only 3 ∼10 training samples by leveraging understanding priors in shared tokens.The method transfers additional information into generation, supporting generation with limited personalized data.
  • 3.4 Stage-3: Deepen Understanding Representation from Generation: Generation-derived differences across timestamps identify relative hard regions containing fine-grained concept information for new understanding tokens.The method uses k-means to localize valuable visual tokens, then trains recognition VQA samples alongside continued T2I updates.

4 Experiment

The experiments evaluate UniCTokens on unified and separate personalized understanding and generation benchmarks, including attribute-reasoning generation and ablations of cross-task training. Results show strong performance across tasks and evidence of mutual benefits between understanding and generation.

  • Experiment Setup: UnifyBench contains 20 concepts spanning persons, pets, and objects, with 10–15 images per concept and extra attributes not visually inferable from training images.Its tests cover understanding, generation, and personalized attribute-reasoning queries.
  • UnifyBench: UniCTokens improves vanilla Show-o’s personalized concept understanding by an average of 8.9% while using fewer tokens.It also achieves decent performance against unified models with substantially more parameters.
  • UnifyBench: UniCTokens achieves state-of-the-art personalized generation among unified models across three evaluation metrics and produces realistic faces with conceptual features.Qualitative comparisons further assess controllable, personalized generation against Yo’Chameleon and GPT-4o.
  • Existing Benchmarks: UniCTokens outperforms Yo’LLaVA(1.3B) on all understanding tasks by an average of 5.13% using 1.3B parameters and fewer training images.The comparison uses separate personalized understanding and generation benchmarks, with multi-concept MC-LLaVA excluded because of unified-model limitations.
  • Ablation Study and Analysis: Improved understanding coincides with enhanced generation, while generation-as-perception yields more detailed sentences and greater attention to provided concepts than alternative initialization strategies.Full-image initialization instead causes attention dispersion, and generic initialization methods remain below the proposed generation-based approach.
  • Ablation Study and Analysis: The three-stage strategy achieves mutual enhancement: omitting Stage 1 significantly degrades personalized attribute-reasoning generation, while Stage 3 improves understanding after Stage 2’s generation-related decline.Stage 3 does not update generative parameters, so its influence on generation is subtle.

5 Conclusion

UniCTokens personalizes a unified VLM through unified concept tokens and a three-stage training strategy, enhancing personalized understanding and generation while preserving conversational capabilities.

  • UniCTokens trains unified concept tokens to personalize a unified VLM for understanding and generation.
  • The three-stage strategy improves personalized understanding and generation through cross-task information transfer.
  • The training setup uses standard autoregressive masked-language-modeling loss to compute answer likelihoods.
  • MMU training data in Stage 2 is essential for maintaining conversational abilities but has minimal impact on generative capabilities.

B Additional Experiment Setup

The additional experiments define evaluation metrics, compare supplementary personalization baselines, and examine unified understanding-generation models. They also assess catastrophic forgetting and position UniCTokens’ staged training as a way to transfer information between tasks.

  • Evaluation Metrics: Understanding is evaluated with recognition recall, VQA and QA BLEU scores, plus LLM-based alignment scores from 0 to 1.Recognition averages positive- and negative-sample recall; VQA and QA additionally use reference-answer alignment.
  • Baseline Comparisons: Supplementary baselines include caption-based LLaVA prompting, Yo’LLaVA, MC-LLaVA, RAP-MLLM, DreamBooth, and Text Inversion.These baselines cover personalized understanding, multi-concept personalization, database-based personalization, generative fine-tuning, and textual inversion.
  • Catastrophic Forgetting: Catastrophic forgetting is evaluated on GenEval, MMMU, and POPE by comparing the personalized model with the original Show-o.The evaluation follows Yo’Chameleon and examines whether newly acquired knowledge affects general multimodal capabilities.
  • Unified Understanding and Generation: Unified-model research seeks information transfer between understanding and generation, but prior approaches often rely on serial processing or extra generation modules.The cited discussion notes that MetaQuery emphasizes understanding aiding generation while neglecting the inverse direction.
  • Bridging Understanding and Generation: UniCTokens introduces a three-stage personalization strategy intended to enable mutual enhancement between understanding and generation.The strategy is presented as addressing the underexplored transfer from generation back into understanding.

E Additional Qualitative Results

Additional qualitative results provide more generated examples from UniCTokens.

  • Qualitative Results: Figure 9 presents additional qualitative images generated by UniCTokens.The figure extends the paper’s visual examination of generated outputs.

F Additional Quantitative Experiments

Additional experiments study token count, training cost, training schedules, and evaluation alignment. They report capacity gains with a trade-off at excessive token counts, favorable UniCTokens cost–capability, benefits from staged training, and consistent judge rankings.

  • Learnable Token Length: Increasing learnable tokens improves personalized understanding and generation, but the CLIP-T score declines when the parameter count becomes excessive, such as with 64 tokens.The reported improvement is attributed to greater learning capacity, while the 64-token setting illustrates that more parameters are not always beneficial.
  • Training Cost: 1.3×10^17 FLOPs and 25 minutes per concept give UniCTokens a favorable cost–capability trade-off among unified models.Generator-only DreamBooth is cheaper at 2×10^15 FLOPs and 7 minutes but is described as limited in scope; UniCTokens takes less time than Yo’Chameleon.
  • Training Scheme: The three-stage schedule consistently outperforms joint training in personalized understanding and generation, with the largest gains in personalized attribute-reasoning generation.The staged schedule is described as non-redundant because it warms up concept bindings, bootstraps generation, and feeds generation back into understanding.
  • Judge Models and Human Alignment: Gemini-2.5-Pro and human judges produce consistent method orderings across VQA, QA, and personalized attribute-reasoning generation.The human study contains 300 samples, and UniCTokens ranks highest where results are reported.

G Details of Dataset

The dataset draws animal and object images from three prior datasets, supplements individual images from Yo’LLaVA and online platforms, and validates all images and generated training data with humans.

  • Animal and object images come from MC-LLaVA, Yo’LLaVA, and MyVLM.
  • Images of individuals are sourced from Yo’LLaVA and various online platforms.
  • All images and generated training data undergo rigorous human validation.The dataset also presents images with extra textual information in Figure 11.

Methods VQA QA PARG

Table 7 reports LLM and human scores on the benchmark, including personalized attribute-reasoning generation.

  • Table 7 reports LLM and human scores, with PARG denoting Personalized Attribute-Reasoning Generation.

H Limitation and Discussion

The method remains constrained by the base model, struggles with domain-shift inputs, and still falls short of recommended facial-recognition similarity thresholds for personalized faces.

  • Limitations: The method inherits Show-o’s difficulty generating outputs in varying styles because of limitations in the underlying base model.
  • Limitations: Specialized domain-shift inputs, particularly in medicine, remain difficult for the model to handle effectively.
  • Limitations: Facial similarity around 0.3xx remains below the recommended 0.4–0.5 threshold for facial recognition.
Loading 2505.14671v3…