Source-linked AI summary
MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization
Chenyang Zhu, Hongxiang Li, Xiu Li, Long Chen
TL;DR
Rare-token concept customization is unstable and does not encode inherent target knowledge, motivating a task that binds natural-language knowledge to visual concepts. MoKus uses cross-modal knowledge transfer through anchor learning and textual knowledge updating, and the paper reports strong evaluations, benchmark improvements, application extensions, and world-knowledge gains.
Problem
Rare tokens can yield unstable generation and fail to convey the target concept’s inherent knowledge, motivating knowledge-aware concept customization.
Method
MoKus learns an anchor representation for the target concept’s visual appearance and updates knowledge-query answers in the LLM text encoder to transfer knowledge to visual generation.
Results
MoKus demonstrates effectiveness in qualitative and quantitative comparisons, supports virtual concept creation and concept erasure, and improves performance on world-knowledge benchmarks.
Takeaways & Limitations
KnowCusBench enables systematic evaluation of knowledge-aware concept customization through concept images, textual knowledge, and diverse generation prompts.
Takeaways & Limitations
Future work includes extending the task to video, improving evaluation metrics, and developing end-to-end methods.
Abstract
from arXiv · showhide
Concept customization typically binds rare tokens to a target concept. Unfortunately, these approaches often suffer from unstable performance as the pretraining data seldom contains these rare tokens. Meanwhile, these rare tokens fail to convey the inherent knowledge of the target concept. Consequently, we introduce Knowledge-aware Concept Customization, a novel task aiming at binding diverse textual knowledge to target visual concepts. This task requires the model to identify the knowledge within the text prompt to perform high-fidelity customized generation. Meanwhile, the model should efficiently bind all the textual knowledge to the target concept. Therefore, we propose MoKus, a novel framework for knowledge-aware concept customization. Our framework relies on a key observation: cross-modal knowledge transfer, where modifying knowledge within the text modality naturally transfers to the visual modality during generation. Inspired by this observation, MoKus contains two stages: (1) In visual concept learning, we first learn the anchor representation to store the visual information of the target concept. (2) In textual knowledge updating, we update the answer for the knowledge queries to the anchor representation, enabling high-fidelity customized generation. To further comprehensively evaluate our proposed MoKus on the new task, we introduce the first benchmark for knowledge-aware concept customization: KnowCusBench. Extensive evaluations have demonstrated that MoKus outperforms state-of-the-art methods. Moreover, the cross-model knowledge transfer allows MoKus to be easily extended to other knowledge-aware applications like virtual concept creation and concept erasure. We also demonstrate the capability of our method to achieve improvements on world knowledge benchmarks.
1 Introduction
The paper identifies instability and missing concept knowledge in rare-token customization, then introduces knowledge-aware concept customization and MoKus to bind textual knowledge to visual concepts. It also presents KnowCusBench and reports strong evaluations, efficient updates, extensions, and world-knowledge improvements.
- Motivation: Rare-token methods can reconstruct reference concepts but produce unstable generations with other prompts and ignore inherent concept knowledge.Rare tokens lack semantic meaning, seldom occur in pretraining data, and are designed independently of knowledge.
- New Task: Knowledge-aware concept customization requires identifying prompt knowledge, integrating it coherently, and binding one or multiple knowledge pieces to a target concept.The task extends concept customization to natural-language knowledge and supports customized content creation.
- MoKus: MoKus transfers textual knowledge to visual generation by updating knowledge-query answers in an LLM text representation of the target concept.The framework uses an LLM text encoder and DiT backbone, with cross-modal transfer as its central observation.
- MoKus: MoKus first learns an anchor representation for visual concept information, then performs textual knowledge updating to bind knowledge to that anchor.The anchor mediates between the target concept and knowledge, while updated knowledge generalizes with other textual inputs.
- Evaluation: KnowCusBench is introduced as the first benchmark for knowledge-aware concept customization, combining concept images, six knowledge perspectives, and four prompt perspectives.The benchmark contains 5,975 images for comprehensive evaluation.
- Results: Qualitative and quantitative comparisons demonstrate MoKus’s effectiveness and superiority, with extensions to virtual concept creation, concept erasure, and world-knowledge benchmarks.The paper reports improvements on benchmarks such as WISE.
2 Related Work
The related work distinguishes knowledge-aware concept customization from established concept-customization and knowledge-editing approaches. It highlights different trade-offs among memory-based, locate-then-edit, and metalearning methods.
- Concept Customization: Concept customization has addressed objects, styles, faces, multi-object composition, and concept swapping using user-provided references.The paper differs by integrating knowledge into customization.
- Knowledge Editing: Memory-based knowledge editing retrieves external cases without changing parameters, but depends on retrieval quality and increases inference costs.
- Knowledge Editing: Locate-then-edit methods modify identified knowledge regions permanently and support batch operations, but edited knowledge can have poor transferability and locality.
- Knowledge Editing: Metalearning methods predict weight adjustments efficiently, but require additional training data and struggle with conflicts among multiple edits.
3 Observation: Cross-modal Knowledge Transfer
The paper studies cross-modal knowledge transfer by updating an LLM text encoder before image generation. Updating a factual answer changes the resulting visual output toward the updated knowledge.
- Motivation: The model can generate an incorrect portrait when prompted for an image involving complex knowledge, motivating proactive internal knowledge updates.The example asks for Ludwig van Beethoven’s favorite instrument.
- Solution: The method edits the LLM text encoder so the answer to a knowledge question changes before using the original question as the image prompt.The example updates Beethoven’s favorite instrument to “guitar.”
- Observation: Comparing generations before and after editing reveals cross-modal knowledge transfer from the text modality to the visual modality.
4 Method: MoKus
MoKus binds a target concept’s visual information to an anchor representation, then updates an LLM’s parameters so textual knowledge queries resolve to that anchor during generation.
- Visual Concept Learning: MoKus uses visual concept learning to bind target-concept information to an anchor representation through fine-tuning.Rare tokens serve as textual inputs, while trainable LoRA parameters are optimized using predicted and ground-truth velocities.
- Visual Concept Learning: The visual-learning pipeline extracts data latents from images, samples noise and diffusion timesteps, and feeds patchified latents with textual guidance to the MMDiT.The target velocity equals the difference between data and noise latents.
- Textual Knowledge Updating: Textual knowledge updating converts each knowledge item into a question paired with the shared anchor representation as its expected output.The LLM encoder supplies hidden states and gradients for computing an updating direction for each question.
- Textual Knowledge Updating: MoKus derives the parameter shift by solving a regularized least-squares problem that minimizes reconstruction error and update magnitude.The resulting closed-form shift is added directly to the pretrained parameters of editable layers.
- Textual Knowledge Updating: Textual knowledge updating supports high-fidelity generation, generalizes when combined with other prompts, and updates one knowledge item within seconds.This efficiency follows from directly applying the computed parameter shift to the editable layers.
- Benchmark: KnowCusBench evaluates knowledge-aware concept customization with concept images, diverse textual knowledge, and generation prompts spanning everyday concepts.The benchmark includes 35 concepts and knowledge generated from six perspectives, with prompts created from four perspectives.
5 Experiments
Experiments evaluate MoKus through reconstruction, customized generation, ablations, and extensions to virtual concepts, concept erasure, and world-knowledge generation. Across these settings, MoKus binds multiple knowledge pieces while preserving fidelity, prompt alignment, preference, and efficiency.
- Qualitative Comparison: MoKus binds several knowledge pieces to one concept and generates varied customized results.The method combines updated knowledge with other text prompts for high-fidelity generation.
- Qualitative Comparison: MoKus captures visual details for high-fidelity reconstruction and binds specific knowledge to generalize across new prompts.Compared with Naive-DB and Enc-FT, the method preserves visual fidelity while maintaining alignment with textual prompts.
- Quantitative Comparison: MoKus surpasses all baselines on CLIP-I-Seg for reconstruction and generation, while achieving the best prompt fidelity, human preference, and efficiency.It performs slightly worse than Naive-DB on reconstruction CLIP-I but leads on segmented concept fidelity.
- Ablation Studies: η = 1e −6 yields the best performance across all metrics and is used in the final model.Both qualitative visualization and quantitative evaluation support this scaling-factor choice.
- Ablation Studies: MoKus maintains robust performance as knowledge increases, with each additional knowledge item adding only about 7 seconds of training time.The ablation reports stable behavior and limited incremental training cost.
- Extensions: MoKus extends to virtual concept creation, concept erasure, multiple-knowledge combination, and improved world-knowledge benchmark performance.Textual knowledge updating creates usable virtual concepts, modifies generated attributes for erasure, combines complex knowledge, and improves all reported WISE metrics.
6 Conclusion
The paper introduces knowledge-aware concept customization and MoKus, which transfers textual knowledge into visual generation through an anchor representation. It also presents KnowCusBench and identifies video, evaluation, and end-to-end methods as future directions.
- Conclusion: Knowledge-aware concept customization represents a target concept with multiple pieces of knowledge and uses cross-modal transfer to influence visual output.Changes in the text modality are transferred to the visual modality during generation.
- Conclusion: MoKus binds visual appearance to an anchor representation, then projects each knowledge item to that anchor by modifying parameters in the LLM encoder.KnowCusBench is introduced as the first benchmark for this task.
- Future Work: Future work includes extending the task to video, improving evaluation metrics, and developing end-to-end knowledge-aware customization methods.These directions are stated as three areas for further exploration.