Source-linked AI summary
Personalized Generation In Large Model Era: A Survey
Yiyan Xu, Jinghao Zhang, Alireza Salemi, Xinting Hu, Wenjie Wang, Fuli Feng, Hamed Zamani, Xiangnan He, Tat-Seng Chua
TL;DR
PGen research is fragmented across modalities and communities, lacking a cross-community synthesis of its methods, data, and evaluation. This survey unifies PGen around personalized contexts, multimodal instructions, and a general workflow, then organizes the field through a multi-level taxonomy and reviews applications and open problems. It concludes that this unified view can support cross-modal knowledge sharing while the rapidly evolving field and lack of universally accepted benchmarks require continuing updates.
Problem
PGen research has developed independently across communities, with no dedicated cross-community survey integrating its modalities, methods, datasets, and evaluation perspectives.
Method
The survey formalizes PGen’s user-specific inputs, objectives, and workflow, then classifies research by modality, personalized context, task, datasets, and evaluation metrics.
Results
The survey provides a comprehensive cross-modal review of PGen’s technical advances, applications, evaluation resources, and open research challenges.
Takeaways & Limitations
A unified view of PGen can foster cross-modal knowledge sharing and collaboration toward more personalized digital experiences.
Takeaways & Limitations
The rapidly evolving field requires continual taxonomy updates, and universally accepted benchmarks across modalities remain an ongoing challenge.
Abstract
from arXiv · showhide
In the era of large models, content generation is gradually shifting to Personalized Generation (PGen), tailoring content to individual preferences and needs. This paper presents the first comprehensive survey on PGen, investigating existing research in this rapidly growing field. We conceptualize PGen from a unified perspective, systematically formalizing its key components, core objectives, and abstract workflows. Based on this unified perspective, we propose a multi-level taxonomy, offering an in-depth review of technical advancements, commonly used datasets, and evaluation metrics across multiple modalities, personalized contexts, and tasks. Moreover, we envision the potential applications of PGen and highlight open challenges and promising directions for future exploration. By bridging PGen research across multiple modalities, this survey serves as a valuable resource for fostering knowledge sharing and interdisciplinary collaboration, ultimately contributing to a more personalized digital landscape.
1 Introduction
Personalized Generation (PGen) shifts content generation toward individual preferences and needs, but research remains fragmented across modalities and communities. This survey addresses the gap with a unified perspective, taxonomy, application overview, and open-problem agenda.
- Motivation: PGen tailors generated content to individual preferences and needs across domains including e-commerce, marketing, and AI assistants.Large generative models have accelerated this shift from generic generation toward user-centric content.
- Research gap: Existing PGen research is fragmented across NLP, computer vision, and information retrieval, while prior surveys provide only model-centric or task-centric partial views.The paper identifies the absence of a survey offering a cross-community overview.
- Research gap: A unified, modality-agnostic framework is needed to organize distinct data structures, challenges, and technical approaches across modalities.The paper notes that modality differences make unification difficult but necessary for communication and knowledge sharing.
- Contributions: The survey formalizes PGen’s key components, core objectives, and general workflow from a unified user-centric perspective.It integrates studies across different modalities into a holistic framework.
- Contributions: It also proposes a multi-level taxonomy, reviews applications in content creation and delivery, and outlines open problems for future research.These contributions are intended to support knowledge sharing and collaboration across PGen communities.
2 A Unified User-centric Perspective for Personalized Generation
The survey defines PGen as generation guided by personalized contexts and multimodal instructions, with quality, instruction alignment, and personalization as shared objectives. Its workflow combines user modeling with modality-specific generative modeling techniques.
- Task formulation: PGen uses personalized contexts and multimodal instructions to guide generative models toward content matching individual preferences and specific needs.Inputs may include textual prompts, voice commands, and other modality-specific instructions.
- Objectives: Across modalities, PGen evaluates high quality, instruction alignment, and personalization as three consistent objectives.Quality includes coherence, relevance, and aesthetics; personalization concerns alignment with user preferences.
- Workflow: PGen workflows first model users from diverse user-specific data, then generate content across modalities.The generative stage is expected to produce high-quality, instruction-aligned, and personalized content.
- User modeling: User modeling commonly combines representation learning, prompt engineering, and retrieval-augmented generation to structure and enrich personalized information.These techniques encode or summarize inputs, organize user information for models, and retrieve relevant external data.
- Generative modeling: Generative modeling uses foundation models, guidance mechanisms, supervised fine-tuning, and preference-based optimization to integrate personalized signals.The surveyed workflow includes LLMs, MLLMs, and diffusion models, along with full fine-tuning, PEFT, and RLHF-based approaches.
- Workflow: Together, these techniques provide an adaptable and scalable workflow for diverse personalized contexts and user instructions.The survey presents this workflow as part of the evolving large-model landscape for PGen.
3 Personalized Generation Across Modalities
PGen research spans modalities, personalized contexts, and tasks, with text-generation work covering information seeking, recommendation, writing assistance, dialogue, and user simulation. The survey organizes these areas alongside their datasets, evaluation approaches, and emerging challenges.
- Taxonomy: The survey’s multi-level taxonomy categorizes PGen by modality, personalized context, task, datasets, and evaluation metrics.Modalities include text, image, video, audio, 3D, and cross-modal generation.
- Personalized text generation: Personalized text generation targets content tailored to user preferences and needs across information seeking, recommendation, writing assistance, dialogue, and user simulation.User interactions can provide implicit preferences and behavioral patterns over time.
- Information seeking: Information-seeking systems use user feedback to tailor generated responses, but generative personalization remains relatively underexplored because standardized metrics and benchmarks are limited.Recent work applies personalized feedback to train language models for tailored summaries.
- Recommendation: Generative recommendation systems use prompting or direct training, while diffusion models have also been explored for recommendation tasks.Recommendation primarily concerns personalized content delivery rather than content generation itself.
- Writing assistance: Writing-assistant benchmarks cover short-form and long-form generation, text completion, and stylistic imitation from user-written documents.Retrieval-augmented generation is described as the dominant approach for extracting writing preferences from historical documents.
- Dialogue and simulation: Personalized dialogue research uses personas, profiles, prompting, in-context learning, fine-tuning, and chain-of-thought reasoning to align responses with user preferences.Datasets include LiveChat, FoCus, and Pchatbot, while user simulation extracts preferences through profile-based agent interactions.
- Evaluation: Evaluation combines human judgment with reference-based and reference-free automatic approaches, including language-model assessment of profile alignment.The paper emphasizes that the target user is best positioned to judge whether generated content matches personal needs.
3.2 Personalized Image Generation
Personalized image generation uses user preferences, interactions, attributes, subjects, and identity-related inputs to tailor images across general and specialized tasks. Research spans preference inference, subject-driven generation, face generation, and virtual try-on, with similarity-based evaluation and growing privacy concerns.
- Overview: Personalized image generation synthesizes images reflecting individual preferences and requirements across general-purpose and specialized tasks.The surveyed tasks include general-purpose generation, fashion design, e-commerce product imagery, subject-driven generation, face generation, and virtual try-on.
- Preference-based generation: Historical interactions and real-time feedback provide visual preference signals for personalized image generation.Methods use historical engagements, interacted images, and multi-turn feedback to infer or progressively refine user preferences.
- Preference-based generation: User behaviors and attributes guide personalized fashion and e-commerce image generation through preference extraction, reward modeling, or user-group conditioning.Examples include preference vectors, LLM-extracted preferences, profile-based reward models, and group-level product-image generation.
- Personalized subjects: Subject-driven text-to-image methods learn representations from a few subject images while following instructions to preserve the subject across new contexts.Optimization-based methods encode subjects with learnable identifiers, while related work extends the setting to multiple subjects, structured prompts, and abstract concepts such as styles or relations.
- Personal face/body: Personal face and body generation creates identity-preserving portraits, avatars, and virtual try-on images under user-specific requirements.Face-generation methods preserve face identity while modifying expressions, actions, backgrounds, or styles; virtual try-on combines body and face images with specified garments.
- Personal face/body: Personalized face generation raises misuse and privacy risks, motivating unlearning, adversarial-attack, and watermark-based protection methods.These methods are investigated specifically to protect user privacy as personalized face-generation capabilities develop.
3.3 Personalized Video Generation
Personalized video generation extends personalization to subject-driven, identity-preserving, talking-head, pose-guided, and video try-on tasks. Methods adapt image-generation models with temporal, motion, identity, garment, or pose mechanisms, and evaluations combine similarity, quality, temporal, and human-assessment measures.
- Overview: Personalized video generation produces tailored videos reflecting individual preferences, traits, and specific needs.The section covers subject-driven text-to-video, identity-preserving generation, talking heads, pose-guided video, and video virtual try-on.
- Subject-driven T2V: Subject-driven text-to-video methods adapt personalized text-to-image models by adding motion and temporal-dynamics modules.Additional approaches use coarse-to-fine subject encoding, dynamic video sampling, motion control, and depth control.
- Face/body-conditioned video: Personal face and body inputs support identity-preserving video, talking-head, pose-guided, and video virtual try-on tasks.These tasks follow multimodal instructions while preserving identity, synchronizing speech, matching poses, or transferring garments through video.
- Evaluation: Video evaluations use LPIPS, SSIM, PSNR, CLIP-based measures, face-similarity metrics, and audio-visual synchronization metrics for personalization and instruction alignment.Face similarity commonly uses ArcFace or CurricularFace backbones, while audio-driven talking heads require synchronization assessment.
- Evaluation: Overall video quality is assessed with frame-level FID and KID, video-level VFID, FID-VID, FVD, and KVD, plus temporal-consistency measures.Many studies also use qualitative case studies and human assessments to capture personalization, instruction alignment, and overall video quality.
3.4 Personalized 3D Generation
Personalized 3D generation transforms visual or textual user contexts into customized 3D assets, most commonly from image-based subjects. The surveyed work covers image-to-3D, identity-preserving face and pose generation, and 3D virtual try-on.
- Overview: Personalized 3D generation transforms users’ visual or textual contexts, including body shapes, facial features, images, and prompts, into 3D assets.The common paradigm begins with image-based personalized subjects and generates corresponding 3D assets.
- Image-to-3D: Personalized image-to-3D methods generate assets that capture the geometry and appearance of supplied subjects.Approaches include cross-domain adaptation between 2D and 3D GANs and score-distillation-based optimization of portrait geometry and texture.
- Image-to-3D: Text prompts are incorporated into some 3D-generation methods to improve multi-view consistency, detail preservation, and alignment.Examples combine multi-view diffusion, DreamFusion and DreamBooth, cross-view attention, NeRF representations, and identity-aware optimization.
- Face and pose generation: Personal face and body images or monocular videos enable identity-preserving 3D asset generation.Tasks include 3D face generation and 3D human pose generation using pose, expression, mesh, or parameter-efficient adaptation mechanisms.
- 3D virtual try-on: 3D virtual try-on creates customized 3D models from minimal inputs such as user images, clothing images, and textual prompts.Applications include personalized facial modeling for eyeglass design and reconstruction of 3D clothing from 2D images.
3.5 Evaluation Metrics
Evaluation of personalized generation combines alignment measures, modality-specific quality metrics, and qualitative assessment. Across audio and 3D settings, metrics separately target personalization, perceptual quality, diversity, geometry, realism, and consistency.
- 3D evaluation: Personalized 3D generation commonly uses LPIPS, SSIM, PSNR, and CLIP scores for personalization and instruction alignment.Task-specific pretrained classifiers can additionally evaluate attributes such as facial characteristics.
- 3D evaluation: 3D geometric quality is evaluated with Chamfer Distance and related point-based measures, while user studies assess realism, texture photorealism, and shape-texture consistency.Objective and subjective evaluations therefore cover both geometric fidelity and perceptual qualities.
- Qualitative evaluation: Qualitative user studies remain important for evaluating subjective properties that objective metrics may not fully capture.In 3D generation, these properties include realism, texture photorealism, and shape-texture consistency.
- Audio evaluation: Personalized audio studies use CLAP scores, Pattern Similarity, and Embedding Distance to measure alignment with target musical styles or speaker characteristics.These measures are applied to tasks such as music transfer and text-to-audio generation.
- Audio evaluation: Speech audio quality is assessed with STOI, ESTOI, and PESQ, while FAD measures realism and diversity.These metrics target intelligibility, clarity, and distributional audio quality.
- Audio evaluation: Music generation evaluations use Precision, Recall, Density, and Coverage to capture creativity and output diversity.The metric set complements alignment measures by evaluating distributional breadth and creative coverage.
3.7 Personalized Cross-modal Generation
Personalized cross-modal generation uses multimodal user contexts and interactions to produce tailored responses, behaviors, captions, comments, and visual-question-answering outputs. Evaluation combines generation metrics, concept matching, task-performance measures, and human judgments.
- Cross-modal systems generate personalized textual responses from multimodal contexts such as images, videos, and historical robot trajectories.
- Robotics: Robotic systems infer user preferences from trajectories and feedback to customize decision-making and behaviors.Variational Preference Learning models diverse preferences through variational inference and RLHF, while Promptable Behaviors uses multi-objective reinforcement learning.
- Caption/Comment Generation: User-created captions and comments support personalized captioning and video-comment generation through targeted questions, historical text, personas, fine-tuning, and user identifiers.
- Cross-modal Dialogue Systems: Personalized visual question answering identifies user-specific subjects in images and infers user intent from accompanying queries.Memory-based methods store subject, context, and word-output memories for retrieval during generation.
- Evaluation Metrics: Text-generation evaluations use term and semantic matching, concept recall, precision, F1-score, and human judgments of emotion, style, and relevance.
- Evaluation Metrics: Personalized agents are evaluated with success rate, success weighted by path length (SPL), distance to goal, episode length, and human assessment.
4 Applications
PGen applications span content creation and content delivery. It helps creators preserve distinctive styles while streamlining workflows, and adapts delivered content to users across marketing, commerce, media, education, gaming, and assistants.
- PGen applications divide into content creation tools and personalized content delivery for end users.
- Content Creation: For creators, PGen analyzes prior content or work to generate tailored ideas, drafts, modifications, headlines, introductions, and branded content while preserving style.
- Content Creation: Individuals can use PGen for personalized email drafting, resume creation, travel planning, workout scheduling, and portrait generation.
- Content Delivery: Personalized delivery adapts generic multimodal content into formats suited to different audiences, helping users navigate information overload.
- Content Delivery: Representative delivery applications include targeted advertising, tailored retail experiences, personalized media, adaptive learning materials, dynamic games, and specialized AI assistants.Examples include virtual try-ons, personalized news and thumbnails, customized learning roadmaps, adaptive game difficulty, and user-specific guidance.
- Content Delivery: Gaming applications adapt storylines, tasks, difficulty levels, and characters to player preferences and behaviors.
- Content Delivery: Personalized AI assistants provide specialized support, including legal assistance, medical advice, and financial guidance.
5 Open Problems
PGen still faces challenges in scaling personalization, reasoning over user preferences, adapting to changing preferences, avoiding filter bubbles, integrating generation with retrieval, evaluating personalization, and ensuring trustworthiness.
- Scalability and Efficiency: Large generative models impose resource costs that limit PGen deployment in real-time, large-scale user scenarios.
- Deliberative Reasoning for PGen: Deliberative reasoning is proposed to analyze user preferences more thoroughly before generation, beyond multi-turn refinement.
- Evolving User Preference: PGen must track evolving user preferences from behavior and respond to preference shifts.
- Mitigating Filter Bubbles: PGen may reinforce existing preferences or beliefs and contribute to polarization, motivating diversity enhancement, user-controllable inference, and multi-agent perspectives.
- Synergy Between Generation and Retrieval: Combining PGen with retrieval-based systems may address cases where existing content does not fully meet users’ needs.
- Evaluation: Traditional metrics such as BLEU and CLIP-I do not fully capture alignment between generated content and user preferences.
- Trustworthiness: Trustworthy PGen requires balancing effective personalization with privacy protection and establishing transparent governance, moderation, and explainable generation.
6 Conclusion
The survey unifies PGen research across modalities through a holistic framework and multi-level taxonomy. It also reviews datasets, metrics, applications, and unresolved challenges to support cross-modal knowledge sharing and future development.
- The survey is presented as a comprehensive review of PGen advancements and trends across multiple modalities.
- Its holistic framework formalizes user-specific data, core objectives, and general workflows for PGen.
- The multi-level taxonomy categorizes methods by modality, user inputs, and specific tasks while summarizing datasets and evaluation metrics.
- The survey examines applications in content creation and delivery and identifies key research challenges that remain to be addressed.
- By unifying multimodal research, the survey supports cross-modal knowledge sharing and collaboration toward a more personalized digital landscape.
Limitations
The survey cannot fully encompass PGen research because the field is rapidly evolving. Its taxonomy therefore requires continuous updates, while universally accepted benchmarks remain an ongoing challenge.
- Rapidly emerging methods, datasets, and evaluation metrics make it difficult for the survey to encompass all PGen research.
- The survey’s taxonomy requires continuous updates as new methods, datasets, and evaluation metrics continue to emerge.
- Universally accepted benchmarks across different modalities remain an ongoing challenge.