PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Image Generation with Reasoning and Reflection

Bo Ni, Ngoc N. Tran, Qinwen Ge, Franck Dernoncourt, Seunghyun Yoon, Samyadeep Basu, Sungchul Kim, Puneet Mathur, Nedim Lipka, Tong Yu, Yu Wang, Ryan A. Rossi, Tyler Derr

arXiv:2610.00737v1cs.CVcs.AI

TL;DR

Personalized image generation has focused on reproducing curated visual concepts rather than inferring identity from users’ accumulated histories. The paper introduces PMH-IG and PEARL’s reason-reflect approach; across both tasks, PEARL outperforms strong personalization baselines.

  • Problem

    Personalized image generation has focused on reproducing visual concepts from a few reference images, while identity-aware generation requires inferring lifestyle and aesthetic preferences from richer histories.

    Source: §1 · p. 1

  • Method

    PMH-IG provides two history-grounded tasks, and PEARL uses a multimodal reasoner to plan, reflect on an initial rendering, and revise it for a frozen image generator.

    Source: §1 · p. 2

  • Results

    Across both PMH-IG tasks, PEARL substantially outperforms strong personalization baselines on image quality, retrieval-based personalization, and MLLM-judge metrics.

    Source: §6 · p. 10

  • Takeaways & Limitations

    PMH-IG evaluates identity inference from behavioral signals in Amazon reviews and aesthetic signals in Instagram posts, rather than reproduction of a single depicted concept.

    Source: §1 · p. 2, §2.1 · p. 3

    Training performs one planner update followed by one reflector update; systematic study of multi-round optimization remains future work.

    Source: §3.3 · p. 7

Abstract

from arXiv · show

Personalized image generation has remained narrowly focused on conditional synthesis from curated visual exemplars, rather than capturing who a user is. In practice, however, a user's personal context is much richer, comprising reviews, posts, images, captions, and metadata accumulated over time. A truly personalized generator should leverage this history to produce images aligned with the user's lifestyle and aesthetic preferences. To this end, we introduce the first unified benchmark for personalized image generation from user histories. The benchmark comprises two complementary tasks and a multi-axis evaluation protocol that assesses target fidelity, visual quality, user distinguishability, semantic alignment with the user's history, and task-specific utility. Grounded in real-world e-commerce and social media settings, the benchmark includes: (1) Personalized Scene Generation, which places a given object in a scene that reflects a user's preferences and lifestyle, motivated by personalized product presentation; and (2) Personalized Creative Generation, which generates a novel image on a specified topic that is faithful to a user's aesthetic and visual identity, motivated by social media content creation. We further propose PEARL, which couples a multimodal reasoner with a frozen image generator in an interleaved reason-reflect loop optimized with differential data reward. Across both tasks, PEARL outperforms strong baselines, achieving an average improvement of 15% across personalization metrics.

1 INTRODUCTION

The introduction argues that image personalization should infer identity from rich user histories, not merely reproduce concepts from curated exemplars. It introduces PMH-IG’s two tasks and PEARL’s reason-reflect framework, reporting gains over strong baselines.

1 INTRODUCTION

  • Prior personalized image generation focused on reproducing visual concepts from a few reference images, whereas real-world personalization requires aligning images with users’ lifestyles and aesthetic preferences (Wei et al., 2025; Ruiz et al., 2023).The introduction motivates this broader scope with personalized product backgrounds and identity-aligned social content.Source: §1 · pp. 1–2
  • PMH-IG evaluates Personalized Scene Generation from Amazon review histories and Personalized Creative Generation from Instagram post histories.The tasks ask models to ground a target product or topic in a user’s history and identity.Source: §1 · p. 2
  • PEARL plans from user history, reflects on an initial rendering to identify personalization mismatches, then re-renders from a corrected plan.Its reasoner is trained through two-stage cross-modal reflection tuning.Source: §1 · p. 2
  • PEARL substantially outperforms strong personalization baselines on both tasks, with ablations showing measurable gains from reasoning and reflection.Evaluation covers image generation quality, personalization, and MLLM-as-a-Judge probing lifestyle alignment.Source: §1 · p. 3

2 A BENCHMARK FOR PERSONALIZED IMAGE GENERATION FROM USER HISTORIES

PMH-IG benchmarks personalized image generation from realistic user histories through complementary scene-generation and creative-generation tasks. It distinguishes a task condition, which specifies what to generate, from user history, which guides how it is personalized.

2.1 PROBLEM DEFINITION

  • Personalized generation maps a user's multimodal history and task condition to an image that follows the condition and reflects the user's preferences, behaviors, and aesthetics; scene and creative tasks probe lifestyle and visual identity.Source: §2.1 · pp. 3–4

2.2 BENCHMARK DESIGN PRINCIPLES

  • PMH-IG uses public, naturally occurring user histories, separates what to generate from how to personalize it, and jointly evaluates condition fidelity, visual quality, and user-level personalization.Source: §2.2 · p. 4

2.3 PERSONALIZED SCENE GENERATION: E-COMMERCE PRODUCT PRESENTATION

  • Amazon review and product-image histories personalize the scene around a preserved target product, allowing the same product to appear in settings aligned with different users' interests and lifestyles.Source: §2.3 · p. 4

2.4 PERSONALIZED CREATIVE GENERATION: SOCIAL MEDIA CONTENT POSTING

  • Instagram image-caption histories guide visual identity while a topic specifies content; the same topic can yield different compositions, framing, colors, settings, and presentation styles.Source: §2.4 · p. 4

2.5 EVALUATION PROTOCOL

  • Shared evaluation measures reference-free aesthetic quality and MLLM-judged style, content, and overall user alignment, averaging Gemini Flash 2.5 and Qwen2.5-VL-32B scores.Source: §2.5 · pp. 4–5
  • Scene personalization uses Hit@5 and MRR retrieval, while creative generation adds held-out-post CLIP, DINO, LPIPS, and MS-SSIM plus inter- and intra-category user retrieval R@1.Source: §2.5 · p. 5

3 PEARL: PERSONALIZED IMAGE GENERATION WITH REASONING AND REFLECTION

PEARL separates history-based planning from render-aware reflection, then trains both policies with silver trajectories and downstream preference optimization.

3 PEARL: PERSONALIZED IMAGE GENERATION WITH REASONING AND REFLECTION

  • PEARL combines a planner that turns user history into a scene plan with a reflector that revises it against a rendered image; both policies are trained against the inference pipeline.The renderer remains frozen, and training uses a task-aligned retrieval reward.Source: §3 · p. 5

3.1 PLANNER: REASONING OVER USER HISTORY

  • The planner aggregates behavioral and aesthetic cues distributed across the user history, producing a reasoning trace and a self-contained scene prompt for the generation condition.A frozen image generator turns the prompt into the initial rendering.Source: §3.1 · p. 5

3.2 REFLECTOR: CROSS-MODAL REFINEMENT FROM RENDERED EVIDENCE

  • The reflector uses the user history and initial rendering to identify personalization mismatches, then emits a revised scene description for the final image.Mismatches can include scene incongruities, missing lifestyle cues, or aesthetic deviations.Source: §3.2 · pp. 5–6

3.3 TRAINING

  • In stage one, a multimodal teacher with access to the target identity signal supplies separate silver trajectories for the planner and reflector, which are independently fine-tuned.The reflector’s teacher input includes a pseudo-initial rendering produced from the planner’s target scene description.Source: §3.3 · p. 6
  • In stage two, the policies are alternately updated with preference optimization, scoring candidate outputs through the downstream rendering pipeline rather than training against a fixed teacher signal.For the planner, candidates are rendered through the reflector and generator and scored with a task-aligned retrieval reward.Source: §3.3 · p. 6
  • Training uses one planner update followed by one reflector update; systematic study of additional alternation is left to future work.Each update is scored through the composed pipeline used at inference.Source: §3.3 · p. 7

4 EXPERIMENTS

Experiments evaluate PEARL against personalization baselines across two PMH-IG tasks, with automatic, human, and qualitative assessment. PEARL leads on the reported task metrics, while ablations find gains from reflection, especially on fine-grained style and content judgments.

4 EXPERIMENTS

  • The experiments assess image quality, personalization, and MLLM judgments that probe lifestyle alignment across both benchmark tasks.The setup uses a held-out target image for scene generation and a held-out post for creative generation.Source: §4 · p. 7, §4.2 · p. 7

4.1 BASELINES

  • The comparison uses two history-based personalization methods and two MLLM baselines, all paired with frozen SDXL for a fair renderer match.PMG uses LLM-extracted style keywords, Pigeon masks user history, while LLaVA and LaVIT provide MLLM-based baselines.Source: §4.1 · p. 7

4.2 EXPERIMENT SETTINGS

  • The setup uses Qwen2.5-VL-7B planner and reflector modules with frozen SDXL, and trains them with LoRA and task-specific composite rewards.The reward combines recommendation rank, identity coverage, and Qwen judgment for scene generation, and CLIP and DINO similarities for creative generation.Source: §4.2 · p. 7

4.3 QUANTITATIVE RESULTS

  • On scene generation, PEARL outperforms baselines on all metrics, with largest margins on H@5 and MRR; on creative generation it leads 7 of 10 columns, including all retrieval and MLLM-judge metrics.The creative-generation results also report PEARL strictly best on CIS and DIS and competitive on the other two target-based image metrics.Source: Table 1 · p. 7, §4.3 · pp. 7–8, Table 2 · p. 8

4.4 HUMAN EVALUATION

  • In a preliminary single-annotator study, PEARL was selected for history fit in 19/25 scene pairs and 16/25 creative pairs.The authors caution that these judgments measure history fit, not target users’ own preferences.Source: §4.4 · p. 8

4.5 QUALITATIVE RESULTS

  • Figure 3 illustrates PEARL translating histories into scene choices, including DIY worktable context and clean kitchenware with a hand-off gesture for food.The authors report that LaVIT and Pigeon fail to preserve product detail and generate meaningful images in the DIY example.Source: §4.5 · p. 8, Figure 3 · p. 9

4.6 ABLATION STUDIES

  • In a cross-task ablation, PEARL beats PEARL-Reflection on four of five aggregated dimensions, with the largest gains on MLLM style and content judgments.PEARL-Reflection replaces alternating-policy DPO with supervised fine-tuning alone; the overall-score drop is under 1%.Source: §4.6 · pp. 8–9, Figure 4 · p. 9

5 RELATED WORK

Prior personalized image generation either reproduces curated visual concepts or conditions on accumulated user history. PEARL differs by explicitly reasoning over multimodal histories to infer implications for the target image.

Conditional Personalized Image Generation

  • Conditional personalized generation reproduces a specified concept from reference images, using per-user optimization or adapters that encode references into a frozen generator.DreamBooth (Ruiz et al., 2023) and Textual Inversion (Yang et al., 2023) require per-user training, while IP-Adapter amortizes that cost.Source: §5 · p. 9

History-Based Personalized Image Generation

  • History-based methods condition on accumulated interactions: Pigeon (Xu et al., 2025) encodes noisy histories, while PMG uses textual descriptions to represent preferences.Personalized Image Editing (Dunlop et al., 2025) uses Collaborative DPO over a preference graph to share aesthetic signals across users with similar tastes.Source: §5 · p. 9
  • Unlike these history-based methods, which fuse histories into generation without explicit inference, PEARL reasons over multimodal histories and their implications for the target image.The contrast is with approaches that treat history as a conditioning signal without explicitly reasoning over its implications.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents PMH-IG as a benchmark grounded in realistic multimodal histories and PEARL as a reasoning-interleaved framework. Across both tasks, PEARL outperforms strong personalization baselines on several evaluation axes, and ablations show measurable gains from reasoning and reflection.

6 CONCLUSION

  • PMH-IG is introduced as a benchmark grounded in realistic multimodal histories from public review and posting platforms.PEARL decomposes generation into identity reasoning over user history and conditioned synthesis through a frozen image generator.Source: §6 · p. 10
  • Across both PMH-IG tasks, PEARL substantially outperforms strong personalization baselines on image quality, retrieval-based personalization, and MLLM-judge metrics.Source: §6 · p. 10
  • Ablations confirm that both reasoning and reflection contribute measurable gains.Source: §6 · p. 10

7 AI USE DISCLOSURE

The authors disclose using generative AI for brainstorming, part of the codebase, preliminary analysis, drafting, figure work, and summarizing related research, with review and auditing of AI-assisted work.

7 AI USE DISCLOSURE

  • Generative AI was used to brainstorm a research idea, implement part of the codebase, and conduct preliminary experiment-result analysis.Source: §7 · p. 10
  • The authors report no use of generative AI for autoresearch, proposing hypotheses, or methodology feedback.Source: §7 · p. 10
  • AI assistance also contributed to paper drafting, teaser-figure and image creation or modification, and research-landscape summaries, which the authors reviewed and audited.They report manual literature checks for plagiarism and audits of LLM-generated claims and code.Source: §7 · p. 10

A LIMITATIONS

The section identifies limits in scene evaluation and optimization, then describes PMH-IG and its e-commerce scene-generation task. The benchmark separates the content condition from the user-history personalization signal.

A LIMITATIONS

  • Personalized Scene Generation lacks a natural ground-truth scene for each user-product pair, so alignment is assessed indirectly; PEARL training uses only one alternating-policy DPO round.The authors leave systematic study of multi-round optimization for future work.Source: A LIMITATIONS · p. 13

B PMH-IG

  • PMH-IG evaluates complementary e-commerce and social-media forms of personalization, with the user history supplying personalization and the product or topic specifying content.Scene generation adapts a product's surrounding context; creative generation extends a user's visual identity to new content.Source: B PMH-IG · p. 13

B.1 PERSONALIZED SCENE GENERATION: E-COMMERCE PRODUCT PRESENTATION

  • Scene generation targets user-level product presentation: retain the product while adapting its visual context to a user's inferred interests and lifestyle.Each target product comes from the user's own review history, and user-disjoint splits test generalization to unseen users.Source: §B.1 · pp. 13–14

B.2 PERSONALIZED CREATIVE GENERATION: SOCIAL MEDIA CONTENT POSTING

Personalized Creative Generation tests whether a model can infer a creator's visual regularities from history and apply them to a new topic. The supplied appendix passages also detail PEARL training and the evaluation probes and metrics.

B.2 PERSONALIZED CREATIVE GENERATION: SOCIAL MEDIA CONTENT POSTING

  • The task asks models to infer latent aesthetic regularities from a creator's visual history and apply them to new topics, producing images that could fit the user's feed.The Instagram dataset uses chronological histories: the two newest posts are targets and the preceding five form the visual history; user-level splits test unseen users.Source: §B.2 · pp. 14–15

C PEARL ALGORITHMS

  • PEARL's training distills teacher-generated silver trajectories before alternating planner and reflector updates with Direct Preference Optimization scored through the downstream pipeline.Each candidate is run through the other frozen policy and renderer, then ranked by task-aligned retrieval reward.Source: C PEARL ALGORITHMS · p. 15

D.1 CONTRASTIVE RETRIEVAL PROBES: SHARED SETUP

  • The shared contrastive retrieval probe trains and freezes a compatibility scorer on real history-target pairs, then ranks generated images against a deterministic distractor pool.Generated images are not part of probe training, avoiding circularity in evaluation.Source: §D.1 · pp. 16–17

D.2 RECOMMENDATION PROBE FOR PERSONALIZED SCENE GENERATION

  • The scene recommendation probe combines review text and product photos in its user encoder, then ranks the target among 99 distractor products.It reports Hit@5 and MRR over 518 test users.Source: §D.2 · p. 17

D.3 STYLE-DISCRIMINATION PROBE FOR PERSONALIZED CREATIVE GENERATION

  • The creative style probe represents users from recent post images and tests whether a generated image retrieves its user against distractors at different similarity difficulties.It reports R@1 at inter-category and intra-category difficulty levels over 314 test pairs.Source: §D.3 · pp. 17–18

D.4 OUTPUT QUALITY

  • Output quality is measured with the LAION aesthetic predictor, a reference-free score on an approximately 1–10 scale.Higher scores correspond to images judged more aesthetically pleasing on the LAION training distribution.Source: §D.4 · p. 18

D.5 MLLM-AS-JUDGE PROTOCOL

  • Two multimodal LLM judges score generated images against user history for style, content, and overall fit on 1–5 scales.The reported score for each dimension averages Gemini Flash 2.5 and Qwen2.5-VL-32B judgments.Source: §D.5 · p. 18

D.6 HUMAN EVALUATION

The human evaluation compares PEARL with task-specific baselines on history fit and task constraints, but its single-annotator sample and differing creative histories limit interpretation.

D.6 HUMAN EVALUATION

  • The evaluation comprises one annotator's 25 comparisons per task and does not establish inter-annotator reliability, population-level preference, or statistical significance.All 50 responses were included in the primary tabulation.Source: §D.6 · pp. 18–19
  • Creative comparisons used different history subsets for PEARL and Pigeon, so they do not isolate method effects under identical conditioning.The evaluation displayed the chronological union of the histories used by the two pipelines, excluding held-out posts.Source: §D.6 · p. 19
  • In the 25 creative comparisons, topic-adherence choices favored PEARL in 18 cases, Pigeon in five, and tied in two; history-fit choices favored PEARL in the sample.The table reports counts and percentages, with Pigeon as the creative-generation baseline.Source: §D.6 · p. 19, Table 4 · p. 19
  • One scene account was flagged for possible target-review leakage, and its original inputs are unavailable, leaving contamination of the original outputs unresolved.Excluding that account in a sensitivity check still yielded 18 of 24 history-fit choices for PEARL.Source: §D.6 · p. 19

E.1 ABLATION STUDIES

Across both tasks, PEARL outperforms its supervised-only variant on MLLM-judge metrics, with the largest margins in style and content. The ablations associate the reflector's gains with fine-grained personalization.

E.1 ABLATION STUDIES

  • The ablation pattern indicates that alternating-policy DPO mainly contributes fine-grained personalization details identified and corrected by the reflector from rendered evidence.The planner alone can already produce coarse scene-layout or subject-matter changes.Source: §E.1 · p. 19
  • Retrieval and image-quality differences are smaller and less consistent, while the overall MLLM judge score drops by under 1% between variants.The authors relate the pattern to IRG's finding that interleaved reasoning yields its largest gains on fine-grained fidelity rather than coarse semantic alignment [Huang et al., 2025].Source: §E.1 · pp. 19–20
  • Across both tasks, PEARL consistently outperforms PEARL-SFT on MLLM judge metrics, with the largest margins on style and content.The scene and creative ablation results are reported in Tables 5 and 6, respectively.Source: §E.1 · p. 19, Table 5 · p. 20, Table 6 · p. 20

E.2 QUALITATIVE RESULTS

Figure 5 compares generations with three users’ histories, showing recurring choices in color, framing, and scene presentation across interiors, food, and quilting. The applications extend the user-history-to-image formulation to product presentation, staging, content creation, recommendation, and identity analysis.

E.2 QUALITATIVE RESULTS

  • Across three users, PEARL’s examples retain history-linked choices in interiors, food, and quilting, while other systems vary in palette, composition, or subject emphasis.For User 1, PEARL keeps a light palette and fireplace; for User 2, it uses a photographic salad presentation; for User 3, it depicts a quilt in a domestic setting.Source: §E.2 · p. 20

H.1 PERSONALIZED ADVERTISING AND PRODUCT PRESENTATION

  • The general user-history-to-image formulation supports personalized product presentation, where reviews guide lifestyle-matched scenes and retrieval metrics offer an offline proxy for engagement.The passage contrasts, for example, a farmhouse countertop for a homesteading user with a minimalist apartment for a compact-living user.Source: H APPLICATIONS AND USE CASES · p. 24, §H.1 · pp. 25–26

H.2 PERSONALIZED VIRTUAL TRY-ON AND LIFESTYLE STAGING

  • Personalized Scene Generation reframes virtual try-on as inferring staging context from user history rather than compositing a garment onto a fixed reference body.Its retrieval and identity-element metrics assess whether the staged context suits the target user, beyond garment fidelity alone.Source: §H.2 · p. 26

H.3 OBJECT-TO-SCENE GENERATION AS INVERSE IMAGE EDITING

  • Personalized Scene Generation reverses conventional editing: it builds a complete user-appropriate scene around an isolated object, requiring global layout, coherent backgrounds, and contextual plausibility.The benchmark evaluates outward scene construction around an object rather than inward modification of an existing scene.Source: §H.3 · p. 26

H.4 CONTENT CREATOR TOOLING

  • Personalized Creative Generation can test whether generated posts extend a creator’s established aesthetic to novel topics, with contrastive retrieval providing a style-fidelity signal for iteration.The passage presents this as a way to quantify style fidelity without costly human evaluation.Source: §H.4 · p. 26

H.5 PERSONALIZED STORYBOARD AND NARRATIVE ILLUSTRATION

  • For narrative illustration, topic supplies the prompt and user history the style; intra-category retrieval tests attribution against confounders sharing the topic but not the visual style.The passage identifies style consistency across generated illustrations as important to coherent visual storytelling.Source: §H.5 · p. 26

H.6 USER IDENTITY MODELING AND UNDERSTANDING

  • Identity element F1 can serve as a standalone, interpretable diagnostic for user modeling by counting which concrete identity cues a generated image captures or misses.The metric extracts recurring objects, brands, settings, and stylistic cues from user history.Source: §H.6 · p. 26

H.7 PERSONALIZED RECOMMENDATION VIA GENERATIVE RETRIEVAL

  • The benchmark’s retrieval-based protocol supports generative recommendation by testing whether generated images carry enough user-specific signal to identify the correct user or product.It complements rating- or click-based evaluation with a visual-fidelity axis.Source: §H.7 · p. 27

H.8 EVALUATION OF PERSONALIZED MULTIMODAL AGENTS

The benchmark offers a standardized way to assess whether multimodal agents produce visual outputs that are both high-quality and user-appropriate. Its reason-then-reflect structure and separate planning and reflection evaluations are presented as a reusable template for other generation settings.

H.8 EVALUATION OF PERSONALIZED MULTIMODAL AGENTS

  • The evaluation suite jointly measures fidelity, user alignment, and output quality to test whether personalized multimodal agents produce high-quality, user-appropriate visuals.The passage names personalized assistants, avatar generators, and interactive design copilots as possible applications.Source: §H.8 · p. 27

H.9 MULTIMODAL REASONING AND PLANNING

  • PEARL’s reason-then-reflect structure may extend to tasks that translate multimodal context into an output specification and verify it against rendered evidence.The protocol separately evaluates planning through fidelity metrics and reflection through reflector ablation.Source: §H.9 · p. 27
Loading 2610.00737v1…