PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation

Yuang Peng, Yuxin Cui, Haomiao Tang, Zekun Qi, Runpei Dong, Jing Bai, Chunrui Han, Zheng Ge, Xiangyu Zhang, Shu-Tao Xia

arXiv:2406.16855v2cs.CV

TL;DR

Personalized image generation needs evaluation of both prompt following and reference-concept preservation, while existing evaluation can be human-intensive or misaligned with people. DREAMBENCH++ combines GPT-based evaluation with a diverse image-and-prompt benchmark. It reports strong human agreement across both capabilities and identifies balancing preservation with prompt adherence as a central goal.

  • Problem

    Evaluating personalized image generation remains challenging: models must preserve reference concepts and follow prompts, while existing evaluations may be human-intensive or fail to assess these capabilities comprehensively.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    DREAMBENCH++ combines GPT-4o instructions and reasoning for automated evaluation with a dataset of diverse images and prompts.

    Source: §1 · pp. 2–3

  • Results

    79.64% human agreement for concept preservation and 93.18% for prompt following, exceeding DINO and CLIP by +32.59% and +37.23%, respectively.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Across the evaluated models, the paper identifies balancing concept preservation and prompt adherence as the primary goal.

    Source: §1 · p. 3

    Evaluation remains a preliminary attempt for personalized image generation and depends heavily on multimodal LLM advances and carefully designed system prompts.

    Source: D LIMITATION & FUTURE WORK · p. 23

Abstract

from arXiv · show

Personalized image generation holds great promise in assisting humans in everyday work and life due to its impressive ability to creatively generate personalized content across various contexts. However, current evaluations either are automated but misalign with humans or require human evaluations that are time-consuming and expensive. In this work, we present DreamBench++, a human-aligned benchmark that advanced multimodal GPT models automate. Specifically, we systematically design the prompts to let GPT be both human-aligned and self-aligned, empowered with task reinforcement. Further, we construct a comprehensive dataset comprising diverse images and prompts. By benchmarking 7 modern generative models, we demonstrate that DreamBench++ results in significantly more human-aligned evaluation, helping boost the community with innovative findings.

1 INTRODUCTION

Personalized image generation must preserve a reference image’s concept while following a text prompt, but the paper asks how to comprehensively evaluate models and compare technical routes. DREAMBENCH++ combines GPT-based human-aligned evaluation with a larger, more diverse dataset and reports strong agreement with human ratings.

1 INTRODUCTION

  • Personalized generation must satisfy both prompt following and preservation of the reference concept, yet the paper asks which technical route is superior and where to head.Preservation includes semantic details such as facial characteristics and higher-level abstractions such as style.Source: §1 · p. 2
  • DREAMBENCH++ achieves 79.64% human agreement for concept preservation and 93.18% for prompt following, exceeding DINO and CLIP by +32.59% and +37.23%, respectively.The benchmark evaluates seven modern models.Source: §1 · pp. 2–3
  • DREAMBENCH++ uses GPT-4o meta-prompts and internal reasoning, followed by task summarization and planning, to generate automated evaluation scores.Its instructions specify the task, scoring criteria, range, and format.Source: §1 · p. 3, Figure 3 · p. 3
  • DREAMBENCH++ expands evaluation to 150 images and 1,350 prompts, covering more varied subjects and difficulty levels than DreamBench.Those totals are 5× the images and 54× the prompts in DreamBench.Source: §1 · p. 3
  • Across seven models, the paper identifies balancing concept preservation and prompt adherence as the primary goal; DreamBooth excels at preserving detailed features while following prompts closely.The evaluated models perform well on animals and styles but struggle with human images and diverse object categories.Source: §1 · p. 3
  • The authors provide open-source code and standardized evaluation, and describe their GPT-based design as transferable to other domains and foundation models.The paper presents the transferability as the authors’ belief.Source: §1 · p. 3

2 DREAMBENCH++

DREAMBENCH++ standardizes GPT-based evaluation with direct ratings and task-specific criteria, and expands personalized-image data across image types and prompt difficulty. Its visualized distribution is reported as more diverse than DreamBench’s.

2.1 PROMPTING GPT FOR AUTOMATED & HUMAN-ALIGNED BENCHMARKING

  • The evaluation uses direct 0–4 ratings because GPT image-comparison judgments can be position-sensitive, while pairwise comparisons require quadratically more runs as methods increase.The paper identifies direct numerical rating as more efficient and scalable for evaluating multiple methods.Source: §2.1 · p. 4
  • GPT receives task descriptions, scoring criteria, score ranges, and format instructions, with criteria tailored separately to concept preservation and prompt following.Preservation criteria cover shape, color, texture, and applicable facial features; prompt-following criteria cover relevance, accuracy, completeness, and context.Source: §2.1 · p. 4
  • A two-step reasoning policy asks GPT to check and summarize task understanding, then plan the evaluation as a generalized form of chain-of-thought reasoning.The authors intend internal thinking to reinforce task understanding and instruction following.Source: §2.1 · p. 5

2.2 SCALING UP PERSONALIZED IMAGE GENERATION BENCHMARKING

  • The dataset categorizes images as objects, living subjects, or styles to address potential evaluation bias from DreamBench’s small scale and CustomConcept101’s limited diversity.The authors scale both image count and diversity.Source: §2.2 · p. 5
  • Images are collected from web sources and authorized contributors, screened using subject-region detection and human review, and curated before prompt generation.The paper notes that image copyright status is verified for academic suitability.Source: §2.2 · p. 5
  • Each image receives nine GPT-4o prompts spanning photorealistic, non-photorealistic, and imaginative content, yielding 150 images and 1,350 prompts.Human calibration checks that prompts are ethical and without flaws.Source: §2.2 · p. 5
  • The t-SNE comparison is reported to show greater image diversity in DREAMBENCH++ than DreamBench, while a second panel displays DREAMBENCH++ image and prompt distributions.The authors also manually balance subject classes and photorealistic versus non-photorealistic styles.Source: Figure 5 · p. 5, §2.2 · p. 6

3 EXPERIMENTS

The experiments compare DREAMBENCH++ with human and traditional metric evaluations, then examine benchmark rankings and prompt-design effects. They also describe the tested models and human-rating procedure.

3.1 EXPERIMENTAL SETUP

  • The evaluation compares three fine-tuning methods and four encoder-based methods, using SD v1.5 and SDXL v1.0 as base models.The authors report following official implementations where possible and tuning parameters for performance on DreamBench.Source: §3.1 · p. 6
  • Seven trained annotators score each instance, with at least two human ratings per instance to reduce noise.Human and GPT scoring use the same task and scheme, and their results are kept isolated to avoid hindsight bias.Source: §3.1 · p. 6

3.2 MAIN RESULTS

  • DREAMBENCH++ aligns better with human evaluations than DINO or CLIP for concept preservation, while CLIP-T aligns with humans on prompt following.DINO and CLIP diverge on concept preservation, with DINO favoring overall shape over color, texture, and facial detail.Source: Figure 6 · p. 6, §3.2 · p. 7
  • The leaderboard reports concept-preservation and prompt-following scores by category and ranks models using their product, CP·PF.The section identifies human images as the lowest-scoring concept category and describes a negative correlation between concept preservation and prompt following.Source: §3.2 · p. 7, Table 2 · p. 7
  • The prompt ablation finds that proposed prompt designs improve alignment, removing internal thinking causes a significant drop, and background-related human priors can degrade performance.The ablation measures alignment with mean Krippendorff’s alpha.Source: §3.2 · p. 7, Table 3 · p. 7

4 DISCUSSIONS

The discussion reports human-alignment results, examines data diversity and prompting choices, and tests consistency across multimodal models. It highlights both strengths and boundaries of the evaluation approach.

4.1 IS DREAMBENCH++ ALIGNED WITH HUMANS?

  • DREAMBENCH++ achieves 83.31% consistency with human evaluation for concept preservation and 98.71% for prompt following, exceeding traditional DINO and CLIP metrics by +32.59% and +37.23%.The alignment study measures mean Pearson correlation.Source: §4.1 · p. 8, Table 4 · p. 8

4.2 IS DATA DIVERSITY NECESSARY?

  • On DREAMBENCH++, TI, DreamBooth, and Emu2 show notable score drops versus DreamBench, despite broadly consistent overall results.The authors link the drops to difficulty with complex or stylized prompts and anime references; Figure 7 presents successful and failure cases for Emu2, DreamBooth, and TI.Source: §4.2 · p. 8, Table 5 · p. 8, Figure 7 · p. 8

4.3 CAN WE USE FREE LUNCH TO IMPROVE DREAMBENCH++ EVALUATION?

  • Chain-of-thought improves prompt-following evaluation but does not improve concept-preservation evaluation, and may shift attention toward background or texture rather than the subject.Figure 8 describes both the prompt-following benefit and the possible cost to visual concept preservation.Source: §4.3 · p. 9, Figure 8 · p. 9, Table 6 · p. 9
  • In-context learning counterintuitively reduces alignment, potentially due to the patching scheme, sample selection, or bias within GPT-4o.Source: §4.3 · p. 9, Table 6 · p. 9

4.4 HOW TRANSFERABLE IS DREAMBENCH++ ACROSS MLLMS?

  • Human–MLLM consistency is reported across models, from 74.00% for Claude-3.5 to 67.22%–84.23% across GPT-4o variants; newer GPT-4o versions show increasing consistency.The reported consistency rises from 79.47% to 84.23% across newer GPT-4o versions.Source: §4.4 · p. 9, Table 7 · p. 9

5 RELATED WORKS

Prior personalized-generation work spans fine-tuning, encoder-based, adapter-free, and multimodal approaches. Image-generation evaluation metrics cover distinct dimensions, but the cited literature leaves automated personalized evaluation unresolved.

Personalized Image Generation

  • Personalized generation developed from pixel-to-pixel translation toward free-form generation conditioned on reference images and prompts.Source: §5 · p. 10
  • Fine-tuning can create specialist personalization models but requires tuning for new subjects, whereas encoder-based methods can generate subject- or style-guided images in one shot.Encoder- and adapter-based approaches inject reference-image embeddings into diffusion cross-attention; adapter-free approaches extract and fuse reference information such as attention maps.Source: §5 · p. 10

Benchmarking Image Generation

  • Existing evaluation metrics address image quality, perceptual similarity, or prompt-image alignment, but automated personalized evaluation remains unresolved.Human-aligned metrics face scaling limits, and DreamBench is described as having only 30 simple objects and lacking diversity comprehensiveness.Source: §5 · p. 10

6 CONCLUSIONS

The paper presents DREAMBENCH++ as a human-aligned benchmark for personalized image generation. Its experiments examine dataset diversity and complexity, metric alignment, and GPT prompting approaches.

6 CONCLUSIONS

  • DREAMBENCH++ is introduced as a human-aligned benchmark for personalized image generation.Source: §6 · p. 10
  • The experiments report advantages in dataset diversity and complexity, alongside metrics that align with human preferences.Source: §6 · p. 10
  • The paper offers insights into prompt design for multimodal GPTs, including the potential and challenges of chain-of-thought prompting and in-context learning.Source: §6 · p. 10

A QUALITATIVE ANALYSIS

The analysis finds that technical routes have different strengths across image types: fine-tuning methods do better on subject-rich cases, while encoder-based methods do better on styles and are stronger at preserving human facial features.

A QUALITATIVE ANALYSIS

  • Fine-tuning methods preserve intricate details better for subject-oriented animals and objects, but often fail to preserve facial and clothing features in person images.The paper characterizes human-image personalization as more demanding for visual concept preservation than for textual following.Source: A QUALITATIVE ANALYSIS · p. 21
  • Encoder-based methods perform better in style-oriented cases where subject details matter less, leveraging high-level visual semantics such as overall shape, style, and theme.Source: A QUALITATIVE ANALYSIS · p. 21
  • Table 8 lists training hyperparameters for DreamBench and DREAMBENCH++, with BS meaning batch size, LR learning rate, and Steps training steps.Source: Table 8 · p. 22

B IMPLEMENTATION DETAILS

The paper reports that its DreamBench reproductions are comparable to or better than official results, documents implementation settings, and motivates single-image personalization as a practical setting.

B IMPLEMENTATION DETAILS

  • Most methods use guidance_scale 7.5 and 100 inference steps; Emu2 uses 3 and 50, while BLIP-Diffusion and IP-Adapter use negative prompts.IP-Adapter additionally uses ip_adapter_scale 0.6; Table 9 provides the negative-prompt templates.Source: B IMPLEMENTATION DETAILS · p. 22, Table 9 · p. 22
  • The reproduced DreamBench results are comparable to or better than the official results.The caption specifies that N/A marks results not reported in the official paper.Source: Table 10 · p. 22, B IMPLEMENTATION DETAILS · p. 22

C.1 ARE MULTIPLE IMAGES FOR EACH INSTANCE NECESSARY?

  • Single-image personalization is more relevant to daily use given limited reference-image availability, and fine-tuning methods perform well with one image.Source: §C.1 · p. 22

C.2 WHICH TECHNICAL ROUTE PERFORMS BETTER?

The preferred technical route depends on the application: encoders are favored for human identity preservation, while fine-tuning can suit prompt following and common objects.

C.2 WHICH TECHNICAL ROUTE PERFORMS BETTER?

  • Encoder-based methods preserve precise facial features better in human-centric generation, while fine-tuning handles common objects and standardized appearances more strongly.Encoder-based approaches provide more consistent performance across diverse object types.Source: §C.2 · p. 22
  • SDXL outperforms SD v1.5 in concept preservation and prompt following; fine-tuning generally follows prompts better than encoder-based methods.The paper attributes the SDXL advantage primarily to its text encoder architecture and diverse training data, and links encoder prompt-alignment challenges to feature injection.Source: §C.2 · p. 23
  • For human identity preservation, encoder-based methods are preferable; for precise prompt following or common objects, fine-tuning approaches may be more suitable.Source: §C.2 · pp. 22–23

D LIMITATION & FUTURE WORK

The authors describe human-aligned evaluation and benchmarking as an emerging, challenging direction, characterize their personalized-image-generation work as preliminary, and identify reliance on multimodal language models and carefully designed prompts.

D LIMITATION & FUTURE WORK

  • The work is a preliminary attempt at human-aligned evaluation for personalized image generation.Source: D LIMITATION & FUTURE WORK · p. 23
  • Evaluation results depend heavily on advances in multimodal large language models and on carefully designed system prompts.Source: D LIMITATION & FUTURE WORK · p. 23
  • Future work will extend human-aligned evaluation to areas including multimodal intelligence, audio, 3D and video generation, autonomous driving, and embodied visual intelligence.Source: D LIMITATION & FUTURE WORK · p. 23

BROADER IMPACT

Personalized T2I models may be misused to generate NSFW content, so the paper calls for safeguards during both use and evaluation. Evaluation content is filtered out by human annotators.

BROADER IMPACT

  • Personalized T2I models may be misused as tools for generating NSFW content.The paper identifies this risk for models built on pretrained T2I foundation models.Source: BROADER IMPACT · p. 23
  • The paper says NSFW detectors are critical safeguards during both model use and evaluation.It also calls for filtering evaluation data to avoid NSFW content.Source: BROADER IMPACT · p. 23
  • Human annotators filtered NSFW content out of the evaluation data.Source: BROADER IMPACT · p. 23
Loading 2406.16855v2…