Source-linked AI summary
UFO: Chain-of-Evaluation for Omni-Condition Alignment in Multi-Modal Image Generation
Danning Zhang, Yijing Lin, Shuhan Zhuang, Mengqi Huang, Shaojin Wu, Shancheng Fang, Zhendong Mao
TL;DR
Subject-driven image generation needs reliable evaluation, yet existing evaluation and benchmark approaches have limited coverage of complex multi-modal alignment. UFO evaluates omni-condition alignment through atomic, modality-relevant units, and reports the highest correlation with human preferences, with a 15.25% average improvement over existing metrics.
Problem
Evaluation is a bottleneck for generative models, and existing customization benchmarks have limited variety in interactions between textual and visual conditions.
Source: §1 · p. 1, §3.1 · p. 5Method
UFO decomposes omni-condition alignment into atomic evaluation units, classifies their modality relevance, and evaluates them with VQA queries or specialized function calls.
Source: §1 · pp. 2–3Results
15.25% average improvement over existing metrics, with UFO achieving the highest correlation with human evaluation preferences and particular advantages in complex scenarios.
Source: §1 · p. 3Takeaways & Limitations
UFO-Bench assesses subject-driven generation across diverse subject categories and hierarchical editing scenarios with conflicting multi-modal conditions.
Source: §1 · p. 3This paper focuses on a basic, well-defined personalized image-generation setting, rather than more complex multi-image and multi-turn scenarios.
Source: §3.2 · p. 7
Abstract
from arXiv · showhide
Multi-modal image generation, particularly subject-driven customization, has garnered growing attention in recent years. Despite the rapid advancement of generative models, their evaluation remains largely lagging. Existing methods, whether embedding-based or Multi-modal Large Language Model (MLLM)-based, evaluate alignment with each modal condition in isolation, which contradicts the simultaneous condition alignment objective of multi-modal image generation, leading to poor consistency with human judgments. To address this challenge, we propose UFO, the first unified framework for omni-condition alignment simultaneous evaluation. Specifically, UFO introduces a novel Atomized Chain-of-Evaluation paradigm, i.e., it first decomposes omni-condition alignment into a sequential chain of fine-grained, disentangled Atomic Evaluation Units (AEUs), categorizes them into distinct modality-relevance classes, and then employs general or dedicated functional calls for accurate verification of different AEU types. Experimental results demonstrate that UFO achieves the highest correlation with human evaluation preferences, delivering an average improvement of 15.25%. Furthermore, we present UFO-Bench, a dedicated benchmark designed to holistically evaluate the performance of existing customization models under the diverse mutual interactions of textual and visual conditions.
1 Introduction
Subject-driven customization requires generated images to satisfy textual edits while preserving unedited subject attributes, but existing evaluators assess modal conditions in isolation. UFO proposes atomized, condition-aware evaluation and UFO-Bench; experiments report stronger human-preference alignment than existing metrics.
1 Introduction
- Isolated evaluation can miss textual edits and fine-grained identity changes, while penalizing valid generations that apply edits and preserve identity.These errors include false positives for outputs that resemble the reference but ignore text, and false negatives for valid edits with reduced visual similarity.Source: §1 · p. 2, Figure 1 · p. 2
- UFO decomposes omni-condition alignment into atomic units, classifies their modality relevance, verifies them with VQA or specialized tools, and aggregates their scores.The framework describes the resulting holistic score as interpretable and uses adaptive importance weighting.Source: §1 · pp. 2–3
- UFO-Bench addresses limited coverage of modality interactions with subject categories and hierarchical editing scenarios, including conflicting conditions.Its design spans three editing-difficulty tiers and includes 81.97% conflicting condition pairs.Source: §1 · p. 3
- 15.25% average improvement over existing metrics, with UFO achieving the highest correlation with human evaluation preferences and particular advantages in complex scenarios.The experiments compare UFO’s alignment with human evaluation preferences against existing metrics.Source: §1 · p. 3
2 Related Work
Prior work spans unified generation models, subject-driven benchmarks, and image-evaluation metrics. The cited benchmark and metric approaches leave gaps in assessing complex, conflicting textual and visual conditions.
2.1 Subject‑Driven Image Generation
- Unified subject-driven generation models explore varied generation and editing capabilities through different architectures and cross-modal strategies.Examples include OmniGen2 [30], UNO [31], Nano Banana, Qwen-Image [29], and BAGEL [4].Source: §2.1 · p. 3
2.2 Subject‑Driven Image Generation Benchmarks
- Existing benchmarks extend from general editing to multi-task and 3D settings, while later work adds human-aligned or hierarchical assessment.DreamBench++ [19] uses strong VLMs, DSH-Bench varies task difficulty, and Beyond the Pixels [24] uses hierarchical VLM prompting for identity preservation.Source: §2.2 · pp. 3–4
2.3 Evaluation Metrics
- Evaluation metrics cover visual quality and text-image consistency, while MLLM approaches decompose scoring but still have limits in image-level or independent-dimension assessment.VIEScore [12] uses fine-grained sub-criteria; DreamBench++ [19] separates concept preservation from prompt following, but relies on absolute scores and may miss fine-grained quality differences and complex preferences.Source: §2.3 · p. 4
3 Method
UFO-Bench targets gaps in the variety of textual–visual interactions represented in customization benchmarks, while UFO evaluates omni-condition alignment through a four-stage chain. The benchmark spans editing scenarios, and UFO decomposes, classifies, scores, and weights atomic evaluation units (AEUs).
3 Method
- UFO-Bench addresses limited interaction variety in existing customization benchmarks, which often omit conflicting conditions and combined global and local edits.These gaps constrain evaluation of identity preservation alongside complex textual instructions.Source: §3 · p. 4, §3.1 · p. 5
3.1 UFO Benchmark
- UFO-Bench covers seven subject categories and four editing paradigms, from non-editing through complex editing.The paradigms test recontextualization, targeted additions, global transformations, and their combination; prompts are prepared iteratively with LLMs.Source: §3.1 · pp. 5–6, Figure 3 · p. 5
3.2 Unified omni‑Condition Alignment (UFO)
- UFO focuses here on personalized image generation and obtains an interpretable holistic alignment score through a four-stage evaluation pipeline.It decomposes alignment into AEUs, assigns modality relevance, scores each unit, and aggregates scores with adaptive weighting.Source: §3.2 · p. 7
3.2.1 AEUs Decomposition
- UFO uses a VLM to jointly reason over the reference image and text, decomposing alignment into fine-grained AEUs at local-component and global-attribute levels.The AEUs make visual and textual consistency measurable as weighted semantic components.Source: §3.2.1 · p. 7
3.2.2 Modality‑relevance Classification
- Each AEU is classified as image-only, text-only, or text-and-image relevant so it is checked against the corresponding condition type.This classification follows from differing modality dependencies across AEUs.Source: §3.2.2 · p. 7
3.2.3 VQA and Function Calls Scoring
- UFO scores AEU-specific VQA queries with Yes/No judgments and invokes specialized function calls, such as ArcFace for facial identity consistency.The VQA judgments map to binary scores; the ArcFace score is normalized to [0, 1], with higher values indicating stronger identity consistency.Source: §3.2.3 · p. 7
3.2.4 Weighted Aggregation
- UFO aggregates AEU scores as a weighted average, assigning weights from 1 to 5 based on the importance of visual attributes and editing requirements.This strategy jointly accounts for condition importance and is presented as supporting more interpretable, human-aligned evaluation.Source: §3.2.4 · p. 8
4 Experiments
Experiments compare UFO with existing automatic metrics and evaluate customization models on UFO-Bench. UFO achieves the highest reported correlation with human evaluations, while ablations support the roles of weighted aggregation and atomic decomposition.
4.1 Experimental Setup
- The setup uses GPT-4o, evaluates closed- and open-source models against LLM- and embedding-based metrics, and measures average per-case Spearman correlation with human rankings.Human evaluation covers 100 randomly sampled test cases and 600 generated images from six models.Source: §4.1 · pp. 8–9
4.2 Main Results
- Doubao (2025-12) achieves the highest total score (0.7439); Qwen-Image (2025-12) scores 0.7252, surpassing Nano-Banana and other community models.Qwen-Image scores 0.7635 on non-editing and 0.7836 on global editing, but 0.6532 on local editing.Source: Table 1 · p. 9, §4.2 · p. 9
- CLIP-T has a negative correlation (-0.1996), while CLIP-I and DINO reach ρ≈0.3–0.4; VIE-Score and DreamBench++ exceed 0.55 overall but are around 0.58 on local editing.The MLLM-based metrics lack an explicit mechanism to separate the edited region from the preservation background.Source: §4.2 · p. 9
- Table 2 reports that UFO achieves the highest correlation with human evaluations.The evaluation compares metric-induced rankings with human rankings.Source: §4.2 · p. 9, Table 2 · p. 10
4.3 Ablation Study
- Replacing adaptive weights with uniform weights reduces correlation from 0.6889 to 0.6253.The authors note that uniform weighting treats attributes equally, potentially over-penalizing trivial violations and under-rewarding important characteristics or instructions.Source: Table 3 · p. 10, §4.3 · p. 10
- Removing AEU decomposition reduces correlation from 0.6889 to 0.5752, as holistic scoring struggles with complex visual details.In this ablation, the image is passed directly to the VLM for a single holistic score.Source: Table 3 · p. 10, §4.3 · p. 10
4.2 Main Results
- Figure 6 contrasts metric behavior on conflicting conditions, subtle subject-detail inconsistencies, and facial-identity inconsistencies.Its examples show existing metrics may favor visual similarity over text instructions, while expert function calls detect identity inconsistencies overlooked by other metrics.Source: §4.2 · p. 10
5 Conclusion
UFO evaluates atomic modality-relevance units and aggregates them into a holistic score, while UFO-Bench covers diverse subjects and hierarchical editing scenarios. Experiments show stronger alignment with human judgment than existing metrics, particularly under complex multi-modal conditions.
5 Conclusion
- UFO evaluates atomic modality-relevance units and aggregates them into a holistic score; UFO-Bench spans diverse subjects and hierarchical editing scenarios.Experiments report stronger alignment with human judgment than existing metrics, particularly under complex multi-modal conditions.Source: §5 · p. 11