Source-linked AI summary
VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation
Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, Wenhu Chen
TL;DR
Evaluating conditional image synthesis remains difficult because existing metrics are task-limited and opaque, while human judgments are hard to scale. VIESCORE uses instruction-guided MLLMs to produce explainable image scores across tasks. GPT-4o and GPT-4v correlate better with human ratings than open-source MLLMs, with weaker performance on editing tasks.
Problem
Existing automatic metrics are narrow or opaque, while human evaluation of synthetic images remains difficult to scale across conditional image tasks.
Method
VIESCORE uses an MLLM backbone that takes instructions, images, and task conditions to generate a rationale before a score, without task-specific metric training.
Results
Across seven tasks, close-source MLLMs such as GPT-4o and GPT-4v were more effective than open-source MLLMs, with correlations over 0.4 to human ratings on most tasks.
Takeaways & Limitations
VIESCORE shows potential as an explainable evaluator for conditional image synthesis, while GPT-4o outperformed existing automatic metrics across all evaluated tasks.
Takeaways & Limitations
VIESCORE has lower correlation on image editing tasks, where MLLMs often miss minor changes and fine-grained color or texture differences.
Abstract
from arXiv · showhide
In the rapidly advancing field of conditional image generation research, challenges such as limited explainability lie in effectively evaluating the performance and capabilities of various models. This paper introduces VIEScore, a Visual Instruction-guided Explainable metric for evaluating any conditional image generation tasks. VIEScore leverages general knowledge from Multimodal Large Language Models (MLLMs) as the backbone and does not require training or fine-tuning. We evaluate VIEScore on seven prominent tasks in conditional image tasks and found: (1) VIEScore (GPT4-o) achieves a high Spearman correlation of 0.4 with human evaluations, while the human-to-human correlation is 0.45. (2) VIEScore (with open-source MLLM) is significantly weaker than GPT-4o and GPT-4v in evaluating synthetic images. (3) VIEScore achieves a correlation on par with human ratings in the generation tasks but struggles in editing tasks. With these results, we believe VIEScore shows its great potential to replace human judges in evaluating image synthesis tasks.
1 Introduction
Existing metrics struggle to evaluate conditional image synthesis because they are task-agnostic and opaque, while human evaluation is difficult to scale. VIESCORE uses instruction-guided MLLMs to produce rationales and scores across tasks.
- Existing metrics can miss task-specific image properties and provide opaque scores with limited explainability.The cited metrics are described as agnostic to the end task and insufficiently interpretable.
- Human evaluation offers visual understanding but faces scalability limits and subjective preferences.
- VIESCORE accepts an instruction, synthesized image, and task conditions, then produces a natural-language rationale before the final score.
- Any suitable MLLM, including GPT-4 or LLaVA, can serve as the scoring function because it follows instructions and generates human-like responses.
- The paper studies how well MLLMs assess synthetic images against human evaluations across image generation and editing tasks.
2 Related Works
Conditional image synthesis spans diverse generation and editing tasks, but existing automatic metrics remain narrow and human evaluation remains difficult to scale. Prior work on LLM-based evaluation motivates using instruction-following models for broader image assessment.
- Conditional image synthesis includes text-to-image generation, inpainting, and text-guided image editing.
- Subject-driven generation and editing, multi-concept composition, and control-guided generation extend conditional synthesis with subject and structural conditions.
- Traditional metrics measure fidelity or text alignment, with CLIP score and BLIP score among commonly used alignment measures.
- Automatic metrics remain narrow in scope, while human benchmarks such as ImagenHub and HEIM improve standardization but retain scalability challenges.
- LLM evaluators draw on emergent reasoning and instruction-following abilities for generated-output evaluation.
3 Preliminary
ImagenHub evaluates conditional image synthesis using human ratings of semantic consistency and perceptual quality. MLLMs offer a possible multi-image, instruction-driven evaluator, while existing metrics cover only limited aspects of image quality.
- 3.1 Evaluation Benchmark: ImagenHub is a standardized human-rated benchmark covering image generation, editing, and conditioned tasks.
- 3.1 Evaluation Benchmark: ImagenHub scores semantic consistency, perceptual quality, and an overall score derived from their instance-level geometric mean.
- 3.2 Multimodal Large Language Models: MLLMs integrate visual capabilities and must process multiple images and lengthy prompts to evaluate complex synthetic-image conditions.
- 3.2 Multimodal Large Language Models: Many open-source MLLMs accept only one image, whereas newer models support interleaved image-text inputs and closed-source models include GPT-4v, GPT-4o, and Gemini.
- 3.3 Existing Auto-metrics: CLIP-Score has a narrow practical range and cannot assess all semantic-consistency aspects, including overediting in image editing tasks.
- 3.3 Existing Auto-metrics: LPIPS captures perceptual resemblance and distortions but misses image naturalness factors such as shadows, lighting, and spatial distance.
- 3.3 Existing Auto-metrics: CLIP-I may confuse different subjects with similar descriptions, while DINO is more sensitive to differences between subjects of the same class.
- 3.3 Existing Auto-metrics: Figure 2 studies correlation between MLLM ratings and human perspectives on image evaluation.
4 Method
VIEScore evaluates synthetic images by feeding task conditions, images, and rating instructions to an MLLM, which produces explainable sub-scores and an overall score. Its rating system separates semantic consistency from perceptual quality and aggregates their sub-scores while emphasizing that all criteria should be met.
- Three human raters score each image aspect using options [0, 0.5, 1].
- All input conditions, synthesized images, and rating instructions are provided to the MLLM together in one pass.The figure illustrates this process for multi-concept image composition.
- Semantic Consistency is evaluated with task-specific sub-scores, while Perceptual Quality separately assesses naturalness and distortion.For multi-concept composition, semantic consistency covers prompt alignment and resemblance to each concept; perceptual quality covers unnatural appearance and visible defects.
- The overall score combines the semantic-consistency and perceptual-quality sub-scores using equal weighting and a minimum operation.The minimum operation emphasizes meeting every criterion without exception; α_i denotes semantic-consistency sub-scores and β_i perceptual-quality sub-scores.
- Semantic-consistency rating receives input conditions and the synthetic image, whereas perceptual-quality rating receives only the synthetic image.This separates condition alignment from image-only visual assessment during MLLM evaluation.
5 Experimental Results
Across conditional image tasks, MLLM-based VIESCORE correlates with human judgments most strongly for generation, while editing remains challenging. GPT-4o generally leads other MLLMs, and extra example images can reduce evaluation reliability.
- Metric-to-Human Correlations: GPT-4o reports significantly higher human-rating correlation than other evaluated MLLMs across image-generation and editing tasks.GPT-4v and Gemini are satisfactory on nearly all tasks, whereas LLaVA correlates much less than human raters.
- Metric-to-Human Correlations: GPT-4v and Gemini remain within 0.2 of human correlations on nearly all tasks and can match humans on text-guided generation.
- In-Context Learning: In-context examples generally diminish correlations, especially for subject-driven generation, subject-driven editing, and multi-concept composition.Only mask-guided editing and control-guided generation show subtle increases.
- In-Context Learning: MLLMs become confused by example images, helping explain degraded correlations and weak performance on tasks requiring multiple-image understanding.The confusion appears in GPT-4v, GPT-4o, Gemini, and LLaVA rationales.
- Ranking Models: GPT-4v generally aligns more strongly with ImagenHub model rankings than LLaVA, although alignment varies substantially across tasks.Both models align on multi-concept composition and control-guided generation, with nearly matching alignment on subject-driven editing.
- Insights and Challenges: Editing-task correlations are generally lower because MLLMs often miss minor changes such as small patch edits and perceive edited images as identical.The authors attribute this limitation to emphasis on high-level features over fine details such as color and texture.
- VIESCORE and Auto-metrics vs Human: GPT-4o achieves the highest correlation with human annotations, while DINO can outperform GPT-4v and CLIP-I for subject-driven semantic consistency.LPIPS is effective for control-guided tasks but less effective for image editing.
6 Conclusion
The paper proposes VIESCORE for evaluating synthetic images across seven conditional image synthesis tasks and finds stronger performance for closed-source MLLMs, especially GPT-4o, than for open-source models and automatic metrics. However, correlations are lower for image editing tasks.
- VIESCORE evaluates synthetic images across seven popular conditional image synthesis tasks using human ratings from ImagenHub.
- GPT-4o and GPT-4v are significantly more effective than open-source MLLMs for assessing synthetic images.
- Correlation exceeds 0.4 with human ratings for most tasks when VIESCORE uses closed-source MLLM backbones.
- VIESCORE shows lower correlation on image editing tasks for most MLLMs, including GPT-4v.
- GPT-4o is more effective than existing automatic metrics across all tasks, while DINO outperforms GPT-4v on subject-driven generation and editing.
- The authors identify distillation models as a future direction for replicating human-like performance in synthetic-image evaluation.
7 Limitations
The evaluation is constrained by GPT-4v’s refusal to process images resembling real people and by the API’s lack of persistent sessions for in-context learning.
- GPT-4v refuses to evaluate AI-generated images resembling real people or photographs, so those results are removed by keyword matching.
- The experiments rely on the OpenAI API because it lacks the persistent session functionality available in the GPT-4v Playground.
- The authors believe the Playground might perform better, especially for in-context learning, but use the API for the large-scale experiment.
8 Potential Risks
Multimodal models may reproduce training-data biases and may inadequately interpret image nuances or cultural sensitivities, producing inappropriate or offensive outputs.
- Multimodal models can inadvertently perpetuate or amplify biases present in their training data.
- Synthetic-image interpretation and evaluation depend heavily on context, including image nuances and cultural sensitivities.
- Insufficient understanding of nuances or cultural sensitivities may lead to inappropriate or offensive outputs.
9 Artifacts
The paper’s datasets and models are publicly accessible for academic use, and the official OpenAI API is available for academic purposes.
- All datasets and models are publicly accessible for academic use.
- The official OpenAI API is available for academic purposes.
10 Computational Experiments
The experiments define task-specific prompts and scoring procedures for evaluating semantic consistency and perceptual quality across seven conditional image-generation tasks. The study compares MLLM-based ratings with human correlations using zero-shot and one-shot settings.
- Prompt Design: The prompt combines a context prompt with a rating prompt to guide each MLLM evaluation.Outputs are requested in JSON containing scores and concise reasoning.
- Perceptual Quality: Perceptual quality is rated using separate 0-to-10 scores for image naturalness and artifacts.The perceptual-quality template applies across tasks.
- Semantic Consistency: Semantic consistency is evaluated with task-specific 0-to-10 scores for prompt adherence and, where relevant, guidance or subject resemblance.The templates cover text-guided, control-guided, subject-driven, and editing tasks.
- Experimental Settings: The study reports Spearman correlations and compares zero-shot with one-shot performance, while few-shot experiments were omitted because GPT-4v failed in that setting.Pearson and Kendall correlations are additionally provided for human-to-human comparison.
- Experimental Coverage: The evaluation includes text-guided, mask-guided, text-guided, subject-driven, multi-concept, and control-guided generation or editing categories.The listed model sets span seven conditional image-generation task groups.
B.5 ImagenHub Human data information
The ImagenHub evaluation uses human ratings and correlation tables to assess MLLM outputs across semantic consistency, perceptual quality, and overall categories. GPT-4o performs best among the tested MLLMs, while several open-source models have format, score, or rationale problems.
- Human Data and Correlations: ImagenHub human ratings are summarized alongside Spearman-correlation comparisons across semantic consistency, perceptual quality, and overall categories.The appendix also reports Pearson and Kendall coefficients for human-to-human comparisons.
- MLLM Observations: GPT-4o-2024-05-13 tends to be the best MLLM, producing reasonable scores and rationale across every VIESCORE task.GPT-4-vision-preview is described as the second-best MLLM, while Gemini-1.5-Pro performs similarly to GPT-4v.
- MLLM Observations: LLaVA-1.5-7B produces reasonable rationale, but its scores concentrate toward certain numbers.This concentration distinguishes its score behavior from its ability to generate rationale.
- MLLM Observations: Qwen-VL-7B outputs JSON-like dictionaries for semantic consistency and perceptual quality, but its rationale is often unreasonable.It does not understand the delimiter used in the instructions.
- MLLM Observations: BLIP-2, InstructBLIP, Fuyu, CogVLM, and OpenFlamingo show failures including invalid formats, constant or zero scores, out-of-range values, or blank outputs.Prompt engineering does not resolve the reported issues for BLIP-2 or CogVLM.