Source-linked AI summary
MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Bowen Zheng, Jintao Lin, Changheng Yi, Haoran Yang, Binxiao Huang, Chenyang Lei, Rui Liu, Xihui Liu
TL;DR
Existing multi-reference image-generation benchmarks cover limited evaluation dimensions despite models’ growing use of multiple references to express complex visual attributes. MDMR-Bench combines diverse capability settings with checklist-based evaluation; its results show performance gaps across model types and task dimensions, while even strong models retain difficulties.
Problem
Existing multi-reference benchmarks provide limited evaluation dimensions and do not capture the diverse challenges of generation conditioned on multiple references.
Source: §1 · p. 2Method
MDMR-Bench evaluates 2,390 cases across diverse multi-reference tasks using global- and atomic-level checklist questions.
Source: §1 · pp. 2–3Results
Proprietary models substantially outperform open-source models, while models are relatively stronger at visible entity fusion than infographic manipulation, visual reasoning, and fine-grained attribute transfer.
Source: §5 · pp. 9–10Takeaways & Limitations
MDMR-Bench provides a basis for measuring capability gaps and guiding future progress toward more faithful, controllable, and reliable multi-reference generation systems.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.
1 Introduction
MDMR-Bench addresses the limited task coverage and evaluation dimensions of existing multi-reference benchmarks with a 2,390-case benchmark spanning under-evaluated capabilities. Its checklist protocol combines global and atomic questions for more fine-grained assessment.
1 Introduction
- Existing multi-reference benchmarks are often small, narrow in task coverage, visually simple, and organized around only one evaluation aspect.Some vary input styles, but these variations do not capture the diverse capabilities required in real-world multi-reference generation.Source: §1 · p. 2
- MDMR-Bench contains 2,390 cases and broadens dedicated task coverage to include multi-element extraction, infographic manipulation, and reasoning-intensive generation.It also covers common entity fusion and attribute transfer, using real-world and high-quality synthetic images.Source: §1 · pp. 2–3
- The benchmark introduces checklist-based evaluation with global- and atomic-level questions to support fine-grained diagnosis of model capabilities and failures.Global questions assess overall prompt requirements, while atomic questions target fine-grained details; the paper reports higher inter-judge agreement than score-based evaluation.Source: §1 · p. 3
2 Related Works
Related work traces multi-reference generation to the limits of text and single-image guidance, then reviews gaps in reference-generation benchmarks and evaluation protocols. The paper positions MDMR-Bench as addressing task coverage and fine-grained assessment.
2.1 From Text and Single-Reference to Multi-Reference Image Generation
- Text and single-reference editing can be insufficient to specify complex visual attributes, motivating multiple reference images; reliable, fine-grained evaluation remains unclear.Representative multi-reference models include Qwen-Image-Edit, DreamOmni2 [48], the GPT-Image series, the NanoBanana series, and the Seedream series [38].Source: §2.1 · p. 3
2.2 Benchmarks for Reference-Based Image Generation
- Existing multi-reference benchmarks vary in scale, task coverage, evaluation protocol, and realism, with some lacking fine-grained attribute transfer or task-level difficulty modeling.MultiBanana explores reference-image variations, while its visual styles are described as relatively homogeneous.Source: §2.2 · p. 4
2.3 Evaluation Metrics and Protocols
- Similarity metrics can be unsuitable when target outputs are not unique, while scalar VLM scores remain coarse-grained and inconsistent across evaluators.Checklist evaluation decomposes prompts into atomic questions and is reported in prior work to offer finer granularity, interpretability, and stronger human-judgment alignment; it remains largely unexplored for multi-reference generation.Source: §2.3 · p. 4
3 MDMR-Bench
MDMR-Bench organizes multi-reference image generation around complementary capability settings, using a scalable construction pipeline and checklist-based evaluation. Its design covers both established and under-evaluated tasks, with scoring that gives equal weight to five task families.
3.1 Challenging Dimensions
- The benchmark covers entity fusion and attribute transfer, multi-element extraction, infographic manipulation, and visual reasoning across complementary multi-reference capabilities.Entity fusion and attribute transfer vary the number of references from two to four; other settings test multi-element composition, structured visual editing, and implicit relationships.Source: §3.1 · pp. 4–5
3.2 Sample Construction
- Samples are built through image-gallery construction, task-bucket filling, and validation and refinement, with specialized procedures for some settings.The gallery combines LAION images above 1024 resolution with synthetic images; 48.2% of cases include at least one synthetic reference, and 29% of reference images are synthetic.Source: §3.2 · pp. 5–6
- The benchmark contains 936 entity-fusion, 982 attribute-transfer, 101 multi-element-extraction, and 278 infographic-manipulation cases, plus visual-reasoning cases.Infographic cases cover Chinese and English content; reasoning cases include analogical and logical inference across transformations such as temporal, stylistic, state, spatial, and layout changes.Source: §3.2 · p. 6
3.3 Evaluation Metrics
- Evaluation uses binary checklists across Following, Consistency, Composition, Minimality, and Quality, with equally weighted global-level and atomic-level question groups.Gemini-3-Pro generates the questions from instructions and reference images; a subset is manually audited.Source: §3.3 · p. 6
- The final score equally averages the five task-family scores, so sample counts do not directly determine each family’s contribution to Overall.Dimension weights are 0.20, 0.25, 0.25, 0.20, and 0.10 for Following, Consistency, Composition, Minimality, and Quality, respectively.Source: §3.3 · p. 7
4 Evaluation Results
Across the benchmark, proprietary models outperform open-source systems, while task- and dimension-level analyses reveal persistent weaknesses in reference fidelity, composition, infographic manipulation, and human-related attribute transfer.
4.1 Experimental Setup
- The evaluation includes six proprietary and five open-source systems, supporting comparison of their capabilities and the difficulty dimensions that remain challenging.Source: §4.1 · p. 7
4.2 Experimental Results
- 88.79 for GPT-Image-2, 85.1 for Nano Banana Pro, and 81.2 for Nano Banana 2 lead Gemini-3-Pro scores; FLUX.2 leads open-source models at 51.3.Proprietary models outperform open-source models across all five dimensions.Source: §4.2 · p. 7
- Proprietary models generally score higher on Following and Quality than on Consistency, Composition, and especially Minimality; open-source models often have plausible images but miss precise constraints.The benchmark therefore assesses whether models faithfully bind, preserve, and compose information from multiple references, not only image realism.Source: §4.2 · pp. 7–8
4.3 Task-Level Results
- Task-level results show proprietary models perform more evenly, multi-element extraction is often easier than using separate reference images, and infographic manipulation is weaker for all but Nano Banana Pro.Nano Banana Pro and Seedream 4 outperform most systems on reasoning-related tasks; BAGEL is relatively strong despite weaker overall image quality.Source: §4.3 · p. 8, Table 3 · p. 8
4.4 Reliability of VLM Judge
- 0.853, 0.854, and 0.849 are the agreement ratios with human judgments for Gemini-3-Pro, GPT-5.4, and Doubao-1.8, respectively.Gemini-3-Pro has the highest Spearman correlation (0.862) and Cohen’s κ (0.642).Source: §4.4 · p. 8, Table 4 · p. 9
4.5 Diagnostic Analysis
- Human-related attribute transfer, including expression, makeup, outfit, pose, and hairstyle, performs substantially worse; GPT-Image-1.5’s strong Following comes with lower Minimality.The latter reflects changes to content that should have remained untouched.Source: Figure 6 · p. 9, §4.5 · p. 9
4.4 Reliability of VLM Judge
- Gemini-3-Pro provides the most balanced reliability across four measures and is the default evaluator; Qwen3-VL-32B reaches an agreement ratio of 0.800 despite lower alignment.Source: §4.4 · p. 9
4.5 Diagnostic Analysis
- 0.106 lower Overall and 0.122 lower Composition are the approximate two-to-four-reference declines on Attribute Transfer, on the plotted 0–1 scale.Source: §4.5 · p. 9
5 Conclusion
MDMR-Bench evaluates multi-reference image generation across diverse tasks using interpretable checklist judgments. Its evaluation finds that proprietary systems outperform open-source models, while significant capability gaps remain.
5 Conclusion
- MDMR-Bench is a 2,390-case benchmark spanning entity fusion, attribute transfer, multi-element extraction, infographic manipulation, and visual reasoning-aware generation.Its checklist protocol decomposes outputs into interpretable, visually verifiable judgments.Source: §5 · p. 9
- Proprietary systems substantially outperform open-source models, but even strong models show failures in reference consistency, composition, and minimality.Source: §5 · p. 9
- Models are relatively stronger at visible entity fusion, while infographic manipulation, visual reasoning-aware generation, and fine-grained attribute transfer remain difficult.More-reference subsets score lower, and instruction following can accompany unintended modifications.Source: §5 · p. 10
A.1 Inter-Judge Agreement Analysis Setting
The analysis compares checklist-based and score-based judge agreement, and separately assesses agreement between VLM and human ratings.
A.1 Inter-Judge Agreement Analysis Setting
- Checklist-based evaluation achieves substantially higher inter-judge agreement than score-based evaluation across most aspects.Separately, Gemini score-based evaluation has Krippendorff’s α = 0.5896 against human scores using the interval metric.Source: §A.1 · p. 14
A.2 Cross-Judge Evaluation
A supplementary analysis tests whether the benchmark’s model conclusions depend on using Gemini as evaluator.
A.2 Cross-Judge Evaluation
- GPT-Image-2, Nano Banana Pro, and Nano Banana 2 remain the top three under GPT-5.4 and Gemini-3-Pro, though the next three models reorder.The authors report that the main conclusions are stable under a non-Gemini evaluator.Source: Table 5 · p. 14, §A.2 · p. 14
A.3 Human Evaluation Protocol
The human evaluation compares checklist judgments with independently collected human labels under blinded, randomized conditions. A separate audit finds that 98.75% of checklist items are valid.
A.3 Human Evaluation Protocol
- Five annotators independently judged 5,924 checklist decisions across 800 outputs, with majority vote as the final human label.The 200 cases were evaluated using four generators, with model identities hidden and output order randomized.Source: §A.3 · p. 14
- Human inter-annotator agreement was Krippendorff’s α = 0.88.Annotators inspected the instruction, reference images, and generated output, using the same evaluation instructions as VLM judges.Source: §A.3 · p. 14
A.4 Checklist Validity Audit
- 98.75% of 2,962 audited checklist items were valid, while 37 items were invalid.Five PhD-level auditors reviewed items from 100 cases spanning all five task families.Source: §A.4 · p. 15
A.5 Controlled Real-versus-Synthetic Inputs
A paired experiment compared real and synthetic reference inputs under the same task structures and evaluation protocol. The authors also repeated generation and judging to examine run-to-run consistency on Multi-Element Extraction.
A.5 Controlled Real-versus-Synthetic Inputs
- The paired input-source experiment used 100 cases with matched task structure and instruction semantics, evaluated by Nano Banana Pro and Seedream 5.0.Real references were captioned and synthesized by Nano Banana Pro; both input versions used the same checklist protocol and Gemini-3-Pro judge.Source: §A.5 · p. 15
- Overall scores increased by 0.43 points for Nano Banana Pro and 0.23 for Seedream 5.0; the gap changed from 4.08 to 4.28 points.Dimension-level changes were mixed; the performance gap changed by 0.20 points.Source: §A.5 · p. 15, Table 8 · p. 15
A.6 Repeated Runs on Multi-Element Extraction
- Nano Banana Pro and Nano Banana 2 led all three independent runs on Multi-Element Extraction, while Seedream 4.0 and 5.0 exchanged positions in run three.The runs independently repeated generation and judging across all 101 cases.Source: §A.6 · p. 15, Table 9 · p. 16
A.7 Limitations
The benchmark’s coverage, construction, and evaluation each have limits: some challenge settings are harder to represent, manual filtering constrains scalability, and automatic checklists cannot capture all aspects of image quality.
A.7 Limitations
- Reasoning-intensive and infographic-oriented cases remain challenging to construct realistically and semantically coherently, so their coverage may lag common entity fusion and attribute transfer.The limitation concerns some challenge settings rather than the benchmark’s coverage overall.Source: §A.7 · p. 16
- Most subsets use only two reference images, except Entity Fusion, Attribute Transfer, and Analogical Reasoning, to test atomic capabilities under controlled settings.The authors prioritize evaluating capabilities such as fine-grained attribute extraction, symbolic editing, and reasoning-aware generation over maximizing reference count.Source: §A.7 · p. 16
- Manual filtering improves benchmark reliability and authenticity but limits dataset-construction scalability.The authors cite current multimodal models’ mistakes on subtle visual differences, source-specific details, and fine-grained attributes as motivation for filtering.Source: §A.7 · p. 16
- Automatic checklist evaluation improves consistency and interpretability but cannot fully replace human judgment, especially for artistic preference, creativity, and cultural nuance.These image-quality aspects remain difficult to assess with deterministic yes/no questions alone.Source: §A.7 · p. 16
A.8 Compute Resources
MDMR-Bench’s supplementary materials describe its computational requirements, intended impacts, released asset, and checklist prompts. They emphasize diagnostic use while noting evaluation and misuse considerations.
A.8 Compute Resources
- Benchmark evaluation requires generating outputs for open-source baselines; those runs used GPUs with 80GB of memory, while proprietary models were accessed through APIs with undisclosed hardware.The benchmark primarily evaluates inference rather than training, and memory use varies with model and reference-image handling.Source: §A.8 · p. 16
- Checklist generation and judging use API calls, so practical cost is dominated by latency, rate limits, and multimodal request volume.The stages can be parallelized across samples, models, GPU workers, and API workers.Source: §A.8 · p. 16
A.9 Broader Impacts
- The benchmark is intended to improve transparency and reliability by diagnosing failures such as identity preservation, attribute transfer, text rendering, and reference composition.The authors frame this diagnostic use as support for understanding model failures before deployment in user-facing settings.Source: §A.9 · p. 17
- More capable multi-reference generation could be misused for misleading media, impersonation, and deceptive advertisements, although this work introduces no generative model or safety-filter bypass tools.The authors encourage responsible use for research, transparency, and safety-oriented assessment.Source: §A.9 · p. 17
A.10 New Asset Documentation
- The benchmark asset records task family, subtask, image order, roles, content types, and related metadata to support auditable, reproducible evaluation.The released materials are to include data-format documentation, field descriptions, and scripts for checklist generation and evaluation.Source: §A.10 · p. 17
- The benchmark is intended for academic evaluation and diagnostic analysis, not identity verification, surveillance, or automated moderation.The authors also document limitations including manual filtering, task-category scope, and possible dependence of proprietary outputs on API versions and availability.Source: §A.10 · p. 17
A.11 Prompt Templates for Checklist Generation and Evaluation
- The two-stage protocol first generates atomic, visually verifiable checklist questions, then has a judge answer each item for a model output.The decomposition into independent yes/no judgments is intended to make failures easier to localize.Source: §A.11 · p. 17, Figure 8 · p. 19, Figure 9 · p. 20
- The generation prompt organizes questions under five dimensions and requires the first two questions per dimension to be global, with later questions atomic.Items are to be objective, visually verifiable yes/no questions grounded in the instruction, metadata, and images.Source: §A.11 · p. 18, Figure 8 · p. 19
B Qualitative Visualization Results
The qualitative examples illustrate multi-reference generation tasks ranging from compositing people and objects into scenes to transferring attributes and carrying out visual transformations. They also include text and infographic edits that require placing or restyling content.
B.1 Visualization Set 1
- Examples combine people, objects, clothing, and scenes from multiple references, including placing a person and chair in a landscape or assembling people in a furnished room.Other prompts combine a person's backpack and pants with a scene and chair from separate images.Source: §B.1 · pp. 22, 27
- Reasoning-intensive prompts apply a poem's historical period to clothing, transfer an action or pose, or reproduce a height ordering across images.A further prompt asks the model to analyze a seasonal transformation and apply it to another scene.Source: §B.1 · pp. 24, 29, 30
- Attribute-transfer prompts apply a reference's style, material, or environmental change to another image's objects or scene.Examples include restyling a plant and table, matching stone structures to a grenade's material, and transferring a seasonal or environmental transformation.Source: §B.1 · pp. 23, 24, 25, 29
- Text and infographic edits place objects in labeled sectors or change lettering to match the style of text in another image.Prompts include positioning an object below a Chinese label and restyling “Love flowers,” “CHICAGO,” and “HELPDESK.”Source: §B.1 · pp. 25, 26, 31
- Other examples extract and reposition multiple elements, replace objects or materials, and adjust attributes using separate references.Prompts include placing a rabbit and cat in a scene, swapping two cups, and changing vase material and bag color.Source: §B.1 · pp. 23, 28