PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation

Yuta Oshima, Daiki Miyake, Kohsei Matsutani, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta

arXiv:2511.22989v2cs.CV

TL;DR

Existing benchmarks offer limited coverage of reference counts and heterogeneous combinations, leaving multi-reference instruction following difficult to assess systematically. MultiBanana evaluates these challenges across reference counts and several mismatch types; its analyses identify distinct failure patterns and difficult task conditions across models.

  • Problem

    Existing benchmarks cover few references and do not systematically evaluate the heterogeneous combinations involved in multi-reference instruction following.

    Source: §1 · p. 2

  • Method

    MultiBanana evaluates multi-reference generation across up to 8 references, domain and scale mismatches, rare concepts, and multilingual textual references.

    Source: §1 · p. 2

  • Results

    As reference difficulty rises, models either omit required subjects or collapse, and cross-domain, scale or viewpoint-gap, rare-concept, and multilingual tasks remain challenging.

    Source: §5 · p. 8

  • Takeaways & Limitations

    MultiBanana is released as an open benchmark to establish a standard basis for fair comparison in multi-reference image generation.

    Source: §1 · p. 2

    The benchmark covers static image generation, not video generation or editing, and natural-language object placement prompts are inherently ambiguous.

    Source: §I · p. 23

Abstract

from arXiv · show

Recent text-to-image generation models have acquired the ability of multi-reference generation and editing; that is, to inherit the appearance of subjects from multiple reference images and re-render them in new contexts. However, existing benchmark datasets often focus on generation using a single or a few reference images, which prevents us from measuring progress in model performance or identifying weaknesses when following instructions with a larger number of references. In addition, their task definitions are still vague, limited to axes such as ``what to edit'' or ``how many references are given'', and therefore fail to capture the challenges inherent in combining heterogeneous references. To address this gap, we introduce MultiBanana, which is designed to assess the edge of model capabilities by widely covering problems specific to multi-reference settings: (1) varying the number of references (up to 8), (2) domain mismatch among references (e.g., photo vs. anime), (3) scale mismatch between reference and target scenes, (4) references containing rare concepts (e.g., a red banana), and (5) multilingual textual references for rendering. Our analysis among a variety of text-to-image models reveals their respective performances, typical failure modes, and areas for improvement. MultiBanana is released as an open benchmark to push the boundaries and establish a standardized basis for fair comparison in multi-reference image generation. Our data and code are available at https://github.com/matsuolab/multibanana .

1. Introduction

Existing benchmarks cover few references and do not adequately represent the challenges of combining heterogeneous references. MultiBanana evaluates these challenges across several dimensions and uses the benchmark to characterize model capabilities and failure modes.

1. Introduction

  • Existing benchmarks typically limit references to 1–4 and use task definitions that miss challenges of combining heterogeneous references.The paper identifies both limits as gaps in evaluating multi-reference instruction following.Source: §1 · p. 2
  • MultiBanana tests up to 8 references alongside domain and scale mismatches, rare concepts, and multilingual textual references.The examples include photo–anime mixtures, red bananas, and English, Chinese, and Japanese rendering references.Source: §1 · p. 2
  • As difficulty increases, closed-source models tend to include all subjects but distort composition, while open-source models often omit subjects despite cleaner outputs.The analysis varies difficulty through more references, cross-domain mixtures, or rare concepts.Source: §1 · p. 2

2. Related Works

Prior work develops controllable text-to-image methods and benchmarks for editing and reference-based generation. Existing benchmark coverage remains limited in reference count, task diversity, and heterogeneous reference combinations.

2.1. Controllable Text-to-Image Generation

  • Text-conditioned diffusion models and fine-tuning or adapter methods have been developed to improve controllability, while multimodal models handle text and image inputs in unified generation and editing.Examples include Stable Diffusion, FLUX, DreamBooth, IP-Adapter, GPT-Image-1, Nano Banana, and Qwen-Image.Source: §2.1 · p. 2

2.2. Benchmarks for Reference-Based Generation

  • Editing and multi-reference benchmarks extend evaluation, but remain limited in metric consistency, task coverage, reference count, or modeling heterogeneous references.The passage notes that OmniContext and DreamOmni2 use only a small number of images and task types and do not account for reference heterogeneity.Source: §2.2 · pp. 2–3, Table 1 · p. 3

3. MultiBanana

MultiBanana extends reference-driven generation tasks to eight references and explicitly tests how models combine heterogeneous references. Its construction balances data and task coverage, while its evaluation scores five aspects of generated images.

3.1. Task Types

  • The benchmark extends reference counts to eight and adds domain gaps, scale mismatches, rare concepts, and multilingual rendering to its task design.Its multi-reference setting defines four task structures for 3–8 references, yielding 24 tasks.Source: §3.1 · pp. 3–4

3.2. How to Construct MultiBanana

  • Real-image sourcing favored backgrounds, so synthetic images were added to expand underrepresented person- and object-related references.Real images came from LAION-5B and were selected for aesthetic scores above 6.25 and resolution above 512 pixels.Source: §3.2 · p. 4
  • Images were filtered for reference suitability, hierarchically categorized, and paired with editing instructions generated and checked by Gemini and human annotators.Task construction retained reference–instruction pairs judged free of visual breakdowns and filtered out nonsensical or trivial tasks.Source: §3.2 · pp. 4–5, Figure 2 · p. 4

3.3. Statistics in MultiBanana

  • Synthetic data expanded underrepresented categories, including animals from 111 to 915 images, clothing design from 40 to 396, and face expression from 10 to 146.The real dataset had 798 background images, about one-fourth of its total, while subtle person attributes such as expression and makeup remained limited.Source: §3.3 · pp. 5–6
  • The benchmark contains 3,769 instruction sets: 264 single-reference, 907 two-reference, and 2,598 multi-reference; Figure 4 shows task balance across reference counts.Each of the four multi-reference task types includes at least 70 sets, exceeding the reported count for multi-reference tasks in OmniContext and DreamOmni2.Source: §3.3 · p. 6, Figure 4 · p. 6

3.4. Evaluation Setting

  • Gemini 2.5 and GPT-5 rate generated images on five criteria scored from 1 to 10, with instruction alignment and reference consistency weighted three times each.The other criteria are background-subject match, physical realism, and visual quality.Source: §3.4 · p. 6

4. Experiments

The experiments compare open- and closed-source models across MultiBanana task types, reference counts, and difficult reference combinations. They find distinct failure patterns as references accumulate, and report that VLM judges correlate strongly with human ratings.

4. Experiments

  • The evaluation includes Nano Banana, GPT-Image-1, Qwen-Image-Edit-2509, DreamOmni2, and OmniGen2, with OmniGen2 tested only at five references.Gemini 2.5 and GPT-5 provide the reported average scores.Source: §4 · pp. 6–7, §4.1 · p. 7

4.1. Per-Task Evaluation

  • Background-replacement tasks score lower for both model families, and closed-source models also outperform open-source models in relatively simple editing scenarios.Table 3 reports average scores on a 10-point scale, averaging Gemini 2.5 and GPT-5 ratings.Source: Table 3 · p. 7, §4.1 · p. 7

4.2. Effect of the Number of References

  • Both model families generally lose score as reference counts rise, but closed models tend toward compositional distortion while open models often omit subjects.Closed models' Instruction Alignment and Reference Consistency decline relatively mildly, while open models' corresponding scores are low even at three references and deteriorate further; open models' overall visual quality remains relatively stable.Source: Figure 5 · p. 7, §4.2 · p. 7, Figure 6 · p. 8

4.3. Analysis of Difficult Reference Combinations

  • Most models score lower on cross-domain, scale-and-viewpoint, rare-concept, and multilingual tasks than on tasks without those combinations.The reported comparison identifies these combinations as challenging even for state-of-the-art closed models.Source: Figure 7 · p. 8, §4.3 · p. 8

4.4. Reliability of VLM Judge

  • GPT-5 and Gemini 2.5 show strong correlation with human ratings, and Qwen3-VL-8B-Instruct is reported as a reliable open-source alternative.The human comparison uses three evaluators and 32 benchmark samples.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Agentic Framework

  • In the agentic framework experiments, GPT's performance improves, whereas Nano Banana shows no improvement and sometimes degrades due to information loss from the initial instruction.The tested IPR setup has a generator create images and a planner rewrite prompts over multiple steps based on those images.Source: §4.5 · p. 8

5. Conclusion

MultiBanana evaluates multi-reference image generation across difficult reference combinations and increasing reference counts. The evaluation finds that models either omit instructed subjects or collapse while trying to follow instructions, and that heterogeneous reference conditions remain challenging.

5. Conclusion

  • The benchmark evaluates multi-reference generation and identifies omission or collapse as failure modes, with cross-domain, scale or viewpoint gaps, rare concepts, and multilingual inputs remaining challenging.The conclusion reports these challenges across different types of reference difficulty.Source: §5 · p. 8

A. Results of Qwen3-VL Judge

The appendix reports Qwen3-VL judge results and details the construction and preparation of the reference collection. It describes filtering unsuitable images and categorizing references to support task-specific combinations.

A. Results of Qwen3-VL Judge

B.1. Image Collection

  • The reference collection combines real LAION-5B images with synthetic images from Nano Banana and GPT-Image-1 to broaden coverage and diversity.Synthetic prompts vary across human, animal, object, and text-containing image categories.Source: §B.1 · p. 15

B.2. Image Filtering

  • Image filtering removes unclear foreground subjects, unsafe or unsuitable content, and nearly identical synthetic images; 48% of images remain after foreground-size and visibility filtering.Automated screening and human review excluded approximately 3% for inappropriate content.Source: §B.2 · p. 15

B.3. Category Classification

  • Hierarchical classification distinguishes human, object, and background attributes so images can be selected for task-specific reference requirements.The protocol applies structured annotations to variable real images; synthetic attributes can be taken directly from their generation prompts.Source: §B.3 · pp. 15–16

B.4. Task Construction

Task construction combines categorized references with generated and reviewed editing instructions, assigning retained tasks to difficulty categories. The benchmark also includes eleven two-reference task types with a minimum sample share for each.

B.4. Task Construction

  • Gemini generates editing instructions from sampled reference categories and examples, after which difficult-to-interpret, overly simple, or breakdown-inducing instructions are removed or reviewed.Remaining samples are assigned to cross-domain, scale-and-viewpoint, rare-concept, and multilingual-text categories. Examples in Figures 10 and 11 are described as almost fully solvable by advanced models such as Nano Banana.Source: §B.4 · p. 16, Figure 10 · p. 17, Figure 11 · p. 17

C. Further Statistics for MultiBanana

D. Comparison with Prior Benchmarks

Existing editing and multi-reference benchmarks are nearing saturation on high-performing models, while the VLM scores used for evaluation show positive cross-model correlations. The reported correlations support consistency between the evaluation criteria, though the supplied passages provide no correlation values.

D. Comparison with Prior Benchmarks

E. Reliability and Cost of VLM Judges

  • The authors use VLM judges as a time- and cost-efficient proxy for human evaluation and report that they correlate strongly with human ratings.The supplied passage refers to the main-text correlation result but does not give its value.Source: E. Reliability and Cost of VLM Judges · p. 18

E.1. Cross-VLM Correlation

  • GPT and Gemini evaluation scores are positively correlated across 10 subsets, supporting the consistency of the benchmark’s evaluation criteria.The mean correlation was computed across 10 partitions of evaluated Nano Banana and GPT-Image-1 results.Source: §E.1 · p. 18

E.2. Self-Consistency Analysis

The evaluation analysis reports stable VLM scores across seeds, sensitivity to fine-tuning gains, and model-specific strengths and weaknesses across tasks. Performance generally falls as reference counts or reference diversity increase, while a source-subset analysis finds no statistically significant bias from synthetic references.

E.2. Self-Consistency Analysis

  • Standard errors on the 1–10 scale stay below 0.1 across three seeds, with no increase as reference count rises, indicating self-consistent VLM judgments.The standard errors concern total scores averaged across GPT and Gemini evaluations of GPT-Image-1 and Nano Banana.Source: §E.2 · pp. 18–19, Table 10 · p. 19

E.3. Sensitivity Analysis

  • The evaluation metrics capture gains by Qwen-Image-Edit-2509 over Qwen-Image-Edit, but this sensitivity comparison uses single-reference tasks.Qwen-Image-Edit supports only a single input image.Source: §E.3 · p. 19, Table 11 · p. 19

E.4. Cost of VLM Judge

  • In January 2026, one evaluation costs approximately $44 with GPT-5 or $29 with Gemini 2.5; Qwen3-VL is reported as a reliable enough API-free alternative.Source: §E.4 · p. 19

F.1. Per-Task Evaluation

  • GPT-Image-1 leads in Instruction Alignment, while Nano Banana scores highest in single-reference Reference Consistency and is on par with GPT-Image-1 on several two-reference editing tasks.Nano Banana performs worse than the other models on background modification in Background Subject Match, Physical Realism, and Visual Quality.Source: §F.1 · p. 19

F.2. The Effect of the Number of References

  • All models’ performance decreases as reference images increase, but DreamOmni2 retains quality-related scores while tending toward the minimum Instruction Alignment and Reference Consistency scores.For X–1 Object + Global style tasks, Background Subject Match, Physical Realism, and Visual Quality do not show severe declines; Qwen-Image-Edit-2509 also produces perceptually invalid outputs when given four or more references, which may be out of distribution.Source: §F.2 · pp. 19–20

F.3. Difficult Reference Combination

  • Cross-domain references lower Reference Consistency across models, and scale or viewpoint differences also consistently reduce that metric.In cross-domain cases, models tend to distort individual reference attributes to achieve a consistent style; scale and viewpoint mismatches may also compromise fine details or pose adjustment.Source: §F.3 · p. 20
  • Rare concepts tend to receive lower Physical Realism scores, while multilingual text rendering lowers all reported metrics except Reference Consistency.The authors associate rare-concept errors with reproducing reference scale without appropriate adjustment, and multilingual declines with limited non-English text rendering; GPT-Image-1 declines less than other models on multilingual tasks.Source: §F.3 · pp. 20–21

F.4. Risks of Model Bias in Synthetic References

  • Mean scores across reference-source subsets remain within their respective 99% confidence intervals, indicating no statistically significant bias from synthetic references.The subsets group references from GPT-Image-1, Nano Banana, or both.Source: §F.4 · p. 21

F.5. Potential Conflict in Cross-Domain Task

The cross-domain evaluation shows that reference consistency and scene coherence need not always conflict, and Nano Banana can improve a prioritized criterion while maintaining the other. The appendix also describes real-world relevance and agentic frameworks, whose refinement experiments show different outcomes for Gemini and GPT.

F.5. Potential Conflict in Cross-Domain Task

  • A generated cross-domain image can score 7 for reference consistency and 9 for background-subject match, showing both criteria can be met together.The example places a photorealistic person in an anime background.Source: §F.5 · p. 21
  • Prioritizing either reference consistency or background-subject match lets Nano Banana improve the chosen criterion while maintaining the other.The comparison uses 20 randomly selected prompts with explicit instructions prioritizing one criterion.Source: §F.5 · p. 21, Table 15 · p. 21

F.6. Alignment with Real-World Use Cases

  • MultiBanana includes a task example presented as directly relevant to real-world use cases such as advertising.The figure caption identifies the example as a benchmark task.Source: §F.6 · p. 21, Figure 13 · p. 22

F.7. Agentic Inference

  • The agentic setups pair a Generator that produces images with a Planner that updates prompts and selects references; GPT and Gemini use different Generator–Planner pairings.GPT uses GPT-Image-1 as Generator and GPT-5 as Planner; Gemini uses Nano Banana and Gemini 2.5 Flash, respectively.Source: §F.7 · p. 22

F.7.1. Iterative Prompt Refinement (IPR)

  • In IPR, the Planner refines the prompt based on the generation result from the previous step.Source: §F.7.1 · p. 22

F.7.2. Context-Aware Feedback Generation (CAFG)

  • In CAFG, the Generator uses the previous generation result as context when producing the next image.Source: §F.7.2 · p. 22

F.7.3. Selective Reference Adaptation (SRA)

  • SRA has the Planner select references insufficiently reflected in the current output, so the Generator receives only those references as context.The approach is expected to reduce task complexity by adaptively decreasing the number of references.Source: §F.7.3 · p. 22

F.7.4. Experiments

  • GPT demonstrates consistent improvements across categories with IPR, whereas Nano Banana shows modest gains in Physical Realism and Visual Quality but unchanged or worse Instruction Alignment and Reference Consistency.The study limits refinement to three steps; the authors suggest Gemini’s planner progressively loses information from the original prompt.Source: §F.7.4 · p. 22

G. Implementations

This material describes MultiBanana’s implementation, its benchmark context and scope, and the prompts and criteria used for agentic refinement and VLM evaluation.

G. Implementations

  • MultiBanana provides a code repository and dataset, and specifies GPT-5 and Gemini API versions for judging, with image-generation APIs current as of November 2025 unless noted.The listed VLM endpoints are gpt-5-2025-08-07 and gemini-2.5-flash.Source: G. Implementations · p. 22

H.1. Controllable Text-to-Image Generation

  • The related-work discussion presents controllable text-to-image methods as using external conditioning modules or training-free approaches to support image-conditioned generation.It names ControlNet and T2I-Adapter as conditioning-module examples, and pix2pix-zero and prompt-to-prompt as training-free examples.Source: §H.1 · p. 23

H.2. Benchmarks for Reference-Based Generation

  • Compared with MultiRef-Bench [9], MultiBanana has 3,769 samples versus 1,990, supports up to 8 references versus 6, and uses raw images rather than boxes or masks.The paper describes raw-image inputs as more practical.Source: §H.2 · p. 23

H.3. Instruction-Following in LLMs

  • The paper relates multi-reference generation to LLM instruction following: both require satisfying multiple constraints, and more references can require reconciling conflicting visual cues.Source: §H.3 · p. 23

I. Limitations and Discussions

  • The benchmark covers static image generation, not video generation or editing, and uses natural-language placement instructions that the paper describes as inherently ambiguous.The discussion says structured layout inputs could enable finer-grained composition control.Source: §I · p. 23

J.1. Prompts of Agentic Framework

  • The IPR planner identifies underrepresented references and proposes a refined prompt; the next generation step uses that prompt with the references and prior output.The planner returns a prompt and indices of references insufficiently reflected in the generated image.Source: §J.1 · p. 24

J.2. Prompts of VLM as Judge

  • The judge scores five independent 10-point criteria: instruction alignment, reference consistency, background-subject match, physical realism, and visual quality.The prompt says an average score should be considered 4 points.Source: §J.2 · p. 25

Related papers

Loading 2511.22989v2…