GIR-Bench: Versatile Benchmark for Generating Images with Reasoning

1 INTRODUCTION

• Earlier benchmarks focused mainly on object attributes and composition, while existing reasoning benchmarks remain limited in evaluation dimensions and protocols.They do not adequately capture unified models’ full capabilities.Source: §1 · p. 2
• Existing evaluations cannot quantify knowledge and reasoning transfer between understanding, generation, and editing, and often rely on MLLM judges.Such dependence couples evaluation results to multimodal models’ own biases and limitations.Source: §1 · p. 2
• Unified models show a persistent understanding–generation gap, even though integrating understanding with generation enables more complex visual tasks.The benchmark also reveals a performance gap between unified and generation-only models on reasoning-driven visual tasks.Source: §1 · p. 3
• GIR-Bench evaluates unified multimodal models across three perspectives using diverse tasks and task-specific evaluation pipelines.The pipelines are designed to provide fine-grained, interpretable metrics beyond MLLM-as-a-Judge evaluation.Source: §1 · pp. 2–3

2.1 GIR-BENCH-UGC

• GIR-Bench-UGC pairs implicit prompts and reference images for 300 real-world entities to assess understanding–generation consistency.Generation is evaluated by average DINOv3 feature similarity to reference image sets; understanding uses reference images for visual question-answering cases.Source: §2.1 · p. 4, Figure 3 · p. 4

2.2 GIR-BENCH-T2I

• GIR-Bench-T2I uses 300 prompts across numerical reasoning, spatial layout, and text rendering, with deterministic tasks and verifiable ground truth.Numerical reasoning requires all detected object counts to match; spatial relations are checked using bounding boxes.Source: §2.2 · pp. 4–5
• For implicit text-rendering prompts, the word-level continuous substring score rewards ground-truth words covered by continuous character spans while allowing extra text.The score divides the number of covered ground-truth words by the number of words in the ground truth.Source: §2.2 · p. 5

2.3 GIR-BENCH-EDIT

• GIR-Bench-Edit evaluates visual puzzles, Sudoku logic, and reasoning perception using input images and corresponding ground truths.Evaluation uses normalized FID for puzzle reconstruction, digit-and-position accuracy for Sudoku, and mask IoU for reasoning perception.Source: §2.3 · pp. 5–6

3.1 EXPERIMENT SETTINGS

• 21 representative models were evaluated across multimodal understanding, image generation, image editing, and unified multimodal categories.Source: §3.1 · p. 6

3.2 EVALUATION ON REAL-WORLD UNDERSTANDING AND GENERATION

• Across models, implicit prompts produced marked performance declines relative to category inputs, pointing to difficulty transferring reasoned constraints rather than rendering named entities.Source: Figure 4 · p. 7, §3.2 · p. 7
• BLIP3o achieved the highest overall understanding score of 0.974, while Janus-Pro scored 0.874; the authors conclude basic recognition and reasoning are not the main bottlenecks.Source: §3.2 · p. 7

3.3 EVALUATION ON REASONING-CENTRIC TEXT-TO-IMAGE GENERATION

• 0.362 was the highest numerical-reasoning score, achieved by Gemini-2.5-Flash-Image; BAGEL w/CoT scored 0.249 versus BAGEL’s 0.057.Source: §3.3 · p. 7
• Gemini-2.5-Flash-Image scored 0.759 in spatial layout; BAGEL w/CoT scored 0.448, above Qwen-Image at 0.368 and Ovis-U1 at 0.356.Source: §3.3 · p. 7
• GPT-Image-1 scored 0.813 and Gemini-2.5-Flash-Image 0.806 on implicit text rendering, while BAGEL w/CoT did not improve over BAGEL (0.120 vs. 0.163).Source: §3.3 · pp. 7–8
• Unified models outperform generation-only systems on reasoning-centric text-to-image generation, while editing gaps are narrower and all models remain constrained.Even the strongest proprietary models often fail at fine-grained local control and pixel-level information maintenance.Source: §3.3 · p. 7, Table 2 · p. 8, §3.4 · p. 8

3.4 EVALUATION ON REASONING-CENTRIC IMAGE EDITING

• Editing results show constrained abilities across models, including frequent failures by the strongest proprietary systems in fine-grained local control and pixel-level information maintenance.The three tasks are visual puzzle, visual logic, and reasoning perception.Source: §3.4 · p. 8

• Generation outputs satisfy only part of coupled constraints, while editing examples show unfaithful puzzle reconstructions, digit errors, and failures to preserve regions.Source: §4 · p. 9
• Qwen-Image renders an entity when named but fails on its implicit prompt; BAGEL w/CoT infers “I Have a Dream” yet fails to render it directly.Refeeding BAGEL’s expanded prompt produces the target text, suggesting transfer into generation—not inference alone—is the challenge.Source: §4 · p. 9
• Judge-based scores can reward superficial resemblance or biased reasoning, whereas task-specific metrics penalize ground-truth mismatches and yield more consistent, interpretable evaluations.Source: §4 · p. 10, Figure 9 · p. 10
• The analysis identifies three proposed drivers: reasoning localized in the LLM, information loss through limited condition tokens, and too little interleaved reasoning-trace data.The paper describes these as factors identified through its quantitative ablations and qualitative case studies.Source: §4 · p. 10

5 CONCLUSION

• GIR-Bench evaluates unified multimodal models across understanding, generation, and editing using explicit task-specific metrics rather than MLLM-as-a-Judge.The experiments find unified models surpass generation-only models on reasoning-intensive tasks but still struggle to transfer reasoning into faithful visual outputs.Source: §5 · p. 10

A ETHICS STATEMENT

• Benchmark images come from public open-source datasets or automated generation pipelines and are filtered to avoid inappropriate or harmful content.Source: A ETHICS STATEMENT · p. 13
• The authors withhold prompts or data that could enable malicious generation and say downstream deployment requires appropriate safeguards.Source: A ETHICS STATEMENT · p. 13

• Unified multimodal research has shifted from modality-specific architectures toward models handling understanding and generation across modalities.Source: §B.1 · p. 14
• Approaches discussed include early-fusion token transformers, pure autoregressive modeling, learnable query interfaces, and diffusion-based unified frameworks.Source: §B.1 · p. 14
• The cited systems include models with interleaved multimodal pretraining and emergent reasoning or image-manipulation capabilities, motivating comprehensive reasoning-centric benchmarks.Source: §B.1 · p. 14

B.2 BENCHMARKS FOR UNIFIED MULTIMODAL MODELS

• Earlier benchmarks span compositional alignment, surface-level editing, knowledge-image alignment, and reasoning prompts, but retain limits in reasoning coverage or evaluation that separates comprehension from generation.The described approaches include MLLM-based judging and evaluations that cannot disentangle understanding from generation.Source: §B.2 · p. 15

C IMAGE EDITING INSTRUCTIONS

• Each GIR-Bench-Edit instruction specifies an editing objective, constraints to preserve, and an output format, with placeholders filled by case-specific descriptions or categories.Figure 12 shows Edit examples, including segmenting a described region, solving Sudoku, and restoring a jigsaw puzzle.Source: C IMAGE EDITING INSTRUCTIONS · pp. 15–16, Figure 12 · p. 16

D MORE QUALITATIVE RESULTS

• Figure 10 presents examples from GIR-Bench-UGC, Figure 11 from GIR-Bench-T2I, and Figure 12 from GIR-Bench-Edit.These figures provide representative qualitative results from all three benchmark components.Source: D MORE QUALITATIVE RESULTS · p. 16
• The cases illustrate typical challenges faced by current models across the benchmark's three components.The passage characterizes them as representative qualitative results.Source: D MORE QUALITATIVE RESULTS · p. 16
• The examples highlight reasoning–generation misalignment across the different tasks.The passage says this pattern is observed across the benchmark components.Source: D MORE QUALITATIVE RESULTS · p. 16

E ROBUSTNESS AND VALIDATION OF AUTOMATED EVALUATION METRICS

• ρ ≈0.96 Pearson correlation with human annotation; MLLM-as-a-Judge approaches show lower consistency and score inflation, especially in lower-performance regimes.The correlation analysis aggregates scores across GIR-Bench tasks; Figure 13 depicts MLLM judge points above the diagonal in the lower-left quadrant.Source: E ROBUSTNESS AND VALIDATION OF… · p. 16, Figure 13 · p. 17
• Rare false negatives arise mainly with highly stylized or occluded objects, or severely distorted text; these edge cases do not disproportionately affect model rankings.The paper reports that relative rankings remain consistent and that the significant performance gap is robust to these metric fluctuations.Source: E ROBUSTNESS AND VALIDATION OF… · p. 17

F EXTENDED DISCUSSION: THE GAP BETWEEN UNDERSTANDING AND GENERATION

• Quantitative ablations and qualitative case studies are used to analyze the understanding–generation gap, identify its root causes, and propose research directions.The analysis draws on Figure 4 and Figure 5.Source: F EXTENDED DISCUSSION: THE GAP BETWEEN… · p. 17

F.1 ROOT CAUSES OF THE PERFORMANCE GAP

• Models may know the relevant entity yet fail to activate that knowledge from an implicit prompt; the LLM's reasoned state may not reach or guide the image generator.The Charging Bull case contrasts success when the entity is explicitly named with failure when it must be inferred.Source: §F.1 · pp. 17–18
• A lightweight interface between an autoregressive LLM and generation head can compress semantic state, diluting or losing fine-grained constraints such as exact counts.The paper describes this information bottleneck in heterogeneous architectures.Source: §F.1 · p. 18
• Static image-text pre-training pairs omit generation processes, leaving models without interleaved reasoning-trace examples for decomposing abstract instructions into visual plans.The proposed trace format links a reasoning step, an intermediate visual state, and a refined image.Source: §F.1 · p. 18

F.2 FUTURE DIRECTIONS

• 0.057 to 0.249 accuracy on Numerical Reasoning for BAGEL w/ CoT; the paper proposes explicit textual plans as a bridge from implicit logic to generator-readable constraints.The paper reports this change as evidence that externalizing reasoning boosts performance.Source: §F.2 · p. 18
• Natively unified architectures, with text and image tokens treated equally in one backbone, are proposed to remove the heterogeneous interface bottleneck.The proposed design would let generation attend directly to uncompressed reasoning states.Source: §F.2 · p. 18
• Reasoning-trace datasets are proposed to supervise the reasoning-to-generation mapping with execution traces as well as final ground-truth images.The passage contrasts this with relying solely on end-to-end alignment.Source: §F.2 · p. 18

G CLARIFICATION FOR THE TEXT RENDERING METRIC (swc)

• In Case B, swc credits valid target text despite chatty additions, while sacc fails because of its length penalty.The comparison covers different generation scenarios for the ground truth “Make It Happen”.Source: Table 3 · p. 18, G CLARIFICATION FOR THE TEXT RENDERING… · p. 19
• In Case C, swc identifies legible words in merged OCR output, while sacc penalizes missing whitespace.Tight layouts or artistic fonts can cause the OCR outputs to merge.Source: Table 3 · p. 18, G CLARIFICATION FOR THE TEXT RENDERING… · p. 19
• In Case D, both metrics penalize missing content, and swc rejects partial matches rather than crediting incomplete generations.The authors describe swc as a strict filter; “Make It Hap” is not credited.Source: Table 3 · p. 18, G CLARIFICATION FOR THE TEXT RENDERING… · p. 19