PhyBench: A Physical Commonsense Benchmark for Evaluating Text-to-Image Models

1 Introduction

• Prior T2I benchmarks emphasize accuracy, safety, and privacy, while quantitative investigation of physical commonsense remains largely unexplored.The introduction argues that evaluation of this capability can help practitioners identify model strengths and weaknesses.Source: §1 · pp. 1–2
• PhyBench covers 4 physical-knowledge types, 31 scenarios, and 700 prompts, and evaluates 6 proprietary and open T2I models.Its categories are optics, mechanics, thermodynamics, and material properties.Source: §1 · p. 2, Figure 2 · p. 3
• PhyBench uses implicit prompts and fine-grained physical principles to assess whether models discern commonsense rather than merely follow explicit instructions.The prompt pipeline draws on textbooks and GPT4 to refine scenarios before GPT4 creates prompts.Source: §1 · p. 2
• PhyEvaler uses GPT-4o with item-specific scoring instructions and aligns well with human ratings.The framework is proposed to evaluate images generated for PhyBench.Source: §1 · p. 2
• Current T2I models struggle to depict physical commonsense except in optical scenarios, while making physical principles explicit in prompts improves image accuracy.The authors interpret this as a gap between following text instructions and reasoning about physical commonsense.Source: §1 · p. 2
• Advanced open-source models such as Stable Diffusion 3 and Stable Diffusion XL show a significant physical-commonsense gap relative to DALLE 3.Source: §1 · p. 2

2.1 Text-to-image generation

• Diffusion models iteratively refine random noise into images, while the paper evaluates four closed-source and two open-source T2I models that still face simple physical-scene challenges.The cited Stable Diffusion series is one example of diffusion models.Source: §2.1 · p. 3

2.2 Benchmarks for text-to-image generation

• Earlier benchmarks test counting, shapes, writing, compositional generation, safety, or bias; PhyBench instead uses implicit prompts to evaluate physical understanding.The position benchmark (Yang et al., 2024b) is also described as using implicit prompts to assess whether models possess required knowledge.Source: §2.2 · pp. 3–4

3 Our Benchmark: PhyBench

• PhyBench contains 700 prompts across four types of physical knowledge and 31 physical scenarios.Source: §3 · p. 4

3.1 PhyBench Construction

• Thermodynamics covers six depictable phase changes, while material properties cover six physical and seven chemical properties using common scenarios for reactions.Source: §3.1 · p. 4
• Researchers manually draft simple law-based scenarios, expand prompts with GPT-4 object substitutions, and filter samples; mechanics and optics each span six types.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2 Image Generation on PhyBench

• Each implicit prompt is generated four times by each of six T2I models to mitigate randomness in image generation.Source: §3.2 · p. 5

3.3 Evaluation Method

• Images receive separate scene scores for prompt adherence and physical-correctness scores for whether depicted principles are correct; GPT-4o applies scene-specific criteria and aligns closely with human ratings.Source: §3.3 · p. 5
• For spatial judgments, GroundingDino bounding boxes are added to GPT-4o scoring prompts to mitigate errors from its hallucinations.Source: §3.3 · pp. 5–6

4.1 Experiments Setup

• Machine and human average scores are reported for scene and physical correctness, with average per-sample Pearson correlation measuring their alignment.Source: §4.1 · p. 6

4.2 Quantitative Evaluation

• Adding object bounding boxes brings average machine scores closer to human evaluations and significantly improves machine–human score correlation.Source: §4.2 · p. 6
• Midjourney exceeds 2.5 in both machine and manual optics scores; nearly all models score highly on simple light laws.Source: §4.2 · p. 6
• 1.31 is Gemini’s score for thermodynamic scenarios, and all models struggle with these prompts.Source: §4.2 · p. 6
• 1.28 is the highest physical-properties score and 0.84 the top chemical-properties score across current models.Source: §4.2 · p. 6
• 1.71 is DALL-E 3’s mechanics physics score, while every other model scores below 1.5.Source: Table 3 · p. 6, §4.2 · p. 6

4.3 Takeway Findings

• Models generally render the designed scenes, but physical correctness is poor outside optics, and open-source models lag proprietary models in physical-commonsense understanding.Source: §4.3 · p. 6

5 Discussion

• PhyBench scoring uses scenario-specific standards generated from template criteria, with GroundingDino bounding boxes added for spatial-relationship assessments.The detailed standards are then used to score each image.Source: Figure 5 · p. 7
• Current T2I models primarily provide superficial text-to-image translation and lack internal knowledge of physical commonsense.The discussion reports robustness analyses involving prompt rewriting and different T2I models.Source: §5 · pp. 6–7

5.1 Ablation Study

• Explicit prompts improve performance across all models; in thermodynamic scenarios, SDXL with explicit prompts outperforms every model using implicit prompts, while DALL-E 3 rises from 1.09 to 2.46.The thermodynamic scores are reported in Table 6.Source: §5.1 · p. 8, Table 6 · p. 8
• RPG's low image-generation success prevents quantitative comparison, so the authors analyze it qualitatively.They compare images generated with original and rewritten prompts.Source: §5.1 · p. 8
• RPG struggles with physical scenarios even under rewritten prompts; the authors attribute this to focus on layout and object details, limited understanding of physical laws, and insufficient training data for such images.Source: §5.1 · p. 8

• PhyBench provides 700 manually reviewed prompts across four physical-knowledge areas and 31 scenarios, and PhyEvaler evaluates six models with strong correlation to human evaluations.Source: §6 · p. 8
• The evaluated T2I models struggle to generate images that adhere to physical commonsense, despite excelling at text-to-image translation.Source: §6 · p. 8

• Extensive experiments indicate that current T2I models lack understanding of fundamental physical commonsense.Source: §7 · p. 9
• Prompt rewriting and agent frameworks do not address the underlying physical-commonsense issue.Source: §7 · p. 9
• Developing a T2I model that autonomously understands and applies knowledge is left for future work.Source: §7 · p. 9

• Even in the simplest scenarios, current T2I models are prone to errors.Source: A Experiments · p. 11
• For a wooden cylinder in front of a mirror, DALL-E 3 and Midjourney depict the reflected objects incorrectly.Source: A Experiments · p. 11
• For objects in a water tank, DALL-E 3, Gemini, and Midjourney misrepresent at least one object's state, while DALL-E 2 omits an object.Source: A Experiments · p. 11

A.1 Experiments Setup

• Prompts are used directly for Midjourney, Gemini, and DALLE 2; DALL-E 3 and Gemini receive a prefix to prevent rewriting, while Stable Diffusion prompts are reformatted by GPT-4.Source: §A.1 · p. 11
• Each image receives scene-accuracy scores from 0–2 and physical-law-adherence scores from 0–3, averaged across 10 human evaluators.Source: §A.1 · p. 11
• Figure 10 contrasts three successful examples with three failures and marks corrected sections after prompt rewriting.Source: Figure 10 · p. 11

B.1 Case Analysis

• Reflection, light scattering, and weightlessness are depicted correctly in the examples, phenomena the authors say models can learn from natural images.Source: §B.1 · p. 11
• In the buoyancy example, DALL-E 3 places both an apple and rock at the tank bottom, but a rewritten prompt yields an image with the apple floating and rock sunk.Source: §B.1 · pp. 11–12
• A basic hyperbaric-chamber prompt produces a smooth can, whereas a revised prompt specifying high external pressure yields a crumpled can.Source: §B.1 · p. 12
• For baking soda mixed with litmus solution, the original prompt produces incorrect milky-white and green liquid, while a revised prompt yields blue liquid.Source: §B.1 · p. 12