Source-linked AI summary
WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation
Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Fanqing Meng, Kunpeng Ning, Bin Zhu, Li Yuan
TL;DR
Existing T2I evaluations largely measure realism and shallow text-image alignment, providing limited evidence about complex semantic understanding and world-knowledge integration. WISE introduces a 1,000-prompt benchmark across 25 subdomains and WiScore, a weighted evaluation protocol emphasizing consistency. Testing 20 models reveals persistent deficiencies in applying world knowledge during image generation, including among unified multimodal models.
Problem
Existing evaluations insufficiently assess complex semantic understanding and world-knowledge integration because they emphasize realism and surface-level text-image alignment.
Method
WISE evaluates knowledge-grounded image generation with 1,000 prompts across three domains and 25 subdomains, using WiScore to weight consistency, realism, and aesthetic quality.
Results
Testing 20 models reveals clear deficiencies in complex semantic understanding and world-knowledge integration, including in unified multimodal models.
Takeaways & Limitations
Unified multimodal models’ stronger understanding capabilities do not fully translate into advantages for world-knowledge-grounded image generation.
Takeaways & Limitations
WISE covers only a sample of knowledge domains, and some prompts may span multiple categories, creating ambiguity in cross-category analysis.
Abstract
from arXiv · showhide
Text-to-Image (T2I) models are capable of generating high-quality artistic creations and visual content. However, existing research and evaluation standards predominantly focus on image realism and shallow text-image alignment, lacking a comprehensive assessment of complex semantic understanding and world knowledge integration in text-to-image generation. To address this challenge, we propose \textbf{WISE}, the first benchmark specifically designed for \textbf{W}orld Knowledge-\textbf{I}nformed \textbf{S}emantic \textbf{E}valuation. WISE moves beyond simple word-pixel mapping by challenging models with 1000 meticulously crafted prompts across 25 subdomains in cultural common sense, spatio-temporal reasoning, and natural science. To overcome the limitations of traditional CLIP metric, we introduce \textbf{WiScore}, a novel quantitative metric for assessing knowledge-image alignment. Through comprehensive testing of 20 models (10 dedicated T2I models and 10 unified multimodal models) using 1,000 structured prompts spanning 25 subdomains, our findings reveal significant limitations in their ability to effectively integrate and apply world knowledge during image generation, highlighting critical pathways for enhancing knowledge incorporation and application in next-generation T2I models. Code and data are available at \href{https://github.com/PKU-YuanGroup/WISE}{PKU-YuanGroup/WISE}.
1. Introduction
Existing T2I evaluations emphasize realism and shallow text-image alignment, leaving complex semantic understanding and world-knowledge integration insufficiently assessed. WISE addresses this gap with a broad benchmark and reports persistent deficiencies across dedicated and unified models.
- Existing T2I benchmarks use simple prompts that do not effectively challenge complex semantic understanding or world-knowledge integration.
- FID emphasizes image realism, while CLIP-based metrics struggle with fine-grained semantics and complex reasoning.
- WISE evaluates implicit semantic understanding and world-knowledge integration through 1,000 questions spanning three domains and 25 sub-areas.
- The evaluation covers 20 models, including 10 dedicated T2I models and 10 unified multimodal models.
- Results reveal deficiencies in complex semantic understanding and world-knowledge integration across existing models, including unified multimodal systems.
2. Related Works
T2I research covers dedicated generators and unified multimodal systems, which process textual and visual inputs and support cross-modal generation and understanding.
- T2I models comprise dedicated systems and unified multimodal models, and the paper reviews both lines before discussing evaluation protocols.
I. Generate Input Pairs
WISE begins with indirect, knowledge-intensive inputs and evaluates generated content across consistency, realism, and aesthetics. Its motivating examples expose conflicts that straightforward benchmarks overlook.
- I. Generate Input Pairs: WISE uses indirect prompts such as “A candle in space” and “A close-up of a maple leaf in summer” to initiate evaluation.
- I. Generate Input Pairs: The benchmark contrasts shallow image-text evaluation with prompts requiring world knowledge and reasoning.
- I. Generate Input Pairs: WISE evaluates generated images using Consistency, Realism, and Aesthetic Quality.
- I. Generate Input Pairs: Its representative cases assign 0 consistency when a candle in space violates oxygen-dependent combustion or a summer maple leaf conflicts with seasonal botany.
- I. Generate Input Pairs: The benchmark is motivated by shortcomings in existing metrics and by limited evidence that unified multimodal understanding transfers to image generation.
3. The World Knowledge-Informed Semantic Evaluation (WISE) Benchmark
WISE constructs knowledge-intensive prompts across broad world-knowledge domains and pairs them with a multifaceted evaluation protocol. WiScore emphasizes semantic consistency while retaining realism and aesthetic quality.
- 3. The World Knowledge-Informed Semantic Evaluation (WISE) Benchmark: Existing benchmarks and alignment metrics focus mainly on straightforward prompts and surface-level semantics.
- 3.1. Building a Benchmark Based on World Knowledge: WISE contains 1,000 prompts across cultural common sense, spatiotemporal reasoning, and natural science, divided into 25 subdomains.
- 3.1. Building a Benchmark Based on World Knowledge: The dataset draws on educational materials, encyclopedias, commonsense problem sets, and LLM-generated data, followed by human refinement.
- 3.1. Building a Benchmark Based on World Knowledge: Prompts often omit target entities and instead provide clues, relations, or conditions that require world-knowledge retrieval before generation.
- 3.2. Discovering the Deep Visual Language Alignment Bottlenecks of Traditional Evaluation: WiScore combines Consistency, Realism, and Aesthetic Quality as α1 × Consistency + α2 × Realism + α3 × Aesthetic Quality.
- 3.2. Discovering the Deep Visual Language Alignment Bottlenecks of Traditional Evaluation: The paper sets α1 = 0.7, α2 = 0.2, and α3 = 0.1, prioritizing consistency while retaining realism and aesthetics.
- 3.2. Discovering the Deep Visual Language Alignment Bottlenecks of Traditional Evaluation: WiScore most closely aligns with human assessment among the evaluated metrics.
4. Evaluation Results
WISE evaluation shows substantial weaknesses in current T2I models’ world-knowledge use, while unified multimodal architectures and CoT achieve stronger performance. Rewriting complex prompts also improves scores, indicating prompt comprehension contributes to the difficulty.
- Experiment Settings: 20 models were evaluated: 10 dedicated T2I models and 10 unified multimodal models.
- Main Results: Most models score below 0.6 on WiScore, revealing deficiencies in complex semantic understanding and world-knowledge use, especially among dedicated T2I models.
- Main Results: Science prompts are generally harder, with Chemistry requiring implicit reasoning about properties, reactions, colors, and corrosion states.
- Main Results: AR+Diffusion deep- and shallow-fusion unified models outperform dedicated T2I models, achieving the best architectural results.
- Main Results: BAGEL achieves the best Table 1 performance when Chain of Thought assists visual generation, supporting CoT as a knowledge-elicitation mechanism.
- Rewritten Prompts: Nearly all models improve substantially after prompt rewriting, with Qwen-Image reaching an overall WiScore of 0.88.
- Rewritten Prompts: Janus-Pro-1B gains +0.34 after rewriting, while rewritten BAGEL reaches 0.73, close to BAGEL+CoT’s original-prompt score of 0.70.
5. Conclusion
WISE evaluates whether T2I systems can use world knowledge beyond direct word-to-pixel mappings and finds clear weaknesses, particularly in dedicated T2I models. Rewriting and CoT results point to prompt interpretation, knowledge elicitation, and tighter understanding-generation integration as important directions.
- WISE benchmarks 1,000 knowledge-intensive questions across domains and evaluates 20 dedicated and unified generative models.
- The evaluated models show clear weaknesses in applying world knowledge during image generation, especially dedicated T2I models.
- Prompt rewriting and Chain of Thought results identify prompt interpretation, knowledge elicitation, and understanding-generation integration as important future directions.
Impact Statement
WISE provides a structured framework for evaluating world-knowledge integration and complex semantic understanding across cultural, spatio-temporal, and natural-science domains. Its coverage is organized into detailed subcategories spanning cultural knowledge, spatial and temporal relations, biological phenomena, physical principles, and chemistry.
- WISE is designed to support more factually grounded and intelligent generative systems through structured evaluation across natural sciences and cultural domains.
- The benchmark covers Cultural Common Sense, Spatio-Temporal Reasoning, and Natural Science as its three main domains.
- Cultural Common Sense: Cultural Common Sense includes art, celebrity, life, festivals, sports, religion-related knowledge, crafts, construction, animals, and plants.
- Cultural Common Sense: Cultural coverage is audited by grouping prompts into global/neutral, Western, and non-Western categories, with the cultural subset balanced between Western and non-Western prompts.
- Spatio-Temporal Reasoning: Spatio-Temporal Reasoning evaluates horizontal and longitudinal time, geographical location, relative position, and different viewpoints.
- Natural Science: Natural Science spans biological state and behavior, mechanics, thermodynamics, optics, physical properties, combustion, metal corrosion, and solution chemical reactions.
- Natural Science: Its science categories include phase transitions, light phenomena, electrical conductivity, flame characteristics, corrosion, and acid-base, redox, and precipitation reactions.
F.1. Rewrite Gain Analysis
The rewrite-gain analysis separates improvements attributable to removable prompt-comprehension difficulty from persistent knowledge-informed generation difficulty. It measures mean rewriting gains across models and partitions originally difficult prompts by threshold.
- The analysis computes across-model mean rewriting gains for each prompt and applies threshold-based partitions to originally difficult prompts.
G. Failure Case Analysis
WISE failure analysis identifies errors beyond surface prompt parsing, including missing associations, violated scientific constraints, and inaccurate visualization of fine-grained states; prompt rewriting does not fully resolve them.
- Failure patterns: Three failure categories dominate: missing implicit associations, scientific constraint violations, and inaccurate visualization of fine-grained states.These patterns indicate that models can recognize prompt words yet fail to retrieve associations, respect physical conditions, or render precise states.
- Prompt rewriting: 74.0% of Chemistry prompts remain ineffective after rewriting, indicating errors beyond indirect wording.The remaining difficulty involves mapping chemical states and material properties into images.
- Implicit associations: For Mother’s Day plant prompts, models may generate generic flowers instead of the culturally associated carnation.The error reflects failure to retrieve the real-world association needed to identify the visual target.
- Scientific constraints: For candles in outer space, models may render a normal flame despite the oxygen-dependent condition required for combustion.The image can appear visually plausible while contradicting a scientific constraint implied by the scenario.
- Fine-grained states: For galvanized steel with early moisture-induced corrosion, models may produce generic red rust instead of localized white corrosion products.This is a fine-grained state-mapping error linked to zinc coating damage.
I. Human Evaluation Details
WiScore is validated through human ratings that use the same three dimensions and weighting scheme as the automated protocol.
- Annotation protocol: Five independent annotators evaluate each image for consistency, realism, and aesthetic quality.Annotators were undergraduate- or PhD-level participants and could use search engines for factual verification.
- Scoring: The final human score averages annotator ratings using WiScore’s weighting scheme.This aligns human evaluation with the dimensions used by WiScore.
J. Additional Architecture Evaluation
Additional evaluations show progress among recent autoregressive models, while AR+Diffusion systems remain stronger overall; unified-resolution testing leaves the ranking trend unchanged.
- Architecture comparison: Recent autoregressive models substantially outperform early AR baselines, including Janus, VILA-U, and Emu3.LongCat-Next, Emu3.5, and NextFlow reach substantially higher WiScore than these earlier models.
- Architecture comparison: AR+Diffusion models remain stronger overall in the supplementary architecture evaluation.This comparison follows the observed gains of recent autoregressive models over early baselines.
- Resolution confound check: Under a unified 512×512 resolution, the overall WISE ranking trend remains unchanged.The only observed change is SD-3.5-medium exceeding SD-3.5-large by 0.01.