PaperlayerView exact revision on arXiv

Source-linked AI summary

TerraVis: Towards Evaluation of World-Grounded Visual Consistency in Text-to-Image Generation via MLLM Workflows

Shuai Fu, Jing Gu, Jian Zhou, Zicheng Duan, Gengze Zhou, Qi Wu

arXiv:2610.02959v1cs.CV

TL;DR

Existing text-to-image metrics can overlook real-world plausibility violations in otherwise appealing or well-aligned images. TerraVis evaluates world consistency through a taxonomy-guided, multi-stage workflow. Across two benchmarks, it achieves the strongest agreement with human judgments among existing metrics.

  • Problem

    Existing metrics do not directly assess whether generated images are visually consistent with real-world constraints, despite their importance for plausible depictions.

    Source: §1 · pp. 1–2

  • Method

    TerraVis uses a taxonomy-guided workflow to check eligibility, detect violations, classify severity, and aggregate a world-consistency score.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Across COCO-T2I and GenAI-Bench, TerraVis achieves the strongest agreement with human judgments, outperforming existing quality, alignment, and preference metrics.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    World consistency is a complementary evaluation dimension for assessing whether generated images conform to real-world constraints.

    Source: §1 · p. 2

    The multi-stage workflow is less efficient than single-pass metrics, and violation detection may produce false positives or miss genuine violations.

    Source: §4.6 · p. 10

Abstract

from arXiv · show

Recent text-to-image models have made substantial progress in photorealism, aesthetics, and text-image alignment. Yet visually appealing images can still violate real-world plausibility, exhibiting malformed object structures, impossible anatomy, physically implausible interactions, or inconsistent spatial relationships. Such failures are not well captured by existing fidelity, aesthetics, preference, or alignment metrics. To address this gap, we introduce TerraVis, a framework for evaluating world-grounded visual consistency in generated images. TerraVis defines a structured taxonomy of world-consistency violations spanning object-, interaction-, and scene-level failures, and employs a multi-stage evaluation framework to identify and quantify them. Given an image, TerraVis first uses an MLLM to assess its eligibility for evaluation, then detects violations across 18 taxonomy-defined types and classifies them as minor or major to derive an overall world-consistency score. Across diverse open-source and proprietary text-to-image models on two widely used benchmarks, TerraVis achieves the strongest correlation with human judgments of world consistency among existing metrics. Our benchmark results further show that models that achieve strong performance on conventional metrics can still exhibit substantial world-consistency failures. These findings highlight world consistency as a complementary evaluation dimension and demonstrate that TerraVis enables systematic quantification, diagnosis, and comparison of such failures. Our code is publicly available at https://github.com/ShyFoo/TerraVis.

1 Introduction

Existing text-to-image metrics do not directly assess whether images obey real-world plausibility, even when they score highly on conventional dimensions. TerraVis introduces a complementary world-consistency evaluation framework and reports stronger correlation with human judgments than existing metrics.

1 Introduction

  • High photorealism, text-image alignment, or preference scores can coexist with implausible structures, interactions, and spatial relationships.Such failures can undermine reliability when images are expected to depict the world plausibly.Source: §1 · pp. 1–2
  • World consistency evaluates whether depicted entities, structures, interactions, and phenomena are visually plausible with respect to the real world.It is presented as complementary to aesthetics, alignment, and preference rather than a replacement for them.Source: §1 · p. 2
  • TerraVis uses a taxonomy of object-, interaction-, and scene-level violations in a multi-stage MLLM workflow to detect, classify, and score inconsistencies.Its contributions focus on perceptible violations of structural, spatial, biological, and physical constraints.Source: §1 · p. 2
  • TerraVis correlates more strongly with human world-consistency judgments than existing quality, alignment, and preference metrics.The framework also outperforms a holistic MLLM baseline using the same taxonomy and instructions; its severity-aware exponential aggregation beats uniform pass-rate aggregation.Source: §1 · p. 2

2 Related Work

Related work covers prompt-conditioned alignment and preference measures alongside prompt-independent technical-quality and aesthetic metrics. The cited work also distinguishes broader world-consistency evaluation from narrower counting-hallucination assessment.

2.1 Prompt-Conditioned Metrics for Generated Images

  • Alignment metrics such as CLIPScore [14] measure text-image semantic similarity, while preference scores such as ImageReward [56] and PickScore [21] capture human judgments of overall quality.Preference metrics are typically trained on large-scale human preference data and complement semantic alignment measures.Source: §2.1 · p. 3

2.2 Prompt-Independent Metrics for Generated Images

  • Technical metrics such as NIQE, BRISQUE, and MUSIQ [20] assess blur, compression artifacts, and distortion, while aesthetic metrics [44] assess visual appeal.The passage says technical metrics have diminished importance as modern text-to-image models generally no longer show obvious low-quality artifacts.Source: §2.2 · p. 3
  • Better FID or MUSIQ scores do not necessarily correspond to fewer counting hallucinations, a specific form of real-world consistency violation [10].World consistency is described as a prompt-independent target covering a broader range of visual failures than counting errors alone.Source: §2.2 · p. 3

3 Evaluation of World-Grounded Visual Consistency Using TerraVis

TerraVis defines world consistency as visual conformity to real-world constraints and organizes violations into a three-level taxonomy. Its scoring protocol combines eligibility checks, violation detection, severity classification, and deterministic aggregation, with MLLMs used to instantiate the judgment stages.

3.1 World-Grounded Visual Consistency

  • World consistency measures how a depiction of a macroscopic object or scene conforms to real-world constraints, including anatomy, interactions, spatial relations, and scene compatibility.Stylized images can be assessed when identifiable real-world content supports plausibility judgments; abstract, symbolic, informational, interface, layout-based, and microscopic content is excluded.Source: §3.1 · pp. 3–4

3.2 TerraVis Violation Taxonomy

  • The taxonomy specifies 18 violation types: seven object-level, eight interaction-level, and three scene-level types.Figure 2 illustrates representative examples, with red ellipses marking observed violation regions.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 TerraVis Scoring Framework

  • The scoring framework checks eligibility, detects taxonomy-defined violations, classifies severity by semantic centrality, and computes a continuous score from major and minor counts.The score is S = exp[−λ(Nmaj + αNmin)]; higher scores indicate stronger consistency, and eligible images with no detected violations receive S = 1. The exponential design penalizes the first violations more strongly, with diminishing marginal decreases.Source: §3.3 · pp. 5–6

3.4 MLLM-based Implementation and Annotation Design

  • The MLLM implementation asks targeted questions in sequence rather than requesting a holistic score, and contrasts that workflow with holistic TerraVis-H.Human reference scores use a five-point Likert scale plus N/A, with three independent annotators rating each image.Source: §3.4 · p. 6

4 Experiment Results

TerraVis is evaluated against human judgments, existing metrics, and alternative evaluation designs across two text-to-image benchmarks. It shows stronger agreement than existing metrics, while model rankings, ablations, case studies, and limitations qualify that result.

4.1 Experiment Setup

  • The experiments use prompts from COCO-T2I and GenAI-Bench, five text-to-image models, and six representative metrics plus GPTScore as a holistic MLLM baseline.COCO-T2I contributes 200 prompts and GenAI-Bench 1,600; human correlation analysis uses a randomly selected 50% subset of generated images.Source: §4.1 · pp. 6–7

4.2 Correlation with Human Judgments

  • TerraVis outperforms all existing metrics by large margins on both benchmarks in correlation with human judgments of world consistency.Among existing metrics, RAHF has the strongest correlations: ρ = 0.29 on COCO-T2I and 0.37 on GenAI-Bench.Source: Table 2 · p. 7, §4.2 · p. 8
  • On COCO-T2I, TerraVis improves over TerraVis-H from ρ = 0.28, τ = 0.24 to ρ = 0.44, τ = 0.37.TerraVis-H has high precision but low recall, whereas TerraVis improves recall while maintaining high precision.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 Multi-Dimensional Evaluation of T2I Models

  • GPT Image 1.5 ranks first in 9 of 16 comparisons, but FLUX.2-dev leads world consistency on GenAI-Bench by both TerraVis and human judgments.The rankings compare five models across technical quality, aesthetics, text-image alignment, human preference, and world consistency.Source: Figure 3 · p. 8, §4.3 · p. 9

4.4 Abalation Study

  • Removing any evaluation level reduces correlations on both benchmarks; the object level contributes most, while the scene level gives the smallest marginal gain.Aggregation ablations compare unweighted pass-rate and exponential rules with severity-weighted variants; TerraVis’s default combines severity weighting and exponential aggregation.Source: §4.4 · p. 9, Table 4 · p. 9

4.5 Case Studies

  • Case studies show human oversight, likely false positives, missed violations, and disagreement among raters on the same image.TerraVis detects some genuine violations missed by all three raters, but also misses hand and arm distortions confirmed by manual review.Source: §4.5 · p. 9, Figure 4 · p. 10

4.6 Discussion and Limitations

  • TerraVis’s multi-stage workflow is less efficient than single-pass metrics, and detection errors, incomplete taxonomy coverage, and annotation variability remain limitations.The authors say continued taxonomy and prompt refinement may be needed as text-to-image models evolve.Source: §4.6 · p. 10

5 Conclusion

The conclusion presents TerraVis as a taxonomy-guided MLLM workflow for evaluating world consistency, validated against human judgments across two benchmarks and five models. Its taxonomy and prompts operationalize eligibility checking, violation detection, and severity classification.

5 Conclusion

  • TerraVis achieves the strongest agreement with human judgments among existing quality, alignment, and preference metrics on COCO-T2I and GenAI-Bench.The evaluation draws on more than 13,000 human annotations across two benchmarks and five text-to-image models.Source: §5 · p. 10

A Detailed Definitions and Criteria for TerraVis Violation Taxonomy

  • The object-level criteria assess entity plausibility independently of broader scene dynamics, while allowing stylized simplifications when identity and basic structure remain recognizable.They include structural, anatomical, non-biological structure, texture, symbolic-content, and object-context violations.Source: A Detailed Definitions and Criteria… · p. 15
  • The interaction criteria cover plausible contact, force, motion, environmental response, and event-dependent behavior, without penalizing stylization that preserves interaction logic.The scene criteria address relative scale, depth ordering and occlusion, and continuity across adjacent regions.Source: A Detailed Definitions and Criteria… · pp. 15–16
  • The taxonomy decomposes world-consistency failures into 7 object-level, 8 interaction-level, and 3 scene-level violation types for more interpretable analysis.It covers individual object plausibility, physically coherent interactions, and global scene configuration.Source: A Detailed Definitions and Criteria… · pp. 15–16

B Prompts Used in TerraVis

  • The appendix states that evaluation prompts are fixed across models and samples unless otherwise specified.Eligibility prompts distinguish representational images of identifiable macroscopic objects or scenes from out-of-scope content, with priority rules for multi-panel images and physical objects containing printed information.Source: B Prompts Used in TerraVis · p. 16, §B.1 · p. 16

B.2 Violation Detection

  • Violation detection instantiates a shared template with concrete, violation-specific questions and bases judgments on visible evidence rather than taxonomy labels alone.Severity classification treats issues affecting semantically central elements as major and peripheral or background issues as minor.Source: §B.2 · p. 17

C Human Annotation

Three independent annotators rated images from five text-to-image models across two benchmarks using a 1-to-5 world-consistency scale and an N/A option. Agreement was moderate to substantial, with different reported levels for the two benchmarks.

C Human Annotation

  • The annotation set contains 13,500 ratings: three annotators for each image across five models and two benchmarks.Annotators were AMT Masters over 18 years old; each annotation was paid approximately $0.08.Source: C Human Annotation · p. 19
  • Annotators first judged whether an image had any perceptible violation, then rated consistency from 1 to 5 with an additional N/A option.The scale ranges from fully consistent with no visible violation to completely inconsistent with multiple major violations or one major plus multiple minor violations.Source: C Human Annotation · pp. 19–20
  • Krippendorff’s α for 5-point ratings was 0.686 on COCO-T2I and 0.513 on GenAI-Bench; binary-label Gwet’s AC1 was 0.603 and 0.555, respectively.Violation-detection precision and recall are computed against human majority-vote labels.Source: C Human Annotation · p. 20, Table A1 · p. 20

D Additional Experimental Results

TerraVis’s structured checks detect more images with human-identified violations than holistic scoring, with some loss of precision. The comparison examines whether explicit violation checks add value beyond task-specific holistic instructions.

D.1 Holistic versus Structured Violation Detection

  • GPTScore uses generic holistic instructions, TerraVis-H uses human-annotator instructions with holistic scoring, and TerraVis decomposes assessment into explicit checks and aggregates violations by severity.Source: §D.1 · p. 20
  • The comparison measures image-level detection against human reference labels, defined as images judged by at least two of three annotators to contain a violation.The evaluators’ positive rates, precision, and recall are compared against those labels.Source: §D.1 · p. 20
  • TerraVis achieved 3.37× and 2.44× TerraVis-H’s recall on COCO-T2I and GenAI-Bench, respectively, with precision decreases of 10.2 and 4.2 percentage points, to 75.8% and 89.1%.TerraVis-H exhibited high precision but low recall, leaving many images containing human-identified violations undetected.Source: §D.1 · p. 20

D.2 Ablation of Violation Types

Ablations indicate that TerraVis’s performance draws on broad coverage across violation types, rather than any single check. Structural distortion has the largest reported marginal contribution.

D.2 Ablation of Violation Types

  • ρ = 0.275 for the best individual check versus 0.428 for the full framework; removing structural distortion yields the largest decrease (∆ρ = −0.0419).The ablation results are aggregated across COCO-T2I and GenAI-Bench.Source: §D.2 · pp. 20–21, Table A2 · p. 21
  • All 18 leave-one-out effects are negative, suggesting each check adds signal and that fine-grained checks capture complementary aspects of world consistency.Source: §D.2 · p. 21

D.3 Sensitivity to Aggregation Hyperparameters

TerraVis’s aggregation remains robust across a broad range of α settings, with its default setting near-optimal. Rankings also show agreement across evaluators, especially within groups whose evaluators align similarly with human judgments.

D.3 Sensitivity to Aggregation Hyperparameters

  • Rank correlations remain stable across a broad range of λ and α, and the default setting (λ = 1, α = 0.5) achieves near-optimal performance.For fixed α, varying λ > 0 only rescales scores monotonically; changing α can affect rankings by adjusting the relative weight of minor violations.Source: §D.3 · p. 21

D.4 Ranking Agreement across Evaluators

  • Group 1 evaluators GPT-5.5 and Gemma 4 31B have τ = 0.80 on both benchmarks; the remaining three evaluators have mean pairwise τ = 0.87 on both.The table reports pairwise Kendall’s τ between model rankings produced by different evaluators.Source: Table A4 · p. 22, §D.4 · p. 22
  • Mean Kendall’s τ across all ten evaluator pairs is 0.64 on COCO-T2I and 0.68 on GenAI-Bench, while between-group agreement averages 0.50 and 0.57, respectively.The authors suggest differences in alignment with human judgments may contribute to the remaining ranking variation.Source: §D.4 · p. 22

D.5 Computational Efficiency Analysis

TerraVis’s multi-stage evaluation has lower throughput than several single-pass MLLM metrics in the reported setting, but processes about one image per second. It is slightly faster than TIFA, another multi-stage method.

D.5 Computational Efficiency Analysis

  • TerraVis processes 0.855 images/s, or 1.17 seconds per image, taking 15.6× and 18.0× the time of VQAScore and HPSv3, and 50.4× that of RAHF.The comparison uses 200 randomly sampled images on a single NVIDIA H100 GPU.Source: §D.5 · pp. 22–23
  • TerraVis is slightly faster than TIFA, another multi-stage method that generates question–answer pairs from the prompt before visual question answering.The authors describe throughput of approximately one image per second as practical for benchmark-scale evaluation under this experimental setting.Source: §D.5 · p. 23

D.6 Distribution of Detected Violations

TerraVis detects violations across object, interaction, and scene levels, with both benchmark-dependent shifts and persistent model-specific profiles. World consistency is distinct from photorealism, prompt faithfulness, and aesthetics.

D.6 Distribution of Detected Violations

  • Interaction- and scene-level violations together account for approximately 25%–50% of violations, despite object-level violations being most common for every model.The ordering object > interaction > scene holds across all ten model–benchmark pairs; removing either interaction or scene violations lowers agreement with human ratings.Source: §D.6 · p. 23
  • Fine-grained types distinguish models that appear similar at coarser levels: GPT Image 1.5 and Nano Banana Pro have higher object-context violation rates, while Qwen-Image and SD 3.5-Large show greater structural distortion.For Qwen-Image and SD 3.5-Large, structural distortion exceeds the combined shares of object identity and object-context violations on both benchmarks.Source: §D.6 · p. 23
  • FLUX.2-dev’s most frequent violation type is symbolic content on both benchmarks, accounting for 35.0% of its COCO-T2I violations.SD 3.5-Large has the largest shares of biological anatomy, texture/surface, optical effect, and region continuity on both benchmarks.Source: §D.6 · p. 23
  • Structural distortion and object identity show the largest cross-model variation on both benchmarks, and removing either yields the largest reductions in agreement with human ratings.The correspondence suggests these dimensions are among those most strongly associated with human judgments.Source: §D.6 · p. 23
  • Biological anatomy and texture/surface violations are absent in the newer models, while symbolic content remains among the three most frequent types for every model.Low rates of contact state, support and stability, thermal response, and energy source may partly reflect the rarity of prompts testing those phenomena.Source: §D.6 · p. 23
  • Benchmark prompt composition shifts violation prevalence: object identity violations rise for all five models from COCO-T2I to GenAI-Bench, while their relative profiles remain broadly similar.The authors attribute the increase partly to GenAI-Bench’s larger share of prompts involving fictional or implausible objects and scenes; per-model shares describe composition, not overall violation frequency.Source: §D.6 · pp. 23–24

E Distinguishing World Consistency from Photorealism, Faithfulness, and Aesthetics

  • World consistency evaluates whether objects, interactions, and scenes conform to real-world constraints, unlike photorealism, prompt faithfulness, or aesthetics.These dimensions are neither equivalent nor mutually implied; a faithful depiction of an impossible event can remain world-inconsistent.Source: E Distinguishing World Consistency… · p. 24

F Additional Case Studies

The case studies show that TerraVis and human raters can disagree in both directions, while prompt alignment, preference, and artifact detection capture different aspects from world consistency. Violation profiles also vary by level and type across models.

F Additional Case Studies

  • High prompt alignment does not guarantee world consistency: an image can faithfully depict a prompt describing content that is implausible in the real world.Conversely, an image that deviates from its prompt can still receive high human world-consistency ratings.Source: F Additional Case Studies · p. 24, Figure A4 · p. 27
  • HPSv3 captures aesthetic preference rather than explicitly assessing real-world consistency, so high preference can coexist with low human world-consistency ratings.Figure A5 also shows the converse pattern: low preference scores can accompany high human world-consistency ratings.Source: F Additional Case Studies · p. 24, Figure A5 · p. 28
  • RAHF’s plausibility head is trained on RichHF-18K [26], limiting coverage to represented artifact types and potentially missing novel artifacts from newer models.Figure A6 illustrates the converse mismatch too: an artifact-free, world-consistent image may receive a low RAHF score.Source: F Additional Case Studies · p. 24, Figure A6 · p. 29
  • Object-level violations dominate every model’s detected violations, but interaction and scene violations together still account for 27.5–50.0%.The 18-type breakdown shows model-specific profiles recurring across benchmarks, including structural distortion for Qwen-Image and object-context and relative scale for GPT Image 1.5.Source: Figure A1 · p. 25, Figure A2 · p. 25
  • TerraVis identifies genuine violations missed by human raters, but can also flag violations in images all human raters judge fully consistent.The examples also include missed violations confirmed by manual review and substantial score differences among human raters.Source: F Additional Case Studies · p. 24, Figure A3 · p. 26
Loading 2610.02959v1…