PaperlayerView exact revision on arXiv

Source-linked AI summary

ACVA-Bench: Beyond Prompt Following in Abstract Concept Visualization Ability for Text-to-Image Models

Mengyan Wang, Jiyuan An, Yining Liu, YaoWen Zhang, Liner Yang ang Erhong Yang

arXiv:2610.32058v1cs.CVcs.AI

TL;DR

Whether T2I models can autonomously visualize abstract concepts without fixed visual referents remains insufficiently evaluated. ACVA-Bench tests this with category-balanced, word-only generation and concept-constrained visual-semantic questions. Across six models, it finds category-dependent semantic gaps, prototype reliance, and failures to preserve fine-grained concept boundaries.

  • Problem

    It remains unclear whether T2I models can autonomously depict abstract concepts without a fixed visual referent while preserving meaning and distinguishing related concepts.

    Source: §1 · p. 1

  • Method

    ACVA-Bench evaluates Chinese abstract concepts using low-intervention word-only generation, three evaluation axes, and dictionary-grounded visual-semantic rubrics.

    Source: §1 · pp. 1–2

  • Results

    Across six models and 2,700 images, the benchmark identifies category-dependent semantic gaps, prototype reliance hidden by surface variation, and failures to preserve fine-grained concept boundaries.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The three-axis framework exposes differences that image quality or a single aggregate score cannot capture and provides concrete targets for improving abstract-concept visualization.

    Source: §6 · p. 8

    The benchmark covers 150 Chinese abstract concepts, validates automatic judgments on 90 human-rated images, and evaluates static images under a standardized word-only prompt.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Text-to-image (T2I) models have achieved remarkable progress in image quality and instruction following, prompting growing research interest in their capacity for deep understanding that goes beyond literal prompt matching. However, without an explicit visual scheme provided, it remains unclear and has not been systematically evaluated whether T2I models can autonomously translate abstract concepts lacking a fixed visual referent, such as "loyalty," into visual representations that are accurate, diverse, and distinguishable from closely related concepts. We define this capability as Abstract Concept Visualization Ability (ACVA) and introduce ACVA-Bench, a category-balanced benchmark grounded in psycholinguistic norms of abstract concepts, which examines models' ACVA along three dimensions semantic sufficiency, representational diversity, and conceptual distinctiveness under a low-intervention setting that withholds concrete objects, scenes, or visual metaphors. Since abstract concepts lack a unique reference image, conventional evaluation metrics that rely on reference-image matching are difficult to apply. To address this, we propose a fine-grained visual-semantic evaluation method constrained by concept semantics, which decomposes concept semantics into independently verifiable visual-semantic questions to assess the semantic representation quality of generated images, with its validity further confirmed by human evaluation. Benchmarking results on mainstream T2I models show that these models rely on a small set of prototypical visual patterns, leaving room for improvement in representational diversity and conceptual distinctiveness for abstract concepts, thereby providing an interpretable and reproducible evaluation foundation for future research.

1 Introduction

Although T2I models have advanced in image quality and instruction following, it remains unclear whether they can choose visual realizations for abstract concepts without a supplied visual scheme. ACVA-Bench frames this as autonomous visual grounding and evaluates it with category-balanced concepts and semantic questions that leave the depiction open.

1 Introduction

  • Abstract concepts such as loyalty lack fixed visual referents, so models must choose entities, relations, and scenes that can diverge or omit essential meaning.Prior evaluations of implicit meaning and reasoning still provide elaborate prompts with task constraints or reasoning cues.Source: §1 · p. 1
  • ACVA is the ability to organize visual entities, relations, scenes, and schemas into recognizable, diverse representations that distinguish related concepts without an explicit visual plan.Source: §1 · p. 1
  • ACVA-Bench evaluates Chinese abstract concepts under a word-only, low-intervention protocol along semantic adequacy, representational diversity, and conceptual distinctiveness.Its category-balanced concepts are grounded in psycholinguistic norms.Source: §1 · p. 1
  • The benchmark’s feature-level scoring decomposes concept semantics into independently judgeable visual questions and is applied to six image-generation models with human validation.The method is designed to constrain semantic conditions without prescribing a single visual depiction.Source: §1 · p. 2

2 Related Work

Related work treats abstract concepts as cognitively heterogeneous, but generative-model studies have not established whether T2I models can organize visual elements for abstract meaning autonomously. Existing T2I evaluation is increasingly fine-grained yet usually measures fidelity to predetermined visual facts.

2.1 Multidimensional Cognitive Representations of Abstract Concepts

  • Abstract concepts draw on partially overlapping sensorimotor, linguistic, emotional, interoceptive, social, and introspective systems, but generative-model work largely treats them as undifferentiated.Psycholinguistic norming of Chinese abstract words identifies cognitive dimensions and categories.Source: §2.1 · p. 2

2.2 Visualizing Abstract Concepts with Generative Models

  • Prior work uses language models to map abstract concepts to concrete prompts or studies indirect grounding and visual variability, leaving autonomous T2I organization of entities and relations unexamined.Source: §2.2 · p. 2

2.3 Text-to-Image Evaluation

  • T2I evaluation spans holistic quality, compositional checks, verifiable visual propositions, and reasoning, but mostly tests fidelity to predetermined visual facts rather than abstract concepts.The paper positions its VQA-style method as evaluating abstract meaning without reducing a concept to one symbol.Source: §2.3 · pp. 2–3

3 ACVA-Bench

ACVA-Bench combines psycholinguistic concept sampling with dictionary-grounded visual-semantic rubrics, low-intervention generation, and three-axis evaluation. Human review refines questions so they constrain semantic conditions without fixing the images’ objects or metaphors.

3 ACVA-Bench

  • ACVA-Bench combines category-balanced concept sampling, rubric construction, low-intervention generation, three-axis evaluation, and human validation.Its construction procedure could extend to other languages with appropriate resources and validation, although the benchmark uses Chinese concepts.Source: §3 · p. 3

3.1 Abstract Concept Set

  • The concept set stratifies 150 Chinese abstract nouns across six categories, drawing 25 concepts per category from norms covering 19 cognitive dimensions.It retains psycholinguistic ratings and includes 30 verified near-synonym pairs.Source: §3.1 · p. 3

3.2 Visual-Semantic Rubric Resource

  • Rubric construction maps dictionary definitions through componential decomposition and frame semantics to visual-semantic constraints and independently judgeable questions.Image schemas connect abstract relations to perceptible spatial structure, while constraints specify semantic conditions rather than fixed objects or symbols.Source: §3.2 · p. 3

3.3 Rubric Question Auditing

  • Seven linguistics graduate students reviewed rubric features, removing unobservable or redundant items and revising question wording and weights.For loyalty, the rubric asks about ongoing relationships and sustained support or allegiance rather than mere proximity, while leaving the visual metaphor open.Source: §3.3 · pp. 3–4

4 Evaluation Methodology: Axes and Metrics

The evaluation separates semantic adequacy, variation among valid depictions, and the ability to distinguish related concepts. Its measures are designed to expose semantic omissions, repeated metaphors, and concept-boundary failures that holistic image–text similarity can miss.

4 Evaluation Methodology: Axes and Metrics

  • Holistic image–text similarity cannot reveal which semantic components are expressed, whether generations reuse a metaphor, or whether nearby concepts remain distinguishable.Source: §4 · p. 4

4.1 Semantic Adequacy

  • Semantic adequacy scores whether an image expresses a target concept’s key components using weighted rubric ratings from 0 to 1, with scores of at least 0.6 passing.Question-level ratings preserve a diagnostic trace of which semantic components are missing, unlike holistic similarity or aesthetic-score combinations.Source: §4.1 · pp. 4–5

4.2 Representational Diversity

  • Representational diversity is measured among groups with at least two passing images, with unfiltered results also reported to expose selection effects.Image-embedding distance, metaphor-description distance, and metaphor clustering distinguish surface variation from variation in metaphorical content; high image distance with low metaphor diversity signals pseudo-diversity.Source: §4.2 · p. 5

4.3 Conceptual Distinctiveness

  • Conceptual distinctiveness reuses independently generated images for 30 near-synonym pairs without definitions, paired prompts, or additional images.It measures image-distribution separability and uses a two-alternative forced choice to assess perceptual readability; the two forms of evidence are reported separately.Source: §4.3 · p. 5

5 Experiments

Experiments evaluate six text-to-image models on semantic adequacy, representational diversity, and conceptual distinctiveness using standardized generations. The results show substantial capability gaps and distinct weaknesses across these axes.

5 Experiments

  • Each of six models receives the same word-only prompt and generates three images per concept, yielding 2,700 images overall.Evaluation uses visual-semantic questions for adequacy and a two-alternative forced-choice judge for perceptual distinctiveness.Source: §5 · p. 5

5.1 Semantic Adequacy: Model and Category Differences

  • GPT-Image-2 passes 90.4% of images, while Qwen-Image and NextStep-1.1 pass 12.7% and 10.2%, respectively.Qwen-Image-2.0 and GLM-Image succeed roughly half the time; their gap, and the Qwen-Image/NextStep-1.1 gap, are not statistically significant.Source: §5.1 · pp. 5–6
  • Physical-spatiotemporal-quantitative concepts score highest (0.641), while interpersonal communication scores lowest (0.364); human imageability is only weakly associated with adequacy (ρ = 0.215).The category means span 0.364–0.641, and imageability’s association is nearly absent for emotion (ρ = 0.011) and interpersonal communication (ρ = 0.030).Source: §5.1 · p. 6

5.2 Representational Diversity: Prototype Reliance and Pseudo-Diversity

  • Among 900 model–concept groups, genuine and low diversity each account for 34.2%, while pseudo-diversity and metaphor differentiation each account for 15.8%.GPT-Image-2 combines leading semantic adequacy with the lowest image distance, metaphor distance, and metaphor entropy; 112 of its 150 groups fall in the low-diversity quadrant.Source: §5.2 · p. 6, Table 2 · p. 6, Figure 5 · p. 7

5.3 Conceptual Distinctiveness: Fine-Grained Precision for Near-Synonyms

  • VLM 2AFC accuracy averages 0.815 across 180 model–pair results, but oath/promise reaches only 0.472, showing that near-synonym boundaries remain difficult.Qwen-Image’s image-embedding accuracy is 0.517 versus VLM accuracy of 0.906, while NextStep-1.1 shows 0.883 versus 0.728, respectively.Source: §5.3 · pp. 7–8, Table 3 · p. 7

5.4 Case Studies

  • Figure 6 illustrates prototype collapse in repeated wedding-ring imagery and obscured oath/promise distinctions, while marriage/family has distinct visual carriers (pair-level accuracy 1.00).The examples connect the quantitative results to generated images across models.Source: §5.4 · p. 8, Figure 6 · p. 8

5.5 What the Three Axes Reveal

  • Adequacy correlates negatively with image diversity (ρ = −0.286 across 900 groups), but has little association with perceptual distinctiveness (ρ = −0.044, p = 0.55).The paper identifies separate improvement targets: semantic relations for low-adequacy models, additional valid situations and metaphors for high-adequacy models with low diversity, and visible distinguishing components for near-synonyms.Source: §5.5 · p. 8

6 Conclusion

The conclusion presents ACVA-Bench as a framework for evaluating abstract visual grounding beyond explicit prompt following. Across six models and 2,700 images, it identifies category-dependent adequacy gaps, prototype reliance, and difficulty preserving fine-grained concept boundaries.

6 Conclusion

  • Across six models and 2,700 images, ACVA-Bench identifies category-dependent semantic gaps, prototype reliance hidden by surface variation, and failures to preserve fine-grained concept boundaries.Its three-axis evaluation distinguishes weaknesses that image quality or a single aggregate score cannot capture.Source: §6 · p. 8

Limitations

The benchmark’s evaluation scope is limited by its concept, generation, validation, and media coverage.

Limitations

  • ACVA-Bench evaluates 150 Chinese abstract concepts with three generations per model–concept pair; broader language coverage and larger samples would extend the evaluation.Automatic judgments are validated on 90 human-rated images, and the benchmark covers static images under a standardized word-only prompt.Source: Limitations · p. 8

Ethical considerations

The section outlines responsible-use considerations alongside reproducibility details and an illustrative rubric example.

Ethical considerations

  • Lexical resources, generated images, and human judgments should respect source licenses and provider terms and protect annotator privacy.The authors note that abstract concepts admit culturally diverse depictions and that the rubrics allow multiple visual realizations.Source: Ethical considerations · p. 8

A Reproducibility and Evaluation Implementation

  • The protocol uses a word-only template, three independent calls per concept, and default non-seeded sampling, producing 2,700 images.Model versions and generation metadata accompany the benchmark; hosted inference hardware is not controlled by the study.Source: A Reproducibility and Evaluation… · pp. 10–11
  • The semantic-adequacy judge shares a model family with two generators, so the reported rankings do not rule out evaluator bias.The authors identify a controlled multijudge study with an unrelated VLM as future work.Source: A Reproducibility and Evaluation… · p. 11

B Category Definitions and an Illustrative Rubric

  • The loyalty example illustrates checkable questions from several semantic frameworks, but its five questions are not the full operational rubric or its calibrated weights.Temporal relations and intentions must be judged from visible evidence rather than inferred backstory.Source: B Category Definitions and an… · p. 11

C Extended Quantitative Results

The extended results show that semantic adequacy can coincide with low diversity, and that imageability is only a weak, uneven predictor of adequacy. They also examine whether diversity patterns persist after adequacy filtering and how separability varies across concept pairs.

C Extended Quantitative Results

  • Table 7 reports model-by-category semantic adequacy means, with each category mean computed from the underlying unrounded values.The supplied passage lists the means in canonical category order but does not identify each value's category.Source: C Extended Quantitative Results · p. 11, Table 7 · p. 15
  • GPT-Image-2 combines the highest semantic pass rate with the lowest form diversity and metaphor entropy across the six models.Figure 8 plots Dimg against metaphor entropy H, with marker size and color encoding semantic pass rate.Source: C Extended Quantitative Results · p. 11, Figure 8 · p. 12
  • Pseudo-diversity accounts for 15.8% of all 900 groups and 15.5% of the 440 groups with at least two passing images.Genuine and low diversity each shift from 34.2% to approximately 34.5%; similar aggregate proportions do not mean the same groups or model composition are unchanged.Source: C Extended Quantitative Results · pp. 11–12

D.1 Imageability and Semantic Adequacy

D Supplementary Statistical Analyses

  • Across 150 concepts, imageability has a weak positive association with mean adequacy (Spearman ρ = 0.215, p = 0.008).The association is uneven by category, ranging from ρ = 0.285 for physical/spatiotemporal/quantitative concepts to ρ = −0.056 for terminological/institutional/epistemic concepts.Source: §D.1 · pp. 12, 15, Figure 10 · p. 15

D.2 Relationships among Evaluation Axes

The evaluation axes capture different patterns: adequacy is negatively associated with diversity, but shows little association with perceptual distinctiveness. Model adequacy comparisons suggest four tiers, while equal and human-calibrated rubric weights produce nearly identical aggregate rankings.

D.2 Relationships among Evaluation Axes

  • At the model–concept level, adequacy correlates negatively with form diversity (ρ = −0.286, p = 1.9 × 10^-18) and metaphor entropy (ρ = −0.160, p = 1.5 × 10^-6).Across six model-level means, the adequacy–diversity correlation is ρ = −0.886, suggesting between-model differences contribute substantially to the aggregate trade-off.Source: §D.2 · p. 15, Figure 11 · p. 15
  • At the model–pair level, mean adequacy and VLM 2AFC accuracy show little association (ρ = −0.044, p = 0.55).The model-level correlation is ρ = 0.029; a near-zero correlation does not establish statistical independence.Source: §D.2 · p. 15, Figure 12 · p. 16

D.3 Uncertainty in Model Comparisons

  • The reported pairwise tests suggest four adequacy tiers rather than six statistically distinguishable model ranks.Three adjacent-model comparisons have reported bootstrap p < 0.0001, while Qwen-Image-2.0 versus GLM-Image and Qwen-Image versus NextStep-1.1 have difference intervals spanning zero.Source: §D.3 · p. 15

D.4 Effect of Rubric Weighting

  • Equal-weight and human-calibrated scores correlate at Spearman ρ = 0.996 over 2,700 images and ρ = 0.986 over 150 concepts, with identical six-model rankings.Reported mean changes are no larger than 0.007; the results show little effect on aggregate comparisons, not that weighting is necessary for the headline ranking.Source: §D.4 · pp. 15–16

E Human Validation Protocol and Metric Interpretation

Human validation supports a reproducible diagnostic signal but does not establish a human-equivalent metric. Supplementary examples show recurring visual schemas and contrasting depictions across concepts and models, while additional plots visualize results reported elsewhere in the paper.

E Human Validation Protocol and Metric Interpretation

  • Automatic scores correlate with human ratings at ρ = 0.362, with 46.3% exact agreement and 91.1% agreement within one rating bin.These results support a reproducible diagnostic signal, but do not establish a human-equivalent metric or guarantee preservation of model rankings.Source: E Human Validation Protocol and Metric… · p. 16, Figure 13 · p. 16

F Extended Qualitative Examples

  • GPT-Image-2’s repeated marriage samples converge on a wedding-ring-and-heart schema, while freedom is repeatedly rendered as a bird in flight.These depictions are often high-scoring but compress the concept into a single prototype rather than exploring alternatives such as shared daily life.Source: F Extended Qualitative Examples · p. 16, Figure 6b · p. 16

F.1 Repeated Generations and Cross-Model Comparisons

  • Across three GPT-Image-2 generations, wedding rings and hearts recur for marriage, while open arms or released birds recur for freedom.The contrasting model outputs show that other depictions are possible.Source: §F.1 · p. 17, Figure 14 · p. 18
  • For oath/promise, three-model examples share motifs such as clasped hands, rings, or luminous gestures; marriage/family examples contrast wedding imagery with family or household scenes.The passage cautions that these examples do not imply every image or concept pair follows the same pattern.Source: §F.1 · p. 17, Figure 15 · p. 19

G Supplementary Result Visualizations

  • For near-synonym distinctiveness, Figure 17 compares image-embedding leave-one-out accuracy with VLM 2AFC accuracy; the rankings are not consistent, and the dashed line marks chance at 0.5.Source: Figure 17 · p. 20

Related papers

Loading 2610.32058v1…