PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Vision-Language Models Agree on the Affective Qualities of Shape? A Cross-Model Audit for Generative Design Interfaces

Luca Bux, Thiago Rios, Ingo Scholtes, Stefan Menzel

arXiv:2608.25876v1cs.HCcs.CV

TL;DR

Generative design controls use affective descriptors, but whether they correspond to stable semantic signals across vision-language models remains an open question. This paper audits cross-model consistency and finds affective axes agree more than an empirical null, though less than geometric controls.

  • Problem

    It remains unclear whether affective controls produce stable signals across models, while scalable human Kansei ratings are limited.

    Source: §1 · p. 1

  • Method

    The paper introduces an offline cross-model audit of affective semantic controls, calibrated against geometric positive controls and unrelated-adjective empirical nulls.

    Source: §1 · p. 2

  • Results

    Affective axes show higher average agreement than the empirical null but lower agreement than geometric controls, with convergence depending on whether category variation aligns with the evaluated semantic direction.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Cross-model convergence is a first-stage consistency check, not a substitute for human evaluation, and can inform which descriptors to expose in design interfaces.

    Source: §5 · p. 9, §7 · p. 10, §6 · p. 10

    The audit measures cross-model consistency rather than agreement with human affective judgments, so its semantic directions are not validated as human Kansei perceptions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative design interfaces increasingly expose semantic controls that let users steer output with concepts such as "more elegant" or "more minimalist," typically encoded by a vision-language model (VLM). A practical question is whether state-of-the-art VLMs represent objects consistently in terms of the same concept. We audit 6 VLMs by ranking untextured 3D objects along Kansei adjective pairs, where Kansei describes affective impressions of product form, with each axis defined as the difference between the text representations of its two poles. Geometric pairs serve as positive controls, and pairs of unrelated adjectives establish an empirical null. Across 10 categories of ShapeNet database, affective axes converge above the null (mean pairwise rank correlation 0.36 vs. 0.14) but below the geometric ceiling (0.44). The agreement between models is partial and highly uneven: on the three axes shared by all categories, mean convergence ranges from 0.21 for bookshelves to 0.51 for jars. Convergence depends primarily on whether a category's representational variation aligns with the semantic direction being evaluated, rather than simply on how much the objects vary in shape overall. Cross-model convergence does not imply agreement with human judgments. Based on our findings, we implement a UI prototype that shows how the audit can inform which Kansei descriptors to expose as controls for a given object class and which to withhold.

1 Introduction

The paper proposes cross-model consistency as a first-stage audit for affective semantic controls in generative design, finding that agreement varies by object category and does not establish agreement with human perception.

1 Introduction

  • Because semantic requests such as “a more elegant chair” act as design controls, their signals should be checked for stability across models before being exposed to users.Existing VLM design work more often evaluates object recognition than affective qualities expressed through shape.Source: §1 · p. 1
  • Cross-model convergence offers an inexpensive offline consistency audit, but it neither establishes correspondence with human perception nor proves independence from shared training data.Direct human evaluation at scale is difficult because affective 3D-shape annotations are limited and collecting reliable ratings across thousands of objects is slow and expensive.Source: §1 · pp. 1–2
  • Across 6 VLMs and 10 ShapeNet categories, affective rankings converge above the unrelated-pair null but below geometric controls, with convergence shaped by category-specific variation alignment.The results indicate that overall shape variation alone does not explain whether models agree on an affective direction.Source: §1 · p. 2
  • The audit evaluates 6 VLMs on multi-view renderings of untextured objects and calibrates affective rankings against geometric positive controls and unrelated adjective pairs.The objects come from 10 ShapeNetCore categories.Source: §1 · p. 2
  • A UI prototype demonstrates how convergence information can guide which affective descriptors to expose as controls for a given object class.The paper presents an offline audit, characterizes when controls yield reproducible signals, and illustrates how that information can inform interface design.Source: §1 · p. 2

2 Related work

Related work spans semantic design controls, Kansei product-form semantics, VLM affective evaluation, embedding-space directions, and cross-model agreement; this study combines these perspectives to audit candidate affective controls before interface use.

2.1 Interactive Control Through Semantic Attributes

  • Because semantic controls can vary by model and object category, the paper proposes auditing candidate controls before exposing them to users.Prior work progressed from relative attribute search and 3D deformation handles to latent-space sliders [10] [15] and language-guided 3D generation. Interface design shapes interaction and feedback, while reliability information may support calibrated reliance.Source: §2.1 · p. 2

2.2 Kansei Engineering and Product Form Semantics

  • The study adapts Kansei adjective pairs by using an LLM to expand category-specific descriptors and testing whether independently developed VLMs rank forms consistently.Traditional Kansei studies relate product forms to human affective judgments, which can vary across demographic groups; language models have also been used to expand or organize Kansei vocabularies.Source: §2.2 · p. 2

2.3 Affective and Aesthetic Evaluation with Vision-Language Models

  • Unlike prior affective image methods, this study probes untextured multi-view 3D renderings to isolate geometry and tests ranking reproducibility across VLMs.CLIP-IQA [51] evaluates abstract image attributes using antonym prompts within one VLM, while prior approaches primarily score photographs, often against human labels.Source: §2.3 · pp. 2–3

2.4 Semantic Directions in Embedding Spaces

  • Building on semantic projection, the study uses text-vector differences to test whether affective directions remain stable across models for each object category.Prior work established vector offsets and word-pair semantic axes [3], extended projection to antonym-defined attributes, and used VLM text-vector differences for manipulation and aesthetic scoring [51].Source: §2.4 · p. 3

2.5 Cross-Model Agreement as an Audit Criterion

  • The paper uses cross-model agreement not as a general similarity measure, but as an audit criterion for deciding whether semantic directions merit interface controls.Agreement has been used in measurement theory, ensemble uncertainty [26], representational comparison [24], and studies of shared vision-model structures; here, geometric positive controls and an empirical null calibrate the audit.Source: §2.5 · p. 3

3 Methodology … 3.5 Directions and Projection

The methodology audits how six pretrained vision-language models rank shape-only 3D objects along geometric and Kansei adjective axes. It builds model-specific semantic directions and compares rankings rather than absolute projection scores.

3.1 Overview

  • The audit renders objects, extracts representations, projects them onto semantic directions, and evaluates cross-model ranking convergence and its relation to within-category shape variation.The methodology describes this as a four-stage process, with evaluation comprising complementary measures.Source: §3.1 · p. 3

3.2 Stimuli and Rendering

  • Across 10 ShapeNetCore categories, 4,950 objects are rendered from eight viewpoints as untextured, uniform grey shapes to isolate form and reduce appearance-based confounds [7].Eight categories reach the 500-object cap; bottle has 498 objects and bookshelf has 452.Source: §3.2 · p. 3

3.3 Encoders

  • Six publicly released VLM encoders are evaluated without fine-tuning, spanning different objectives, architectures, organizations, and training datasets.Five use Vision Transformers, while ALIGN uses a convolutional EfficientNet; training objectives and provenance also vary.Source: §3.3 · p. 4

3.4 Probe Design

  • Probe controls include seven visually verifiable geometric adjective pairs as positive controls and 20 unrelated sensory or abstract pairs as an empirical null.The geometric axes test whether the pipeline captures basic shape attributes; the unrelated pairs calibrate spurious agreement.Source: §3.4 · p. 4
  • The primary Kansei tier contains 32 affective or stylistic pairs, including three shared across all categories and category-specific descriptors grounded in prior studies and LLM-assisted vocabulary expansion.Pairs differing only by a negating affix are excluded because their text embeddings are near-identical.Source: §3.4 · p. 4

3.5 Directions and Projection

  • For each bipolar adjective pair, the normalized difference between its pole text embeddings defines a direction from the negative pole to the positive pole [3] [51].The method uses bare-adjective text embeddings and assumes that text and image representations move roughly parallel for the same concept.Source: §3.5 · p. 4
  • Vector differences cancel a shared anisotropic embedding component, while model-native projection scores are not directly comparable, so evaluation uses rank orderings.Object representations are projected onto the semantic direction to produce scores along the axis.Source: §3.1 · p. 3, §3.5 · p. 4

3.6 Cross-Model Convergence … 3.11 Reproducibility

These sections define cross-model convergence and describe analyses of its relationship to category variation, view reliability, object specificity, prompting, and reproducibility.

3.6 Cross-Model Convergence

  • Cross-model convergence measures how consistently six encoders rank objects along an axis using pairwise Spearman rank correlations.The authors relate this measure to convergent validity.Source: §3.6 · pp. 4–5
  • To assess uncertainty across categories, the authors bootstrap objects with replacement for 1,000 replicates and report 95% confidence intervals; category differences require nonoverlapping intervals.They report common-language effect sizes rather than relying on individual p-values because axes are not fully independent.Source: §3.6 · p. 5

3.7 Variation-Subspace Alignment

  • The variation-subspace analysis tests whether an axis aligns with directions of object-embedding variation, correlating alignment with convergence and comparing both against random directions.The authors interpret alignment as a correlation, not a causal mechanism, and note that principal components need not be visually interpretable geometric axes.Source: §3.7 · p. 5

3.8 View Reliability

  • View reliability is estimated by splitting each object's eight views into two four-view sets, correlating their rankings, applying Spearman–Brown correction, and disattenuating encoder-pair correlations.The correction is intended to make remaining convergence less likely to reflect view-sampling noise alone.Source: §3.8 · pp. 5–6

3.9 Object Specificity

  • The object-specificity test compares convergence for each category-specific affective axis on its authored category versus other categories, excluding the three shared-core pairs.Higher convergence on the authored category would indicate that the affective signal is specific to that object class.Source: §3.9 · p. 6

3.10 Prompting Sensitivity

  • Prompting sensitivity is tested by recomputing convergence for all three tiers using either a positive-pole embedding alone or a sentence template naming the adjective and category.These alternatives are compared with the study's bare adjective–antonym pairs and bipolar difference direction.Source: §3.10 · p. 6

3.11 Reproducibility

  • The study uses publicly released model weights and states that its probe vocabulary, rendering pipeline, embedding extraction, analysis code, and precomputed audit values will be released.The complete probe vocabulary is given in Appendix B.Source: §3.11 · p. 6

4 Results

Affective axes converge more than the irrelevant null but less than geometric controls, and agreement varies with both the object category and the semantic axis. Convergence is associated with alignment to shape variation, while robustness checks show that category-specific axes and bipolar text inputs affect the observed results.

4.1 Convergence across tiers

  • Mean pairwise Spearman convergence was 0.441 for geometric axes, 0.364 for affective axes, and 0.135 for the irrelevant null.Affective axes outconverged irrelevant axes in about 9 of 10 comparisons (CL = 0.906); the geometric–affective gap was real but small (CL = 0.658).Source: §4.1 · p. 6, Figure 2 · p. 6
  • Removing OpenCLIP raised mean affective convergence only from 0.365 to 0.379, indicating that encoder redundancy did not drive the result.CLIP–OpenCLIP correlation was 0.34, below the mean pairwise value, while the strongest reported pairs reached 0.47.Source: §4.1 · p. 6

4.2 Convergence depends on both the axis and the category

  • Affective convergence varied by category and axis: shared-core means ranged from 0.21 for bookshelves to 0.51 for jars, while full-vocabulary means ranged from 0.26 to 0.53.Across the three shared axes, minimalist–ornate converged at 0.53, whereas natural–artificial reached the null at 0.14; category rankings also shifted with vocabulary.Source: §4.2 · p. 7, Figure 3 · p. 8
  • Split-half reliability averaged 0.76 across categories, but disattenuated convergence remained spread across categories (0.32–0.58) and was uncorrelated with reliability (ρ = 0.10).Bookshelves had the lowest reliability (0.61) and shared-core convergence (0.21); correction raised their convergence but they remained lowest among categories.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 Alignment with shape variation

  • Alignment with shape variation predicted convergence for geometric axes (ρ = +0.78) and affective axes (ρ = +0.72), but less strongly for irrelevant axes (ρ = +0.24; p < 0.001 throughout).The relationship remained after controlling for score spread and measurement noise (Kansei ρ = 0.47; geometric ρ = 0.88; irrelevant ρ = 0.15), and was positive for affective axes in every category.Source: §4.3 · p. 7, Figure 4 · p. 7
  • For bookshelves, clocks, and cabinets, geometric convergence exceeded affective convergence, whereas affective axes were not aligned with the geometric axes tested.The strongest affective–geometric relationship was simple–complex versus minimalist–ornate (mean |cos| = 0.24), sharing less than one-fifth of their variance.Source: §4.3 · p. 7, Table 3 · p. 8

4.4 Robustness to encoders and prompts

  • Category-specific axes converged more on their own category than on foreign categories for most classes (0.35 vs. 0.28; CL = 0.64, p = 0.009), with bottles, clocks, and tables reversing the pattern.Convergence among five encoders predicted agreement with the held-out sixth encoder (overall ρ = 0.67).Source: §4.4 · p. 8, Table 4 · p. 8
  • Using a single-pole embedding raised null convergence to 0.21 and lowered affective convergence to 0.27, shrinking their gap from 0.23 to 0.06.Sentence templates raised convergence in all three tiers while preserving the absolute gap at 0.23; the study retained bare bipolar pairs, which kept the null near zero.Source: Table 5 · p. 8, §4.4 · p. 9

5 Discussion

The discussion presents cross-model convergence as a consistency check for semantic controls and shows how category-specific audit results could guide their presentation in generative design interfaces. It also cautions that model agreement is not a substitute for checking descriptors with intended users.

5 Discussion

  • High-convergence descriptors can serve as more reproducible controls, while low-convergence descriptors should be flagged for further evaluation.The audit is proposed as a practical consistency check for semantic controls in AI-assisted design systems.Source: §5 · p. 9

5.1 From audit to interface

  • The category-axis map reveals that a descriptor may have consistent model agreement for one object class but not another, and new encoders require only recomputing projections.The procedure needs neither human labels nor model retraining and uses the existing rendered dataset.Source: §5.1 · p. 9
  • The interface shows each selected object’s projections from six encoders and per-object agreement score so users can assess whether a control reflects shared or model-specific patterns.It makes agreement and disagreement visible during interaction; the design is motivated by evidence that communicating uncertainty can encourage analytical engagement and reduce over-reliance.Source: §5.1 · pp. 9–10
  • The admission rule exposes luxurious–cheap for jars (¯𝜌= 0.47) but withholds it for bookshelves (0.10) and cabinets (0.14).The irrelevant tier sets an empirical floor of ¯𝜌= 0.14; controls are exposed when the lower bootstrap interval bound exceeds the null, marked provisional when it overlaps, and withheld when the estimate is at or below the null.Source: §5.1 · p. 10
  • Future systems could use convergence to rank candidate controls, flag descriptors near the empirical null, or communicate uncertainty about semantic edits.The audit is framed as inexpensive preprocessing infrastructure, not an end-user evaluation.Source: §5.1 · p. 10

5.2 Practical and Societal Implications

  • Shared training data and biases, plus cultural and contextual variation in Kansei meanings, limit what cross-model agreement can establish.Low-agreement controls should be flagged as uncertain rather than removed, and selected descriptors should be checked with the intended user group before culturally sensitive or consequential use.Source: §5.2 · p. 10

6 Limitations

The audit measures agreement among models, not alignment with human Kansei judgments, and the interface prototype has not been evaluated with users.

6 Limitations

  • The audit provides no human validation, so its semantic directions cannot be taken as evidence of human Kansei perception.Cross-model convergence is treated as a model-based consistency criterion that can assist human judgments; how it relates to those judgments remains future work.Source: §6 · p. 10
  • The interface has not been evaluated with users, leaving its effects on designer performance, decision quality, calibration, and trust unknown.Controlled studies comparing convergence-aware interfaces with conventional semantic-control interfaces are needed to determine their practical value.Source: §6 · p. 10

7 Conclusion

The audit finds that affective axes show greater average agreement across six vision-language models than an empirical null, but less than geometric controls. It offers a first-stage consistency assessment—not a substitute for human evaluation—to guide which descriptors are exposed in design interfaces.

7 Conclusion

  • Affective axes show higher average agreement across six vision-language models than an empirical null, but lower agreement than geometric controls.Agreement depends on whether the semantic direction captures variation present among the objects being compared.Source: §7 · p. 10
  • The audit provides a first-stage consistency assessment, not a substitute for human evaluation, and can identify reproducible descriptors without human labels.It scales to new object categories and models and can be recomputed as vision-language models evolve, supporting more transparent, uncertainty-aware design interfaces.Source: §7 · p. 10
  • Future work should validate convergence against human Kansei judgments and study word optimization, richer representations, and reliability information in interactive design.The proposed directions include incorporating materials, textures, and context, and evaluating effects on designer decision-making.Source: §7 · p. 10

GenAI Usage Disclosure

LLMs supported probe-vocabulary construction, manuscript editing, and figure generation, while the authors retained responsibility for the experimental design, analysis, interpretation, and scientific claims.

GenAI Usage Disclosure

  • LLMs suggested candidate affective adjectives and antonyms where source literature provided insufficient descriptors; authors reviewed and curated all suggestions.The final vocabulary is reported in full.Source: GenAI Usage Disclosure · p. 10
  • An LLM was used to polish manuscript phrasing and improve clarity.Source: GenAI Usage Disclosure · p. 10
  • An LLM-based coding assistant supported figure generation.Source: GenAI Usage Disclosure · p. 10
  • The authors stated that experimental design, data analysis, results interpretation, and scientific claims were their own and that they take full responsibility for the paper.Source: GenAI Usage Disclosure · p. 11

A Encoder Agreement Matrix

The section reports mean pairwise Spearman rank correlations between encoder pairs, averaged across all Kansei axes.

A Encoder Agreement Matrix

B Vocabulary

The probe vocabulary combines tiers shared across categories with category-specific Kansei axes, including three axes shared by every category.

B Vocabulary

  • Table 7 lists the complete probe vocabulary: geometric and irrelevant tiers are shared across ten categories, while Kansei axes are category-specific with three shared-core axes.The remaining Kansei axes are drawn per category from the Kansei literature and the LLM-assisted expansion described in Section 3.4.Source: B Vocabulary · p. 12, Table 7 · p. 13

Related papers

Loading 2608.25876v1…