PaperlayerView exact revision on arXiv

Source-linked AI summary

Organization of Valence and Arousal in Vision-Language Representations of Built Environments: Insights from the EMOIS Dataset

Madoka Yonekura, Katsunori Kohda, Nobuhiko Muramoto, Takahiro Yamaguchi

arXiv:2609.06870v1cs.CV

TL;DR

How affective information is organized in vision-foundation representations of built environments remains underexplored, and suitable domain coverage is limited. The paper introduces EMOIS and analyzes CLIP-based predictions and representation geometry; valence shows more coherent organization than arousal, with differences also observed against OASIS.

  • Problem

    How affective information is organized across dimensions and visual domains in foundation-model representations remains unclear, with limited affective-dataset coverage of built environments.

    Source: §I · p. 1

  • Method

    The study constructs EMOIS, a 1,544-image built-environment dataset, and uses CLIP representations for affective prediction and geometric analysis.

    Source: §I · pp. 1–2, §III · p. 3, A. Spatial Emotion Prediction Model · p. 5

  • Results

    Valence showed more coherent global organization than arousal within EMOIS, and comparison with OASIS suggested additional organizational differences between datasets.

    Source: §VI · p. 10, §VII · p. 13

  • Takeaways & Limitations

    EMOIS and the example-based retrieval framework provide a resource and qualitative means to study and interpret affective predictions for built environments.

    Source: §VII · pp. 12–13

    Ratings came from participants whose nationality and longest residence were both in Japan, which may limit generalizability to other populations.

    Source: §VI · p. 12

Abstract

from arXiv · show

Visual perception of built environments contributes to the affective impressions that people form in everyday life. However, how these impressions are represented within vision foundation models remains largely unexplored. To support the systematic investigation of this subject, we introduce the Emotional Impression of Spaces (EMOIS) dataset, comprising 1,544 real-world built-environment images. Each image is annotated with image-evoked valence and arousal ratings collected from Japanese adults by conducting a large-scale web-based survey, with approximately 120 ratings per image. Using Contrastive Language--Image Pre-training (CLIP) representations, we perform predictive and geometric analyses to systematically investigate how valence and arousal are encoded and organized within the representation space. These analyses reveal that valence exhibited stronger and more coherent organization than arousal. Cross-dataset analyses with the Open Affective Standardized Image Set (OASIS), a benchmark dataset of general affective photographs, reveal differences in affective organization between the two datasets. Regression analyses demonstrate high predictive performance for valence and arousal within EMOIS, with mean coefficients of determination of 0.865 and 0.807, respectively, across repeated internal hold-out evaluations. Finally, we present an example-based interface illustrating how learned representations can support qualitative interpretation of predicted affective values. These findings can help elucidate affective representations of built environments and establish EMOIS as a densely annotated resource for future affective computing research in this domain.

I. INTRODUCTION

The introduction motivates studying how affective information is organized in foundation-model representations, especially for built environments, and introduces EMOIS and the study’s analyses. It also situates valence and arousal within dimensional models of emotion.

I. INTRODUCTION

  • Existing work has modeled emotion from visual stimuli, but foundation-model studies have emphasized prediction more than the organization of affective information in their representations.The study frames representation organization as an open question beyond predictive performance.Source: §I · p. 1
  • Affective information’s organization across dimensions and visual domains remains unclear, and investigating it requires datasets that adequately cover the target domain.Existing affective datasets offer limited built-environment coverage, leaving few opportunities to study representational organization in these spaces.Source: §I · p. 1
  • The study introduces EMOIS and uses CLIP representations to test affective prediction and analyze how affective information is organized.Its contributions include comparing organization across valence and arousal and between built-environment and general affective-image datasets, plus an example-based interpretation interface.Source: §I · pp. 1–2

A. Emotion Representation Models

  • Dimensional emotion models represent responses in continuous affective space; the VA circumplex model uses valence and arousal as its dimensions.The passage contrasts dimensional models with categorical models that describe emotions as discrete classes.Source: A. Emotion Representation Models · p. 2
  • Emotional experience may have both categorical and continuous structure, rather than being fully captured by either discrete categories or low-dimensional affective representations alone.This point is presented through prior work on emotion categories connected by continuous affective gradients.Source: A. Emotion Representation Models · p. 2

B. Affective Computing and Representation Learning

Prior work suggests that learned visual representations contain information relevant to subjective evaluation, but evidence about built environments and the organization of affective information remains limited. Research on architectural perception identifies multiple spatial and experiential factors, while computational studies of diverse real-world spaces remain relatively underexplored.

B. Affective Computing and Representation Learning

  • Modern foundation representations have been linked to subjective evaluation and affective judgments, but this evidence comes primarily from general photographic and object-centric datasets.Prior studies include CLIP-based aesthetic assessment and perceptual-attribute prediction, as well as visual models explaining affective responses.Source: B. Affective Computing and… · p. 2
  • Whether foundation representations encode affective characteristics of built environments, and how that information is organized geometrically, remains unclear.Existing affective datasets largely use general photographic or object-centric imagery, while scene datasets often lack valence and arousal annotations.Source: B. Affective Computing and… · p. 2

C. Emotional Perception of Architectural Space

  • Architectural affective responses are associated with geometric properties, color, lighting, materials, and the sequence of spaces encountered.Prior work also links experiential dimensions such as fascination, coherence, and hominess with affective evaluations.Source: C. Emotional Perception of… · pp. 2–3
  • Computational modeling of emotional impressions across diverse real-world built-environment images remains relatively underexplored.Prior approaches include controlled or simplified virtual environments and a CLIP-based office evaluation using social, tranquil, and inspirational dimensions.Source: C. Emotional Perception of… · p. 3

D. Example-Based Interpretation of Affective Predictions

The paper proposes interpreting affective predictions through visually similar examples and their subjective ratings. EMOIS supplies a varied collection of built-environment images, gathered and screened to increase diversity and reduce redundancy.

D. Example-Based Interpretation of Affective Predictions

  • The proposed interpretation framework presents visually similar reference images with their subjective affective ratings to contextualize predictions.It complements approaches that identify influential features or image regions by offering concrete examples for interpreting predicted values.Source: D. Example-Based Interpretation of… · p. 3

III. EMOIS DATASET

  • EMOIS contains 1,544 indoor and outdoor built-environment images annotated for valence and arousal on a seven-point Likert scale.The annotations use the VA dimensional framework.Source: §III · p. 3

A. Image Collection

  • Image collection used keywords for space functions and atmospheres, included varied design styles and vegetation, and excluded people and text.The collection strategy aimed to diversify function and atmosphere while focusing impressions on spatial characteristics rather than social or semantic cues.Source: A. Image Collection · p. 3
  • A post-collection embedding-similarity screen identified 70 images in 33 groups depicting the same environments; these images were retained with common group identifiers.Pairs with cosine similarity greater than 0.9 were visually inspected.Source: A. Image Collection · p. 3

B. Image Preprocessing

Images were square-cropped for annotation and feature extraction, and Japanese adults rated valence or arousal under a web-based survey protocol. A frozen CLIP representation supports prediction, analysis, and retrieval; the prediction model uses standardized features and Bayesian ridge regression.

B. Image Preprocessing

  • The same square-cropped images were used for annotation and feature extraction, after cropping and resizing survey stimuli to 1,000 × 1,000 pixels.The square format was chosen for CLIP inputs; licensing restrictions prevent public redistribution of the original or cropped images.Source: B. Image Preprocessing · p. 4

C. Emotion Annotation

  • Japanese adults rated either valence or arousal on a seven-point scale, with each participant evaluating 50 randomly ordered images.After quality-control exclusions, 4,205 participants were retained for each dimension with the sample balanced across sex and age groups; split-half reliability was assessed over 1,000 random splits.Source: C. Emotion Annotation · p. 4

IV. AFFECTIVE REPRESENTATION LEARNING AND ANALYSIS FRAMEWORK

  • Frozen CLIP encodes spatial images, and the resulting representation space is used for affective prediction, representation analysis, and example-based interpretation.The framework retrieves similar reference images together with their subjective ratings for interpretation.Source: §IV · p. 4, Figure 1 · p. 4

A. Spatial Emotion Prediction Model

  • CLIP ViT-B/16 features are 512-dimensional and standardized using training-set statistics before Bayesian ridge regression predicts mean valence and arousal.The study also evaluates supervised ViT and ResNet representations without dimensionality alignment, and uses 100 shared group-wise 8:2 train–test splits.Source: A. Spatial Emotion Prediction Model · p. 5

B. Local Affective Consistency

The analysis evaluates whether nearby images in each visual representation share affective ratings. It estimates each image’s affect from retrieved neighbors and tests retrieval errors across representations.

B. Local Affective Consistency

  • For each query image, cosine similarity ranks all other images after excluding candidates from the same built-environment group; the five nearest ratings are averaged as the affective estimate.The interface displays only the top three retrieved examples, while the consistency analysis uses five.Source: B. Local Affective Consistency · p. 5
  • Pearson correlations between query ratings and retrieval-based estimates measure local affective consistency.Valence and arousal ratings of the retrieved images are averaged separately.Source: B. Local Affective Consistency · p. 5
  • Image-wise absolute retrieval errors are summarized with MAE and compared using Friedman tests, followed by Holm-corrected Wilcoxon tests when the main effect is significant.The authors use non-parametric tests because parametric-test distributional assumptions were not satisfied.Source: B. Local Affective Consistency · p. 5

C. Global Affective Organization

The analyses visualize and quantify affective organization within and across representation spaces. A separate distance analysis tests how representation distances relate to affective differences.

C. Global Affective Organization

  • CLIP and ViT features were embedded independently with UMAP, then orthogonally Procrustes-aligned so valence and arousal could be projected for visual comparison.The two-dimensional UMAP used 30 neighbors, cosine distance, and minimum distance 0.1.Source: C. Global Affective Organization · p. 5
  • Spatial autocorrelation was measured in the original L2-normalized feature spaces using k=10 nearest neighbors, excluding images from the same environment group.A row-standardized binary neighbor-weight matrix supported computation of Moran’s I and Geary’s C; higher I and lower C indicate stronger organization.Source: C. Global Affective Organization · p. 6
  • Robustness used 100 shared subsamples of 1,000 EMOIS images, while cross-dataset analyses jointly embedded OASIS and EMOIS and sampled 800 images per dataset in each of 100 iterations.The same autocorrelation procedure was applied to CLIP representations; EMOIS neighbors from the same environment group were excluded.Source: C. Global Affective Organization · p. 6

D. Representation–Affect Distance Relationships

  • For unique image pairs outside the same environment group, representation distance was one minus cosine similarity, and affective distance was the absolute valence or arousal difference or their VA-space Euclidean distance.The analysis also compared representation distance with each affective distance using Spearman correlations for ViT and CLIP.Source: D. Representation–Affect Distance… · p. 6
  • The CLIP–ViT correlation difference was assessed with a group-level jackknife, omitting each same-environment group or singleton unit and its associated image pairs in turn.A normal approximation to the jackknife standard error yielded 95% confidence intervals for the correlation difference.Source: D. Representation–Affect Distance… · p. 6

E. Example-Based Interpretation Framework

EMOIS ratings cover a range similar to OASIS but have a different distribution. CLIP predicts affective ratings, and the paper illustrates their interpretation with an interface showing similar rated spaces.

E. Example-Based Interpretation Framework

  • The Flask-based interface displays predicted valence and arousal with uncertainty, retrieves three visually similar EMOIS images with subjective ratings, and locates the prediction in VA space.Three ChatGPT-generated spatial images served as case-study inputs.Source: E. Example-Based Interpretation… · p. 6

A. Affective Characteristics of the EMOIS Dataset

B. Spatial Emotion Prediction Performance

  • CLIP mean R2 was 0.865 for valence and 0.807 for arousal, exceeding ViT-B/16 and ResNet50 on both dimensions.Figure 3 shows true-versus-predicted mean ratings for the median repeated hold-out evaluation, with R2 values of 0.867 and 0.807, respectively.Source: B. Spatial Emotion Prediction… · p. 7, Figure 3 · p. 7
  • Across visual representations, prediction was consistently higher for valence than arousal; the authors note that broader valence ratings and lower inter-rater variability may have contributed.Source: B. Spatial Emotion Prediction… · p. 7

C. Local Affective Consistency in Visual Representations

Local retrieval preserves affective ratings more strongly for valence than arousal, while global analyses show stronger organization for valence, particularly in CLIP.

C. Local Affective Consistency in Visual Representations

  • Valence retrieval consistency increased from ResNet50 to ViT to CLIP: correlations were 0.837, 0.877, and 0.914, and MAEs were 0.334, 0.299, and 0.260.Top-five neighbors excluded images from the query’s built-environment group; all pairwise retrieval-error differences were significant (p<0.001).Source: C. Local Affective Consistency in… · pp. 7–8, Table III · p. 8
  • Arousal retrieval correlations were 0.767, 0.805, and 0.799 for ResNet50, ViT, and CLIP, with MAEs of 0.314, 0.294, and 0.292; ViT and CLIP did not differ significantly.Both transformer representations differed significantly from ResNet50 for arousal retrieval errors.Source: C. Local Affective Consistency in… · p. 8

D. Global Affective Organization of Visual Representations

  • In UMAP, valence showed a smoother apparent gradient, especially for CLIP, while arousal formed multiple localized regions in both representations.Figure 4 colors aligned ViT and CLIP embeddings by mean valence in the top row and arousal in the bottom row.Source: D. Global Affective Organization of… · p. 8, Figure 4 · p. 8
  • For valence, CLIP had higher Moran’s I than ViT (0.826 vs. 0.736) and lower Geary’s C (0.209 vs. 0.303); arousal differences were smaller.Arousal Moran’s I was 0.590 for CLIP and 0.553 for ViT, while Geary’s C was 0.393 and 0.404, respectively; the analyses and UMAP visualizations provide complementary evidence.Source: D. Global Affective Organization of… · p. 8

E. Relationship Between Representation Distance and Affective Distance

Representation distance generally increased with affective distance in both ViT and CLIP, with the strongest relationship for combined valence-arousal distance and the weakest for arousal. CLIP showed stronger correspondence than ViT for valence and combined distance, but not clearly for arousal.

E. Relationship Between Representation Distance and Affective Distance

F. Cross-Dataset Comparison of Affective Organization

EMOIS and OASIS showed different affective organization patterns in CLIP representation space: EMOIS organized valence more strongly, while OASIS organized arousal more strongly. The example-based interface retrieves affectively rated visual references, illustrated through three built-environment cases.

F. Cross-Dataset Comparison of Affective Organization

G. Case Study of the Example-Based interpretation Interface

  • The interface presents visually similar EMOIS reference spaces with subjective affective ratings so users can interpret a query’s predicted affective values by comparison.It supplements numerical predictions rather than explaining why a prediction was generated.Source: G. Case Study of the Example-Based… · p. 10
  • The office-meeting-room query received moderate valence (4.05±0.28) and moderate-to-low arousal (3.83±0.28), with office-like retrieved examples and cosine similarities of 0.93–0.94.Source: G. Case Study of the Example-Based… · p. 10
  • The washitsu query fell in the high-valence, low-arousal region, and retrieved Japanese-style rooms had valence above 5.0 and arousal from 2.42 to 3.05.Source: G. Case Study of the Example-Based… · p. 10
  • The factory-like query had low valence and elevated arousal near the edge of the affective distribution, where retrieved references had lower similarity (0.87–0.90) and more varied ratings.Source: G. Case Study of the Example-Based… · p. 10

VI. DISCUSSION

Across analyses, valence was more coherently organized than arousal within EMOIS, while EMOIS and OASIS differed in their affective organization. The discussion notes scope limits and frames example retrieval as interpretive support rather than an explanation of model predictions.

VI. DISCUSSION

  • Within EMOIS, valence showed more coherent global organization than arousal; CLIP prediction achieved mean R2 values of 0.865 for valence and 0.807 for arousal.Source: §VI · pp. 10–11
  • The authors suggest consistent perceptual characteristics may contribute to valence’s stronger predictability and organization, but the characteristics underlying this tendency remain unclear.Source: §VI · p. 11
  • EMOIS showed stronger organization for valence than arousal, whereas OASIS was more balanced, with slightly stronger arousal organization; explanations based on dataset differences remain speculative.Source: §VI · pp. 11–12
  • The retrieval interface may support interpretation through rated visual examples, but it does not explain why predictions arise, and its practical usefulness awaits controlled user studies.Source: §VI · p. 12
  • EMOIS ratings came from participants whose nationality and longest residence were both in Japan, which may limit generalizability; its comparisons also use one built-environment and one general-image dataset.Source: §VI · p. 12
  • Valence and arousal may not fully characterize emotional experience; future work could examine richer impressions such as coziness, openness, healing, and tension.Source: §VI · p. 12

VII. CONCLUSION

The conclusion reports that CLIP representations predict affective impressions of built environments and organize valence and arousal distinctly; comparison with general affective images also suggests differing affective organization. It presents example-based retrieval as an interpretation aid and calls for independent and cross-cultural validation.

VII. CONCLUSION

  • CLIP representations achieved high prediction performance for built-environment affective impressions under repeated group-wise hold-out evaluation.Source: §VII · pp. 12–13
  • Valence and arousal showed distinct organizational patterns in CLIP representation space, and comparison with a general affective image dataset suggested differing affective organization.Source: §VII · p. 13
  • Visually similar reference environments paired with subjective affective ratings can support interpretation of model predictions.Source: §VII · p. 13
  • Independent built-environment datasets and cross-cultural annotations are needed to assess robustness and generalizability.Source: §VII · p. 13

GENERATIVE AI USE DISCLOSURE

  • The authors report using ChatGPT for language editing, grammar refinement, code-development assistance, and example-image generation, with authors validating the analyses and manuscript.Source: GENERATIVE AI USE DISCLOSURE · p. 13
Loading 2609.06870v1…