Source-linked AI summary
How AI Experiences Art: Emergent Aesthetic Structure in a Self-Supervised Multimodal Embedding Space
Corey D. C. Heath
TL;DR
The paper addresses how AI forms aesthetic categories for human-produced multimodal media without human labels or cross-modal pairs. It trains a self-supervised shared embedding and iterative clustering framework, finding 28 semantically coherent clusters that partially track but diverge from human registers. The authors identify labeling and source-collection biases as important limitations.
Problem
The study asks how AI categorizes aesthetic experience across modalities without explicit human labels and whether its categorizations meaningfully diverge from human categories.
Method
A self-supervised framework projects text, audio, images, and video into a shared 256-d space and iteratively refines HDBSCAN pseudo-labels with SupConLoss.
Results
28 semantically coherent clusters emerge, with NMI=0.40, ARI=0.15, and purity=0.70 against human-aligned clusters.
Takeaways & Limitations
AI-generated structure partially tracks human aesthetic registers while discovering finer-grained and orthogonal organization across modalities.
Takeaways & Limitations
Human labels were assigned at the collection level, while uncurated sources introduced bias, especially toward canonical Western literary works in text.
Abstract
from arXiv · showhide
Aesthetics are an important part of the symbolism of artistic works. Although subjective, humans categorize art based on the emotion evoked regardless of modality. What remains under-explored is how AI models form their own aesthetic categorization of human-produced media without explicit labels or cross-modal supervision. We present a self-supervised framework that projects four modalities (text, audio, image and video) into a shared 256-dimensional embedding space and applies iterative clustering to discover aesthetic structure. We discuss the divergence between AI-generated cluster assignments and human affective register labels on a weakly supervised multimodal dataset. This work has applications in understanding how AI structures cross-modal similarity, organizing heterogeneous media collections for Retrieval-Augmented Generation (RAG), and automated data labeling.
1 Introduction
The paper asks how AI categorizes aesthetic experience across modalities without explicit human labels and whether those categorizations diverge from human judgments. It introduces a self-supervised multimodal framework for discovering aesthetic organization.
- The study examines whether AI understanding extends to aesthetics across text, audio, images, and video without explicit human labels.
- The framework embeds multimodal content, projects modalities into a shared space, and uses clustering to categorize samples.
- The dataset is weakly supervised and based on publicly available media evaluated across multimodal content.
- The primary contributions are self-supervised aesthetic self-organization and divergence analysis between AI and human aesthetic topologies.
2 Related Works
Prior work studied self-supervised aesthetics mainly through photographic quality and modality-specific affective classification. This paper instead develops cross-modal aesthetic alignment without paired samples or explicit cross-modal supervision.
- Earlier self-supervised aesthetic research focused on photographic quality, whereas this work studies affective-experiential representations in media.
- The project grounds aesthetic classification in the valence-arousal scale and prior research on emotional responses to images and audio.
- Unlike ImageBind, the architecture requires no cross-modal pairs and learns shared-space alignment from self-supervised clustering alone.
- The framework extends self-supervised cluster discovery from a single visual modality to four modalities simultaneously.
3 Methodology
The methodology collects weakly supervised text, audio, image, and video data, projects modality-specific embeddings into a shared 256-dimensional space, and iteratively refines clusters. Convergence yields stable pseudo-labels used to organize and describe the data.
- Dataset: The dataset contains six valence-arousal-inspired registers across poems, prose, audio, images, and video, with long audio and video sampled for 120 seconds.
- Embedding extraction: Modality-specific backbone models first extract embeddings from each sample before projection into the shared representation.
- Shared embedding: MLPs project each modality into a 256-d unit-norm representation, while frozen backbone encoders provide the initial embeddings.
- Iterative training: HDBSCAN creates pseudo-labels, then SupConLoss trains the MLPs for 50 epochs by bringing same-cluster samples together and separating different-cluster samples.
- Convergence: Training repeats pseudo-labeling and MLP optimization until sequential iterations exceed an Adjusted Rand Index of 0.9.
4 Results
The framework discovers 28 multimodal aesthetic clusters without human supervision, producing finer-grained and more homogeneous partitions than six human-aligned registers. AI clustering moderately overlaps with, but substantially reorganizes, human labels, revealing text-dominated structures that cut across emotional categories.
- Cluster structure: 28 semantically coherent aesthetic clusters emerge by convergence, compared with six human-aligned registers, and the AI partitions are more homogeneous.The converged clusters have mean purity 0.77, versus 0.72 for the initial modality-dominated clusters.
- Divergence from human labels: NMI is 0.40, ARI is 0.15 and purity is 0.70 between AI-generated and human-aligned clusters.These values indicate moderate shared information, different structuring, and non-arbitrary assignment, respectively.
- Modality structure: Of 28 converged clusters, 25 are predominantly anchored to a single modality, while three have more balanced modality representation.The exceptions are cross-human-register rather than cross-modal clusters.
- Cross-register structure: Timeless-Nostalgia and Soulful-Contemplation form cross-register exceptions, drawing from all six human registers while remaining text-dominated.The corresponding human-aligned region is the most heterogeneous, containing samples from every assignable label.
- Aesthetic dimensions: Soulful-Contemplation spans elegiac, grotesque and idyllic registers, suggesting an internally motivated and reflective dimension independent of explicit emotion.These registers occupy three valence-arousal quadrants, while sublime, uncanny and pastoral are less represented.
- Limitations: Older language and writing in public-domain sources may influence categorization of contemporary data and contribute to poor clustering on human assignments.The paper frames this as a possible bias related to literary temporality.
5 Conclusion
The framework discovers 28 semantically coherent multimodal aesthetic clusters without cross-modal pairs or human supervision. Its structure partially tracks human aesthetic registers while also producing finer-grained, orthogonal organization, though collection-level labels introduce noise into the divergence baseline.
- 28 semantically coherent clusters emerge from multimodal data without cross-modal pairs or human supervision.The framework uses clustering signal alone to organize text, audio, image, and video data.
- NMI=0.40, ARI=0.15, and purity=0.70 indicate partial alignment with human registers alongside finer-grained, orthogonal AI structure.The low ARI and higher purity characterize AI assignments as different but non-arbitrary relative to human-aligned clusters.
- Collection-level human register labels introduce noise into the divergence baseline.The paper identifies item-level annotation as a needed improvement for cleaner ground truth.
Ethics and Privacy Statement
The study used publicly available archives and automated collection without intervention involving living human subjects. Its data practices followed platform rules and excluded private or personally identifiable information.
- The study collected data from publicly available archives using an automated web tool.
- Because collection involved no intervention, interaction, or solicitation with living human subjects, formal IRB oversight was not required.
- Collection followed terms of service, robots.txt protocols, and user agreements while excluding PII, proprietary private records, and restricted credentials.Downloaded samples were processed, stored, and analyzed in aggregated, de-identified form.