Source-linked AI summary
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, Anders Søgaard
TL;DR
The paper asks whether text-only language models encode perceptual structure, a question it studies through color-term representations and CIELAB color-chip coordinates. Using representation-similarity and linear-mapping analyses, it finds topological correspondence, with warmer colors aligning better on average and alignment varying with collocational and syntactic usage.
Problem
Whether text-only language models reflect perceptual structures such as color topology remains poorly understood.
Method
The study compares text-derived representations of color terms with CIELAB representations of corresponding color chips using representation similarity analysis and linear mapping.
Results
Text-only color-term representations show topological correspondence with perceptual color space, with warmer colors better aligned on average and alignment associated with collocational and syntactic usage.
Takeaways & Limitations
Perceptual color-space structure is partly reflected in distributional statistics and more closely in language-model representations, while alignment varies across the color spectrum and language use.
Takeaways & Limitations
Complete isomorphism is unlikely because color terms are not simply denotational and language is influenced by factors beyond perception.
Abstract
from arXiv · showhide
Pretrained language models have been shown to encode relational information, such as the relations between entities or concepts in knowledge-bases -- (Paris, Capital, France). However, simple relations of this type can often be recovered heuristically and the extent to which models implicitly reflect topological structure that is grounded in world, such as perceptual structure, is unknown. To explore this question, we conduct a thorough case study on color. Namely, we employ a dataset of monolexemic color terms and color chips represented in CIELAB, a color space with a perceptually meaningful distance metric. Using two methods of evaluating the structural alignment of colors in this space with text-derived color term representations, we find significant correspondence. Analyzing the differences in alignment across the color spectrum, we find that warmer colors are, on average, better aligned to the perceptual color space than cooler ones, suggesting an intriguing connection to findings from recent work on efficient communication in color naming. Further analysis suggests that differences in alignment are, in part, mediated by collocationality and differences in syntactic usage, posing questions as to the relationship between color perception and usage and context.
1 Introduction
The paper asks whether text-only language models encode perceptual structure grounded in the world, using color as a case study. It evaluates alignment between language-based color representations and the perceptual topology of CIELAB space.
- Research question: Language models encode linguistic, semantic, factual, and relational information, but their reflection of perceptual structures such as color topology remains poorly understood.Few studies have empirically examined whether text-only models capture structures grounded in human perception.
- Motivation: The study uses color because perceptual color spaces relate Euclidean distances to reported perceptual differences, while color-term semantics may reflect constraints from visual perception.Color naming has also been studied extensively in cognitive science and linguistic theories of universal constraints.
- Approach: The experiments compare linguistic representations of English color terms with corresponding color-chip coordinates in CIELAB using two structural-alignment methods.The setup measures correspondence between the two spaces through representation similarity analysis and learned linear mappings.
- Findings: Language-model representations show alignment with CIELAB structure, indicating that they encode some approximation of perceptual color-space topology.The paper studies this correspondence through a dataset of color terms and corresponding chips represented in CIELAB.
- Findings: Simple co-occurrence statistics capture part of the distributional signal, while large-scale language-model pretraining encodes the topology more strongly.The comparison includes PMI-based models and BERT.
- Findings: Larger models align better than smaller ones, low-dimensional representation subspaces explain much of CIELAB variance, and warm colors align better than cool colors on average.The warm–cool difference connects to findings that warmer colors are communicated more efficiently.
2 Methodology
The methodology combines a color-naming dataset, controlled language-model representations, perceptual CIELAB coordinates, and baseline distributional representations. Structural correspondence is evaluated with representation similarity analysis and linear mapping.
- Perceptual color space: CIELAB represents each color with lightness, red–green position, and blue–yellow position, with distances corresponding to perceptual differences.The space is three-dimensional and is used as the perceptual reference.
- Language models: The study analyzes BERT, RoBERTa, and ELECTRA, which use masked-language-modeling or token-replacement-detection objectives.These models provide the primary language-model representations.
- Baselines: Baselines include PMI vectors computed with window sizes 1, 2, and 3, plus word-type FastText embeddings trained on Common Crawl.PMI vectors quantify color-term co-occurrence likelihoods with vocabulary items.
- Representation extraction: Color-term representations are extracted without context or from sampled or controlled contexts, using identical templates to reduce contextual variation across terms.The templates include copula, possession, and spatial frames, aggregated over object nouns.
3 Evaluation
The evaluation compares color-term representations with perceptual CIELAB structure using RSA and learned linear mappings, alongside controls for probe memorization and correspondence.
- Representation Similarity Analysis: RSA measures global topological alignment by correlating representational similarity matrices from language-model embeddings and CIELAB space.Kendall’s τ summarizes the correspondence between pairwise similarity structures.
- Representation Similarity Analysis: Color-term centroids are computed in CIELAB and language-model embedding spaces, excluding terms used fewer than 100 times.The resulting analysis retains the 18 most commonly used color terms.
- Representation Similarity Analysis: The CIELAB similarity function converts squared perceptual distances into bounded similarity values using a scaling factor c = 0.001.Distances are measured with ∆E_CMC∗.
- Linear mapping: Regularized linear regression maps color-term embeddings X ∈ R^n×dLM to CIELAB coordinates Y ∈ R^n×3, with probe complexity varied through the nuclear norm of W.The mapping uses a linear transformation with lasso regularization.
- Controls: Randomly remapping CIELAB codes breaks the chip-term correspondence, and probe selectivity is defined as explained variance in the standard condition minus the control condition.The control results average over 10 random remappings.
4 Results
Language-model representations show significant correspondence with CIELAB structure across configurations, while alignment varies by color, context, and representation setting.
- RSA results: Significant RSA correlations occur across language-model configurations, and randomly permuting CIELAB centroids removes significance.The permutation control supports alignment with CIELAB structure rather than arbitrary correlation.
- Color-specific alignment: Colors such as violet, orange, and purple have higher correlations, driven largely by aligned similarity and dissimilarity clusterings.Their partial orderings within clusters are not consistently aligned.
- Representation settings: NC embeddings show low alignment with the other representation settings, while their layerwise results have large variance and near-zero mean correlation.The paper attributes this pattern to language models likely not encountering isolated color-term tokens during training.
- Linear mapping results: CC achieves the highest selectivity scores among the evaluated context settings and baselines in the linear mapping experiments.The authors interpret controlled sentence context as revealing increased alignment to color space.
- Linear mapping results: High selectivity across probe complexities indicates that probe fitness reflects information in color-term representations rather than memorization.This conclusion is based on comparisons between the experimental condition and the control task.
5 Analysis and Discussion
The analysis finds that color-space alignment varies systematically with model size, color temperature, surprisal, collocationality, and syntactic usage. Warmer colors align better with CIELAB, while varied syntactic contexts improve alignment and fixed collocations reduce it.
- Dimensionality of color subspace: ∼0.4 of CIELAB chip-code variance is explained using ∼10 dimensions, while 30–40 dimensions explain ∼0.7.The latter is nearly the proportion explained by the full representations.
- Model size: Alignment measured by both RSA and linear mapping progressively increases with model size.The authors interpret growing model complexity as moving color-term representational geometry toward isomorphism with CIELAB.
- Color temperature: Warm color-term embeddings are better aligned to CIELAB than cool ones across models and configurations.The analysis connects this pattern to findings that warmer colors are communicated more efficiently.
- Connection to listener surprisal: Lower ranking correlates significantly with higher chip surprisal across all language models and configurations (0.12 ≤ρ ≤0.17, p < 0.05).Higher surprisal indicates lower naming consensus among speakers.
- Corpus-derived predictors: Terms occurring in more fixed collocations are less aligned to the perceptual space, with pmi-col significantly improving fit beyond log frequency and receiving a negative coefficient.The authors associate this with metaphorical contexts such as Red Army or Black Death.
- Corpus-derived predictors: RSA is higher for terms used in more varied dependency relations and modifying more diverse syntactic heads, while adjectival-modifier frequency also predicts higher scores.POS entropy, adjective proportion, and copula proportion do not provide the same significant evidence.
- Vision-and-Language models: Preliminary Vision-and-Language experiments find no major differences from the text-only models.
6 Related Work
Related work studies how distributional representations encode semantic and factual information, while newer research tests whether language-model representations connect text with visual properties and representations. These studies motivate examining perceptual structure beyond conventional lexical or relational information.
- Earlier work evaluated distributional representations against human semantic similarity and relatedness judgments, while questioning whether distributional statistics capture all semantic information equally.
- Subsequent studies queried language models for factual and commonsense knowledge and examined representations of object properties and affordances.
- Research on color-reporting bias found that language models can model relationships between color terms and modified nouns.
- Vision-and-language studies find that language models can retrieve semantically aligned image patches but still substantially underperform humans.
7 Outlook
The paper asks whether text-only representations can align with human perceptual color structure, finding topological correspondence while rejecting complete isomorphism. Alignment varies with color temperature and corpus usage, motivating further study of how language, perception, and context interact.
- Direct sensory experience is not the only route to color understanding, as congenitally blind people can develop striking knowledge of color through language and semantic associations.
- Text-derived color-term representations show topological correspondence with perceptual color spaces, with language-model representations aligning more closely than simple co-occurrence statistics.
- Warm colors show more alignment than cool colors on average, connecting the result to research on communication efficiency in color naming.
- Lower topological alignment correlates with higher color-chip surprisal, suggesting that the contexts in which colors occur help determine alignment.
- Complete isomorphism is unlikely because language and color terms are not purely denotational and are shaped by factors beyond perception.
F Linear mapping results by munsell color chip
Figure 14 breaks down linear-mapping results by Munsell color chip across all models and configurations.
- Linear-mapping results are shown separately for each Munsell color chip across all models and configurations.
G Linear mapping control task and probe complexity
Figure 15 reports results across probe complexities for both the standard experimental condition and a random control task.
- Results are evaluated over a range of probe complexities for the standard experimental condition and the random control task.
H Dimensionality of color subspace
This section examines explained variance as a function of dimensionality, alongside model-size settings used in the experiments.
- The dimensionality analysis plots the proportion of explained variance against the number of dimensions receiving 95% of the linear-regression coefficient weights.
- The model-size experiments use BERT-mini, BERT-small, BERT-medium, and BERT-base configurations ranging from 4 to 12 layers.
J Linear Mixed Effects Model
The analysis uses linear mixed-effects models alongside alignment analyses to relate color-term properties to representation–CIELAB correspondence. Results are organized by model/configuration, color temperature, chip identity, and experimental controls.
- Model construction: Linear mixed-effects models use model type as a random effect and add linguistic predictors through four nested levels.The sequence begins with color-term log frequency, adds PMI collocationality, then entropy measures and dependency-based proportion terms.
- Representation similarity: Figure 5 compares representation-similarity structure across models and configurations using Kendall’s correlation with CIELAB.The comparison uses each model’s layer maximally correlated with CIELAB and distinguishes random-context, controlled-context, and non-context configurations.
- Color-temperature analyses: The reported temperature analyses break down linear-mapping explained variance and RSA Kendall’s τ across warm and cool color chips.These breakdowns are shown separately for baselines and language models.
- Diagnostics and controls: Additional analyses track color-term log frequency, syntactic-distribution entropy, chip-level rankings, and explained variance under randomly permuted controls.The chip-level plot ranks predicted chips by Pearson distance from gold, while the control comparison contrasts normal and randomly permuted color-term assignments.