Source-linked AI summary
Large language models predict human sensory judgments across six modalities
Raja Marjieh, Ilia Sucholutsky, Pol van Rijn, Nori Jacoby, Thomas L. Griffiths
TL;DR
The paper asks how much perceptual information language can provide and whether language relates to perceptual representations. It elicits LLM similarity judgments and tests color naming in English and Russian, finding human-correlated judgments across six domains and language-dependent color categorization.
Problem
The paper examines how much information about perceptual experience can be extracted from language and whether perceptual representations vary with language.
Source: Color naming study · p. 2Method
The authors elicited LLM pairwise similarity judgments across perceptual domains and compared English- and Russian-language color naming with human data.
Source: Similarity study · p. 1, Color naming study · p. 2Results
LLM similarity judgments correlated significantly with human data across all six domains, and GPT-4 reached inter-rater-reliability-level performance on newly collected pitch and consonant datasets without fine-tuning.
Source: Similarity study · pp. 1–2, Discussion · p. 3Takeaways & Limitations
Different color categorizations for the same hex codes in English and Russian support the view that language can reshape perception.
Source: Discussion · p. 3The study focuses on population-level averages, and whether its evidence for cross-cultural differences extends to other languages remains to be tested.
Source: Discussion · p. 4
Abstract
from arXiv · showhide
Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science. We show that state-of-the-art large language models can unlock new insights into this problem by providing a lower bound on the amount of perceptual information that can be extracted from language. Specifically, we elicit pairwise similarity judgments from GPT models across six psychophysical datasets. We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality. To study the influence of specific languages on perception, we also apply the models to a multilingual color-naming task. We find that GPT-4 replicates cross-linguistic variation in English and Russian illuminating the interaction of language and perception.
Results
LLM similarity judgments correlated significantly with human data across all six modalities, with familiar perceptual structures emerging in pitch, consonants, and color. In color naming, GPT-4 captured some English–Russian differences but still diverged from human naming patterns.
Similarity study.
- Human–model Pearson correlations were significant across all six modalities; GPT-4 reached r = .92 for pitch and r = .89 for colors, while GPT-3.5 reached r = .89 for loudness.Source: Similarity study · p. 1
- GPT-4 ranked among the top two models in five of six domains, but its gains over other models were not especially large for colors.The reported improvement was Δr = .15 for colors, compared with Δr = .16 for pitch and taste and Δr = .14 for consonants; gains were absent for loudness and timbre.Source: Similarity study · p. 2
- MDS recovered the pitch spiral, with its twists corresponding to 12-semitone octave separations.The similarity pattern also showed a spike at the octave, consistent with octave equivalence.Source: Similarity study · p. 2
- GPT-4’s MDS solutions showed a familiar color wheel and a production-based consonant representation, consistent with its explanations invoking hue, brightness, color spectra, and articulation.Its pitch explanations referred to octaves, ratios, and harmonic relations.Source: Similarity study · p. 2
Color naming study.
- GPT-4’s Russian naming had an adjusted Rand index of 0.54, while its turquoise category covered 46 Munsell colors versus 15 in human data.The authors note that LLM color naming remained below a separate lab-based experiment’s constrained and free naming results.Source: Color naming study · p. 3
- GPT-4 captured Russian blue and light-blue distinctions, with sínij dominant for 18 Munsell colors versus 29 for humans and golubój dominant for 33.The supplied results also report GPT-4’s Russian violet and lilac categories as dominant for 27 and 20 Munsell colors, respectively, versus 32 and 18 for humans.Source: Color naming study · p. 3
Discussion
The discussion interprets the models as a way to extract quantitative psychophysical signals from text and notes that GPT-4’s color naming varies between English and Russian. The methods describe prompting, stimulus datasets, participant recruitment, and data availability, while the authors identify limits on generalization and interpretation.
Discussion
- The authors present flexible LLM prompting as a way to extract quantitative psychophysical signals from text and frame the work as exploring what information about the world language can recover.They also highlight combining human and machine experiments to study questions in cognitive science.Source: Discussion · pp. 3–4
- GPT-4’s color categorizations differed between English and Russian for the same hex codes, while it achieved IRR-level performance without fine-tuning on newly collected pitch and vocal-consonant datasets.The authors say language dependence suggests the physical stimulus alone is insufficient to explain the model’s naming behavior.Source: Discussion · p. 3
- The authors identify other perceptual measures, individual-level variation, cross-cultural coverage beyond English and Russian, and inherited biases as limitations or directions for further research.They advise benchmarking model behavior against human data.Source: Discussion · p. 4
GPT prompt elicitation.
- For color naming, GPT-4 first generated 15 basic color names, then named World Color Survey hex codes from shuffled lists in English and Russian across repeated prompts.The procedure elicited ten names per color and re-queried invalid outputs, discarding errors after ten attempts.Source: GPT prompt elicitation · p. 4
Stimuli.
- The stimulus section describes six similarity datasets spanning colors, pitch, vocal consonants, loudness, taste, and timbre, sourced from prior studies and newly collected data.Confusion matrices were converted to similarity scores using normalized confusion probabilities and a stated formula.Source: Stimuli · pp. 4–5
Behavioral experiments.
- The behavioral experiments recruited 55 participants for pitch, 64 for vocal consonants, and 154 for color naming in Russian and British English.Color-naming participants included 103 UK participants and 51 Russian participants.Source: Behavioral experiments · p. 5
Code and Data availability.
- The paper makes its data and code available, except for the Lindsey and Brown color-naming dataset, which is available from its authors upon request.It also provides interactive visualizations of modality spaces and human color-naming data.Source: Code and Data availability · p. 5
Extended Methods
The extended methods give prompt formats for eliciting numerical similarity ratings from GPT models across perceptual tasks. Prompts describe the task and rating scale, provide examples, and leave a target pair for the model to rate.
Explicit GPT prompts.
- GPT-3 used the Text Completion API, while GPT-3.5 and GPT-4 used Chat Completion; temperatures were 0.7 for similarity judgments and 0 for color naming.The paper states that the exact prompt formats are shown in the following examples.Source: Explicit GPT prompts · p. 6
Similarity judgments.
- Prompts specified the stimulus type and a 0–1 similarity scale, then used example stimulus pairs and ratings before an unrated target pair.For example, the color prompt supplied hex codes and requested a numerical similarity rating.Source: Similarity judgments · p. 6
- The examples cover pitch frequencies, loudness in decibels, flavors, instrument timbres, and vocal consonants, with target-pair fields left for model ratings.The consonant prompt identifies sounds using the International Phonetic Alphabet and requests a numerical response.Source: Similarity judgments · p. 6
Color naming.
The color-naming prompts elicit a list of basic color terms and ask the model to assign a name to each World Color Survey color from a fixed list. The procedure repeats shuffled-list trials and filters invalid responses.
Color naming.
- The English and Russian prompts ask for 15 basic color names, then ask which listed name best describes a supplied hex code.The task restricts responses to a name from the list.Source: Color naming · p. 7
- For each World Color Survey color, the prompt was repeated ten times with the basic-color list shuffled each time, and invalid outputs were re-queried.Responses still invalid after ten attempts were marked “error” and later discarded.Source: Color naming · p. 7
Similarity Experiments: Participants.
The pitch and vocal-consonant similarity studies recruited US-based Mechanical Turk participants, with prescreening and listening-quality checks before rating tasks. Participants rated randomly assigned stimulus pairs on a seven-point scale.
Similarity Experiments: Participants.
- The pitch study had 55 participants and the vocal-consonant study had 64, recruited in the United States through Amazon Mechanical Turk.Participants were paid $9–12 per hour and provided informed consent under the stated institutional approvals.Source: Similarity Experiments: Participants · p. 7
- Participants had to pass a headphone check and have completed at least 3,000 Mechanical Turk tasks before random assignment to stimulus pairs.They rated similarity from 0, completely dissimilar, to 6, completely similar; participants averaged 80 pitch judgments and 55 consonant judgments.Source: Similarity Experiments: Participants · p. 7
Similarity Experiments: Procedure.
Participants judged randomly assigned sound pairs for similarity, with task-specific instructions for pitch and consonants. Repeated trials provided a consistency-based quality measure used to calculate a small bonus.
Similarity Experiments: Procedure.
- After consent and a headphone check, participants rated randomly assigned sound pairs on seven-point scales; pitch listeners could replay sounds and were told no expertise was needed.The pitch instructions described responses from 0 (“Completely Dissimilar”) to 6 (“Completely Similar”).Source: Similarity Experiments: Procedure · p. 7
- Five repeated random trials supplied a Spearman consistency score used as a quality proxy, with bonuses capped at 10 cents.The bonus was min(max(0.0, 0.1s), 0.1), where s was the correlation between original and repeated responses.Source: Similarity Experiments: Procedure · p. 7
- For consonant judgments, instructions excluded vowel sounds and voice height, and participants rated the similarity of randomly paired consonant sounds.The instructions defined consonants as speech sounds produced by partly blocking airflow from the vocal tract.Source: Similarity Experiments: Procedure · p. 7
Similarity Experiments: Model Evaluation.
The evaluation compared model and human similarity matrices using correlations with bootstrap confidence intervals. The color-naming participants were native English or Russian speakers recruited in the UK and Russia.
Similarity Experiments: Model Evaluation.
- Pearson correlations between the flattened upper triangles of model and human similarity matrices measured agreement, with 95% confidence intervals bootstrapped over 1,000 repetitions.Each bootstrap repetition averaged model predictions before correlating the resulting matrix with human data.Source: Similarity Experiments: Model Evaluation · p. 8
Color Naming Experiments: Participants.
- The color-naming study recruited 103 UK participants and 51 Russian participants.Participants were paid 9 GBP per hour and gave informed consent under an approved protocol.Source: Color Naming Experiments: Participants · p. 8
- Participants had to be raised monolingually and speak the target language as their mother tongue; Russian-language materials were translated and manually checked by a native speaker.All experiment interface text was presented in each participant’s native language.Source: Color Naming Experiments: Participants · p. 8
Color Naming Experiments: Procedure.
The color-naming procedure elicited basic color terms before asking participants to choose labels for colored squares from a fixed list. The study checked response consistency and compared results with a control excluding incomplete sessions and color-blind participants.
Color Naming Experiments: Procedure.
- Participants first freely elicited at least eight basic color names, entered in lowercase without spaces, numbers, or special characters.The obtained lists highly overlapped with GPT-4’s list, which was used as the basis for the subsequent naming task.Source: Color Naming Experiments: Procedure · p. 8
- Among participants’ top 15 color terms, 11 English terms and 12 Russian terms overlapped with GPT-4’s list, which the researchers chose as the basis for the word-naming task.Source: Color Naming Experiments: Procedure · p. 8
- For each colored square, participants selected the best-fitting term from a shuffled list of 15 basic color names in English or Russian.The instructions warned that some colors could recur to check response consistency.Source: Color Naming Experiments: Procedure · p. 8
- Excluding color-blind participants and incomplete sessions produced nearly identical results: adjusted Rand index 0.92 for English and 0.97 for Russian.The main analysis retained participants who failed the color-blindness test and those who abandoned the experiment early.Source: Color Naming Experiments: Procedure · p. 8
Color Naming Experiments: Analysis.
The analysis selected dominant labels for each Munsell color and encoded agreement levels in the resulting color maps.
Color Naming Experiments: Analysis.
- For each Munsell color, the most frequently reported term was selected as its dominant label, with at least 10 responses collected per LLM variant and per human color.Human response means were 19.30 for English and 12.17 for Russian.Source: Color Naming Experiments: Analysis · p. 8
- Figure 3B’s legend and clustered colors used the average RGB values of colors assigned the same term, and the figure listed the number of Munsell colors associated with each term.The figure also conveyed agreement for each color through the dominant term’s selection frequency.Source: Color Naming Experiments: Analysis · p. 8
- Agreement marks indicated dominant-term selection below 50% (“-”) or from 50% through 90% (“*”); unmarked colors exceeded 90%.Source: Color Naming Experiments: Analysis · p. 8
Adjusted Rand index.
The adjusted Rand index compares color-name clusterings despite different vocabularies, with chance correction. Human English naming data aligned strongly with prior lab results, and comparisons across languages and experiments were consistently positive.
Adjusted Rand index.
- All adjusted Rand index values were strictly positive, indicating consistency across languages and experiments.Confidence intervals were computed from 1,000 pairs of bootstrap datasets, resampling each color’s responses and recomputing its dominant name.Source: Adjusted Rand index · p. 9
- The adjusted Rand index compares clustering partitions without relying on aligned labels, correcting the raw Rand index for chance agreement.It is close to 0 for random labeling, 1 for identical clusterings up to permutation, and reaches -0.5 for orthogonal clusters.Source: Adjusted Rand index · p. 9
Lindsey and Brown dataset.
- Human English naming matched Lindsey and Brown’s lab data: constrained naming had ARI = 0.73 95% CI [0.65, 0.74], and free naming had ARI = 0.75 95% CI [0.66, 0.75].The authors note that this supports the quality of online color-naming data despite less control over color presentation.Source: Lindsey and Brown dataset · p. 9