Bouba Beats Kiki, and Other Regularities in the Subjective Preferences of LLMs

1 Introduction

• The study examines choices among acceptable alternatives rather than asking which output is objectively right, including subjective factors in tasks such as naming a child.Expert feedback may be costly and may leave the relative importance of factors such as novelty and capital requirements uncertain.Source: §1 · p. 2
• The study focuses on short-form comparisons of numbers, human names, and fictional city names to make features clearer than in longer text.The authors describe the findings as empirical regularities in generating option lists and comparing items.Source: §1 · pp. 2–3
• The authors investigate 29 potential biases in pairwise choices across domains, including preferences for particular lengths, sounds, or associations.The analysis also considers biases arising when the model generates the list from which comparison pairs are drawn.Source: §1 · p. 2
• Small prompt changes can alter comparison outcomes and activate or depress particular biases, sometimes without an obvious explanation.Asking which item the model would select first may activate a preference for alphabetically earlier items.Source: §1 · p. 3
• Prompts can favor implicit categories in generated city lists, so comparisons may depend on candidate structure beyond fixed item features.Generated lists can contain contiguous runs of names based on themes such as seasons or compass directions.Source: §1 · p. 3

Comparable objects

• The study uses 200 sampled numbers, 100 common female and 100 common male US names, and 199 fictional city names.The names come from 2020 US Census data; the fictional city names were generated by Gemini 2.5 Flash.Source: §2 · p. 3

Prompt expansion

• The authors create 50 prompts through iterative paraphrasing of a seed question, reusing them across domains with the relevant item type substituted.Each round of prompt generation avoids duplicates by providing the model with prompts already generated.Source: §2 · p. 3

Combination

• For each domain, 200 objects form 200 unordered pairs; each pair is tested in both orders and with all 50 prompts, yielding 20,000 comparisons.The pairs are generated from a random cycle so each object appears in exactly two pairs.Source: §2 · pp. 3–4

Experimental settings

• Comparisons use temperature 1 and model-default thinking settings unless otherwise specified; figures often show Gemini 2.5 Flash, with broader model results in appendices.The authors report that results for other state-of-the-art models are broadly similar.Source: §2 · p. 4

3.1 Observed Preferences

• The analysis estimates feature-linked win fractions and logistic-regression coefficients, using pair-level tests with Bonferroni correction across 183 comparisons.The feature set was robust but non-comprehensive; the regression adjusts for correlations among features, and each unique unordered pair contributes once to significance tests.Source: §3.1 · p. 4
• Models consistently favored larger and prime numbers, recent-decade baby names, and phonetic features including vowel starts, higher vowel fractions, and soft “bouba” sounds.Reported win rates were 0.70–0.79 for larger numbers, 0.64–0.73 for primes, and below 0.42 for historical names; vowel starts reached 0.76 for names and 0.64 for most cities, while soft sounds exceeded 0.57.Source: §3.1 · p. 5
• Preferences also diverged by domain and model: longer city names were disfavored, some models avoided number multiples, and presentation-order effects differed between names and cities.Names showed consistent primacy (0.82 for Haiku), while cities showed recency for Gemini 3 Flash (0.47) and GPT-5 Mini (0.39).Source: §3.1 · p. 5

3.2 Prompt variation

• Prompt choice significantly activated or suppressed preferences: all 21 per-prompt winning-fraction distributions differed from expected noise (p < 0.001).Levene’s statistic tested whether the observed dispersion could be explained by natural noise under a null of no prompt effect.Source: §3.2 · p. 6
• Prompt influence was stronger for most number preferences than for names or cities; prompt preferences also correlated across datasets, with “put first” wording an interpretable first-sorted outlier.The same prompt was likely to show a first-presented preference across names and cities; prompt 23 elicited alphabetically or numerically first choices.Source: §3.2 · p. 6
• Prompt wording features tracked different biases: imperative prompts favored first-presented items, while parentheticals favored second-presented items for words.Prompts categorized as “Decisive” or “Judgmental” also showed stronger preferences than those categorized as “Aesthetic” or “Psychological.”Source: §3.2 · p. 8

3.3 Ideation

• Generated lists can form contiguous thematic categories, and newer models may impose such structure even when not requested.The 200-city list contained thematic runs; the authors note that newer model families often partition long lists into categories, suggesting this structure is not limited to an older model.Source: §3.3 · pp. 8–9
• Those endogenous city categories were differentially preferred: ancient, mythological, and Sims 4 cities were favored, while medieval-trade and chess-strategy names were disfavored.The category preferences appeared across essentially all prompts relative to randomly shuffled category assignments, with robustness checks addressing position effects.Source: §3.3 · p. 9

Conclusion

• The study finds that preference strength and direction in short-form comparison tasks can depend on prompts, and that ideation can spill over into comparison.When models generate long lists, they iterate through thematic categories; options from some categories are then preferred over others, with preferences mediated by prompts.Source: §4 · pp. 9–10

Limitations

• The study uses a selected set of frontier or near-frontier models, emphasizing smaller models to reduce computational cost.The authors suggest that a wider range, including more powerful and expensive models, could be studied.Source: §4 · p. 10
• The paper covers a particular set of subjective comparison tasks, while noting that the broader task space remains open to study.The authors describe their framework as a roadmap adaptable to other subjective comparison tasks.Source: §4 · p. 10

Further directions

• Future work could extend the prompt taxonomy and examine ideation and pruning of longer texts.The authors note that their taxonomy of prompt effects is incomplete and that longer texts offer more structure for models to use as preferences.Source: §4 · p. 10

A Related Work

• Prior LLM-judge research documents preferences such as positional bias, sycophancy, and self-preference, while this work studies how such preferences interact and identifies others systematically.The cited work also reports that chain-of-thought text may not correspond to a model’s decision basis (Turpin et al. [2023]).Source: A Related Work · p. 15
• Other research relates LLM preferences to human behavioral biases, including in-group bias, gendered language, multilingual bias, and pseudo-random number generation.Jones and Steinhardt [2022] suggest using human cognitive biases as an analogy for probing LLM behavior.Source: A Related Work · p. 15
• Bias measurement and evaluation frameworks have been proposed, with applications including model assessment, translation quality, and natural-language critiques.Examples include Prometheus, PandaLM, and Auto-J.Source: A Related Work · p. 15
• Ideation research spans domains such as business and research ideas, baby names, and product or fictional-character naming; this study focuses on shorter-form tasks to isolate option properties.Li et al. [2025b] survey approaches to generating candidate lists before refinement.Source: A Related Work · p. 15

B.1 Monte Carlo Permutation Test

• The test repeats 25,000 random shuffles of entities per domain, using each simulated ranking to resolve the same pairwise comparisons and calculate feature win rates.This generates a null distribution for preferences based on rankings unrelated to the feature being assessed.Source: §B.1 · p. 16
• The procedure tests whether observed feature preferences could be explained by a hidden ranking unrelated to the feature.Each simulated ranking resolves the same pairwise comparisons as the observed analysis.Source: §B.1 · p. 16
• Figure 6 displays Monte Carlo win-rate histograms, gray 95% confidence intervals, and colored vertical lines for the LLM-observed win rates reported in Table 1.The figure compares the simulated distributions with observed rates; the supplied description does not state a specific outcome of that comparison.Source: §B.1 · p. 16

B.2 Causal Evaluation via Minimal Pairs

• The evaluation varies one target feature in each minimal pair while holding other identified features as constant as possible, sampling across covariates with K-Means stratification.Source: §B.2 · p. 16
• The observational logistic regression predicts causal win rates with Pearson r = 0.941 and MAE = 0.051 for numbers, and Pearson r = 0.884 and MAE = 0.078 for names.Source: §B.2 · p. 16
• In numbers, Mult 10 has a causal win-rate increase of +31% to +35% relative to a baseline of 0.5; in names, Higher Frac of 2020 Babies increases by +25% to +36%.Source: §B.2 · p. 16

• ARI of 0.50–0.65 across five shuffled lists indicates robust recovery of the original 19 city categories, regardless of list order.The re-categorizations showed high semantic agreement with the original labeling.Source: §B.3 · p. 16

• The crossed-effects GLMM accounted for variation across both prompts and item pairs to distinguish prompt-level variance from sampling noise.Source: §B.4 · p. 16
• 1–20% of observed win-rate variance typically came from prompts, while 3–43% came from item pairs; prompt variance exceeded sampling noise for nearly all models and datasets (p < 0.001).The analysis decomposed per-prompt win-rate variance and used Levene’s test to compare prompt variance with sampling noise.Source: §B.4 · pp. 16–17, Table 2 · p. 17

B.5 Model Consistency: Feedback Arc Sets

• A minimum feedback arc set counts edges that conflict with the best possible item ordering; its size measures inconsistency in preference judgments.The study compares majority-vote graphs, per-prompt graphs, and graphs with randomly directed edges.Source: §B.5 · p. 17
• Individual prompts were more consistent than random guessing, but per-prompt FAS sizes varied: names generally ranged from 10–28% of edges, while number prompts ranged from 0.2% to 1/3.For names, the two most inconsistent prompts had FAS sizes of approximately 28%; for numbers, the prompts ranged from nearly consistent to highly inconsistent.Source: §B.5 · pp. 17–18
• 17% of edges were in the majority-vote FAS for numbers, versus 9% for cities and 7% for names; this analysis used an older dataset and different pair design.The older names dataset used an LLM-generated list rather than the census-based list used in Section 3.1.Source: §B.5 · p. 17, Figure 7 · p. 20

C.1 Cities and categories

• The city-category table reports each category’s range of positions in the original generated list; Gemini assigned categories without context about how the names were generated.Source: §C.1 · p. 21, Table 3 · p. 22

D Prompt questions for comparisons: set of 50

• Table 4 lists all comparison-question prompts and their relevant categories assigned by Gemini; Tables 5 and 6 give Gemini’s definitions of the linguistic categories and semantic domains.Source: Table 4 · p. 23, D Prompt questions for comparisons:… · p. 23, Table 5 · p. 24

E Other prompts used

• The prompt appendix includes instructions to generate ten paraphrases of the base question while avoiding duplicates among earlier prompts.It also documents a city-categorization prompt that prioritizes clear, meaningful categories over list-order patterns.Source: E Other prompts used · p. 25

E.1 Subjective Preference Comparison Prompt Template

• The standardized comparison template inserts the question phrasing and two candidates, then requests only the selected item; a separate prompt extracts decision factors from thinking traces.Source: §E.1 · pp. 25–26

F Summary Table of Preference Features

• The feature summary defines preference features across domains, including digit order, vowel fraction, length, census frequency, and alphabetical order.Source: F Summary Table of Preference Features · p. 27

H Factors named in thinking traces

• The thinking-trace analysis used a separate, older LLM-generated names dataset, and the names analysis was limited to 200 unique pairs.A second LLM extracted and ranked factors in the traces; the number results are shown in Figure 8.Source: H Factors named in thinking traces · p. 28
• For names, model reasoning frequently cited phonetics, cultural associations, and perceived “modernity” as decisive factors.The analysis covered 200 unique name pairs and is shown in Figure 9.Source: H Factors named in thinking traces · p. 28, Figure 9 · p. 28

I Broader Impacts

• For short-form comparisons, the paper recommends heterogeneous prompts to offset preferences inherent to any one prompt or prompt style.The authors say the comparison methodology can be applied to a wider range of subjective comparisons.Source: I Broader Impacts · p. 29

J.1 Tables of Results for Other Datasets and Model Settings

• Table 8 reports win fractions across temperatures T = 0.1, 0.7, and 1.0 for several models; the selected GPT model is shown only at T = 1.0.The supplied text does not provide the table’s cell values.Source: §J.1 · p. 31, Table 8 · p. 31
• Table 9 reports biases for fictional city names across all models, complementing Table 1’s results for Numbers and Names.The caption notes Bonferroni-corrected significance markers for 183 tests.Source: §J.1 · p. 31, Table 9 · p. 31

J.2 Regression Coefficients for Other Datasets

• Regression coefficients for the Numbers and Cities datasets complement the Names results in Figure 1.Figures 10 and 11 show the coefficients with 95% confidence intervals.Source: §J.2 · p. 31, Figure 10 · p. 32, Figure 11 · p. 32

J.3 Prompt Variation Histograms

• Per-prompt winning-fraction distributions are shown across all three datasets for Gemini 3 Flash, GPT-5 Mini, and Qwen 3.5 35B.These figures correspond to Figure 2 in the main text.Source: §J.3 · p. 31, Figure 12 · p. 33, Figure 13 · p. 33

J.4 Prompt Robustness Across Datasets

• Prompt-dependent preferences are correlated between datasets for Gemini 3 Flash and GPT-5 Mini.Figures 15 and 16 present these correlations, corresponding to Figure 3 in the main text.Source: §J.4 · p. 32, Figure 15 · p. 34, Figure 16 · p. 35

J.5 Prompt Behavior Correlations Across Models

• Prompt-dependent preferences are compared across model families for the Names and Numbers datasets.Figures 17 and 18 show PairGrid correlations across all evaluated SOTA models for Names and Numbers, respectively.Source: §J.5 · p. 35, Figure 17 · p. 36, Figure 18 · p. 37

NeurIPS Paper Checklist

• The checklist describes the study’s short-form, stylized scope, reproducibility provisions, and discussion of potential societal impacts.It states that prompts and experimental details are provided, but not all code can be released, and notes discussion of both positive and negative impacts, including prompt exploits.Source: NeurIPS Paper Checklist · pp. 38, 39, 41
• The authors answer “Yes” that LLM use is an important, original, or non-standard research component, citing city-name generation, prompt expansion, and categorization.The justification locates these uses in Sections 2 and 3.2.Source: NeurIPS Paper Checklist · p. 42