Source-linked AI summary
A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premium
Dmitrij Żatuchin
TL;DR
The paper asks whether a model selector favors its own model’s text beyond general generator and selector preferences. It rebuilds the crossed matrices from repository counts and tests an own-model term; the results show no detectable premium overall, within the scope of the five models studied.
Problem
The crossed design leaves open whether a selector favors its own model’s text beyond generator and selector effects, a distinction relevant to whether preference is model-specific or shared.
Source: §1 · pp. 1–2Method
Results
The pooled own-model premium is +0.019 share points, with 17 of 120 relabellings reaching or exceeding it, and no dataset reliably shows a generator’s own model as its keenest selector.
Source: §7 · p. 8Takeaways & Limitations
The matrices support a shared taste for model-written text, but do not show models recognizing and favoring their own prose.
Source: §7 · p. 8The findings concern five named models and do not address whether models can recognize their own text when asked or models released after 2024.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Laurito et al. (PNAS 2025) showed that large language models choosing between two descriptions of the same product, paper or film prefer the description written by a language model over the one written by a person, by a wide margin over what human judges do. Their design crosses five generators with the same five models as selectors, which permits a second question the paper does not headline: does a selector prefer text from its own model beyond what the generator and selector main effects predict? We rebuild the three 5x5 matrices from the per-item counts in the authors' public repository (21,828 valid trials; every cell matches the published value) and fit a two-way fixed-effects model with an own-model term gamma, tested by the exact permutation test over the 120 relabellings of the selectors. The premium is +0.013 on products (exact one-sided p = 0.24), -0.010 on paper abstracts (p = 0.74), +0.054 on films (p = 0.07) and +0.019 pooled (p = 0.14; 95% interval -0.008 to 0.046). The same-vendor term for the GPT-3.5 and GPT-4 pair is negative in all three datasets. Position bias moves single cells by up to 0.42 share points in either direction, and the own-model contrast is unchanged once order-driven items are removed. The design would have detected a premium of 0.05 with 82% (products), 88% (papers), 42% (films) and 97% (pooled) power; the minimum detectable effect at 80% power is 0.034 pooled. The absence is informative down to about 0.04 share points and silent below that. The 4x4 matrix of Tan et al. (ACL 2024) gives gamma = +0.148 at the smallest p its 24 relabellings allow, with a same-family term of the same size. The main result of Laurito et al. stands: models share a taste for model-written text, with GPT-4's descriptions chosen 77% to 95% of the time by every selector on products. What these data do not show is a model recognising and favouring its own prose.
1 Introduction
The analysis asks whether a selector favors its own model’s text beyond generator and selector effects, a question made possible by crossing the same five models in both roles. This matters because model-specific preferences would make text tuned for one engine work against others, unlike a shared preference for model-written text.
1 Introduction
- The original design crossed five generators with the same five selectors on products, paper abstracts, and film summaries, each paired with a human description.The item counts were 109 products, 100 paper abstracts, and 250 film plot summaries.Source: §1 · p. 1
- The motivating study found that selectors preferred model-written product descriptions over human-written ones by 26 to 59 share points across ten cells with human comparisons.The finding was called AI–AI bias and drew a published letter and reply.Source: §1 · p. 2
- The diagonal of the five-generator-by-five-selector matrices isolates whether a model favors its own output beyond generator and selector effects.The question is relevant to self-preference [4] [5]: model-specific preferences would tune text for one engine against others, whereas shared preference could serve them all.Source: §1 · p. 2
- The re-analysis rebuilds three matrices from per-item counts, tests the diagonal with an exact permutation test, and separates same-vendor effects.It also examines position bias, power, and the comparable matrix of Tan et al..Source: §1 · p. 2
2 Data
The analysis reconstructs the published generator-by-selector cells from run-level repository counts, selecting runs that reproduce the published rounded shares. It aggregates valid and invalid choices rather than using per-trial position records for every run.
2 Data
- Each repository run records per item the counts choosing model text, human text, or giving no valid choice; some January 2025 runs also record first-listed choices.Other runs lack per-trial position rows, so the analysis uses aggregates over items or cells and releases derived tables without source text.Source: §2 · p. 2
- All 75 cells match the published rounded values, using 21,828 valid and 1,280 invalid trials across the selected runs.The largest difference between a rebuilt share and its published rounding is 0.005.Source: §2 · p. 2
- For each cell, the analysis selects the latest run whose LLM-win share rounds to the published value; 29 cells have multiple distinct 2025 runs for sensitivity analysis.The paper dataset’s 100 papers appear as 93 items in every paper run.Source: §2 · p. 2
- The paper cell comparing GPT-4 text judged by GPT-3.5 uses the January 2024 run behind the published 0.69 value, rather than the 2025 rerun’s 0.71.Human-selector values are taken from the article’s figures and cover only the GPT-3.5 and GPT-4 texts.Source: §2 · p. 2
3 Method
The method separates generator and selector effects from an own-model diagonal premium in the crossed matrices, then tests whether selector identity predicts that residual. It also assesses position, same-vendor effects, trial-level inference, and the design’s power.
3 Method
- The fixed-effects model estimates each cell share from an overall mean, generator effect, selector effect, and own-model premium on the diagonal.In a full k × k matrix, the premium equals the mean diagonal share minus the mean off-diagonal share, in share points.Source: §3 · pp. 2–3
- The exact one-sided permutation test relabels the five selector columns in all 120 ways under the null that model identity adds no information beyond row and column effects.The pooled test double-centres the three matrices and fits a shared premium under a common relabelling; standard errors provide intervals without adding distributional assumptions to the permutation test.Source: §3 · p. 3
- A family indicator captures the two off-diagonal cells between GPT-3.5 and GPT-4, the only same-vendor pair.The family and self indicators are fitted together and permuted together.Source: §3 · p. 3
- The trial-level comparison fits binomial logistic regression to per-item counts with item, generator, and selector fixed effects, clustering standard errors by item.It also calculates cell-level Pearson dispersion against the binomial model’s predicted variance.Source: §3 · p. 3
- Position analysis uses 27 runs with first-choice records to distinguish content-decided from position-decided items and compares diagonal cells with off-diagonal cells in the same row.It reports first-choice rates, position-decided item shares, and LLM-win shares among content-decided items; 62 of 3,645 items with indeterminate position splits are excluded.Source: §3 · p. 3
- Power simulations add a diagonal premium to each matrix, then combine estimated cell-level interaction noise with binomial sampling at observed trial counts and rerun the exact test.The minimum detectable effect is the premium rejected at one-sided α = 0.05 in 80% of simulations; the larger-design power table uses pooled interaction noise, a base share of 0.7, and random row and column effects.Source: §3 · p. 3
4 Results
After accounting for generator and selector effects, the own-model premium is small and not statistically detectable in Laurito et al.’s three matrices. Position can substantially shift individual cells, while the study had high power for a pooled premium of 0.05; Tan et al.’s different matrix shows a larger premium alongside a similar same-family effect.
4.1 The own-model premium
- +0.013 on products (p = 0.24), −0.010 on paper abstracts (p = 0.74), +0.054 on films (p = 0.07), and +0.019 pooled (p = 0.14; 95% interval −0.008 to 0.046).Only 4 of 15 generator rows had their own model as the selector that liked its text most.Source: §4.1 · p. 3, Table 1 · p. 4
4.2 Same vendor
- −0.022 pooled (p = 0.72) for the same-vendor term; its estimates are negative in products, papers, and films.With only one same-vendor pair, the test has little power and cannot separate vendor from within-vendor size effects.Source: §4.2 · p. 4
4.3 Trial-level inference and why it overstates
- Trial-level estimates overstate evidence for a premium: cell shares vary three to seven times more than binomial sampling predicts, and film cells are especially noisy.Across distinct runs, the median within-cell spread was 0.012 share points and the maximum was 0.161; using mean or latest runs per cell leaves the conclusion unchanged.Source: §4.3 · pp. 4–5
4.4 Position bias
- Position-decided items shift a cell’s share by 0.145 on average and up to 0.424, but the mean own-cell contrast is nearly unchanged after their removal.Across four rows, the contrast is −0.026 raw versus −0.025 on content-decided items; position-decided shares are 0.42 on-diagonal and 0.41 off-diagonal.Source: Table 2 · p. 5, §4.4 · p. 5
4.5 What the design could have seen
- At γ = 0.05, power is 82% for products, 88% for papers, 42% for films, and 97% pooled; the pooled 80%-power minimum detectable effect is 0.034.In a future design, adding models buys more power than adding trials: eight models at 200 trials per cell give 86%, while 3,000 trials per cell in a five-model design give 85% for γ = 0.05.Source: §4.5 · pp. 5–6, Table 3 · p. 6
4.6 Tan et al. beside it
- Tan et al.’s 4×4 matrix has γ = +0.148 at the smallest attainable one-sided p = 0.042, but its same-family coefficient is nearly as large as its self coefficient.The matrices measure different things: Tan et al.’s own context also restates the reader’s parametric belief, unlike descriptions of the same item from the same facts.Source: §4.6 · p. 6
5 Related work
Prior work finds self-preference to be heterogeneous and often smaller under quality controls, while studies of family effects and retrieval preferences report results that vary by setting.
5 Related work
- Quality-controlled studies report smaller or less consistent self-preference, including effects reduced by style matching and mixed directions across judges.Other studies tie preference to familiarity or find no consistent self-recognition, while some report model-specific effects.Source: §5 · pp. 6–7
- Studies excluding self-cells report same-family lifts of 3.4 to 8.4 points and sibling lifts of 6.6 and 11.5 points, but these matrices cannot resolve effects of that size.The present design has one vendor pair and a negative same-vendor coefficient.Source: §5 · p. 7
- Retrieval studies report self-preference under style-based prompts but not fact-centric prompts, while web-content preference rules are mostly shared across engines.The reported shared-rule rate is 79% to 84%, with engine-specific increments of 1% to 8%.Source: §5 · p. 7
6 Limitations
The re-analysis is limited by its small, narrow design, incomplete position records, and substantial invalid responding in one film-selector condition.
6 Limitations
- Five models, one vendor pair, one prompt per role, and one model text per item constrain the re-analysis; GPT-3.5 on films returned no valid choice in 108 to 142 of 500 trials per cell.The same-vendor test has only two cells per matrix and cannot separate vendor from within-vendor size effects.Source: §6 · pp. 7–8
- Position is recorded for only 27 of 75 cells’ worth of runs, and the diagonal comparison rests on four rows.The same-vendor test has two cells per matrix.Source: §6 · pp. 7–8
- The analysis does not address explicit self-recognition, judging open-ended outputs, or models released after 2024.The finding applies to GPT-3.5-turbo, GPT-4-1106-preview, Llama 3.1 70B, Mixtral 8x22B, and Qwen2.5 72B.Source: §6 · p. 8
7 Conclusion
The reanalysis supports the shared preference for model-written text but finds no reliable own-model premium. Its released materials recompute the analysis from public counts, alongside disclosures about the author’s interests and AI-tool use.
7 Conclusion
- +0.019 share points is the pooled diagonal residue, with 17 of 120 relabellings doing at least as well; no dataset shows a reliably keenest own-model selector.The paper’s conclusion is that these data show shared taste for model-written text, not models favouring their own voice.Source: §7 · p. 8
- The package includes extraction, analysis, figure, and number-check scripts, plus per-item and per-cell counts derived from the authors’ repository.It contains no text from the source study.Source: §7 · p. 8
- The author runs a company that measures and sells AI-engine brand-description measurement, a business relevant to whether models share or specialize their text preferences.The author states that the data are the original authors’ and that the numbers recompute from their public counts.Source: §7 · p. 8
- An AI assistant was used for grammar and language editing and to draw figures from analysis output; no study data were labelled or generated with AI tools.The author reviewed the output and takes responsibility for it.Source: §7 · p. 8