Empirical evidence of Large Language Model's influence on human spoken communication

Comparing Word Preferences of Humans and LLMs

• ChatGPT’s lexical preferences were measured with per-word GPT scores from log-odds differences between human-written and GPT-edited texts, identifying delve and related preferred words.Delve was favored over explore or examine and ranked near the top across early GPT models.Source: Comparing Word Preferences of Humans… · p. 3, Figure 1 · p. 4
• A synthetic-control design matched each treated word’s pre-release frequency trajectory using untreated donor words, estimating post-release changes in podcast usage.The podcast corpus was filtered for conversational, unscripted speech using an audio classifier with approximately 90% held-out accuracy.Source: Comparing Word Preferences of Humans… · p. 3

Relationship between ChatGPT’s word preferences and human adoption

• ∼44% above the synthetic-control counterfactual, delve usage rose in Science & Technology podcasts during months 13–18 after ChatGPT’s release.The 95% CI was [+22%, +63%], with in-space placebo p = 0.010; removing high-usage channels left the rise essentially unchanged.Source: Relationship between ChatGPT’s word… · pp. 5–6
• Education and Business podcasts showed mean elevations of 32% and 31%, whereas the category-independent sample rose 9% and Sports declined 7% over months 13–18.The elevated period coincided with GPT-3.5’s dominance in the consumer market.Source: Figure 2 · p. 5, Relationship between ChatGPT’s word… · p. 6
• The lexical elevation later reverted: delve fell below baseline in several categories, while the change-point analysis rose only when the candidate window reached ChatGPT’s actual launch.The later decline coincided with GPT-4o’s free release and growing awareness of delve as an LLM-associated word.Source: Figure 2 · p. 5, Relationship between ChatGPT’s word… · p. 6
• Among 3,535 words, the top 1% of GPT-score words had mean post-release slope βPost +0.030, with 28 of 36 increasing and 13 showing credible increases.The association was strongest at the highest scores and remained after excluding delve; words including boast, meticulous, and showcase maintained sustained uptake.Source: Relationship between ChatGPT’s word… · p. 6, Figure 3 · p. 7

A brief chatbot interaction entrenches lexical choices

• During interaction, participants used the AI-introduced variant at 61% versus 17% when the AI used the alternative, isolating an effect of randomly assigned exposure.The association was statistically significant: b = 0.44, 95% CI [0.42, 0.46], p < .001.Source: A brief chatbot interaction entrenches… · p. 8
• After a 3-minute distractor, participants used AI-introduced variants in descriptions of novel images at 58% versus 23% for alternatives.The effect survived distraction and generalized beyond the images seen during interaction.Source: A brief chatbot interaction entrenches… · p. 8
• Participants selected AI-introduced variants in 63% of forced-choice trials, and uptake held across nouns, verbs, and adjectives.This provided converging evidence for an explicit lexical preference shift.Source: A brief chatbot interaction entrenches… · pp. 8–9
• Only 15 of 496 participants noticed the chatbot’s vocabulary pattern, suggesting that the lexical shift was implicit rather than deliberate.The remaining 481 participants reported nothing unusual or only incidental features.Source: A brief chatbot interaction entrenches… · p. 9

Population-level influence

• Across 737,083 podcast hours, ChatGPT-preferred words rose in unscripted speech after release, with strongest early signals in academic-adjacent domains and indications of diffusion elsewhere.The association was observed beyond scripted or formal speech and was linked causally to ChatGPT’s public release.Source: Discussion · p. 9

Individual-level mechanism

• A brief AI interaction increased AI-introduced words in later spontaneous speech by 36 percentage points, with choices persisting after distraction and selected in 63% of forced-choice trials.The pattern supports lexical entrenchment rather than transient conversational alignment.Source: Discussion · p. 9

Cultural evolution

• The corpus and experiment together position LLMs as cultural models whose variants humans internalize, integrating machine and human cultural production into a coupled process.Because many users interact with relatively few generative systems, hub-like AI providers may concentrate exposure and amplify propagation beyond directly exposed users.Source: Discussion · p. 10

Specificity of lexical shifts

• The identified words lack obvious referential ties to language models and occupy a frequency range where ordinary lexical change is slow, distinguishing the shift from topical or technology-specific vocabulary.The corpus also showed expected surges for pandemic, technical, and LLM-activity terms, but the focal words fit none of those categories.Source: Discussion · p. 10

Dynamic social meaning

• Adoption was not monotone: some words later moderated or fell below baseline as AI associations became culturally legible, while later GPT versions removed delve during editing operations.The authors connect this pattern to avoidance, authenticity signaling, and possible effects on perceived credibility and competence.Source: Discussion · p. 10

Limitations

• The evidence is limited to English-speaking, self-selected public podcast participants and online native-English participants, with the main analysis covering only 18 months after ChatGPT’s release.Fragmentation across models and providers complicates attribution, and the single experimental session captures only an early entrenchment signal.Source: Discussion · pp. 10–11

Methods

• ChatGPT-preferred words were analyzed with GPT scores and per-word synthetic-control and Bayesian change-point models, with in-word-space and in-time placebo procedures testing specificity.The GPT score quantifies each word’s affinity for ChatGPT output, while post-release shifts combine synthetic-control matching with Bayesian change-point regression.Source: Methods · p. 11, Placebo test · p. 17, Change-point regression of the… · p. 18

Data collection

• 824,634 podcast episodes passed a two-stage spontaneity filter requiring multi-speaker dialogue and an audio-classifier score above 3.0.The corpus began with 1,407,131 candidate episodes; 931,450 passed the dialogue screen and 824,634 met the stricter criterion.Source: Data collection · p. 12, Filtering and transcription · pp. 12–13

Filtering and transcription

• Spontaneity classification used diarization plus Whisper-large-v3 audio features trained on human annotations, achieving ICC = 0.87 (95% CI 0.81–0.91).Episodes were retained only when they had at least two speakers, four or more exchanges, and an episode-level spontaneity score exceeding 3.0.Source: Filtering and transcription · pp. 12–13

Log-odds ratio estimation

• GPT scores compare word document frequencies in human and LLM-edited texts across datasets, models, and prompts, using Laplace smoothing and a document-level Bernoulli definition.The main reference score uses pre-GPT-4o production models and a Bayesian hierarchical weighting scheme with Dirichlet priors, summarized as the median log-odds ratio across 1,000 Monte Carlo samples.Source: Log-odds ratio estimation · p. 15, Calculation of a weighted GPT score · pp. 15–16

Estimating causal influence of ChatGPT

• Synthetic controls estimate each GPT-preferred word’s counterfactual trajectory from 100 matched donor words after excluding semantic neighbors and words with non-neutral GPT scores.Donors are selected from the 50,000 most frequent word2vec vocabulary items using smoothed pre-treatment trajectory distance, then combined with non-negative weights summing to one; the design relies on no anticipation, stable pre-release fit, and no interference [89].Source: Estimating causal influence of ChatGPT · p. 16, Donor selection · p. 17

Change-point regression of the synthetic-control gap

• Post-release effects are estimated as changes in synthetic-control gaps at the 30 November 2022 change point, with placebo-based confidence intervals and permutation nulls quantifying uncertainty.The analysis fits hierarchical Bayesian change-point regressions to words averaging at least 20 pre-treatment episodes per month, reports 95% HDIs, and evaluates nested GPT-score slices with 1,000 permutations.Source: Change-point regression of the… · p. 18, Placebo test · p. 18

Controlled referential communication experiment

• The preregistered study recruited 500 Prolific participants and retained a final N = 496 after excluding four participants for color vision deficiency.The sample had a mean age of 40.5 years, 50% women, and 91% reporting English as their first language; recruitment exceeded the power-analysis target of Nrequired = 352.Source: Controlled referential communication… · p. 19
• The between-subjects design crossed two stimulus groups and two AI-used synonym modes across 12 within-subject trials.The experiment used 18 synonym pairs spanning nouns, verbs, and adjectives; each group contained nine pairs, with the chatbot consistently using one synonym from each pair.Source: Controlled referential communication… · p. 19
• Participants interacted with a GPT-4o-based chatbot in a 12-trial picture-selection game, recorded spoken descriptions, completed a 3-minute reasoning distractor, and then performed delayed speech and forced-choice tests.The procedure also included a language-awareness check, background questionnaire, and browser-based speech transcription with manual correction during the Test Phase.Source: Controlled referential communication… · pp. 19–20
• The primary outcome, ∆p, measured each participant’s difference in AI-variant usage between introduced and non-introduced synonym pairs.Inference used a two-tailed 10,000-permutation Monte Carlo sign-flip test, 95% cluster-bootstrap confidence intervals with 10,000 replications, and a participant- and word-pair-level mixed-model cross-check.Source: Controlled referential communication… · p. 20

Appendix A Word preferences of Large Language Models

• GPT-family word-preference patterns remain notably stable across model versions, suggesting a shared training-pipeline origin, although delve weakens sharply in newer models.Delve’s odds ratio declines from approximately 380:1 with GPT-3.5-turbo to 100:1 with GPT-4-turbo and 40:1 with GPT-4o, reaching approximately 0.7:1 with GPT-5 on arXiv abstracts.Source: Appendix A · p. 20
• Rephrasing instructions produce closely similar LORs for every reference word, with clarify consistently milder but never reversing a preference.The compared prompts are polish, improve, and clarify.Source: Appendix A · p. 20, Figure 5 · p. 21
• LOR magnitudes vary substantially across source corpora, primarily because baseline human word choices differ between datasets.For example, GPT revisions frequently introduce underscore even in essays, where humans rarely use it.Source: Appendix A · pp. 20–21
• In scientific abstracts, delve is the most strongly overused term, followed by underscore, comprehend, bolster, boast, inquiry, swift, meticulous, pinpoint, and groundbreak.The weighted GPT score aggregates across models, prompts, and datasets; delve has LOR > 4, while the following terms have LOR > 2.5.Source: Appendix A · p. 21

Appendix B Population-level replication in academic YouTube talks

• The replication mirrors the podcast pipeline for transcription, synthetic control, and change-point analyses, but omits diarization and spontaneity classification because academic talks are predominantly prepared monologues.Videos are restricted to durations between 20 minutes and approximately 3.0 hours.Source: Appendix B · p. 21
• The academic YouTube replication reproduces the delve effect, with a placebo test yielding p = 0.010 after the ChatGPT release.The analysis covers 360,445 talks from channels associated with 20,622 research institutions.Source: Appendix B · p. 21, Figure 6 · p. 22
• Comprehend, boast, swift, and meticulous also show significant post-release uptake in piecewise-linear fits centered on ChatGPT’s release.These words extend the replication beyond the single-word delve result.Source: Appendix B · p. 22, Figure 6 · p. 22

This PDF file includes:

• The supplementary materials include figures S1–S17 and tables S1–S7.These materials accompany the appendix and main analyses.Source: This PDF file includes · p. 30

Datasets to compute Log Odds Ratios of human and LLM word usage

• The LOR analysis draws on arXiv, bioRxiv, Nature, Enron emails, student essays, Wikipedia, and pre-ChatGPT podcast excerpts.The datasets span academic, historical, web, and spontaneous spoken language sources, with sampling sizes specified for each corpus.Source: Datasets to compute Log Odds Ratios of… · p. 31
• Podcast excerpts are sampled from pre-release English episodes that pass the study’s spontaneity threshold, with one 300-word excerpt taken from each of 2,000 episodes.The sampling supports computation of word preferences before testing post-release spoken-language changes.Source: Datasets to compute Log Odds Ratios of… · p. 31

Spontaneity annotation and classification

• Spontaneity is operationalized through disfluencies, with three blinded coders rating three 30-second clips per episode on a four-point scripted-to-spontaneous scale.A threshold of five disfluencies in a 30-second sample separates scripted from spontaneous codes.Source: Spontaneity annotation and… · p. 32, Annotation protocol · p. 32

Audio-based spontaneity classifier

• An audio classifier trained on 399 annotated episodes uses frozen Whisper embeddings and a multilayer perceptron to predict four spontaneity classes from 25 contiguous middle-recording chunks.Source: Audio-based spontaneity classifier · pp. 32–33
• Episodes are retained when they pass the dialogue filter and have a weighted spontaneity score above 3.0, a threshold associated with approximately 90% held-out accuracy.A binary-vote alternative produces a near-identical retained set.Source: Audio-based spontaneity classifier · p. 33

Word-sense audit

• The word-sense audit removes proper nouns, brands, artifacts, and unused dictionary senses symmetrically in human and GPT usage before estimation.Senses are retained only when both corpora attest them at least twice, and the resulting audited counts feed the synthetic-control and change-point analyses.Source: Word-sense audit · p. 34

Trends not linked to ChatGPT

• Pandemic terms such as pandemic, vaccine, corona, and virus show steep declines because the COVID-19 news cycle waned, not because of LLM properties.Other declines reflect platform rebranding or changing topic composition in the talks.Source: Trends not linked to ChatGPT · p. 35
• Rising words include both GPT signatures such as delve and ordinary technical terms such as gpu, python, vector, queries, and transcript.A word can shift because of changing subject matter or field growth even when its GPT score is near the distribution’s middle.Source: Trends not linked to ChatGPT · p. 35
• Because change-point detection alone cannot identify causes, the main design first restricts analysis to words GPT systematically over- or under-uses before testing post-release shifts.This separates the LLM-linked lexical signal from pandemic, product-launch, and research-topic trends.Source: Trends not linked to ChatGPT · p. 35

GP smoothing of the input series

• Pre-treatment Gaussian-process smoothing is used only for donor selection and synthetic-control fitting, while downstream change-point and placebo statistics use the raw monthly series.The pre-treatment-only fit prevents post-release dynamics from leaking into estimation.Source: GP smoothing of the input series · p. 35

In-time placebo: Change-point date sweep

• A change-point date sweep evaluates candidate release months using 24-month pre-windows and 18-month post-windows, with an OLS proxy closely matching the Stan estimator.The proxy achieves Pearson correlation 0.98 across the top-1% GPT-score slice.Source: In-time placebo: Change-point date sweep · pp. 35–36
• Across 28 pre-GPT candidate dates, the in-time placebo test yields permutation p = 0.034.The null consists of candidate change points whose full 18-month post-window predates ChatGPT’s release.Source: In-time placebo: Change-point date sweep · p. 36

• Median |βPre| was 0.001 versus median |βPost| of 0.015 across 3,535 Science & Technology words, indicating minimal pre-treatment trends relative to post-release changes.Source: Synthetic-control pre-trends check · p. 36
• Pre-treatment RMSPE medians were 0.008 for smoothed series and 0.038 for raw monthly data, showing close reproduction of treated words’ trajectories.Source: Synthetic-control pre-trends check · p. 36
• The analysis ranks words using the 95% HDI bound nearest zero, yielding a conservative lower bound on the slope change attributable to ChatGPT’s release.Source: Synthetic-control pre-trends check · p. 36

• Four control variants alter donor screening, donor-pool size, weighting, or count auditing to test whether the headline survives alternative synthetic-control specifications.C1 removes pre-screening, C2 uses ten donors, C3 uses inverse-distance weights, and C4 uses un-audited counts.Source: Robustness checks · pp. 36–37
• C2’s ten-donor design floors the empirical placebo p-value at 1/(10 + 1) ≈0.091 because placebo targets come from the same donor pool.Source: Robustness checks · p. 36
• The Main, C1, C2, C3, and C4 specifications are rendered side-by-side for the delve synthetic-control panel and the score–effect relationship.Source: Robustness checks · p. 37

Robustness to channel and episode outliers

• The largest channel contributed 0.46% of episodes and the ten largest contributed 1.4%, so no single channel dominates the 930,812-episode dataset.The analysis spans 38,294 distinct shows, with a median of five episodes per channel.Source: Robustness to channel and episode… · p. 37
• Excluding the ten channels using delve most leaves its post-adoption increase at the same 1.5-fold factor, indicating broad adoption rather than concentration in a few outliers.The comparison covers months 13–18 after release and uses token-normalized rates.Source: Robustness to channel and episode… · p. 37
• After excluding delve, the score–effect relationship remains essentially unchanged, with top-1% and top-2% GPT-score slices above the permutation null at p = 0.001.Source: Robustness to channel and episode… · p. 37

Participant demographics

• The final sample contained 496 participants, including 91% whose first language was English and 96% who reported prior AI-chatbot use.Source: Participant demographics · p. 37

Power analysis

• An a priori power analysis targeted d = 0.30 with α = .05 and power = .80, requiring 88 participants per cell and 352 overall.Recruitment exceeded the target, yielding a final sample of N = 496 after exclusions.Source: Power analysis · p. 37

Synonym pair selection

• The experiment used 18 familiar synonym pairs with no strong register differences, spanning nouns, verbs, and adjectives and suitable for cartoon-based referential guessing.The pairs were divided into two groups of nine, with accepted surface forms specified for scoring.Source: Synonym pair selection · p. 38

System implementation

• The experiment used a custom web application to generate matched image stimuli, constrain chatbot vocabulary, capture speech, verify transcripts, and score synonym use.The chatbot used gpt-4o with base, per-trial, and global vocabulary rules; human-verified transcripts supported the primary outcome.Source: System implementation · pp. 38–39

Participant flow details

• Participants completed chatbot interaction, distraction, chatbot-free speech tests, forced-choice synonym selections, awareness checks, and questionnaires in a preregistered sequence.The detection check preceded forced choice so responses reflected interaction impressions rather than explicit synonym-choice framing.Source: Participant flow details · p. 39

Theoretical model

• The noisy-voter model represented synonym preferences on a Watts–Strogatz network and compared a committed hub with a randomly placed committed speaker.Simulations used 2,000 agents, 50 independent runs, and hub exposure fractions of 0.5%, 2%, 5%, and 10%.Source: Theoretical model · p. 39

Supplementary Figures

• The delve post-release gap remained under four robustness specifications, with placebo p-values from 0.010 to 0.091 compared with 0.010 for the main specification.The checks varied donor selection, weight aggregation, and the counts substrate.Source: Figure S1 · p. 40
• Across top-1% GPT-score words, 28 of 36 showed positive slopes and 13 were credibly positive, while high-score acceleration persisted after removing delve.The top-1% mean βPost was 0.030, versus 0.027 excluding delve, with permutation p = 0.001 for both.Source: Figure S2 · p. 41, Figure S7 · p. 46
• The launch-date placebo placed the cross-word effect inside the null for pre-GPT candidate dates and above it only when the window included ChatGPT’s release.The true launch date was 2022-11-30, with permutation p = 0.034.Source: Figure S5 · p. 44, Supplementary Figures · p. 44
• The supplementary figures also document spontaneity annotation, smoothing selection, word-level shifts, synonym priming, forced-choice outcomes, and a population-diffusion simulation.These materials specify the measurement interfaces, visualization conventions, and model conditions used across the supplementary analyses.Source: Figure S8 · p. 47, Figure S9 · p. 48, Figure S10 · p. 49

Supplementary Tables

• Table S1 records the GPT-family models and exact OpenAI API snapshots used to compute GPT scores.Source: Table S1 · p. 58
• The chatbot’s base prompt established the image-guessing task, while an appended vocabulary-rule layer constrained designated synonym variants during intervention trials.Source: Figure S19 · p. 59, Figure S20 · p. 60
• Synthetic-control tables report pre-treatment fit metrics, donor-word weights for delve, category-specific gaps, and the design features shared across the main and robustness specifications.The delve donor table states that the top five donors account for 82.4% of total weight.Source: Table S2 · p. 61, Table S3 · p. 62, Table S4 · p. 62
• The supplementary materials report a full Science & Technology fit with smoothed RMSPE median 0.0076 and raw RMSPE median 0.0382.Source: Supplementary Tables · p. 61
• The final experimental sample comprised 496 participants, and Table S7 lists all 18 synonym pairs organized by group and lexical category.Source: Table S6 · p. 64, Table S7 · p. 65