Human heuristics for AI-generated language are flawed

Introduction

• Large language models and AI writing applications can generate coherent text and conversations at scale, reducing effort while enabling plagiarism, manipulation, and deception when their output is mistaken for human writing (1, 2, 1, 3–8).Source: Introduction · p. 2
• The study operationalizes verbal self-presentation as online-profile self-descriptions, where impression formation helps establish trust in social interactions (9, 10, 14, 15).The introduction notes that AI-generated self-presentations may undermine signals such as tone or compositional skill (16), and suspicion of AI use can reduce interpersonal trust (17).Source: Introduction · p. 2
• The study reconstructs and tests heuristics people use to detect AI-generated language, then examines whether AI can predict and manipulate perceptions of humanness.It combines qualitative, quantitative, and computational methods.Source: Introduction · p. 2

Results

• Across three main experiments, participants identified self-presentation sources with only 50 to 52% accuracy, even with incentives (51.6%) or feedback (51.2%).No demographic group performed better than others; the experiment breakdown is shown in Fig. 1.Source: Results · pp. 2–3
• Judgments were consistent beyond chance (Fleiss’ kappa = 0.07, P < 0.0001), and self-reports commonly cited content (40%), grammar (28%), and tone (24%) cues.Family and life experiences were often taken as evidence of human authorship; participants also associated first-person pronouns and grammatical mastery with humans, and warm, genuine tone with humanity.Source: Results · p. 3
• A classifier using crowdworker nonsensical or repetitive labels reached 58.8% source accuracy, versus 51.7% for participants directly judging source in the main experiment.Crowdworkers rated AI-generated profiles as nonsensical more often (13.6% vs. 9.6%) and repetitive more often (12.7% vs. 7.1%), but as having grammatical issues less often (14.8% vs. 19.6%).Source: Results · pp. 3–4
• Grammatical issues made profiles 5% more likely to be judged AI, despite being 15% less likely to be AI; first-person and family cues were not diagnostic.Nonsensical and repetitive content were valid cues, but participants also relied on cues such as long words and rare bigrams that were commonly found in human-written profiles.Source: Results · p. 4
• Feature-based regression predicted participants’ judgments at 57.6% accuracy, while a sequence-classification language model reached 58.1% on hold-out data.These results indicate that the flawed heuristics people used made their impressions somewhat predictable to AI systems.Source: Results · p. 4
• Across validation experiments, optimized AI profiles were judged human more often than regular AI profiles (65.7% vs. 51.6%) and human-written profiles (65.7% vs. 51.7%).The increase was strongest in the professional context, where optimized profiles were perceived as human 71% of the time.Source: Results · p. 5

Discussion

• Across contexts and demographics, people’s judgments of AI-generated self-presentations stayed near chance, even though a separate task found detectable nonsensical and repetitive features.The authors argue that AI language at the time of the study was not entirely indistinguishable from human writing; the difficulty was identifying it when directly asked whether it was AI-generated.Source: Discussion · p. 5
• 58.8% detection accuracy was achievable using functional cues alone, but misleading cues—including grammatical issues, rare bigrams, and long words—offset them and left judgments near chance.Features such as family words and first-person pronouns were equally present in human- and AI-written self-presentations, yet people associated them with human writing.Source: Discussion · p. 5
• Family-topic emphasis can make AI text seem more human without increasing machine intelligence; the authors frame failed detection as human vulnerability, not evidence of intelligence.They note that language-model decoding can introduce statistical anomalies detectable by machines [35], and that domain expertise may outperform personal intuition somewhat [23].Source: Discussion · p. 6
• In three validation experiments, AI systems used people’s flawed heuristics to manipulate judgments and produce language perceived as “more human than human”; malevolent actors could exploit this for automated impersonation (8), targeted disinformation (37, 38), fraud, and identity theft (3–8).People were more likely to disclose private information to and follow recommendations from nonhuman entities they perceived as human (36), and may distrust communicators they believe use AI-generated language (17).Source: Discussion · p. 6
• Education and detection tools may help only somewhat, while changing models can invalidate learned heuristics and disclosure remains difficult in AI-mediated communication [18, 35, 16].Policy proposals call for notice and explanation [42, 43], but applying disclosure to tools such as autocorrection, smart replies, or translation is difficult, and users often ignore notices [44].Source: Discussion · p. 6
• Self-disclosing AI could preserve intuitive judgment by serving communicative functions without imitating human language, including through dedicated dialects or accents.The authors suggest avoiding informal, colloquial cues wrongly associated with humanity; a dedicated AI accent could signal origin without interrupting communication or undermining human cognition.Source: Discussion · pp. 6–7

Experiment design

• Participants rated human- and AI-generated self-presentations in six experiments, and responses were grouped into likely-human versus suspicious judgments.Participants evaluated 16 presentations in the main hospitality experiment and 12 in the dating and professional experiments; using the full rating scale produced similar results. Bonus incentives and feedback did not improve performance.Source: Materials and Methods · p. 7

Collecting and generating self-presentations

• The researchers sampled real platform self-presentations and fine-tuned context-specific language models to generate counterparts, finding no substantial plagiarism.The corpora included 28,890 Airbnb host profiles, 37,450 Guru profiles, and 59,940 OkCupid essays; 95% of generated-text sentences were absent from training data.Source: Materials and Methods · pp. 7–8

Predicting responses

• Researchers built about 180 language features and crowdworker labels, then compared a lasso-selected logistic regression with a GPT-2 sequence classifier on held-out presentations.Feature selection used 20% of presentations, model fitting used 70%, and evaluation used the remaining 10%.Source: Materials and Methods · p. 8

Generating language optimized for perceived humanity

• For validation, context-specific classifiers selected the top 20% of AI-generated presentations predicted to seem human-written.Each validation experiment compared separate samples of 100 human-written, 100 AI-generated, and 100 optimized presentations; the study used regression, language-model, or ensemble classifiers across contexts.Source: Materials and Methods · pp. 8–9

Participant recruitment

• The main experiments recruited 2,000 hospitality participants and 1,000 each for dating and professional contexts, alongside separate crowdworkers and validation samples.The study recruited 1,300 crowdworkers to label 7,000 presentations and 200 participants for each validation experiment.Source: Materials and Methods · p. 9

Limitations and ethics statement

• The authors caution that findings reflect current language models and human heuristics, while disclosure of the vulnerability raises ethical risks and prevention needs.They note that technological and cultural change may alter both AI language and people’s heuristics.Source: Materials and Methods · p. 9

Data sharing

• Data and code for the three main studies and three validation experiments are publicly available through an Open Science Foundation repository; the work used previously published data (47).Repository: https://osf.io/284yv/.Source: Data sharing · p. 10

Figures and Tables

• Participants’ ability to detect AI-generated self-presentations remained close to chance across three social contexts, even with incentives or feedback.Figure 1 summarizes 6,000 to 16,000 judgments per bar across 2,000 to 3,000 self-presentations per bar.Source: Figure 1 · p. 13
• Optimized AI-generated self-presentations were more likely to be perceived as human than human-written profiles or regular AI-generated profiles.The comparison is shown across three validation experiments.Source: Figure 2 · p. 13
• Only nonsense, repetition, and conversational words aligned as functional cues for perceived AI authorship and actual AI authorship; other heuristic-associated features were inversely related or unrelated to actual source.Table 1 compares logistic regression models predicting perceived AI generation and actual AI generation.Source: Table 1 · p. 14

Supporting Information

• Across contexts, groups, and treatments, participants could not detect AI-generated self-presentations; participants over 50 achieved 53% accuracy versus 51% among younger participants.Table S3 reports the auxiliary group analysis.Source: Supporting Information · p. 15
• No gender or ethnic group performed better; university-degree holders performed about 1% worse, technical knowledge was uncorrelated with accuracy, and judgment time or profile length did not predict higher accuracy.The analysis found no subgroup that reliably detected AI-generated self-presentations.Source: Supporting Information · p. 15
• Participants most often cited content (40%), grammar (28%), and tone (24%), associating family experiences, first-person pronouns, and warm language with human writing and generic or impersonal text with AI.Some participants associated grammatical errors with either weaker AI or fallible human authors.Source: Supporting Information · p. 15
• Participants’ accuracy did not change after explaining judgments in any of the three contexts, and validation experiments without explanations showed similar outcomes.Open-ended explanations were solicited only in the three main experiments.Source: Supporting Information · p. 15
• Crowdworkers labeled generated self-presentations as repetitive more often than human-written text (12.7% vs. 7.1%%, p<0.0001), particularly in the professional context.Source: Supporting Information · p. 16
• Crowdworkers labeled AI-generated text as having grammatical issues less often than human-written text (14.8% vs. 19.6%, p<0.0001), especially in dating and professional profiles.Those contexts used GPT-3 to generate self-presentations.Source: Supporting Information · p. 16

• Supplementary materials overview hospitality, dating, and professional studies and provide human- and AI-generated example self-presentations.Source: Table S1 · p. 16, SI References · p. 16, Table S2 · p. 17
• 0.986** (0.976, 0.996) for education and 1.006 (0.982, 1.030) for technical knowledge were reported as accuracy-prediction coefficients; no group performed much above chance.Source: Table S3 · p. 18, SI References · p. 19
• Participants’ performance in identifying generated self-presentations did not change throughout the experiment.Error bars represent 95% confidence intervals across 6,000–16,000 judgments per bar.Source: Figure S1 · p. 32
• In a separate labeling task, participants rated AI-generated self-presentations as nonsensical and repetitive more often than human-written ones.The figure summarizes 1,898–4,704 judgments per bar.Source: Figure S2 · p. 32