PaperlayerView exact revision on arXiv

Source-linked AI summary

The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions

Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicol`o Pagan, Hannah Metzler, Silvia Giordano, Anikó Hannák, David Garcia

arXiv:2511.08592v1cs.CLcs.AIcs.SI

TL;DR

Whether LLMs can convincingly mimic multi-user social-media conversations remains insufficiently tested. The authors compare authentic Reddit discussions with conversations generated by GPT-4o and Llama 3 70B, finding that generated discussions were mistaken for human-made ones in 39% of cases. The findings point to potential for social simulation alongside risks of generated content blending into genuine discourse.

  • Problem

    Whether LLMs can mimic human behavior in multi-user social-media conversations remains insufficiently validated, despite the relevance of generative agents to social simulation.

    Source: Introduction · pp. 1–2

  • Method

    The study presents participants with authentic Reddit discussions alongside same-topic conversations generated by GPT-4o or Llama 3 70B and asks them to identify the human conversation.

    Source: Introduction · p. 2, Discussion · p. 4

  • Results

    39% of LLM-generated Reddit discussions were mistaken for human-authored ones, exceeding the stated Turing Test threshold; Llama 3 70B was harder to distinguish than GPT-4o.

    Source: Discussion · p. 5, Conclusion · p. 6

  • Takeaways & Limitations

    The findings suggest potential for LLM-powered agents to emulate human dialogue in social simulations, while raising concern that generated content could blend into genuine online discourse undetected.

    Source: Discussion · p. 5, Conclusion · p. 6

    LLMs struggled to reproduce emotionally charged discourse, personal storytelling, and polarization, which are central to many social-media phenomena.

    Source: Conclusion · p. 6

Abstract

from arXiv · show

Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39\% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56\% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content.

Introduction

Agent-based models can help study social phenomena, but oversimplified behavior, scarce data, and human variability limit their practical calibration. This study tests whether LLMs can mimic multi-user social-media conversations using authentic Reddit discussions.

Introduction

  • Generative agent-based models offer a controlled setting for studying social-media interventions, but empirical grounding is needed to improve their practical applicability.The motivation includes ethical concerns about large-scale platform experiments and restrictions on public data access.Source: Introduction · p. 1
  • LLMs have prompted growing interest in simulating social networks and testing feed algorithms, but validation of their social-media behavior remains an open need.Prior work includes frameworks for simulated social networks and analytical comparisons of human and LLM replies.Source: Introduction · pp. 1–2
  • Validation remains needed to establish whether generative agents represent social-media users, respond comparably to humans, and produce similar group behavior.The broader problem is that empirical grounding is difficult when data are scarce and human behavior varies.Source: Introduction · pp. 1–2
  • Prior GPT-3 testing used crowdworker-written comparison conversations rather than authentic Reddit threads, limiting the experiment’s external validity.The authors motivate using conversations created by active Reddit users as a more authentic control.Source: Introduction · p. 2
  • The study compares authentic Reddit conversations with same-topic conversations generated by GPT-4o and Llama 3 70B, shown side by side at varying lengths.Participants identify which conversation is human-authored; the study preregistered the experiment.Source: Introduction · p. 2

Results

Participants mistook LLM-generated discussions for human-made ones in 39% of cases, with Llama 3 70B harder to distinguish from human conversations than GPT-4o. Conversation length generally had no significant effect, while participants’ reported authorship cues most often concerned style.

Results

  • 39% of LLM-generated discussions were mistaken for human-made; overall participant success in identifying the human conversation was 0.61.The analysis included 1,205 annotations from 203 participants after excluding attention-check failures.Source: Results · p. 2
  • 0.56 vs 0.66 (P < 0.001): participants were less successful identifying the human conversation when the paired simulation was generated by Llama 3 70B rather than GPT-4o.Figure 1 presents predicted probabilities by model and fixed-effect estimates for numerical conversation length.Source: The impact of model and conversation… · pp. 2–3, Figure 1 · p. 3
  • Length generally had no significant effect on success; in the categorical analysis, only length = 8 was significantly higher than length = 1.The numerical-length model found no significant effect (p-value = 0.36); Figure 2 shows model-specific predicted probabilities and categorical-length effects.Source: The impact of model and conversation… · p. 3, Figure 2 · p. 4
  • The exploratory cue analysis classified participant comments into Format, Style, and Content, then examined responses from 58 high-performing participants.Two researchers independently annotated that subset and resolved discrepancies by consensus.Source: Differences between human and AI… · pp. 3–4
  • 82.5% of analyzed responses mentioned a textual-style cue; content cues appeared in 49.1%, while format cues appeared in 17.5%.Tone was the most frequently mentioned style subcategory (49.1% of total comments); response length was the most frequent format cue (14%).Source: Differences between human and AI… · p. 4

Discussion

Using real Reddit conversations and minimal prompting, the study found that participants could not identify the human conversation in 39% of cases. It also reports model differences, a nonlinear conversation-length pattern, and recurring cues that expose limitations in simulating emotionally charged or polarizing discourse.

Discussion

  • With minimal prompting on real Reddit data, participants failed to identify the human conversation in 39% of cases, exceeding the 30% Turing-test threshold.The authors describe this as evidence that basic prompting can yield conversations that convincingly imitate human interactions.Source: Discussion · pp. 4–5
  • Llama 3 70B conversations were significantly harder to distinguish from human ones than GPT-4o conversations, while raising concerns about artificial consensus and social polarization.The authors note these outputs could blend into genuine online discourse and could be exploited through coordinated AI swarms.Source: Discussion · p. 5
  • Detection accuracy followed an inverse-U pattern, peaking at eight comments before declining; natural-sounding variance or participant fatigue are proposed explanations.The authors state that both explanations warrant further investigation.Source: Discussion · p. 5
  • Participants flagged over-politeness, formality, missing slang, weak emotional nuance, and over-agreement; Tables 1–2 organize these authorship cues.The authors connect these cues to limits in simulating toxic or emotionally charged behavior, emotional or persuasive discourse, and disagreement or polarization.Source: Discussion · p. 5, Table 1 · p. 5, Table 2 · p. 6
  • Careful prompt engineering, model selection, and possibly persona-driven prompting are identified as ways to improve realism, especially for contentious or emotionally rich topics.The authors frame these findings as showing both potential and limitations for current LLMs in replicating online human behavior.Source: Discussion · p. 5

Conclusion

Across models and conversation lengths, participants mistook LLM-generated Reddit discussions for human-authored ones in 39% of cases. The paper recommends Llama 3 70B for naturalistic interaction simulations while emphasizing remaining limitations and directions for further evaluation.

Conclusion

  • Across models and conversation lengths, participants mistook LLM-generated Reddit conversations for human-authored ones in 39% of cases, surpassing the Turing Test threshold.The authors identify this as evidence that LLMs can convincingly simulate human discourse on social media and as a potential for generative agent-based modeling.Source: Conclusion · p. 6
  • The authors recommend Llama 3 70B for simulations aiming to replicate naturalistic human interaction because model selection significantly affects performance.The recommendation follows the finding that Llama 3 70B outperformed GPT-4o in producing more human-like conversations.Source: Conclusion · p. 6
  • LLMs still struggle to simulate emotionally charged discourse, personal storytelling, and polarization, limiting their fit for these dynamics.The authors identify these factors as central to many social media phenomena.Source: Conclusion · p. 6
  • Future evaluations could include additional proprietary and open-weight LLMs, other languages, and multiple persona-driven agents in conversation.The authors propose assessing whether persona-driven setups improve authenticity, personal storytelling, or emotionally intense or controversial discourse.Source: Conclusion · p. 6
  • Researchers should apply LLM-based social simulations critically, especially when studying sensitive phenomena such as polarization, misinformation, or emotional contagion.The paper presents empirical benchmarks, practical model comparisons, and methodological guidance for computational social scientists.Source: Conclusion · p. 6

Methods

The study built paired human and AI Reddit conversations from real discussions, then tested them with participants using a preregistered annotation design. Conversation structure and approximate comment length were held constant, and success was analyzed with multilevel logistic models.

Collecting and generating stimuli

  • Human comment structure and word counts were preserved while AI comments replaced the human text in Reddit discussions sampled from December 2023.The researchers selected 16 posts across eight topics and scraped the first 20 comments for each.Source: Collecting and generating data for… · p. 7
  • GPT-4o and Llama 3 70B generated comments at three temperatures, with each new instance given the post and comments generated so far.The prompt specified the corresponding human comment’s approximate word count and reply target; the same generated conversations were later shown at six lengths.Source: Collecting and generating data for… · p. 7

Experimental setting

  • The experiment used a Reddit-like interface; participants who passed consent and screening completed six annotations and an additional attention test.Sampling ensured that a participant did not see the same post more than once.Source: Experimental setting · p. 7
  • Participants saw side-by-side human and AI conversations matched on comment count, users, usernames, profile pictures, and approximate word count, and selected the human conversation.They also rated confidence, commented as if participating on Reddit, and later described cues they used to distinguish the conversations.Source: Experimental setting · p. 7

Preregistration deviations

  • Two preregistration deviations were reported: a typo in the first hypothesis and omission of two movie-topic conversations, leaving 504 stimuli presented rather than 576 planned.The study recruited 250 English-fluent participants to account for the pilot’s observed attention-test failure rate.Source: Experimental setting · p. 8

Statistical analysis

  • A multilevel logistic model predicted correct identification while accounting for repeated annotations with participant-level varying intercepts.The preregistered model included model and numerical conversation length; a second model treated length categorically to test nonlinear effects.Source: Statistical model · p. 8

Funding statement

The study received support from Baden-Württemberg, the Swiss National Science Foundation, and the European Union’s Horizon Europe programme.

Funding

  • Funding came from Baden-Württemberg’s Ministry of Science, Research and the Arts, the Swiss National Science Foundation’s CARISMA project, and the EU Horizon Europe WHAT-IF programme.The authors note that the views expressed are theirs and do not necessarily reflect those of the European Union.Source: Funding statement · p. 11

A Collective Turing Test of LLMs

The supplied passages describe prompt instructions for generating Reddit-style replies and an exploratory test of whether model and conversation length interacted. That interaction was not statistically significant.

Model prompt

  • The generation prompt instructed models to write original Reddit-style replies, avoid repetitive agreement or excessive punctuation, and approximate the source comment’s length.It also requested varied style, occasional typos, and rare emoji use.Source: A Collective Turing Test of LLMs · pp. 12–13
  • The prompt asked models to use paragraphing and occasional typos while restricting emoji use to rare instances.It allowed them to refer to existing comments or replies when present.Source: A Collective Turing Test of LLMs · p. 13

Interaction model with conversation length

  • The exploratory analysis found no statistically significant interaction effect between LLM model and conversation length on success rate.The study examined whether the relationship between model and identification success varied with conversation length.Source: A Collective Turing Test of LLMs · p. 18
Loading 2511.08592v1…