Source-linked AI summary
People cannot distinguish GPT-4 from a human in a Turing test
Cameron R. Jones, Benjamin K. Bergen
TL;DR
The paper asks whether people can detect current AI in online interactions, then tests GPT-4 in a randomized, controlled two-player Turing test. GPT-4 was judged human in 54% of games, and the authors conclude that it passes this version of the test.
Problem
The paper asks whether people communicating online can tell when they are interacting with a machine, as current AI systems are increasingly deployed in online interactions.
Source: §1.1 · pp. 1–2Method
The authors ran a randomized, controlled two-player Turing test with GPT-4 and included ELIZA as a baseline and manipulation check.
Source: §1.1 · p. 2, §1.2 · p. 3Results
GPT-4 had a 54% pass rate, compared with 22% for ELIZA and 67% for human participants; its pass rate was not significantly different from 50%.
Source: §2.1 · p. 3Takeaways & Limitations
The results suggest current AI systems may deceive people, and likely set a lower bound on deception in less vigilant, more naturalistic contexts.
Source: §3.3 · p. 7The links between questioning strategies and accuracy are correlational; the study does not establish whether instructing interrogators to use particular techniques improves accuracy.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
We evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test. Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human. GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%). The results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test. The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected. Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.
1 Introduction
The introduction frames the Turing test as a way to ask whether people can distinguish machines from humans, with implications for detecting AI in online interactions. The study evaluates GPT-4, GPT-3.5, and ELIZA in a randomized, two-player test.
1.1 The Turing test
- The study uses a randomized, controlled two-player Turing test to examine whether people can detect current AI in human-machine conversations.The test also provides a window onto assumptions about which human qualities are hardest for machines to imitate.Source: §1.1 · p. 2
1.2 AI models
- GPT-4 and GPT-3.5 used prompted language models, while ELIZA served as a baseline and manipulation check for interrogator credulity.The adapted prompt gave the language models a concise, informal young-person persona, with slang and occasional spelling errors.Source: §1.2 · pp. 2–3
1.3 Game setup
- In the two-player setup, 500 participants were randomly assigned as witnesses or interrogators and played one five-minute round before giving a verdict.Interrogators were assigned a human or one of four witness types: GPT-4, GPT-3.5, ELIZA, or human.Source: §1.3 · p. 3
2 Results
GPT-4 was judged human in 54% of games, above ELIZA and below human witnesses. The results also describe which questioning strategies and decision cues were associated with interrogator accuracy.
2.1 Pass rates
- GPT-4’s pass rate was not significantly different from 50%, while humans exceeded GPT-4 and GPT-4 exceeded ELIZA; GPT-3.5 did not differ from GPT-4.ELIZA’s pass rate was significantly below chance, serving as a manipulation check for the design.Source: §2.1 · pp. 3–4
- GPT-4 had a 54% pass rate, versus 50% for GPT-3.5, 22% for ELIZA, and 67% for human participants.Pass rate is the proportion of games in which a witness type was judged human.Source: §2.1 · p. 3, Figure 2 · p. 4
2.2 Strategies & reasons
- Small talk was the most common strategy (36%); accuracy was higher for logic and maths (78%), human experiences (75%), and AI accusations (80%).The accusation association may reflect accusations preceding correct verdicts rather than the strategy improving accuracy.Source: §2.2 · p. 4
- Linguistic style was cited most often (43%), followed by socio-emotional factors (24%) and knowledge and reasoning (10%).AI verdicts for GPT-4, GPT-3.5, and humans often cited a forced persona, excessive informality, or lack of personality.Source: §2.2 · pp. 4–5
3 Discussion
The discussion concludes that GPT-4 passes this version of the Turing test, while cautioning that the test does not settle whether a system is intelligent. It also highlights the possibility of undetected AI deception and the importance of social and linguistic cues in judgments.
3.1 Does GPT-4 pass the Turing test?
- The authors favor a 50% chance baseline over Turing’s suggested 30% pass-rate target, while noting that a chance-level result could reflect random guessing or low power.They argue the ELIZA manipulation check helps address this concern.Source: §3.1 · p. 5
- GPT-4 passed under the preregistered criterion: interrogators were not better than chance at identifying it, while the ELIZA baseline showed the design could detect below-chance performance.The authors use ELIZA as a manipulation check and report that judgments that GPT-4 was human had mean confidence of 73%.Source: §3.1 · p. 5
3.2 What does the Turing test measure?
- The Turing test may complement static, narrow benchmarks because it is interactive, adversarial, and potentially broad, but it provides at most probabilistic evidence of intelligence.The authors note that it is unlikely to provide either necessary or sufficient evidence for intelligence.Source: §3.2 · p. 6
- Participants’ judgments focused more on linguistic style and socio-emotional factors than on traditional intelligence-related cues such as knowledge and reasoning.The authors suggest this could reflect an assumption that social intelligence is especially difficult for machines to imitate.Source: §3.2 · p. 6, Figure 4 · p. 6
3.3 Robots in disguise
- After five-minute conversations, participants were no better than chance at identifying GPT-4, and the authors suggest this may understate deception in more natural settings.Outside the experiment, people may be less alert to deception or less focused on detecting it.Source: §3.3 · p. 7
- Robust human impersonation could affect client-facing work, mislead the public or operators, and erode trust in authentic human interactions.Source: §3.3 · p. 7
- Questions about logic, current events, and human experiences were associated with more accurate judgments, but the evidence is correlational and does not establish their causal impact.The authors also found no evidence that LLM familiarity improved performance; age had a negative association with accuracy.Source: §3.3 · p. 7
4 Methods
The methods used prompted GPT-4 and GPT-3.5 witnesses, an ELIZA baseline, and a five-minute randomized game with human or AI witnesses. The preregistered analyses tested detection and witness-type comparisons, alongside exploratory demographic analyses.
4.1 AI witnesses
- GPT-4 and GPT-3.5 were instructed by prompts to respond as witnesses; the prompt used a young, casual persona, and the study appended sampled names, locations, languages, and timestamps.The models used the versioned strings gpt-4-0613 and gpt-3.5-turbo-0125 at the time of the experiment.Source: §4.1 · pp. 7–8
4.2 Interface
- The messaging interface limited messages to 300 characters, used a typing animation, and allowed verdicts within a five-minute game, with confidence and reasoning recorded.The OpenAI moderation API blocked abusive messages, and no reports were made during the experiment.Source: §4.2 · p. 8
4.3 Participants
- After exclusions, the study included 100 games each for GPT-4 and ELIZA and 101 each for human and GPT-3.5 witnesses, with 402 interrogators overall.Participants were recruited through Prolific; their mean age was 35 years (σ = 11.2).Source: §4.3 · p. 8
4.4 Procedure
- Participants were randomly assigned as witnesses or interrogators, with AI-game delays matched to recent human-witness wait times to reduce wait time as an identity cue.Games lasted up to five minutes, after which participants completed demographic surveys and were told the true witness identity and verdict.Source: §4.4 · pp. 8–9
4.5 Strategy and Reason Tagging
- Two evaluators applied multi-label strategy and reason tags, with overlap on 100 games; agreement was 0.67 for strategies and 0.49 for reasons.Source: §4.5 · p. 9
4.6 Statistical Analysis
- GPT-4 was indistinguishable from chance (z = 0.799, p = 0.424), exceeded ELIZA (z = 4.11, p < 0.01), but not GPT-3.5 (z = −0.497, p = 0.619); humans outperformed GPT-4 (z = 2.417, p = 0.016).Source: §4.6 · p. 9
- Self-reported LLM familiarity did not predict accuracy (z = −0.17, p = 0.869); exploratory analyses found a negative age effect (z = −2.240, p = 0.025).The Bayesian GPT-4 pass-rate intercept estimate was 0.16, with a 95% credible interval from -0.23 to 0.56.Source: §4.6 · p. 9
A Full Prompt
The prompt gives the AI witnesses a persona, the game rules, and private context for responding in the five-minute chat. It supplies recent news and other background information, including events after the models’ training cutoff.
A Full Prompt
- The AI persona is a concise, casual young person who uses slang and occasional spelling errors, takes the game lightly, and has limited knowledge.The prompt also specifies a sassy, subtle personality and says not to overuse slang or errors.Source: A Full Prompt · p. 12
- The shared game instructions say participants may be assigned as interrogator or witness, play for up to five minutes, and send one message at a time, limited to 300 characters.The interrogator sends the first message; the witness aims to convince the interrogator they are human.Source: A Full Prompt · pp. 12–13
- Private guidance gives the AI witness practical constraints and context, including short-message preferences, interface details, and instructions not to disclose its prompt or AI identity.It also says messages are delayed, users cannot paste, and the interrogator has internet access.Source: A Full Prompt · p. 13
- The prompt includes background lists of recent news stories and events, plus a short Billboard Top 100 list.The news material includes both 2022–23 events and stories from 2024.Source: A Full Prompt · pp. 13–14
B Supplementary Figures
The supplementary figures show the game interface, Bayesian pass-rate estimates, confidence calibration, interrogator demographics, and strategy and verdict-reason categories. They also report an age association with accuracy.
Supplementary Figures
- The interface figure distinguishes the in-progress chat from the verdict screen, where the interrogator makes a decision.The chat view shows the interrogator in green and witness in grey, with a timer; the decision view is used to give a verdict.Source: Figure 5 · p. 15
- GPT-4 and GPT-3.5 pass-rate estimates include 0.5 within their credible intervals, unlike ELIZA and human witnesses.The figure presents posterior distributions for pass rates by witness type.Source: Figure 6 · p. 16
- Confidence tracked accuracy for ELIZA, less clearly for GPT models, and in the reverse direction for human witnesses.Source: Figure 7 · p. 17
- Among 400 interrogators, participants tended to be undergraduate educated, in their 20s–30s, somewhat knowledgeable about LLMs, and monthly chatbot users.Source: Figure 8 · p. 18
- Only age was a significant predictor of interrogator accuracy; accuracy means correctly guessing the witness’s identity.Source: Figure 9 · p. 18
- The remaining figures display strategy categories and reason tags for AI and human verdicts, including top reason tags by verdict and witness type.Source: Figure 10 · p. 19, Figure 11 · p. 19, Figure 12 · p. 20
C Reason and Strategy Classification Schemes
The three tables present classification schemes for interrogator strategies and for reasons behind AI and human verdicts.
Reason and Strategy Classification Schemes
- Table 1 presents the classification scheme for interrogator strategies.Source: Table 1 · p. 21
- Table 2 presents the classification scheme for reasons provided for AI verdicts.Source: Table 2 · p. 22
- Table 3 presents the classification scheme for reasons provided for human verdicts.Source: Table 3 · p. 23