PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond pass or fail: an information-processing account of the Turing Test

Qiwei Zhao, Oriana Pansardi, James W. Strachan, Eugenio Scaliti, Guido Manzi, Giulia Borghini, Fabio Manzi, Alessandro Rufo, Frédérique de Vignemont, Michael S. A. Graziano, Stefano Panzeri, Cristina Becchio

arXiv:2610.32868v1cs.HC

TL;DR

Behavioral indistinguishability in the Turing Test can arise because source information is absent or because an interrogator fails to extract information that is present. This paper compares encoding, readout, and meta-readout by placing humans and AI in symmetrical interrogator roles. GPT-5 interrogators read source information more accurately, but their reported feature weightings aligned less well with their actual readout than human interrogators’ reports did.

  • Problem

    Behavioral indistinguishability can reflect either absent source information or a failure by the interrogator to extract information that is present, making a pass-or-fail outcome ambiguous.

    Source: Introduction · p. 3

  • Method

    The authors invert the Turing Test and compare source information in responses, interrogators’ use of that information, and their explicit access to their own readout computations.

    Source: Introduction · p. 3

  • Results

    GPT-5 interrogators achieved high readout alignment but lower meta-readout alignment than humans, who showed modest readout alignment and comparatively higher meta-readout alignment.

    Source: Discussion · pp. 11–12

  • Takeaways & Limitations

    Separating encoding, readout, and meta-readout provides a mechanistic account of how human and artificial systems process information about generating sources.

    Source: Discussion · p. 13

    GPT-5 interrogators’ above-chance reporting of used features did not rule out that their explanations were partly post-hoc rationalizations decoupled from readout.

    Source: Discussion · p. 13

Abstract

from arXiv · show

The Turing Test is passed when a human interrogator cannot distinguish machine-generated responses from human ones. Yet behavioral indistinguishability can reflect two fundamentally different failures: a failure of encoding, in which machine and human output distributions overlap so closely that source information is effectively absent; or a failure of readout, in which source information is present but the interrogator fails to extract it. To dissociate these possibilities, we inverted the Turing Test, placing humans and AI systems in symmetrical interrogator roles. Using a quantitative information-processing framework, we separated encoding, the information contained in the text; readout, the extraction of that information by an interrogator; and meta-readout, the interrogator's explicit access to the features guiding its own judgments. Across two series of experiments involving more than 3800 human interrogators and two generations of GPT models (GPT-4 and GPT-5), we found a striking dissociation across processing levels and interrogator types. As AI-generated responses became more human-like from GPT-4 to GPT-5, source information became more difficult for human interrogators to extract. GPT-5 interrogators, by contrast, read out source-discriminating information more accurately than GPT-4 interrogators. This improvement, however, came with a specific limitation: GPT-5 explanations only partially aligned with the feature weightings that actually predicted their judgments. Human interrogators showed the complementary pattern, with modest readout alignment but comparatively higher meta-readout alignment. Together, these findings suggest a double dissociation between readout and meta-readout across human and GPT interrogators.

Introduction

Behavioral indistinguishability can reflect either absent source information in the responses or an interrogator’s failure to extract information that is present. The paper reframes the Turing Test as an information-processing problem, comparing encoding, readout, and interrogators’ explicit access to their judgments.

Introduction

  • Indistinguishability may reflect overlapping human and machine output distributions, leaving source information effectively absent, or a readout failure despite information being present.In the latter case, another interrogator might still discriminate between the sources.Source: Introduction · p. 3
  • The paper proposes separating these explanations by recasting the Turing Test as an information-processing problem of encoding and readout.Standard implementations conflate the two possibilities.Source: Introduction · p. 3
  • The framework tests encoding in responses, readout by human and artificial interrogators, and whether interrogators can report the features guiding their choices.The final level, meta-readout, tests whether interrogators have access to their own readout computations.Source: Introduction · p. 3
  • Across two GPT generations and 3,824 human participants, the authors report that GPT-5 interrogators read source information more accurately, while their explanations aligned only partly with their actual feature weightings.Human interrogators showed the complementary pattern: modest readout alignment but comparatively higher meta-readout alignment.Source: Introduction · pp. 3–4

Results

Human and AI interrogators distinguished human- from AI-generated responses above chance, but their sensitivity and response biases differed across tasks and GPT generations. GPT-5 interrogators outperformed humans in both tasks, whereas GPT-4 interrogators were less sensitive than humans across both tasks.

Results

  • The two study series paired responses to Strange Stories with human and GPT-4 interrogators in the first series and human and GPT-5 interrogators in the second.Interrogators chose which of two consecutive response intervals contained the target source and gave a free-text explanation.Source: Results · p. 4

Behavioral (in-)distinguishability

  • Human interrogators distinguished human- from GPT-4-generated responses above chance, with d’ = 1.36 in the AI task and d’ = 1.10 in the Human task.They had a small AI-task advantage (p = 0.018) and no systematic criterion bias in either task.Source: Behavioral (in-)distinguishability · p. 4
  • GPT-4 interrogators had d’ = 0.79 in the AI task but d’ = -1.11 in the Human task, with lower sensitivity than human interrogators in both tasks.Their criterion was negative in both tasks, indicating a tendency to report that the target appeared in the first interval.Source: Behavioral (in-)distinguishability · pp. 4–5, Figure 1 · p. 5
  • Human interrogators distinguished human- from GPT-5-generated responses above chance in both tasks, with d’ = 0.77 in the AI task and d’ = 0.27 in the Human task.Sensitivity was higher in the AI task (p = 0.004).Source: Behavioral (in-)distinguishability · p. 5
  • GPT-5 interrogators achieved d’ = 2.09 in the AI task and d’ = 2.01 in the Human task, outperforming human interrogators in both tasks.Their sensitivity did not reliably differ between tasks (p = 0.094); criterion bias was absent in the AI task and small and negative in the Human task.Source: Behavioral (in-)distinguishability · p. 5, Figure 1 · p. 5

Encoding

Encoding models found source information in responses from both GPT generations, with GPT-5 responses more linguistically similar to human responses than GPT-4 responses. Readout varied by interrogator: GPT-5 interrogators used informative features more accurately than human interrogators, while GPT-4 interrogators showed task-dependent alignment.

Encoding

  • Encoding accuracy was 0.983 for human versus GPT-4 responses and 0.925 for human versus GPT-5 responses, with significantly lower accuracy for the GPT-5 comparison.Lower accuracy indicates greater overlap in the feature space captured by the encoding model.Source: Encoding · p. 6, Figure 2 · p. 7
  • GPT-4 response differences were characterized mainly by greater length, fewer typos, and increased redundancy; GPT-5 length differences were reduced, while punctuation density and typos became more informative.The encoding model predicted generating source from linguistic-feature differences; higher accuracy indicates greater separability in the modeled feature space.Source: Encoding · p. 6, Figure 2 · p. 7

Readout

  • GPT-4 interrogators’ alignment was 0.569 in the AI task but -0.756 in the Human task, where they reversed the mapping between response length and source.Human interrogators consistently weighted longer responses toward AI and had alignment of 0.782 in the AI task and 0.701 in the Human task.Source: Readout · p. 8
  • Human interrogators showed lower overlap and alignment than GPT-5 interrogators in both tasks, and relied modestly on informative punctuation and typographical errors.In the AI task, human readout overlap and alignment were 0.688 and 0.619; in the Human task they were 0.572 and 0.346.Source: Readout · p. 8
  • GPT-5 interrogators showed high readout overlap and alignment in both tasks, with alignment of 0.924 in the AI task and 0.919 in the Human task.They associated denser punctuation and greater redundancy with AI responses, and typos with human responses.Source: Readout · p. 8, Figure 3 · p. 9

Meta-readout

Meta-readout compares the features interrogators reported using with the features and weightings inferred from their choices. Human interrogators showed comparatively higher alignment between reported and actual weighting, while GPT-5 interrogators identified features they used but described their weighting less accurately.

Meta-readout

  • Human interrogators’ meta-readout overlap and alignment were moderate in the first series, with alignment of 0.639 in the AI task and 0.624 in the Human task.Overlap and alignment did not vary with chatbot use in the AI task; overlap was higher with more frequent use in the Human task.Source: Meta-readout · p. 10
  • GPT-4 interrogators showed significant meta-readout overlap and alignment in the Human task, but not in the AI task, and scored below human interrogators on both measures across tasks.The authors interpret this as limited explicit access to which features guided decisions and how they were weighted.Source: Meta-readout · p. 10
  • In the second series, human meta-readout overlap was significant in the Human task but not the AI task; GPT-5 overlap was above chance in both tasks.GPT-5 overlap exceeded human overlap in the AI task (p = 0.036).Source: Meta-readout · p. 10
  • GPT-5 meta-readout alignment was lower than human alignment in both tasks, at 0.297 in the AI task and 0.296 in the Human task.GPT-5 reported some features it used but sometimes described their direction opposite to its actual readout weights, including analytical tone, readability, and detail.Source: Meta-readout · p. 10

Discussion

The framework separates source information in responses from interrogators’ use of that information and their access to their own readout. GPT-5 responses were more similar to human responses, while human and GPT-5 interrogators showed contrasting readout and meta-readout patterns.

Discussion

  • Encoding accuracy declined from GPT-4 to GPT-5, although human- and AI-generated responses remained statistically distinguishable and became more similar in the measured linguistic feature space.GPT-4 responses were primarily longer than human responses; GPT-5 responses instead had denser punctuation and fewer typographical errors, with no length difference.Source: Discussion · pp. 11–12
  • Human interrogators used GPT-5’s most diagnostic features weakly while retaining substantial weight on response length, which no longer reliably distinguished GPT-5 from human responses.Participants reporting greater chatbot use did not show the same legacy weighting and correctly weighted punctuation density and typographical errors.Source: Discussion · p. 12
  • GPT-5 interrogators achieved high and symmetric readout overlap and alignment across task conditions despite the greater difficulty of distinguishing GPT-5 from human responses.The authors speculate that this may resemble expertise, but frame that interpretation as a possibility.Source: Discussion · p. 12
  • GPT-5 interrogators showed high readout alignment but lower meta-readout alignment than humans, whose readout alignment was modest and meta-readout alignment comparatively higher.The authors suggest meta-readout misalignment is limited to features including analytic tone, readability, and detail.Source: Discussion · pp. 12–13
  • A localized breakdown in alignment may contribute to coordination failures when users infer that a model’s reasoning matches their own despite divergent feature-to-conclusion mappings.The authors note this may matter when people consult or delegate decisions to LLMs.Source: Discussion · p. 13
  • Inverting the test and separating encoding, readout, and meta-readout offers a mechanistic alternative to binary pass-or-fail judgments, with metrics the authors suggest may extend beyond the Turing Test.The framework examines source information in text, how interrogators use it, and their access to those computations.Source: Discussion · p. 13

Methods

The studies compared human and GPT interrogators across two series, one using GPT-4 responses and one using GPT-5 responses.

Methods

  • The first study series evaluated human- and GPT-4-generated responses with both human and GPT-4 interrogators; the second did the same with GPT-5 responses and interrogators.The research protocol received local ethics committee approval and complied with the revised Helsinki Declaration.Source: Methods · p. 14
  • The research protocol was approved by the Ethik-Kommission der Ärztekammer Hamburg.The study reported compliance with the principles of the revised Helsinki Declaration.Source: Methods · p. 14
  • The studies used symmetrical interrogator roles to evaluate responses from the corresponding GPT generation and from humans.The first series paired GPT-4 responses with GPT-4 interrogators, and the second paired GPT-5 responses with GPT-5 interrogators; both also included human interrogators.Source: Methods · p. 14

Experimental models and subject details

Human interrogators were recruited online, while GPT-4 and GPT-5 interrogators were queried through the OpenAI API.

Experimental models and subject details

  • The final human samples included 1,904 participants for the GPT-4 comparison and 1,920 for the GPT-5 comparison, each contributing one judgment.Participants were native English speakers aged 18–70 recruited through Prolific and were instructed to use a laptop or desktop.Source: Experimental models and subject details · p. 14
  • Human participants with prior participation in the earlier study or prior completion of the Strange Stories test for the GPT-5 comparison were excluded.Mobile-browser users were excluded and replaced; participants reporting psychiatric conditions or dyslexia were ineligible.Source: Experimental models and subject details · p. 14
  • GPT-4 (‘gpt-4-0613’) and GPT-5 (‘gpt-5-2025-08-07’) served as the respective AI interrogators, with data collected in November 2024 and September 2025.Both were queried through the OpenAI API with temperature 0.8 and top-p 0.9.Source: Experimental models and subject details · p. 14

Stimuli

The experiments used correct responses to Strange Stories social scenarios, pairing human answers with GPT answers within items.

Stimuli

  • Strange Stories prompts present short social scenarios and ask for open-ended descriptions of a character’s mental state or social action.Scenarios include deception, irony, persuasion, misunderstanding, and mistaken identity.Source: Stimuli · p. 14
  • Only responses scored as correct under the original test criteria (score 2) were retained, preventing discrimination based on incorrect mental-state reasoning.The test was selected because it elicits extended textual answers suitable for linguistic-feature analysis.Source: Stimuli · p. 14
  • The GPT-4 set contained 119 correct human–GPT-4 response pairs, sampled and paired within item.One GPT-4 answer was excluded for failing the correctness criterion; human correct responses were downsampled from 295.Source: Stimuli · p. 14
  • The GPT-5 set contained 120 correct human–GPT-5 response pairs, sampled and paired within item.All 120 GPT-5 responses met the correctness criterion; 267 correct human responses were downsampled.Source: Stimuli · pp. 14–15

Turing and Inverse Turing Test protocol

Each interrogator makes an authorship choice between two responses, then rates confidence and explains the decision. The procedure uses a two-interval forced-choice judgment, with task instructions specifying whether to identify the human or AI response.

Turing and Inverse Turing Test protocol

  • Each trial presents a Strange Stories item followed by two responses to the same question, one human-generated and one AI-generated.The responses appear in either human–AI or AI–human order.Source: Turing and Inverse Turing Test protocol · p. 15
  • Interrogators choose which response matches the instructed source—human or AI—in a two-interval forced-choice judgment.They then rate confidence from 1 to 100 and provide a written explanation of their choice.Source: Turing and Inverse Turing Test protocol · p. 15

Procedure

Human and LLM interrogators completed matched authorship judgments, confidence ratings, and written explanations. The studies counterbalanced response order and task instruction, then quantified discrimination and linguistic features using specified modeling and coding procedures.

Procedure

  • Each trial proceeded through a source decision, confidence rating, and written explanation, with human and LLM procedures matched in sequence.Human trials ran online; LLM trials used chained prompts so confidence and explanation followed the decision.Source: Procedure · p. 15
  • Each response pair, order, and task combination received four human judgments and four LLM runs.The GPT-4 comparison included 119 pairs and 1,904 observations per interrogator type; the GPT-5 comparison included 120 pairs and 1,920 observations per type.Source: Procedure · p. 15
  • Signal detection theory estimated sensitivity to source and criterion, the tendency to select one response position over the other.Indices were computed separately by stimulus pair, interrogator type, and task.Source: Procedure · pp. 15–16
  • The analyses quantified 12 linguistic features, selected from cues reported in explanations and measured with LIWC-22, Python, or manual coding.Features included length, detail, typos, redundancy, punctuation, lexical diversity, readability, tone, emotion references, social awareness, and factual errors; analytic tone was also specified.Source: Procedure · p. 16
  • Redundancy and factual errors were manually coded, while punctuation frequency was measured with LIWC-22 and factual errors were rare.Punctuation, typos, and other automated features were normalized as described in their operational definitions.Source: Procedure · pp. 16–17
  • Lexical diversity used moving-window type-token ratios, while readability used sentence length and syllables per word; lower readability scores indicate more complex sentence structure.Conversational tone combined normalized conversational terms and contractions; emotion and social-awareness measures used LIWC-22.Source: Procedure · pp. 16–17

Encoding and readout models

Logistic models quantified source information in linguistic differences and how interrogators used that information. The analyses also compared model-derived strategies with features interrogators explicitly reported, using validation and resampling procedures.

Encoding and readout models

  • Encoding models predicted whether the human response appeared first, while readout models predicted interrogators’ choices from signed linguistic feature differences.Regression weights specify the feature combination associated with source encoding or with the interrogator’s choice, respectively.Source: Encoding and readout models · p. 17
  • Encoding and readout models used ridge-regularized logistic regression, with predictors z-scored and a shared λ =1.23285e-2 selected by cross-validated balanced accuracy.The regularization search evaluated 100 λ values from 10^-3 to 10^3; ridge regularization retained all features while addressing potential multicollinearity.Source: Encoding and readout models · pp. 17–18
  • Encoding–readout overlap measures whether the same features receive weight, while alignment measures whether their signed directions correspond.Overlap uses cosine similarity of absolute vector values; alignment uses signed vectors, with positive values indicating directionally consistent use.Source: Encoding and readout models · p. 18
  • GPT-4o-assisted coding classified feature mentions and directions; agreement with two experimenters on 379 explanations reached mention AC1=0.833 and category AC1 = 0.967.Agreement between the 10% subset and its full-dataset coding was mention AC1 = 0.958 and category AC1 = 0.982.Source: Encoding and readout models · pp. 18–19
  • Explanations were L1-normalized into directional feature vectors and averaged by interrogator type and task to form meta-readout vectors.Overlap and alignment compared these explicit explanation vectors with the corresponding readout vectors.Source: Encoding and readout models · p. 19
  • Bootstrap estimates resampled stimulus pairs, while permutation-derived null values accounted for non-zero chance levels in encoding, readout, and meta-readout measures.Analyses used 1,000 bootstrap resamples and, unless otherwise stated, 10,000 permutations; human interrogators were also analyzed by low- versus high-frequency chatbot use.Source: Encoding and readout models · pp. 19–20

Supplementary Information

The supplementary figures organize readout-model accuracy and human-interrogator measures by comparison, task, interrogator, and chatbot familiarity. They display shuffled-reference values and bootstrap uncertainty for the specified analyses.

Supplementary Information

  • Figure S1 separates the human–GPT-4 and human–GPT-5 comparisons, with interrogator in columns and task on the x-axis.Bars show balanced accuracy for readout models predicting choices from linguistic feature differences; horizontal lines mark the average 1,000-shuffle reference and error bars show ±1 SD across 1,000 stimulus-pair bootstrap resamples.Source: Figure S1 · p. 24
  • Figure S2 divides human interrogators into low- and high-LLM-familiarity groups and presents results separately for the GPT-4 and GPT-5 study series.Its panels include criterion and sensitivity, encoding–readout overlap and alignment, meta-readout weights, and readout–meta-readout overlap and alignment; error bars show ±1 SD across bootstrap estimates.Source: Figure S2 · p. 25, Supplementary Information · p. 26
Loading 2610.32868v1…