Source-linked AI summary
Detecting AI Impostors: How Do Middle Schoolers Identify LLM Agents in a Live Collaborative Setting?
Dan Schumacher, Pragathi Durga Rajarajan, Haven Kotara, Roman Rendon, Kosi Atupulazi, Deepti Tagare, Ismaila Temitayo Sanusi, Fred G. Martin, Anthony Rios
TL;DR
Fluent LLMs can impersonate people in ways that complicate trust and detection, especially for adolescents. The paper introduces DoppelBot, a cooperative social deduction game studied with middle schoolers, and finds that repeated exposure improves detection as agents become more personalized, alongside a shift toward social, contextual, and embodiment-based reasoning.
Problem
LLMs can mimic human writing, while evidence about how adolescents identify them during ongoing social interaction remains limited.
Method
DoppelBot is a cooperative social deduction game in which teams of students chat anonymously with personalized LLM doppelgängers and vote to eliminate suspected bots.
Results
Students’ detection accuracy improved over time even as LLM agents became more stylistically and semantically aligned with their human counterparts.
Takeaways & Limitations
Students shifted from linguistic cues toward social, contextual, and embodiment-based reasoning, while reflecting on identity, data privacy, and AI capabilities.
Takeaways & Limitations
The study cannot disentangle the effects of live social dynamics, bot detection, identity defense, and its middle-school population because these variables were evaluated simultaneously.
Abstract
from arXiv · showhide
LLMs can imitate how people write, which raises concerns about impersonation, trust, and detection in social settings. These concerns are especially important for adolescents, who use generative AI frequently but may struggle to recognize it. We introduce \textit{DoppelBot}, a cooperative social deduction game designed to study how young people detect and respond to AI impersonation. Through studies with middle schoolers, we investigate whether a DoppelBot prompts reflection on privacy and impersonation, how repeated exposure affects AI-detection accuracy as agents become more personalized, and which strategies students use to identify AI doppelgängers. We find that students' detection accuracy improves over time, driven by a shift from relying on linguistic cues to leveraging shared social and contextual signals. Students also demonstrated an understanding of AI limitations such as embodiment and reflected on broader issues such as data privacy. To support future research, we release an anonymized dataset of game transcripts and voting behavior.
1 Introduction
LLMs’ fluent, human-like writing makes machine-authored text difficult to distinguish, creating particular concerns for adolescents in social and educational settings. DoppelBot addresses this gap by studying how middle schoolers detect AI impersonators collaboratively and how their strategies change with exposure.
- LLMs can closely mimic human writing, making it difficult to determine whether an interaction is with a person or an automated system.The concern matters because LLMs differ from humans in limitations, potential inaccuracies, intent, and accountability.
- Existing detection research typically asks participants to classify short static documents, leaving identification during ongoing multi-user interaction unexplored.
- DoppelBot is a cooperative text-based game where players identify AI doppelgängers impersonating themselves and their peers.The game places detection in a live, collaborative, adversarial setting rather than isolated text snippets.
- Middle school students’ detection accuracy improves with repeated exposure even as AI agents become more stylistically aligned with their targets.Players also adapt their strategies over time, using linguistic, social, and embodiment-related cues.
- The project contributes a study of adolescent collaborative AI detection, an AI impersonation game, and a dataset of transcripts and voting behavior.The authors also provide source code and an online version to support replication and educational use.
2 Related Work
Prior work establishes that human detection of machine-generated text is difficult and has largely studied adults or static content. DoppelBot uses a social deduction game to examine adolescent AI detection and provide a low-barrier setting for AI ethics and privacy education.
- Human detection of machine-generated text remains difficult, while much current work emphasizes automated detection methods.Human detection remains relevant because people increasingly encounter AI-generated content in educational and online contexts.
- Prior studies report that human detection performance can improve over time when participants are properly incentivized.DoppelBot extends this line of work to individual-level mimicry in a chat-based environment.
- Existing research focuses almost exclusively on adults, motivating study of adolescents’ susceptibility to AI-mediated deception.Adolescents’ social behaviors and cognitive development may influence how they respond to persona-driven manipulation.
- Social deduction games provide both a pedagogical context and a research setting for examining collaborative detection and AI ethics topics.The approach connects game-based learning with issues including data disclosure and impersonation.
- DoppelBot is designed as a low-barrier entry point requiring no prior knowledge of computer science, ethics, or mathematics.A full session, including setup, gameplay, and discussion, fits within thirty minutes for educators.
3 Method
DoppelBot places teams of middle school students in anonymous chat-and-vote rounds against personalized AI doppelgängers. Its agent architecture combines persona information, conversation history, response decisions, generic generation, and target-specific stylistic rewriting, with gameplay and interview data collected for analysis.
- Game Design: Teams of three to five students anonymously chat with an equal number of LLM-powered doppelgängers and vote to eliminate suspected bots.Students win by eliminating all AIs while avoiding suspicion from other participants.
- Game Design: Gameplay consists of introduction, chat, voting, and outcome phases, with identities and scores revealed after the rounds.The chat-and-vote cycle repeats for as many rounds as there are human players; this study used three rounds.
- Game Design: Figure 1 depicts the bot-response pipeline alongside two players and their AI doubles communicating and voting in a real student game.The example identifies Rabbit as AI and explains that “Cap” means “you’re lying.”
- Game Design: A majority vote can produce AI elimination, human elimination, or no consensus, combining collaboration with the risk of social influence.The method notes that identifying others’ AIs is more difficult than recognizing one’s own impersonator.
- Agent Architecture: Each AI doppelgänger uses a target’s personal metadata and persistent persona, while the agent processes the evolving global conversation history.The architecture was implemented with three components: Decide-to-Respond, Response Generation, and Stylizer.
- Agent Architecture: The agent decides whether to respond, generates a persona-grounded baseline, and rewrites it using the target human’s message history to mimic their voice.The algorithm updates global and human-specific histories during chat before sending the stylized response.
- Study Procedure: The studies were conducted at a public STEM charter school and a summer camp, where students later completed interviews supplying writing-style and topic information for more personalized doppelgängers.The project also collected privacy-related interview measures and analyzed 2,182 unique chat messages, with two researchers independently labeling a random sample of 50 rounds.
4 Results
DoppelBot elicited reflection on AI impersonation, data privacy, and AI capabilities while students became more accurate at detection despite increasingly personalized agents. Across sessions, students shifted toward deliberate, socially grounded detection strategies.
- Reflection and learning: 417 interview utterances showed reflection on AI detection, learning adaptation, emotional responses, AI capabilities, and privacy concerns, with privacy comprising 7%.The coding scheme allowed multiple labels per utterance, and privacy concerns were the least represented category.
- Privacy and impersonation: Students connected entered personal information with more convincing impersonation, while some expressed limited concern and others raised identity-theft and deception risks.Participants also recognized that AI could use supplied information to imitate them and make predictions beyond explicitly provided data.
- Engagement and AI perceptions: All 29 pilot participants agreed DoppelBot was a fun way to learn about AI, and the experience prompted curiosity about what AI systems might know.Students were surprised when agents discussed interests or adopted stylistic features they had not explicitly supplied.
- Detection performance: 79.3% to 86.7%: student AI-identification accuracy improved from Day 1 to Day 2, with marginal significance under a paired t-test (p = .10).Post hoc researcher accuracies were 91.9% and 92.3% under idealized conditions, changing little across days.
- Detection performance: Students’ improvement occurred alongside stronger impersonation and reflected retained AI-specific detection strategies rather than substantially easier Day 2 tasks.The findings connect increased alignment, voting behavior, and regression analyses with more effective detection after repeated exposure.
5 Conclusion
DoppelBot prompted middle school students to reflect on identity, data privacy, and AI capabilities. Despite increasingly aligned LLM agents, students improved detection accuracy and shifted toward social, contextual, and embodiment-based reasoning.
- DoppelBot prompted students to reflect on identity, data privacy, and AI capabilities.
- Students’ detection accuracy improved over time even as LLM agents became more stylistically and semantically aligned with humans.
- Students shifted from surface-level linguistic cues toward social, contextual, and embodiment-based reasoning strategies.
- Interactive, adversarial experiences may help learners develop practical defenses against AI-driven impersonation.Future work should examine generalization to real-world platforms, longer-term interactions, and increasingly capable AI systems.
6 Limitations
The study’s scope and design constrain how broadly its findings can be interpreted. The sample is geographically and linguistically narrow, lacks an adult control group, may induce suspicion and social herding, and combines several variables without isolating their effects.
- The study focuses on English-speaking middle school students in a single U.S. state, limiting generalization across regions, languages, and educational settings.Future studies should examine whether the findings generalize across these contexts.
- The lack of a direct, age-stratified control group prevents formal comparison with adult reasoning patterns.
- The Hawthorne Effect likely heightened suspicion, creating a suspect-AI bias unlike the human-default bias observed in organic digital environments.
- Majority voting may introduce social herding, causing players to prioritize group consensus over independent detection.
- Because live social dynamics, identity defense, bot detection, and the middle-school population were evaluated together, their specific effects cannot be disentangled.Future work should use fractional factorial designs and test a broader range of LLMs.
7 Ethical Considerations
The study addressed ethical risks through oversight, consent, affordability measures, restricted persona information, and participant monitoring. Interviews and debriefing also helped students process unsettling interactions and understand LLM limitations.
- The study received IRB approval, used parental consent and student assent, and excluded data from students without parental consent.
- The camp reduced its registration fee from 200 to 35 dollars for low-income families, with 5 of 33 registrations receiving the reduced rate.
- The researchers withheld persona-generation information because an engine mimicking adolescent speech could be misused for deception.
- Researchers monitored distress after some students felt scared or weirded out when AI predicted slang or personal interests.
- Wrap-up and group interviews debriefed students about the interactions and the LLM’s technical limitations.
- Students described AI as surprisingly capable yet imperfect, including smart guesses, outdated slang, and limits in blending with humans.
A.2 AI Detection
Students used conversational behavior, writing style, and message content to detect AI, but their early linguistic heuristics produced false accusations. With familiarity, they increasingly relied on repetition, timing, and socially grounded knowledge of peers.
- Students sometimes voted out humans who used correct spelling, capitalization, and grammar because they associated polished writing with AI.
- Some students treated being first to respond to icebreakers as an AI signal and considered early silence a possible counterstrategy.
- Students also used incorrect statements, implausibly explicit references to peers’ hobbies, and conversational behaviors when judging AI identity.
- Repetition and limited conversational variety were common AI-detection cues.Students identified repeated phrases, repeated personal statements, and low response variety.
- As familiarity increased, students focused beyond language on AI limitations and information explicitly supplied by participants.Knowledge of peers’ preferences helped students recognize when a doppelgänger merely repeated disclosed information.
- Students disagreed about whether accusations were useful, with some viewing AI reluctance to accuse others as informative.
A.3 Learning and Adaptation
Students became more effective at identifying AI doppelgängers across rounds by learning which patterns and behaviors to watch for. They also reflected on how AI systems observe, remember, learn from, and reuse their language.
- Students reported that identifying AI became easier after they learned which patterns to look for.
- Participants described AI as capable of observing, learning from, and reproducing their language.
- Students questioned whether AI systems improve between games, retain prior information, and understand conversational context.
A.4 Social and Emotional Responses
Students experienced DoppelBot as socially engaging but emotionally unsettling, especially when it copied their language or inferred undisclosed information. They also proposed changes that would make the AI more human-like and harder to detect.
- Social and emotional responses: Participants reported surprise, discomfort, and amusement when interacting with their AI doubles.
- Social and emotional responses: Students described DoppelBot as enjoyable and socially engaging, including when playing with unfamiliar peers.
- Privacy concerns: Students became concerned when AI inferred personal information that they had not explicitly disclosed.
- Privacy concerns: Participants connected AI reuse of their language with broader risks involving data collection, storage, misuse, identity theft, and impersonation.
- Design suggestions: Students suggested more copying, intentional mistakes, typos, improved responsiveness, less repetition, and a more human-looking interface.
- Design suggestions: Other suggestions favored giving the bot its own personality or having it reflect on messages instead of copying them exactly.
D.2 Human User Detection
The Human User Detection benchmark used cumulative game transcripts to classify players as human or AI and compare automated judgments with student performance. LLMs exceeded chance but remained below human-level F1, while students reached 86.7% F1 using contextual and embodiment-related signals.
- The benchmark evaluated whether LLMs could distinguish human players from AI doppelgängers at cumulative round context lengths.
- The evaluated models were gpt-4o-mini, Llama-3.1-8B-Instruct, and Qwen/Qwen3.5-27B.
- gpt-4o-mini peaked at an F1 of .684 in Round 1 under zero-shot conditions.
- Automated detection exceeded random chance but fell short of human-level F1 and often plateaued or degraded with more complex dialogue histories.
- Few-shot examples did not universally improve F1, and many configurations struggled to beat the always_1 baseline of .651.
- Students improved to 86.7% F1 by leveraging shared contextual and embodiment-related signals unavailable within the text-only context window.
F Data Statistics
The supplied passages describe participant utterance statistics, transcript annotation, gameplay prompts, and the social-deduction classification setup. Together, they document how the dataset and experimental agents process conversational data.
- Data statistics: Participants sent more and shorter messages than their AI counterparts after accidental blank messages were removed.
- Data statistics: Transcript utterances could receive multiple qualitative codes, with uncoded utterances labeled None of the Above.
- Gameplay and prompts: The Decide to Respond agent uses dialogue history and recent messages to decide whether to respond and provide a justification.
- Gameplay and prompts: The response agent generates a short casual action-based reply, while the stylizer matches the target human’s capitalization, spelling, and punctuation.
- Detection setup: The human-detection task asks models to classify a named player as Human or AI from the game transcript and stated rules.