Source-linked AI summary
QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents
Ye Yuan, Rui Song, Weien Li, Zeyu Li, Haochen Liu, Xiangyu Kong, Changjiang Han, Yonghan Yang, Zichen Zhao, Zixuan Dong, Fuyuan Lyu, Bowei He, Haolun Wu, Jikun Kang, Xue Liu
TL;DR
Existing social-deduction benchmarks emphasize outcomes and text interaction, limiting direct evaluation of whether agents’ language reflects their perceptions and actions. QUACK audits this grounding across outcomes, trajectories, and utterances, finding systematic failures across three frontier VLMs, including hallucinated spatial claims and mostly unsupported accusations.
Problem
Existing environments emphasize game outcomes and largely text-only interaction, limiting direct assessment of whether dialogue matches agents’ perceptions and actions.
Method
QUACK reconstructs agents’ ground-truth trajectories from logs and automatically verifies discussion claims within a three-tier evaluation framework.
Results
Across three frontier VLMs, grounding failures arise systematically; GPT-5.5 hallucinates 16.4% of spatial claims and makes 54.5% of accusations without grounded evidence.
Takeaways & Limitations
Groundedness is a distinct capability axis not captured by task success and should be measured directly in social-deduction and multi-agent language settings.
Takeaways & Limitations
Claim extraction is slightly conservative, and the study does not isolate visual modality because it lacks a text-only ablation.
Abstract
from arXiv · showhide
Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and makes over half of its accusations without grounded evidence. We release the full engine, evaluation framework, toolkit, and logs at https://github.com/AAAAA-Academia-Attractions/QUACK.
1 Introduction
QUACK introduces an open-source multimodal social deduction environment and evaluation framework that audits whether VLM agents’ language remains grounded in their perceptions and actions. Its replayable trajectory reconstruction and Statement Verification Pipeline evaluate outcomes, behavior, and utterance-level consistency while automatically measuring four grounding failures.
- Motivation: Existing social deduction environments mainly score game outcomes, making it difficult to distinguish grounded reasoning from hallucinated evidence or plausible dialogue.Without reconstructable trajectories, important reasoning failures remain hard to identify systematically.
- QUACK Environment: QUACK provides an open-source multimodal social deduction environment with partial observability and fully replayable logs for grounded VLM-agent evaluation.Agents navigate configurable graph-based maps and interact under controlled research conditions.
- Evaluation Framework: The three-tier framework scores game outcomes, behavioral trajectories, and utterance-level consistency, extending evaluation beyond win rates toward language grounding.Tiers 1 and 2 provide outcome and behavioral context, while Tier 3 audits claims in discussion utterances.
- Statement Verification Pipeline: The Statement Verification Pipeline reconstructs trajectories, extracts structured discussion claims, and verifies them against the reconstructed world state.It operationalizes spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency, with validation against human annotation.
- Empirical Findings: 3 frontier VLM-powered agents were evaluated across 270 games in homogeneous and cross-model adversarial settings, revealing systematic grounding failures.All three models hallucinated a substantial fraction of spatial claims and made the majority of accusations without grounded evidence.
2 Related Work
QUACK bridges social-deduction research on multimodal agent interaction with efforts to evaluate social agents beyond game outcomes. It addresses gaps in existing resources by combining interactive partial observation with reconstructable trajectories for auditing grounded claims.
- Social deduction games as environments: Werewolf/Mafia-style games are widely used to study deception, persuasion, and strategic communication in LLMs.
- Social deduction games as environments: Werewolf Among Us provides an observational corpus of human gameplay for modeling persuasion, rather than an interactive vision-language environment.
- Social deduction games as environments: QUACK couples a playable, partially observed multimodal environment with reconstructable ground-truth trajectories, requiring agents to perceive, act, and justify claims.
- Evaluating social agents: Most social-deduction benchmarks emphasize outcomes such as winning, survival, or voting accuracy, revealing little about why agents succeed or fail.
- Evaluating social agents: Recent evaluation work extends beyond outcomes toward strategy quality, human alignment, opponent and belief modeling, and collaboration–competition metrics.
3 The QUACK Environment
QUACK is a partially observable Markov game on a graph-structured map, with hidden-role teams, multimodal observations, VLM agents, phase-dependent actions, and event-driven meetings. Its design supports grounded social deduction by combining private role/task information, local perception, trajectory memory, and structured state transitions.
- Teams and roles: QUACK models n agents on a graph-structured map as a partially observable Markov game with hidden-role Geese and Ducks.Agents are privately informed of their roles; Ducks additionally know fellow Ducks, while Geese know only team sizes.
- Teams and roles: In the standard configuration, n = 6 agents include m = 1 Duck, while Geese complete location-bound tasks or identify and eject all Ducks.Ducks win at voting parity and can eliminate co-located Geese subject to a cooldown, while performing indistinguishable fake tasks.
- Map and state space: The map is an undirected weighted graph of rooms and corridors, where corridor weights specify traversal time in ticks and designated rooms support tasks or emergency meetings.The released example map contains 14 weighted corridors with travel times of 1 −3 ticks.
- Multimodal observation space: Each agent receives two rendered images plus a structured textual summary encoding its spatial view, local perceptions, movement events, tasks, communication, and role-specific status.The global map supports spatial orientation without showing other players, while the local view shows currently perceived players, bodies, and witnessed arrivals or departures.
- Agent: VLM policies map observations and running trajectory memory to actions and utterances, conditioning meeting statements and votes on prior transcripts and outcomes.Memory includes occupied rooms, witnessed movements, encountered players, and previous meeting records, grounding discussion claims in accumulated partial recollection.
- Actions and transition dynamics: Actions are phase-, role-, and situation-dependent: free roam permits waiting, movement, task work, reporting, and meeting calls, while meetings switch to discussion and voting.The engine exposes legal actions with each observation, and free-roam actions mutate state through movement, kills, task progress, and proximity chat.
4 Automated Evaluation Framework
QUACK automatically evaluates social-deduction agents from engine event logs at three complementary levels: game outcomes, behavioral trajectories, and utterance groundedness. Its Statement Verification Pipeline extracts structured claims and verifies them against reconstructed trajectories, enabling auditable measurements of grounding failures.
- Three-tier evaluation: QUACK computes game outcomes, behavioral trajectories, and utterance groundedness automatically from each game’s engine-level event log.Tier 1 captures outcomes, Tier 2 reconstructs behavior, and Tier 3 audits what agents say.
- Three-tier evaluation: Tier 2 reconstructs spatial trajectories to measure behavior that outcomes miss, including task efficiency, spatial coverage, voting accuracy, kill rate, self-report, and post-kill displacement.Metrics differ by role: Geese receive movement, task, voting, and reporting measures, while Ducks receive killing and post-kill behavior measures.
- Statement Verification Pipeline: The verification pipeline parses utterances into LOCATION, ROUTE, SIGHTING, ACTIVITY, ACCUSATION, and DEFENSE claims, then checks them against reconstructed ground-truth trajectories over resolved time windows.Verification uses logged occupancy, visibility, task, movement, and accusation evidence, storing supporting evidence for auditable verdicts.
- Statement Verification Pipeline: 99.5% of sampled claims received correct extraction and verdict judgments, with human checks finding 199 correct cases out of 200.Recall was separately assessed by comparing extracted claims with claims listed by humans across sampled utterances.
- Grounding-failure metrics: The framework operationalizes spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency as measurable grounding failures.It also reports lie detection rate by linking a Duck’s verifiable lies in meetings to subsequent ejections, while separating accusation outcome from accusation groundedness.
5 Experiments
QUACK evaluates three frontier VLM agents across homogeneous and cross-model adversarial regimes totaling 270 games. Despite generally balanced outcomes, the audit reveals systematic grounding failures across spatial claims, accusations, deception, and language-action consistency that win rates alone obscure.
- 5.1 Setup: Three agents—GPT-5.5, GEMINI-3.1-PRO, and CLAUDE-OPUS-4.7—receive identical prompts and act from multimodal observations through the same action interface.Experiments use a 10-room map with n = 6 agents and m = 1 Duck across homogeneous and cross-model adversarial regimes.
- 5.1 Setup: 270 games cover 9 settings, with 30 games per setting and shared initialization seeds across settings.Results aggregate model metrics over 90 games in each role, reporting means over games unless noted.
- 5.2 Overall outcomes: Across 9 settings, Geese win 63.3–93.3% of games, Ducks win 6.7–36.7%, and ejection accuracy ranges from 51.7–93.3%.These results indicate an adversarial task rather than a trivially crew-favored one, while deduction remains unreliable.
- 5.3 Grounding failure modes: 15.1% is the pooled spatial hallucination rate, meaning roughly one in seven verifiable location or sighting claims contradicts the agent’s trajectory.The finding is consistent with long-horizon reasoning difficulty under partial observability.
- 5.3 Grounding failure modes: 53.5% is the pooled unsupported accusation rate, with more than half of accusations lacking evidence from the accuser’s grounded observations.This failure is described as remarkably stable across crews and shared qualitatively by all three frontier models.
- 5.3 Grounding failure modes: 22.1% is the pooled Duck deception rate, while language-action inconsistency appears when stated tasks or paths conflict with logged actions.Duck lies are almost never subtle near-misses, and faked-task alibis can contradict recorded activity.
- 5.3 Grounding failure modes: 75.2% of meetings with a verifiable Duck lie end in Duck ejection, falling to 58.4% when CLAUDE-OPUS-4.7 is the crew.Thus, mechanically falsifiable evidence is often not converted into correct deduction.
6 Conclusion and Discussion
QUACK is an open-source framework for auditing whether multimodal social-deduction agents’ language remains grounded in their perceptions and actions. It evaluates grounding across outcomes, behavioral trajectories, and utterance-level consistency, positioning groundedness as a capability distinct from task success.
- Framework: QUACK audits whether multimodal social-deduction agents’ language stays grounded in what they actually perceived and did.It is introduced as an open-source environment and evaluation framework.
- Framework: QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency.This extends beyond benchmarks scored almost entirely by game outcomes.
- Broader takeaways: Groundedness is a distinct capability axis in social-deduction and multi-agent language settings that task success does not capture.The authors argue it therefore deserves direct measurement.
- Broader takeaways: Social-deduction games provide a convenient setting for studying grounded generation because they incentivize verifiable claims and lies while preserving a recoverable world state.The passage presents this combination as especially useful for studying grounded generation.
- Future work: QUACK is intended as both a diagnostic for current agents and a substrate for future work on groundedness optimization and richer environments.Suggested directions include explicitly optimizing discussion for groundedness and extending verification beyond the current setting.
Limitations
The claim-extraction component relies on an LLM and is slightly conservative, occasionally dropping claims. Human validation indicates high precision and recall, while missed claims reduce coverage rather than corrupting scored verdicts.
- Claim extraction relies on an LLM and is slightly conservative, occasionally dropping claims.This limitation can lead to incomplete claim coverage.
- 199/200 sampled claims had correct extractions and verdicts in human validation.The pipeline was precise on the validation sample.
- 220/223 claims were recovered, indicating high recall despite occasional missed claims.Missed claims reduce coverage rather than corrupting verdicts, so reported rates may slightly undercount total claims made.
Ethical Considerations · A Demonstration of Essential Prompts
QUACK confines deception research to a synthetic AI-agent game and frames its evaluation for auditing and red-teaming, while releasing prompts that specify multimodal gameplay, role strategies, response formats, and claim extraction.
- Ethical Considerations: All interactions occur between AI agents in a synthetic game, without human subjects, personal data, or real-world social relationships.The work studies deception detectability rather than optimizing persuasion or manipulation.
- Ethical Considerations: The framework has dual-use potential because measuring lie detectability could inform efforts to make deception harder to detect.The authors judge auditing grounded behavior especially valuable as VLM agents are deployed to report what they perceived and did.
- Ethical Considerations: The authors release the environment, evaluation framework, and logs openly to support responsible auditing of agent claims.Open release enables scrutiny of whether reported behavior is grounded in observed trajectories.
- A Demonstration of Essential Prompts: The shared system prompt defines Goose Duck roles, objectives, team composition, player identities, game rules, multimodal observations, and exact action, discussion, and voting formats.Agents receive a global map with room layout and task locations plus a local view showing only their current room and immediate surroundings.
- A Demonstration of Essential Prompts: The Goose strategy prioritizes task completion, paired travel, observation of player behavior, evidence-based discussion, and skipping when no strong suspect exists.Its discussion guidance asks agents to report routes, encounters, locations, activities, victim sightings, and timing before accusing.
- A Demonstration of Essential Prompts: The Duck strategy instructs impostors to eliminate Geese, create alibis through fake tasks and plausible routes, manipulate discussion, protect teammates, and avoid revealing their role.It also advises moving away after kills, generally withholding body reports, and pushing votes toward Geese.
- A Demonstration of Essential Prompts: The claim-extraction prompt instructs the Tier-3 pipeline to output all verifiable meeting claims as JSON arrays covering locations, routes, sightings, activities, accusations, and defenses.It requires exact room names, temporal references, preserved route order, no vague claims or duplicates, and JSON-only output.
B All Metrics
Table 3 summarizes all metrics by tier.
- Table 3 summarizes all metrics by tier.
C Full Evaluation Results
The appendix provides complete evaluation metrics across nine model-pair settings, organized into three tiers covering outcomes, behavior, and statement grounding. Values are averaged over 30 games per setting, with rates reported as percentages and other measures as raw means.
- Evaluation Setup: The evaluation covers 9 settings, each defined by the model playing the Geese and the model playing the Duck.Each setting’s values are means over 30 games.
- Evaluation Tiers: Table 4 reports Tier 1 outcome metrics, Table 5 reports Tier 2 behavior metrics, and Table 6 reports Tier 3 statement-grounding metrics.The three tables divide the complete metric set by evaluation tier.
- Metric Reporting: Rate-style metrics are percentages, while counts, durations, and ratios are reported as raw means.This reporting convention applies across the appendix’s complete metric set.
D Case Studies of Grounding Failures · E LLM Usage · Tier 1 — Game Outcomes
The case studies illustrate four grounding failures identified by the verification pipeline, while the paper reports limited LLM use and defines Tier 1 game-outcome metrics for evaluating agents.
- D Case Studies of Grounding Failures: Spatial hallucination occurs when a Goose claims to have seen a player alive after that player was already killed.The representative utterance describes Diana alive in Security and moving toward Weapons despite the ground truth contradicting the sighting.
- D Case Studies of Grounding Failures: Unsupported accusation occurs when a Goose reports seeing no one but names a suspect using only second-hand statements.The cited example explicitly says the agent saw neither Diana nor anyone else before the report, then reasons about Bob and Alice from timing.
- D Case Studies of Grounding Failures: Deception collapse occurs when a Duck fabricates a sighting for an alibi but claims to have seen a player who was already dead.The example claims Eve was alive in Security around tick 12 to support the Duck’s defense.
- D Case Studies of Grounding Failures: Language–action inconsistency occurs when a Duck claims to have completed a task in a room where the engine records no such task.The subject performed tasks in medbay and weapons, not in the claimed room.
- E LLM Usage: LLMs were used only to refine wording, verify grammar, improve clarity and readability, and accelerate code building.The paper states that LLMs did not assist with method design or results analysis.
- Tier 1 — Game Outcomes: Tier 1 records Goose win rate as the fraction of games won by the Geese and logs the winning team and ending condition.Ending conditions include completed tasks, all Ducks ejected, voting parity, or timeout.
- Tier 1 — Game Outcomes: Tier 1 also measures game duration, task completion rate, kills and meetings, ejection accuracy, and end-of-game survival counts.These metrics capture engine ticks, completed versus total Goose tasks, kill and meeting statistics, correct ejections, and players alive at termination.
Tier 2 — Behavioral Trajectories
Tier 2 evaluates agents through behavioral trajectories rather than game outcomes or statement grounding. Its metrics quantify voting, task execution, movement, kills, and post-kill displacement.
- Behavioral Trajectories: Tier 2 measures behavioral trajectories, complementing Tier 1 outcome metrics and Tier 3 language-grounding audits.The framework separates outcomes, behavior, and statement groundedness into three evaluation tiers.
- Behavioral Trajectories: Behavioral metrics include voting accuracy, skip rate, cooldown utilization, and task efficiency.Voting accuracy, skip rate, and cooldown utilization are percentages, while task efficiency is a raw mean.
- Behavioral Trajectories: Trajectory metrics also track distinct rooms visited per team, kills per game, and post-kill displacement.These quantities are reported as raw means.