PaperlayerView exact revision on arXiv

Source-linked AI summary

Clueing up LLMs with Tool-Augmented Deductive Reasoning

Rebecca Ansell, Autumn Toney-Wails

arXiv:2609.18736v1cs.AI

TL;DR

Extended interactions make it difficult for LLMs to maintain consistent deductive beliefs. The paper tests a structured possibility matrix in multi-agent Clue games, finding near-perfect accuracy for tool-augmented agents and improved accuracy for baseline agents in mixed games.

  • Problem

    LLMs often underperform in game environments that require extended memory and multi-step reasoning.

    Source: §1 · p. 1

  • Method

    The paper evaluates a possibility matrix that externalizes evolving game state and deductive constraints for six LLM players in Clue.

    Source: §1 · p. 2

  • Results

    Tool-augmented agents achieve near-perfect accusation accuracy, while baseline agents achieve higher accusation accuracy in mixed games than in baseline games.

    Source: §5 · p. 6

  • Takeaways & Limitations

    Structured belief tracking can support sustained deductive inference, though the implemented tool does not necessarily directly affect agents’ decision-making autonomy.

    Source: §5 · p. 7

    The possibility matrix is designed for Clue’s deductive structure, and its generalizability to other reasoning domains remains for future work.

    Source: §6 · p. 7

Abstract

from arXiv · show

Despite recent advances in large language models (LLMs), performing logically consistent deductive reasoning over extended interactions remains challenging. Tasks that require integrating evidence across multiple reasoning steps, maintaining consistency with prior inferences, and updating beliefs under new constraints can surface limitations in current models while providing a useful testbed for evaluating reasoning enhancements. In this paper, we implement a text-based, multi-agent version of the classic board game Clue as an environment to evaluate multi-step, agentic deductive reasoning. In this setting, agents must infer hidden information from a sequence of observations, maintain consistency across turns, and reason over an evolving set of logical constraints. We instantiate six LLM-based agents (GPT-4o-mini and Gemini-2.5-Flash) as players that engage in turn-based gameplay; using three agents per model family, we establish baseline performance across repeated games. We then introduce a tool-augmented approach in which a structured possibility matrix converts implicit game state from generated reasoning logs into an explicit representation of remaining possibilities. The possibility matrix encodes extended-turn memory and deductive constraints, offloading these tasks from the agent. We compare this approach against the baseline to evaluate how tool augmentation supports reasoning quality and task success for autonomous agents in a strategic reasoning environment.

1 Introduction

The paper evaluates whether a structured possibility matrix can support deductive reasoning in interactive Clue games, where agents must track evidence across turns. Tool-augmented agents achieve near-perfect gameplay, and tool use also benefits baseline agents in mixed games.

1 Introduction

  • LLMs often struggle with extended-memory and multi-step reasoning in games, motivating techniques that support agentic reasoning.The paper situates game analysis as a way to examine decision-making, planning, and interaction dynamics.Source: §1 · p. 1
  • The possibility matrix externalizes belief tracking and deductive constraints, updating the prompt with an explicit representation of the evolving game state.The approach is also applicable to LLMs without native tool-use functionality.Source: §1 · pp. 1–2
  • The evaluation uses six GPT-4o-mini and Gemini-2.5-Flash players across repeated baseline, tool-augmented, and mixed Clue games.The study poses questions about gameplay performance, effects on non-tool agents, and decision strategies.Source: §1 · p. 2
  • Tool augmentation yields near-perfect gameplay, improves non-tool agents’ performance in mixed games, and reduces accusation delay without lowering accuracy.The introduction reports these overall findings without specifying numerical values.Source: §1 · p. 2

2 Related Work

The related work identifies a gap in evaluating coherent deductive reasoning across interdependent inferences and in game environments that isolate deduction. It positions structured belief tracking as a way to study this challenge without requiring agents to learn effective tool invocation.

2.1 Deductive Reasoning in LLMs

  • Long reasoning chains can contain logical inconsistencies, while isolated-event benchmarks cannot assess coherent beliefs across interdependent inferences.The cited analyses also report that correct answers can arise from unfaithful intermediate steps.Source: §2.1 · p. 2

2.2 Games as Deductive Reasoning Testbeds

  • Many game environments test persuasion, intent modeling, or reasoning under uncertainty rather than deduction; prior text-based Clue work still found models struggled (Ansell and Toney [2026]).The paper frames extended-memory deductive reasoning as an open challenge in games.Source: §2.2 · p. 2

2.3 Tool-Augmented Reasoning

  • The proposed approach injects a structured belief state into gameplay prompts, offloading extended-memory tracking without requiring agents to learn when or how to call tools.Source: §2.3 · p. 2

3 Experimental Design

The study uses a text-based, six-player Clue environment with partially observed card reveals and two LLM families. Its possibility matrix tracks card beliefs and constraints throughout play, while repeated game versions and four metrics support evaluation.

3.1 Clue Game Environment

  • Players make public suggestions but receive at most one private card reveal per turn, requiring inference from partial information.The six-player game uses 21 cards, with one suspect, weapon, and room hidden in the solution envelope and three cards dealt to each player.Source: §3.1 · pp. 2–3

3.2 LLM Agents

  • GPT-4o-mini and Gemini-2.5-Flash players receive identical prompt structures and configurations, without distinct personas or role-specific behaviors.Source: §3.2 · p. 3

3.3 Possibility Matrix Tool

  • The possibility matrix records YES, NO, or MAYBE for each card-holder pairing and updates beliefs from private reveals, suggestions, and disproof behavior.It also flags when only one envelope candidate remains in each card category, indicating readiness for an accusation.Source: §3.3 · p. 3
  • At each turn, the tool injects the updated matrix into the agent prompt and revises it after turn actions and card reveals.The figure compares baseline and tool-augmented player turns.Source: §3.3 · p. 3, Figure 2 · p. 4

3.4 Game Implementation

  • The experiment runs six games each for baseline, matrix, and mixed versions, and evaluates accusation accuracy, deduction quality, knowledge accumulation, and final accusation delay.In mixed games, each model family has one tool user and two baseline players; player order is shuffled.Source: §3.4 · p. 4

4 Results

MATRIX players identified the hidden solution more accurately than BASELINE players, despite accumulating knowledge more slowly. Results also show delays between solution confirmation and final accusation.

4.1 Accusation Accuracy

  • 94.5% of MATRIX players identified the correct solution, while BASELINE players never reached it; 47% of BASELINE players identified one card and 36% identified none.Gemini-2.5-Flash achieved perfect accuracy once the matrix tool was introduced.Source: §4.1 · p. 4, Figure 3 · p. 5

4.2 Deduction Quality

  • MATRIX reduced incorrect deductions to 0.1 per game for Gemini-2.5-Flash and 1.7 for GPT-4o-mini, versus 2.4 for each model in BASELINE.In MIXED, baseline-prompted players' correct deduction rates were slightly below BASELINE; matrix-prompted players had profiles similar to MATRIX, with GPT-4o-mini slightly worse and Gemini Flash slightly better.Source: §4.2 · pp. 4–5

4.3 Knowledge Accumulation

  • Tool-augmented players accumulated knowledge more slowly per round but used it to make correct accusations more efficiently; BASELINE players relied on longer games to build a near-complete picture.BASELINE approached the knowledge ceiling by rounds 17–18, while MATRIX games ended by round 11 with curves around 14 cards; MATRIX players solved more accurately despite learning fewer cards overall.Source: §4.3 · p. 5

4.4 Solution Suggested to Final Accusation

  • In MATRIX, confirmed suggestions occurred on turns 18–23, with correct accusations 7–19 turns later; only 2 of 6 games were won by the first suggester.In MIXED, confirmation occurred on turns 13–35, with accusations 7–31 turns later, and the first suggester won 3 of 6 games. In MATRIX, 33 of 36 players received accusation-ready status before game end, yet mean gaps to accusation were 11.1 turns for GPT-4o-mini and 8.2 for Gemini-2.5-Flash.Source: §4.4 · pp. 5–6

5 Discussion

Tool-augmented agents made near-perfect accusations with less accumulated knowledge than baseline agents, while mixed games also improved baseline players’ accuracy. The discussion links structured belief tracking to fewer unsupported deductions but notes that it does not directly control agents’ decision timing.

5 Discussion

  • Tool-augmented games achieved near-perfect accusation accuracy despite accumulating substantially less knowledge, while baseline agents approached the knowledge ceiling but failed to convert evidence into correct accusations.Figure 5 tracks cards known per round against an 18-card ceiling.Source: §5 · p. 6, Figure 5 · p. 7
  • Gemini-2.5-Flash reduced incorrect deductions to near zero and achieved perfect accusation accuracy, while GPT-4o-mini retained similar incorrect-deduction rates across conditions.The authors suggest Gemini treats the matrix as a hard constraint; both models had comparable win counts in MATRIX games.Source: §5 · p. 6
  • Baseline-prompted players had higher accusation accuracy in MIXED than BASELINE games, possibly because tool-augmented players supplied more informative suggestions to all players.The authors describe this as a possible game-environment effect, with deduction profiles otherwise fairly similar.Source: §5 · p. 6
  • In MIXED games, a minority of tool users brought group accusation accuracy close to fully tool-augmented play, but every win went to GPT-4o-mini players.The authors suggest willingness to accuse may influence wins more than matrix use.Source: §5 · p. 6
  • Figure 6 tracks the gap between first undisproved solution suggestions and correct accusations; agents retained decision autonomy, and changing accusation timing would likely require prompt intervention.Agents were not required to accuse when they received the accuracy signal.Source: §5 · pp. 6–7, Figure 6 · p. 7
  • The authors conclude that offloading extended-memory reasoning and belief tracking improves sustained deductive inference, while tool augmentation does not necessarily change agents’ decision-making autonomy.They also report that the possibility matrix reduced unsupported inferences characteristic of baseline reasoning.Source: §5 · p. 7

6 Limitations and Future Work

The authors identify limits in the possibility matrix’s task-specific design, the small evaluation scale, and the baseline comparison. These constrain claims about generality and make it difficult to isolate which matrix features account for the improvements.

6 Limitations and Future Work

  • The possibility matrix is designed for Clue’s deductive structure, so its generality to other reasoning domains and differently structured games remains untested.It encodes card-ownership constraints, maintains beliefs, and signals when one solution remains.Source: §6 · p. 7
  • The evaluation covers two model families and 18 games, with six games per condition, making the results preliminary for a stochastic multi-agent setting.Larger evaluations across additional models and simulations, including larger reasoning-oriented models, are proposed to assess robustness and generality.Source: §6 · pp. 7–8
  • Because the baseline uses natural-language histories, the comparison cannot isolate whether improvements come from shorter context, explicit constraints, better memory retention, or their combination.The authors suggest comparing summarized histories, model-generated tables, and alternative memory-augmentation strategies.Source: §6 · p. 8

7 Conclusion

The conclusion evaluates belief-state offloading in Clue, a dynamic multi-agent setting, and reports substantial gameplay gains, including near-perfect accuracy for tool-augmented agents and benefits in mixed settings.

7 Conclusion

  • The evaluation targets extended-memory deductive reasoning in Clue, where outcomes depend on player interactions, strategies, and information exchange rather than fixed-answer question answering.Source: §7 · p. 8
  • Tool augmentation substantially improves gameplay performance, with near-perfect accuracy for tool-augmented agents and benefits for baseline agents in mixed settings.Source: §7 · p. 8
  • Succinct, structured belief states over natural-language reasoning logs are associated with lightweight agents performing near-optimally in complex interactive settings.Source: §7 · p. 8

Related papers

Loading 2609.18736v1…