PaperlayerView exact revision on arXiv

Source-linked AI summary

Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People

Gabriel Grand, Valerio Pepe, Jacob Andreas, Joshua B. Tenenbaum

arXiv:2510.20886v2cs.CLcs.AI

TL;DR

The paper asks whether language models can seek information strategically rather than only answer users’ queries. It pairs a collaborative dialogue-and-decision task with Bayesian-inspired inference-time strategies, reporting improved information-seeking and game outcomes, including wins over human and frontier-model opponents.

  • Problem

    It is unclear whether language models optimized to answer queries can instead ask useful questions and reason about uncertainty while seeking information in large hypothesis spaces.

    Source: §1 · p. 1

  • Method

    The paper evaluates agents in Collaborative Battleship and develops Bayesian-inspired inference-time strategies for question selection, move selection, and decisions between exploring and acting.

    Source: §1 · p. 2

  • Results

    Combined Bayesian strategies improve targeting and enable weaker LMs to win against humans and GPT-5, with similarly significant gains reported on Guess Who?.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that resource-rational strategies are important for agents to scale beyond benchmarks and collaborate with people on real-world problems.

    Source: §6.2 · p. 9

    The strategies rely on efficiently drawing conditional samples from a generative world model, which may need to be learned in settings beyond hand-implementable Battleship.

    Source: §6.1 · p. 9

Abstract

from arXiv · show

Many emerging applications of AI--from scientific discovery to medical diagnosis--require agents to seek information strategically: forming hypotheses, asking targeted questions, and making decisions under uncertainty. In high-stakes settings with limited resources, do language models (LMs) behave like rational agents? Drawing on insights from human cognition, we develop methods to evaluate and enhance agentic information-seeking. First, we introduce a decision-oriented dialogue task called Collaborative Battleship, in which a Captain must balance exploration (asking questions) and action (taking shots), while a Spotter must supply accurate, contextually-grounded answers. Compared to human players (N=42), we find that many LM agents struggle to ask informative questions, produce accurate answers, and identify high-utility actions. To address these gaps, we develop novel Monte Carlo inference strategies for LMs inspired by Bayesian Experimental Design (BED). For Spotter agents, our approach boosts accuracy by up to 14.7% absolute over LM-only baselines; for Captain agents, it raises expected information gain (EIG) by up to 0.227 bits (94.2% of the achievable noise ceiling). Combined, these components yield sharper targeting (+0.303-0.374 F1), and enable weaker LMs, such as Llama-4-Scout, to outperform both humans (8% -> 82% win rate) and frontier models (0% -> 67% win rate vs. GPT-5) at ~1% of GPT-5's cost. We replicate these findings on Guess Who?, where our methods significantly boost accuracy (+28.3-42.4 p.p.), demonstrating their general applicability for building information-seeking agents.

1 INTRODUCTION

The paper asks whether language models can ask goal-directed questions and act under uncertainty, and introduces a human-comparable dialogue task plus Bayesian-inspired strategies to evaluate and improve them. Its reported results span question answering, information gain, game performance, and transfer to Guess Who?.

1 INTRODUCTION

  • The authors ask whether LMs optimized to answer queries can instead ask useful questions and improve their information-seeking at inference time.They frame this question in applications that search large hypothesis spaces, including scientific experiments, theorem conjecture, and drug discovery.Source: §1 · p. 1
  • Collaborative Battleship makes Captains choose between questions and shots, while Spotters answer yes/no questions using full-board knowledge; questions can be evaluated as Python programs over sampled game states.This enables comparison of human and model behavior with idealized Bayesian strategies and measurement of expected information gain.Source: Figure 1 · p. 2
  • BATTLESHIPQA comprises 126 human-human game trajectories from 42 participants and supports grounded-answering and strategic-gameplay evaluations.SpotterQA contains 931 gold yes/no questions with expert answer annotations; CaptainQA tests play under limited questions and shots.Source: §1 · p. 2
  • Python code generation improves SpotterQA accuracy by 14.7% over direct-answering and chain-of-thought baselines, while Bayesian sampling raises CaptainQA question EIG by up to 0.227 bits.The EIG gain is reported as 94.2% of the information-theoretic ceiling.Source: §1 · p. 2
  • Combined Bayes-QMD strategies improve targeting and let weaker LMs outperform humans and GPT-5, while the approach also improves Guess Who? accuracy.Reported targeting gains include +0.397 F1 for Llama-4-Scout and +0.332 F1 for GPT-4o; Guess Who? gains are +42.4 p.p. and +28.3 p.p., respectively.Source: §1 · p. 2
  • The paper contributes a reusable evaluation harness, a multimodal dataset, and Bayesian-inspired inference-time strategies intended for information-seeking in other discovery settings.The introduction also reports that the methods eliminate redundant questions with zero information gain.Source: §1 · p. 2

2 THE BATTLESHIP GAME

Collaborative Battleship adapts Battleship into a two-player, multi-turn dialogue task. The Captain explores by asking questions or exploits information by shooting; the Spotter sees the full board and answers questions.

2 THE BATTLESHIP GAME

  • The partially informed Captain alternates between revealing a tile and asking a question, while the fully informed Spotter must answer questions in context.The task adapts a paradigm previously used to study human information-seeking to both human and language-model agents.Source: §2 · p. 3
  • The game extends earlier Battleship paradigms from static snapshots to full multi-turn trajectories, adding a Spotter to answer the Captain’s questions.Questions are represented as Python programs, and Spotter answers are restricted to yes or no to preserve strategic balance.Source: §2 · p. 3

3 FORMAL FRAMEWORK: BAYESIAN EXPERIMENTAL DESIGN

The framework represents uncertainty over feasible hidden boards, models questions as binary functions, and scores questions by expected information gain under a noisy-answer model. It then uses the current belief to choose questions, shots, and whether to explore or act.

3 FORMAL FRAMEWORK: BAYESIAN EXPERIMENTAL DESIGN

  • The framework defines the hidden board as a random variable, conditions feasible boards on the observed partial board, and represents the Captain’s belief over boards given the dialogue history.A question maps a board deterministically to a yes/no answer.Source: §3 · p. 3
  • Because Spotters or question-to-code translation may err, the observation model treats answers as a binary symmetric channel with flip probability ε.The model allows errors from misreading the board, misinterpreting the question, or miscommunicating the answer.Source: §3 · p. 3
  • After an observed answer, the Captain updates board beliefs by reweighting each feasible board according to how likely that answer is under the noise model.Boards consistent with the answer gain weight, while alternatives retain some weight when ε > 0.Source: §3 · pp. 3–4

B SEQUENTIAL MONTE CARLO APPROXIMATION

  • Sequential Monte Carlo approximates the belief with weighted particles because exact sums over feasible boards are typically intractable.The particle approximation is updated via sequential Monte Carlo with per-turn resampling.Source: §3 · p. 4

3 FORMAL FRAMEWORK: BAYESIAN EXPERIMENTAL DESIGN

  • Expected information gain measures the board uncertainty expected to be removed by a question, subtracting channel uncertainty from the expected noisy-answer uncertainty.The framework gives a closed form for a binary symmetric channel in terms of binary entropy; information gain is maximized when the predicted answer is about equally likely to be yes or no.Source: §3 · p. 4

3.1 BAYESIAN STRATEGIES FOR EXPLORATION AND ACTION

  • Bayes-QMD combines question selection, move selection, and a decision strategy that compares asking with shooting using discounted one-step lookahead.These strategies maximize expected utility under the current belief and are described as resource-rational, not necessarily globally optimal.Source: §3.1 · pp. 4–5
  • Bayes-Q samples candidate questions, for example from an LM, and selects the question with the highest expected information gain.The question-scoring approach evaluates expected information from the Spotter’s noisy answer.Source: §3.1 · p. 4, §3 · p. 4
  • Bayes-M selects the unrevealed tile with the highest current-belief hit probability; Bayes-D uses one-step lookahead to choose whether to ask or shoot.With γ = 1, the policy asks only when expected one-step improvement in next-turn MAP hit probability exceeds the current MAP chance; smaller γ favors acting sooner.Source: §3.1 · pp. 4–5

4 EXPERIMENTS

The experiments combine a human behavioral study with SpotterQA grounded-answering tests and CaptainQA full-game tests. They assess human information-seeking and evaluate language models’ question-answering and strategic play.

4.1 EVALUATING HUMAN INFORMATION-SEEKING BEHAVIOR

  • The human study recruited 42 participants; 931 agreed-upon gold answers set a 92.5% baseline, with questions classified as simple or history-dependent complex.The study collected data from 18 pre-sampled 8×8 boards; question categories include discourse-dependence, state-dependence, vagueness, and ambiguity.Source: §4.1 · p. 5

4.2.1 RESULTS

  • SpotterQA accuracy ranges from 52.5% for GPT-4o-mini to 92.8% for o3 mini; o3, o4-mini, and GPT-5 meet or exceed the 92.5% human mean.Figure 3 reports accuracy across 15 language models and marks the mean human accuracy.Source: Figure 3 · p. 6, §4.2.1 · p. 6

4.2 SPOTTERQA: MODELING QUESTION-ANSWERING

  • SpotterQA evaluates grounded answers using question, answer, observed-board, and game-history inputs, comparing direct answering with CoT, code translation, and combined CoT + Code.The evaluation considers a broad, non-exhaustive set of frontier language models.Source: §4.2 · pp. 5–6

4.2.1 RESULTS

  • CoT + Code yields 14.7% absolute accuracy-point gains over Base across models; GPT-4.1 rises from 75.2% to 90.9%.Code alone improves accuracy by 13.2 percentage points over Base; the reported differences are significant (p < 0.001).Source: §4.2.1 · p. 6, Figure 3 · p. 6
  • Human accuracy remains roughly consistent on simple and complex questions, while GPT-4o drops from 72.8% to 60.4% and the best model, o3, remains below humans on complex questions.On complex questions, o3 reaches 87.4% versus 91.9% for humans; code generation partially mitigates the gap.Source: §4.2.1 · p. 6, Figure 3 · p. 6

4.3.1 RESULTS

  • With +Bayes-QMD, Llama-4-Scout rises from 0.367 to 0.764 F1 and reaches 0.82–0.83 win rate against humans and 0.67 against GPT-5.Both Llama-4-Scout and GPT-4o outperform humans and GPT-5 with the full Bayesian model; Llama-4-Scout costs approximately 1% of GPT-5.Source: §4.3.1 · p. 7, Figure 4 · p. 8
  • QBayes raises EIG by up to 0.227 bits per question, but Llama-4-Scout and GPT-4o gain only +0.021–0.026 F1 in targeting.MBayes explicitly marginalizes over the implications of each question when computing πt, offering a more reliable mechanism for translating information into moves.Source: §4.3.1 · p. 8
  • DBayes mitigates weaker players’ tendency to front-load all 15 questions; humans and GPT-5 ask fewer overall (8.0–8.2 vs. 14.1–14.9).Humans and GPT-5 tend to ask several questions early but reserve some for later turns.Source: §4.3.1 · p. 8, Figure 4 · p. 8

5 GENERALIZED INFORMATION-SEEKING GAMES

The paper tests its Bayesian strategies in Guess Who?, a game with richer object-relational semantics and more complex entity and attribute reasoning than Battleship. Both evaluated models improve with QMBayes, supporting generalization to information-seeking games with combinatorial hypothesis spaces.

5 GENERALIZED INFORMATION-SEEKING GAMES

  • GPT-4o success rose from 0.617 to 0.900 and Llama-4-Scout success from 0.300 to 0.724 with QMBayes.The figure caption characterizes the improvements as consistent and as replicating the core Battleship findings.Source: §5 · p. 8, Figure 5 · p. 8
  • Guess Who? provides a distinct testbed with richer object-relational semantics and more complex reasoning about entities and attributes than Battleship.Source: §5 · p. 8
  • The results suggest the framework generalizes to information-seeking environments with combinatorial hypothesis spaces.Source: §5 · p. 8

6 RELATED WORK, DISCUSSION, AND CONCLUSION

The discussion situates the work among research on language-to-code inference, information-seeking, and resource-rational cognition, while identifying limits in pragmatic reasoning and world-model sampling. The conclusion reports gaps in smaller models and costly human-level-or-better performance from larger reasoning models, and argues for resource-rational strategies in real-world collaboration.

6 RELATED WORK, DISCUSSION, AND CONCLUSION

  • The work connects language-to-code methods for probabilistic inference with recent approaches to planning and rational agent behavior.Source: §6 · p. 9
  • Prior information-seeking research spans structured prompting, entropy reduction, Bayesian inference, and constraint satisfaction approaches to eliciting preferences and clarifying ambiguity.Source: §6 · p. 9
  • The modeling approach draws on resource-rational accounts and human tendencies to use greedy heuristics, consider few hypotheses, and prefer interpretable queries.Source: §6 · p. 9

6.1 LIMITATIONS AND FUTURE WORK

  • The task’s rich pragmatic behaviors are not explicitly modeled, and fixed ε could miss reliability differences across individual Spotters.The authors suggest inferring ε to account for Spotter reliability and note that agents may need to adapt to different levels of aleatoric uncertainty in scientific settings.Source: §6.1 · p. 9
  • The strategies depend on efficiently drawing conditional samples from a generative world model, which may need to be learned for code- or image-represented states beyond Battleship.Source: §6.1 · p. 9

6.2 CONCLUSION

  • Smaller models such as Llama-4-Scout and GPT-4o struggle to explore and act coherently, while GPT-5 approaches or exceeds human-level performance at steep resource costs.The authors argue that resource-rational strategies are essential for agents intended to scale beyond benchmarks and collaborate with people on real-world problems.Source: §6.2 · p. 9

ETHICS STATEMENT

The paper reports ethical review and informed consent for its human-participant study, and states that its code, data, and BATTLESHIPQA dataset are available. Its appendix describes the Collaborative Battleship task and its extensions to earlier Battleship paradigms.

ETHICS STATEMENT

  • The human-participant protocol received MIT COUHES review and approval, participants gave informed consent, and their privacy was protected.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

USE OF AI ASSISTANCE

  • The authors used language models for manuscript editing, literature review, figure preparation, and coding assistance, and report reviewing and editing the final content themselves.Source: USE OF AI ASSISTANCE · p. 10

A.1 EXTENDED GAME DESCRIPTION AND PRIOR WORK

  • Collaborative Battleship adapts prior Battleship-like human information-seeking tasks into a two-player game where a partially informed Captain navigates exploration and action with a Spotter.Source: §A.1 · p. 20
  • The task extends prior work with full multi-turn games, a Spotter for real-time answers, Python-program questions, and larger 8×8 boards.Source: §A.1 · p. 20
  • The Spotter is restricted to yes-or-no answers, creating an enforceable information bottleneck rather than allowing open-ended replies.The authors say this restriction is important when participants receive monetary incentives and may form ad-hoc conventions across rounds.Source: §A.1 · p. 20

A.2.1 STUDY SETUP

The human study recruited 42 participants for synchronous, two-player sessions and used a balanced incomplete-block design to allocate Battleship boards. Participants received instructions on collaborative play, thoughtful questions, and study procedures.

A.2.1 STUDY SETUP

  • The study recruited 42 U.S.-based, English-fluent Prolific participants with at least a 95% approval rate and ran synchronous two-player sessions from November 2024 to January 2025.Participants used desktop or laptop computers; mobile devices were not supported.Source: §A.2.1 · p. 21

A.2.2 STIMULUS AND BLOCK DESIGN

  • The experiment sampled 18 boards, each an 8×8 grid with four ships, and allocated six boards to each of 21 pairs so every board appeared in seven pairs.Within pairs, players alternated Captain and Spotter roles across rounds and board order was randomized.Source: §A.2.2 · p. 21

A.2.3 PARTICIPANT INSTRUCTIONS

  • Participants were instructed to work together to reveal all ships efficiently, ask thoughtful questions, and avoid low-quality or copy-pasted questions and move spamming.Bonuses depended on team performance, with fewer moves yielding a higher bonus.Source: §A.2.3 · p. 22
  • The study instructions required timely participation and attention checks, and described how to end the study, receive payment, handle matchmaking delays, and report technical issues.Source: §A.2.3 · pp. 22–23

A.3 BATTLESHIPQA DATASET

The appendix documents BATTLESHIPQA’s expert-labeled human-question dataset, illustrates its pragmatic categories, and specifies the game prompts and inference procedures used in the experiments.

A.3 BATTLESHIPQA DATASET

  • 931 questions with unanimous annotator agreement formed the gold set, yielding a 92.5% human accuracy baseline and a taxonomy of simple, stateful, discourse, vague, ambiguous, and unanswerable questions.Unanswerable questions were removed; reported inter-annotator agreement ranged from 94.0–99.6%.Source: §A.3 · p. 23

A.4 ILLUSTRATED EXAMPLES

  • Examples distinguish board-state-only questions from questions requiring game history or prior dialogue, and illustrate vague, ambiguous, and non-binary unanswerable questions.For instance, “it” depends on the earlier discourse, while “where are the ships?” cannot be answered yes or no.Source: Figure 8 · p. 25, Figure 9 · p. 25, Figure 10 · p. 26
  • Other examples show a Spotter’s contextually helpful false answer, players monitoring perceived reliability, and Captain side comments that build rapport.The examples characterize benevolent lying, epistemic vigilance, and phatic communication, respectively.Source: Figure 14 · p. 27, Figure 15 · p. 27, Figure 16 · p. 28

B SEQUENTIAL MONTE CARLO APPROXIMATION

  • Because exact inference over the large board space is intractable for quantities such as question probabilities and EIG, the method maintains a sequential Monte Carlo particle approximation to the posterior.The particles represent possible board configurations and carry weights updated as observations arrive.Source: B SEQUENTIAL MONTE CARLO APPROXIMATION · p. 29

C.1.1 GAME SETUP PROMPT

  • In this Battleship variant, teams collaborate: the Captain asks the Spotter questions about the board or guesses a tile, while the Spotter sees the full board and answers only yes or no.The board is an 8×8 grid with four colored ships, each 2–5 tiles long.Source: §C.1.1 · p. 30

C.2.1 TASK PROMPTS

  • Captain task prompts request a choice between questioning and moving, a likely ship tile, or an information-seeking yes/no question, with remaining questions, shots, and ship status supplied in the full template.The template also offers a step-by-step reasoning instruction.Source: §C.2.1 · p. 31, §C.2.2 · p. 31

C.3.1 VARIANTS

  • Spotter code prompts require a generic Python function over true and partial boards, so answers can reflect which ship tiles remain unrevealed and the game context.The function returns a Boolean interpreted as yes or no.Source: §C.3.1 · p. 32

C.3.2 FULL TEMPLATE

  • The Spotter template combines the partial board, hidden full board, role objective, question, and selected answering and reasoning variants.The prompt supplies the Captain’s question alongside the board information.Source: §C.3.2 · pp. 32–33

D.1 SPOTTERQA EVALUATIONS

The appendix presents SpotterQA’s accuracy breakdown by question type and example Python programs for answering questions from true and partial boards.

D.1 SPOTTERQA EVALUATIONS

  • Table 3 reports SpotterQA accuracy by question type and valid-format response rates; all models except Gemini-2.5-Flash readily followed the answer format.The caption identifies frequent invalid answers as the exception for Gemini-2.5-Flash.Source: Table 3 · p. 34

D.1.1 EXAMPLE SPOTTER PROGRAMS

  • Example programs operationalize questions as Python functions over true and partial boards, including checks for unrevealed ship tiles and ship orientation.The examples cover simple, stateful, discourse-dependent, vague, and ambiguous questions.Source: §D.1.1 · pp. 35–36
  • For vague “close” and ambiguous row questions, the example code makes an explicit interpretation, such as defining closeness as four-neighbor adjacency.The row example interprets “after 5” as columns 6–8 and row G as the whole row.Source: §D.1.1 · p. 36

D.2 CAPTAINQA EVALUATIONS

The appendix describes CaptainQA’s reported metrics and visualizations, then outlines a constrained Guess Who? adaptation and its corresponding strategy choices.

D.2 CAPTAINQA EVALUATIONS

  • Table 4 reports CaptainQA targeting F1, moves, questions, EIG, and redundant-question rate; its caption gives an EIG ceiling of approximately 0.531.Figure 18 defines each heatmap cell as the row Captain’s win rate over the column Captain; 0.50 is balanced.Source: Table 4 · p. 39, Figure 18 · p. 39
  • The question-embedding visualization shows human and model Captains in overlapping but systematically distinct regions, with points representing questions and colors identifying LMs.The map uses t-SNE coordinates from PCA-reduced embeddings.Source: Figure 19 · p. 40

E.1 THE Guess Who? GAME

  • Guess Who? uses a one-player, one-guess-loss setting; to discourage trivial enumeration, the experiment expands the character set to 100 and limits questions to eight.The original environment has 24 characters and a 40-question maximum.Source: §E.1 · p. 43

E.3 MODELING QA BEHAVIOR

  • With only one guess, the Guess Who? decision function has the player ask available questions before guessing, and the Bayes-QMD condition is removed because the setting lacks rational decision-making ability to model.The appendix notes that Random and Greedy baselines would guess immediately without prior information.Source: §E.3 · p. 44

E.4 EXPERIMENTAL DETAILS

The Guess Who? experiments evaluate GPT-4o and Llama-4-Scout as players against GPT-5 as gamemaster, running each strategy for 60 games.

E.4 EXPERIMENTAL DETAILS

  • GPT-4o and Llama-4-Scout played against GPT-5 as gamemaster, and each strategy was run for 60 games using the OpenRouter API with default parameters.GPT-5 was not evaluated as a player model for cost reasons.Source: §E.4 · p. 44

E.5 RESULTS

In Guess Who?, Bayesian strategies substantially improve success for both tested models, and EIG-oriented conditions increase question informativeness. The results point to generality across two information-seeking domains.

E.5 RESULTS

  • 0.617→0.900 for GPT-4o and 0.300→0.724 for Llama-4-Scout in Guess Who? success rate; Bayes-QM lifts weak LMs past stronger pure-LM agents.Source: §E.5 · pp. 44–45, Table 9 · p. 45
  • Bayes-Q and Bayes-M separately improve over both base conditions, suggesting base LMs neither optimally use prior answers nor reliably ask the optimal question.Source: §E.5 · p. 45
  • EIG rises from 0.454 to 0.506/0.512 for GPT-4o and 0.413 to 0.478/0.478 for Llama-4-Scout under Bayes-Q/Bayes-QM.Source: §E.5 · p. 45
  • EIG-optimizing conditions eliminate redundant questions (EIG = 0); GPT-4o had none in any condition, while Llama-4-Scout had 0.6% only at baseline.Source: §E.5 · p. 45

E.6.1 SYSTEM PROMPT

  • Guess Who? gives players eight questions, followed by one guess that determines whether they win or lose.Source: §E.6.1 · p. 45

E.6.2 PLAYER PROMPTS

  • Player prompts request information-seeking yes/no questions, discourage repeats, and include single-question and batched-question formats.Source: §E.6.2 · p. 46

E.6.3 GAME/SAMPLER PROMPTS

  • The sampler identifies characters consistent with the latest question and returns a JSON mapping of characters to yes/no answers.Source: §E.6.3 · pp. 46–47

Related papers

Loading 2510.20886v2…