Source-linked AI summary
Confident at the moment of action: belief miscalibration in LLM play under hidden information
Bhushan Kashinath Joshi
TL;DR
Agentic systems rely on stated confidence to decide when to act, but it is unclear whether that confidence tracks correctness at the moment of action under hidden information. The paper tests this with a hidden-information chess variant that separately elicits beliefs and moves and scores beliefs against recoverable ground truth, finding extreme high-confidence capture errors that outcome-only evaluation can miss.
Problem
Whether a model’s stated confidence tracks correctness at the moment of acting under hidden information is rarely tested directly because standard evaluations do not separate belief elicitation from action.
Method
The paper elicits a move and a probability distribution over the opponent’s hidden royal piece every turn in a chess variant with secretly relocatable royal status, then scores beliefs against logged ground truth.
Results
1 of 62 captures made at high stated confidence about the hidden royal piece were correct across two independent batches, with the pattern scoped across additional configurations and providers.
Takeaways & Limitations
Separately measuring belief and action can reveal severe confidence–correctness failures that outcome-only game evaluation misses, even when the model wins the game.
Takeaways & Limitations
Only S1 received the complete calibration battery, while the other four seats support only a narrow capture-time high-confidence hit-rate comparison.
Abstract
from arXiv · showhide
Agentic systems increasingly gate actions on a model's own stated confidence, which assumes confidence tracks correctness at the moment of acting. We test this in a hidden-information chess variant where royal status can be secretly, repeatedly relocated between pieces, and where an agent's stated probability distribution over the opponent's hidden royal piece -- elicited every turn, separately from the move it chooses -- is scored against ground truth recoverable after the game. Across two independent batches, captures made at high stated confidence ($\geq 0.5$) about the hidden piece's location were correct in 1 of 62 cases. The calibration deficit is concentrated almost entirely in these events: 99.3% of it in the original batch, 98.7% in the replication. The same pattern, in weaker form, orders consistently (point estimates only; most pairwise gaps are not statistically distinguishable at this sample size) across four further model configurations spanning a second provider -- reported as scope for the finding, not as evidence that capability predicts calibration: a same-model comparison at a fixed external leaderboard score shows a deliberation-budget change alone moves the metric by nearly as much as a large cross-model gap. In a separate seat, conventional evaluation axes -- legality, cost, latency, completion rate -- can dissociate entirely from belief quality, with the configuration winning on every conventional axis producing the worst belief quality tested. A model exhibiting this pattern can still win the game its belief was about, which is why outcome-only evaluation would not detect it.
1. Introduction
The paper addresses whether stated confidence tracks correctness at the moment of action under hidden information. It introduces a chess setting that separately elicits beliefs and moves, finding severe high-confidence capture errors that outcome-only evaluation can miss.
- Motivation: Standard evaluations collapse belief and action into one scored output, preventing direct tests of whether a stated belief guides a correct action.QA scores an answer with its stated probability, while competitive games usually score actions or outcomes without eliciting hidden-state beliefs.
- Contribution: The study builds a chess variant with secretly and repeatedly relocated royal status, eliciting a move and a probability distribution over the opponent’s hidden royal piece every turn.The true hidden state remains logged and is recoverable after the game, allowing belief scoring against fact and comparison with the same-turn move.
- Finding: 1 of 62 high-confidence captures about the hidden royal piece were correct across two independent batches.The calibration deficit was concentrated almost entirely in these high-confidence events rather than distributed evenly across beliefs.
- Implication: A model can still win the game its belief concerned, so outcome-only evaluation does not reveal belief–action misalignment.The paper therefore measures stated belief separately from the action taken on it.
- Scope: The scope analysis spans five seat-configurations across two model providers, while the paper does not claim the pattern is universal.The study also examines whether conventional evaluation axes can dissociate from belief quality.
2. Related Work
Related work uses games to resist contamination and saturation and studies calibration, hidden information, deception, and theory of mind. This paper distinguishes its instrument by scoring beliefs about a logged, relocatable hidden state separately from in-game actions.
- Game-based evaluation: Competitive-game evaluations address contamination and benchmark saturation by testing performance in games rather than static answer settings.Kaggle Game Arena expanded from chess to poker and Werewolf alongside other games, with game-specific leaderboards and metrics.
- Distinction: The paper’s target is belief quality against ground truth, adding a measurement layer beyond game outcomes or self-reported honesty.This layer is designed to separate what an agent believes from what it does.
- Calibration and uncertainty: Prior calibration work studies whether verbalized confidence tracks answer correctness and shows sensitivity to elicitation protocol, unfamiliar tasks, and suggestibility.Other approaches fine-tune confidence with Brier-score loss or combine normalized confidence with self-consistency signals.
- Distinction: This work elicits a structured move and hidden-state belief inside an evolving game, whereas related targets concern future events or static facts.Its free-control result also cautions that isolated-position elicitation can overstate in-game performance by 28.4 points on one seat and 41.2 on another.
- Hidden information: Hidden-information benchmarks probe opponent modeling, deception, and detection, including settings where honesty is judged from speakers’ self-reports.The paper contrasts these with its model-independent, internally logged ground truth.
3. Setting and Method
The study uses Regent Chess, a hidden-information variant with logged Crown Shifts, structured belief elicitation, and separate capture-only and every-ply scoring. A fixed source-visible opponent supports controlled comparisons and tractability analysis.
- Setting: Regent Chess makes royal status transferable between pieces, and winning requires capturing the opponent’s current Regent rather than necessarily the Original King.Crown Shift is the secret action that transfers royal status.
- Setting: Every Crown Shift is logged invisibly and replayable, making the true Regent recoverable for any ply after the game.This enables scoring stated beliefs against an exact fact rather than a proxy.
- Elicitation: Each turn produces a chosen move plus a top-k probability distribution over which live opponent piece is the Regent, including residual probability for other pieces.The agent receives its own known state but never the opponent’s hidden state; top-k coverage remains a limitation of the format.
- Opponent: The fixed deterministic heuristic opponent makes seat comparisons and depth curves meaningful and supports the oracle-versus-public tractability bound.Its source-visible policy allows public predictors to be compared with an oracle having full internal game-state access.
- Scoring: Capture-only scoring pairs probability with Regent status when the model captures a square, whereas every-ply scoring evaluates every named candidate on every well-formed ply.Both are compared with a per-ply population-matched uniform prior; capture-only is exposure-biased, while every-ply supplies more events.
- Assumption: The first four turns provide a rule-based advantage because the Original King remains the Regent until each side’s fourth turn.This structurally unfair window is excluded from inference-sensitive interpretation.
- Method: The study preregistered four expectations with stated falsifiers and scored them verbatim rather than adapting them to the results.The expectations covered initial calibration, illegal moves, belief–action gaps, and tracking versus narrative quality.
- Scope: Only S1 received the complete calibration battery; the other four seats support only the narrow capture-time high-confidence hit-rate comparison.Five seat-configurations were scored for capture-time calibration, while S1 alone received capture-only, every-ply, and ground-truth-scored belief-quality evaluation.
4. Results
Across replicated batches and broader configurations, belief quality fails at action time and is not captured by conventional performance metrics or static screening. The degradation also worsens with game depth, with a measurable residual specific to hidden-state tracking.
- 4.1 Primary finding: 0 of 22 original and 1 of 40 replication high-confidence captures were correct; pooled, only 1 of 62 was correct.The replication agreed in direction and magnitude with the original, with Brier ratios of 7.127 and 7.094 versus matched uniform baselines.
- 4.1 Primary finding: 99.3% of S1’s capture-time Brier gap came from high-confidence events in the original batch, versus 98.7% in replication.Lower-confidence events contributed essentially nothing to the miscalibration.
- 4.2 The gap is not confined to capture time: S1’s every-ply beliefs were worse than a matched uniform prior from board-ply ≥8, with deltas of +0.059 original and +0.070 replication.The gap increased to +0.098–+0.112 at ply≥14, showing the failure was not confined to captures.
- 4.3 Scope: Point estimates ordered monotonically across five seat-configurations spanning Google and OpenAI, but most pairwise gaps were not statistically distinguishable.The paper treats this as scope evidence rather than evidence that external capability predicts calibration.
- 4.3 Scope: A 10.8-point same-model change from deliberation budget was comparable to the 14.9-point gap between configurations spanning a 635.7-point external leaderboard difference.S6 and S6B shared the same model and external score of 1258.32, differing only in max_tokens: 4096 versus 16384.
- 4.4 Conventional metrics: The configuration winning on legality, cost, latency, and completion rate produced the worst belief quality tested.This within-model result supports separating conventional play metrics from accurate belief-tracking.
- 4.5 Depth-dependent degradation: Well-formed elicitation fell from 85.6% to 47.8% with depth, while illegal attempts rose from 1.2% to 20.0%.The deep bucket was under-powered at n=39.
- 4.5 Hidden-state residual: In the well-powered mid-game bucket, Regent Chess scored 47.8% versus 65.7% for standard-chess control (p = 1.6×10−6).The two conditions were statistically indistinguishable in the opening bucket, while Regent Chess’s decline was 17.7 points steeper overall.
Figures
The figures visualize capture-time reliability, five-seat calibration ordering, depth-related well-formed-rate degradation, and the relationship between leaderboard score and calibration.
- Figure 1 plots stated confidence against capture-time hit rate across 394 capture-only events, with marker area proportional to bin size.The diagram includes a corrected reference tracker curve and omits lines across empty bins.
- Figure 2 presents capture-time calibration ordering across five seats with 95% Wilson confidence intervals.
- Figure 3 compares well-formed-rate degradation by ply bucket in Regent Chess and the no-crown control.
- Figure 4 supplements the analysis by relating leaderboard score to capture-time calibration and should not stand alone as evidence for any claim.
5. Limitations
The paper’s scope is constrained by limited seat coverage, a single opponent regime, non-deterministic sampling, and the absence of transfer validation. These boundaries prevent interpreting the reported ordering as evidence that external capability predicts calibration or that the findings generalize beyond the tested setting.
- Scope of evidence: Only S1 received the complete calibration battery; the five-seat comparison supports only capture-time high-confidence hit rate.Every-ply calibration, capture-time Brier ratio, coverage, and depth-degradation results are S1-only findings.
- Interpretation of comparisons: A same-model deliberation-budget change moved capture-time calibration by 10.8 points, comparable to a 635.7-point cross-model leaderboard gap.Because S1 was absent from the leaderboard and leaderboard scores proxy chess capability rather than calibration, the ordering cannot support capability-predicts-calibration claims.
- Experimental setting: All results used one fixed, moderate-difficulty heuristic opponent, leaving stronger, position-responsive, and greater-depth settings untested.The depth-degradation residual was measurable in only one ply bucket against this opponent.
- External validity: No result establishes that this instrument predicts behavior on real-world or downstream tasks; reported numbers remain domain-internal.Transfer validation is explicitly absent.
- Reproducibility: Provider-default non-deterministic sampling prevents exact replay from a fixed seed, although rerun replication agreed with the original batch.The replication used the same configuration rather than an exactly replayable random sequence.
- Measurement design: Joint elicitation may let the belief request interact with move generation, while split elicitation measures belief in a separate call and context.Which architecture best captures the belief associated with the action remains an empirical question.
6. Discussion
The discussion argues that outcome-only game evaluation can miss severe belief–action dissociation and presents Crown Shift as a distinct endogenous hidden-state test. It also frames tractability and deliberation limits as important qualifications on interpretation.
- Why outcome metrics do not surface this: A model can win despite a badly miscalibrated hidden-state belief, so outcome, legality, and move-count metrics can record a belief-driven failure as success.The paper gives both a representative game and a controlled within-model comparison supporting this dissociation.
- Endogenous vs. exogenous hidden state: Crown Shift tests concealment of a hidden fact the player creates and can repeatedly revise, unlike settings with fixed exogenous hidden information.This endogenous structure makes coverage of decisions about the agent’s own hidden state expressible.
- Tractability: Oracle-versus-public tractability testing found that solvability is opponent-policy-dependent, addressing whether failure could instead reflect task design.The paper identifies this tractability bound as a methodological contribution independent of any model score.
- Deliberation caps as an evaluation bias: A fixed output-token budget can penalize longer deliberation and weakens absolute claims about calibration ceilings when longer budgets are not tested for every seat.One seat truncated on roughly a third of turns while another almost never did.
7. Conclusion
The conclusion reports a replicated, high-confidence capture failure measured by separately eliciting belief and action against recoverable ground truth. It limits the claim to this instrument while identifying opponent responsiveness and out-of-domain transfer as key open questions.
- Conclusion: High-confidence captures were correct in only 1 of 62 cases across two independent, direction-agreeing batches.The result concerns confidence at the moment of action, measured separately from the chosen move.
- Open questions: Whether the failure holds against position-responsive opponents and predicts behavior outside this platform remains unaddressed.The fixed opponent’s near-random shift timing makes the hidden state unmeasurable past a certain point in the present setting.
Use of AI tools
The paper used generative AI across review, research direction, literature search, implementation, analysis, interpretation, drafting, and figure creation, under author direction and review. The game data were real model-play logs, not synthetic datasets, and substantive research responsibility remained with the author.
- Research and review: AI tools supported review, research-direction collaboration, literature search, manuscript correction, and claim assessment across the project.The author approved review rounds and independently verified resulting corrections.
- Implementation and methods: AI assistance supported implementation of the game engine, agent harness, elicitation pipeline, and analysis code under author direction.The rules were author-designed, and rule correctness was checked by a test suite covering each rule clause.
- Data and mathematics: The analyzed dataset consists of real model-play logs against a deterministic opponent at metered API cost, not AI-generated synthetic data.The paper also states that no novel mathematical proofs were formulated or assisted.
- Manuscript production: AI assistance was used to create and modify figures, summarize literature, format references, structure and draft the manuscript, and edit readability.These uses are listed as recommended-disclosure categories.
- Verification: Headline figures come from version-controlled analysis code, a canonical numbers document, and automated tests against committed data.This workflow followed an internal rule adopted after an earlier ad hoc result could not be exactly reproduced.
- Responsibility: The author retained responsibility for substantive research decisions, stopping rules, data-collection closure, claim selection, and final content.This includes artifacts produced with generative AI assistance.
Appendix: Regent Chess Rules
The appendix reproduces the frozen Golden Rules ruleset, version 1.2, used for every reported game, with selected non-result-bearing sections omitted.
- Version 1.2 of the frozen “Golden Rules” ruleset governed every game reported in the paper.Internal version-history annotations were removed, but no rule text was altered.
Core principle
Regent Chess makes royal status transferable and hidden, so victory depends on capturing whichever piece currently holds that status rather than necessarily the Original King.
- Royal status can be secretly transferred between pieces in Regent Chess.The objective is tied to the opponent’s current Regent, not permanently to the king.
- Capturing the opponent’s Regent wins immediately, whether or not that piece is the Original King.
Terminology
The rules distinguish the current royal piece, the piece that began as king, the secret transfer action, and the two resulting modes of play.
- The Regent is the piece currently holding royal status, and capturing it ends the game immediately.
- The Original King is the piece that began the game as the king.
- A Crown Shift secretly transfers royal status to another piece.
- King Mode means the Original King remains the Regent, while Regent Mode means another piece holds royal status.The terminology separates standard king restrictions from the rules applying to a non-king Regent.
Setup
Games begin from standard chess, then add privately tracked and repeatedly available Crown Shifts that alter royal status, check restrictions, victory conditions, and draw procedures.
- Setup: The game starts from the standard chess position with both players in King Mode, while hidden state is privately tracked and logged for recovery.
- Crown Shift: Crown Shift first becomes available at a player’s 4th turn and recharges after 15 further completed moves.Each player tracks move counts independently, and availability is unaffected by the opponent’s timing.
- Crown Shift: Crown Shift is a secret own-turn action that can designate any own piece as Regent, including the Original King.If the Original King has been captured while demoted, returning to King Mode is permanently impossible.
- Crown Shift: Available Crown Shift can escape checkmate by assigning royal status to a non-Original-King piece, after which the player must make a legal move.If the action is unavailable when checkmate occurs, the game ends immediately as a loss.
- Modes and victory: In King Mode the Original King obeys standard check restrictions, whereas in Regent Mode the current Regent is immune to them and its capture ends the game.A demoted Original King may enter attacked squares, cannot castle, and can be sacrificed without immediately losing.
- Other rules: A promoted Regent pawn silently becomes the new Regent, while all other standard chess rules remain unless specifically modified.Castling remains restricted to King Mode and does not recover rights lost while the Original King or relevant rook was demoted and moved.
- Signals and draws: Check announcements reveal attacks on the Original King in either mode, and passes, Crown Shifts, captures, and move counters determine special draw and availability behavior.Traditional stalemate is abolished; consecutive passes draw, while the 50-move and 75-move rules use different reset and pass-counting conventions.