PaperlayerView exact revision on arXiv

Source-linked AI summary

The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation

Makoto Fukushima, Hua-Dong Xiong, Ehsan Moradi Pari

arXiv:2609.11489v3cs.AIcs.HC

TL;DR

AI-AI benchmarks may measure compatibility with other AIs rather than humans, whose cooperation can rely on conventions beyond literal messages. The paper defines an exactly computable Hanabi convention gap and finds it positive in human and human-AI pairs but slightly negative in AI pairs.

  • Problem

    AI-AI benchmarks evaluate self-play or cross-play, while human cooperation may rely on implicit conventions; a quantitative metric isolating that channel across cooperation settings was lacking.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The convention gap compares the literal-hint posterior probability of life loss with the observed loss rate, using Hanabi’s finite deck and deterministic hints for exact computation.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    +26.2 pp in human pairs, +16.4 pp in human-AI pairs, and −0.7 pp in AI pairs; the gap concentrates where literal information is absent.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Convention compatibility, rather than AI-AI performance, may predict an AI’s effectiveness with human partners, so partnership claims based on AI-AI evaluation should be validated with humans.

    Source: §1 · p. 2, §5 · p. 14

    The measurement is retrospective rather than causal, and the human-AI data covers only three AI types from a single study.

    Source: §5 · pp. 13–14

Abstract

from arXiv · show

Cooperative AI agents are evaluated against other AIs, yet human cooperation relies on implicit conventions -- shared protocols for reading meaning beyond the literal message -- which AI-AI benchmarks may not capture. We propose the convention gap, the difference between the failure probability predicted from the literal content of communication and the observed failure rate, as a metric of implicit communication. In the card game Hanabi, the finite deck and deterministic hint constraints make this posterior exactly computable. We replayed about 101,000 play actions from three public datasets of human-human (an online Hanabi platform), AI-AI (HOAD), and human-AI (HanabiData) games. The gap was +26.2 percentage points (pp) in human pairs, -0.7 pp in AI pairs, and +16.4 pp in human-AI pairs, and was concentrated on plays of cards that had received no hints (+46 pp in human pairs). Within human-AI play, the literal information available to humans was similar across the three AI partners (mean predicted failure 38-41%), but human failure rates ranged from 14.4% to 34.4% and the gap from +24.1 to +6.2 pp; the partner eliciting the largest gap produced the fewest human failures. Game score carried different information: it depended on each corpus's roster composition, whereas the gap separated human from AI play at the agent level. As a known-answer check, Off-Belief Learning agents, whose convention content is controlled by construction, gave a gap of +1.6 pp at the convention-free level, rising monotonically to +21.7 pp. These results suggest that convention compatibility, rather than AI-AI performance, may predict an AI's effectiveness with human partners.

1 Introduction

AI-AI benchmarks may measure compatibility with other AIs rather than with humans, whose cooperation can rely on conventions beyond literal messages. The paper proposes an exactly computable Hanabi metric and reports distinct convention gaps across human, AI, and mixed pairs.

1 Introduction

  • AI-AI self-play and cross-play may assess compatibility with other AIs rather than compatibility with human partners.The paper frames the convention gap as a way to isolate implicit communication across human-human, AI-AI, and human-AI play.Source: §1 · pp. 1–2
  • The convention gap is the difference between a life-loss posterior based only on literal hints and the actual outcome.Hanabi’s finite card set and deterministic hint constraints make this posterior exactly computable by enumeration.Source: §1 · p. 2
  • +26.2 pp in human pairs, +16.4 pp in human-AI pairs, and −0.7 pp in AI pairs; the gap concentrates where literal information is absent.It approaches zero once two or more hints have touched the played card.Source: §1 · p. 2

2 Related Work

Prior work documents limits in transferring AI-AI performance to human-AI cooperation and the role of implicit conventions, but lacks a quantitative metric that isolates implicit communication across settings.

The Hanabi challenge and human-AI studies

  • Earlier Hanabi work found that strong self-play scores do not ensure transfer to unfamiliar partners, and human-AI studies have examined preferences and perceived cooperativity.Other-Play AIs outperforming self-play AIs with humans was an early indication that AI-AI rankings may not transfer [Hu et al., 2020].Source: §2 · pp. 2–3

Zero-shot coordination and ad hoc teamplay

  • Zero-shot coordination methods build agents for unfamiliar partners, whereas this paper takes a diagnostic approach to measuring compatibility without new AI training.Prior work also links behavioral differences between agents with ad-hoc cooperation failure in Hanabi (r = −0.978) [Jeon and Kim, 2023].Source: §2 · p. 2

Related work synthesis

  • The convention gap addresses the missing objective metric by comparing literal-information posteriors with actual outcomes across human-human, human-AI, and AI-AI settings.Related work connects the metric’s literal-versus-pragmatic distinction to grounded belief in Off-Belief Learning [Hu et al., 2021] and to literal and pragmatic listeners in RSA [Frank and Goodman, 2012, Goodman and Frank, 2016].Source: §2 · p. 3

3 Method

The method replays standard two-player Hanabi games to compute literal-information failure posteriors and compare them with observed outcomes, using play actions and hint properties as units of analysis.

3.1 The Convention Gap

  • The convention gap is the mean posterior probability of life loss from literal information minus the observed loss rate across play actions.Each play action is pooled as one observation, without first averaging by game or player.Source: §3.1 · pp. 3–4

3.2 The Posterior

  • The posterior enumerates feasible card identities using hint constraints and remaining copies, then conditions jointly on the player’s full hand.The finite deck and deterministic hints make the computation exact; the information set excludes convention-based inference about hint meaning or partner intent.Source: §3.2 · pp. 4–5

3 Method

  • The analysis replays standard two-player Hanabi games action by action through a custom engine to reconstruct information states before each play.The game uses a 50-card deck, and the score is the number of successfully played cards.Source: §3 · p. 3, §3.3 · p. 5

3.3 Data Sources

  • Three public datasets supply human-human, AI-AI, and human-AI games, replayed to reconstruct the players’ information states.The human-human corpus includes 425 completed games and 9,017 plays; the AI-AI corpus includes 62,890 plays, and the human-AI corpus includes 29,049 plays.Source: §3.3 · p. 5

3.4 Failure-Rate Metrics

  • Player Loss measures an AI’s own failed plays, while Partner Loss measures its partner’s failed plays when paired with that AI.The metrics are computed from the HOAD games for AI-AI play and HanabiData games for human-AI play.Source: §3.4 · p. 6

3.5 Hint Quality Metrics

  • Hint quality is measured by disambiguation power and playability rate: candidate identities eliminated and touched cards currently playable, respectively.The analysis logs these two metrics per hint to examine differences in convention gaps across AIs.Source: §3.5 · p. 6

4 Results

Across human-human, human-AI, and AI-AI play, the convention gap varies with cooperation setting and is concentrated where literal hints provide little information. Within human-AI play, it varies by AI partner, while game-score comparisons depend on corpus composition; the OBL hierarchy provides a known-answer validation.

4.1 Convention Gap Across Cooperation Settings

  • +26.2 pp in human-human play, +16.4 pp in human-AI play, and −0.7 pp in AI-AI play; human-human calibration falls below the literal posterior across bins.The headline gaps were nearly unchanged under play-, game-, and player/participant-level clustering.Source: §4.1 · pp. 6–7, Figure 1 · p. 7, Table 2 · p. 7

4.2 Where the Gap Resides: Hint-Count Localization

  • +46 pp in human-human plays with zero hints, compared with −12 pp in human-AI and −13 pp in AI-AI; with two or more hints, the gap is near zero in all settings.With exactly one hint, the gap was +36 pp for human-human, +25 pp for human-AI, and +1 pp for AI-AI.Source: §4.2 · p. 7, Figure 2 · p. 8

4.3 Per-AI Convention Gap Within Human-AI Play

  • With similar mean posterior information across AI partners (37.9–40.6%), human convention gaps ranged from +24.1 pp with Full to +6.2 pp with Outer.On AI play actions, Full had a +16.9 pp gap, while Intentional and Outer were near zero.Source: §4.3 · pp. 7, 9, Figure 3 · p. 9

4.4 The Convention Gap Distinguishes Human-AI from AI-AI Where Game Score Does Not

  • Game-score ordering depends on roster composition and can reverse under lenient scoring, while convention gaps vary little with composition and distinguish actor types.Human-AI teams scored 7.2 versus 10.7 for AI-AI teams under strict scoring, despite gaps of +16.4 pp versus −0.7 pp; lenient scoring reverses the score comparison.Source: §4.4 · pp. 9–10, Figure 4 · p. 10

4.5 Known-Answer Validation on a Designed Convention Hierarchy

  • +1.57 pp at the convention-free OBL root, rising monotonically to +21.68 pp at OBL5, as predicted by the designed hierarchy.The OBL1 near-zero gap also held in cross-play across independently trained seeds (+1.22 pp).Source: §4.5 · p. 11, Table 5 · p. 11, Figure 5 · p. 11

5 Discussion

The discussion argues that convention gap captures beyond-literal coordination that game scores can obscure, and that AI partnership claims warrant validation with human partners. Its interpretation is bounded by observational evidence and differences among datasets and partners.

Discussion

  • The gap separated human from AI play by 27 pp (+26.2 vs. −0.7), while game-score ordering depended on corpus composition and scoring choices.Within-setting score variation exceeded between-setting differences; under lenient scoring, the human-AI/AI-AI ordering reversed.Source: §5 · p. 12

What accounts for the human gap?

  • Human zero-hint plays succeeded at 78% despite a 0.69 posterior, whereas AI-partner zero-hint play succeeded at 16%, consistent with a missing shared convention system.Candidate mechanisms include positional conventions, action-based inference, good-touch assumptions, known-trash elimination, and hint-timing inference.Source: §5 · p. 12

Human-AI partnerships

  • Mean literal posterior was similar across AI partners (37.9–40.6%), but Outer’s hints touched playable cards less often (43.5%) than Full and Intentional (57–59%).These patterns are consistent with hint interpretability, rather than the AI’s own play accuracy, contributing to partnership quality.Source: §5 · p. 12

Beyond hint-level playability

  • Full’s 3.2 pp lower Partner Loss than Intentional did not replicate within participants, leaving that micro-ordering and its explanations unresolved.The paired mean difference was −3.4 pp, with 95% CI [−7.8, +0.9] and sign-test p = 0.63.Source: §5 · p. 13

Intent-based play as convention alignment

  • Full applies intent reasoning to interpret received hints, producing convention-like behavior; its own-play gap also appeared with all five tested LLM configurations.The authors relate partner compatibility to legibility: interpretable actions may matter beyond communication efficiency.Source: §5 · p. 13

Implications for cooperative AI evaluation

  • Benchmarks should report convention gap alongside score and validate AI-AI partnership claims with human partners, since AI-AI Partner Loss can underestimate Human-AI Partner Loss.The proposed recipe extends to domains where a literal-information baseline can be constructed, though the exact Hanabi baseline does not transfer.Source: §5 · p. 14

Limitations

  • The evidence remains bounded by limited AI-partner coverage, cross-dataset confounding, and observational measurement that cannot distinguish shared convention from unilateral adaptation.The human-AI data covers three AI types from one study; the three settings also come from different platforms, populations, and interfaces.Source: §5 · pp. 12–15

6 Conclusion

Across Hanabi cooperation settings, the convention gap varied systematically and was concentrated in zero- and single-hint plays. It also varied by AI partner, while game score and the gap carried different information.

6 Conclusion

  • +26.2 pp in human pairs, +16.4 pp in human-AI pairs, and −0.7 pp in AI pairs; the gap was concentrated in zero- and single-hint plays and varied by AI partner.Game score and the convention gap carried different kinds of information; the authors conclude that convention compatibility is a dimension benchmarks should measure.Source: §6 · p. 15

A Bot Screen and Human-Human Corpus Definition

  • +26.2 pp under the two-tier bot-screening policy and +27.5 pp under a stricter game-level-only policy, with the difference described as within between-corpus noise.The retained human-human corpus contains 425 games and 9,017 human play records from 155 players.Source: A Bot Screen and Human-Human Corpus… · p. 18

B Per-Actor Attribution in AI-AI Per-Agent Statistics

  • Per-agent AI-AI statistics use the actor identified on each play, while pooled AI-AI statistics, including the −0.7 pp headline gap, aggregate across all AI plays.A game-level pairing label would mix both agents’ plays rather than isolate the target agent.Source: B Per-Actor Attribution in AI-AI… · p. 18

C Hanabi Rules

  • In the standard two-player game, players see partners’ cards but not their own, and a failed play costs a life token.Three life losses end the game; the score counts cards successfully played, from 0 to 25.Source: C Hanabi Rules · p. 19

D Replay Pipeline and Game Engine

  • At each play, the replay engine snapshots the acting player’s hint knowledge and computes a literal-information posterior before applying the play to observe its outcome.The engine reconstructs game state from deck order and actions, including visible cards, hint constraints, and fireworks.Source: D Replay Pipeline and Game Engine · pp. 19–20

E The Joint Posterior: Exact Computation, Verification, and the Per-Card Approximation

  • Exact joint inference conditions on the acting player’s full hand; the per-card approximation differs on some plays but moves no headline gap by more than 2.1 pp.The joint method was verified against exhaustive enumeration on 936 plays with maximum absolute discrepancy 0; its Brier score improved over the approximation in all three main settings.Source: E The Joint Posterior: Exact… · p. 21

F AI Agent Descriptions

  • The AI-AI corpus uses seven rule-based agents, while the human-AI corpus uses Intentional, Outer, and Full; Full incorporates bidirectional intent reasoning.Full has no HOAD counterpart.Source: F AI Agent Descriptions · pp. 21–22

G Logistic Regression

H Statistical Methods

The paper uses logistic and hierarchical models, bootstrap intervals, and cluster-robust sensitivity checks. In human-human play, convention gap varied by pair type, with the expert–beginner stratum omitted as too small for a stable estimate.

H Statistical Methods

  • Play-level bootstrap intervals were checked with game- and player/participant-cluster resampling, and the three levels agreed closely for headline and per-partner gaps.The authors report that no conclusion depended on the resampling level.Source: H Statistical Methods · p. 23
  • Logistic models predict binary life loss from the posterior, with model variants adding pair skill or human-AI partner type; hierarchical models assess convention-gap change with game sequence.The human-human hierarchical model uses 415 game-player observations from 20 players; the human-AI model uses 1,376 observations from 23 participant–AI trials.Source: H Statistical Methods · p. 24
  • The human-AI hierarchical model includes partner-specific intercept and slope shifts, but its 284 divergences persist with only 10 participants.The passage reports fixed-effect R-hat ≤1.02 and says none of the population-level slopes is near a decision boundary.Source: H Statistical Methods · p. 24

I Skill Modulation

  • Expert–Expert pairs had a +29.1 pp convention gap versus +21.1 pp for Beginner–Beginner pairs; the three-game Expert–Beginner stratum was omitted as unstable.Expert and intermediate players had similar mean posteriors (31.6% vs. 30.2%), while actual loss rates differed (2.5% vs. 4.8%).Source: I Skill Modulation · p. 24

J Hint Quality Metrics

Hint-level metrics compare the three HanabiData AIs with a pooled HOAD baseline. The partner-related convention-gap difference is concentrated at one-hint plays.

J Hint Quality Metrics

  • Figure A2 compares disambiguation power, playability rate, and the recipient’s next action across three HanabiData AIs against the pooled HOAD baseline, using 20,088 human-AI hints.The pooled AI-AI baseline had 36.4% disambiguation power and a 48.0% playability rate.Source: J Hint Quality Metrics · pp. 24–25, Figure A2 · p. 25
  • Humans made zero-hint plays in 4.6% of plays with Full and 7.6% with Intentional, and these plays failed at 77–87%.The pooled AI-AI baseline was intermediate between Outer and the other two AIs on disambiguation power and playability rate.Source: J Hint Quality Metrics · p. 25

K Convention Gap by AI Partner Type

  • Table A5 reports each AI partner’s human-play convention gap and 95% bootstrap intervals, while Table A6 decomposes gaps by hint-action count.The overall human-human average (+26.2 pp) is the denominator for the “% of H-H” comparison.Source: Table A5 · p. 25, Table A6 · p. 26
  • The partner difference in convention gap is concentrated at one-hint plays: Full is +32.3 pp and Outer +10.8 pp; at two or more hints, gaps are near zero regardless of partner.At zero hints, all three partners had negative gaps.Source: K Convention Gap by AI Partner Type · p. 25, Table A6 · p. 26

L Within-Participant Partner Analysis

Within-participant contrasts show positive convention-gap differences for Full and Intentional relative to Outer for most participants who tried multiple AI types. The Full–Intentional contrast remains unresolved.

Participant × partner structure

  • The participant–partner data are self-paced and heavy-tailed: the median participant played two games total, and the median participant×partner cell contained one game.The metadata covers 2,040 analyzed games and 228 observed participants.Source: L Within-Participant Partner Analysis · p. 26

Paired within-participant contrasts

  • Full–Outer and Intentional–Outer gap contrasts were positive for 57 of 69 and 58 of 64 participants, respectively; Full–Intentional was small and not significant in either direction.Among participants who played all three AI types, gap(Full) exceeded gap(Outer) for 83.3%, and both Full and Intentional exceeded Outer for 77.1%.Source: L Within-Participant Partner Analysis · p. 26, Table A7 · p. 26
  • Because each participant serves as their own control, the Outer contrasts cannot be explained by different populations self-selecting different AI partners.At a threshold of at least three games per cell, the Full–Outer contrast was positive for all 22 participants.Source: L Within-Participant Partner Analysis · p. 26, Table A7 · p. 26

M Robustness: Lives-Remaining Stratification and the Literal-Information Ceiling

The gap remains robust when last-life plays are excluded, and the literal-information ceiling shows how much of the available convention information each group uses. Human-AI partners shift the fraction of that ceiling humans attain substantially.

M Robustness: Lives-Remaining Stratification and the Literal-Information Ceiling

  • +26.6 pp versus +26.2 pp for human-human pairs when last-life plays are excluded versus included; the corresponding human-AI values are +17.4 versus +16.4 pp and AI-AI values are −0.5 versus −0.7 pp.The gap is largest with all three lives remaining and smallest on last-life plays; those plays make up 6.4% of human-human, 17.9% of human-AI, and 8.1% of AI-AI actions.Source: M Robustness: Lives-Remaining… · p. 26, Table A8 · p. 27
  • 92.1% for expert-expert pairs and 83.2% for the human-human corpus overall are the fractions of the literal-information ceiling attained.The gap is bounded above by the mean posterior, and the bound is attained when a group never fails.Source: M Robustness: Lives-Remaining… · pp. 26–27, Table A9 · p. 27
  • 62.7% with Full versus 15.3% with Outer is the fraction of the literal-information ceiling humans attain; AI-AI play is −8.5%.The AI-AI value is slightly below calibration.Source: M Robustness: Lives-Remaining… · p. 27

N AI Play-Time Mechanism Statistics

  • Table A10 reports each HanabiData AI's play-time posterior, hint count on the played card, and fraction of plays extending the fireworks.The posterior is the Bayesian life-loss estimate, hints-on-card counts hints touching the card, and success records whether the play extended the fireworks.Source: N AI Play-Time Mechanism Statistics · p. 27, Table A10 · p. 27

O Convention Gap Across All Settings

The convention gap declines across human-human, human-AI, and AI-AI play, with AI-AI values generally near zero. Human-AI and human-pair values overlap, while Flawed accounts for a distinct high-risk pattern among AI plays.

O Convention Gap Across All Settings

  • +21.1 pp for Beginner-Beginner pairs is below Full's +24.1 pp; the seven AIs range from +0.7% for Bergh to −5.8% for Flawed.The other five AIs are within ±0.6 pp of zero.Source: O Convention Gap Across All Settings · pp. 27–28
  • Flawed's plays at posterior risk 0.6–0.8 fail 10–12 pp more often than predicted, while the other six AIs show only small departures in either direction.These Flawed plays number 4,191; they make up 95% of the plays in Figure 1c's 0.5–0.8 bins. Flawed plays cards its rule-following partner chose not to hint.Source: O Convention Gap Across All Settings · p. 28
  • +21 to +29 pp in human-human, +6 to +24 pp in human-AI, and approximately 0 pp in AI-AI play form the cross-setting gradient.Figure A3 shows human plays and AI plays separately for each human-AI partner type.Source: O Convention Gap Across All Settings · p. 27, Figure A3 · p. 28

P Game Score Analysis

Strict end scores differ across the three settings, but decomposing them shows that human-AI teams score higher than AI-AI teams when games survive. Score distributions also depend on censoring and roster composition, while AI-AI risk and score rankings diverge.

P Game Score Analysis

  • Intentional scores highest in human-AI games at 12.1, followed by Full at 7.2 and Outer at 2.9; AI-AI scores range from IGGI at 13.7 and Piers at 13.6 to Flawed at approximately 0.Human-human scores range from E-E at 23.8 and E-I at 23.4 to I-B at 13.6 and B-B at 9.9.Source: P Game Score Analysis · p. 28
  • Human-AI game-score ranking (Intentional > Full > Outer) differs from Partner Loss ranking (Full < Intentional < Outer), so the two rankings capture different aspects of cooperative quality.Figure A7 shows the human-AI score distributions by partner type.Source: P Game Score Analysis · p. 28
  • 20.5% of human-human, 57.4% of human-AI, and 26.3% of AI-AI games strike out; among games reaching non-zero scores, means are 23.3, 16.9, and 14.6, respectively.Thus human-AI teams score higher than AI-AI teams when they complete a game; the pooled mean ordering reverses because human-AI has a higher strike-out rate.Source: P Game Score Analysis · pp. 28–29
  • Lenient scores reverse the human-AI versus AI-AI mean ordering: 12.1 versus 11.6, and human-AI becomes broadly unimodal after strict zeroing is removed.The human-AI bin proportions are 1/16/22/24/33/3% across [0, 1–5, 6–10, 11–15, 16–20, 21–25]; its struck games had a mean pre-strike tableau of 8.4. Lenient scoring is diagnostic only; headline analyses retain strict scoring.Source: P Game Score Analysis · p. 29, Figure A4 · p. 29
  • AI-AI scores remain bimodal under lenient scoring because the low mode is carried entirely by Flawed games, reflecting roster composition rather than censoring.Figure A5 decomposes lenient end-score distributions by group.Source: P Game Score Analysis · p. 29, Figure A5 · p. 30
  • 18.5 for human-human, 10.7 for AI-AI, and 7.2 for human-AI are the mean strict end-of-game scores across settings.All three pairwise contrasts are significant; Figure A6 presents mean scores with ±1 SD across games.Source: P Game Score Analysis · p. 28, Figure A6 · p. 30

Q Convention Gap Cross-Matrices

  • In AI-AI cross-play, Flawed has high posterior risk regardless of partner, while most other agents have near-zero risk with cooperative partners and elevated risk with Flawed.Piers and IGGI achieve the highest scores across most partners, while Flawed depresses scores across the board; Figure A8 shows the subject-partner matrices.Source: Q Convention Gap Cross-Matrices · p. 30

R Failure Mode Taxonomy

Convention failures are the dominant classified cause of life-loss events across settings. Following such events, the convention gap declines, with the largest drop for Outer.

R Failure Mode Taxonomy

  • 40–51% of life-loss events are convention failures, the dominant mode in every setting.These are plays made despite hints and a posterior indicating risk greater than 0.5.Source: R Failure Mode Taxonomy · p. 30
  • With Outer, convention failures account for 51.4% of losses, alongside 30.1% desperate plays.The causes are classified for every life-loss event.Source: R Failure Mode Taxonomy · p. 30
  • The mean convention gap over the five turns after a life-loss event falls by 7.6 pp with Outer, 4.3 pp with Full, and 3.0 pp with Intentional.The graded contrast is consistent with opaque communication degrading most under pressure.Source: R Failure Mode Taxonomy · p. 30

S Convention Learning

Convention-gap analyses suggest weak human-human learning but no credible partner-specific learning with AI; tests with LLM partners find that Full’s positive gap persists while reciprocal non-literal interpretation is mostly absent.

S Convention Learning

  • Human-human convention learning was probable but not statistically credible, while human-AI play showed no credible learning trend with any partner type.For human-human play, P(β > 0) = 0.98 but the 95% HDI narrowly included zero; human-AI P(β > 0) values were all ≤0.63.Source: S Convention Learning · pp. 31–32

T The Posterior as a Rational-Agent Benchmark

  • The posterior treats literal-hint inference as a rational-agent baseline, so humans’ better-than-predicted risky plays reflect an additional convention channel.The posterior enumerates hint-consistent card identities weighted by their remaining copy counts.Source: T The Posterior as a Rational-Agent… · p. 32

U Predicting Human-AI Partner Loss from Hint Structure

  • AI-AI hint profiles suggest a testable hypothesis: hints optimized for information rather than action-readiness may elevate human Partner Loss.Simple had the highest AI-AI Partner Loss (21.9%) and 44.3% playability, making it a candidate for elevated loss with humans.Source: U Predicting Human-AI Partner Loss… · p. 32

V Convergence with Subjective Cooperativity Ratings

  • A controlled Piers-versus-RL experiment likewise found more errors and lower perceived cooperativity for the RL partner, paralleling Outer’s degradation with humans.The prior study had N = 8; the authors describe the convergence with their log analysis as consistent with the gap capturing a property related to subjective cooperativity.Source: V Convergence with Subjective… · p. 33

W Testing the Intent-Based Interpretation with LLM Partners

  • Full’s own-play gap exceeded +11.8 pp in all five LLM pairings, while only Qwen3.6-A3B showed a nonzero LLM-side gap, reduced with CoT.The LLM-side Qwen3.6-A3B gap was +5.8 pp; with CoT it was +1.9 pp. The experiment covered five LLM configurations.Source: W Testing the Intent-Based… · p. 33, Figure A9 · p. 34

X Known-Answer Validation on a Designed Convention Hierarchy (OBL)

Off-Belief Learning provides a designed test of the convention gap: it is near zero for a convention-free policy and rises through the hierarchy as convention interpretation is introduced.

X Known-Answer Validation on a Designed Convention Hierarchy (OBL)

  • OBL1 is trained against beliefs induced by a uniformly random partner, removing incentive to form or read conventions; higher levels reintroduce interpretation stepwise.This construction predicts a near-zero gap at OBL1 and a monotonic increase across levels.Source: X Known-Answer Validation on a… · p. 34
  • +1.57 pp at OBL1 rose monotonically through OBL2–OBL5 to +21.68 pp, matching the designed convention hierarchy.The mean literal posterior of chosen plays rose from 7.8% to 24.1%, while realized loss stayed at 2–3%; OBL1’s cross-play gap was +1.22 pp.Source: X Known-Answer Validation on a… · pp. 34–35
  • OBL1’s residual gap decays toward zero with higher-level partners because grounded hints provide harvestable playability correlation that convention-reading partners do not.With OBL5, OBL1’s gap was +0.1 pp [−0.2, +0.5], whose CI includes zero; 72% of the decay came from within-stratum changes.Source: X Known-Answer Validation on a… · p. 35
  • Levels 2–5 use one released seed each, and seed-invariance evidence is available only for OBL1.The monotonic trend also holds across published levels 1–4 without the unpublished OBL5.Source: X Known-Answer Validation on a… · p. 35
Loading 2609.11489v3…