PaperlayerView exact revision on arXiv

Source-linked AI summary

Do LLMs Act on What They Know? From Partner Representations to Cooperative Actions

Yuhwan Jeong, Jinnyeong Yang, Kuk-Jin Yoon

arXiv:2610.08129v1cs.LG

TL;DR

Cooperation with unfamiliar partners requires adapting to conventions that may differ from one’s own, and the paper tests whether LLMs both decode those conventions and use them in receiving decisions. In a controlled Hanabi-derived environment, convention information stated as rules has limited effects, while translating it into action recommendations yields larger gains.

  • Problem

    The paper asks whether LLMs can infer an unfamiliar partner’s communication convention and use it to adapt receiving decisions.

    Source: §1 · pp. 1–2

  • Method

    The study probes receiver representations and tests rule-statement and action-translation interventions in a controlled Hanabi-derived environment with frozen LLM weights.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Convention information is more readily decoded on the intent than target axis, but receiving choices do not consistently follow it; combining true conventions with action translation yields target doing of 0.962 and intent doing of 0.867.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    Convention readout, behavioral adaptation, and game score require separate evaluation, and using inferred partner information in decisions remains an open challenge.

    Source: §6 · p. 10

    Probe errors bound the gains from linking convention information to current decisions, and turning available partner information into receiving decisions remains an open challenge.

    Source: §6 · p. 10, J Additional Discussion and Limitations · p. 26

Abstract

from arXiv · show

Cooperation with unfamiliar partners requires adapting to communication conventions that are not known in advance. We study this problem in a controlled Hanabi-derived environment with scripted hint generation, LLM-controlled receiving decisions, and frozen model weights. Across eight LLMs, linear probes recover intent conventions substantially more accurately than target conventions, yet receiving choices do not consistently agree with the sender's convention. We compare probe-predicted and ground-truth conventions presented either as general rules or as externally computed action recommendations. Rule statements yield modest and model-dependent changes in cooperation, whereas action translation produces larger gains on average. In a Qwen3-8B case study, matched-state statement reversals reveal much greater sensitivity to action recommendations than to rule statements. Activation transfers from oracle-action and non-oracle hint-restatement donors improve intent accuracy on both action classes, but the tested alternatives do not reliably reproduce these benefits. Together, these results distinguish convention decodability, sensitivity to convention information, and cooperative performance, and highlight limitations in turning available partner information into receiving decisions.

1 Introduction

The paper asks whether LLMs can infer unfamiliar partners’ conventions and use that information when receiving hints. In a controlled Hanabi-derived setting, it separates convention decodability from behavioral use and finds that translating conventions into action recommendations helps more than stating them as rules.

1 Introduction

  • Unfamiliar partners may share a goal while using different strategies or communication conventions, making previously effective responses inappropriate (Bard et al., 2020; Carroll et al., 2019).The paper focuses on adapting from interaction history without partner-specific updates to base-model weights.Source: §1 · p. 1
  • Accurate partner prediction need not translate into behavioral adaptation, motivating tests of both convention decodability and its influence on decisions (Riemer et al., 2025).The authors ask whether partner conventions are decodable from hidden representations and whether making that information available changes interpretation of the current hint.Source: §1 · p. 2
  • The controlled Hanabi-derived environment uses scripted hint generation and LLM receiving decisions to isolate convention-dependent interpretation and action selection.The sending policy is controlled, and the base LLM weights remain frozen.Source: §1 · p. 2
  • The study probes hidden representations and tests textual and activation-based interventions, separately measuring convention decodability, convention-following choices, and game score.Figure 1 schematically distinguishes Knowing, decoding the sender’s convention, from Doing, whether target and intent choices agree with it; the components are not identified model subspaces.Source: §1 · p. 2, Figure 1 · p. 2
  • Convention information is more readily decoded on the intent axis than the target axis, yet receiving decisions do not consistently follow the partner’s convention.Rule statements have limited effects on cooperation, while translating conventions into action recommendations yields larger gains.Source: §1 · p. 2
  • The contributions include a controlled setting separating convention inference, hint interpretation, and receiving decisions, plus matched-state and activation analyses of behavioral use.The authors also distinguish convention information from changes in default action preference.Source: §1 · p. 2

2 Related Work

Prior work studies partner-aware coordination, LLM cooperation, and internal representations, but behavioral performance alone cannot show whether partner information is absent or unused. This paper pairs readouts with interventions that test whether partner information changes decisions in line with a convention.

Partner-aware coordination

  • Coordination with unfamiliar teammates has been studied through symmetry-based training, cross-environment and population objectives, and general formulations for unseen partners (Hu et al., 2020; Jha et al., 2025; Hui et al., 2026; Dizdarević et al., 2025).Related approaches ground cooperation in human behavior, including imitation-based methods and influence-based team steering (Carroll et al., 2019; Jacob et al., 2022; Meta Fundamental AI Research (FAIR) et al., 2022; Sheng and Paleja, 2026).Source: §2 · p. 2
  • LLM cooperative-game studies and explicit partner models complement work on co-player inference and partner generalization (Zhang et al., 2024a; Cross et al., 2025).Behavioral performance alone does not reveal whether partner information is absent or simply unused; this paper tests whether feeding readouts back changes decisions in line with the partner’s convention.Source: §2 · p. 3

Probing internal representations

Activation patching and steering

  • Activation patching transplants internal states, while steering adds learned or synthesized directions at inference time; the robustness of steering remains an active concern (Zhang and Nanda, 2024).Related applications include game strategies and theory-of-mind behavior (Li et al., 2026a).Source: §2 · p. 3

3 Task and Experimental Setup

The task uses a Hanabi-derived game in which each player has a fixed communication convention that the other must infer from interaction history. Scripted hints and a controlled model interface isolate the LLM’s receiving decisions.

3.1 Convention-Based Cooperation Task

  • Players choose plays, discards, or color/rank hints in a Hanabi-derived game; hints mark cards by game rules, while conventions specify intended responses.Three colors and five ranks yield a maximum shared score of 15, and an incorrect play costs one shared life.Source: §3.1 · p. 3
  • Each player’s convention specifies a target rule and an intent mapping, and the receiver must infer the partner’s convention from interaction history.A successful play alone does not establish that the receiver selected the card intended by the sender.Source: §3.1 · p. 3

3.2 Interaction Settings

  • Scripted bots generate hints in both scripted interactions and controlled games, while LLMs make target-slot and PLAY/DISCARD choices at designated turns.The surrounding policy controls turn selection and other actions, isolating receiving decisions rather than fully autonomous gameplay.Source: §3.2 · p. 4

3.3 Model Interface

  • The base model weights remain fixed, and the model selects between two offered options using their next-token logits at the answer position.Prompts include task rules, the acting player’s convention, interaction history, and visible game state; option order is randomized in games.Source: §3.3 · p. 4

4 Probing and Interventions

The method reads partner conventions from receiver representations, then tests whether predicted or true convention information changes receiving choices as rules or action recommendations. Activation transfers and other representation transformations provide additional intervention tests.

4.1 Convention Probes

  • Linear probes predict the sender’s convention from concatenated embedding and transformer-layer representations of the receiver’s observation, with separate probes for each convention axis.The probe labels identify the sender’s assigned convention, independently of marked slots and option order.Source: §4.1 · p. 4
  • The primary last-token readout uses the final prompt-body token before the action question, while an auxiliary event-aggregated readout combines predictions from selected history-line tokens.The last-token readout is used for both axes and readout-guided interventions.Source: §4.1 · p. 5

4.2 Convention Information and Action Selection

  • Conditions vary whether convention information is probe-predicted or oracle and whether the convention is stated as a rule or translated into current action recommendations.In loop + verb, external code applies the predicted convention to the current hint; the LLM still makes the final choice.Source: §4.2 · p. 5

4.3 Activation Interventions

  • Activation interventions transfer attention-head outputs from oracle-action or hint-restatement donor runs and compare transfers of donor attention weights paired with recipient value vectors.Other tests include head-output scaling, convention-contrast injection, fitted maps, averaged directions, query replacement, and a minimal-context self-donor; the base model remains frozen.Source: §4.3 · p. 5

5 Results

Across eight LLMs, probes recover intent conventions more accurately than target conventions, but receiving behavior depends on how convention information is presented and applied. Action translation raises cross-play scores more than rule statements, while matched-state and activation analyses expose limits in using convention information.

5.1 Models and Evaluation

  • Evaluation covers eight open-weight LLMs, with Qwen3-8B used to develop the setup and for matched-state and activation analyses; game comparisons use matched seeds and configurations.Probe readouts and prompt-level conditions are evaluated across all eight models, with knowing, doing, and game score reported separately.Source: §5.1 · p. 5

5.2 Knowing

  • Across LLMs and both settings, intent conventions are decoded more accurately than target conventions; Qwen3-8B controlled-game last-token turn averages are 0.814 for intent and 0.569 for target.The event-aggregated readout gives 0.869 for intent and 0.607 for target, but the readouts differ in position, selection, and aggregation.Source: §5.2 · p. 6, Table 1 · p. 6

5.3 Doing

  • Under told + verb, target and intent doing reach 0.962 and 0.867, game score reaches 7.78 (+2.15 over raw), and the result remains 0.67 below the code ceiling.Rule-only statements have limited effects; applying the predicted convention to the current hint raises intent doing, while a corrected rule raises target doing.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 Self-Play and Cross-Play

  • Across 56 cross-play pairings, mean score is 5.1 for raw, 5.3 for loop, 5.4 for told, and 7.0 for told + verb; predicted loop + verb raises the mean from 5.3 to 5.9.Action-translation gains also appear in self-play; Qwen3-32B scores 8.4 under told + verb versus 2.2 under raw and 2.9 under told.Source: §5.4 · p. 7

5.5 Responses to Convention Information

  • Reversing an action recommendation shifts the answer logit by 1.378 on average and changes the selected action in 59% of states, versus 0.066 and 10% for rule statements.Rule-statement effects concentrate in color hints; scaling an attention head raises intent accuracy by choosing PLAY more often, helping PLAY states but harming DISCARD states.Source: §5.5 · p. 8, Table 3 · p. 9

5.6 Effects of Activation Interventions

  • The best single-layer donor transfers improve intent accuracy for both action classes, while fixed transformations do not reproduce the benefit and several shift preference toward PLAY.Loop readouts recover hint type and predicted convention more reliably than the resulting intent decision at attention-block outputs.Source: §5.6 · p. 9, Table 4 · p. 9

5.7 Learning the Application Step

  • Fine-tuning on 4,000 receiving decisions raises told-condition intent doing to 0.998 and game score to 8.43, near the 8.45 ceiling, but the original probe degrades on adapter states.Retraining the probe raises score to 6.47, and omitting the poorly decoded target statement raises it to 6.72; the remaining limitation shifts to convention recovery.Source: §5.7 · p. 9

6 Conclusion

The paper separates convention readout from behavioral adaptation: probes decode partner conventions, but receiving choices do not consistently follow them. Rule statements have limited effects, decision-linked convention information helps more, and readout, adaptation, and game score require separate evaluation.

6 Conclusion

  • Convention readout, behavioral adaptation, and game score require separate evaluation because receiving decisions do not consistently follow decoded partner conventions.Rule statements had limited effect; linking convention information to the current decision helped more, though probe errors bounded gains.Source: §6 · p. 10

A Evaluated Models

  • The evaluation covers eight open-weight instruction-tuned LLMs, using only Gemma3-12B’s language decoder rather than its vision tower.The models are loaded from their Hugging Face repositories; Llama-3.1-8B uses a mirror with weights identical to the gated Meta release.Source: A Evaluated Models · p. 14
  • Each probe input concatenates the residual stream from every layer, including the embedding output, at one token position.Its dimension is the number of layers plus one, multiplied by the hidden size.Source: A Evaluated Models · p. 14

B Environment and Measurement Constants

The appendix specifies the board design, prompt and response interfaces, probe readout, and example interactions used to make the measurements reproducible. It also describes how matched comparisons and readout availability are handled.

B Environment and Measurement Constants

  • The seed formula assigns boards across four convention configurations, while the agent’s own convention is generated independently of the partner’s.Board ranges are disjoint; probes train on training boards, readout numbers use test boards, constants use selection boards, and game comparisons use main-stage boards.Source: B Environment and Measurement Constants · p. 14
  • Each game score difference is paired on the seed, and 95% bootstrap intervals resample games because decisions within a game are not independent.The three-color, five-rank environment scores from 0 to 15; the code-based ceiling replaces the receiver with a deterministic policy applying the sender’s convention under the fixed sending policy.Source: B Environment and Measurement Constants · p. 15

C.1 Prompt Format and NOTES

  • Matched prompt comparisons verify token-for-token agreement outside the designed difference, while NOTES uses blank filler when a condition writes nothing.Rule statements describe the convention generally; action translation applies it to the current hint and gives a slot and action.Source: §C.1 · pp. 15–16

C.2 Answer Interface

  • Every main-text number uses the single-token interface; free generation is a separate check because its generated prefix changes the matched positions needed for probing and patching.Single-token decisions use the logit difference between the two option tokens in one forward pass, with option order counterbalanced.Source: §C.2 · p. 16

C.3 Event-Aggregated Readout

  • The event-aggregated readout sums probe logits from received-side history lines and predicts by the sign of that sum.It excludes the receiver’s sent hints and the partner’s reactions, uses observable records only, and makes no prediction when no received-side line is present.Source: §C.3 · p. 16

C.4 Probe Training

  • Each probe standardizes features using training-set statistics and fits an L2-regularized logistic regression with C = 0.01.Optimization uses L-BFGS for at most 300 iterations.Source: §C.4 · p. 16

D Prompt Examples

  • The prompt examples show a receiver choosing a marked slot consistent with the partner’s convention, then selecting an action according to its own convention.The example receiving prompts share the same body and differ only in their NOTES lines across conditions.Source: D Prompt Examples · pp. 16, 18
  • In this environment, partners’ assigned hint rules remain unstated and fixed within a game, and receivers are instructed to infer them from hints and responses.A hint’s sender meaning and receiver interpretation can differ because each follows that player’s own rules.Source: D Prompt Examples · p. 17

E Turn Curves

The turn-curve analysis compares readouts only where both produce predictions and reports how event-aggregated readout availability changes over turns. Final-turn table values do not correspond to the figure’s rightmost point.

E Turn Curves

  • The event-aggregated readout is available for 0.94 of perspectives by turn eight; Figure 3 restricts both curves to turns when both readouts predict.This avoids comparing the event-aggregated readout on its own turns against an easier sample; availability is absent at the first turn and present for half of perspectives at the second.Source: E Turn Curves · p. 19, Table 8 · p. 19
  • The rightmost point of Figure 3 averages only four perspectives reaching turn 36, unlike the final-turn columns of Table 1.Each perspective contributes at its own last turn; 150 of 480 controlled-game perspectives end at turn 33 and 128 at turn 34.Source: E Turn Curves · p. 19
  • Curves use matched observations and average over observations rather than perspectives.The released tables report accuracy under both averaging rules, counts, sample sizes, distinct perspectives and games, and bootstrap intervals.Source: E Turn Curves · p. 19

F Per-Model Doing and Game Score under Every Condition

The appendix table reports doing and game score by model across receiving conditions, with the code-based ceiling excluded. The supplied passages describe the table’s scope and scales but do not provide its cell values.

F Per-Model Doing and Game Score under Every Condition

  • Table 9 reports doing and game score for Qwen-family and other models across receiving conditions, with chance at 0.500 and scores on a 0–15 scale.The table scope excludes the code-based ceiling and includes five LLM receiving conditions and two threshold variants, each evaluated on 240 main-stage games per condition.Source: F Per-Model Doing and Game Score under… · p. 19, Table 9 · p. 20

G.1 Matched-State Statement Reversals

Matched-state reversals show that Qwen3-8B responds far more to action recommendations than to rule statements. The section also examines donor-transfer scope and reports intent-accuracy gains from oracle-action and non-oracle restatement transfers.

G.1 Matched-State Statement Reversals

  • The matched-state diagnostic used 800 receiving states, with 348 color-hint and 452 rank-hint states, and kept key prompt and answer conditions fixed.The action-recommendation and rule-statement pairs were matched in token length.Source: §G.1 · p. 20
  • Action recommendations shifted the answer logit in the recommended direction in all 800 states, while rule statements had a mean convention-aligned change of 0.066.The paired comparisons fixed the history, selected slot, question, and answer options.Source: §G.1 · pp. 20–21

H.1 Scope and Donor Conditions

  • The donor-transfer analysis used distinct state sets, so its effect sizes should not be treated as a ranking under one common evaluation distribution.The oracle-transfer analysis used 899 states; development analyses used 926 states.Source: §H.1 · p. 22

H.2 Donor Activation Transfers

  • The oracle-action donor transfer raised intent accuracy from 0.603 to 0.694 at layer 24, while the oracle-rule donor transfer gained only 0.010 there.Using oracle attention weights with recipient value vectors improved accuracy by 0.071; restoring recipient attention at NOTES positions retained a 0.069 improvement.Source: §H.2 · p. 22
  • The non-oracle restatement transfer improved intent accuracy by 0.057 on 926 development states, and the full restatement run improved it by 0.078.Both improvements occurred on both action classes; the transfer requires a separate restatement execution for each decision.Source: §H.2 · p. 22

H.3 Recipient-Side Transformations

Recipient-side transformations and mechanistic diagnostics tested whether fixed changes could reproduce donor-transfer benefits. Tested scaling, averaged directions, and other interventions did not reliably preserve convention-following behavior, while a Qwen3-8B adapter was trained to apply stated rules.

H.3 Recipient-Side Transformations

  • Fixed recipient-side transformations did not reliably reproduce donor-transfer benefits: the best fitted map added only three correct decisions out of 452, and averaged restatement directions lost the donor’s action-class gains.Scaling could improve aggregate accuracy by favoring PLAY while reducing accuracy on DISCARD states.Source: §H.3 · p. 23

H.4 Additional Mechanistic Diagnostics

  • A layer sweep found that rule-statement reversal moved the answer logit by 0.105 on average, with no layer carrying more than 0.045 of that shift.Random-vector injections at the same positions moved the logit by 1.4 to 2.8.Source: §H.4 · p. 24

H.5 Why the Tested Fixed Transformations Do Not Reproduce the Transfer

  • Fixed transformations captured only part of donor-output differences, and the recipient-run readouts did not decode the convention-consistent intent decision well.Linear readouts decoded that decision at 0.58 to 0.63 from the recipient run, versus 0.966 from the oracle action run from layer 7.Source: §H.5 · p. 24

H.6 Restating the Hint Type

  • Restating hint type raised loop intent doing from 0.612 to 0.682, but it lowered raw game score from 5.63 to 5.22 by changing the PLAY rate.The rule statement given with the restatement gained +0.75 in score; restatement alone was treated as a bias, not a method.Source: §H.6 · pp. 24–25

H.7 Confidence Threshold on the Target Axis

  • Omitting low-confidence target sentences raised loop score to 5.95, but the threshold reduced target doing on constant-selection boards and was not treated as a method.The threshold was based on a Qwen3-8B readout-logit magnitude of 2.93, with per-model medians for other LLMs.Source: §H.7 · p. 25

I.1 Training Data and Objective

  • The adapter’s 4,000 training decisions used randomly sampled rule statements as supervision for target-slot and intent decisions derived from the rule and current hint.The training used neither assigned conventions nor probe predictions.Source: §I.1 · p. 25

I.2 Adapter Configuration

  • The adapter used rank 16 and scaling 32 on attention and MLP projections, with 43.6M trainable parameters and one epoch of training.The base model weights remained unchanged during training; the adapter was merged into them in games.Source: §I.2 · p. 25

I.3 Probe Retraining

  • Because the base probe lost intent accuracy on adapter hidden states, the authors retrained probes on scripted interactions replayed through the adapter model.The retrained probes achieved 0.921 intent accuracy and 0.650 target accuracy on probe test boards.Source: §I.3 · p. 25

I.4 Game Results

On constant-selection states, the Qwen3-8B adapter followed the provided intent statement at 0.998. The supplied passage notes that the adapter also changed behavior without a statement and was evaluated only on Qwen3-8B, without cross-play.

I.4 Game Results

  • The adapter raised raw game score from 5.63 to 6.21, so its statement conditions were compared with adapter-under-raw as well as base-LLM-under-raw.The supplied table caption identifies 240 games per condition but does not provide the score cells.Source: §I.4 · p. 25, Table 12 · p. 26
  • On constant-selection states, the adapter followed the intent statement it was given at 0.998.Its intent doing under loop equaled the probe’s accuracy.Source: §I.4 · p. 25
  • The adapter was trained and evaluated on Qwen3-8B only, was not placed in cross-play, and changed sending behavior in unmeasured ways..Source: §I.4 · p. 25

J Additional Discussion and Limitations

The discussion separates access to partner-convention information from its use in receiving decisions, and cautions that accuracy or game-score gains alone do not establish greater partner sensitivity. Its conclusions are bounded by a controlled receiving setup and by the scope of the tested interventions.

J Additional Discussion and Limitations

  • Decodable convention information is not enough: rule statements change the intent decision little, while external action translation performs the step of applying the estimate to the current hint.The discussion argues that improving the readout alone is unlikely to close the gap.Source: J Additional Discussion and Limitations · p. 26
  • Higher accuracy or game score alone does not establish greater partner sensitivity; the paper separately reports knowing, doing, score, and effects on PLAY- and DISCARD-required states.Interventions can raise aggregate accuracy by favoring the more frequent action.Source: J Additional Discussion and Limitations · p. 26
  • A single layer carries much of the donor benefit, but tested transformations do not reproduce it, and readouts recover predicted conventions more reliably than resulting intent decisions.The tested alternatives include scaling, contrastive, fitted, and averaged transformations; other forms are not excluded.Source: J Additional Discussion and Limitations · p. 26
  • Because the setting uses fixed conventions and scripted hints in a two-player Hanabi-derived game, findings concern controlled receiving decisions, not autonomous or larger-team cooperation.Games with more than two players are left to future work.Source: J Additional Discussion and Limitations · p. 26
  • Probe transfer to model-generated histories depends on available evidence; donor gains use recorded states, while mechanistic analyses cover only Qwen3-8B and linear answer-position attention-block outputs.Residual-stream and MLP outputs, earlier positions, and nonlinear forms are not excluded.Source: J Additional Discussion and Limitations · p. 26

Related papers

Loading 2610.08129v1…