Source-linked AI summary
Amadeus: When Models of People Meet
Karl Hanna
TL;DR
The paper asks whether independently learned models of people can reproduce their interactions, since individual fidelity may not guarantee interaction fidelity. It learns models of eight elite chess players while withholding their pairwise games, then composes them on those dyads. Recovery is partial and varies by method and behavioural dimension.
Problem
It is less explored whether accurately modelled individuals reproduce their interactions, which may depend on the specific partner or other interaction effects.
Source: §1 · p. 1Method
Amadeus learns player models from games outside an eight-player target cohort, seals all target-pair games, and evaluates composed models against those withheld interactions.
Source: §1 · p. 1Results
M1 more clearly improves WDL fidelity, while M2 substantially improves opening-family fidelity; M2 and M3’s correct identity assignment ranks closest among 40,320 assignments.
Source: §5 · p. 7Takeaways & Limitations
Independently learned models can recover aspects of previously unseen interactions, but recovery varies by modelling method and behavioural dimension.
Source: §5 · p. 7The study covers only eight elite chess players and evaluates fidelity with WDL and opening-family distributions, so its patterns may not hold for other populations, measures, or domains.
Source: §7 · p. 8
Abstract
from arXiv · showhide
With the constant advancements in AI, one possibility is to model agents after humans and, in turn, use these agents to carry out synthetic interactions. Such models could be used to predict interactions between their real counterparts, or potentially interactions at larger scales. In this paper, we test a more controlled version of this question through chess. We use 8 elite chess players, seal their direct pairwise games, learn each player independently using different methods, and then compose the resulting models on the withheld dyads. To evaluate the generated interactions, we use two measurements: opening-family total variation distance and win-draw-loss (WDL) total variation distance. M1 primarily improves WDL fidelity while producing smaller opening-family improvements, whereas M2 produces much larger opening-family improvements while having little effect on WDL-TV. For opening-family behaviour under M2, the correct assignment of the eight learned player identities also gives the closest match among all $8! = 40{,}320$ possible assignments. These results show that at least some properties of previously unseen interactions can be recovered from independently learned individuals. The partial recovery observed here may reflect limitations of the current individual modelling methods rather than a fundamental limit on compositional interaction recovery. An additional post-hoc method that combines the two mechanisms improves both measurements, suggesting that recovery across these behavioural properties is not necessarily mutually exclusive.
1 Introduction
The paper asks whether independently learned models of individuals can reproduce interactions that were withheld during training, since individual fidelity need not guarantee interaction fidelity. Using sealed chess pairings, it finds that recovery varies by method and behavioural dimension.
1 Introduction
- Individual fidelity may not imply interaction fidelity because behaviour can depend on the partner, relationships, and other interaction-specific effects.If individual models compose successfully, they could support simulations of real individuals, teams, or larger social systems.Source: §1 · p. 1
- Chess provides observable discrete actions, fixed rules, repeated player and pair observations, and a way to withhold target-pair interactions cleanly.Outcomes and openings can also be measured quantitatively, with less ambiguity about trajectories than in open-ended conversation.Source: §1 · p. 1
- Amadeus trains a ChessFormer-based model from scratch, learns player-specific behaviour with M1 and M2, then composes models on sealed interactions.The paper also explores M3, which combines the two already-trained mechanisms without additional training.Source: §1 · p. 1
- M1 more clearly improves outcome fidelity, while M2 produces much larger improvements in opening-family fidelity.The paper also reports that the correct player-identity assignment gives M2’s closest opening-family match among 40,320 assignments.Source: §1 · p. 1
- The study tests whether independently learned models can reproduce real interactions when those interactions are completely withheld during training.It introduces a controlled chess setup with eight elite players and strict pairwise sealing.Source: §1 · p. 1
2 Related Work
Prior work models human chess behaviour at population and individual levels, and individual behaviour has also been modelled beyond chess. Other generative-agent work studies social interaction, while Amadeus tests whether models of specific people reproduce their withheld real interactions.
2 Related Work
- Chess research has moved from human move prediction and skill-conditioned policies [4] [12] toward player-specific modelling, stylometry, and more efficient personalization [6] [13] [11] [1].ChessFormer provides Maia-3 human-move models and reports 57.1% move-matching accuracy [7].Source: §2 · pp. 1–2
- Individual-level behavioural modelling also spans chess and Rocket League [8] and LLM agents representing 1,052 real people [10].The combined interview-and-survey agents reached 86% of participants’ test–retest consistency on held-out survey questions [10].Source: §2 · p. 2
- Generative Agents [9], SOTOPIA [15], and Concordia [14] study social or emergent interaction, whereas Amadeus evaluates composed models against the real interactions withheld during learning.Amadeus’s target is whether models of specific real individuals reproduce those individuals’ interactions.Source: §2 · p. 2
3 Methodology
The paper tests whether independently learned player policies can be composed to approximate sealed interactions, using eight elite chess players and withholding all target-player games from training, tuning, and calibration. It compares policy-level and candidate-level personalization, evaluates individual behavior and composed interactions, and explores a post-hoc hybrid and robustness analyses.
3.1 Problem Formulation
- The goal is to test whether composing independently learned policies for players A and B produces a synthetic distribution resembling their sealed real interactions.The interaction process C combines the two policies in a shared chess environment.Source: §3.1 · p. 2
3.2 Base Chess Model / 3.3 Target Cohort and Interaction Sealing
- The base ChessFormer model is trained from scratch, then fine-tuned on Broadcast games excluding the eight targets; all target-player games remain sealed.Each target’s eligible games against outsiders are split at the game level for training and individual behavioral validation.Source: §3.2 · p. 2, §3.3 · p. 3
3.4.1 Policy-Level Personalization (M1).
- M1 personalizes the frozen base model by optimizing one player-specific 128-dimensional vector that replaces the mover’s Elo embedding.The opponent’s Elo conditioning remains unchanged; generation applies a Stockfish-based sampling adjustment.Source: §3.4.1 · p. 3
3.4.2 Candidate-Level Personalization (M2).
- M2 reranks the frozen base policy’s top-five legal candidates using a player-specific residual learned from shared move-style features and player vectors.The base-model parameters remain frozen; during training, the observed move is appended if absent from the raw candidate set, while inference uses the raw top five.Source: §3.4.2 · p. 3
3.5 Individual Behavioral Validation
- Individual validation compares correct-player, generic, and other-player representations on held-out nonsealed games, using method-specific NLL measures.M1 uses full-action NLL; M2 uses candidate-matched NLL, so their values are not directly comparable.Source: §3.5 · pp. 3–4
3.6 Exploratory Hybrid Composition (M3)
- M3 is an exploratory, post-hoc hybrid that uses M1 candidates and applies M2’s player-specific residual reranking without further training or tuning on sealed interactions.Its learned components remain frozen and use the same production sampler as M1 and M2.Source: §3.6 · p. 4
3.7 Compositional Interaction Protocol
- The sealed-dyad protocol compares generic-both, personalized-A-only, personalized-B-only, and personalized-both conditions against the same real distribution for each dyad.The generic baseline is the frozen base adapted to elite Broadcast games with all targets excluded; each dyad-condition generates 10,000 games across both color orientations.Source: §3.7 · p. 4
3.8 Evaluation Metrics / 3.9 Post-hoc robustness analyses
- Evaluation measures total variation distance for win–draw–loss outcomes and opening-family frequencies across 1,609 sealed games covering all 28 dyads.WDL outcomes are defined by player identity and color orientations receive equal weight; opening families use the deepest recognized opening, with unrecognized games assigned Unknown. Robustness analyses estimate finite-sample TV and compare matched identities with all 40,320 global permutations and one-player substitutions.Source: §3.8 · p. 4, §3.9 · pp. 4–5
4 Results
The learned player representations distinguish individual behaviour, and their composed interactions show method-dependent fidelity gains. Opening-family improvements are especially consistent across dyads and identity-specific, while finite-sample effects explain part—but not all—of the measured TV.
4.1 Individual Behavioural Validation
- For both M1 and M2, the correct player representation ranks first for all 8/8 target players; mean NLL decreases versus the generic representation by 0.006802 for M1 and 0.009278 for M2.M2’s raw top-5 candidates contain the observed move for 94.98% of positions on average; the two methods’ NLL definitions differ and should not be compared directly.Source: §4.1 · p. 5
4.2 Sealed Pairwise Interaction Fidelity
- M1 lowers WDL-TV from 0.164171 (GG) to 0.150148 (AB); M2’s opening-family TV change is -0.117509 (AB−GG), while its WDL-TV moves from 0.163166 (GG) to 0.163408 (AB).M1 also reduces opening-family TV from 0.554433 (GG) to 0.536973 (AB).Source: §4.2 · p. 5, Table 2 · p. 6
4.3 Exploratory Hybrid Model (M3)
- M3 reduces WDL-TV by -0.011037 and opening-family TV by -0.109967 from its own GG values, retaining gains descriptively close to M1 and M2, respectively.The reported AB values are 0.152036 for WDL-TV and 0.444428 for opening-family TV.Source: §4.3 · p. 5, Table 3 · p. 6
4.4 Dyad-Level Variation
- M2 improves opening-family TV for 27/28 dyads (96.4%; median change -0.105987), while WDL-TV improves for 13/28 (46.4%; median change 0.001650); M1 and M3 also show dyad-level variation.M1 improves both metrics for 22/28 dyads; M3 improves WDL-TV for 19/28 and opening-family TV for 27/28 dyads.Source: §4.4 · p. 5, Table 4 · p. 7
4.5 Identity Specificity of Compositional Recovery
- For opening-family behaviour under M2 and M3, the correct identity assignment has the lowest mean TV among all 8! = 40,320 assignments, and matched compositions outperform approximately 90% of one-player substitutions.The corresponding WDL identity effects are substantially weaker, particularly for M2.Source: §4.5 · p. 6, Table 5 · p. 7
4.6 Effect of Limited Real-Game Samples
- Finite-sample simulations show that limited real-game samples contribute substantially to observed TV, especially for opening-family behaviour, but do not fully account for the synthetic–real gap.The simulated opening-family TV shifts downward from GG to AB for M2 and M3, while WDL-TV changes little between conditions.Source: §4.6 · p. 7, Table 6 · p. 7
5 Discussion
The personalization methods recover different aspects of interaction fidelity: M1 improves WDL fidelity, while M2 improves opening-family fidelity and M3 retains some of both. The remaining gap cannot yet be attributed to imperfect individual models or pair-specific behaviour.
5 Discussion
- M1 improves WDL fidelity, M2 substantially improves opening-family fidelity, and M3 retains some of both.For M2 and M3, the correct assignment of player identities ranks first among all 8! = 40,320 assignments for opening-family behaviour.Source: §5 · p. 7
- The study cannot determine how much of the remaining interaction-fidelity gap reflects imperfect individual models versus genuinely pair-specific behaviour.The finite-sample analysis finds that limited sealed-game counts contribute substantially to measured TV, particularly for opening-family behaviour, but do not fully account for the gap.Source: §5 · p. 7
- The study is limited to eight elite chess players, WDL and opening-family measurements, and an exploratory post-hoc M3 hybrid.Future work includes stronger individual models, more repeated pairwise interactions, richer behavioural metrics, and evaluation in other domains.Source: §5 · pp. 7–8
6 Conclusion
The paper tests whether independently trained models of individuals recover aspects of unseen pairwise interactions. It finds partial, method- and behaviour-dependent recovery, while leaving open how much interaction structure is specific to the pair.
6 Conclusion
- M1 improves WDL fidelity, M2 substantially improves opening-family fidelity, and M3 retains some of both.For M2 and M3 opening-family behaviour, the correct identity assignment gives the closest match among all 8! = 40,320 assignments.Source: §6 · p. 8
- Interaction recovery depends on the modelling method and behavioural property, and the extent recoverable from strong individual models remains an open question.The paper asks how much of an interaction can be recovered from its participants and how much reflects interaction-specific structure.Source: §6 · p. 8
7 Limitations
The study’s conclusions are bounded by its players, domain, behavioural measures, and pairwise setting. Imperfect individual models and small sealed-game samples also limit attribution and precision.
7 Limitations
- The study covers eight elite chess players and measures interaction fidelity only through WDL and opening-family distributions.The observed patterns may not hold for weaker players, more volatile populations, other measurements, or domains outside chess.Source: §7 · p. 8
- Imperfect individual models make it difficult to distinguish modelling errors from behaviour specific to a particular interaction.The models include board-defined termination but omit actions such as resignation or agreed draws, which may carry player- or dyad-specific signal.Source: §7 · p. 8
- The study considers only pairwise interactions, and each sealed dyad has just 25–131 games.Samples of this size can produce substantial TV, particularly for opening-family distributions, limiting estimates of the underlying real interaction distributions.Source: §7 · p. 8
A Data construction and target resolution
The data pipeline fixes an eight-player cohort, filters and samples training games, excludes target-player games from training, and extracts deduplicated target-pair games for evaluation.
A Data construction and target resolution
- The cohort contains eight prominent players, selected by exhaustive pairwise-coverage analysis to balance mutual broadcast-game coverage against cohort size.The selection compared cohorts of size 6–9 using each cohort’s minimum mutual-game count across pairs.Source: A Data construction and target… · p. 9
- Games are retained only when they satisfy the validity rules, including standard variant, parseable player Elo ratings, a valid result, and at least one move.A missing variant header is treated as Standard; custom starting positions are excluded.Source: A Data construction and target… · p. 9
- The population corpus uses five June Lichess Standard-Rated archives and balances games across 22 mean-Elo bins, capping each bin within sequential 20,000-game chunks.The archives are from 2017 and 2018–2021; the cap is at most 10 games per bin per chunk.Source: A Data construction and target… · p. 9
- Games involving a target player’s known Lichess account are excluded from the population corpus before further processing.The username check normalizes case and whitespace.Source: A Data construction and target… · p. 9
- Training positions are sampled only before the first ply with a pre-move clock below 30.0 seconds, with up to 32 eligible plies sampled per game on each pass.Missing clock annotations do not trigger the cutoff.Source: A Data construction and target… · p. 9
- Target-vs-target games are removed from training and validation, while deduplicated sealed games cover all 28 dyads with 25–131 games per dyad.Broadcast adaptation excludes any game involving a target; the sealed cache contains 1,609 games.Source: A Data construction and target… · p. 9
B Base-model implementation and training
The base model encodes canonicalized board histories alongside Elo embeddings, then uses a transformer policy/value network. Training proceeds in two stages, with a fixed Stage-2 checkpoint reused in later experiments.
B Base-model implementation and training
- The input combines the current and seven preceding positions with mover and opponent Elo embeddings, yielding 352 features per square.Positions are canonicalized so the side to move is presented as White.Source: B Base-model implementation and training · p. 9
- The ChessFormer has eight 1024-dimensional transformer blocks with geometric attention bias, a 4,352-move policy head, and a three-way win/draw/loss value head.It has about 77.8M trainable parameters.Source: B Base-model implementation and training · p. 9
- Training uses population pretraining followed by Broadcast domain adaptation, with the Stage-2 checkpoint selected by validation loss and frozen for all later experiments.Stage 1 carries forward its final-step checkpoint without validation-based selection; Stage 2 selects from saved checkpoints using a 2% validation split.Source: B Base-model implementation and training · p. 10, Table 7 · p. 10
C Personalization implementation
Both personalization methods use one-target games while keeping the base model frozen. M1 learns player-specific policy vectors, M2 reranks a restricted candidate set using player style, and exploratory M3 combines their artifacts at generation time.
C Personalization implementation
- M1 and M2 train on each player’s games against non-target opponents, with game-based 80/20 splits and phase-balanced decisions in both splits.The base model stays frozen in both methods.Source: C Personalization implementation · p. 10, Table 8 · p. 11
- Each player receives a fixed representative Elo, computed as the median of their recorded ratings in accepted non-cohort Broadcast games.The value conditions generic play, opponent-side conditioning when the other side is personalized, and M1 initialization.Source: C Personalization implementation · p. 10
- M1 learns one 128-dimensional vector per player, replacing the mover’s Elo embedding while leaving opponent conditioning unchanged.Only the vector is trained, using legal-move-masked cross-entropy over the full 4,352-action policy space; each player is trained independently.Source: C Personalization implementation · p. 10
- M2 reranks the frozen base model’s top five legal moves using learned player-style representations, so inference cannot select a move outside that candidate set.Training and validation append the human move when it is absent from the top five; the base model remains frozen.Source: C Personalization implementation · pp. 10–11
- Exploratory M3 combines M1-personalized policy logits with M2’s top-five selection and style-residual reranking at generation time, without additional training.For generic play, M3 uses the plain frozen base model.Source: C Personalization implementation · p. 11
D Individual-behavior validation protocol
The validation diagnostic tests whether each learned representation predicts its player’s held-out moves better than generic and other-player representations. Synthetic games use a shared generation design and strength guardrail across methods.
D Individual-behavior validation protocol
- The diagnostic uses phase-balanced validation positions, but those same splits selected checkpoints, so it is not an untouched independent test.M1 evaluates 47,136 positions and M2 46,950; the difference reflects their different splitting procedures, not leakage.Source: D Individual-behavior validation… · p. 11
- For both M1 and M2, the correct player representation ranks first by mean NLL for all 8 of 8 players in the individual-behavior diagnostic.The comparison includes the generic baseline and the other players’ representations; the M1 and M2 NLLs use different action spaces and must not be compared or combined.Source: D Individual-behavior validation… · pp. 11–12
E Synthetic interaction generation
- Each of 28 dyads is generated in four conditions and both color assignments, with 5,000 games per assignment; M1, M2, and M3 share this design.This gives 10,000 games per dyad-condition and method.Source: E Synthetic interaction generation · p. 12
- Every reported method and condition uses the same post-hoc strength guardrail, which reduces but does not eliminate cross-player behavioral differentiation on held-out data.The guardrail value was selected on nonsealed held-out data to meet a fixed human-comparable-strength criterion.Source: E Synthetic interaction generation · p. 12
- Games use ordinary chess termination rules, with a 500-ply cap; capped games are censored and excluded from outcome denominators.Seeds are derived deterministically from the dyad, condition, color assignment, and game index.Source: E Synthetic interaction generation · p. 12
F Evaluation implementation
Evaluation compares generated and sealed interactions using WDL and opening-family distributions, with equal-weight aggregation across colors and dyads. Uncertainty intervals are computed per dyad and metric, not for the aggregate.
F Evaluation implementation
- The evaluation measures total variation distance between real and generated WDL distributions and between their opening-family distributions.Opening families come from a fixed, versioned reference database; games with no recognized opening receive an explicit “Unknown” category.Source: F Evaluation implementation · p. 12
- Color assignments are averaged 50/50, and the 28 dyads receive equal weight in the overall summary.Dyads with different numbers of real games therefore count equally.Source: F Evaluation implementation · p. 12
- The 95% percentile intervals use 10,000 paired bootstrap replicates resampling only real sealed games; the aggregate across 28 dyads has no confidence interval.Resampling is performed within each color assignment while preserving its original sample size.Source: F Evaluation implementation · p. 12
G Reproducibility summary
- The reproducibility summary is presented in Table 10.Source: Table 10 · p. 13