Source-linked AI summary
Aligning Superhuman AI with Human Behavior: Chess as a Model System
Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, Ashton Anderson
TL;DR
The paper addresses how AI can model granular human decisions for collaboration, focusing on move-level behavior in chess. It develops Maia, a human-game-trained AlphaZero framework, and neural networks for predicting moves and errors, finding stronger human alignment than existing engines and competitive baselines.
Problem
Existing chess engines and standard strength adjustments are poorly suited to predicting human moves at particular skill levels, limiting fine-grained human-AI alignment.
Method
The authors repurpose AlphaZero to predict human moves from human games without tree search, add game-history and architectural changes, and train residual networks to predict human errors.
Results
Maia achieves much higher human-move prediction accuracy than existing engines, while the error-prediction networks significantly outperform competitive baselines.
Takeaways & Limitations
Human-aligned AI can be tuned to match particular levels of human skill, and granular decision modeling extends beyond move prediction to human-error prediction.
Takeaways & Limitations
The work relies on a one-dimensional chess rating scale, leaving open whether finer alignment can reveal additional distinctions among players with the same rating.
Abstract
from arXiv · showhide
As artificial intelligence becomes increasingly intelligent---in some cases, achieving superhuman performance---there is growing potential for humans to learn from and collaborate with algorithms. However, the ways in which AI systems approach problems are often different from the ways people do, and thus may be uninterpretable and hard to learn from. A crucial step in bridging this gap between human and artificial intelligence is modeling the granular actions that constitute human behavior, rather than simply matching aggregate human performance. We pursue this goal in a model system with a long history in artificial intelligence: chess. The aggregate performance of a chess player unfolds as they make decisions over the course of a game. The hundreds of millions of games played online by players at every skill level form a rich source of data in which these decisions, and their exact context, are recorded in minute detail. Applying existing chess engines to this data, including an open-source implementation of AlphaZero, we find that they do not predict human moves well. We develop and introduce Maia, a customized version of Alpha-Zero trained on human chess games, that predicts human moves at a much higher accuracy than existing engines, and can achieve maximum accuracy when predicting decisions made by players at a specific skill level in a tuneable way. For a dual task of predicting whether a human will make a large mistake on the next move, we develop a deep neural network that significantly outperforms competitive baselines. Taken together, our results suggest that there is substantial promise in designing artificial intelligence systems with human collaboration in mind by first accurately modeling granular human decision-making.
1 INTRODUCTION
The paper frames granular modeling of human decisions as distinct from maximizing AI performance, using chess to study move prediction across skill levels and human-error prediction. Existing engines fail to align well with human moves, motivating Maia and a neural network for predicting blunders.
- Motivation: Granular human-alignment research targets instance-by-instance decisions and skill-specific behavior rather than aggregate performance or an absolute performance standard.The paper identifies this as relevant to domains including medicine, law, hiring, and lending.
- Chess as a model system: Chess provides superhuman AI, abundant recorded human games, and ratings spanning a wide range of expertise for evaluating fine-grained alignment.Online games record positions, moves, and move times, while ratings measure player skill.
- Research aims: The study predicts which move a human will make at a given skill level, rather than which move is most likely to win.It also targets prediction across many player skill levels.
- Research aims: The study also predicts whether a chess player will make a significant mistake on the next move, because this could guide human partners.This is the paper’s second principal modeling task.
- Existing approaches: Stronger depth-limited Stockfish versions generally predict human moves better than weaker versions, so depth-limiting does not target lower-skill human behavior.The finding challenges the use of bounded rationality as a route to human-like engine behavior.
- Contributions: Maia repurposes AlphaZero-style learning on human games without tree search, achieving higher move-prediction accuracy than Stockfish or Leela and tunable alignment to player skill.A separate deep residual network significantly outperforms competitive baselines for predicting individual and population-level mistakes.
- Implications: The results suggest that designing AI for human collaboration can benefit from accurately modeling granular human decision-making.The conclusion concerns systems intended to align algorithmic behavior with human behavior.
2 RELATED WORK
The paper connects human-aligned chess modeling to interpretability and research on modeling human decisions for assistance, error reduction, and education. Chess serves as a model system for studying alignment between algorithmic and human approaches.
- Interpretability: The paper differs from interpretability work by aligning an AI system’s observable chess behavior with observable human behavior.The focus is behavioral alignment rather than defining understandability or explainability alone.
- Human decision-making: Human-computer interaction and human-factors research models human decisions to support assistance, reduce human error, and target educational content.These applications motivate related efforts to represent human decision-making.
- Chess as a model system: Chess has long served as a model system for artificial intelligence and cognitive psychology, including studies using chess engines to examine human decisions and errors.This paper uses chess specifically to study alignment between algorithmic and human approaches.
3 DATA AND BACKGROUND
The study uses human games from Lichess and combines two contrasting engine paradigms: Stockfish’s heuristic search and Leela’s AlphaZero-derived neural architecture. Human ratings provide a skill measure across players and game formats.
- Chess engines: Stockfish uses heuristic position evaluation with classical alpha-beta game-tree search, while Leela is an open-source AlphaZero implementation.Stockfish evaluations support move-matching and blunder analyses.
- Chess engines: For human move prediction, the study repurposes Leela’s neural architecture but trains it on real human games instead of self-play.This separates the prediction target from the original AlphaZero training setup.
- Data source: Lichess supplies over 1 billion human games and more than 1 million games per day across players ranging from amateurs to the world champion.The platform provides live games at quick time controls.
- Data source: Lichess organizes games into formats including HyperBullet, Bullet, Blitz, Rapid, and Classical, with different time controls.Each format has its own player rating.
- Skill measurement: Each Lichess player has a format-specific Glicko-2 rating, with higher values indicating higher skill and updates after every game.The ratings provide a skill measure for analyzing behavior across expertise levels.
4 MOVE-MATCHING
The move-matching task evaluates whether chess systems predict the moves humans actually make, including at specific skill levels. Existing engines provide limited alignment, whereas Maia, trained on human games without tree search, predicts human moves more accurately and targets skill levels.
- Task definition: Move-matching treats predicting the move played in a human game as a classification task rather than selecting the move most likely to win.The main objective is to model granular human decisions at different skill levels.
- Existing engines: 33–41%: Stockfish engines match human moves at this rate, but their prediction curves rise with human rating instead of targeting particular skill levels.Depth-limited Stockfish versions differ in playing strength yet show similar performance among the strongest tested versions.
- Existing engines: 46%: the strongest tested Leela model reaches this move-matching accuracy, exceeding Stockfish, but Leela prediction curves remain essentially constant or weakly sloped across skill levels.Leela versions vary substantially in strength, while their human-skill targeting remains limited.
- Maia: Maia replaces AlphaZero self-play training with human-game training and predicts moves directly with its policy network, omitting tree search.The architecture also incorporates recent move history to improve human-move prediction.
- Maia: 46% to over 52%: Maia’s move-matching accuracy ranges from its lowest cross-rating result to its highest, surpassing the strongest existing engines.Each Maia model peaks near the rating range on which it was trained, with performance degrading as test rating diverges.
- Model comparisons: Within the 1100–1900 training ranges, every Maia model outperforms every tested Stockfish and Leela model; Maia 1900 still leads the best existing models by 5 percentage points.Beyond the trained rating range, Maia degrades while strong Leela and Stockfish models become competitive.
- Maia: 3 percentage points and 5–10 percentage points: recent history and removing tree search provide large Maia performance boosts, while more rollouts do not affect performance.The comparison uses Maia models trained on 1100- and 1900-rated players.
- Model comparisons: 65–79%: Maia models agree with one another at rates higher than agreement involving any non-Maia model, forming a distinct predictive subspace across training skill levels.Leela and Stockfish agreement depends more strongly on similarity in model strength.
5 PREDICTING ERRORS
The paper evaluates whether models can predict human blunders from chess positions and player metadata, finding that residual neural networks outperform competitive baselines on individual and grouped-position tasks.
- Individual blunder prediction: The task labels a move as a blunder when it worsens the player’s win probability by at least 10 percentage points.Win probabilities are derived from Stockfish evaluations through an empirical lookup table.
- Individual blunder prediction: Models are evaluated in board-only and board-and-metadata settings, where metadata includes ratings, time remaining, and board evaluation.These features are intended to be available to a live observer of the game.
- Individual blunder prediction: 63% accuracy is achieved by the best random-forest baseline using board state and metadata, compared with 56.4% using only the board.These baselines were selected by validation-set AUC after training on 3M moves.
- Individual blunder prediction: 71.7% accuracy is achieved by the AlphaZero-based residual CNN for individual blunder prediction.The model outperforms the fully connected network, with metadata supplied as additional input channels.
- Grouped-position blunder prediction: 76.9% accuracy is achieved by the deep residual CNN when predicting whether more than 10% of players blundered in recurring positions.Grouped-position prediction is easier than individual prediction, possibly because grouping reduces noise.
6 CONCLUSION
The paper frames human-aligned AI as a distinct goal from superhuman performance and develops Maia to model chess behavior at specific skill levels. It also extends this approach to human-error prediction while identifying future work on skill dimensions and other domains.
- Conclusion: Human-aligned AI is presented as a distinct goal from maximizing performance against an absolute standard.The paper emphasizes matching human output instance by instance and across levels of expertise.
- Conclusion: Maia is a chess model designed to predict human move-level behavior rather than optimize moves solely for winning.Its parameterization is intended to maximize alignment at a tuneable level of human skill.
- Conclusion: The methods are extended beyond move prediction to the prediction of human error.This broadens the model’s target behavior from chosen moves to situations in which people may go wrong.
- Future directions: Future work includes identifying additional dimensions of human skill beyond the one-dimensional rating scale.The authors also propose studying the approach in high-stakes, online, and physical-world domains.
7 SUPPLEMENT
The supplement documents the data-processing pipeline, model configurations, evaluation setup, and specialized blunder-prediction models used in the study.
- Reproducibility: The authors provide source code, data, models, and hosted tabular files to support reuse of the computationally intensive dataset-processing pipeline.Processing the April 2017–December 2019 data into tabular form required about four days on a 160-thread, 80-core server and 2.5 TB of memory.
- Data preparation: Training and validation games were binned into 100-point rating ranges, with bullet games and moves made under severe time pressure excluded.December 2019 was reserved for testing, and moves after either player had less than 30 seconds were removed.
- Move prediction model: The move-prediction model uses residual convolutional blocks and outputs an 8×8×73 move encoding, considering only legal moves during testing.The architecture uses six blocks, 64 filters, and separate policy and value heads.
- Win-probability conversion: The study estimates human win probability by empirically mapping rounded Stockfish centipawn scores to observed winning-player frequencies.Scores are rounded to the nearest 10 centipawns, with a discontinuity near zero attributed to positions where following a loop is optimal for both players.
- Blunder prediction: Baseline blunder-prediction models include decision trees, logistic regression, linear regression, random forests, and naïve Bayes, while SVM and perceptron models did not finish within two days.The baselines were implemented with Scikit Learn.
- Blunder prediction: Specialized blunder-prediction models substantially outperformed attempts to reuse the move-prediction models for blunder prediction.The individual blunder models used fully connected and residual convolutional architectures, with separate grouped-board preprocessing and training configurations.