PaperlayerView exact revision on arXiv

Source-linked AI summary

Transcendence: Generative Models Can Outperform The Experts That Train Them

Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, Eran Malach

arXiv:2406.11741v4cs.LGcs.AI

TL;DR

Because generative models imitate human-generated data, it is unclear whether they can outperform the experts whose behavior they learn. This paper formalizes transcendence and studies low-temperature sampling as a way to denoise expert errors. In chess experiments, low-temperature models can surpass the players who produced their training data.

  • Problem

    Generative models are trained to match human-labeled output distributions, raising the question of whether they can surpass the experts whose behavior generated their data.

    Source: §1 · p. 1

  • Method

    The paper formalizes transcendence and proves that low-temperature sampling can denoise expert errors through implicit majority voting.

    Source: §1 · p. 2

  • Results

    Low-temperature ChessFormer models sometimes surpass training players; at τ = 0.001, expected reward (probability of winning) improves by 2.15±0.17%.

    Source: §4.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    Within the studied setting, diverse expert data enables low-temperature denoising to consolidate varied knowledge, and diversity is necessary for effective transcendence.

    Source: §1 · p. 2, §6 · p. 10

    The theoretical framework assumes test-time game conditions match training conditions, limiting its direct extension to composition or reasoning without relaxing that assumption.

    Source: §6 · p. 10

Abstract

from arXiv · show

Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence can be enabled by low-temperature sampling, and rigorously assess this claim experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.

1 Introduction

The paper asks whether models trained to imitate human behavior can surpass their expert sources, despite being optimized to match human-labeled distributions. It investigates transcendence through chess experiments and theory connecting low-temperature sampling, expert denoising, and majority voting.

1 Introduction

  • The paper assumes specialized knowledge and capacities are supplied by cross-entropy training, while setting aside post-training methods such as RLHF.This assumption limits the scope of the introduction’s discussion of generative models.Source: §1 · p. 1
  • ChessFormer 1000 and 1300 significantly surpass the maximum ratings of their training players, demonstrating transcendence in chess.Figure 1 reports ratings across sampling temperatures for models trained on games capped at different player ratings.Source: §1 · p. 2, Figure 1 · p. 2
  • The theory connects low-temperature sampling to implicit majority voting, which can denoise expert errors and enable transcendence.The analysis covers a single expert making uniformly random mistakes and collections of experts skilled in different domains.Source: §1 · p. 2
  • The chess experiments confirm the prediction that models exceed their training experts’ maximum rating only at low temperatures.The models are trained on chess transcripts with players capped at specified skill levels.Source: §1 · p. 2
  • Lower temperatures improve rewards substantially on a relatively small subset of states, including states likely to determine game outcomes.The paper attributes increased performance to large improvements on a relatively small portion of states.Source: §1 · p. 2
  • Less diverse datasets prevent ChessFormer from transcending, making dataset diversity necessary for practical majority-vote gains.The paper also notes that diverse expert perspectives support the denoising effect.Source: §1 · p. 2

2 Definition of Transcendence

The section formalizes transcendence by specifying inputs, outputs, experts, and rewards. It defines the phenomenon as a learned predictor achieving better reward than the best expert generating its data, within an idealized learning setting.

2 Definition of Transcendence

  • The framework represents variable-length inputs and finite outputs, with predictors mapping each input to a conditional output distribution.The function class F consists of functions from X to probability distributions over Y.Source: §2 · p. 3
  • Training examples come from a full-support input distribution and experts selected uniformly at random, inducing a mixture distribution over input-output pairs.The experts share the same input distribution in this setup.Source: §2 · p. 3
  • Prediction quality is measured by average reward over a chosen test distribution, with rewards varying across outputs for each input.The framework defines a reward function on input-output pairs and a test distribution over inputs.Source: §2 · p. 3
  • Transcendence means the learned predictor beats the best data-generating expert on reward, in an idealization with infinite data and unrestricted predictors.The paper notes that this framework also assumes shared expert input distributions, full-support training inputs, and uniform expert sampling.Source: §2 · p. 3

3 Conditions for Transcendence

The section establishes conditions under which a model trained on expert labels can or cannot surpass its expert sources. It shows that direct distribution modeling cannot transcend, while sufficiently low-temperature sampling can under specific conditions.

3.1 Low-Temperature Sampling is Necessary for Transcendence

  • Without temperature adjustment, the learned mixture cannot exceed every expert’s reward: some expert always matches or beats it.This result assumes experts are sampled uniformly for each input; the authors leave possible transcendence with Bayesian weighting for future work.Source: §3.1 · pp. 3–4

3.2 Transcendence with Low-Temperature Sampling

  • Transcendence occurs at all sufficiently low temperatures exactly when the arg-max predictor beats every expert; low-temperature sampling acts like majority vote across experts.When experts assign non-negligible mass to the best actions, this vote may select them.Source: §3.2 · p. 4

3.3 Denoising a Single Expert

  • For a single correct but noisy expert, sufficiently low-temperature sampling achieves transcendence when the expert mixes random outputs with optimal predictions.The noisy expert chooses a random output with probability ρ and otherwise follows the optimal expert.Source: §3.3 · pp. 4–5

3.4 Transcendence from Multiple Experts

  • Complementary experts can be transcended when the test distribution assigns positive mass to at least two subsets where different experts perform well.Each expert performs well on its own subset and behaves randomly on the others.Source: §3.4 · p. 5

4 Experiments

The experiments test whether low-temperature sampling can produce transcendence in chess and examine dataset diversity as a condition. Chess models trained on lower-rated games sometimes exceed their source players, while additional SQuAD and synthetic experiments report temperature-related gains.

4.1 Experimental Setup

  • ChessFormer trains blindly on PGN moves and outcomes, while its latent space captures state advantage and player identity, addressing the theory's full-coverage assumption.The theory assumes experts are defined over the entire input space, unlike chess data after roughly move 15.Source: §4.1 · pp. 5–6, Figure 3 · p. 6

4.2 Experimental Results

  • Around 1500 rating, ChessFormer 1000 and 1300 transcend at τ = 0.001, while ChessFormer 1500 does not.The rating number refers to the maximum player rating in the training data.Source: §4.2 · p. 6
  • 2.15±0.17% average expected-reward gain at τ = 0.001 accompanies a right-shift concentrated in a few states, sometimes exceeding 5%.The reported reward is probability of winning.Source: §4.2 · pp. 7–8
  • The <1500 dataset has less measured action-distribution diversity than <1300 and <1000, which the authors suggest explains its failure to transcend.Entropy is measured only on common states with more than 100 actions, excluding most midgame states after move 16.Source: §4.2 · p. 8

4.3 Additional Settings

  • Across SQuADv2, temperature denoising improves reported exact-match, semantic-match, and F1 performance for tested language models.The evaluation covers models from 163M to 7B parameters and multiple temperatures.Source: §4.3 · p. 9, Figure 6 · p. 9
  • In a Gaussian-input linear-classification toy task, transcendence occurs with high expert diversity and low temperature.Source: §4.3 · p. 9

5 Related Work

The paper situates its chess testbed in a long history of chess AI and connects its theory to research on diverse learners and ensembles. It distinguishes its imitation-learning approach from offline reinforcement learning.

Chess and AI

  • Chess AI progressed from early proposals by Shannon and Turing to Deep Blue defeating Kasparov and neural approaches such as AlphaZero surpassing Stockfish; this work builds on chess models trained with Lichess data.The paper also cites transformer-based sequential chess agents as inspiration.Source: §5 · p. 10

Diversity beats Strength

  • The theory links low-temperature imitation learning to majority voting and ensemble diversity, alongside prior work on bagging, boosting, and model averaging.The related work also discusses diverse Go teams, ensemble distillation, ensemble self-training, and weak-to-strong generalization.Source: §5 · p. 10

Offline Reinforcement Learning

  • Unlike offline reinforcement learning, this work does not train on an objective to improve a fixed behavior-policy dataset and does not assume reward labels.The passage notes that reinforcement-learning objectives can introduce training instabilities.Source: §5 · p. 10

6 Discussion and Future Work

The paper presents transcendence as generative models outperforming the experts whose data they imitate, with low-temperature sampling linked theoretically to denoising and combining diverse knowledge. It reports validation in several settings while identifying limits on generalization and on claims about novel reasoning.

Discussion

  • The paper reports that low-temperature sampling can denoise expert biases and consolidate diverse knowledge, with empirical validation in chess, natural-language question answering, and toy Gaussian models.It also emphasizes dataset diversity and varied expert perspectives.Source: §6 · p. 10
  • The authors identify dataset diversity as necessary for transcendence, emphasizing the role of varied expert perspectives.This is stated as a finding about the paper’s supported setting.Source: §6 · p. 10

Limitations

  • Generalization beyond chess remains open, and the theoretical framework assumes test-time game conditions match training conditions.Extending the findings to composition or reasoning would require foregoing that assumption.Source: §6 · p. 10

Future Work

  • The authors identify practical implementations and ethical considerations for deployed generative models as future work.They say the findings lay groundwork for broader applications, not that those applications are established.Source: §6 · p. 10

Broader Impact

  • The reported denoising does not establish that low-temperature sampling yields novel abstract reasoning or solutions beyond human experts’ abilities.The authors characterize the effect as denoising errors.Source: §6 · p. 10

A Proofs

The proofs delimit when transcendence is impossible or possible under imitation learning and low-temperature sampling. They show how argmax predictions and multiple experts covering disjoint test subsets can support transcendence.

A Proofs

  • Transcendence cannot arise through imitation learning alone when all experts are sampled uniformly across the input distribution.Source: A Proofs · p. 13
  • When an argmax prediction outperforms the best expert, low-temperature sampling can enable transcendence because softmax converges to argmax as temperature approaches zero.Source: A Proofs · p. 13
  • A single expert that makes uniformly random mistakes can be transcended when its argmax prediction achieves higher reward than the expert.Source: A Proofs · p. 13
  • The paper characterizes low-temperature sampling across experts as majority voting.Source: A Proofs · p. 13
  • Multiple experts can enable transcendence when the test distribution is spread across disjoint input subsets.The proof concludes that the combined predictor equals the optimal predictor in this case.Source: A Proofs · p. 14

B Additional Denoising Visualizations

The additional chess examples illustrate how lowering sampling temperature can steer the model toward strategically stronger moves. Each example highlights a different tactical or positional consequence of the move choice.

B Additional Denoising Visualizations

  • In Figure 8, decreasing temperature raises expected reward and converges on rook to e4, the only move avoiding heavy material loss.The rook is pinned to the queen.Source: Figure 8 · p. 15
  • In Figure 9, decreasing temperature converges on queen to d4, taking a pawn and checking the black king, rather than tempting moves that lose the queen.The alternatives either lose the queen immediately or allow a pawn threat that forces it away from a free pawn.Source: Figure 9 · p. 15
  • In Figure 10, decreasing temperature converges on rook to g1, which opens the promotion square without immediate risk and can permit a later capture of the rook on a1.The alternative rook move to f1 allows White to take the rook and blocks the black pawn’s promotion.Source: Figure 10 · p. 15

C Intuition of low temperature sampling inducing transcendence

The toy example illustrates how averaging diverse expert output distributions and then sampling at a temperature below 1 can favor a correct, high-reward action. The discussion connects this mechanism to findings on label disagreement and to other forms of transcendence in trajectory models.

C Intuition of low temperature sampling inducing transcendence

  • Averaging the two experts’ distributions puts most probability mass on the correct high-reward action, and τ<1 shifts more weight toward the high-probability action to increase expected reward.Each expert individually assigns non-negligible mass to the high-reward action but samples a low-reward action most of the time.Source: Figure 13 · p. 16, Figure 14 · p. 16, C Intuition of low temperature… · p. 16

D.1 Label Disagreement

  • Studies report that training on disagreeing labels can improve models, and the authors’ theoretical claims extend this possibility to improving on the diverse labelers themselves.Xie et al. [36] report gains from adding random noise to teacher-generated labels; Uma et al. survey improvements from retaining disagreeing labels.Source: §D.1 · p. 17

D.2 Offline Reinforcement Learning

  • Decision Transformer can prompt a model to outperform the best training trajectory by conditioning on trajectory performance at inference time, a distinct form of transcendence.The passage identifies this as a promising direction for further study.Source: §D.2 · p. 17
  • In off-policy learning, setting the target policy to the exploration policy’s greedy version is equivalent to temperature 0.The passage gives choosing the argmax action of the exploration policy as an example [21].Source: §D.2 · p. 17

E Training Details

The training setup largely follows a prior hyperparameter set, with a lower batch size, and the authors provide code openly.

E Training Details

  • The batch size was lowered to 125K, at which training remained stable.The remaining hyperparameters largely follow [37].Source: E Training Details · p. 18
  • The authors release their code openly to support further research into transcendence.The code builds on work by Karvonen [12] and Karpathy.Source: E Training Details · p. 18
  • Table 2 presents the hyperparameters used for ChessFormer.Source: Table 2 · p. 18

F Compute Resources

The authors report the hardware used for training and the approximate training time per model.

F Compute Resources

G Full t-SNE

The t-SNE visualizations show that learned latent representations relate to game reward and length, although reward is not directly provided during training.

G Full t-SNE

  • Reward-colored t-SNE shows high absolute-reward states more likely near one another, suggesting the model learned some representation of reward.Reward is not directly given to the model during training.Source: G Full t-SNE · p. 20
  • The symmetric regions of high-reward states also point to evidence of an equivariant representation of player identity.Source: G Full t-SNE · p. 20
  • Games with high reward tend to be longer.The authors note that outcomes tend to become clearer as games progress.Source: G Full t-SNE · p. 21
Loading 2406.11741v4…