PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Superhuman Models with Consistency Checks

Lukas Fluri, Daniel Paleka, Florian Tramèr

arXiv:2306.09983v3cs.LGcs.AIcs.CRstat.ML

TL;DR

As models may become superhuman, evaluating individual decisions is difficult when ground truth is unavailable or hard to assess. The paper tests decisions against human-interpretable consistency rules and finds contradictions in chess valuations and language-model forecasts.

  • Problem

    As models may become superhuman, how can their decisions be evaluated when correctness is difficult to measure directly?

    Source: §1 · p. 1

  • Method

    The paper tests related model decisions against human-interpretable consistency rules and applies the framework to chess, forecasting, and legal judgments.

    Source: §1 · pp. 1–2

  • Results

    Chess engines sometimes value semantically identical positions differently, while GPT-3.5-turbo and GPT-4 make sporting-record forecasts that fail to improve monotonically.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Proactively testing logical consistency can help ground trust in models and detect or debug failures even when the incorrect decision is unknown.

    Source: §1 · p. 1

    Inconsistencies may be rare and require adversarial search, while efficient detection of important violations remains unclear.

    Source: §8 · p. 12

Abstract

from arXiv · show

If machine learning models were to achieve superhuman abilities at various reasoning or decision-making tasks, how would we go about evaluating such models, given that humans would necessarily be poor proxies for ground truth? In this paper, we propose a framework for evaluating superhuman models via consistency checks. Our premise is that while the correctness of superhuman decisions may be impossible to evaluate, we can still surface mistakes if the model's decisions fail to satisfy certain logical, human-interpretable rules. We instantiate our framework on three tasks where correctness of decisions is hard to evaluate due to either superhuman model abilities, or to otherwise missing ground truth: evaluating chess positions, forecasting future events, and making legal judgments. We show that regardless of a model's (possibly superhuman) performance on these tasks, we can discover logical inconsistencies in decision making. For example: a chess engine assigning opposing valuations to semantically identical boards; GPT-4 forecasting that sports records will evolve non-monotonically over time; or an AI judge assigning bail to a defendant only after we add a felony to their criminal record.

1 Introduction

As models may become superhuman, evaluating individual decisions is difficult when humans cannot determine ground truth. The paper proposes testing decisions against human-interpretable consistency rules and demonstrates inconsistencies in chess, forecasting, and legal judgments.

1 Introduction

  • Consistency rules can expose errors even when decision correctness is impossible to assess: related inputs imply a logical relation that model outputs should satisfy.A violation establishes that at least one tested output is wrong, though it may not identify which one.Source: §1 · pp. 1–2
  • Superhuman chess engines Leela Chess Zero and Stockfish sometimes assign sharply different valuations to semantically identical positions, revealing novice-recognizable decision blunders.The paper frames such inconsistencies as rare but potentially severe individual-decision errors.Source: §1 · p. 2
  • GPT-3.5-turbo and GPT-4 make inconsistent forecasts, including sporting-record predictions that fail to improve monotonically over successive years.Because future outcomes are not yet knowable, the models’ accuracy requires waiting for forecast resolution dates.Source: §1 · p. 2
  • GPT-3.5-turbo sometimes assigns bail more readily to a suspect after adding crimes, despite the difficulty of assessing legal decisions against disputed or unobserved outcomes.The paper notes that disagreement can involve fairness as well as accuracy.Source: §1 · p. 3

2 Related Work

Prior work checks consistency during training and testing, studies related fairness and metamorphic-testing properties, and seeks oversight or truthfulness. This paper extends consistency checks to domains where ground truth is beyond human knowledge.

Related Work

  • Training-time semi-supervised and self-supervised methods enforce output invariances, typically when labels are expensive rather than fundamentally unknowable.Test-time work also studies invariance under natural or adversarial transformations; this paper extends such checks to broader consistency notions in domains without ground truth.Source: §2 · p. 3
  • Many fairness metrics test prediction invariance across people or populations, while metamorphic testing checks logical relations across program runs.The paper distinguishes its focus on settings where ground truth is explicitly beyond human knowledge, not merely expensive to obtain.Source: §2 · p. 3
  • Prior work finds adversarial strategies against superhuman game agents through self-play or semantically equivalent boards, whereas this paper seeks decision bugs without a better ground-truth proxy.The comparison concerns eliciting flaws in individual decisions rather than relying on end-to-end outcomes.Source: §2 · p. 3
  • Scalable-oversight methods use human-model interaction to extract confidently correct answers; this work complements them by probing for confidently incorrect, human-verifiable bugs.Source: §2 · p. 3
  • Consistency tests could help detect dishonest language-model answers under the assumption that truthful answers are easier to keep consistent.Source: §2 · p. 3

3 Consistency Checks without Ground Truth

The framework evaluates models by checking whether outputs on related inputs obey human-verifiable logical constraints, even when output correctness is unknown. A counterexample can reveal an error without proving overall consistency.

3 Consistency Checks without Ground Truth

  • The framework checks whether model outputs across related inputs satisfy human-verifiable rules, potentially showing at least one output incorrect despite unknown individual ground truth.Source: §3 · p. 4
  • A consistency rule requires that when input predicate P holds, output predicate Q holds for the corresponding model decisions.The framework defines consistency as satisfying this implication for all inputs.Source: §3 · p. 4
  • Checks can test invariance or more complex relations, such as forecasts of the 100m world record not increasing over time.A single counterexample establishes inconsistency, while proving consistency is hard for many properties.Source: §3 · p. 4
  • The framework prioritizes soundness with hard constraints that should always hold, accepting that this may find fewer bugs than relaxed checks.Relaxed violations would require further human vetting; randomized models can also be self-inconsistent across repeated calls.Source: §3 · p. 4

4 Applications Overview

The paper applies consistency checks to chess decisions, language-model forecasts, and legal predictions. Each setting reveals logical inconsistencies, and adversarial search finds more failures than ordinary in-distribution checks.

4 Applications Overview

  • The chess application tests evaluations of individual board positions and moves rather than measuring performance end-to-end across whole games.Source: §4 · p. 4
  • The forecasting application checks whether GPT-3.5-turbo and GPT-4 forecasts reflect a logically consistent internal world model.Source: §4 · p. 4
  • The legal-prediction application tests language models on detecting human-rights violations and making bail decisions.Source: §4 · p. 4
  • Across the applications, the authors find clear logical inconsistencies, and adversarial search can uncover significantly more failures than ordinary checks.Inconsistencies are rare for in-distribution data, especially for chess models.Source: §4 · p. 4

5 Superhuman Chess AIs

Chess engines can play at superhuman levels yet violate simple invariances, revealing severe errors in individual evaluations. Consistency failures persist across Leela and Stockfish, and adversarial search finds more of them than random search.

5.1 Logical Consistency Checks in Chess

  • Chess checks compare evaluations after forced moves, board transformations where pawns and castling do not interfere, position mirroring, and the model’s recommended move.For the recommended-move check, small discrepancies from reduced uncertainty as the game progresses are not counted as failures.Source: §5.1 · p. 5

5.2 Experimental Setup

  • Leela evaluations use 400 MCTS nodes by default, and experiments test 400,000 database positions plus 200,000 synthetic pawnless positions for board transformations.The evaluation predicts the expected game outcome under optimal play on a scale from -1 to 1.Source: §5.2 · p. 5

5.3 Results

  • Leela’s rotation and mirroring failures change winning chances by up to 69%, and its recommended Re8 move drops Black’s estimated win probability from 68% to 0%.The forced-move and recommended-move examples also show drastic evaluation changes; in a Master-level game, Re8 offered White a mating opportunity.Source: §5.3 · pp. 5–6, Figure 2 · p. 6
  • An 8× increase to 3,200 search nodes reduces failures only 3–6.6×, while adversarial search finds up to 9× more board-transformation failures than random search.The second adversarial run found an evaluation difference larger than 1, exceeding anything found by random search.Source: §5.3 · pp. 6–7, Table 2 · p. 7

5.4 Consistency Tests for Other Chess AIs

  • Stockfish is consistent on average, with most evaluation differences at or below 0.25, but still exhibits multiple failures across all tested constraints.Stockfish results cannot be directly compared with Leela’s evaluation differences because of technical differences.Source: §5.4 · p. 7

5.5 Summary

  • Across the chess experiments, consistency tests reveal humanly verifiable failures in superhuman models, and consistency levels vary across designs without necessarily tracking standard performance measures.The authors also report that adversarially guided search may be needed to uncover the most pernicious bugs.Source: §5.5 · p. 7

6 Forecasting Future Events with Large Language Models

Forecasting future events has inherently unknown ground truth, so the paper tests GPT models against logical consistency rules instead. Both models show strong violations, and prompting improves negation consistency without reliably improving other checks.

6.1 Logical Consistency Checks in Forecasting

  • Forecasting checks test complementary event probabilities, paraphrase invariance, monotonic change over time, and whether conditional probabilities satisfy Bayes’ rule.For example, the number of people who have climbed Everest should not decrease over time, while the men’s 100m record should not increase.Source: §6.1 · p. 8

6.2 Experimental Setup

  • The benchmark contains 380 forecasting questions and 1,220 variants, with violation metrics normalized to [0, 1]; experiments use GPT-3.5-turbo and GPT-4 at temperatures 0 and 0.5.The study samples negation, paraphrasing, monotonicity, and Bayes’ rule cases, using repeated runs and median forecasts to reduce variance.Source: §6.2 · pp. 8–9

6.3 Results

  • GPT-3.5-turbo and GPT-4 show many strong forecasting violations; GPT-4 is more consistent except on Bayes’ rule, and stochasticity accounts for less than 20% of strong violations.Strong violations exceed ε = 0.2; the authors conclude the forecasts are inherently unreliable despite being unable to verify their correctness.Source: Figure 3 · p. 8, Table 3 · p. 8, §6.3 · p. 9

6.4 Prompting for Consistency

  • Negation prompting improves both models’ negation metrics, with GPT-4 nearing the non-adversarial threshold, but paraphrasing worsens slightly and Bayes’ rule changes little.The authors treat this as evidence that improving one consistency check need not improve consistency in general.Source: §6.4 · p. 10
  • Prompting did not significantly improve the full paraphrasing check, possibly because the model failed to derive a shared canonical paraphrase or the instructions confused it.The authors note that paraphrasing has more degrees of freedom than negating a question and may therefore be harder to prompt or train for.Source: §6.4 · p. 10

7 Legal Decision-making

The section tests legal-model consistency through paraphrasing case facts and checking whether bail decisions worsen when criminal records improve. The checks reveal decision flips, including approval of bail after a suspect’s record is made worse.

7.1 Logical Consistency Checks in Legal Decisions

  • The legal checks test whether paraphrasing changes a case judgment and whether a bail model favors a defendant after their record becomes worse.The section presents these as two consistency checks despite the difficulty of assessing legal correctness.Source: §7.1 · p. 11, §7 · p. 11

7.2 Experimental Setup

  • For human-rights judgments, the study fine-tuned a binary classifier on ECHR cases and tested random-fact and most-important-fact paraphrases verified as semantically unchanged.The evaluation sampled 500 cases from the ECHR test set and used GPT-3.5-turbo to paraphrase facts; importance was measured using the model’s final attention layer.Source: §7.2 · p. 11
  • For bail judgments, GPT-3.5-turbo assessed ProPublica COMPAS suspects, then was tested on counterfactual records made worse or better by changing crime severity or prior-crime counts.The study created 10 counterfactual suspects for each of 1,560 suspects and tested temperatures 0 and 0.5 for cases where a worse record led to bail approval.Source: §7.2 · p. 11

7.3 Results

  • Randomly paraphrasing a case fact usually preserved the human-rights decision, with flips occurring only for predictions near 50%.Some decisions still flipped under the random-paraphrase check.Source: §7.3 · p. 11
  • Paraphrasing the fact the model considers most important flipped 50% of non-violation predictions; flips in the opposite direction occurred in 7%.Random-fact paraphrases produced fewer violations, with flips limited to predictions near 50%.Source: §7.3 · p. 11, Figure 5 · p. 12
  • At temperature 0, GPT-3.5-turbo changed from denying bail to approving it in 78 of 1,560 cases (5%) when given an objectively worse record.Figure 6 illustrates approval after an additional crime; the authors suggest the lower violation count may reflect the small number of input features, and note these decisions are not necessarily legally correct.Source: §7.3 · p. 12, Figure 6 · p. 12

8 Limitations and Future Outlook

The authors identify limits in finding, validating, and interpreting consistency violations. In particular, the checks may miss bugs and do not account for multiple fixed points in performative settings.

Efficiency

  • Random black-box search may miss rare inconsistencies, and it remains unclear whether important violations can be found efficiently even with adversarial search.White-box search reveals more violations, but polynomial verifiability in the limit does not establish efficient detection.Source: §8 · p. 12

Soundness

  • Automated generation of supposedly consistent inputs can produce unsound checks, and manual review may miss cases where paraphrases describe different events.Relaxing soundness could improve completeness, but discovered bugs would then need further vetting.Source: §8 · p. 12

Feedback loops

  • The framework is not ready-made for performative predictions because it queries model predictions in parallel and does not account for multiple distinct fixed points.The authors identify this issue as critical when testing superhuman models used for high-stakes decisions.Source: §8 · pp. 12–13

False negatives

  • Passing consistency checks does not establish trustworthiness: this incomplete bug-finding technique can miss bugs.The authors caution against absolutely trusting future, better models merely because they pass simple checks.Source: §8 · p. 13

A Costs and Compute

The appendices document experimental costs and Leela’s consistency checks, setup, and search-scaling results. Increasing search strength reduces failures, but not proportionally to the increase in nodes.

A Costs and Compute

  • Leela experiments used 8 NVIDIA RTX A6000 GPUs and 73.5 single-GPU run-days; forecasting and bail experiments cost under $2,000 in API tokens.ECHR paraphrase generation cost under $100.Source: A Costs and Compute · p. 19

B.1 Examples of Consistency Checks

  • Leela checks board transformations and mirroring for equal evaluations, and forced and recommended moves for evaluations that are exact negatives.The sign reversal reflects Leela scoring from the perspective of the player to move.Source: §B.1 · p. 19, Figure 7 · p. 19

B.2 Leela Chess Zero Experimental Setup

  • The Leela experiments used 400,000 master-game positions for three checks and synthetic eight-piece positions for board transformations.The transformation test requires positions without pawns or castling rights; the master-game positions were selected using four stated middle-game criteria.Source: §B.2 · pp. 19–20

B.3 Additional Leela Chess Zero Results

  • An 8× increase to 3,200 MCTS nodes reduced recommended-move failures by only 3–6.6×, with log failure counts scaling almost linearly with log node count.The scaling pattern held across inconsistency thresholds.Source: §B.3 · p. 21, Table 6 · p. 21, Figure 8 · p. 22

B.4 Stockfish Experimental Setup

The Stockfish setup tests whether consistency checks generalize across a different chess engine and evaluation functions. Its scores are transformed for evaluation, but direct comparison with Leela is not possible.

B.4 Stockfish Experimental Setup

  • Stockfish uses principal variation search and supports both NNUE and classical handcrafted evaluations, enabling a test of whether the consistency method generalizes.The experiments reuse Leela’s data and checks.Source: §B.4 · pp. 22–23
  • The classical Stockfish experiment tested 200,000 positions rather than 400,000 because its 4.1-million-node PVS search required more resources.The classical and NNUE node counts were tuned to match Leela’s strength; the classical evaluation required about 4,100,000 nodes, versus about 81,000 for NNUE.Source: §B.4 · pp. 22–23
  • Stockfish centipawn scores were transformed into win-draw-loss probabilities and then Leela-style q-values because centipawn differences do not map linearly to win probabilities.For example, the score difference between 200 and −200 can have a different practical meaning from the same-sized difference between 200 and 600.Source: §B.4 · p. 23
  • Direct evaluation-gap comparisons between Stockfish and Leela are impossible because their scoring policies differ.Leela assigns ±1 only for a forced checkmate, whereas Stockfish can output 100% win or loss probabilities at sufficiently high confidence.Source: §B.4 · p. 23

B.5 Additional Stockfish Results

Both Stockfish evaluation versions show consistency failures despite generally small evaluation differences. The classical version is more consistent than NNUE at its tuned strength, while reducing its search nodes removes that advantage.

B.5 Additional Stockfish Results

  • Most Stockfish positions had evaluation differences ≤0.25, but all tested constraints still exposed failures, including a larger fraction of extreme cases than for Leela.The authors attribute part of the larger extreme-failure fraction to Stockfish’s inflated evaluation differences, which also suggests its score-to-win-probability mapping is not calibrated.Source: §B.5 · p. 25
  • The weaker classical Stockfish evaluation was more consistent than the modern NNUE evaluation in the reported comparison.The paper presents examples of strong failures for both versions.Source: §B.5 · p. 25, Figure 11 · p. 26, Figure 12 · p. 27
  • At 81,000 PVS nodes, classical Stockfish had roughly equal or worse violation counts than at its higher node count, and performed much worse on board transformations than NNUE.The authors take this as slight evidence that more search nodes matter more for consistency than a well-trained evaluation function.Source: §B.5 · p. 25

C.2 Examples of Forecasting Consistency Checks

  • The forecasting check examples pair event and negation questions, paraphrases of the same event, record forecasts across years, and conditional election outcomes.The negation pair is consistent when probabilities sum to one; the Bayes’ rule example uses products of unconditional and conditional forecasts.Source: §C.2.1 · p. 30, §C.2.2 · p. 30

C.2.3 Monotonicity; C.2.4 Bayes’ Rule

  • The monotonicity examples ask for the men’s 100-meter record in 2025, 2028, 2032, 2036, and 2040; the Bayes’ rule examples compare election and popular-vote forecasts.The Bayes’ rule example is consistent when f(q1)f(q3) equals f(q2)f(q4).Source: §C.2.3 · p. 30, §C.2.4 · p. 30

C.3 Additional Results

Additional forecasting analyses compare model consistency, test repeat-question baselines, and examine limits of the checks. GPT-4 is more consistent than GPT-3.5-turbo on all checks except Bayes’ rule, while generated test questions may make results optimistic.

C.3.1 Violation Histograms

  • GPT-4 was more consistent than GPT-3.5-turbo on every forecasting check except Bayes’ rule, and temperature showed no significant effect.Figure 13 presents violation-metric histograms by check, model, and temperature.Source: §C.3.1 · p. 31, Figure 13 · p. 31
  • GPT-3.5-turbo’s negation violations had a heavy tail of very high scores, often caused by misunderstanding or misreporting negation; this pattern was absent in GPT-4.These failures produce high violation scores when the original-event probability is far from 50%.Source: §C.3.1 · p. 32

C.3.2 Baselines and Controlling for Randomness

  • Sampling each query three or six times and taking the median controls for randomness but does not eliminate it.The authors consider random outputs inconsistent for evaluating high-stakes models because such outputs are inherently unreliable.Source: §C.3.2 · p. 32
  • Only 6% of repeated-identical-question tests were strong violations above ε = 0.2, versus around 30% for paraphrasing tests.The baseline repeated the same question four times instead of testing four phrasings.Source: §C.3.2 · p. 32, Figure 15 · p. 32
  • The study tests monotonicity of median forecasts, stronger than requiring some consistent points within prediction intervals but weaker than requiring the full intervals to be consistent.Even the weaker existence-of-points criterion still reveals GPT-4 inconsistencies in the illustrated tests.Source: §C.3.2 · pp. 32–33, Figure 16 · p. 32

C.3.3 Discontinuities in Predicted Probabilities

  • Probability forecasts often use multiples of 5%, which can create up to 0.05 irreducible inconsistency, but the observed GPT-4 violations are too large for rounding to explain them.The authors relate the rounding pattern to probability tokens such as 50% being more common in training data than 51%.Source: §C.3.3 · p. 33

C.4 Generating Consistency Checks for GPT-4 Using GPT-4

  • Some GPT-4 test questions were generated partly or wholly by GPT-4, so the authors regard results as conservative because the model may perform better on questions from its own output distribution.A model’s failure on such a test remains a bug, but passing may be a false positive if the question is already familiar.Source: §C.4 · p. 33

C.5 Consistency Prompting

  • The authors make no claim that their prompting methods are optimal, and fixed-prompt benchmarks can underestimate a model’s best possible performance.The appendix illustrates canonical-paraphrase and negation prompts with Senate forecasts.Source: §C.5 · pp. 33, 35, 36

D.1 Experimental Setup

The appendices test ECHR case classifications under single-fact paraphrases and bail decisions under severity-changing counterfactuals.

D.1 Experimental Setup

  • The ECHR model uses pre-trained legal-BERT-sc to encode case facts, then classifies them with a fine-tuned self-attention and linear head.The authors report comparable performance metrics to the cited setup while requiring less fine-tuning compute.Source: §D.1 · p. 37
  • GPT-3.5-turbo generates paraphrases intended to preserve each legal fact’s information and formal style.The appendix provides the generation prompt and an example paraphrase.Source: §D.1 · p. 37, §D.2 · p. 38
  • Random-fact tests exclude facts under 120 characters and the common first fact, then run 30 tests per case; salient-fact tests paraphrase the attention-highest fact three times.Each individual test changes only one case fact.Source: §D.1 · p. 37

D.2 Additional Results

  • Figure 18 shows two legal cases where paraphrasing one fact, while leaving the others unchanged, flips the model’s classification.The figure marks removed words in red and added words in green.Source: §D.2 · p. 37, Figure 18 · p. 39

E.1 Experimental Setup

  • Bail counterfactuals make cases less or more severe by changing charges or counts of prior and juvenile crimes.For each case, five variants in each direction are created by changing one or two sampled features.Source: §E.1 · p. 39

E.2 Additional Results

In bail prediction, models sometimes violate severity-consistency checks, although blatant decision flips are rare. The appendices frame consistency testing as an evaluation tool, not a safety guarantee.

E.2 Additional Results

  • 78 of 1,560 cases showed wrong-direction outputs at temperature zero; blatant bail-decision flips were 0.1–0.6%.The reported 78 violations count wrong-direction outputs on counterfactual defendants; the table presents results by experiment and violation type.Source: §E.2 · pp. 39–40, Table 16 · p. 40
  • The authors attribute fewer bail-task violations partly to its small feature space, which may make simple, mostly consistent rules easier to learn, though not necessarily legally correct ones.Counting deferred or undecided outputs reveals more inconsistencies.Source: §E.2 · p. 40

X-Risk Sheet

  • The safety-risk sheet cautions that its individual answers do not decisively establish the work’s relevance to existential-risk reduction.Source: X-Risk Sheet · p. 41

Long-Term Impact on Advanced AI Systems

  • The authors present consistency checks as a possible tool for detecting deceptive behavior, while stressing that the paper offers no safe-system design recommendations.They describe possible future checks on AI statements about behavior and mistakes that humans may not readily detect, including for systems controlling critical infrastructure.Source: Long-Term Impact on Advanced AI Systems · p. 41

Safety-Capabilities Balance

Elaborations and Other Considerations

Related papers

Loading 2306.09983v3…