PaperlayerView exact revision on arXiv

Source-linked AI summary

Great Models Think Alike and this Undermines AI Oversight

Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, Jonas Geiping

arXiv:2502.04313v2cs.LGcs.AIcs.CL

TL;DR

Human preferences and annotations are costly, motivating AI oversight through model-based evaluation and supervision. This paper introduces CAPA to study model similarity and finds affinity bias in AI judges, while complementary knowledge contributes to weak-to-strong training gains.

  • Problem

    Human preferences and expert annotations are slow and expensive, motivating model-based evaluation and supervision through AI oversight.

  • Method

    The paper proposes CAPA, a probabilistic model-similarity metric based on overlap in model mistakes, and applies it to AI judges and weak-to-strong training.

  • Results

    LLM-as-a-judge scores correlate with model similarity (average Pearson r=0.84), while weak-to-strong gains increase when supervisor and student errors are less correlated.

  • Takeaways & Limitations

    AI oversight evaluations should account for affinity bias, and complementary knowledge from weak supervisors contributes beyond elicitation in weak-to-strong generalization.

  • Takeaways & Limitations

    The paper establishes correlations rather than causation, because methods to reduce similarity without harming capabilities remain an open problem.

Abstract

from arXiv · show

As Language Model (LM) capabilities advance, evaluating and supervising them at scale is getting harder for humans. There is hope that other language models can automate both these tasks, which we refer to as ''AI Oversight''. We study how model similarity affects both aspects of AI oversight by proposing Chance Adjusted Probabilistic Agreement (CAPA): a metric for LM similarity based on overlap in model mistakes. Using CAPA, we first show that LLM-as-a-judge scores favor models similar to the judge, generalizing recent self-preference results. Then, we study training on LM annotations, and find complementary knowledge between the weak supervisor and strong student model plays a crucial role in gains from ''weak-to-strong generalization''. As model capabilities increase, it becomes harder to find their mistakes, and we might defer more to AI oversight. However, we observe a concerning trend -- model mistakes are becoming more similar with increasing capabilities, pointing to risks from correlated failures. Our work underscores the importance of reporting and correcting for model similarity, especially in the emerging paradigm of AI oversight.

1. Introduction

The paper studies whether model similarity affects AI oversight, where language models evaluate outputs and provide training annotations. It finds similarity biases judging, affects weak-to-strong gains, and increases correlated failure risks as capabilities rise.

  • AI oversight uses language models for both training-data annotation and model-output evaluation as human supervision becomes slow and expensive.
  • CAPA measures functional similarity through chance-adjusted overlap in model predictions and mistakes.
  • LLM judges assign higher scores to models that are more similar to themselves, even when accounting for capability.
  • Weak-to-strong training gains are higher when weak supervisors and strong students are more different, suggesting complementary knowledge matters.
  • As frontier-model capabilities increase, their mistakes become more correlated, indicating risks from shared blind spots and failure modes in AI oversight.

2. Methodology: Measuring LM Similarity

The paper introduces CAPA to measure functional similarity while correcting chance agreement, distinguishing different mistakes and incorporating output probabilities. Simulations show CAPA reports zero similarity for independent models with uncorrelated errors.

  • Error consistency measures overlap in samples where two models err, but raw agreement can be inflated by high accuracy and can treat different wrong answers as agreement.
  • CAPA is the only compared metric satisfying all three desiderata: adjusting for accuracy, distinguishing different mistakes, and incorporating prediction probabilities.
  • CAPA uses model output probabilities to capture disagreement across incorrect options rather than relying only on top predictions.
  • CAPA adjusts observed probabilistic agreement against agreement expected from independent models with the models’ accuracies.
  • In simulations with independent, uncorrelated models, CAPA reports 0 similarity while alternative metrics increase with the second model’s accuracy.
  • A CAPA value of 0 indicates agreement matching independent models, while positive values indicate correlated errors and greater functional similarity.

3. Affinity Bias in LLM-as-a-Judge

The study tests whether LLM judges favor models that are functionally similar to themselves, using CAPA to relate model similarity to judgment scores. Across benchmarks and controls, similarity predicts higher judgments even beyond accuracy, including binary preference evaluation.

  • 3.1. Experimental Setup: Judges evaluated 8,707 free-text-compatible MMLU-Pro questions, with MCQ results used to compute CAPA across 9 judges and 39 models.Judgment scores were compared with verifiable accuracy measurements.
  • 3.2. Results & Discussion: Average Pearson r=0.84 links LLM-as-a-judge scores with model similarity across all judges, showing affinity bias beyond self- or family-preference.The relationship is significant at p < 0.01 and is observed across model families.
  • 3.2. Results & Discussion: Similarity retains a significant positive association with judgment scores after controlling for evaluated-model accuracy.Multiple regression finds significant positive effects for both accuracy and similarity, while partial correlations confirm similarity effects across judges.
  • 3.2. Results & Discussion: The similarity effect is especially large for judges smaller than 32B, while accuracy coefficients increase for more capable judges.These results indicate that accuracy and similarity contribute independently to judgment scores.
  • 3.2. Results & Discussion: Affinity bias persists across individual MMLU-Pro categories and transfers to AlpacaEval binary preferences, rather than being limited to aggregated QA scoring.The AlpacaEval result uses Elo scores derived from judges’ binary preferences.

4. Learning from Complementary Knowledge of LM Annotators

The study finds that complementary knowledge between weak supervisors and strong students helps explain weak-to-strong gains beyond elicitation. Less similar errors predict larger gains, while complementary knowledge can raise the achievable ceiling.

  • 4.1. Experimental Setup: The analysis studies 4 weak 1–3B models, 3 strong 7–9B models, 12 pairs, and 15 binary-classification tasks.The smaller models have lower pre-finetuning accuracy across tasks.
  • 4.2. Results & Discussion: Similarity is inversely correlated with weak-to-strong improvement, including after controlling for the accuracy gap.The partial correlation is r = −0.35 with p < 0.01.
  • 4.2. Results & Discussion: 42.2% average accuracy in the quadrant where the elicited strong model is wrong reflects complementary knowledge transfer, while 71.9% where the weak supervisor is wrong reflects elicitation.These values aggregate 15 tasks and 12 model pairs.
  • 4.2. Results & Discussion: The union of correct predictions from the weak supervisor and strong elicited model yields a higher ceiling than elicitation alone.This ceiling highlights complementary knowledge as a concrete avenue for improving weak-to-strong training.

5. Models are making more similar mistakes as capabilities increase

The paper examines whether model errors become more correlated as capabilities improve. Across 130 models, similarity rises with capability, creating potential risks for AI oversight through shared blind spots and correlated failures.

  • 5.1. Experimental Setup: The analysis uses 130 official models, grouped into five performance-percentile buckets, and computes within-bucket mean similarity across developers.MMLU-Pro and BBH provide broad, nonsaturated capability evaluations.
  • 5.2. Results & Discussion: Model capability has a strong positive correlation with κp, indicating that more capable models make more similar errors.The trend also holds across individual categories in both MMLU-Pro and BBH.
  • 5.2. Results & Discussion: If the trend continues, increasing similarity could amplify judge affinity bias, reduce inter-model training gains, and compromise LM-jury independence.The authors identify correlated failures as the most concerning safety implication.
  • 5.2. Results & Discussion: The correlation is not primarily explained by question difficulty, is stronger in instruction-tuned models, and is not removed by alternative Mamba architectures.These are preliminary analyses of possible explanations.

6. Related Work

Related work distinguishes representation- and function-based model similarity, prior work on judgment and weak-to-strong training, and applications of model-difference analysis. This paper contributes sample-level metrics for quantifying LM similarity and studies correlated mistakes.

  • Model similarity: Error consistency measures chance-adjusted overlap in model mistakes, motivating extensions that account for prediction differences and probabilistic outputs.The paper positions its similarity metric as a functional alternative to representation-based measures.
  • LLM-as-a-Judge: Prior work documents affinity and other biases in LM judges, while human evaluators also show biases toward similar candidates and other presentation features.These findings motivate studying similarity-based judge preferences.
  • Training on LM annotations: Weak-to-strong research shows that larger students can outperform weaker annotators, while prior work links student disagreement or model diversity to training outcomes.This paper uses supervisor–student similarity to predict gains from weak-to-strong training.

7. Conclusion, Limitations, Future Work

The paper introduces CAPA to measure functional similarity and uses it to identify affinity bias, complementary knowledge, and increasingly correlated errors in AI oversight. It concludes that similarity should be measured and accounted for, while noting causal, task-domain, and application-scope limitations.

  • CAPA measures functional similarity in language models and reveals affinity bias in AI judges, complementary knowledge in LM-annotation training, and increasingly correlated errors as capabilities improve.These findings motivate measuring similarity when evaluating or training models through AI oversight.
  • Limitations: Similarity correlates with both evaluation and training supervision, but establishing causality requires making models less similar without harming their capabilities.The paper identifies this intervention as a challenging open problem.
  • Limitations: The analysis is limited to multiple-choice tasks because precise evaluation of free-text responses remains an evolving area.Existing model-free and model-based metrics can be sensitive to syntax and style.
  • Future Work: Future work considers similarity in generator-verifier settings, safety-intervention composition, qualitative descriptions of model differences, and post-intervention decorrelation.These directions extend similarity analysis beyond the paper’s primary applications.

Impact Statement

The paper develops CAPA as a chance-adjusted probabilistic similarity metric that accounts for model accuracy, prediction probabilities, and multiple-choice option structure. It motivates this design through comparisons with existing agreement and divergence metrics.

  • CAPA extends error consistency by using prediction differences and output probabilities to quantify functional similarity while adjusting agreement for chance and model accuracy.Its observed and expected agreement are computed from model probabilities at the sample level.
  • For multiple-choice tasks, CAPA adjusts incorrect-answer agreement for the number of options instead of assuming models place all incorrect probability on the same option.The method uses a uniform prior over incorrect options, which is justified for datasets where answer options are equally distributed.
  • The framework extends to multi-model comparisons using first-principles derivation rather than Fleiss’ κ’s chance-agreement assumptions.The multi-model formulation averages pairwise observed and expected agreements across models.
  • CAPA also has formulations for classification and exact-match settings, while regression requires task-specific distance and error-distribution assumptions.For unconstrained exact-match outputs, the paper recommends a discrete CAPA variant because probabilistic agreement is difficult across multiple tokens.
  • CAPA retains differences in model accuracies when computing chance agreement, unlike Scott’s π, producing more desirable similarity behavior in the illustrated comparison.The supplied passages state that Scott’s π would treat the pairings as equally similar while κ distinguishes them.

A.6. Theoretical bounds for CAPA

Theoretical analysis bounds CAPA and characterizes how its probabilistic agreement behaves near perfect agreement and under finite-confidence predictions. The results show that CAPA approaches, but does not generally attain, its upper bound of 1.

  • CAPA’s probabilistic observed agreement is strictly greater than 0 and strictly smaller than 1 when model probabilities are positive and normalized.
  • The chance-agreement term ranges from 0 to 1, with its upper bound reached as both models’ correct-answer probabilities approach 1.
  • CAPA’s upper bound is 1, but the metric remains strictly below 1 for finite probabilities and approaches 1 as model probabilities for correct answers approach certainty.The paper argues that this does not undermine the metric because probabilistic agreement captures finer distinctions.
  • For binary probabilities, stronger and more confident agreement yields greater probabilistic observed agreement than weaker agreement, which CAPA captures more precisely than error consistency.The comparison contrasts [0.999..., 0.000...1] with [0.8, 0.2] for otherwise matching model pairs.
  • The theoretical lower bound for CAPA is -1, although achieving it in practice is impossible because probabilistic observed agreement remains greater than zero.

A.7. CAPA comparison with other inter-rater metrics

CAPA is compared with alternative inter-rater metrics in simulations of increasing agreement and disagreement. The experiments show that existing metrics miss aspects of probabilistic similarity, while CAPA has a lower-bound limitation under extreme disagreement.

  • Simulation Experiments: CAPA and competing metrics are evaluated on binary simulations covering agreement, disagreement, accuracy changes, and correlated or uncorrelated errors.The study compares five metrics using simulated binary classification data.
  • Agreement behavior: Earlier metrics fail to capture the full range of increasing similarity, whereas CAPA is designed to reflect progressively closer output distributions.Error consistency saturates when accuracies match, Jensen–Shannon similarity initially overestimates agreement, and probabilistic κ and π are ill-suited in the tested setting.
  • Limitations of CAPA: CAPA’s main limitation is that it never reaches −1 when models become maximally dissimilar, a behavior also observed for error consistency.The limitation follows from the original Cohen’s κ formulation and is considered less problematic when models are trained for accuracy.
  • Adjusted CAPA: Adjusted CAPA addresses the negative-domain limitation by changing the expected-agreement term, making the metric approach −1 as models become increasingly dissimilar.The adjustment is computed following equation 18.

B.1. Extended Multiple Regression Analysis with Model Size as a Confounder

The appendix extends the judge-bias analysis with additional comparisons and regression controls. Judgment scores remain related to model similarity, while free-form and multiple-choice evaluations largely agree but differ in difficulty.

  • Extended regression analysis: The regression analysis controls for ground-truth accuracy while testing whether judgment scores remain associated with judge–model similarity.The extended analysis additionally considers standardized model size as a possible confounder.
  • Extended similarity analysis: Judgment scores correlate more strongly with CAPA than with original error consistency, with CAPA reaching mean Pearson r = 0.9.The comparison uses filtered MMLU-Pro evaluations.
  • Open-style and Multiple Choice Correlate: Free-form accuracy closely follows multiple-choice accuracy for most models, but open evaluation is typically 5–10% lower.Qwen2.5 and Gemma-2 instruction-tuned models are exceptions because they perform particularly well in free-form responses.
  • Capability-Dependent Affinity Effects: Most judges rank stronger and weaker models correctly, yet they often label too many incorrect responses as correct, with the bias varying by judge capability.Qwen2.5-72B-Instruct identifies errors in weaker models more effectively but shows the same positivity bias for stronger models.

B.4. Judge Preference in Chat Generation Tasks Against Similarity

The chat-generation extension tests whether similarity predicts judge preferences beyond multiple-choice evaluation. CAPA similarity on MMLU-Pro positively correlates with AlpacaEval judgment Elo, and the category-level analysis shows the pattern is broadly consistent.

  • Experimental setup: The experiment evaluates 23 instruction-tuned models with nine judges over AlpacaFarm response pairs using an AlpacaEval 2.0 reimplementation.Judge-model similarities are taken from the earlier MMLU-Pro evaluation because AlpacaEval is not naturally an MCQ benchmark.
  • Free-form Judge Preference: CAPA similarity on MMLU-Pro positively correlates with judgment Elo on AlpacaEval binary preferences despite the benchmark domain gap.This provides initial evidence that similarity measured on a multiple-choice benchmark can predict judge preferences across tasks.
  • Category analysis: The category-level analysis finds a positive CAPA–judgment-score relationship across MMLU-Pro categories, indicating affinity bias is not confined to the aggregate benchmark.Except for one engineering result, category-level Pearson correlations exceed 0.75 and are statistically significant.

B.6.2. MULTIPLE REGRESSION

The regression and weak-to-strong analyses examine how similarity relates to judge scores and training gains. Similarity remains significant after accounting for accuracy, while less-similar model pairs obtain larger weak-to-strong gains, subject to dataset and model-selection boundaries.

  • Multiple regression: For Qwen2.5-7B-Instruct, similarity has a significant positive effect of β = 0.59, while the model achieves R2 = 0.91.Both similarity and accuracy are significant at p < 0.001, with F(2, 35) = 182.9.
  • Multiple regression: Similarity coefficients remain statistically significant after controlling for model accuracy, confirming affinity bias persists beyond actual capability.Across judges, similarity coefficients range from β = 0.35 to β = 1.15, while all regressions have high explanatory power.
  • Similarity versus weak-to-strong gain: Weak-to-strong gains are consistently higher on tasks where the weak supervisor and strong student are less similar.CAPA shows the highest correlation among the tested metrics, although probabilistic information adds little in the binary setup.
  • Scope boundary: The reported weak-to-strong analysis excludes several weak models that produced negative gains and excludes Hellaswag because both model groups had low accuracy.These exclusions constrain the scope of the reported training results.
  • Weak-to-strong outcomes: After weak-to-strong training, the strong student surpasses the weak supervisor but remains below the ground-truth elicitation ceiling.A union of correct predictions from the weak supervisor and strong elicited model yields an even higher ceiling.

C.5. Weak-to-strong Accuracy Value Details in Elicitation vs Complementary Knowledge Analysis

Weak-to-strong training surpasses the weak supervisor but remains below the elicitation ceiling, with complementary knowledge explaining beyond-chance gains not accounted for by elicitation.

  • Complementary knowledge transfer explains much of the beyond-chance accuracy not accounted for by elicitation.Randomly finetuning on a different training subset flips about 11% of predictions, indicating that some observed accuracy can arise by chance.
  • The weak-to-strong trained student surpasses the weak supervisor across datasets but remains below the ground-truth elicitation ceiling.Figure 17 averages accuracies across 12 model pairs.
  • The underlying accuracy tables partition test examples by whether the weak supervisor and strong elicited model make relative mistakes.Table 11 averages results across 15 datasets and reports four parts of the test distribution.
  • On the train distribution, weak-to-strong accuracy is nearly equal on only-elicitable and complementary-knowledge samples, with a slight lean toward the latter.This comparison is reported in Table 12 across the 15 studied datasets.

D.2.3. CAN CHANGING ARCHITECTURE REDUCE MODEL SIMILARITY?

Changing architecture does not uniformly reduce model similarity: models from the same family remain more similar, while instruction tuning is associated with higher similarity. Similarity increases with capability across datasets, subjects, and most tasks, though task-level BBH estimates are less reliable.

  • D.2.3. CAN CHANGING ARCHITECTURE REDUCE MODEL SIMILARITY?: Same-family Falcon3 Mamba and Transformer models show higher similarity than Falcon3 Mamba paired with different-family Mistral or Llama Transformers.The analysis suggests training data and fine-tuning may matter more than architectural differences alone.
  • D.2.3. CAN CHANGING ARCHITECTURE REDUCE MODEL SIMILARITY?: Instruction-tuned models exhibit higher similarity overall and a steeper capability-similarity trend than base models.The comparison uses matched-size or comparable-capability models and the same binning strategy.
  • Alternative similarity measures, including discrete κp and κp for M > 2, also increase with capability, while κp for M > 2 measures agreement across all models in a bin.JSD shows high variance and a flat trend, whereas error consistency increases with capability.
  • Similarity increases with capability across all MMLU Pro subjects and in most BBH tasks, so the MMLU trend is not confined to one domain.The BBH task-level trend is weaker in some tasks because each has at most 250 questions.
Loading 2502.04313v2…