PaperlayerView exact revision on arXiv

Source-linked AI summary

Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?

Richard Ren, Steven Basart, Adam Khoja, Alice Gatti, Long Phan, Xuwang Yin, Mantas Mazeika, Alexander Pan, Gabriel Mukobi, Ryan H. Kim, Stephen Fitz, Dan Hendrycks

arXiv:2407.21792v3cs.LGcs.AIcs.CLcs.CY

TL;DR

AI safety benchmarks are intended to measure properties distinct from general capabilities, but the relationship between them is unclear. The paper analyzes benchmark correlations with capabilities and compute across models and finds that many benchmarks track those factors, motivating clearer empirical distinctions between safety and capability gains.

  • Problem

    Safety benchmarks are intended to measure properties distinct from general capabilities, but their relationship to those capabilities is unclear.

    Source: §1 · p. 1

  • Method

    The paper analyzes common safety benchmarks’ correlations with a general capabilities component and training compute across models.

    Source: §1 · p. 2

  • Results

    Around half of AI safety benchmarks often capture latent factors tied to general capabilities and raw training compute.

    Source: §8 · p. 21

  • Takeaways & Limitations

    New safety evaluations should report correlations with capabilities, and benchmarks can incentivize differential safety progress when they identify properties decorrelated from capabilities.

    Source: §7 · p. 21

    The critique targets abstract top-down verbal arguments about deep-learning phenomena, while speculation about AI risks can still help horizon-scan and identify failure modes.

    Source: §7 · p. 20

Abstract

from arXiv · show

As artificial intelligence systems grow more powerful, there has been increasing interest in "AI safety" research to address emerging and future risks. However, the field of AI safety remains poorly defined and inconsistently measured, leading to confusion about how researchers can contribute. This lack of clarity is compounded by the unclear relationship between AI safety benchmarks and upstream general capabilities (e.g., general knowledge and reasoning). To address these issues, we conduct a comprehensive meta-analysis of AI safety benchmarks, empirically analyzing their correlation with general capabilities across dozens of models and providing a survey of existing directions in AI safety. Our findings reveal that many safety benchmarks highly correlate with both upstream model capabilities and training compute, potentially enabling "safetywashing"--where capability improvements are misrepresented as safety advancements. Based on these findings, we propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context as a set of clearly delineated research goals that are empirically separable from generic capabilities advancements. In doing so, we aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.

1 Introduction

The paper asks how AI safety progress can be distinguished from generic capability gains when safety properties and capabilities are intertwined. It empirically tests benchmark correlations and argues for clearer, measurable safety goals.

1 Introduction

  • Safety properties and upstream capabilities are intertwined, making differential safety progress difficult to distinguish and allowing capability improvements to be misrepresented as safety research.The paper calls technical improvements that disproportionately improve safety relative to other attributes differential safety progress, and terms misrepresentation safetywashing.Source: §1 · p. 2, Figure 2 · p. 3
  • Existing top-down alignment theory and bottom-up flaw-patching approaches can both select research topics that remain entangled with latent upstream capabilities.The paper gives LLM alignment to human preferences [5] and distribution-shift robustness as examples.Source: §1 · p. 2
  • The paper measures correlations between common safety benchmarks and both a capabilities component and training compute, rather than relying on intuitive arguments.A high correlation suggests a benchmark measures capabilities as a latent upstream factor, while a low correlation alone does not establish benchmark quality.Source: §1 · p. 2
  • Across dozens of models and benchmarks, many safety benchmarks correlate highly with capabilities, so capability scaling can raise benchmark scores without establishing differential safety progress.The paper also finds intuitive alignment-philosophy arguments can be disconnected from empirical measurements.Source: §1 · p. 2
  • The paper calls for empirical clarity about AI safety as clearly delineated research goals separable from generic capabilities research.It recommends reporting safety benchmarks’ correlations with upstream capabilities.Source: §1 · p. 3

2 Related Work

Prior evaluation work developed platforms, scaling analyses, benchmark factor analyses, and downstream-capability predictions, but had not empirically meta-analyzed safety benchmarks’ entanglement with upstream capabilities. Related work frames differential safety progress as relative improvement in safety versus overall capabilities.

2 Related Work

  • Prior evaluation research covered open-source platforms, scaling, factor analysis, downstream-capability prediction, and benchmark comparisons, but had not meta-analyzed safety benchmarks’ entanglement with upstream capabilities.The passage cites open-source platforms [10], scaling [7] [8] [11] [12] [13] [14] [15] [17] [18] [19] [20], factor analysis [22], capability prediction [23] [24] [26] [27] [28] [29] [30] [31] [32], and concurrent PCA analysis [34].Source: §2 · p. 3
  • Differential safety progress means safety properties advance relative to overall capabilities, and some methods have improved robustness without necessarily increasing upstream capabilities.The passage describes such methods as targeted improvements orthogonal to the default capability-driven trajectory [36] [37] [38].Source: §2 · p. 3

3 Methods

The methods construct a general capabilities score from benchmark performance and test how safety scores relate to capabilities and training compute. They apply the analysis across language and vision models and orient safety metrics so higher scores indicate greater safety.

3 Methods

  • The capabilities score projects each model’s standardized benchmark results onto the first principal component, capturing the direction along which models vary most.The benchmark matrix has m models and b capabilities benchmarks; columns are normalized to mean 0 and variance 1.Source: §3 · pp. 3–4
  • The analysis builds model-by-benchmark scores, derives a capabilities score, and tests safety-benchmark correlations with capabilities and compute.Figure 3 summarizes the three-stage procedure; the paper describes the methodology as general for determining benchmark entanglement.Source: §3 · p. 3, Figure 3 · p. 4
  • For each safety benchmark, the method standardizes scores and computes Spearman correlation across models between safety performance and the capabilities score.Metrics are redefined so higher scores indicate improved safety; high correlation suggests capabilities tracking, while negative correlation indicates worse safety scores as capabilities rise.Source: §3 · p. 4
  • Compute correlation is Spearman correlation across models between estimated training FLOPs and safety-benchmark scores.FLOPs come from public estimates or the estimate 6 · Training Tokens · Params following [12]; estimates were available for 21 of 26 chat fine-tuned models.Source: §3 · p. 4
  • Safety metrics are oriented so higher scores mean greater safety, including converting successful-attack proportions into failed-attack proportions for adversarial robustness.This adjustment supports comparison of safety scores under a consistent direction.Source: §3 · p. 4
  • The language-model capabilities component uses 12 benchmarks, and the analysis includes diverse base and chat/instruct model classes.The component captured 78.3% of variance for base models and 70.8% for chat models; instruct fine-tuned models are the default, and results generally did not change markedly with base models.Source: §3 · pp. 4–5
  • For vision models, ImageNet accuracy serves as the capabilities component, with adversarially trained models used for robustness and standard models for calibration.The analysis uses 63 adversarially trained models and 44 standard models.Source: §3 · p. 5

4 Human Values

The human-values section evaluates alignment, machine ethics, and bias benchmarks against upstream model capabilities and compute. Findings differ across areas: alignment benchmarks track capabilities strongly, while machine-ethics and bias results vary by benchmark.

4.1 Alignment

  • Human-preference alignment benchmarks correlate highly with upstream capabilities and compute in chat models, making them liable to safetywashing.The authors note that capability gains may raise these scores, while safety-critical alignment edge cases may remain uncaptured.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 Machine Ethics

  • ETHICS has high capabilities and compute correlations, MACHIAVELLI has low correlations and slope, and sycophancy worsens as capabilities and compute increase.The authors distinguish ethical-knowledge tests from propensity-focused measures, but caution that benchmark correlations must still be measured empirically.Source: §4.2 · p. 9, Table 3 · p. 9

4.3 Bias

  • Most evaluated bias benchmarks have low capabilities and compute correlations, but BBQ Disambiguated (76.8%) and Winogender (75.6%) are exceptions.Low correlations do not by themselves establish that a bias dataset is high quality.Source: §4.3 · p. 10, Table 4 · p. 10

5 Truthfulness

Truthfulness research spans accurate answers and resistance to misconceptions, but its broad use overlaps with general capabilities. The benchmark analyses find strong capability links in misconception avoidance and scalable oversight, while calibration depends on the metric.

5 Truthfulness

  • Truthfulness benchmarks cover accurate question-answering and misconception avoidance, but the term also encompasses scalable oversight and calibration.TruthfulQA probes whether models reproduce common misconceptions or false beliefs [3].Source: §5 · p. 11, §5.1 · p. 11

5.1 Misconception Avoidance

  • TruthfulQA MC1 is highly determined by upstream capabilities and training compute, and misconception benchmarks are highly liable to safetywashing.The paper identifies low capability and compute correlations in Sycophancy and MACHIAVELLI as a hint that situational dishonest tendencies could be measured more distinctly.Source: §5.1 · p. 12

5.2 Scalable Oversight

  • GPQA and QuALITY correlate 77.7% and 88.8% with capabilities, respectively, and scalable-oversight evaluations are liable to safetywashing.The paper argues current evaluations do not isolate safety properties from capabilities, and reports that the term increasingly describes process-based feedback and math performance [84].Source: §5.2 · pp. 13–14, Table 5 · p. 13

5.3 Calibration

  • Calibration datasets assess whether models convey uncertainty accurately; a perfectly calibrated forecaster predicting 70% rain should be right on 70% of those days.The section investigates two calibration metrics.Source: §5.3 · p. 14

1. Brier Score: EX

Across models, Brier Score tracks accuracy much more closely than RMS calibration error, making it more liable to safetywashing. The paper attributes this pattern to Brier Score combining calibration and refinement, and recommends RMS calibration error instead.

1. Brier Score: EX

  • Brier Score is highly correlated with accuracy across vision and chat language models, whereas RMS calibration error is not.For language models, Brier Score accuracy correlations are 95.5% for chat models and 98.6% for base models; RMS calibration error correlations are 20.1% and 2.5%, respectively.Source: Table 6 · p. 14, §1 · pp. 14–15, Figure 11 · p. 15
  • Using Brier Score to compare calibration methods on one model may isolate calibration, but comparing models with it is misleading because it effectively proxies accuracy.The paper finds Brier Score more liable to safetywashing, while RMS calibration error is far less liable.Source: §1 · p. 15
  • Brier Score combines accuracy and calibration through its calibration-error and refinement terms, while RMS calibration error more directly measures calibration.The paper says the refinement term is minimized when accuracy is high and recommends RMS calibration error in both settings.Source: §1 · pp. 14–15

6 Security

The security analysis covers adversarial robustness and weaponization capabilities. Its results distinguish benchmarks that track general capabilities from jailbreak, gradient-based, and weaponization evaluations that appear to measure more distinct properties.

6.1 Adversarial Robustness

  • The evaluation tests six language-model robustness benchmarks and two vision evaluations, including human-written jailbreaks on HarmBench and PGD attacks on ImageNet.The PGD attack uses an ε = 8/255 budget.Source: §6.1 · pp. 15–16, AdvGLUE++ · p. 16

AdvGLUE++

  • Greater intelligence does not guarantee adversarial robustness: attackers can also improve as defenses become more powerful.The authors describe this as a persistent problem under the red queen’s hypothesis.Source: AdvGLUE++ · p. 17
  • Traditional perturbation and natural-adversarial benchmarks track capabilities, unlike jailbreak and gradient-based benchmarks; ImageNet-A correlates 97.9% with accuracy, while PGD is −41.8%.The authors report the same broad distinction across vision and language models.Source: AdvGLUE++ · p. 17, Table 7 · p. 18, Figure 13 · p. 18

Capabilities

  • Weaponization benchmarks are not prone to safetywashing; WMDP scores are strongly anticorrelated with capabilities and compute across its three fields.WMDP uses inverted scoring, so higher scores indicate more effective suppression of specific harmful capabilities.Source: Table 8 · p. 18, Capabilities · p. 19

7 Discussion

The discussion argues that benchmark scores can reward generic capability gains rather than differential safety progress, while safety priorities should account for how risks change with scale. It recommends empirically checking benchmark correlations and avoiding safety claims unsupported by differential progress.

7 Discussion

  • Capability gains can raise many safety-benchmark scores without reducing overall risk, making benchmark incentives and safety claims vulnerable to safetywashing.Benchmarks shape research effort and resource allocation, while capability gains can also increase weaponization and catastrophic malicious-use risks.Source: §7 · pp. 19–21
  • The authors argue that abstract alignment arguments are poor predictors of empirical correlations, while preserving a role for speculation in horizon-scanning and identifying failure modes.They object specifically to multi-step top-down verbal claims about deep-learning phenomena, not all alignment philosophy.Source: §7 · p. 20
  • Safetywashing can arise through sociological labeling, public-relations claims, and grant incentives that frame capability advances as safety progress.The discussion says public-relations safetywashing is especially pronounced under public pressure or regulatory scrutiny.Source: §7 · p. 20
  • For base models, capabilities score and training compute have a Spearman correlation of 96.5%.Training FLOP is approximated as 6 × params × train_tokens.Source: Figure 15 · p. 20
  • Safety priorities should anticipate which risks disappear, worsen, or emerge as capabilities scale, and favor methods that improve safety independently of scale.The discussion cautions that capabilities can improve truthfulness while also increasing weaponization and catastrophic malicious-use risks.Source: §7 · p. 21
  • The authors recommend reporting safety evaluations’ capabilities correlations, designing decorrelated benchmarks, and avoiding safety claims without differential progress.They also call for reassessing benchmark adequacy and capabilities entanglement as training techniques are integrated into models.Source: §7 · p. 21

8 Conclusion

The conclusion reports that many safety benchmarks track general capabilities and training compute, while correlations vary across safety areas. It argues for empirical measurement rather than alignment theory as a guide for machine-learning safety research.

8 Conclusion

  • Around half of AI safety benchmarks often capture latent factors tied to capabilities and training compute; alignment, scalable oversight, truthfulness, and static adversarial robustness correlate highly, while bias, dynamic robustness, and calibration correlate relatively weakly, and sycophancy and weaponization risk correlate negatively.The conclusion says upstream capabilities are difficult to avoid measuring in safety benchmarks and calls empirical measurement a replacement for alignment theory in guiding ML safety research.Source: §8 · p. 21

Contributions

The contribution statement credits the team across technical implementation, research engineering, writing, advising, and project framing.

Contributions

  • Richard and Steven led technical implementation, with Adam, Alice, Long, and Xuwang contributing as research engineers.Richard, Adam, and Mantas wrote the paper; Dan wrote the “Arguments For and Against” sections.Source: Contributions · p. 22
  • Gabe, Alex, and Stephen provided general advising, while Ryan assisted with analysis and writing processes.Source: Contributions · p. 22
  • Dan contributed the paper idea, suggested experiments, and made major framing and presentation decisions.Source: Contributions · p. 22

A.1.1 List of Language Models

The appendix documents evaluated model sets and expands the paper’s capability-correlation analyses across language and vision evaluations. It reports that strict instruction following depends more on capabilities in easier settings, while hallucinations generally decrease as capabilities improve.

A.1.1 List of Language Models

  • The appendix lists evaluated chat and base language models, vision models, and adversarially trained vision models for robustness evaluations.The listed vision evaluations include gradient-based adversarial robustness and natural adversarial examples.Source: §A.1.1 · p. 31, §A.1.2 · p. 31, §4 · pp. 32–33

A.2 Capabilities Scores

  • Capabilities scores are relative comparison measures; the appendix reports separate runs for base and chat models.Source: §A.2 · p. 34, Table 9 · p. 34

A.3 Capabilities Evaluations: Correlation Matrices

  • Many capabilities benchmarks correlate strongly with one another, with clusters suggesting some may measure similar capabilities and fine-tuning changing correlation patterns.MATH and LAMBADA show lower correlations with other tasks and may measure more distinct capabilities.Source: §A.3 · p. 35, Figure 16 · p. 35

A.4 Strict Instruction Following

  • IFEval and RuLES Basic show moderate positive capabilities correlations of 57.8% and 41.6%, while RuLES Benign and Red Team correlations are weaker.The appendix says instruction-following ability in challenging or potentially problematic contexts does not seem to improve with capabilities.Source: §A.4 · p. 36, Table 10

A.5 Hallucinations and Misconceptions: Expanded

  • As general capabilities improve, hallucinations decrease across base and chat models in both generative and classification tasks.The appendix characterizes factual reliability and hallucination reduction as generally prone to safetywashing.Source: Table 11 · p. 37, §A.5 · p. 37

A.6.3 Machine Ethics

  • MoralChoice is reported as saturated near 100% for most models, while MACHIAVELLI has a low slope across models.Source: Table 14 · p. 39

A.6.7 Adversarial Robustness

  • AdvGLUE does not significantly decorrelate from GLUE for chat models, while AdvGLUE++ does so to a small extent.Source: Table 18 · p. 40

A.7 Closed Source Model Evaluations: GPT-4o Capabilities Score

  • GPT-4o is evaluated alongside open-source models using recalculated capabilities scores, and the appendix reports a clear capabilities gap.The recalculation excludes BBH and LAMBADA from the capabilities score.Source: Figure 17 · p. 43, §A.7 · p. 43
Loading 2407.21792v3…