PaperlayerView exact revision on arXiv

Source-linked AI summary

Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Christopher M. Stewart, Preston Botter, Natalie Sarabosing, Muye Zhang, Rachel Phinnemore, Shalini Ghosh, Hong Shen, Hoda Heidari

arXiv:2610.12409v1cs.AI

TL;DR

Safety benchmarks are often read as measuring attributes, although it may be unclear whether a dataset isolates one. This paper tests harmful refusal in HELM Safety with construct-validity and psychometric analyses; three candidate datasets are saturated, and HarmBench’s responses are not explained by a single dimension.

  • Problem

    A composite safety score can mask sharp differences in specific behaviors, while it is often unclear whether even a component dataset measures one attribute such as harmful refusal.

    Source: §1 · p. 2

  • Method

    The paper applies construct-validity tests to HELM Safety’s candidate harmful-refusal datasets, then uses dimensionality modeling and ability-matched differential item functioning to audit HarmBench.

    Source: §1 · p. 2, §2 · p. 3, §3.2 · p. 5

  • Results

    HarmBench needs more than one dimension: confirmatory 3D reduces held-out log-loss by 0.064 on average relative to the strongest unidimensional 3PL.

    Source: §1 · p. 2, §4 · p. 7

  • Takeaways & Limitations

    HarmBench’s composite score cannot be read as measuring a single construct like harmful refusal in the current model pool, so a score should earn that interpretation before comparison.

    Source: §5 · p. 9, §6 · p. 9

    The model pool is a convenience sample clustered by developer, so the DIF finding describes these models at this snapshot rather than either developer generally.

    Source: §7 · p. 9

Abstract

from arXiv · show

Safety benchmarks typically report one overall score for a suite of datasets, each of which may target one or more safety-related attributes, so models with similar overall scores can have very different attribute profiles. Comparing models is more tractable at the level of individual attributes, yet it is often unclear whether even a single dataset's scores isolate any single attribute. One plausible candidate for such an attribute is harmful refusal, a model's tendency to refuse dangerous or policy-violating prompts. We examine whether it constitutes a single, measurable attribute in HELM Safety. Using a construct validity framework that stipulates that an attribute must exist before a test can measure it, we start with HELM Safety's four datasets that might plausibly target harmful refusal, but find that three are saturated. We subject the remaining dataset, HarmBench, to two psychometric tests to determine if a single attribute like harmful refusal could stand behind its score. First, multidimensional item response theory modeling strongly suggests that HarmBench does not measure a singular attribute. Second, a differential item functioning analysis finds items where models from different developers with the same refusal ability score differently. These flags largely disappear under scope-specific matching, a pattern consistent with aggregation effects but not sufficient to rule out domain-specific developer differences. Zooming out, HarmBench collapses distinct harm behaviors into one score, and the overall HELM safety aggregate further collapses HarmBench and scores from other datasets into a single top-line number. Any safety score that averages over datasets and items can hide saturation and conflate behaviors this way. We argue that a score should earn its single-attribute reading before models are compared with it.

1 Introduction

A composite safety score can rank models without supporting claims about any specific safety attribute. This paper asks whether harmful refusal is a single construct measurable in HELM Safety, and tests whether HarmBench’s responses are organized by it.

1 Introduction

  • Under Borsboom’s construct-validity framework, a test measures a construct only if the construct exists and causally produces variation in its outcomes.The paper treats harmful refusal as a hypothesized construct, not one explicitly proposed by HELM Safety’s developers.Source: §1 · pp. 1–2
  • Two models can share an overall safety score yet differ sharply on a specific behavior, so a composite ranking cannot show which model is safer on that attribute.Safety decisions often require component-specific claims that a single overall score cannot support.Source: §1 · p. 2
  • Three of HELM Safety’s four candidate harmful-refusal datasets have mean pass rates of 0.92–0.94, leaving only HarmBench with enough variation for item-level modeling in this pool.Saturated datasets may still measure the construct well for weaker models.Source: §1 · p. 2
  • HarmBench’s responses are not organized by a single harmful-refusal dimension, and developer-linked item differences lessen under scope-specific matching without excluding domain-specific differences.The other three candidate datasets are saturated in the HELM Safety model pool, leaving HarmBench as the dataset with enough variation for item-level testing.Source: §1 · p. 2

2 Construct validity tests

A HarmBench score supports a claim about harmful refusal only if one construct connects item responses to that claim. The section sets out two psychometric tests of that warrant: dimensionality within the response matrix and item differences across developers matched on ability.

2 Construct validity tests

  • The score is a tally of item pass/fail responses; reading it as harmful refusal requires the warrant that one construct drives those responses.If several constructs drive the responses, the score still exists but measures a blend rather than harmful refusal alone.Source: §2 · p. 3
  • The dimensionality test asks whether one latent dimension accounts for pass/fail patterns; needing several suggests the score combines constructs rather than measuring one.The theoretical construct is the intended attribute, while a latent dimension is the statistical axis estimated from responses.Source: §2 · p. 3
  • The DIF test compares item responses from models matched on refusal ability; developer-linked differences after matching indicate construct-irrelevant variance, not which developer is safer overall.The test is conditional: overall safety differences alone do not constitute differential item functioning.Source: §2 · pp. 3–4
  • Item response theory estimates item difficulty and discrimination to distinguish different constructs from differences in how difficult items are.The same estimated ability provides a basis for comparing developers’ models.Source: §2 · p. 4

3 Data and methods

The methods prepare a binary model-by-item response matrix, then test dimensionality and developer-linked item differences using item response theory. The analyses include exploratory and confirmatory models, held-out predictive comparisons, and conditional DIF screens.

3.1 Data and scope

  • Strictly requiring a unanimous perfect judge score for a pass removes 2 of 400 HarmBench items; the resulting pass rate is 0.67.Each item’s continuous score averages two LLM judges applying a five-point rubric.Source: §3.1 · p. 4
  • HarmBench classifies items by response process—standard, contextual, or copyright—and by seven semantic harm domains, with the two taxonomies overlapping but not perfectly.All 100 copyright items share both the functional and semantic copyright classifications.Source: §3.1 · p. 4

3.2 Models and estimation

  • The primary confirmatory model assigns standard, contextual, and copyright items to three factors; a seven-factor model based on semantic harm domains provides a sensitivity check.Both models estimate factor correlations to assess whether their named factors are meaningfully distinct.Source: §3.2 · p. 5
  • Five repeated 80/20 response-level splits compare held-out log-loss as the primary criterion and Brier score as a secondary calibration check.The matrix has 81 models and 398 items; the analysis uses binary 2PL as the primary, simpler response model.Source: §3.2 · p. 5
  • A Bernoulli-null eigenvalue check and exploratory 1D–10D 2PL models test whether extra latent dimensions predict held-out responses better than one scale.The exploratory models use no preassigned item-to-factor labels and do not themselves name the constructs.Source: §3.2 · pp. 4–5
  • DIF screens condition on estimated ability: Mantel-Haenszel compares groups within three ability bands, while ridge-penalized logistic regression uses continuous ability as a sensitivity check.Developer groups are restricted to OpenAI and Anthropic, with 21 and 11 models respectively; the screens also compare closed/API-accessed with open-weight-like models.Source: §3.2 · pp. 5–6
  • The small, dependent developer groups leave group-specific item estimation, anchor selection, and uncertainty calibration unestablished; graded-response model recovery also needs validation beyond predictive fit.The authors retain simpler DIF screens rather than multiple-group IRT with anchor purification, and frame the response model choice as limited scope rather than a solution to these issues.Source: §3.2 · pp. 5–6

4 Results

Across the HELM Safety v1.17.0 model pool, HarmBench responses are not organized by one harmful-refusal dimension. Developer-linked item flags appear under single-score matching but mostly disappear under scope-specific matching, with that reduction not ruling out genuine developer differences.

Dimensionality

  • Several later eigenvalues exceed the one-factor random-data cutoff, while exploratory 2D–10D models improve held-out log-loss over 1D but do not identify a stable best dimensionality or name the constructs.The 1D 2PL log-loss is 0.470 (SD 0.003), compared with about 0.28 for every exploratory model from 2D through 10D.Source: §4 · p. 6
  • Confirmatory 3D reduces held-out log-loss by 0.064 (SD 0.003; range 0.060–0.067) versus the strongest unidimensional 3PL, with all five splits favoring 3D.Brier score falls from 0.096 (SD 0.002) to 0.078 (SD 0.002), a paired reduction of 0.0176 (SD 0.0005); the 7D model is slightly better on log-loss, but 3D uses fewer fitted quantities and wins AIC and BIC.Source: Table 1 · p. 6, §4 · pp. 6–7
  • The confirmatory factors are related but not interchangeable: standard and contextual correlate at 0.785, while copyright correlates with them at 0.451 and 0.603.Non-copyright harm domains in the 7D model are generally more tightly correlated, often about 0.65–0.85, while copyright correlations range from 0.34 to 0.64.Source: §4 · p. 7
  • Without copyright items, the 2D standard/contextual model beats 1D in all five splits, with held-out log-loss 0.278 (SD 0.007) versus 0.293 (SD 0.009), supporting residual dimensional structure without establishing two substantive constructs.Rasch/1PL variants also do not rescue a one-dimensional score, and 3PL variants do not improve the preferred confirmatory 2PL models on held-out log-loss.Source: §4 · p. 7

Differential item functioning

  • Under the single score, matched OpenAI and Anthropic models differ on 13 items by Mantel-Haenszel and 17 by the logistic check; flags fall to 1 and 2 under 3D matching.The 7D screens are similar in substance, with residual flags concentrated in cybercrime/intrusion and misinformation/disinformation; provenance groups show no family-wise flags despite mean scores of 0.776 versus 0.507.Source: §4 · pp. 7–8, Table 2 · p. 8
  • These results apply to the HELM Safety v1.17.0 model pool, and the reduction in DIF flags is consistent with aggregation effects but does not establish their cause.Scope-specific ability estimates use the same tested items and can absorb domain-specific developer differences, so the findings do not show that genuine developer differences are absent.Source: §4 · pp. 6, 8

5 Discussion

HarmBench’s composite score cannot be read as a single harmful-refusal measure: its items reflect distinct behaviors, and its response patterns require more than one dimension. The authors recommend keeping behaviors separate and note limits on what their analyses establish.

5 Discussion

  • The two psychometric checks can be run on benchmarks with item-level responses, using the response matrix for dimensionality and model-family grouping for ability-matched DIF.The authors say the checks are cheap and fast enough to run on any model release.Source: §5 · p. 8
  • The dimensionality evidence rejects one dimension but does not determine an exact count; three- and seven-dimension models predict about equally well.The authors report these as the most parsimonious structures that capture the separation.Source: §5 · p. 8
  • Copyright items mix refusal disposition with memorization: reproducing protected text depends mostly on training memorization, not judging a request harmful and declining.The authors describe this separation as consistent with what the construct predicts once the label is taken apart.Source: §5 · p. 9
  • For the current model pool, HarmBench’s composite score cannot be read as measuring a single construct, so safety scores should keep behaviors separate.HELM Safety’s harmful-refusal signal rests on HarmBench, the only dataset still separating models in its latest pool.Source: §5 · p. 9

6 Conclusion

The conclusion reports that the HELM Safety datasets relevant to harmful refusal do not support a single-attribute reading: three are saturated, and HarmBench shows multidimensional responses. A score should earn its single-attribute interpretation before it is used to compare models.

6 Conclusion

  • HarmBench responses require more than one dimension, while developer-linked DIF flags lessen under scope-specific matching without ruling out domain-specific differences.Three of the four HELM Safety datasets plausibly targeting harmful refusal were saturated.Source: §6 · p. 9
  • The conclusion treats harmful refusal as a candidate construct but says it could not be found in the HELM Safety dataset.The remaining dataset examined was HarmBench.Source: §6 · p. 9
  • A score intended to measure one attribute should earn that interpretation before it is used to compare models.For HarmBench, the supported reading is the narrower one that keeps behaviors apart.Source: §6 · p. 9

7 Limitations

The evidence is bounded by the model sample, scoring procedure, and benchmark setting. The authors also note uncertainty from judge-based item scores and thresholding.

7 Limitations

  • The 81-model convenience sample is clustered by developer, with small groups including 21 OpenAI and 11 Anthropic models, so DIF findings describe this snapshot rather than either developer generally.Shared training pipelines also mean within-developer models are not independent examinees and may inflate observed developer DIF.Source: §7 · p. 9
  • The conclusions apply to HELM’s operationalization of HarmBench; the original classifier-based attack-success pipeline was not used for replication.The item-level responses, dimensional structure, and DIF pattern could differ under the original pipeline.Source: §7 · p. 9
  • Item scores come from two language-model judges, binarize a five-point rubric at a strict threshold, and cover one benchmark at one release.Judge biases may affect the response matrix, and another cutoff could surface different gradations.Source: §7 · p. 9

A Software, seeds, and reproducibility

The analyses use Python packages for variational MIRT estimation, data handling, statistical tests, alignment, and figures. The appendix documents the fitting and restart procedure, with code and HELM Safety data available online.

A Software, seeds, and reproducibility

  • The analysis uses py-irt, pyro, numpy, pandas, scipy, and matplotlib; specific package versions are listed in the code repository’s requirements file.The notebook and supporting code are available in the repository, and HELM Safety data are downloaded from Stanford CRFM on first run.Source: A Software, seeds, and reproducibility · p. 11
  • Across five splits, every confirmatory 3D restart had lower held-out log-loss than every unidimensional 3PL restart.Mean within-split SD of held-out log-loss across starts was 0.00037 for unidimensional 3PL and 0.00035 for confirmatory 3D 2PL.Source: A Software, seeds, and reproducibility · p. 11
  • Each model–split fit used 20 random initializations and 2,000 epochs of stochastic variational inference, retaining the restart with highest training log-likelihood.NaN-loss fits were retried at learning rate 0.005 for 3,000 epochs; held-out responses were not used to select restarts.Source: A Software, seeds, and reproducibility · p. 11

B Bernoulli-null eigenvalue check

The Bernoulli-null check finds several later HarmBench eigenvalues above a matched-random cutoff, arguing against a clean one-dimensional interpretation. The robustness analyses compare response-process and harm-domain partitions under a shared evaluation protocol.

B Bernoulli-null eigenvalue check

  • Several later observed eigenvalues exceed the matched-Bernoulli cutoff, screening against a clean one-dimensional interpretation.The main-text MIRT comparisons provide stronger model-based evidence and interpretable factor structures.Source: B Bernoulli-null eigenvalue check · pp. 11–12

C Model-family and content-partition robustness

  • Robustness checks vary the item-response family and item partition to test whether the main conclusion survives more constrained, flexible, or taxonomy-aligned models.The purpose is not to select the most elaborate model based on one predictive metric.Source: C Model-family and content-partition… · p. 12

C.1 Models compared

  • Exploratory multidimensional 2PL models let items load on every dimension, testing predictive gains but yielding rotationally ambiguous, less interpretable factors.The confirmatory 3D 2PL instead fixes items in advance to standard, contextual, or copyright response-process factors; the 7D sensitivity model assigns items to HarmBench’s seven harm domains.Source: §C.1 · pp. 12–13

C.2 Evaluation protocol

  • The evaluation uses the same response matrix and five repeated 80/20 response-level splits, with held-out log-loss primary and Brier score secondary.Training-set AIC and BIC serve as parsimony checks.Source: §C.2 · p. 13

C.3 Results

Robustness checks support the conclusion that one dimension is too weak to represent HarmBench, while the three-factor response-process structure captures much of the dependence. The section also details ability-matched developer and access-type comparisons using Mantel-Haenszel and logistic DIF screens.

C.3 Results

  • Held-out log-loss falls from 0.330 for unidimensional 1PL to 0.263 for confirmatory 3D 1PL, so multiple dimensions help even with equal item slopes.The unidimensional 1PL is a floor rather than a serious competitor to the response-process models.Source: §C.3 · p. 13
  • Confirmatory 3D 2PL has held-out log-loss 0.258 versus 0.261 for 3D 3PL, while exploratory 4D 3PL reaches Brier 0.076 without improving held-out log-loss or information criteria.The 4D variant adds thousands of fitted quantities, leaving the simpler confirmatory 3D 2PL preferred among the 3PL variants.Source: §C.3 · p. 13
  • The 7D model records log-loss 0.255 and Brier 0.077 versus 0.258 and 0.078 for 3D 2PL, but 3D 2PL has lower AIC and BIC.AIC/BIC are 13,707 and 22,183 for 3D 2PL versus 13,961 and 25,080 for 7D.Source: §C.3 · p. 13, Table 3 · p. 13
  • Across model-family and partition checks, one dimension remains too weak, while the standard/contextual/copyright structure captures much of the dependence left behind by a single score.The seven-domain model yields a small predictive gain but not enough to displace the more parsimonious three-factor structure.Source: §C.3 · p. 14

D Differential item functioning details

  • DIF tests condition on fitted ability: under a single-score interpretation, equally able models should have equal item pass probabilities regardless of developer or access type.OpenAI versus Anthropic and closed/API versus open-weight-like models are the pre-specified comparisons; raw means are descriptive before matching.Source: D Differential item functioning details · p. 14, §D.1 · p. 14

D.2 Mantel-Haenszel screen

  • The Mantel-Haenszel screen groups models into coarse ability bands and compares pass odds between groups within each band.For each scope and comparison, 5,000 no-DIF matrices are simulated with group labels fixed; the family-wise cutoff is the 95th percentile of the maximum item statistic.Source: §D.2 · pp. 14–15

D.3 Logistic sensitivity screen

  • The ridge-logistic sensitivity screen keeps ability continuous and tests whether the group label still shifts item pass probability after ability is included.Its ridge penalty stabilizes estimates in the small model sample, and it is treated as corroborating evidence rather than a replacement for Mantel-Haenszel.Source: §D.3 · p. 15

D.4 Main DIF results

Developer DIF is concentrated in the single-score comparison and mostly disappears when items are scored by response-process scope. Seven-domain analyses show a notable remaining OpenAI-versus-Anthropic signal for cybercrime/intrusion.

D.4 Main DIF results

  • Under the single HarmBench score, OpenAI-versus-Anthropic comparisons flag 13 items by Mantel-Haenszel and 17 by logistic screening; response-process scopes reduce these to one and two flags, respectively.The access-type comparison has no FWER-level flags in the single-score or response-process scopes despite substantially different raw means.Source: Table 5 · p. 15, §D.4 · p. 15

D.5 Seven-domain sensitivity

  • Within the seven-domain scopes, most domains have no FWER-level DIF, but cybercrime/intrusion retains OpenAI-versus-Anthropic flags under both screens.Logistic screening also leaves one developer flag in misinformation/disinformation; no seven-domain access-type comparison has an FWER-level flag.Source: Table 6 · p. 16, §D.5 · p. 16

E Model list and cohort composition

The analysis cohort contains 81 models from the HELM Safety leaderboard, which listed 87. Six were excluded because they lacked complete per-item data on at least one benchmark.

E Model list and cohort composition

Loading 2610.12409v1…