PaperlayerView exact revision on arXiv

Source-linked AI summary

SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks

Yanhang Li, Zhichao Fan, Zexin Zhuang

arXiv:2605.25492v1cs.LG

TL;DR

SafetyRepro asks whether benchmark-based pairwise model rankings remain identified when evaluator-controlled configuration choices vary. It introduces a finite-envelope test and applies it through a commit-stamped evaluation protocol, finding that configuration choice alone can flip the pairwise verdict on every tested benchmark. The paper concludes that strict pairwise rankings must be interpreted jointly with the model and harness within the measured envelope.

  • Problem

    Benchmark claims such as “A is safer than B” rely on harness choices that benchmark papers often under-specify, leaving the stability of strict pairwise orderings insufficiently tested.

  • Method

    The paper defines a configuration-conditional pairwise-disagreement rate, computes it exactly on a commit-stamped practice-derived envelope, and uses SHAP only to localize influential axes.

  • Results

    Configuration choice alone moves the pairwise verdict on every benchmark in scope, and XSTest permits all six orderings of the three evaluated models.

  • Takeaways & Limitations

    A strict pairwise ranking is free of strict reversal on the configuration grid iff ρflip=0, so benchmark verdicts require disclosure of both configuration and harness.

  • Takeaways & Limitations

    The empirical findings are confined to three 7–9B open-weight models, five benchmarks, one fixed item subset per benchmark, NF4 quantization, and one in-house harness.

Abstract

from arXiv · show

Pairwise model comparisons drawn from foundation-model benchmarks ("A is safer than B") are read as quantitative verdicts but hinge on harness choices benchmark papers under-specify. We close one theory-benchmark loop on this primitive: a finite-envelope proposition tying a measurable pairwise-disagreement rate to whether the strict ordering admits a configuration-pair reversal, paired with a commit-stamped evaluation protocol that operationalises it on widely cited alignment benchmarks. On every benchmark we test, configuration choice alone can flip the pairwise verdict; the proposition isolates this strict-reversal failure mode.

1. Introduction

SafetyRepro argues that pairwise benchmark verdicts depend on under-specified harness choices, then tests this dependence with a commit-stamped configuration envelope. Across the evaluated benchmarks, configuration choice alone can reverse model pairwise verdicts, while implementation-package differences provide separate supporting evidence.

  • Motivation: Benchmark pairwise comparisons depend on prompt template, decoding, few-shot level, scoring rule, and quantization choices that are often under-specified.The paper frames this as a reproducibility problem for TruthfulQA, BBQ, ToxiGen, CrowS-Pairs, and XSTest.
  • Scope: The study evaluates three instruction-tuned 7–9B open-weight models on five alignment-related benchmarks using a 612-cell grid within a bounded NF4 consumer-hardware regime.Claims are restricted to three models, one fixed item subset per benchmark, and the stated configuration envelope.
  • Main findings: Configuration choice alone moves the pairwise verdict on every benchmark in scope within the commit-stamped practice-derived envelope.The envelope includes core harness defaults plus stress settings, and the rate is descriptive over the observed configurations rather than a population claim.
  • Contributions: The operator-controllable pairwise-disagreement rate is introduced as a configuration-conditional rank-flip metric computed exactly on the observed envelope.The contribution distinguishes this metric from the separate LightGBM/SHAP explanatory analysis.
  • Contributions: The paper also reports implementation non-equivalence across three mainstream evaluation packages as a separate case study.The packages are compared on nominally similar tasks, but the paper does not claim package-level harmonization or a single-axis attribution.
  • Contributions: A bounded conservative-core scale probe addresses a single-family confound without claiming a generic scaling law.The probe covers two model families and is explicitly limited in scope.

2. Background

The paper builds on work treating benchmark implementations and protocol choices as sources of measurable variance, then positions its contribution as a configuration-conditional rank-flip metric alongside existing single-harness sensitivity studies.

  • Prior work: Prior work documents benchmark-score sensitivity to prompt design, fragile evaluation harnesses, and the need for variance accounting.The cited literature motivates treating evaluator choices as part of the empirical protocol rather than as optional details.
  • Conceptual background: Construct-validity and generalizability traditions model evaluator implementation as a method facet, while researcher-degrees-of-freedom analyses explain why protocol choices matter.This background supplies the conceptual basis for measuring implementation-related variance.
  • Positioning: SafetyRepro contributes a rank-flip rate computed exactly on a commit-stamped practice-derived configuration envelope, plus a limited three-package implementation case study.The paper presents these contributions alongside, not above, existing variance work within single harnesses.

3. Experimental Grid

The experimental grid varies practical harness axes across three comparable open-weight models and five alignment-related benchmarks, while fixing or restricting factors to make configuration variance measurable. The resulting design comprises 612 model-by-benchmark-by-harness cells and 1,764 inference calls.

  • Models: The model set contains Qwen-2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Yi-1.5-9B-Chat, treated as a three-level factor within the 7–9B band.The paper avoids claims beyond this model scale and open-weight setting.
  • Benchmarks: The five benchmarks cover truthfulness, social bias, implicit hate, and exaggerated refusal, while jailbreak robustness and multiturn adversarial suites remain out of scope.A fixed random item sample is shared across configurations to isolate harness variance from data-subsampling variance.
  • Implementation axes: The grid varies prompt template, decoding, few-shot level, and scoring, with practical levels chosen to reflect common practice rather than maximize variance.Templates span minimal, role-framed, detailed, and chain-of-thought-style instructions; few-shot levels are 0, 3, and 5.
  • Scoring: Free-form scoring extracts a label from generated text, whereas logprob scoring ranks fixed candidate options; the two paths therefore score different objects.Logprob scoring is restricted to greedy decoding, and CrowS-Pairs admits only the logprob path.
  • Quantization: NF4 quantization is fixed in the main grid as a deployment-realism axis for 16 GB-class consumer pipelines.A separate BF16 paired sweep reports median |sBF16−sNF4|=1.7 pp and matching three-model orderings on 17/18 paired cells.
  • Grid accounting: The valid grid contains 612 model×benchmark×harness cells and produces 1,764 total inference calls after accounting for stochastic five-seed averaging.The per-benchmark rank-flip envelope contains 48 shared configurations for four benchmarks and 12 for CrowS-Pairs.

4. Metrics

The paper defines metrics for harness-driven score dispersion, compliance disagreement, ranking instability, and variance attribution, alongside a finite-envelope test for strict pairwise reversals. It explicitly limits interpretation where ties, envelope size, scoring paths, or sampled model levels affect identifiability.

  • Score Dispersion Index (SDI): SDI measures the max−min score range across differently implemented labs, normalized by the mean score.The paper reports SDI as (max−min)/mean and warns that its ratio is unstable as the mean approaches zero.
  • Compliance Flip Rate (CFRθ): CFRθ counts grid-configuration pairs whose pass/fail verdicts differ at threshold θ.The reported threshold summaries use θ ∈ {0.5, 0.7}, with fuller threshold tables in the appendix.
  • Ranking concordance (τR): τR averages Kendall τ across every pair of configurations’ induced three-model rankings.Under the three-model uniform null, expected total-order mismatch is 5/6 ≈ 83.3%, while the per-pair inversion rate is 50%.
  • Configuration-identifiability: The paper distinguishes total-order mismatch from pairwise disagreement, denoting the latter ρflip and treating ties separately when forming three-model rankings.Tie cells receive deterministic alphabetical tiebreaking only for the induced total ordering.
  • Variance attribution (ρ): Variance attribution ρ compares implementation-attributable with model-attributable raw η², while model effects remain conditional on the sampled model trio.Large interaction shares make the partition a qualitative robustness reading rather than an identified estimate.
  • Configuration-identifiability: Proposition 1 tests strict pairwise reversal-identifiability using the counts of positive, negative, and tied configuration signs.The paper states that ρflip=0 is the one-line test for absence of strict reversal, but this does not establish total-order identifiability.

5. Findings

The study evaluates strict pairwise rank instability over a commit-stamped, practice-derived configuration envelope. Configuration choice can reverse verdicts across benchmarks, with the strongest reachability on XSTest, while core-tier results attenuate without disappearing.

  • Evaluation envelope: The commit-stamped envelope includes documented benchmark or harness defaults and practice-derived stress choices, while excluding claims about representative community practice.The core tier covers templates, decoding, and few-shot choices; the stress tier adds chain-of-thought and diverse T=0.7 decoding.
  • Metric: ρflip is the minority strict-sign mass: the share of admissible configurations whose pairwise verdict opposes the majority direction, bounded above by 0.5.It is also a lower bound on per-configuration-pair strict-disagreement rate and is reported as operator-controllable rank-flip rate.
  • Headline findings: Configuration choice alone moves pairwise verdicts on every benchmark in scope, and XSTest reaches all six orderings of Qwen, Mistral, and Yi under the full envelope.Figure 2 reports both per-pair disagreement and the number of reachable total orderings.
  • Scoring-path stratification: Within free-form scoring, ρmax_flip is 36% on TruthfulQA and 25% on BBQ, while ToxiGen and XSTest remain at least as unstable within free-form as in the mixed envelope.The reported values are scoring-path slices, not a controlled intervention because their decoding support differs.
  • Scoring-path stratification: 47.9% is the maximum TruthfulQA pairwise-disagreement rate on the full mixed-scoring-path envelope, compared with 36.1% within free-form scoring alone.The difference partly reflects switching scoring paths rather than strict within-measurand instability.
  • Envelope robustness: Core-tier attenuation preserves instability: TruthfulQA is roughly 40% and XSTest retains 5 of 6 orderings, while all six XSTest orderings depend specifically on diverse T=0.7 decoding.Dropping T4 alone preserves 6/6 XSTest orderings at ρmax=0.361; dropping diverse decoding reduces reachability to 5/6.
  • Robustness checks: Parse-clean filtering leaves the main claims intact: TruthfulQA remains ρmax_flip=0.479, BBQ 0.438, CrowS-Pairs 0.250, and XSTest preserves six-of-six ordering reachability.ToxiGen rises from 0.354 to 0.361, while XSTest falls from 0.292 to 0.282 after filtering.
  • Interpretation: Implementation main effects explain at least as much aggregate variance as model identity on three of four free-form benchmarks, with TruthfulQA the exception and interactions large throughout.A separate package case study reports score spans of 21.7 percentage points on TruthfulQA and 22.7 on BBQ, interpreted as implementation non-equivalence rather than clean package effects.

6. Threats to validity and limitations

The empirical claims are bounded by a narrow, hand-curated evaluation envelope, fixed subsets and draws, parser degradation, and unresolved cross-package attribution. The paper therefore treats several findings as qualitative or suggestive rather than population-level or fully decomposed conclusions.

  • Scope: The study covers three 7–9B open-weight models, five benchmarks, one fixed item subset per benchmark, NF4 quantisation, and one in-house harness.It does not sweep larger or closed-weight models, additional item subsets, or a full NF4-versus-BF16 grid.
  • Evidence status: The cross-package case study, BF16 comparison, Type-III check, and scale probe are presented as suggestive rather than as replacements for the main bounded evidence.The paper separately identifies population-level rank stability, closed-weight models, and several broader evaluation settings as out of scope.
  • Statistical interpretation: The variance ratio ρ is qualitative because analyses use fixed item subsets, one exemplar draw, no within-cell replication, and large interaction shares of 41–54%.The reported sensitivity intervals are within-grid configuration-bootstrap ranges, not super-population confidence intervals.
  • Parser: T4 parse rates fall to 0.88 on ToxiGen and 0.91 on XSTest, motivating parse-clean headlines restricted to cells with parse rate ≥0.95.The paper reports parse-clean results separately to address this parser limitation.
  • Terminology: The paper distinguishes variance from uncertainty, disagreement from error, and alignment-related evaluation from the broader label of safety.These terminology choices constrain how the findings should be interpreted.
  • Code provenance: All reported numbers come from a v3.0 rerun after an eight-issue code audit, with prompt truncation identified as the dominant source of v2.2-to-v3.0 changes.This provenance documents a substantial implementation correction rather than eliminating the paper’s scope boundaries.
  • Scope and follow-ups: The envelope is hand-curated rather than representative, and cross-package spread cannot be decomposed into per-axis contributions without changing task definitions.Priority follow-ups include larger models, independent item subsets, and per-axis decomposition of cross-package differences.

7. Conclusion

The paper gives a finite-grid test for strict pairwise rank reversals and applies it to alignment benchmarks. Every tested benchmark fails the test, showing that the observed verdict depends on both model and harness and requires disclosure of the configuration.

  • Conclusion: A pairwise ranking is free of strict reversal on a configuration grid if and only if ρflip=0.The proposition isolates strict reversals but does not establish total-order identifiability or exclude ties.
  • Conclusion: Every alignment benchmark in the commit-stamped envelope fails the strict-reversal test.This is the paper’s cross-benchmark conclusion about configuration-conditional pairwise verdicts.
  • Conclusion: On XSTest, varying harness choices reaches all six orderings of three open-weight 7–9B models.The result demonstrates the strongest reported form of configuration-conditional rank instability.
  • Conclusion: The paper concludes that a “model A is safer than B” verdict is a property of the model–harness pair, not of the models alone.Reproduction therefore requires disclosure of both the models and the evaluation configuration.
  • Conclusion: The framework applies to pairwise verdicts drawn from any finite configuration envelope, while identifying published rankings requires no new theory or compute.The contribution is a one-line certification test for the observed envelope rather than a population-level guarantee.
  • Disclosure: The protocol records resolved axis values, point scores, score neighbourhoods, verdict stability, and ranking stability in a GRID card.The artefact bundle also commits to code, environment, model and dataset pins, per-cell results, and reproducibility manifests.
  • Disclosure: The evaluation bundle includes 612 per-cell records, metric tables, figure sources, and per-item generation records, but its release is a commitment rather than an attachment at submission time.The paper explicitly avoids presenting unreleased artefacts as already solving reproducibility.

D. Regex-parser parse rates per (benchmark, template)

Parser quality varies across benchmark templates, especially for chain-of-thought free-form configurations, but parse-clean filtering preserves the principal rank-flip conclusions. The appendix defines the score-range quantities used for these checks.

  • D. Regex-parser parse rates per (benchmark, template): Parse rates are reported as means and minima across valid free-form configurations for each benchmark–template cell.T4 is chain-of-thought; CrowS-Pairs is excluded because it has only a logprob path and no free-form configurations.
  • D. Regex-parser parse rates per (benchmark, template): On 579 of 612 parse-clean cells, SDI and CFR maxima remain unchanged, τR shifts by at most ±0.15, and only XSTest’s ρ changes materially.XSTest shifts from 2.4 to 1.7 while remaining above 1; the listed BBQ, TruthfulQA, and ToxiGen values are otherwise unchanged.
  • D. Regex-parser parse rates per (benchmark, template): Parse-clean rank-flip calculations restrict the ρflip and ordering analysis to cells where all three models have parse rate ≥0.95.This filtering tests whether parser degradation drives the headline reversal findings.
  • D. Regex-parser parse rates per (benchmark, template): TruthfulQA remains at ρflip=0.479, BBQ at 0.438, and CrowS-Pairs at 0.250 after parse-clean filtering.The first two use |C|=48 shared configurations, while CrowS-Pairs uses |C|=12.
  • D. Regex-parser parse rates per (benchmark, template): ToxiGen’s parse-clean envelope shrinks to |C|=36 and its ρflip rises slightly to 0.361.Twelve of the original 48 cells are excluded.
  • D. Regex-parser parse rates per (benchmark, template): XSTest’s parse-clean envelope shrinks to |C|=39 and its ρflip falls to 0.282.Nine cells are excluded, but all six distinct model orderings remain reachable.
  • D. Regex-parser parse rates per (benchmark, template): The appendix concludes that T4 parse degradation on ToxiGen and XSTest does not contaminate the headline operator-controllable claims.The preserved six-of-six XSTest ordering is the strongest supporting check.
  • E. Absolute score ranges per slice: For score ranges, n counts valid configurations and the reported values are smin, smax, mean, and absolute range, with range defined as smax−smin.Scores are bounded in [0,1].

F. ANOVA specification

The ANOVA analysis separates model, implementation, interaction, and residual contributions, while the per-axis analysis uses error-corrected ω2. Robustness checks preserve the qualitative benchmark ordering but caution against treating individual ρ magnitudes as precise.

  • F. ANOVA specification: The variance decomposition fits an OLS model on the free-form slice with model, template, decoding, few-shot, and selected two-way interactions.Scoring and quantization are held constant in this slice.
  • F. ANOVA specification: Two variance-share estimators are computed from the same Type-II ANOVA table.The headline ratio ρ uses raw η2 shares.
  • F. ANOVA specification: The implementation-to-model ratio ρ compares implementation η2 with model η2, while model, implementation, interaction, and residual shares sum to 100% by construction.The interaction share aggregates included two-way interaction terms, including model-by-axis interactions.
  • F. ANOVA specification: The per-axis breakdown uses Hays’ error-corrected ω2, which penalises terms whose sum of squares is consistent with residual noise.This estimator addresses which individual axis explains measurable score variance rather than overall variance partitioning.
  • F. ANOVA specification: CrowS-Pairs is excluded from the ANOVA because its free-form slice is empty.The appendix describes the analysis as a deliberate first pass.
  • F. ANOVA specification: The bootstrap summary uses 1,000 resamples over configurations for the reported metric ranges.The paper distinguishes these within-grid resampling intervals from inferential population claims.
  • F. ANOVA specification: The BBQ variance row reports model 9.1%, implementation 34.7%, and ρ=3.8.The row’s interval information is reported alongside these point values.
  • F. ANOVA specification: No single implementation axis dominates any benchmark, so one-at-a-time robustness sweeps under-report the joint configuration envelope.The per-axis analysis is therefore complementary to the joint implementation variance share.

K. Per-benchmark summary

Configuration and implementation choices produce benchmark-specific instability, with TruthfulQA, BBQ, ToxiGen, and XSTest showing substantial score or ordering sensitivity while CrowS-Pairs is degenerate by construction.

  • TruthfulQA: TruthfulQA reaches 102.4% SDI on Qwen-2.5-7B, with scoring-method swings up to 0.468 and τR≈0.TruthfulQA is the only benchmark where model identity dominates variance.
  • BBQ: BBQ is implementation-dominated (ρ = 3.8×), with CFR0.5 up to 43.9% on Yi-1.5-9B and CFR0.7 up to 49.6% on Qwen-2.5-7B.
  • ToxiGen: ToxiGen has the highest ρ overall at 14.9×, but its 78.3% total-order mismatch rate remains below the 83.3% chance baseline.The model-variance share is only 1.8%, making the ratio unstable near zero.
  • XSTest: XSTest is implementation-dominated under Type-II analysis (ρ = 2.4), but its ratio drops to 1.3 under Type-III sums of squares and 1.7 on the parse-clean subset.It is therefore best read as boundary-close rather than cleanly implementation-dominated.
  • CrowS-Pairs: CrowS-Pairs has low SDI and undefined CFR because its pair format admits only one scoring path and one decoding setting.The resulting fixed-threshold flip rate is algebraically degenerate rather than informative.
  • Implementation checks: The v3.0 rerun corrected right-side truncation and head-only extraction, with individual-cell bugfix deltas reaching 0.51 absolute score points.The fs5/T4 corner accounts for most of the tail, while Yi-1.5 is most bugfix-sensitive on average.

O. BF16 conservative-core sweep (NF4 vs BF16, paired cells)

The paired NF4/BF16 sweep shows mostly small conservative-core score changes but a heavy Yi-1.5 tail, while ordering stability is high on the matched sub-envelope and broader claims remain untested.

  • Paired score changes: Mean |sBF16 −sNF4| is 2.58 pp, with median 1.70 pp, maximum 20.68 pp, and 5 of 54 cells exceeding a 5-pp gap.Yi-1.5-9B is the outlier, including a 20.68-pp maximum on Yi/XSTest/T1/logprob.
  • Paired score changes: Most conservative-core cells move by less than 3 pp under NF4→BF16, but BF16 produces larger within-envelope ranges for Yi/ToxiGen and Yi/XSTest.The reported absolute ranges increase from 0.102→0.319 and 0.139→0.342, respectively.
  • Ordering stability: NF4 and BF16 induce identical 3-model orderings on 17 of 18 conservative-core cells, yielding 94.4% ordering stability.The single divergent cell is ToxiGen/T3/logprob, where the Qwen–Yi pair flips.
  • Scope: The BF16 ordering result does not extend to the full |C|=48 adversarial envelope because T2/T4, few-shot, and diverse-decoding cells lack matched BF16 evaluations.
  • Cross-family probe: A two-family scale probe finds the same three-vs-two collapse-versus-persist partition: ToxiGen, XSTest, and CrowS-Pairs narrow at larger models, while TruthfulQA and BBQ retain or grow range.The pattern is shared by Qwen-2.5 and Yi-1.5 but is not presented as a scaling law.
  • Cross-package context: The cross-package case study remains illustrative because packages score different candidate objects and do not isolate prompt, parser, scorer, or chat-template effects.

U. Cross-package case study: non-equivalence of package-default evaluation stacks

Holding a nominal anchor constant across three evaluation packages still yields materially different scores because the packages implement non-equivalent task definitions and candidate objects.

  • Design: The case study reruns one Qwen2.5-7B-Instruct anchor across packages, using native adapters and package-specific item sampling.The nominal anchor is bf16, greedy, 0-shot, 300 examples, and seed 42.
  • Non-equivalence: The packages differ in native benchmark coverage and score different candidate objects, including canonical-answer logprobs, joint concatenated choices, and A/B/C/D letter logprobs.
  • Observed spread: A community user can observe scores differing by 22 points on the same model and benchmark despite holding the nominal configuration constant.The paper treats this as evidence that configuration disclosure alone is insufficient for cross-package reproduction.
  • Limitation: The cross-package spread cannot be decomposed into prompt, parser, scorer, and chat-template contributions without modifying task definitions inside the runners.

V. Aggregate variance partition: SDI, CFR, ρ

Aggregate analyses show that configuration choices can substantially disperse scores and alter pass/fail or pairwise conclusions, while robustness checks narrow the supported scope of those findings.

  • SDI: Mean SDI is 54%, reaching 102.4% on Qwen-2.5-7B/TruthfulQA, while CrowS-Pairs remains below 35% SDI on all three models.
  • CFR: At θ=0.5, CFR peaks at 51.1% on Mistral-7B/TruthfulQA; at θ=0.7, it peaks at 49.6% on Qwen-2.5-7B/BBQ.These cells reflect implementation-grid balance rather than a stable model-benchmark property.
  • Variance partition: Implementation-to-model variance ratios are 3.8 on BBQ, 14.9 on ToxiGen, 0.7 on TruthfulQA, and 2.4 on XSTest.The directional conclusion is robust, but the ToxiGen magnitude is unstable because model-variance share is only 1.8%.
  • Scoring path: Switching only the scoring path moves scores by up to 0.468 on Qwen-2.5-7B/TruthfulQA because the paths score non-equivalent measurands.
  • Scope: The study does not claim a generic scaling trend, transfer beyond the tested families and models, or population-level evaluation reproducibility.
  • Robustness: On three stratified 80% TruthfulQA subsamples, all four conservative-core orderings remain invariant, with per-cell raw-score range median 1.49 pp and maximum 3.83 pp.One Qwen/Yi tied cell was resolved by deterministic alphabetical convention rather than a strict ordering.
  • Robustness: The conservative-core rank-flip check preserves reversal existence but caps the metric at 0.5 for |C|=4, so its saturated values do not quantify full-envelope magnitude.
Loading 2605.25492v1…