PaperlayerView exact revision on arXiv

Source-linked AI summary

When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels

Sushant Gautam, Finn Schwall, Annika Willoch Olstad, Fernando Vallecillos Ruiz, Birk Torpmann-Hagen, Sunniva Maria Stordal Bjørklund, Leon Moonen, Klas Pettersen, Michael A. Riegler

arXiv:2605.06652v1cs.LGcs.AIcs.CL

TL;DR

The paper asks how deployments can compare language-model safety when no suitable labeled benchmark exists. It formalizes benchmarkless comparative safety scoring, implements and validates an instrumental-validity chain in SimpleAudit, and finds reliable separation, target-dominated variance, and rerun stability under fixed configurations.

  • Problem

    Deployments may need comparative safety evidence for a language, sector, or regulatory regime before a suitable labeled benchmark exists.

  • Method

    The paper fixes the scenario pack, rubric, roles, sampling, and rerun budget, then validates responsiveness, target sensitivity, and reproducibility in SimpleAudit.

  • Results

    AUROC remains ≥0.89 across tested local configurations, target identity explains η^2 = 0.52 of variance, and ten reruns stabilize scores within roughly one point.

  • Takeaways & Limitations

    Comparative evidence should report scores, deltas, critical rates, uncertainty, and auditor and judge configuration together under a fixed instrument.

  • Takeaways & Limitations

    Passing the validity chain does not establish that a scenario pack captures the deployment construct or that scores reflect real-world safety.

Abstract

from arXiv · show

Many deployments must compare candidate language models for safety before a labeled benchmark exists for the relevant language, sector, or regulatory regime. We formalize this setting as benchmarkless comparative safety scoring and specify the contract under which a scenario-based audit can be interpreted as deployment evidence. Scores are valid only under a fixed scenario pack, rubric, auditor, judge, sampling configuration, and rerun budget. Because no labels are available, we replace ground-truth agreement with an instrumental-validity chain: responsiveness to a controlled safe-versus-abliterated contrast, dominance of target-driven variance over auditor and judge artifacts, and stability across reruns. We instantiate the chain in SimpleAudit, a local-first scoring instrument, and validate it on a Norwegian safety pack. Safe and abliterated targets separate with AUROC values between 0.89 and 1.00, target identity is the dominant variance component ($η^2 \approx 0.52$), and severity profiles stabilize by ten reruns. Applying the same chain to Petri shows that it admits both tools. The substantial differences arise upstream of the chain, in claim-contract enforcement and deployment fit. A Norwegian public-sector procurement case comparing Borealis and Gemma 3 demonstrates the resulting evidence in practice: the safer model depends on scenario category and risk measure. Consequently, scores, matched deltas, critical rates, uncertainty, and the auditor and judge used must be reported together rather than collapsed into a single ranking.

1 Introduction

The paper defines benchmarkless comparative safety scoring for deployments lacking suitable labeled benchmarks and proposes an instrumental-validity chain for interpreting scenario-based scores. SimpleAudit instantiates the approach as a local-first measurement instrument whose claims remain bounded by its fixed configuration and deployment context.

  • Motivation: Benchmarkless comparative safety scoring addresses model-safety comparisons when domain-specific labeled benchmarks are unavailable or too costly to construct.The setting includes language-, sector-, policy-, and infrastructure-specific deployment constraints.
  • Validation chain: The instrumental-validity chain requires responsiveness to a controlled safety contrast, target-dominated variance, and stability across reruns.The paper instantiates these requirements with safe-versus-abliterated targets, variance decomposition, and bootstrap stability analysis.
  • Scope: The chain does not establish construct validity for Norwegian public-sector safety; deployment validity remains the deploying team’s responsibility.The narrower claim is that the instrument responds to target behavior rather than noise or apparatus artifacts.
  • Reference implementation: SimpleAudit is a local-first Python library and versioned scenario pack that operationalizes the validity chain as a measurement instrument.It reports verdicts, scores, matched deltas, critical-rate differences, uncertainty, transcripts, and token usage.
  • Validation and application: The paper applies the same validation chain empirically and uses it for a Norwegian public-sector model comparison.The contribution list identifies empirical validation, a Petri generalization check, and deployment evidence as separate components.

2 Background and Positioning

The paper positions benchmarkless comparative scoring between static labeled benchmarks, discovery-oriented auditing, and LLM-as-judge methods. These neighboring approaches leave unmet the need for local, repeatable, uncertainty-aware comparative numbers in constrained deployments.

  • Static safety benchmarks: Static safety benchmarks provide calibrated comparison when labeled data exists but are expensive to build, release-fixed, and commonly English-first.The paper contrasts their annotation and coverage requirements with the needs of local, jurisdiction-specific deployments.
  • Discovery-oriented auditing: Discovery-oriented audits generate attacks, transcripts, and hypotheses for review, emphasizing speed and breadth rather than procurement-ready scores.Petri is presented as the most directly comparable discovery-oriented artifact.
  • Deployment need: Procurement teams instead need governance-relevant numbers with error bars that are comparable across reruns and producible locally.This need is especially relevant for regulated deployments and long-tail languages.
  • LLM-as-judge reliability: LLM-as-judge systems face position, verbosity, and self-enhancement biases, so comparative tools must report deltas and uncertainty rather than assume judge reliability.The paper extends variance decomposition from judge selection to the joint target–auditor–judge stack.

3 Problem Formulation

The paper formulates benchmarkless comparative safety scoring as a fixed-instrument claim-making problem without ground-truth labels. Its contract defines the roles, score outputs, and validation requirements that bound permissible comparisons.

  • Problem formulation: Benchmarkless comparative scoring occupies the gap where ground-truth labels are unavailable and discovery findings cannot become procurement-comparable scores for small local teams.The category fixes an instrument and reruns it under disclosed conditions.
  • Role structure: Target, auditor, and judge are independent roles so experiments can quantify whether scores reflect target behavior or apparatus artifacts.Independence permits variation of each role as an experimental factor.
  • Instrument definition: The measurement instrument consists of a versioned scenario population, rubric, auditor and judge instructions, turn budget, sampling parameters, and rerun count.Scores from different instruments are not directly comparable.
  • Reported quantities: Severity labels are remapped to a 0–100 safety scale, while critical rate and confidence intervals are reported alongside target-to-target deltas.The critical rate separately captures the proportion of most-severe failures.
  • Validation requirements: Instrumental validity requires responsiveness to a capability-matched safety contrast, target sensitivity against auditor and judge artifacts, and reproducibility across reruns.The variance and stability requirements address distinct ways an LLM-on-LLM scoring stack can fail.
  • Comparison choice: Absolute deltas are the paper’s default comparison, while relative or log-transformed deltas are left for deployments with different preferences over score regions.The alternative delta choices would weight score differences differently.

4 SimpleAudit: A Reference Implementation

SimpleAudit packages a fixed scenario-based audit into a repeatable, inspectable measurement instrument. Its bounded workflow separates probing, target response, and judging while supporting local execution, modular roles, explicit configuration, and uncertainty reporting.

  • Instrument design: SimpleAudit fixes the scenario pack, rubric, auditor, judge, target model, and sampling configuration so outputs can be repeated and statistically characterized.The design is explicitly not a general auditing platform.
  • Workflow: Each scenario runs as a bounded multi-turn interaction in which the auditor generates probes, the target responds, and the judge grades the transcript.Figure 1 presents these three stages as the core workflow.
  • Modularity and outputs: SimpleAudit allows target, auditor, and judge replacement through a common provider interface while otherwise pinning the instrument.It reports aggregate scores, severity distributions, critical rates, target deltas, bootstrap intervals, transcripts, and token usage.
  • Claim population: Replacing the scenario pack creates a new instrument because the pack defines the deployment population over which claims are made.The claim population therefore changes with the scenario distribution.
  • Claim contract: The claim contract licenses comparative claims under fixed configuration but excludes universal-safety claims, complete hazard coverage, and deployment certification.Changing any component yields a new instrument with a new claim population.

5 Results: Validating and Configuring the Scoring Instrument

SimpleAudit’s validation chain tests known safety sensitivity, target-dominated variance, and rerun stability, then configures judges and auditors around comparative rather than absolute scores. The results support local deployment when reporting target ordering, critical status, deltas, and uncertainty under a fixed instrument configuration.

  • Validation chain: The validation chain asks whether scores respond to a known contrast, reflect target rather than apparatus effects, and stabilize across reruns.The experiment uses local models for validation and admits XL only as a reference for configuring the local stack.
  • Target sensitivity: AUROC = 1.00 (XS), 0.98 (S), and 1.00 (M) at J=A=L separates safe from abliterated targets across sizes.Each size uses 10 safe and 10 abliterated runs.
  • Target sensitivity: Target variance dominates with η2 = 0.52, while auditor and judge contributions are substantial but not ordered by this analysis.The reported target interval is [0.41, 0.62], with overlapping auditor and judge confidence intervals.
  • Rerun stability: By n=10, scores stabilize within roughly one point on the 0–100 scale, below the 5–20 pp deltas used in procurement.Safe-target deviation falls from 8.3 points at one run to 0.9 at nine runs; abliterated targets drop below 2 points from k=3.
  • Judge selection: M and L judges are viable for governance use, whereas XS has a 44% critical-miss rate and S fails at governance-compatible rates.M and L have critical-miss rates near 10% against XL, compared with a 4% XL self-agreement floor.
  • Judge and auditor configuration: Reliable judges preserve target ordering and critical status despite disagreements about absolute single-run severity levels.The claim contract therefore emphasizes comparative deltas and critical-versus-non-critical status rather than exact absolute scores.
  • Judge and auditor configuration: The default configuration is J = A because judge variance largely cancels under deltas, while auditor variance remains consequential and drives much of the non-target variance.At J=A=L, token budget is approximately 48% target, 34% auditor, and 18% judge.

6 Discovery Outputs as Comparative Scores

The validity chain is tool-agnostic: it admits Petri as well as SimpleAudit. Their meaningful differences arise upstream, because SimpleAudit fixes scoring contracts and local deployment conditions while Petri prioritizes flexible behavioral discovery.

  • 6.1 The chain admits Petri: The validation chain admits Petri, so tool differentiation occurs upstream in the constraints each instrument enforces by construction.The comparison treats chain passage as an existence claim, not a selection task.
  • 6.1 The chain admits Petri: Petri is a discovery-oriented Inspect-based tool with a multi-turn auditor and 38 default scoring dimensions for surfacing behaviors for human review.Its authors frame the tool around speed and breadth rather than a single committed comparative score.
  • 6.2 What SimpleAudit enforces upstream: SimpleAudit commits the scenario pack, rubric, aggregation rule, turn budget, sampling configuration, and rerun count before release-over-release comparison.Petri allows users to add or substitute dimensions and choose aggregation, shifting those decisions to the user.
  • 6.2 What SimpleAudit enforces upstream: SimpleAudit’s local judge–target–auditor stack keeps prompts, transcripts, and policies inside the deployment environment.This directly fits the local deployment setting described for benchmarkless comparative scoring.
  • 6.2 What SimpleAudit enforces upstream: Petri uses roughly 1.7× more tokens overall, with the difference dominated by its tool-using auditor and compounded across reruns.The reported cost difference is a concrete deployment trade-off between the tools.
  • 6.2 What SimpleAudit enforces upstream: Petri’s richer behavioral surface supports discovery, while SimpleAudit’s enforced contracts make it the operational fit for the stated deployment niche.The paper presents discovery and scoring as complementary contracts rather than interchangeable objectives.

7 Norwegian Procurement Case Study

The Norwegian case compares Borealis and Gemma 3 under a fixed local instrument, showing that model preference varies by size, scenario category, and risk measure. The evidence is therefore a bundle of matched comparisons and uncertainty measures, not an unconditional ranking.

  • Study design: The evaluation compares Borealis Instruct and Gemma 3 IT across four model sizes on a fixed 36-scenario Norwegian pack, with five turns and n=10 reruns.The judge and auditor are Qwen3.5-122B-A10B-Q4 K S, and the comparison concerns local candidates rather than a frontier reference.
  • Size scaling: 43.7% is Borealis’s 27B full-pack score, up from 4.6% at 1B, with most gains occurring below 12B.The intermediate scores are 27.9% at 4B and 42.3% at 12B; the 12B→27B increase is only +1.4 pp, below either cell’s rerun standard deviation.
  • Matched comparison: +0.8, +14.2, +7.0, and +6.0 pp are Borealis–Gemma full-pack score deltas from 1B through 27B, while critical-rate deltas favor Borealis at every matched size.The 1B score gap is within rerun uncertainty, whereas the 4B–27B gaps exceed the per-cell rerun envelope.
  • Category structure: The 4B Borealis advantage is category-specific: Healthcare is +24.0 pp, Public Sector +15.2 pp, and Safety +10.6 pp, while Language is weakest.Borealis has higher Language critical rates than Gemma at 1B, 12B, and 27B: +3.8, +2.5, and +3.8 pp.
  • Decision evidence: 15.3% versus 22.8% are Borealis-27B and Gemma-27B full-pack critical rates, but Borealis-27B still has only a 43.7% mean score and 21.7% pass rate.The procurement artifact combines scores, uncertainty, matched and category deltas, critical rates, and rerun caveats; it is not deployment clearance.

8 Discussion

The discussion frames benchmarkless scores as conditional, reproducible comparative evidence rather than absolute model properties. It requires disclosure of the instrument, role assignments, stability evidence, and non-claims so others can rerun or challenge the evaluation.

  • Evidence contract: A fixed instrument can provide comparative evidence without ground-truth labels when it is repeatable, stress-tested, and reported with uncertainty.Evaluation artifacts should be judged by the rankings they produce, the claims they license, and the assumptions under which those claims remain valid.
  • Disclosure: The minimum disclosure set includes the scenario pack, rubric, turn budget, sampling, rerun count, target–auditor–judge roles, versions, capability constraints, and stability evidence.It also includes matched deltas and critical-rate differences, alongside explicit non-claims such as universal safety, legal compliance, hazard completeness, and deployment certification.
  • Falsifiability: The artifact is falsifiable because another group can rerun the instrument, substitute components, or challenge the scenario population without treating scores as absolute target properties.Deployment claims should rest on matched deltas, critical-rate differences, and judge-sensitivity analyses interpreted against local priorities.

9 Limitations

The validation chain supports instrument-level claims, not construct validity or deployment claims. Its evidence is bounded by scenario design, studied configurations, and the choice of auditor.

  • Scope of validity: Passing the instrumental-validity chain is necessary but not sufficient for deployment claims because local policy expertise must assess whether the scenario pack captures the deployment construct.The chain establishes response to a controlled contrast, target-driven variance, and rerun stability, but not real-world safety in a specific domain.
  • Empirical boundaries: Empirical breadth is bounded to the studied languages, packs, judges, and model families, and the abliterated contrast tests refusal-trained safety differences rather than all unsafe behavior.Explicit eval-awareness mitigations are not implemented, and their relevance at the deployment scales studied remains open.
  • Scenario design: The method assumes scenario authors specify deployment concerns at the right abstraction level; narrow packs yield precise evidence about incomplete constructs, while broad packs complicate category interpretation.The supported use is iterative rather than one-shot.

10 Conclusion

The paper defines benchmarkless comparative safety scoring as a distinct evaluation category for deployments lacking ground-truth benchmarks. It offers a no-label validation chain and reports conditional comparative evidence, while retaining domain expertise and future labeled benchmarks as necessary complements.

  • Conclusion: Benchmarkless comparative safety scoring addresses deployments needing model comparisons before ground-truth safety benchmarks exist, especially in low-resource, regulated, or jurisdiction-specific settings.The paper treats it as a distinct category with its own claim contract, validation requirements, and deployment shape.
  • Conclusion: The resulting evidence is reported as scores, deltas, critical rates, uncertainty intervals, and sensitivity checks rather than an unconditional model ranking.The tool does not replace domain expertise, scenario authoring, or labeled benchmarks when those become feasible.
  • Future work: A next step is to stress-test the validation chain with deliberately degenerate instruments and calibration against human judgment, because its value depends on rejecting as well as admitting tools.

Supplementary Materials

The supplementary analyses test whether SimpleAudit’s comparative scores reflect target behavior, remain stable across reruns, and transfer to Petri under the same validity chain. They also identify deployment boundaries: apparatus choices affect scores, and Petri’s target dominance is dimension-dependent.

  • Validation chain: AUROC remains ≥0.89 across reliable local judge–auditor combinations and target sizes, with 1.00, 0.98, and 1.00 at J=A=L for XS, S, and M.The lone overlap is one low-scoring safe run at T=S.
  • Variance decomposition: Target identity is the largest variance factor in all three SimpleAudit refits, with η2 = 0.518 mixed, 0.771 abliterated-only, and 0.859 safe-only.The result persists when the safety contrast is removed and targets differ only by size.
  • Variance decomposition: Auditor and judge effects remain material, spanning partial η2 values from 0.25 to 0.73, so replacing either component requires rerunning validation.The instrument is not judge-invariant or auditor-invariant on absolute-level metrics.
  • Judge selection: M and L judges approach XL self-agreement, whereas XS and S have critical-miss rates incompatible with deployment use.Judge comparison therefore requires critical-miss analysis rather than rank correlation alone.
  • Stability: Ten runs stabilize SimpleAudit per-scenario scores to roughly one point on the 0–100 scale, while abliterated targets converge faster.The steepest gains occur between k=1 and k=5.
  • Deployment fit: Petri’s total token spend averages 717K per run versus SimpleAudit’s 422K, with Petri using roughly 3.2× more auditor tokens.Petri’s auditor-heavy profile is driven by its tool-using auditor and scratchpad, while SimpleAudit uses more target tokens.
  • Petri comparison: Petri’s discovery-oriented findings support relative scores and capability-bounded auditor effectiveness, motivating application of the same validation chain to both tools.The tools differ in contract and deployment fit even when both can be assessed instrumentally.

J.3 Safe-vs-abliterated separation

Petri’s safe-versus-abliterated responsiveness depends on the dimension: concerning separates reliably, while u bizarre behavior fails despite target-dominant variance. The chain therefore admits Petri for concerning, but not uniformly across dimensions.

  • Safe-vs-abliterated separation: AUROC reaches 0.83 only at T = XS for u bizarre behavior, then falls below chance at T = S and near chance at T = M.CIs at T ∈{S, M} span 0.5, so responsiveness fails despite target dominance in the variance step.
  • Safe-vs-abliterated separation: AUROC is ≥0.99 for concerning in every cell, with point estimates of 1.000 in 10 of 12 cells.This result holds across the four reliable judge–auditor combinations and three local target sizes.
  • Chain admission: The chain admits Petri at concerning because responsiveness combines with the variance and stability results.Other dimensions, including needs attention and concerning with no mitigating factors, also pass all three requirements.
  • Comparison to SimpleAudit: The Petri variance profile differs from SimpleAudit, but the protocol does not support a direct performance comparison between the tools.The authors leave the causes of the difference unresolved and identify chain admission, not tool ranking, as the load-bearing claim.
  • Rubric scope: Petri’s 14 active dimensions carry comparative signal, while the remaining 24 support discovery without contributing to a scoring instrument.The classification uses matched safety and size contrasts under A=L, J=L, and n=10.

K.2 Per-dimension contrasts and the construct split

Petri’s active dimensions separate safety-related and capability-related constructs rather than measuring one underlying property. Cross-pack results likewise show that model family, language, scenario distribution, and risk measure can reverse comparative outcomes, requiring fixed protocols and contextual reporting.

  • Per-dimension contrasts: Three dimensions carry safety signal with small size signal, while scenario realism shows strong size signal and no safety signal.The safety-sensitive dimensions are cooperation with human misuse, user over sysprompt, and counterfactual bad behavior.
  • Construct split: Concerning is among the largest safety-active dimensions, whereas u bizarre behavior shows size but near-zero safety signal and fails the safe-vs-abliterated AUROC test.Target dominance for u bizarre behavior therefore reflects capability rather than safety; concerning is the most safety-active intended-use dimension.
  • Responsiveness criterion: Three dimensions fire above 5% but fail the contrast criteria, showing that activity alone does not anchor a comparative score.The chain rejects such dimensions at the responsiveness step before variance or stability checks.
  • Construct structure: 52.7% of variance is explained by PCA’s first component, with cumulative variance reaching 80% at k=4 and 90% at k=6.The active output has lower effective dimensionality than Petri’s nominal 38-dimension surface.
  • Cross-pack scenario effects: Qwen models score 20–45 pp higher on English packs, while Gemma models score 10–22 pp higher on Norwegian packs at every evaluated size.The reversal occurs across all sizes, and Qwen’s English advantage narrows roughly log-linearly with parameter count.
  • Cross-pack scenario effects: The sparse Q35B-A3B model scores about −12 pp lower on safety and −11 pp lower on Norwegian-language tasks than dense Q27B.The largest gap is −40 pp on hallucination scenarios, suggesting active parameter count matters more than total parameter count for safety robustness.
  • Interpretation and reporting: These findings are deployment-relevant descriptive patterns, not cross-family safety rankings, because another judge or scenario pack could shift absolute scores.The consolidated view instead uses aggregate scores with pack-level deltas to identify which deployment risks account for an average difference.
  • Interpretation and reporting: Scenario packs encode a local view of deployment risk and require iterative revision, transcript inspection, version control, and rerun comparisons.A fixed claim contract specifies which comparative claims are licensed and which universal-safety or certification claims are excluded.
Loading 2605.06652v1…