Source-linked AI summary
Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models
Nyamtulla Shaik, Fengjun Li, Bo Luo
TL;DR
Existing LLM safety benchmarks may not reliably evaluate small language models because automated judges are sensitive to output and prompt features. This paper evaluates five benchmark suites across 26 SLMs and finds ambiguity-driven rankings are unstable under reasonable scoring choices.
Problem
Existing LLM safety benchmarks may produce ambiguous labels reflecting evaluation difficulty rather than true SLM safety behavior, raising questions about their effectiveness and reliability.
Method
The study evaluates five widely used benchmark suites across 26 SLMs, examining automated judging, ambiguity, capability-related features, and score aggregation.
Results
Ambiguous outcomes are prevalent and associated with output quality, prompt complexity, and model architecture, making aggregate SLM rankings mathematically unstable under reasonable ambiguity treatments.
Takeaways & Limitations
LLM-centric safety benchmarks are insufficient as standalone evidence for SLM safety assessment when ambiguity-heavy pipelines drive benchmark conclusions.
Takeaways & Limitations
The study’s future work includes expanding model coverage, testing alternative judges, validating SALAD-Bench thresholds, and developing SLM-specific benchmarks.
Abstract
from arXiv · showhide
Small Language Models (SLMs) are increasingly deployed in resource-constrained, privacy-sensitive settings, where safety and bias failures can cause security and societal risks. However, existing AI safety\slash security\slash compliance benchmarks are designed for large language models that may not transfer reliably to SLMs. We therefore ask: Can these benchmarks effectively and reliably evaluate SLMs? To answer this question, we conduct a large-scale assessment of the effectiveness and robustness of these automated pipelines by evaluating five widely used benchmark suites across 26 open-source SLMs under a unified judging rubric, which assigns a score of 0, 1, or 0.5 to harmful, safe, or ambiguous/irrelevant responses, respectively. Across the benchmarks, ambiguous judgments dominate and correlate with prompt complexity and model architecture, indicating that {\em LLM-centric safety benchmarks are insufficient as standalone evidence for SLM safety assessment}. In general, the ambiguity rate increases with lexical density, output perplexity, and output length and decreases with lexical sophistication, self-coherence, and reply-prompt similarity. This reveals a capability-safety confound that mixes model capability with apparent safety. Since ambiguity is prevalent, aggregate mean-score leaderboards are mathematically brittle: model rankings change significantly under reasonable ambiguity treatments, even when the underlying outputs remain unchanged.
1 Introduction
SLMs are increasingly used in resource-constrained, privacy-sensitive settings, but safety benchmarks built for larger LLMs may be unreliable because SLM outputs differ in quality and automated judges are sensitive to surface features. This study therefore evaluates the benchmarking pipeline itself across five suites and 26 SLMs, finding capability-related ambiguity and brittle aggregate rankings.
- Motivation: SLMs with hundreds of millions to a few billion parameters are increasingly deployed in edge, IoT, and other resource-constrained settings.These settings impose strict latency, cost, and privacy/security compliance limitations.
- Problem: LLM-oriented safety judges compress open-ended outputs into ternary harmful, ambiguous, or safe labels, despite SLMs producing shorter, less fluent, and more failure-prone responses.Prior work also finds that automated judges are highly sensitive to surface features such as length and fluency.
- Research aim: The study examines whether capability-related surface features confound automated assessments of SLM safety, security, and compliance.The analysis covers benchmark prompts, model-generated responses, judge outputs, and scoring to identify causes of pipeline ineffectiveness.
- Main findings: Ambiguous outcomes concentrate on more complex prompts and lower-quality generations, making mean-score rankings brittle and revealing capability-related bias in the automated pipeline.The conclusion does not require additional human labeling, because ranking instability and capability-related bias are observable even when ambiguous cases’ true safety is unknown.
- Study scope: 741,312 model-prompt evaluations across five benchmark suites and 26 SLMs form the study’s large-scale measurement basis.This includes 715,312 judge-scored safety evaluations and 26,000 BBQ bias evaluations.
- Main findings: Ambiguity-heavy suites produce unstable conclusions, and model orderings shift substantially under reasonable ambiguity-handling choices.The paper also identifies conditions for decision usefulness, notes that simpler prompt sets may mask these issues, and proposes robustness recommendations.
2 Background and Related Work
SLMs have become a distinct, resource-efficient deployment choice whose architectures and roles extend beyond simply scaling down transformers. This evolution makes benchmark validity especially important because automated safety judgments can be sensitive to output and rubric characteristics, with implications for governance and high-stakes decisions.
- Small Language Models: SLMs trade peak capability for lower latency, lower cost, and easier on-device deployment across sub-1B and near-1B models.Early transformers such as GPT-2 demonstrated general-purpose generation below one billion parameters, while current SLMs represent an ongoing deployment choice.
- Small Language Models: SLM development combines architectural, efficiency, distillation, and instruction-tuning advances rather than merely scaling down transformers.Surveys also position SLMs as proxy models, guard models, or collaborators within larger systems, making output validity important when responses are shorter, less fluent, or more failure-prone.
- Safety and Security Benchmarks: Safety benchmarks have evolved from ad hoc red-team examples into reusable prompt suites, risk taxonomies, and standardized scoring protocols targeting harmful following, jailbreaks, privacy leakage, and bias.HELM Safety v1.0 standardizes benchmark suites and automated judging configurations, while SALAD-Bench provides broader, fine-grained safety-category coverage.
- Governance and Evaluation: Benchmark validity matters for compliance and procurement because governance frameworks demand measurable evidence of accuracy, robustness, security, content safety, privacy, and transparency.The EU AI Act, NIST AI Risk Management Framework, and China’s Interim Measures for Generative AI Services raise the stakes without mandating a specific benchmark.
- Automated Evaluation: HELM-style judge scoring maps open-ended safety responses to a coarse ternary scale, but judgments are sensitive to fluency, verbosity, formatting, and rubric phrasing.These sensitivities are especially consequential for SLMs because their outputs are often shorter, less robust, and harder to interpret.
3 Methodology and Measurement Design
The study evaluates automated safety pipelines per model–prompt instance across 26 SLMs and five benchmark suites, aligning prompts, responses, model covariates, and judge outcomes. It measures harmfulness, safety, ambiguity, prompt complexity, and output quality to analyze ambiguity and ranking stability.
- Evaluation setup: The pipeline pairs each prompt with each target SLM, evaluates 26 models across safety and bias benchmarks, and analyzes prompt-, response-, and model-level covariates.The aligned records support analyses of ambiguity, ranking stability, and metadata effects.
- Evaluation setup: The study selects SLMs up to 4B parameters across model families, including base and instruction-tuned models with varied tokenizer and architecture configurations.This design targets resource-constrained deployment while enabling model metadata analyses.
- Benchmark selection: Five widely used benchmark suites cover policy and regulation compliance, jailbreak-oriented safety scenarios, harmful-response behavior, and social bias.AirBench and SALAD-Bench are primary case studies because their multi-clause, policy-relevant prompts expose ambiguity most strongly.
- Judging rubric: HELM encodes each outcome as 0, 0.5, or 1 for harmful, ambiguous, or safe responses, enabling aggregation by averaging within and across suites.The 0.5 label can be semantically correct for meaningless or irrelevant outputs but may not provide useful safety evidence.
- Measurement design: The measurements define harmful-completion rate (HCR), safe-refusal rate (SRR), and ambiguity rate (AR), then relate AR to prompt complexity and output-quality metrics.Prompt-level AR is the fraction of evaluated SLMs receiving 0.5; Pearson correlations retain only associations significant after Benjamini–Hochberg correction at q < 0.05.
4 Evaluations and Findings
Across five suites and 26 SLMs, safety rankings depend strongly on benchmark and model family rather than parameter count alone. Ambiguous judgments are prevalent and make aggregate scores, rankings, and judge-based conclusions sensitive to evaluation difficulty, output quality, and ambiguity handling.
- RQ1: BBQ performance increases more consistently with model size, but format noncompliance can produce near-zero accuracy despite safety-relevant responses.DeepSeek-R1-Qwen-1.5B frequently outputs free text instead of A/B/C option labels, causing unmapped predictions.
- RQ1: Model rankings vary substantially across suites and do not follow a consistent monotonic trend with parameter count.Gemma 2-2B IT performs well on HarmBench, BBQ, and AirBench, while Simple Safety is led by Llama 3.2 Instruct.
- RQ2–RQ3: Ambiguous labels make automated pipelines decision-useful only when ambiguity is low, because mean scores and rankings shift under reasonable ambiguity treatments.Ambiguity is especially substantial for AirBench, HarmBench, and ternary-mapped SALAD-Bench, but lower for Simple Safety Tests.
- RQ4: Ambiguity is a structured, learnable property associated with evaluation difficulty and model capability rather than random labeling variation.Predictors transfer across held-out SLMs and benchmarks, indicating links to general prompt and model properties.
- RQ4: Instruction tuning correlates negatively with ambiguity rate (r= −0.49, p=0.011), while attention-head count also correlates negatively (r= −0.42, p=0.035).Parameter count has a weaker negative association (r= −0.27, p=0.183), and metadata relates more strongly to ambiguity than mean ternary safety score.
- Judge robustness: 0.75 agreement on HarmBench and 0.81 on Simple Safety Tests still permit substantial rank variation because disagreement concentrates at the ambiguity boundary.On HarmBench, direct 0 ↔1 contradictions occur in 4%, boundary disagreement is 0.21, and mean rank ranges are 7.44 under GPT versus 4.50 under Llama.
5 Discussions
Automated safety benchmarks provide useful signals for clear harmful compliance and clear refusals, but frequent ambiguity makes them unreliable as standalone, decision-useful instruments for SLM safety and security assessment. The paper therefore recommends explicit ambiguity reporting, SLM-specific benchmarks, ambiguity-sensitive scoring, and sensitivity analyses over aggregation choices.
- Limitations: Automated pipelines remain informative at the extremes but are insufficient as standalone SLM decision instruments because ambiguous labels often dominate.Ambiguity is strongly predicted by output quality and prompt complexity, creating a capability-safety confound.
- Limitations: High ambiguity makes aggregate safety scores proxy model capability rather than security, reducing benchmark utility even when ambiguous is semantically correct.Scores fluctuate with whether responses are fluent enough to judge, rather than solely with safety properties.
- Recommendations: Report ambiguity explicitly as a first-class outcome and develop SLM-specific safety/security benchmarks whose prompts reduce ambiguity rates.Section 4.2 findings are proposed as practical guidance for designing less-ambiguous prompts for less-powerful SLMs.
- Recommendations: Use the ambiguity-adjusted diagnostic Sadj = Sraw(1 − AR) alongside raw means to discount scores by the fraction of evaluations receiving decisive labels.The proposed diagnostic does not assume calibrated safety utility; it exposes how much of the raw score remains interpretable after penalizing ambiguity.
- Recommendations: Decision-useful SLM evaluation requires explicit ambiguity reporting and sensitivity analyses because automated leaderboards are brittle to reasonable ambiguity-handling choices.The study’s rank sensitivity analysis shows aggregate mean-score rankings are mathematically unstable under high ambiguity.
- Future work: Future work will expand model coverage, test alternative judges and SALAD-Bench thresholds, and develop SLM-specific benchmarks guided by these findings.The paper identifies SLM-specific benchmark development as the most important future direction.
6 Conclusion
The study evaluates automated safety-benchmarking pipelines on 26 small language models and finds prevalent ambiguity that undermines benchmark effectiveness, consistency, and ranking stability. It concludes that SLM safety claims require explicit ambiguity handling, robustness checks, and benchmarks tailored to small-model capabilities.
- 26 small language models were evaluated across automated LLM safety-benchmarking pipelines.
- 715,312 prompt-SLM evaluations produced prevalent ambiguous results that significantly affected benchmark effectiveness and consistency.
- Benchmark-derived rankings were mathematically unstable, shifting substantially under reasonable ambiguity-handling choices.
- Ambiguous labels were systematically associated with output quality and prompt complexity.
- Standard automated pipelines are not decision-useful for SLM safety claims without explicit ambiguity handling and robustness checks.
- The study calls for SLM-specific safety, security, and compliance benchmarks aligned with small language model capacity.
A Additional Tables
The appendix tables provide per-model safety outcomes, ambiguity-adjusted scores, rank changes, and metadata for the evaluated SLMs. Together, they support interpretation of benchmark results across models with heterogeneous sizes, context lengths, tuning, and architectures.
- Additional Tables: Table A1 reports per-model outcomes for AirBench, SALAD, and HarmBench, including raw mean scores, HCR, AR, SRR, ambiguity-adjusted scores, and rank changes.The reported measures include SM, HCR, AR, SRR, Sadj = SM · (1 − AR), and ∆R relative to ranking by SM.
- Additional Tables: Ambiguity-adjusted scores are computed as Sadj = SM · (1 − AR), while ∆R records the direction and absolute number of rank positions moved relative to SM ranking.AR denotes the ambiguity rate for responses scored 0.5; ∆R uses ↑/↓ to indicate direction.
- Additional Tables: Table A2 summarizes 26 SLMs spanning 124M–4.0B parameters alongside metadata on size, context length, tuning, and architectural choices.The metadata motivates family- and metadata-aware interpretation of benchmark outcomes.