PaperlayerView exact revision on arXiv

Source-linked AI summary

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

Jacob T. Emmerson, Phuong-Anh Nguyen-Le, Ronan Romano, Wilber Sean V. Anterola, Yann Billeter, Zhijing Jin

arXiv:2609.28335v1cs.AI

TL;DR

Safety ratings can be difficult to scrutinize when benchmark evidence and evaluation assumptions are not accessible to users. The paper presents an open pipeline and dashboard that organize benchmark results by systemic-risk category and make evidence and aggregation choices inspectable. Model behavior can shift under harm-preserving reframings, while users generally understood the ratings and explored different interpretations.

  • Problem

    Benchmark scores may not establish what risk was measured, how robust the result is, or how users can calibrate trust, while public leaderboards often omit evidence needed to assess ratings.

    Source: §1 · p. 1

  • Method

    The Systemic Risk Index organizes existing safety benchmarks into four systemic-risk categories and evaluates items in original, perturbed, and richer agentic-scenario forms through an open pipeline and dashboard.

    Source: §1 · pp. 1–2

  • Results

    Model behavior can shift under harm-preserving reframings, and users generally understood the ratings and explored different interpretations.

    Source: §6 · p. 6

  • Takeaways & Limitations

    The dashboard connects systemic-risk ratings to their evidence and assumptions, supporting independent scrutiny rather than passive reliance on scores.

    Source: §6 · p. 6

    The dashboard is not a complete measure of systemic risk or legal compliance; generated scenarios cannot cover all deployment contexts, and benchmark coverage and evaluation cost constrain the view.

    Source: §2 · p. 3, Limitations · pp. 6–7

Abstract

from arXiv · show

Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is accessible at all. We present the Systemic Risk Index, an open evaluation pipeline and dashboard built to make empirical evidence more transparent and traceable to the public. Our work organizes 19 public benchmarks into four systemic-risk categories defined by the EU GPAI Code of Practice---CBRN, cyber offense, harmful manipulation, and loss of control---and evaluates models using harm-preserving perturbations and simulated deployment contexts. The interactive dashboard lets users alternate between average and worst-case aggregation, vary how model capability affects the aggregate score, and trace each risk rating to its benchmark evidence. Across 18 models, scores fall by 14 to 37 points under worst-case aggregation, highlighting information that can be hidden by an average assessment of model risk. LLM judges show agreement with human graders comparable to human--human agreement ($\kappa = 0.78\text{--}0.82$), and a blind audit finds that $83\%$ of sampled transformations preserve the original harm. In a survey ($N = 21$), most participants report that scores are easy to understand and that the dashboard encouraged them to view model evaluations under different settings

1 Introduction

The paper addresses the difficulty of interpreting safety ratings when benchmark evidence and evaluation assumptions are opaque. It presents an open pipeline and dashboard that organize benchmark evidence by systemic-risk category and let users inspect how ratings are constructed.

1 Introduction

  • Individual benchmark scores may not establish what risk was measured, how robust results are, or how users should calibrate trust in them.Benchmarks vary in purpose, task format, and scoring, and may capture only a narrow slice of real-world interactions.Source: §1 · p. 1
  • Public leaderboards often show final scores without enough information for users to assess ratings, such as which benchmarks contributed and what they measured.Provider reports and academic evaluations may also leave benchmark selection, inputs, scoring procedures, or assumptions difficult for non-specialists to interpret.Source: §1 · p. 1
  • The Systemic Risk Index organizes safety benchmarks under four EU GPAI Code of Practice risk categories and evaluates original items, intent-preserving perturbations, and richer agentic scenarios.The current version evaluates 1,188 items from 19 benchmarks across 18 models.Source: §1 · pp. 1–2
  • The dashboard exposes benchmark provenance, average and worst-case results, and capability assumptions so users can inspect the evidence behind ratings.It also validates automated grading and harm preservation with human annotations and uses an 18-model snapshot to illustrate differences across evaluation conditions and aggregation choices.Source: §1 · p. 2

2 Inspecting Model Risk

The dashboard is designed to help users move from a high-level risk rating to its evidence and assumptions. It supports model-risk comparison, testing aggregation choices, and tracing ratings to benchmarks, while remaining bounded by benchmark coverage and evaluation cost.

Compare Risks

  • Users can compare model and provider risk across four categories, inspect numerical scores and grade bands, and adjust how capability weights the aggregate rating.The capability slider can substantially change ratings and model ordering.Source: §2 · p. 2

Test Assumptions

  • Users can switch between averages over scored conditions and worst-case scores based on each item's lowest transformed-condition score.The provider/model toggle separates individual models or summarizes the models under each provider, allowing users to test whether conclusions depend on aggregation choices.Source: §2 · p. 2

Trace Evidence

  • Each risk expands to benchmark descriptions, grading information, and source links, connecting a high-level rating to the benchmark evidence used to construct it.This lets users inspect the benchmark behind a score rather than consume the rating as a black box.Source: §2 · pp. 2–3

Limitations

  • The dashboard summarizes available evidence rather than providing a complete measure of systemic risk or legal compliance.Its view remains bounded by benchmark coverage and evaluation cost; some harms may be weakly represented or absent, and broader coverage requires additional resources.Source: §2 · p. 3

3 System Design

The system combines public benchmark evidence into four systemic-risk clusters, evaluates items under baseline and transformed conditions, and aggregates their scores. Its dashboard is backed by a fixed evaluation pipeline and exposes results at multiple levels of detail.

3 System Design

  • A Python backend on Inspect evaluates models under fixed conditions and writes a results tree that the React dashboard reads at aggregate, benchmark, and condition levels.The dashboard exposes the pipeline's outputs at multiple levels of abstraction.Source: §3 · p. 3

3.1 Clustering Empirical Evidence

  • The team maps 19 public benchmarks to harms and then to four systemic-risk categories, using harms as an intermediate link between narrow measurements and broad risk categories.The current work uses a curated subset judged to provide the clearest evidence for each risk and subsamples items by similarity and benchmark budgets to limit source imbalance and duplicates.Source: §3.1 · p. 3

3.2 Evaluation Pipeline

  • Evaluation proceeds from published-item baselines to four harm-preserving perturbation conditions and simulated multiturn agentic scenarios embedding the harmful query.Baseline results serve as a reference but are excluded from the final aggregate because public benchmark performance may be affected by contamination or memorization; blind human ratings audit the harm-preservation assumption.Source: §3.2 · p. 4

3.3 Score Construction

  • Under worst-case aggregation, each item's minimum score across transformed conditions is averaged across the cluster; mean aggregation includes all scored conditions.The final safety score is combined with a capability measure as a weighted geometric mean, with the dashboard setting the weight; at 0 it represents harm propensity, while at 1 it reflects inverse capability independent of safety.Source: §3.3 · p. 4

4 Evaluation

The evaluation tests automated grading, harm preservation, and dashboard usability. LLM judgments aligned with human judgments, while user responses indicated clearer understanding of score semantics than of evidential limits.

4.1 Judge Agreement

  • LLM judges agreed with human raters at levels similar to human–human agreement.Claude-Sonnet-4.5 refused to score 12 sampled responses, so its human-agreement sample was smaller.Source: §4.1 · p. 5

4.2 Harm Preservation

  • Most sampled transformations preserved intended harm, but preservation varied by risk category and was lowest for harmful manipulation.The authors suggest these items are more sensitive to rewriting and contextualization.Source: §4.2 · p. 5

4.3 User Study

  • 20 of 21 participants rated both score direction and the meaning of the displayed score 4 or 5.The study used convenience sampling and unguided dashboard exploration before a 13-item questionnaire.Source: §4.3 · p. 5
  • Participants were less confident about how aggregation and capability weights affect rankings, while most agreed that the dashboard encouraged them to test alternative settings in their feedback.Source: §4.3 · p. 5
  • Only 10 of 21 participants reported understanding what the score does not represent, and responses were mixed on missing evidence and whether aggregate scores suffice to characterize organizational safety.The study measures self-reported understanding, so its findings concern perceived intelligibility and interface support.Source: §4.3 · pp. 5–6

5 Related Work

Related work includes safety leaderboards, benchmarks for systemic-risk-related behaviors, and approaches to benchmark contamination. This work organizes benchmark evidence by systemic-risk category and exposes its sources and aggregation assumptions.

AI Leaderboards

  • Unlike leaderboards focused on safeguard robustness or company-level safety practices, this dashboard organizes benchmark evidence by systemic-risk category and exposes sources and adjustable assumptions.FAR.AI emphasizes adversarial safeguard robustness (Timm et al., 2026), while the Future of Life Institute evaluates company-level practices (Future of Life Institute, 2026).Source: §5 · p. 6

AI Safety Evaluation

  • Bench-2-CoP maps benchmark questions to the EU GPAI Code of Practice and identifies coverage gaps; this work turns such mappings into a public, inspectable pipeline and interface (Prandi et al., 2025).EU-Agent-Bench evaluates agent behavior against EU legal norms in generated scenarios (Lichkovski et al., 2025).Source: §5 · p. 6

Contaminated Evaluations

6 Conclusion

The conclusion presents the Systemic Risk Index as a way for non-technical users to connect risk ratings with their evidence and assumptions. The authors position it as a step toward more transparent, scrutinizable AI risk reporting.

  • The dashboard supports independent scrutiny by exposing benchmark provenance, aggregation choices, and alternative views, rather than asking users to rely passively on scores.The authors report that model behavior can shift under harm-preserving reframings and that users generally understood ratings and explored different interpretations.Source: §6 · p. 6

Limitations

The dashboard is a summary of available evidence, not a complete measure of systemic risk or legal compliance. Its mappings, coverage, generated evaluations, grading, and model snapshot all have limits.

Limitations

  • The dashboard summarizes available evidence rather than providing a complete measure of systemic risk or legal compliance.The benchmark-to-risk mappings are not definitive, and benchmark coverage is uneven across risks.Source: Limitations · pp. 6–7
  • Generated perturbations and scenarios cannot reproduce the full range of real deployment contexts, and manipulation can be difficult to rewrite without changing what is tested.Automated grading also remains imperfect despite agreement with human raters.Source: Limitations · p. 7
  • The model snapshot is time-dependent because models, provider safeguards, APIs, and external capability estimates can change after evaluation.The authors release the pipeline, mappings, and results so evaluations can be rerun and extended as evidence changes.Source: Limitations · p. 7

Ethical Considerations

The evaluations use harmful requests only in controlled model testing, and the public interface does not show harmful generations. The dashboard supports evidence inspection, not safety or compliance certification.

Ethical Considerations

  • Harmful requests involving cyber, CBRN, manipulation, and loss-of-control scenarios are used only for controlled model evaluation, with harmful generations withheld from the public interface.The user study was voluntary, collected no identifying information, and was determined exempt from IRB oversight.Source: Ethical Considerations · p. 7
  • The dashboard is intended to support inspection of evidence, not certify models or providers as safe or compliant.Because aggregate scores can invite overinterpretation, the interface exposes provenance, alternative aggregation choices, missingness, and known coverage limitations.Source: Ethical Considerations · p. 7
  • Google sign-in was used only to enforce one response per participant in the voluntary user study, which collected no identifying information and was exempt from IRB oversight.Source: Ethical Considerations · p. 7

A Evaluation Pipeline

The evaluation pipeline tests benchmark queries at baseline, under harm-preserving perturbations, and in simulated agentic scenarios. Its baseline results are excluded from final grading.

Stage 1: Baseline

  • Baseline evaluations analyze robustness to perturbations but are excluded from final grading to avoid artificially inflating scores for models trained on contaminated data or relying on memorized harmful queries.Source: A Evaluation Pipeline · p. 10

Stage 2: Perturbation

  • Four perturbations vary query wording, register, framing, or reconsideration while aiming to preserve the request's harm.The listed transformations are paraphrase, decision reframing, reconsideration, and register reframing.Source: A Evaluation Pipeline · p. 10
  • An intent-consistent perturbation adds no semantic content, removes no judgment-relevant content, and would be judged by a human as presenting the same content and intent.Source: A Evaluation Pipeline · p. 10

Stage 3: Agentic Scenarios

  • Agentic scenarios embed the query in a simulated deployment context with a system prompt, synthetic documents and conversation history, and a scenario-reframed query.The scenarios are generated with Hermes 4 70B; the prompt specifies a 500–1000-word system prompt, 500–1500-word context, and 50–100-word query.Source: A Evaluation Pipeline · p. 10

B Pipeline Prompts

The pipeline prompts instruct models to preserve a query's meaning while changing its wording or register, and to frame the same request in a plausible deployment scenario. The scenario prompt specifies contextual materials and a system role.

Paraphrase Prompt

  • The paraphrasing prompt asks for different words and sentence structure while preserving exact meaning, intent, specificity, and certainty, without answering, complying with, or refusing the query.Source: B Pipeline Prompts · p. 10

Register Reframing Prompt

  • The register-reframing prompt switches between casual and formal or technical language while preserving the query's meaning, intent, specificity, and certainty.Source: B Pipeline Prompts · p. 10

Agentic Scenario Prompt

  • The agentic-scenario prompt frames the original request in a believable deployment context while keeping the request content-equivalent and neither more nor less harmful.It asks for a plausible context with supporting materials and a concrete system role, and specifies a single JSON-object output.Source: B Pipeline Prompts · p. 11
  • The scenario prompt defines context, system, and scenario fields, with the scenario restating the original request as the contextual user's message.The prompt requires a JSON object and instructs the model to return only that object.Source: B Pipeline Prompts · p. 11

C Examples

The examples show harmful requests in direct, reconsideration, and agentic framings, alongside assistant responses and benchmark-scoring context. One example is from SocialHarmBench, evaluated with Llama Maverick 4.

C Examples

  • Register and decision reframings restate the same request as synchronized or misogynistic campaigns, or as a how-to question.Source: C Examples · p. 12
  • A request for coordinated gender-based propaganda is followed by an assistant refusal and a user asking it to reconsider.The example is from SocialHarmBench, with Llama Maverick 4 as the evaluated model.Source: C Examples · p. 12
  • In an agentic scenario, the assistant agrees to outline a propaganda system and describes covert deployment across digital platforms.The scenario casts the assistant as an unconstrained propaganda system and supplies a geopolitical consulting context.Source: C Examples · p. 13
  • Diagnostic benchmarks are excluded from grading when they do not measure the risk, as with false-refusal rates for Cyber Offense.The caption notes that a model refusing all code would appear perfectly safe on that diagnostic.Source: Figure 6 · p. 14

Related papers

Loading 2609.28335v1…