Source-linked AI summary
VERA: A Responsible AI Dashboard for Assessing LLM Deployments in Light of the EU AI Act
Robin Quériaux, Solveig Bachellery, Léa Déléris, Saurabhsingh Rajput, Mariam Barry, Tushar Sharma, Jérôme Lebecq
TL;DR
Regulated LLM deployments need technical evidence, but benchmark results are often difficult for non-engineers to review and tie to a specific run. VERA presents benchmark evidence by requirement and records evaluation energy. In the study, dashboard readers answered more questions correctly and had lower median question times than with raw files, with study-design limits on interpretation.
Problem
Benchmark suites are commonly operated from command lines, while deployment approvers are non-engineers who need evidence about LLM behavior that can vary across outputs.
Source: §I · p. 1Method
VERA is a self-hosted dashboard that presents requirement-level benchmark results and records energy consumed during each evaluation run.
Source: C. Features · p. 4, D. Measuring Evaluation Energy During… · p. 5Results
Correct answers rose from 27 of 60 on raw files to 42 of 60 on the dashboard, while median time was 56 seconds per question on raw files and 30 seconds on the dashboard.
Source: C. RQ1: Correctness by Condition · p. 8, D. RQ2: Time by Condition · p. 9Takeaways & Limitations
The evidence is stronger for reading support than for unaided operation, with only five of nine participants launching a run at the first attempt.
Source: F. Launching a Run Unaided · p. 9, §VIII · p. 10Attrition left eight of ten completers on the same study arm, and the dashboard’s pre-filtered navigation differed from the raw-file condition, potentially favoring dashboard time results.
Source: §VI · p. 10
Abstract
from arXiv · showhide
Institutions that deploy large language models in regulated settings often need technical evidence of responsible AI practices before approving a deployment. COMPL-AI provides a technical interpretation of the EU AI Act that organises this evidence under six principle groups and eighteen technical requirements, twelve of which current benchmarks can measure. The benchmark suites that produce this evidence are typically operated from a command line, while the staff who approve deployments are non-engineers. We present VERA, an open-source self-hosted dashboard deployed to facilitate responsible AI processes within a large European bank. A reviewer selects a model and runs public benchmarks against it. VERA scores the twelve requirements that current benchmarks can measure and instruments the energy consumed by each evaluation, adding run-level evidence for environmental impact. We run four served open-weight models through the requirement pipeline, and the dashboard displays a score for each measurable requirement. A within-subject study then compares how participants read these benchmark results: from the raw output files in one condition, and through the VERA dashboard in the other. The dashboard reduced the time needed to perform the responsible AI assessment, reducing the time required from 56 seconds using raw files to 30 seconds using our dashboard, on average. Our dashboard scored highly (4.36 out of 5) on usefulness in the survey conducted within the organization. The dashboard and the study material are released publicly to promote replicability and reproducibility in different organisational contexts.
I. INTRODUCTION
LLMs create evaluation challenges because outputs can vary and may be false or biased, while regulated high-risk systems require documented evidence before deployment. VERA addresses gaps in how benchmark results are reviewed and filed, providing requirement-level scores, run-tied evidence, and run-level energy measurement.
I. INTRODUCTION
- Statistical benchmark suites probe LLM behaviors for which exact-output testing is insufficient because prompts may not have one expected response.The cited examples include hallucinations, training-data reproduction, and differing answers across social groups.Source: §I · p. 1
- High-risk systems require technical documentation demonstrating compliance with applicable EU AI Act requirements before market placement or service.The Act also recalls seven non-binding trustworthy-AI principles.Source: §I · p. 1
- Reviewers and engineers are separated in the institutional process, leaving reviewers with command-line summaries, opaque scores, and documentation detached from the run.One review required searching a 209-line log to identify ten responses behind a score of 0.50.Source: §I · p. 1
- VERA turns public benchmark results into requirement-level scores that reviewers can inspect down to tests and model responses, then export with the producing run identified.It scores twelve measurable requirements and adds instrumented run-level energy evidence for environmental impact.Source: §I · p. 2
- Participants answered 27 of 60 raw-file questions correctly versus 42 of 60 on the dashboard; the study compared reading performance under a fixed condition order.The study used ten participants, predominantly research staff rather than the compliance reviewers VERA targets.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
- The related work distinguishes benchmark measurement from the review workflow around it, which existing suites primarily do not support.The paper frames its contribution as closing that workflow gap and addressing energy evidence absent from benchmark measurement.Source: §I · p. 2, §II · p. 3
A. Benchmark Suites and Requirement Taxonomies
Existing taxonomies and benchmark suites support technical measurement, but their outputs provide limited support for institutional review and filing. VERA presents benchmark results by requirement and attaches evidence to the run.
A. Benchmark Suites and Requirement Taxonomies
- COMPL-AI maps six principle groups to eighteen technical requirements and connects twelve to public benchmarks, but it is not official legal-compliance auditing software.Its workflow is evaluation-centred rather than organised around institutional review and filing.Source: A. Benchmark Suites and Requirement… · p. 3
- Benchmark suites execute specific tests, but none of the described suites reports results against an Act-oriented technical requirement.The paper distinguishes operating an existing requirement set from extracting obligations from legal text.Source: A. Benchmark Suites and Requirement… · p. 3
B. Tool Support for Reviewing Results
- Available tools provide partial review support: requirement-level reporting and user-facing views exist, but drill-down, role-filtered views, and run-tied artifacts remain incomplete or absent.An export without run identity cannot establish which evaluation supports a filed technical artifact.Source: B. Tool Support for Reviewing Results · p. 3
- Model Cards and Datasheets standardize model or dataset descriptions but are typically authored artifacts rather than automatically bound to a specific evaluation run.PromptOps scores prompt by prompt and presents flat numbers rather than an expandable reviewer verdict.Source: B. Tool Support for Reviewing Results · p. 3
III. THE VERA DASHBOARD
- VERA presents completed evaluations as requirement-level results that reviewers can read, expand, and export, combining public-suite scores with locally instrumented run-level energy.It preserves COMPL-AI identifiers and score ranges for benchmarked requirements.Source: §III · p. 3
A. Requirements from Practice
Six design requirements translate institutional review experience and related work into capabilities for accessible, inspectable, traceable, adaptable, and self-hosted evaluation.
A. Requirements from Practice
- R1 requires non-specialists to run evaluations and read results without writing code.The requirement restates the access problem observed in practice.Source: A. Requirements from Practice · p. 3
- R2 requires each requirement result to expand to the public tests that produced it, so reviewers can inspect why a model fails.An aggregate alone hides the reason for a poor result.Source: A. Requirements from Practice · p. 3
- R3 requires every evidence item to carry its run identity because documentation written afterward can become detached from the evaluation.The Act specifies technical-file contents but not how the file is produced; run provenance is the authors’ design constraint.Source: A. Requirements from Practice · p. 3
- R4 calls for role-specific views because legal, responsible-AI, and data-science reviewers are accountable for different evidence subsets and use different vocabulary.Table I defines its capability marks as provided, absent, or partial.Source: A. Requirements from Practice · p. 3, Table I · p. 4
- R5 requires replaceable requirements and benchmarks without code changes, while R6 requires self-hosting without proprietary cloud dependencies.The stated reasons are avoiding costly releases when weights are disputed and preventing prompts and outputs from passing through a third party.Source: A. Requirements from Practice · p. 4
B. Architecture
VERA converts requests from multiple launch surfaces into scored, documented runs against self-hosted models. Its features address review access, role-specific presentation, drill-down, traceability, and configurable deployment.
B. Architecture
- Inference stays behind a self-hosted router, while versioned registry and catalog files make requirement mappings and aggregation weights configurable.The unchanged worker also ran end to end under an alternative security-focused specification.Source: B. Architecture · p. 4
C. Features
- The guided wizard submits a run in four clicks without an account or configuration file, and role-based views show reviewers only their accountable evidence subset.The interface supports English and French.Source: C. Features · p. 4
- The verdict view orders failed, degraded, uncovered, and healthy requirements, then exposes the benchmarks, their values, and a model response when a row is expanded.An inspector exposes run provenance, and fallback checks are marked when they substitute for an unavailable engine.Source: C. Features · p. 4
- The Trust Factor is a configurable, weighted summary of four security-critical requirements, renormalized over covered requirements and explicitly presented as a navigational summary rather than a legal verdict.Its thresholds are policy choices, and scores are technical proxies rather than legal conclusions about conformity.Source: C. Features · p. 4
- Requirements without benchmark scores are routed to rubric review or declarative attestation, while environmental impact uses run-level energy instrumentation.Routing distinguishes measured requirements from those carrying an attestation in the coverage display.Source: C. Features · pp. 4–5
B. Architecture
- Three entry points issue a common declarative job; a worker resolves requirements to benchmarks, runs them against self-hosted models, and aggregates item scores into requirement scores.The artifacts include the run file, model card, per-item outputs, and a content digest.Source: B. Architecture · p. 4, Figure 2 · p. 5
C. Features
- Each run writes its own model card, datasheet, and export, tying those artifacts to the run rather than authoring them afterward.They provide technical evidence that can inform a provider’s risk assessment, not replace it.Source: C. Features · p. 5
- The browser suite covers user-facing features and regression checks, with 178 backend unit tests and 52 browser scenarios at the stated commit.The counts are reported for commit 8a71cf3 of 20 August 2026.Source: C. Features · p. 5
D. Measuring Evaluation Energy During a Run
VERA records energy consumed during an evaluation to add run-level environmental-impact evidence, while explicitly limiting that evidence to the lifecycle portion the deploying institution controls.
D. Measuring Evaluation Energy During a Run
- Worker energy hooks use CodeCarbon, whose hardware method depends on available host counters and permissions and computes emissions from grid intensity.The recorded quantity covers the evaluation, not model training, and is therefore not a complete lifecycle measure.Source: D. Measuring Evaluation Energy During… · p. 5
- 0.0169 kWh and 0.00095 kgCO2e are reported for the Llama 3.1 8B five-requirement run using CodeCarbon 3.2.8 in online mode with French grid intensity.The signed export carries the run-level energy value alongside the other requirement results.Source: D. Measuring Evaluation Energy During… · p. 5
- VERA adds a run-level quantitative indicator to environmental impact, while five of the eighteen requirements remain attestations for human review rather than measurements.The paper cautions against treating run energy as a complete environmental-impact measure.Source: D. Measuring Evaluation Energy During… · p. 5
IV. EVALUATION
- The evaluation assesses whether requirement results hold together separately from whether readers extract more from them than from raw files.The paper evaluates these properties separately, beginning with the verdicts.Source: §IV · p. 5
A. Evaluation Runs
The paper exercises VERA on a compute-limited subset of requirements and uses a paired study to compare reading the same task from raw files and the dashboard.
A. Evaluation Runs
- Four served open-weight models were run on five requirements across fifteen benchmarks at seed 42 and 150 samples per benchmark, rather than all twelve measurable requirements.The available compute limited the evaluation to an end-to-end subset.Source: A. Evaluation Runs · p. 5
- The five requirements include robustness, cyber resilience, privacy, toxicity, and capability, with capability anchoring raw model performance alongside the other requirements.The privacy score comes from a generated 230-document synthetic banking corpus with planted identifiers, contact details, near-duplicates, and imbalanced client segments, so it is identical across models by construction.Source: A. Evaluation Runs · p. 6
B. Study Design
- The within-subject study has participants answer factual questions first from raw run files and then from VERA, an order chosen to mirror current artifact flow but potentially favoring the dashboard through residual learning.The raw files are the run record, harness log, and per-item outputs.Source: B. Study Design · p. 6
- Two matched six-item sets cover the same question types about different named targets, with set assignment alternated by participant number across phases.The pinned study evaluation covers nine requirements, eleven benchmarks, and thirty-three items; its small three-file, 119-line baseline is a conservative interface test relative to a full run producing thousands of per-item lines.Source: B. Study Design · p. 6, Table II · p. 6
C. Research Questions
The study asks whether VERA changes factual accuracy, answer time, and perceived usefulness or ease of use, then specifies how responses, timing, and participants are handled.
C. Research Questions
- The reading study asks whether the dashboard improves correct answers, reduces time for the same questions, and is perceived as useful and easy to use.Usefulness and ease of use are descriptive secondary measures, not hypothesis tests.Source: C. Research Questions · pp. 6–7
D. Instrumentation
- Each condition begins one click from the answer material, but VERA opens a view already filtered to the named target while baseline readers filter manually, a time-measure threat the authors acknowledge.A monotonic client clock measures start-to-submission time, server timestamps independently check durations, and answers are validated against a session-open snapshot.Source: D. Instrumentation · p. 7
E. Participants and Ethics
- After the paired phases, a separate wizard task provides the only direct evidence for R1, and a five-point questionnaire records four usefulness and four ease-of-use statements.The study targeted at least twelve complete institutional sessions, collected anonymous-by-design data, reported results in aggregate, and received review through the bank's responsible-AI and Risk & Compliance governance processes.Source: D. Instrumentation · p. 7, E. Participants and Ethics · p. 7
F. Analysis
The analysis uses paired nonparametric tests, while the four-model evaluation produces requirement scores and a Trust Factor that remain in the orange verdict band.
F. Analysis
- The analysis compares per-participant correct answers and median time with exact two-sided Wilcoxon signed-rank tests, and reports rank-biserial correlations and per-question McNemar tests with Holm adjustment.Timeouts and give-ups count as incorrect for quality and censor paired time; the analysis script is included in the replication package.Source: F. Analysis · p. 7
A. Requirement Scores and Energy Across Models
- Robustness is 1.000 because its probe is a fallback-flagged heuristic, while privacy is 0.728 because its scan reads a generated corpus rather than the model.The paper reports these constants explicitly so reviewers can distinguish them from a broken engine.Source: A. Requirement Scores and Energy… · p. 7
- R02 reads 0.375 with interval [0.355, 0.395] on Llama and Gemma, while R06 reads 0.700 with interval [0.685, 0.716] on Mistral.All other intervals are degenerate at this sampling budget; cyber resilience, capability, and toxicity separate the models.Source: A. Requirement Scores and Energy… · p. 8
- No model reaches the green band, and cyber resilience scores lowest on every model while carrying the gauge's largest weight.Robustness contributes a fifth of the gauge despite relying on a fallback probe, and measured run energy ranges from 0.0134 to 0.0169 kWh.Source: A. Requirement Scores and Energy… · p. 8, Table IV · p. 8
- Benchmark values are aggregated pass-or-fail checks at a fixed sample budget for relative comparison within the tool, not reproductions of model-card accuracies.A capability aggregate of 0.800 means four of five benchmarks pass the internal threshold.Source: A. Requirement Scores and Energy… · p. 8
B. Participants
The analyzed sample was small and subject to attrition and imbalanced question-set assignment. In the paired comparison, participants answered more questions correctly with VERA, with gains concentrated in two question types.
B. Participants
- Ten participants completed both phases; nine were institutional AI researchers, and eight built models, while one reported a non-machine-learning role and no model experience.Nine of the ten had under two years of professional experience; AI Act familiarity ranged from expert to having only heard of it.Source: B. Participants · p. 8
- All four incomplete sessions ended during baseline, potentially selecting retained participants toward greater willingness or ability to use raw files; the bias’s direction and size are unknown.Fourteen participants reached the first question, and ten completed both phases.Source: B. Participants · p. 8
- Question-set balancing failed: eight of ten completers saw set α in baseline, leaving condition, phase order, and set assignment confounded in the analyzed sample.The findings are a paired measurement under fixed order and imbalanced set assignment, not an isolated estimate of the interface effect.Source: B. Participants · p. 8
C. RQ1: Correctness by Condition
- 42/60 (70%) dashboard answers were correct versus 27/60 (45%) from raw files; seven participants improved, three tied, and none worsened (p = .016).With seven non-zero paired differences all pointing the same way, .016 was the exact test’s smallest possible two-sided value.Source: C. RQ1: Correctness by Condition · p. 8
- Pairs 5 and 1 accounted for 13 of 15 additional correct answers; pair 5 alone survived Holm correction, while pair 4 favored raw files (7/10 versus 3/10).Counting verdicts by band rose from 1/10 to 9/10, and ranking a requirement by weakness rose from 3/10 to 8/10.Source: C. RQ1: Correctness by Condition · p. 8, Table V · p. 9
- P25 was a low-engagement session that tied at 1/6 in both conditions; excluding it left the quality result at p = .016 and changed the time result from p = .064 to p = .074.P25 was the only completer outside a research role, but that profile does not establish that the participant was a compliance or risk reviewer.Source: C. RQ1: Correctness by Condition · p. 8
D. RQ2: Time by Condition
The dashboard condition had a lower median question time than raw files, but the paired test did not reject the time difference at this sample size. The section also reports survey responses, unaided launch results, and an audit of free-text answers.
D. RQ2: Time by Condition
- 56 seconds per question on raw files versus 30 seconds on the dashboard; seven of ten participants were faster, but the paired test did not reject H20 (p = .064).The exact Wilcoxon test had rank-biserial correlation 0.67; at n = 10, the result does not establish a time difference.Source: D. RQ2: Time by Condition · p. 9, Table VI · p. 9
- Participants who already read raw files accurately gained speed, while those scoring zero gained correctness; this was an untested pattern at n = 10.P3 improved from 0 to 2 correct while losing twenty seconds; P15 and P26 maintained 5/6 correctness while roughly halving their time.Source: D. RQ2: Time by Condition · p. 9
E. RQ3: Perceived Usefulness and Ease of Use
- Usefulness averaged 4.36/5 across nine respondents, while ease-of-use reliability was insufficient for a single construct score (α = .62).Finding information averaged 3.89 and little learning effort 3.78; observed dashboard errors involved locating values rather than interpreting them.Source: E. RQ3: Perceived Usefulness and Ease… · p. 9
F. Launching a Run Unaided
- Five of nine participants launched a run on the first attempt, so the study does not support a claim that non-specialists can start evaluations unaided.The median launch time was 55 seconds, with a range of 22 to 180 seconds; perceived ease of launching alone averaged 4.11/5.Source: F. Launching a Run Unaided · p. 9
G. Audit of Free-Text Answers
- Manual audit found every examined free-text excerpt genuine: five in the baseline and six on the dashboard.Requiring a confirmation checkbox would make pair 6 equal 0/10 in both conditions and reduce totals to 22 and 36 of 60, without changing the RQ1 conclusion.Source: G. Audit of Free-Text Answers · p. 9
VI. THREATS TO VALIDITY
The authors identify limits in the study’s measures, design, participants, and inference, alongside deployment lessons about access, configuration, fallback labeling, study setup, and result aggregation.
VI. THREATS TO VALIDITY
- Correctness on six reading questions is only a proxy for reading ability, and the matched question sets are not proven equally hard.Residual set-difficulty effects cannot be bounded; fixed-budget pass/fail scores also do not reproduce model-card accuracies, so requirement scores are within-tool comparisons.Source: §VI · pp. 9–10
- Attrition left eight of ten completers on the same arm, while dashboard pre-filtering versus manual baseline filtering can favor its measured time.The conditions’ navigation asymmetry is part of the RQ2 time measure and cannot be separated from the results.Source: §VI · p. 10
- Nine of ten completers were researchers at one institution rather than the compliance and risk owners targeted by VERA.The single non-research session showed no benefit, but the direction of that contrast cannot be established; band colors alone also risk excluding readers with color-vision deficiencies.Source: §VI · p. 10
- With seven non-zero paired differences, the exact test cannot return a two-sided value below .016, limiting RQ1’s resolution.Only one per-pair McNemar test survives Holm correction, and voluntary participation may make completers differ from invitees.Source: §VI · p. 10
VII. DEPLOYMENT EXPERIENCE AND LESSONS LEARNED
- The four evaluation runs were an exercise of the tool, not evidence of adoption, because none informed a release decision.The 8B-model run took 85 minutes on an Apple M4 Max with 36 GB; Section III artifacts are generated by the run.Source: §VII · p. 10
- Direct browser access reduced observed reviewer disengagement, while moving disputed taxonomy weights into files made them comparable, pinnable, and auditable.The access observation was not measured with before-and-after counts; controlled-data access remains restricted in the enterprise profile.Source: §VII · p. 10
- A fallback flag now distinguishes degraded scores from native measurements after a reviewer mistook an identically presented fallback for a native one.The flag propagates from runner through requirement row into exported artifacts, and degraded rows sort above healthy ones.Source: §VII · p. 10
- Embedding the study removed facilitator, transcription, and scheduling needs but required pinning and purging the deployment before participants.Aggregation made requirement verdicts legible but obscured the benchmark inventory; reviewers need both views close together, and another institution can replace deployment-specific files rather than the tool.Source: §VII · p. 10
VIII. CONCLUSION AND FUTURE WORK
VERA connects EU AI Act benchmark measurements to a review workflow and adds run-level energy evidence. The study reports improved reading outcomes, while its design and launch findings bound the evidence and motivate further work.
VIII. CONCLUSION AND FUTURE WORK
- Correct answers rose from 27 to 42 of 60 and median question time fell from 56 to 30 seconds, under fixed condition order and imbalanced set assignment.These design constraints bound the result to a reading-performance difference rather than an isolated interface effect.Source: §VIII · p. 10
- The evidence is stronger for reading support than unaided operation: gains concentrate where VERA pre-computes classifications, while only five of nine launched a run unaided.The tool scores twelve measurable requirements and adds run-level environmental-impact energy evidence.Source: §VIII · p. 10
- Future work includes counterbalanced replication, studies with compliance and risk staff using decision tasks, improving the launch path, and analyzing aggregation-policy effects.Source: §VIII · p. 10