Source-linked AI summary
A Four-Axis Trustworthiness Benchmark for LLM-as-Judge in Principle-Based Regulation
Dipankar Sarkar
TL;DR
Principle-based regulation requires LLM judges that are auditable, calibrated, and robust to paraphrase and adversarial gaming, not merely accurate on average. This paper benchmarks these four axes and finds that no method dominates, while the strongest judge collapses from 0.74 to 0.27 on keyword-stuffed Consumer Duty inputs.
Problem
LLM-as-judge for evaluative regulatory principles lacks evidence on auditability, calibration, paraphrase robustness, and adversarial robustness.
Method
The paper introduces Principle-Bench and Ceca, a calibrated assessor that evaluates four trustworthiness axes and provides exact per-exemplar counterfactual attributions.
Results
No method dominates all four axes; the LLM-judge drops from 0.74 to 0.27 on adversarial Consumer Duty inputs, with cross-family agreement falling to Cohen’s kappa = 0.16.
Takeaways & Limitations
Deployment-grade principle-based LLM judges should report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy and auditable attribution.
Takeaways & Limitations
The LLM-judge is not a deployable assessor by itself because its confidences require calibration and its narrative rationales lack exactly reproducible per-exemplar contributions.
Abstract
from arXiv · showhide
Principle-based regulation, with evaluative standards such as "fair, clear, and not misleading" or "deliver good outcomes", cannot be reduced to binary predicates, and LLM-as-judge is increasingly used as the substitute. Our position is that any such judge must be evaluated on four axes: accuracy, paraphrase robustness, adversarial robustness, and calibration. We release Principle-Bench, 168 cryptoasset financial-promotion scenarios mapped to two UK FCA principles, with paraphrase, adversarial keyword-stuffing, and boundary perturbations authored under a pre-registered rubric; the first benchmark covering all four axes for principle-based regulation. We also introduce Ceca (Calibrated Exemplar-Cluster Assessment): a calibrated, auditable assessor that emits exact per-exemplar counterfactual attributions. Across keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, and a calibrated cascade, no method dominates all four axes. A 120B LLM-judge, strongest on benign inputs, loses 47 accuracy points (0.74 to 0.27) on keyword-stuffed Consumer Duty inputs: "compliance theatre." A second judge from a different model family agrees only at Cohen's kappa = 0.16 on that split, localising the failure to the model rather than the corpus. Any deployment-grade LLM-judge for principle-based regulation must report per-principle adversarial deception and post-hoc calibration alongside aggregate accuracy.
1 Introduction
Principle-based regulations such as the FCA’s Consumer Duty require evaluative judgments that binary rule-checking cannot reliably provide. LLM-as-judge is a promising alternative, but trustworthy deployment requires evaluations that are auditable, reproducible, calibrated, and robust to gaming.
- Motivation: FCA principles require promotions to be “fair, clear, and not misleading” and firms to “deliver good outcomes,” rather than satisfy a single boolean predicate.A promotion may include every required disclaimer while remaining misleading overall.
- Problem: Binary rule-based systems dominate automated compliance checking but cannot adequately verify outcome-based, evaluative regulatory principles.This paradigm underpins Datalog verifiers, semantic-graph checkers, and many production reg-tech stacks.
- Motivation: LLM-as-judge offers an alternative to direct principle evaluation, yet is opaque, poorly calibrated by default, and lacking regulator-interrogable per-example justification.The deployment question concerns whether judgments are auditable, reproducible, calibrated, and robust to gaming.
2 Contributions
This section introduces Principle-Bench, a perturbation-based benchmark for principle-based regulatory assessment, and Ceca, an auditable calibrated assessor. It also defines a four-axis comparison spanning heuristic, embedding, LLM-judge, and cascade methods.
- Principle-Bench: Principle-Bench contains 168 scenarios across two UK FCA principles, including original, paraphrase, adversarial, and boundary cases.Its perturbation suite comprises 100 original, 30 paraphrase, 28 adversarial, and 10 boundary scenarios.
- Ceca: Ceca is a calibrated exemplar-cluster assessor that provides exact per-exemplar counterfactual attribution for auditable assessment.The paper presents it as a transparency-by-design alternative to an opaque judge.
- Four-axis comparison: The study compares keyword counting, three sentence-transformer embedders, an open-weight LLM-judge, a second model-family judge, and a calibrated cascade across four axes.The comparison includes both an open-weight judge and a second judge from a different model family.
3 Principle-Bench and Ceca
This section introduces Principle-Bench, a preregistered 168-scenario benchmark for principle-based regulation, and Ceca, an auditable calibrated assessor. Assessors are evaluated across accuracy, paraphrase robustness, adversarial robustness, and calibration using exemplar-traceable confidence outputs.
- Ceca: Ceca represents each principle through weighted positive and negative exemplar clusters, producing calibrated compliance probability, per-cluster confidences, and an escalation flag.An assessment is auditable when every component traces to a specific exemplar contribution that a regulator could remove or contest.
- Evaluation axes: Assessors are evaluated on accuracy, paraphrase robustness, adversarial robustness, and calibration.Paraphrase robustness preserves accuracy under exemplar-token-free rewrites, while adversarial robustness preserves accuracy under surface-keyword attacks.
- Principle-Bench: 168 scenarios cover COBS 4.5A.3R and PRIN 2A/FG22–5 Consumer Duty across original, paraphrase, adversarial, and boundary splits.The splits contain 100 original, 30 paraphrase, 28 adversarial, and 10 boundary scenarios; two additional paraphrase scenarios were refused by the generator’s safety filter.
- Principle-Bench: The benchmark scenarios were generated from 22 author-prepared summaries of public FCA material with target verdicts and a ban on verbatim exemplar phrases.Per-cluster rubrics were committed and tagged v0-rubric-prereg before any scenario was scored, creating a tamper-evident pre-registration record.
- Ceca: Ceca computes raw cluster confidence from positive-minus-negative mean cosine similarity, averages weighted cluster scores, and applies Platt scaling to a 20-scenario dev split.Averaging makes the score Lipschitz in any single exemplar, while calibration maps narrow raw-score outputs to the regulator-visible probability.
4 Experimental Setup
The experiment compares six judging methods, including keyword, embedding, LLM-based, and cascade approaches, with calibration and cascade settings tuned on a 20-scenario development subset.
- Methods: Six methods are compared: Keyword, MiniLM, BGE, Ollama-mxbai, LLM-judge, and a keyword-primary cascade with LLM fallback.The LLM-judge uses gpt-oss:120b at temperature 0 with seed 42, prompted by the principle text, full exemplar set, and input.
- Tuning: Calibrators and the cascade band are tuned on a 20-scenario development subset of the original scenarios.The cascade invokes the LLM-judge within the band [0.4, 0.6].
5 Results
Results show that no assessor dominates across benign, paraphrase, adversarial, and calibration dimensions: the LLM-judge leads on benign inputs but collapses under keyword stuffing. Cross-family disagreement and cascade failures indicate model-side and architectural vulnerabilities, while Ceca provides auditable counterfactual attribution.
- Primary results: No method dominates: on original COBS, LLM-judge and cascade reach accuracy 0.96 vs. 0.94 and AUC 1.00 vs. 0.99, while embedders sit at chance.On Consumer Duty, accuracy widens to 0.74 for the LLM-judge and 0.67 for the cascade.
- Primary results: 0.04–0.10 deployment-relevant test ECE follows Platt scaling, down from raw 0.22–0.27, showing calibration is necessary.Scaling was fit on a 20-scenario dev split.
- Adversarial robustness: 47 points: the LLM-judge falls from 0.74 to 0.27 on adversarial Consumer Duty while retaining 0.92 on COBS, inverting the benign ranking.Keyword stuffing inserts short factual assertions into substantively non-compliant promotions, which the judge over-credits as evidence.
- Adversarial robustness: 0.67 Consumer Duty adversarial accuracy makes uncalibrated sentence-transformer embedders the most robust because they ignore injected phrases, while keyword scoring fails 100% on COBS adversarial.The result reflects the embedders’ inability to treat inserted phrases as evidence.
- Inter-judge agreement: κ=0.16 on Consumer Duty adversarial contrasts with κ=0.85 on original COBS and 0.86 on paraphrase, localising keyword-stuffing susceptibility to the model rather than the corpus.The comparison uses gpt-oss:120b and the different-family judge kimi-k2.6.
- Auditability and cascade: 10×: removing Ceca’s top-attributed exemplar shifts the score by ten times a random exemplar’s effect and flips 4% [CI 1–8%] of keyword-backend verdicts.The attribution supports regulator-facing counterfactual questions about which exemplar drove a verdict.
- Auditability and cascade: 0%: the cascade scores COBS adversarial inputs because its keyword primary confidently mis-scores stuffed cases outside the deferral band and never consults the judge.Its 89% deferral cascade approaches LLM-judge accuracy on benign and paraphrased inputs, with gaps of 2–14 points, but inherits the primary’s vulnerability.
6 Discussion
The discussion argues that a naive “just use the LLM” deployment is unsafe because raw confidence is poorly calibrated and rationale lacks reproducible per-exemplar auditability. Deployment-grade assessment should therefore combine adversarial-deception and post-hoc calibration reporting with an auditable, contestable attribution layer.
- Why “just use the LLM” is unsafe: Raw ECE of 0.22–0.27 makes unscaled LLM-judge confidences unusable as probabilities for regulators.Platt scaling is required before interpreting verdict confidence probabilistically.
- Why “just use the LLM” is unsafe: Narrative rationale does not provide the ranked, exactly reproducible per-exemplar contributions required for auditability.By the paper’s criterion, the judge remains a strong baseline rather than a deployable assessor.
- Recommendation: Deployment-grade judges should report per-principle adversarial deception rates and post-hoc calibration alongside aggregate accuracy.The verdict should also be paired with an auditable, contestable attribution layer.
7 Conclusion · A Related Work
Principle-based regulation requires LLM judges evaluated beyond average correctness, across calibration, paraphrase robustness, adversarial robustness, and counterfactual auditability. Related work spans principles-based regulation, automated legal compliance, legal NLP, calibration, prototypical explanations, and LLM-as-judge evaluation, while Principle-Bench targets evaluative regulation with purpose-built perturbations.
- 7 Conclusion: Principle-based regulation requires assessors that are calibrated, paraphrase-robust, adversary-robust, and counterfactually auditable.No single method wins across all four axes, and headline accuracy can conceal that the strongest judge is easiest to game.
- A Related Work: Principles carry comparative weight rather than all-or-nothing operation, creating a direct limit for rule-encoding compliance systems.The cited literature links principles-based regulation to cases where rules over- or under-reach.
- A Related Work: Horn clauses, defeasible reasoning, LegalRuleML, and modal-verb extraction represent major strands of automated compliance and legal NLP.Hashmi’s modal-verb approach targets terms such as “must” and “shall,” which principle-based regimes do not use.
- A Related Work: Sentence-BERT, legal-domain BERT, and LegalBench advance legal-text representation and reasoning, whereas this work targets evaluative regulation with purpose-built perturbations.The distinction is the benchmark’s focus on principle-based evaluation rather than general legal reasoning tasks.
- A Related Work: Ceca extends exemplar clusters from prototypical networks and prototype-based explanation to regulatory text.Its contribution is positioned within prior work on prototypes and explanation for structured assessment.
- A Related Work: Modern classifiers are overconfident, post-hoc scaling helps, and LLM judges can approach human agreement while remaining biased.This work further identifies poor calibration and adversarial gameability in regulatory settings.
B Ceca Method Details · C Principle-Bench Construction
Ceca combines exemplar-cluster scoring, calibration, counterfactual attribution, and selective LLM escalation into a modular assessment method. Principle-Bench scenarios are generated from public FCA-informed summaries under a preregistered rubric, with author-assigned labels and disclosed validation and overlap limitations.
- B Ceca Method Details: Ceca is built in four layers on a pluggable embedder 𝜙: String → R𝑑.
- B Ceca Method Details: Averaging exemplar similarities makes 𝜌 robust to a single spurious match, with a single positive-exemplar shift bounded by 𝜖/(4|𝐸+|).The symmetric negative-exemplar bound is 𝜖/(4|𝐸−|), and ˆ𝑠 inherits the bound through a convex combination.
- B Ceca Method Details: Platt calibration fits s= 𝜎(𝑎ˆ𝑠+ 𝑏) on a held-out development split, making 𝜏 an empirical-probability cut-off.Isotonic regression is also reported as a robustness check.
- B Ceca Method Details: 4% [95% CI 1–8%] of inputs flip verdicts after removing the top-attributed exemplar, which shifts scores by 10× the mean random-exemplar shift.Ceca’s closed-form counterfactual exactly matches rerunning the assessor with the exemplar removed.
- B Ceca Method Details: The cascade defers to an LLM-judge when primary confidence falls within a development-tuned band [ℓ,ℎ] and otherwise emits the primary verdict.It provides per-exemplar attribution for the high-confidence majority, LLM rationale for the low-confidence minority, and bounds judge invocations.
- C Principle-Bench Construction: Each scenario is anchored on one of 22 author-prepared summaries informed by public FCA material and generated with gpt-oss:120b under channel, product, and audience constraints.The model is forbidden from using exemplar phrases verbatim, and the target verdict is the ground truth.
- C Principle-Bench Construction: Before authoring or labelling, researchers preregistered two principle-specific rubrics with per-cluster 1–5 Likert criteria and an escalation rule.The tagged rubric commit v0-rubric-prereg predates every label commit, providing a tamper-evident record that criteria were fixed before scoring.
- C Principle-Bench Construction: Labels are author-assigned via the rubric without external annotator validation, while scenarios are anchored on public FCA summaries to constrain verdicts by external regulatory practice.A regulator panel for inter-annotator agreement is left to successor work; corpus/judge overlap is partly mitigated by a different-family second judge, kimi-k2.6.
D Full Experimental Setup
The experiments compare keyword, embedding, LLM-judge, and cascade methods, with calibration and cascade tuning performed on a 20-scenario development subset. Evaluation covers original and perturbation splits using classification, ranking, compliance, and calibration metrics with bootstrap confidence intervals and paired significance testing.
- Methods: The benchmark evaluates a keyword bag-of-clusters embedder, MiniLM, BGE, Ollama-mxbai, an LLM-judge, and a keyword-primary cascade.The LLM-judge is gpt-oss:120b via Ollama Cloud at temperature 0 and seed 42; the cascade falls back to the judge when s∈[0.4, 0.6].
- Methods: MiniLM scores ranged only [0.476, 0.522], whereas the keyword embedder ranged [0.39, 0.66], motivating keyword-primary cascade tuning.A 20-sample Platt fit on MiniLM was degenerate, with slope a≈−773; the keyword embedder’s wider range supported stable band tuning.
- Procedure and metrics: Calibrators and the cascade band were tuned on 20 original scenarios, while metrics used the remaining 80 original scenarios plus all perturbation splits.For 7–16% of original scenarios, malformed judge JSON was excluded from judge cells; metrics included accuracy, precision, recall, F1, per-cluster AUC, false-compliance rate, ECE, and Brier score.
- Procedure and metrics: All point estimates used bootstrap 95% CIs from 104 resamples, and paired comparisons used McNemar’s exact test.The reported metric set included accuracy, precision, recall, F1, per-cluster AUC, false-compliance rate, ECE, and Brier score.
E Full Results · F Ethical Considerations
The full results show that keyword and LLM-judge methods provide the strongest per-cluster signal, while counterfactual faithfulness and cascade deployment reveal important calibration and cost trade-offs. Ethical considerations emphasize author-labelled evaluation, reproducibility limits, and the dual-use risk of optimizing promotions against the metric.
- E Full Results: Keyword and LLM-judge methods carry the strongest per-cluster signal, while sentence-transformer embedders are near-random or inverted.Several Ollama-mxbai clusters score 0.03–0.09, below the 0.5 threshold.
- E Full Results: 0.018 and 0.0045 mean score changes follow removal of the top-|Δ| exemplar for keyword and MiniLM, respectively.These are 10× and 2.3× the random-exemplar control.
- E Full Results: 4% [CI 1–8%] and 0% of verdicts flip for keyword and MiniLM, respectively, with only keyword moving scores enough to cross a binary decision.Attribution ranks influence correctly for both backends, but MiniLM’s compressed range limits score movement.
- E Full Results: 89% of inputs are deferred by the cascade’s dev-tuned band [0.40, 0.60].The cascade’s ECE exceeds the LLM-judge ECE because poorly calibrated keyword outputs are mixed with well-calibrated judge verdicts.
- E Full Results: 17.6s is the expected independently scheduled latency, approximately 11% below the judge’s 19.7s, despite no systematic same-environment wall-clock reduction.The bound is 0.89 · 19.7 + 0.01 ≈ 17.6s; deployment savings primarily concern judge-call count.
- F Ethical Considerations: Author-assigned labels follow a pre-registered rubric and corpus-freeze tags, but inter-annotator agreement is not reported because no external annotator was used.The paper maps a path toward a regulator-validated successor.
- F Ethical Considerations: A principle-assessment tool has dual-use potential: regulators may triage compliance, while promoters may optimize against the metric without changing practice.The adversarial experiment is described as directly quantifying this risk.
G Reproducibility
The authors release the code, prompts, raw responses, calibrators, benchmark, and pre-registration tags, while documenting sweep caching and bootstrap procedures. Supporting diagnostics report paraphrase gaps, ROC curves, escalation behavior, and Ceca exemplar attributions.
- Reproducibility: The release includes method code, prompts, raw LLM responses, fitted calibrators, the full benchmark, and pre-registration tags establishing that the rubric pre-dates labelling.The full sweep covers 6 methods × 4 splits × ≤2 principles, caches one embedding per (model, exemplar) pair, and uses 104 bootstrap resamples.