PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Accuracy: A Dual-Judge Evaluation Protocol for Vision-Language Models in Legally Grounded Tasks

Su Myat Noe, Ha Thanh Nguyen, May Myo Zin, Ken Satoh

arXiv:2608.24258v1cs.AI

TL;DR

Legally accountable AI evaluations need to assess not only response quality but also equivalence to an applicable reference. This paper introduces a dual-judge protocol for UK traffic-sign interpretation and finds that high quality scores are least trustworthy under heavy occlusion.

  • Problem

    Existing legal-AI benchmarks and single quality judges provide limited evidence about whether responses are semantically equivalent to a fixed, human-curated legal reference.

  • Method

    The protocol pairs a standard 0–10 quality judge with a binary semantic-equivalence judge and evaluates UK traffic-sign responses across visibility and occlusion conditions.

  • Results

    8.0% of evaluations showed the asymmetric disagreement pattern, rising to 54–63% among high-scoring answers under heavy occlusion at v ≤ 0.3.

  • Takeaways & Limitations

    One additional LLM call surfaces a visibility-dependent disagreement signal that single-judge protocols do not report.

  • Takeaways & Limitations

    The equivalence judge’s legal grounding is unestablished because the reference is an author paraphrase rather than statutory text and no legally trained annotator participated.

Abstract

from arXiv · show

AI systems are increasingly evaluated for legally accountable settings, where correct outputs must also be justifiable against an applicable legal standard. Existing legal-AI benchmarks and LLM-as-judge protocols provide important infrastructure for measuring task performance and open-ended response quality. We contribute one additional evaluation signal: a dual-judge protocol that pairs a standard 0-10 quality judge with a strict binary semantic-equivalence judge against a human-curated reference. We study a controlled, visually grounded regulatory task - UK traffic-sign interpretation, whose meaning is a codified question with a known reference for every input - and measure not merely whether the two judges disagree (by construction they must) but how much and where. On 4,680 evaluations under seven visibility levels and two occlusion modes, the two judges are moderately associated (point-biserial r = 0.644), while revealing an asymmetric Type II pattern affecting 8.0% of all evaluations. Its distribution is instructive: the marginal rate peaks at high visibility (14.2% at v = 0.8) simply because high-scoring answers are common there, but conditioned on the answer already scoring above 7, the rate is highest under heavy occlusion (54-63% at v <= 0.3), so a high quality score is least trustworthy when the input is most degraded. We are explicit that the signal is a property of this judge and reference: a 49-row human check shows the 0-10 judge aligns closely with everyday-reader judgement (Pearson r = 0.81; r = 0.80 with the LLM accuracy sub-score), while the equivalence judge is fairly but one-directionally stricter. The protocol adds one LLM call per evaluation and surfaces a signal single-judge protocols do not report. We release the prompt template, occluded variants, and full evaluation results.

1. Introduction

The paper proposes a dual-judge protocol that supplements standard quality scoring with strict semantic-equivalence checking against human-curated references. It demonstrates the protocol on a controlled UK traffic-sign interpretation benchmark and characterizes its disagreement signal.

  • Motivation and gap: The study addresses a gap in legal-AI evaluation, where benchmarks and LLM-as-judge methods typically report a single accuracy-style or quality score.The strict question of whether a response matches the reference is not part of the default protocol.
  • Testbed and scope: 4,680 evaluations span 30 UK traffic signs, 7 visibility levels, 2 occlusion modes, and 4 VLM-based systems in a controlled regulatory benchmark.The benchmark uses traffic-sign interpretation under the UK Road Traffic Act and TSRGD, with the applicable rule known for every input.
  • Contribution: The dual-judge protocol pairs a standard 0–10 LLM-as-judge with a strict binary semantic-equivalence judge against human-curated gold descriptions.The protocol adds one LLM call per evaluation and releases the equivalence-judge prompt template.
  • Validation: 49-row human evaluation found Pearson r = 0.81 with the 0–10 judge and r = 0.80 with the LLM accuracy sub-score.The equivalence judge applies a stricter, judge-dependent reference-match criterion.
  • Empirical signal: 8.0% Type II disagreement was characterized by size, visibility distribution conditioned on the base rate, and judge-dependence.The paper presents this signal as an addition to existing AI-for-Law benchmark design rather than a replacement for current infrastructure.

2. Related Work

Prior work spans collaborative LLM reasoning, traffic-sign recognition, LLM-as-judge evaluation, and legal-AI benchmarking. This paper extends these lines by evaluating semantically granular traffic-sign interpretation under controlled occlusion and auditing judge bias.

  • Multi-agent and chain-of-thought reasoning: Multi-agent frameworks and chain-of-thought prompting seek to improve reasoning beyond single-agent prompting, but gains are mixed when integration remains the bottleneck.Reported benefits are most consistent when sub-task decomposition is genuinely orthogonal.
  • Traffic-sign recognition: Traffic-sign research has progressed from classical detection pipelines toward VLMs interpreting signs at semantic rather than object-detection granularity.Existing evaluations commonly use clean, fully visible benchmarks and often assume contextual applicability.
  • Traffic-sign recognition: Controlled occlusion extends prior VLM traffic-sign evaluations, which have studied occlusion mainly through standard object-detection metrics.The extension targets degraded visibility and contextual applicability in VLM-based systems.
  • LLM-as-judge evaluation: LLM-as-judge protocols are widely used but remain vulnerable to position, verbosity, self-enhancement, and label-matching biases.The methodology audit identifies a label-matching bias against verbose systems and proposes a structured-output fix.
  • Legal-AI benchmarks and evaluation: LexGLUE and LegalBench standardize evaluation across broad legal-language and legal-reasoning tasks, establishing infrastructure for legal-AI assessment.LexGLUE covers seven sub-tasks, while LegalBench comprises 162 tasks spanning six types of legal reasoning.

3. Methodology Audit and Metric Correction

An audit of the initial single-judge pipeline found systematic inconsistencies between descriptive quality scores and binary correctness flags, including contradictory system rankings.

  • Audit findings: 4.8% of evaluations scored overall ≥ 7 while receiving correctly identified = 0, concentrated in longer-output systems such as Multi-Agent and Chain-of-Inference.The audit identified this mismatch in the initial single LLM-as-judge pipeline.
  • Audit findings: Judge-score ranking produced CoT > CoI > SA > MA, whereas binary-flag ranking produced SA > CoI > CoT > MA.The two metrics therefore yielded different orderings of the evaluated systems.
  • Audit findings: The most thorough reasoner ranked third by the judge score, illustrating the practical consequence of metric disagreement.This conclusion follows from the reported ranking discrepancy and the passage’s characterization of the most thorough reasoner.

4. Experiment: Recognition Under Occlusion

The experiment evaluates four Azure GPT-4o-based systems on 30 UK traffic signs across randomized and bottom-up occlusion, using 390 variants and 4,680 evaluations against human-curated references. CoT performs best on the 0–10 quality judge, while judge disagreement reveals that high quality scores become least trustworthy under heavy occlusion.

  • System comparison: +0.39 paired mean difference favors Single Agent CoT over the Single Agent baseline (p < 10−5), while Multi-Agent scores −1.53 lower (p < 10−29) and CoI scores −0.21 lower (p = 0.032).The two large effects remain under the n = 390 paired analysis, although the variant cells are not fully independent across a sign.
  • Dual-judge evaluation: 0.644 is the point-biserial association between the 0–10 quality judge and binary equivalence judge.Mean overall scores are 8.44 when responses are marked equivalent and 3.62 when they are rejected.
  • Occlusion effects: 54–63% is the conditional Type II rate under heavy occlusion, compared with 22–32% at high visibility among answers scoring above 7.The marginal Type II share peaks at 14.2% for v = 0.8 because high-scoring answers are more common there; conditioning reverses this pattern.
  • Human validation: 0.81 is the Pearson correlation between human judgement and the aggregated 0–10 judge across 49 checked responses.The equivalence judge is stricter in one direction, rejecting 17 responses humans marked as matches while never accepting a human non-match.
  • Human validation: The equivalence judge rejects generally correct responses when they omit specific reference meaning, such as the requirement to be immediately stoppable.This illustrates why everyday readers may treat a response as equivalent while a legal auditor identifies a substantive omission.

5. Discussion

The discussion presents the dual-judge protocol as a complementary audit signal for open-ended, legally grounded responses, while limiting empirical claims to a closed, visually grounded verbalisation task. It also identifies shared-model bias, narrow data and model coverage, and the need for independent human or model corroboration.

  • What the dual-judge protocol contributes: r = 0.644, 91.4% of rows fell outside disagreement quadrants, with 8.0% Type II and 0.6% Type I disagreements.The authors interpret the asymmetric disagreement as the protocol’s informative signal: two judges asking different questions usually provide compatible answers, but not symmetrically.
  • Verbalisation, not interpretation: The protocol evaluates rule verbalisation against a closed, codified reference, not the open-textured legal question of whether a rule applies to facts.The authors therefore do not claim that it evaluates legal reasoning broadly.
  • The Type II signal interpreted for legal AI: The 0–10 judge assesses fluency and informativeness, whereas the equivalence judge applies a strict wording-match criterion relevant to auditability against an applicable reference.The data indicate that the equivalence judge is stricter than the everyday reader and rejects in only one direction rather than disagreeing symmetrically.
  • How this complements LexGLUE and LegalBench: The dual-judge protocol complements LexGLUE and LegalBench by evaluating open-ended responses when a gold reference exists but exact label-match is not the natural protocol, adding one semantic-equivalence call per evaluation.The structured-output fix independently removes a parsing artefact in which fuzzy matching can over-credit terse descriptions whose surface form matches ground-truth labels.
  • Scope of generalisation: The mechanism is modality-independent, but the empirical findings are scoped to the visual testbed’s occlusion-based difficulty gradient and closed, codified references.The authors state that transfer to text is technically straightforward, while textual law lacks a clean analogue for occlusion and often has contested equivalence criteria.
  • Limitations: Both judges and all four evaluated systems are GPT-4o, while the dataset contains 30 base signs and the human study is a single-annotator sanity check.The authors call for corroboration by a different model family and future extension to GPT-5, open VLMs, and multi-annotator evaluation with legally trained annotators.

6. Conclusion

The paper presents a dual-judge protocol for visually grounded legal-regulatory tasks, pairing a 0–10 quality judge with a strict binary semantic-equivalence judge for one additional LLM call per evaluation. Across 4,680 traffic-sign evaluations, the protocol surfaces visibility-dependent disagreement that single-judge protocols do not report, while human validation supports the quality judge’s everyday-reader alignment and the equivalence judge’s one-directional strictness.

  • Protocol: The protocol pairs a standard 0–10 LLM-as-judge with a strict binary semantic-equivalence judge against human-curated gold descriptions.It adds one additional LLM call per evaluation.
  • Findings: 14.2% was the marginal Type II rate at v = 0.8, while conditioning on scores above 7 shifted the highest rates to heavy occlusion at low visibility.The marginal peak reflects the greater prevalence of high-scoring answers at high visibility, whereas conditional rates reached 54–63% at v <= 0.3.
  • Validation: Pearson r = 0.81 showed that the 0–10 judge aligned with everyday-reader judgement, while the equivalence judge was fairly but one-directionally stricter.The human-evaluation validation contained 49 rows.
  • Practical contribution: A single extra LLM call per response surfaces a visibility-dependent disagreement signal that single-judge protocols, by construction, do not report.The protocol is presented as a low-cost complement to, rather than a replacement for, existing legal benchmarks.

Impact Statement

The paper advocates reporting strict semantic equivalence alongside quality scores for legally accountable AI evaluation, while cautioning that its equivalence judge is only fairly validated. It reports no specific ethical concerns but rejects treating the judge as an authoritative legal standard without independent validation.

  • The paper aims to advance evaluation of AI systems intended for legally accountable deployment.
  • Benchmarks should report a strict equivalence signal alongside a quality score because the two can dissociate in practice.
  • The equivalence judge is only fairly validated and should not be treated as an authoritative legal standard without further independent validation.
  • The paper identifies no specific ethical concerns.
Loading 2608.24258v1…