Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion

Surya Saka

Source-linked AI summary

Calibrated Trust, Not Sharper Prediction: An Empirical Test of Uncertainty Fusion

Surya Saka

arXiv:2608.14617v1Jul 11, 2026cs.LGcs.AIcs.CL

TL;DR

Legal AI has proposed combining multiple uncertainty tools to produce superior legal-outcome predictors, but errors in legal work can create professional liability. This paper tests that proposal on ECtHR cases and finds no sharper discrimination, while a recalibrated, risk-controlled engine provides calibrated selective automation.

Problem
The paper examines whether combining multiple uncertainty instruments produces a superior legal-outcome prediction engine, a consequential question because errors in legal work can create professional liability.
Method
It tests the fusion proposal on 1,000 ECtHR cases using two frontier LLMs and a strong baseline, then evaluates recalibrated selective automation with risk control.
Results
The proposal does not improve discrimination and includes an unsafe component, but after removal and recalibration the engine auto-clears cases at 96.8% accuracy, with 0.5% of errors escaping and 96.3% caught.
Takeaways & Limitations
The contribution is calibrated trust: an auditable, risk-controlled way to decide which cases the system should resolve and which it should escalate.
Takeaways & Limitations
Evidence is limited to binary violation prediction at the ECtHR, so selective-prediction conclusions should be retested on contract-obligation and patent tasks.

Abstract

from arXiv · show

A recurring proposal in legal AI is to improve case-outcome prediction by fusing uncertainty tools (evidence graphs with belief propagation, sequential Bayesian odds updating, Dempster-Shafer combination, and conformal prediction) into one pipeline. We test this on 1,000 real European Court of Human Rights cases from LexGLUE and FairLex, predicting whether the Court found a Convention violation from the case's fact paragraphs. We compare three families across two frontier LLMs (Claude Opus 4.8 and GPT-5.5) as per-fact evidence estimators: (A) the raw LLM, (B) the LLM routed through the fusion pipeline, and (C) a term-frequency baseline through the same pipeline. Across roughly 4,750 tests we find: (1) on discrimination (AUROC around 0.83) the pipeline yields no improvement over either the raw LLM or the baseline; a frontier LLM used directly is the strongest single discriminator. (2) Naively composing an LLM with Bayesian-odds and Dempster-Shafer fusion more than doubles calibration error (ECE from about 0.16 to 0.46) via a prior-mismatch mechanism that replicates across both models. (3) Dempster-Shafer fusion is actively unsafe on long chains, committing confidently to wrong labels at below-chance accuracy; we recommend removing it. (4) The pipeline's genuine value is operational: routed through a conformal selective-prediction layer, the system decides which cases to automate and which to escalate. After removing Dempster-Shafer, recalibrating, and applying class-conditional risk control on the full 1,000-case set, the tuned engine auto-clears at 96.8 percent accuracy with 0.5 percent errors escaping and 96.3 percent caught for review, versus 85.9 / 3.8 / 72.1 for an untuned baseline. The contribution of such pipelines in law is calibrated trust, not sharper prediction.

1 Introduction · 2 Related work

The paper tests a rarely evaluated fusion proposal for legal outcome prediction, separating the contributions of frontier LLMs, uncertainty mathematics, and conformal selective prediction. Its central deployment focus is calibrated automation: safely auto-clearing easy cases while escalating uncertain ones.

1 Introduction: The paper prioritizes safe selective automation because confidently wrong legal dispositions can create professional liability, while escalating every case defeats the tool’s purpose.The deployment objective is to reduce human review burden by automating genuinely easy cases and escalating cases requiring lawyers.
1 Introduction: The study evaluates uncertainty fusion on 1,000 real ECtHR cases using two frontier LLMs, a term-frequency control, and approximately 4,750 controlled tests.The evaluation separates the value of the LLM, fusion mathematics, and conformal selective prediction.
1 Introduction: Dempster–Shafer fusion is reported as actively harmful on long legal evidence chains, producing below-chance accuracy on cases to which it commits most confidently.The paper recommends treating this as a documented negative result rather than merely a lack of improvement.
1 Introduction: Naively combining an LLM with Bayesian-odds fusion destroys calibration through prior mismatch, although single-parameter recalibration restores it.Bayesian odds updating relies on a base rate and sequential likelihood ratios, while its conditional-independence assumption is a known weakness.
1 Introduction: The numerical fusion instruments do not understand legal text; understanding and numerical encoding occur separately through either an LLM or a term-frequency statistic.The LLM contributes imperfect comprehension, whereas term frequency contributes only word-occurrence correlations learned from training data.
2 Related work: The adopted judgment-prediction framing comes from ECtHR research and is instantiated through the LexGLUE and FairLex benchmarks.The underlying task includes binary violation prediction alongside related article and case-importance tasks in prior work.
2 Related work: Conformal prediction differs from evidence-fusion methods by wrapping scores with distribution-free, finite-sample coverage guarantees under exchangeability.Conformal Risk Control and related frameworks provide the basis for certifying an auto-clear gate at a chosen error level.
2 Related work: Selective prediction formalizes abstention through risk–coverage evaluation, while learning-to-defer research addresses routing rejected cases under practical costs and capacity constraints.Calibration is necessary because confidence thresholds are meaningful only when reported probabilities are calibrated; prior work also studies selective prediction for legal cases and LLMs.

3 Data and task

The study uses real European Court of Human Rights cases from LexGLUE and FairLex, represented as ordered fact paragraphs for binary outcome prediction. Severe class imbalance motivates balanced sampling for LLM-scored comparisons.

Datasets: The data combine LexGLUE’s ECtHR Task A with FairLex’s ECtHR configuration, using real judgments released through HUDOC.LexGLUE provides 9,000 training, 1,000 validation, and 1,000 test cases; FairLex adds applicant gender, age, and respondent-state attributes.
Task: Each case is represented as an ordered list of fact paragraphs, with each paragraph treated as one evidence item for predicting the binary outcome from facts alone.Cases contain roughly 20–25 fact paragraphs on average.
Class balance: 89.8% of the full 1,000-case test set are violation cases, leaving about 10% no-violation cases and making separation of the minority the central discriminative challenge.The study therefore treats accuracy and calibration metrics as difficult to interpret under the severe imbalance.
Class balance: 200 balanced test cases—100 violation and 100 no-violation—form the stratified sample for all LLM-score arm comparisons.The full test set contains 1,000 cases, but LLM-scored comparisons use the balanced sample.

4 Methods

The study evaluates a two-stage architecture that converts fact paragraphs into violation scores and aggregates them into case decisions, optionally with conformal prediction. Its methods vary the reader, aggregator, experimental arm, and evaluation criteria to isolate the effects of fusion.

System architecture: Every system first maps fact paragraphs to violation-leaning scores in [0, 1], then aggregates those scores into a case-level decision, optionally using conformal prediction.The architecture separates evidence reading from case-level aggregation.
Readers: Readers comprise TF–IDF logistic regression, Claude Opus 4.8, and GPT-5.5, with LLMs producing holistic case probabilities and per-paragraph scores.The TF–IDF reader uses 1–2 grams, a 20,000-term vocabulary, sublinear term frequency, and L2-regularised logistic regression.
Aggregators: Aggregators include mean and maximum baselines, Bayesian odds, damped belief propagation, and sequential Dempster–Shafer combination.Bayesian odds use per-fact likelihood ratios; belief propagation discounts correlated evidence; Dempster–Shafer combines mass functions with conflict normalisation.
Selective prediction: Split conformal prediction converts case scores into prediction sets: singleton sets proceed as committed predictions, while two-element sets signal review.The method uses class-conditional nonconformity scores and targets coverage 1 − α.
Evaluation: Evaluation covers AUROC, Brier score, ECE, risk–coverage, AURC, SAC, and triage outcomes including auto-clear, proceeded-subset accuracy, caught errors, and escaped errors.The study uses 200-resample bootstrap 95% confidence intervals for AUROC and ten equal-width bins for ECE.
Experimental arms: Arms A, B, and C compare raw LLM probabilities, LLM per-fact scores through fusion, and TF–IDF scores through the identical pipeline.Controls B2 and C2 use simple mean aggregation to isolate fusion-mathematics effects on identical reader evidence.

5 Results: discrimination, calibration, coverage

Fusion did not sharpen discrimination: the raw frontier LLM was strongest, while routing it through fusion reduced AUROC. The main value was calibrated trust, but prior mismatch harmed calibration and Dempster–Shafer produced dangerously confident errors.

Discrimination: GPT-5.5 raw achieved AUROC 0.851, versus 0.783 after fusion and 0.831 for the term-frequency baseline through the identical pipeline.Every confidence interval overlapped the term-frequency control, and no arm dominated.
Calibration: ECE rose to 0.457 for Opus and 0.463 for GPT-5.5 when Bayesian-odds and Dempster–Shafer aggregation composed with the LLM.The degradation was attributed to reinjecting the 89.8% training prior into scores from a balanced sample, creating prior mismatch.
Calibration: ECE fell from 0.158 for the raw Opus LLM after single-parameter recalibration of the fused score.The recalibration removed essentially all of the confidence–accuracy gap and was adopted in the tuned engine.
Coverage: Coverage met or exceeded every target 1 − α∈{0.80, 0.90, 0.95} across arms and models, including 0.913 against 0.90 for Opus and 0.921 for GPT-5.5.All 25 term-frequency method-by-α cells also satisfied coverage.
Aggregator safety: Dempster–Shafer collapsed to AUROC 0.541, and its committed singleton predictions fell below chance, being wrong on roughly three-quarters of cases.Belief propagation achieved AUROC 0.820 and Bayesian odds 0.817, versus 0.834 for the mean baseline; Dempster–Shafer was removed from subsequent analysis.

6 Results: selective prediction and human-in-the-loop triage

The conformal layer adds value by reframing prediction as triage: cases proceed to automation or escalate to human review. Its benefit appears in selective operation, where calibrated combination scores improve coverage at accuracy floors and fusion mathematics reduce committed-case error.

Triage framework: The conformal layer routes each case either to Proceed for automation or to Review for human escalation.Triage requires automated cases to be accurate while uncertain cases are escalated.
Selective prediction: 0.42 coverage at a 95% accuracy floor and 0.575 at 90% show meaningful selective performance for the calibrated term-frequency-plus-combination score.The raw-LLM SAC at a 95% accuracy floor is near zero, indicating the combination’s value lies in selective operation rather than ranking.
Human-in-the-loop triage: About 87% of would-be errors are caught for review with the combination, versus about 68% for the raw LLM.This comparison uses the 90% confidence setting (𝛼= 0.10) with Dempster–Shafer removed.
Human-in-the-loop triage: About 4.7% of wrong decisions escape to auto-decision with the combination, versus about 7.3% for the raw LLM.The pattern is near-identical across Opus and GPT-5.5, with the combination being more cautious about coverage.
Fusion contribution: 0.31–0.35 abstention error reduction for fusion mathematics is roughly double the 0.11–0.17 achieved by B2 and the raw LLM.The curves for Opus and GPT-5.5 are nearly coincident across confidence settings.

7 A tuned, risk-controlled triage engine

The tuned triage engine prioritizes calibrated scores, class-conditional thresholds, and explicit risk control rather than additional fusion. With sufficient calibration data, it substantially improves selective automation, although minority-class guarantees remain weak.

Design: Recalibration, class-conditional thresholds, and a Clopper–Pearson accuracy floor maximize auto-clearing subject to class-conditional error bounds.The design repairs prior mismatch and uses separate thresholds for the rare, hard no-violation class.
Score construction: λ = 1/K damping restores AUROC from 0.68 to ≈0.81, while the simple mean achieves AUROC 0.830 without saturation pathology.The undamped product drives 96% of cases to posterior 0.000 after roughly 25 likelihood ratios; the mean is an equally defensible substitute.
Selective performance: 96.8% accuracy, 0.5% errors escaping, and 96.3% errors caught characterize tuned-engine auto-clearing at a 95% target, versus 85.9% / 3.8% / 72.1% for baseline.The tuned engine auto-clears 14% of cases versus 29% for the baseline, reflecting the cost of the certified floor.
Out-of-sample control: 0.015 majority-class floor-violation frequency confirms the guarantee out of sample once calibration data are sufficiently large.This remains comfortably below the nominal δ = 0.05.
Limitations: 0.46 minority-class floor-violation frequency persists at n = 1000, requiring a larger calibration corpus or less stringent minority target.On the balanced 200-case sample, about 25 minority cases per split still admitted a 7% upper bound despite zero calibration errors.

8 Discussion

The fusion pipeline does not sharpen case-outcome prediction: discrimination remains flat across frontier models and a strong baseline, with the strongest single system being a frontier LLM used directly. Its practical contribution is calibrated trust through selective, auditable automation, provided unsafe fusion components are removed and review is governed carefully.

Calibrated trust, not sharper prediction: AUROC ≈0.83 remains flat across two frontier models and a strong baseline, while a frontier LLM used directly is the most accurate single system.The fusion proposal promises better prediction but does not deliver it.
Calibrated trust, not sharper prediction: Removing Dempster–Shafer, recalibrating scores, and applying risk control yields calibrated trust: auditable, guarantee-backed separation of cases for automation versus escalation.The pipeline’s value is operational rather than improved average discrimination.
Deployment architecture: A cascade can contain costs by using a cheap reader and calibrated combination for every case, escalating only uncertain cases to the expensive LLM.The expensive model is deployed where it can change the decision, making its use economically defensible.
Human review: Human review is not automatically safer: its value depends on presentation and reviewers’ private information, although randomized evidence supports conformal sets over fixed-size sets.The review interface should therefore surface calibrated confidence and relevant information.
Operating point: Auto-clearing is preferred when calibrated uncertainty makes its expected error cost no greater than the cost of human review.Formally, auto-clearing is preferred when min(s, 1 − s) ≤ c_rev/c_err, equivalently when arg-max confidence exceeds 1 − c_rev/c_err.
Implications for legal practice: Procurement should favor defensible claims about calibrated selective automation and auditable error floors over unsupported claims of superior outcome prediction.The accuracy floor also functions as a governance instrument.
Fusion mathematics: Bayesian odds and belief propagation do not add information; belief propagation can be calibration-neutral after damping, whereas Bayesian odds can harm calibration through prior mismatch.Dempster–Shafer is presented as a distinct combination rule but unsafe on long, mutually reinforcing legal evidence chains.

9 Limitations and threats to validity

The study’s validity is limited by its single jurisdiction and task, noisy term-frequency labels, small LLM comparison sample, in-sample curve calibration, possible LLM familiarity with ECtHR doctrine, and incomplete minority-class certification. These constraints temper generalization while leaving some with-/without-fusion contrasts and the full-set tuned-engine validation informative.

Generalizability: Single-jurisdiction evidence covers binary ECtHR violation prediction, so selective-prediction conclusions require retesting on contract-obligation and patent tasks.Cascade and calibration findings are considered more portable than the specific AUROC values.
Label quality: Case-level labels inherited by every paragraph make the term-frequency reader’s per-fact training labels noisy and handicap arm C in absolute terms.The handicap remains constant across with-/without-fusion contrasts supporting the conclusions.
Sample size: 200 cases underpin the LLM-scored comparisons, producing AUROC confidence intervals of approximately ±0.05, although full-set tuned-engine validation is not similarly limited.The limitation concerns the balanced-sample LLM arms rather than the 1,000-case validation.
Calibration: In-sample isotonic recalibration shapes the risk–coverage, reliability, and SAC curves, while the rigorous out-of-sample guarantee comes separately from Table 5’s calibration/test split.The figures therefore should not be treated as the source of the formal guarantee.
LLM exposure: Residual pre-training familiarity with ECtHR jurisprudence cannot be excluded despite facts-only instructions and withheld labels, potentially inflating the raw-LLM arm.If present, this would strengthen the finding that fusion mathematics adds no discrimination over the raw LLM.
Minority-class certification: n=1000 still does not certify a tight floor for the rare no-violation class, so deployments requiring that guarantee need a larger calibration corpus.This limitation specifically concerns minority-class certification.

10 Conclusion

Across 1,000 real ECtHR cases and roughly 4,750 controlled tests, the proposed uncertainty-fusion pipeline failed to improve legal-outcome prediction on its stated terms. Its components introduced safety and calibration failures despite evaluation with two frontier LLMs and a strong baseline.

The proposal was tested on 1,000 real ECtHR cases using two frontier LLMs, a strong baseline, and roughly 4,750 controlled tests.
The fusion pipeline did not improve discrimination over the tested alternatives.
One pipeline component was actively unsafe.
Naively composing the pipeline with an LLM destroyed calibration.

Ethics statement

The study uses only publicly released, anonymised ECtHR judgments and frames its decision-support system for human-reviewed use, cautioning against unsupervised deployment and assuming the review path is risk-free.

The study uses publicly released, anonymised ECtHR judgments from HUDOC through LexGLUE and FairLex, collecting no private or personally identifying data.
The decision-support system is intended to operate with a human reviewer in the loop, not through unsupervised deployment.
The paper cautions that the human-review path should not be assumed risk-free.

Data and reproducibility

The study uses public ECtHR datasets and a Python implementation whose procedures and figures are organized for reproducible tracing from reported numbers to saved result artefacts.

Data: The datasets are public LexGLUE ECtHR Task A and FairLex ECtHR, both sourced from the ECtHR HUDOC database.
Implementation: Aggregators, conformal and risk-control procedures, and calibration methods are implemented in Python.
Reproducibility: Figure-generation code uses NumPy, scikit-learn, NetworkX, and Matplotlib, with each reported number traceable to a saved result artefact.

A Detailed experimental protocol · B Full coverage and abstention tables · C Selective-prediction summary

The study specifies readers, aggregation rules, conformal risk control, and resampling procedures, then evaluates coverage, abstention error reduction, AURC, and selective accuracy. Coverage meets all targets, while combination methods show their main value in selective prediction rather than ranking.

A Detailed experimental protocol: The term-frequency reader used TF–IDF 1–2 grams and L2-regularised logistic regression, while Claude Opus 4.8 and GPT-5.5 scored 200 balanced cases.Training used 9,000 LexGLUE cases and a 20,000-term vocabulary with sublinear term frequency and minimum document frequency 3.
A Detailed experimental protocol: The protocol set the training positive base rate to b = 0.898 and specified damping rules for belief propagation and the chain-aware tuned engine.The internal damping was λ = 1/(1 + 0.15 · ln(1 + K)); the chain-aware variant used λ = 1/K.
A Detailed experimental protocol: Dempster–Shafer assigned uncertainty mass 0.25 per fact, used the pignistic transform, and was excluded after Section 5.4.Its reporting was limited to Section 5.4.
A Detailed experimental protocol: The tuned engine maximised auto-clearance subject to a class-conditional Clopper–Pearson error bound with confidence 1 − δ and δ = 0.05.Validation used 200 random 500/500 calibration/test partitions of the full 1,000-case set.
A Detailed experimental protocol: AUROC intervals used 200 bootstrap resamples, while triage and tuned-engine statistics averaged over 30–200 splits and abstention reduction over 30 seeds.The number of random splits varied as indicated by the analysis.
B Full coverage and abstention tables: Coverage met or exceeded every target, and fusion arm B roughly doubled abstention error reduction versus the identical-evidence no-fusion control B2.Targets were 1 − α ∈ {0.95, 0.90, 0.80}; control C2 also showed a large reduction attributed to poorer base calibration.
C Selective-prediction summary: For the Opus reader, calibrated term-frequency-plus-combination achieved the lowest AURC and highest SAC, indicating value concentrated in selective prediction rather than ranking.AURC is lower-is-better, whereas SAC is higher-is-better, evaluated at three accuracy floors.