PaperlayerView exact revision on arXiv

Source-linked AI summary

Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority

M. Mikail Demir

arXiv:2610.29961v1cs.AIcs.CLcs.CY

TL;DR

Legal citation errors include nonexistent authorities and real cases cited for unsupported propositions, burdening courts. This study tests decision models against generative LLMs on CaseHOLD with missing-authority items. Generative LLMs often still choose a holding, while the most cautious decision model does so in fewer than one in three cases; pairing models halves wrong citations.

  • Problem

    Legal filings cite nonexistent authorities or real authorities that do not support the cited proposition, requiring citation checks and adding work for courts.

    Source: §1 · pp. 1–2

  • Method

    The study adds a missing-authority condition to CaseHOLD and compares four System One decision models with six generative LLMs.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    Most generative LLMs still pick a holding when none applies, while the most cautious decision model does so in fewer than one in three cases; pairing models halves wrong citations.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Calibrated probabilities provide a user-controlled caution dial rather than correctness, with honesty beginning when a research tool says it cannot find the authority.

    Source: §7 · p. 12

    The abstain condition tests one constructed form of absence: replacing the correct holding with a real holding from another case, not other forms of missing authority.

    Source: §6 · p. 11

Abstract

from arXiv · show

Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law. System One decision models are built on large language models but, instead of generating text, return a calibrated probability for each option. This suits the verification of legal authority, where the question is closed: a given holding, or none of them. We test whether such models abstain when the correct authority is missing. We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%. On standard CaseHOLD, a decision model matches the best generative LLM (88.1% vs. 88.2%) at 1/55 of the cost. Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model. Legal AI should be judged by its wrong citations, not by accuracy alone.

1 Introduction

Legal citation errors include both nonexistent authorities and real authorities cited for propositions they do not support. The paper tests whether decision models abstain when the correct legal authority is absent, and how they compare with generative LLMs.

1 Introduction

  • Wrong legal citations can involve nonexistent authorities or real authorities that do not support the cited proposition, adding work for lawyers and courts.Courts may need to check citations, hold hearings, and decide on sanctions.Source: §1 · pp. 1–2
  • Accuracy-focused evaluation rewards guessing, although a wrong citation can reach a filing while an unnecessary abstention costs an extra lookup.The paper argues that confidence should help users decide which answers to check.Source: §1 · p. 2
  • Most evaluations assume the correct authority is among the candidates and treat errors equally, unlike the practical task of checking a citation.The paper asks whether models commit to a wrong authority or abstain when the right one is missing.Source: §1 · p. 2
  • System One decision models return probabilities rather than generated text, a fit for the closed choice of a holding or none of them.The paper notes that whether they are more trustworthy when the authority is unavailable had not been tested.Source: §1 · p. 3
  • The study extends CaseHOLD with missing-authority items and compares four decision models with six generative LLMs, including combinations and probability-only training of a local model.The authors also report a contamination check and describe the distilled Qwen3-8B model as well calibrated.Source: §1 · p. 3

2 Related Work

Prior legal NLP datasets support verifiable-answer tasks, while abstention research and System One models provide related approaches. The paper positions its work at the intersection of missing legal authority and model decisions.

2.1 Legal Datasets with Verifiable Answers

  • Legal datasets such as CaseHOLD, ContractNLI, and MAUD pose questions with source-checkable answers, but usually assume a correct candidate and equal error costs.CaseHOLD asks which of five holdings a court gave for a cited case.Source: §2.1 · pp. 3–4

2.2 Abstention and Unanswerable Questions

  • Prior abstention work studies unanswerable questions and finds that models often choose an option anyway; this paper brings that test to legal authority.The passage notes that reasoning fine-tuning can worsen abstention across twenty datasets [25].Source: §2.2 · p. 4

2.3 System One Decision Models

  • System One models return option probabilities, and prior work studies training, readout, calibration, and model combinations, but legal authority selection had not been evaluated.Existing legal comparison on ContractNLI found Jev cheapest and fastest but less accurate than hosted models [54].Source: §2.3 · p. 4

3 Method

The study evaluates two CaseHOLD conditions, compares decision models with generative LLMs, scores their outputs, tests combinations, and trains a local model from decision-model probabilities.

3.1 Task and Data

  • CaseHOLD items pair a citing-opinion excerpt with five candidate holdings, one from the cited case and four similar distractors.The evaluation uses the full test split of 5,314 items.Source: §3.1 · p. 5

3.2 Two Conditions

  • In the abstain condition, 1,670 of 5,314 items replace the correct holding with a similar real holding and make a sixth, none-of-these option correct.The remaining 3,644 items retain the correct holding, with the sixth option wrong; replacements have mean similarity 0.28 to the removed holding.Source: §3.2 · p. 5

3.3 Models

  • Four System One models return probabilities for each option, while six generative LLMs return one option key.The tested generative models span five vendors and a range of prices.Source: §3.3 · pp. 5–6

3.4 Prompts and Settings

  • Both model types receive the same one-sentence task instruction, but use different input formats and structured outputs.Generative LLMs run at temperature 0 with each model’s lowest available reasoning setting above none; their reasoning tokens and billed costs are recorded.Source: §3.4 · p. 6

3.5 Metrics

  • The evaluation reports accuracy and macro F1, plus wrong-pick and false-abstention rates, calibration error, and AUROC for decision models.Confidence intervals use 2,000 bootstrap resamples; paired comparisons use exact McNemar tests.Source: §3.5 · p. 6

3.6 Combining Decision Models and Generative LLMs

  • Three combination policies use veto thresholds, agreement-based review, or a System One-first cascade that calls the LLM on remaining items.Policies are scored as k·wrong citations + reviews, with illustrative k values of 1, 5, and 10; thresholds are tuned on 30% and evaluated on the remaining 70%.Source: §3.6 · pp. 6–7

3.7 Contamination Checks

  • Following Arvin [1], contamination checks compare passages with case names and citations anonymized, with only the cited-case citation, or with that citation removed.The procedure and results are reported in Appendix C.Source: §3.7 · p. 7

3.8 Distillation

  • Qwen3-8B is fine-tuned on 10,000 CaseHOLD items to match Jev’s probabilities without labeled data, alongside base and gold-label versions.The probability-trained and base versions use no labeled data; the student’s next-token probabilities over option letters support calibration measurement.Source: §3.8 · p. 7

4 Results

On standard CaseHOLD, the best decision model matched the best generative LLM’s accuracy at about one-fifty-fifth the cost, while models often chose a wrong holding when none applied. Agreement gating reduced wrong citations at the cost of more review, and probability distillation produced a well-calibrated Qwen3-8B without gold labels.

4.1 Accuracy and Cost

  • 88.1% for pplx-decider and 88.2% for Gemini 3.8 Flash (difference −0.2 points, 95% CI [−1.2, 0.8], p=0.74) came at about 1/55 of the cost for the decision model.Three of four decision models were more accurate than every generative LLM except Gemini; all models lost 9.9 to 18.7 accuracy points in the abstain condition.Source: §4.1 · p. 7

4.2 Committing When No Holding Is Correct

  • 30.1% of missing-holding items drew a wrong pick from Jev, versus 37.8% to 85.6% for generative LLMs; decision-model AUROC ranged from 0.73 to 0.89.Jev abstained on 26.2% of items that had a correct holding. At a 4.2% false-abstention rate, pplx-decider recognized 57.7% of missing-holding items, against Gemini’s 62.2%.Source: §4.2 · pp. 7–8, Figure 3 · p. 8

4.3 Combining Decision Models and Generative LLMs

  • 9.4% wrong citations with the agreement gate, down from Gemini 3.8 Flash’s 18.4%, while unnecessary reviews rose from 3.0% to 12.7%.For k=5, a combined policy was best for every tested pair; at k=10, the best policy for most pairs sent more than half of items to review and reduced wrong citations to about 2% or less.Source: Figure 4 · p. 9, §4.3 · p. 9

4.4 Contamination

  • Anonymizing case names and citations cost the strongest models only 0.5 to 5.7 accuracy points, despite their above-chance performance using citation alone.The authors conclude that models know much of the case law but do not depend on it.Source: §4.4 · p. 9

4.5 Distillation

  • Trained only on Jev’s probabilities, Qwen3-8B reached 63.8% accuracy, near Jev’s 63.9%, and calibration error of 0.025, below Jev’s.Gold-label training reached 74.9% accuracy; Jev’s probabilities closed about two-thirds of the accuracy gap from the untrained model without labeled data. The comparison covers the abstain condition, with each version trained once on 10,000 items.Source: §4.5 · p. 10, Table 3 · p. 10

5 Discussion

Decision models and generative LLMs have complementary failure patterns, so legal AI should be assessed by its wrong citations and unnecessary abstentions, not accuracy alone. Pairing models can reduce wrong citations, while local distillation offers a way to run the check on a firm’s hardware.

5.1 Two Ways to Fail

  • Jev’s caution means it abstains on about one in four items that do have a correct holding, and its confidence is not always a reliable sign of missing knowledge.The decision models’ probability of NONE separates items with and without a correct holding, with AUROC 0.73 to 0.89.Source: §5.1 · p. 10

5.2 Accuracy Is the Wrong Scoreboard for Legal Authority

  • Accuracy counts a wrong citation and an unnecessary review equally, although a wrong citation can reach a filing and end in sanctions.The acceptable risk depends on the user, matter, and court, so the operating point belongs to the lawyer or firm.Source: §5.2 · pp. 10–11

5.3 A Cheap Check, Not a Substitute

  • Pairing a decision model with a generative LLM halved wrong citations while keeping nearly all correct answers in the test.The decision model costs a small fraction of a generative LLM call, making a second opinion on every answer feasible.Source: §5.3 · p. 11
  • A combined system can direct a lawyer’s reading toward answers most likely to be wrong, but it does not remove the duty to read and verify cited authorities.Courts require authorities to be read and verified regardless of their source.Source: §5.3 · p. 11
  • Even the strongest models often identified a case’s holding from its citation alone, but legal authority depends on what the cited opinion actually held.The paper notes that prior knowledge can override the provided text in verification.Source: §5.3 · p. 11

5.4 Running the Check Locally

  • A local Qwen3-8B model reproduced Jev’s accuracy and abstention behavior and became well calibrated after 90 minutes of training on one workstation GPU, using Jev’s probabilities without labels.Its teacher’s probabilities can be collected on public material, but the student inherits the teacher’s mistakes and should be tested on a firm’s own work.Source: §5.4 · p. 11

6 Limitations

The evaluation has limits in its training-data certainty, constructed absence condition, error-cost assumptions, and scope. The distillation result is also based on a single training run.

Contamination

  • The closed models’ undisclosed training data means the study cannot rule out exposure to CaseHOLD’s public opinions, decided before 2021.Perturbation checks show that models know much of the case law but do not depend on recognizing the case.Source: §6 · p. 11

A constructed absence

  • The abstain condition tests one constructed absence: replacing a correct holding with a real holding from another case, not cases that are overruled, from the wrong jurisdiction, or nonexistent.Source: §6 · p. 11

The cost of errors

  • The reported values of k are illustrative; the authors know of no empirical estimate of how much worse a wrong citation is than an unnecessary review, and the value varies by practice setting.Source: §6 · p. 11

Scope

  • The study tests one task—English-language U.S. case law—and one prompt per model type; generative-model behavior may differ with other prompts, reasoning settings, or retrieval.The results describe model versions available in October 2026.Source: §6 · p. 11
  • Each distillation configuration was trained once on 10,000 items, so its results carry more uncertainty than those of the hosted models.Source: §6 · p. 12

7 Conclusion

The paper compares generative LLMs and System One decision models when the correct authority is absent. Generative LLMs often still choose a holding, while pairing models halves wrong citations and probability distillation supports a well-calibrated local model.

7 Conclusion

  • Most generative LLMs still choose a holding when the authority is absent, while the most cautious decision model does so in fewer than one in three cases at a fraction of the cost.Source: §7 · p. 12
  • Pairing a decision model with the strongest generative LLM tested halves its wrong citations.Source: §7 · p. 12
  • A decision model’s probabilities can teach a local model to behave similarly without labeled data.Source: §7 · p. 12
  • Calibrated probabilities provide a dial for users to choose how cautious a research tool should be, including when it cannot find the authority.Source: §7 · p. 12

A Prompts

The appendix gives the shared task prompt and describes how each model type receives the item. Decision models receive a choice question, while generative LLMs reply with an option key under a constrained format.

A Prompts

  • The shared prompt asks which holding statement a citing court gave after replacing the cited case’s holding parenthetical with <HOLDING>.Source: A Prompts · p. 14
  • Decision models receive the citing passage as state and the holdings as choice criteria, including NONE when no listed holding matches the passage.Source: A Prompts · pp. 14–15
  • Generative LLMs receive the citing passage and lettered options and are constrained to reply with only an option key through a JSON schema.Source: A Prompts · p. 15

B Pilot Study and Additional Results

A pilot found that GPT-6 Luna often chose a holding with high stated confidence when none was correct, while decision-model probabilities also improved a smaller encoder’s calibration. The results show both the limits of stated confidence and the potential of probability distillation.

Stated confidence

  • 60% of items without a correct holding drew a GPT-6 Luna pick, with mean stated confidence of 0.91 on those wrong picks.On the same items, its expected calibration error was 0.29, versus 0.07 and 0.08 for the two decision models.Source: B Pilot Study and Additional Results · p. 15
  • A 0.90 confidence threshold reduced GPT-6 Luna’s wrong picks to a level similar to a decision-model veto, but required a threshold chosen on labeled data.The main study therefore asked generative LLMs for an answer only, since they do not state confidence in normal use.Source: B Pilot Study and Additional Results · p. 15

Distillation into a smaller encoder

  • Adding Jev’s probabilities raised legal BERT’s original-task accuracy by 0.7 points and halved calibration error from 0.043 to 0.022.On the abstain condition, Jev’s probabilities made the model abstain more often without improving accuracy; Jev added little because BERT was already nearly as accurate as Jev on CaseHOLD.Source: B Pilot Study and Additional Results · p. 15

C Contamination Checks

The contamination checks test whether performance depends on recognizing cited cases, using citation-only, anonymized, and citation-removed variants. Strong models scored well from citations alone, but their accuracy changed little under anonymization or citation removal.

Contamination checks

  • Anonymization is a proxy for dependence on case recognition, not a direct measure of whether closed models’ training data contained CaseHOLD.The cases in CaseHOLD predate the tested models, while closed-model training data is undisclosed.Source: C Contamination Checks · p. 15
  • An earlier CaseHOLD anonymization test reported GPT-4o macro F1 falling from 0.744 to 0.728, with 88% of answers unchanged.The present study repeats anonymization with GPT-6 Luna as the rewriter and checks the rewrites automatically.Source: C Contamination Checks · pp. 15–16
  • The authors anonymized 500 items, retained 406 rewrites that passed automatic checks, and compared original, anonymized, citation-removed, and citation-only variants.The variants test whether accuracy survives removal of case-name and citation cues, or is high when only the citation is shown.Source: C Contamination Checks · p. 16, Table 4 · p. 16
  • 74.4% for Gemini 3.8 Flash and 71.2% for pplx-decider on citation-only items; anonymization cost 0.5–5.7 points and citation removal cost 0.5–5.2.Decision models lost the least under anonymization, at 0.5–2.2 points; the authors conclude pplx-decider’s strong results are not an artifact of case recognition.Source: C Contamination Checks · p. 16, Table 4 · p. 16

D Distillation Settings

The Qwen3-8B distillation experiment compares training on correct-option labels with training to match Jev’s option probabilities. It uses LoRA fine-tuning on a sample of CaseHOLD training items.

Distillation settings

  • Qwen3-8B was fine-tuned with LoRA on 10,000 randomly sampled training items for one pass, using an NVIDIA RTX A6000.The setup used rank 16, α=32, dropout 0.05, bfloat16, and gradient checkpointing.Source: D Distillation Settings · p. 16
  • The Jev-trained model minimized KL divergence from Jev’s probabilities, while the gold-trained model minimized cross-entropy against the correct option.Both models saw the same instruction, with options A–E and F for none, and thinking disabled.Source: D Distillation Settings · p. 16
  • Training passages were truncated to their last 600 tokens and options to 160 tokens, although one CaseHOLD holding exceeds 20,000 tokens.Training used AdamW with a 10^-4 learning rate, cosine schedule, 5% warmup, and effective batch size 16.Source: D Distillation Settings · p. 16
Loading 2610.29961v1…