PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving versus Verifying: Catching Contradictions in Tax Reasoning Systems

Albert Sadowski, Jarosław A. Chudziak

arXiv:2609.05928v1cs.CLcs.AI

TL;DR

Real legal inputs may omit required facts or contain contradictions, and accuracy on clean cases does not show how models handle those defects. We evaluate six models on SARA-derived tax cases and find that verification catches many contradictions missed during solving; a self-verification gate turns that detection into abstention at little clean-accuracy cost.

  • Problem

    Existing abstention benchmarks primarily test underspecification, leaving unclear whether models distinguish missing facts from contradictions in statutory tax reasoning.

    Source: §1 · pp. 1–2

  • Method

    We evaluate six models on SARA-derived tax cases with missing-fact and contradiction perturbations, and test a same-model verifier that gates solver outputs on contradiction labels.

    Source: §1 · p. 2, §3.6 · p. 5

  • Results

    The strongest models abstain on missing facts but compute through contradictions, returning the clean answer 63-76% of the time; the gate recovers 70-83% contradiction abstention at a 0-5.1 percentage-point clean-accuracy cost across six models.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    High accuracy on well-formed inputs alone is an incomplete measure of reliability, while a single self-verification call can recover missed contradiction detection as abstention without training or external tooling.

    Source: §7 · p. 10

    The evaluation uses stylized SARA-derived cases from one benchmark and one legal domain, and does not disentangle defect type from fact category.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models now compute correct tax liabilities on over 90% of well-formed cases in statutory benchmarks, which makes them candidates for the tax-advisory and compliance systems that consume such an answer directly. Real legal inputs, however, are frequently defective: required facts are missing, or stated facts contradict one another. Accuracy on clean benchmarks says nothing about how a model behaves then, and a system that computes straight through a defective input returns a confident number with no sign that anything is wrong. This raises two questions: does a model asked to solve a case abstain when the input is defective, and when it does not, can the same model catch the defect when asked instead to verify the input? We study six recent models on SARA-derived tax cases under missing-fact and contradictory-fact perturbations. The strongest models abstain when a fact is missing but compute through injected contradictions, returning the clean-input answer 63-76% of the time with no signal of the conflict; asked instead to verify the same input, they flag most of those contradictions. We wire that verification call into a simple contradiction gate: one extra call that abstains when the model reports a conflict. Across all six models it recovers most of the missed contradiction abstention at a clean-accuracy cost of at most about 5 percentage points, with no training and no external tooling. High accuracy on well-formed inputs is therefore an incomplete measure of reliability, and the detection the solver misses is cheaply recoverable with a single self-check.

1 Introduction

High accuracy on well-formed legal inputs leaves model behavior on defective inputs unclear, even though real legal cases are often incomplete or inconsistent. This study tests whether models abstain on missing or contradictory facts and whether verification can catch conflicts missed during solving.

Motivation

  • Accuracy on well-formed inputs tells little about behavior on defective inputs, which are common in legal practice because clients omit facts and records conflict.A system that computes through such inputs risks returning an answer without that risk being captured by standard accuracy benchmarks.Source: §1 · p. 1

Research gap

  • Existing abstention work largely tests underspecification, leaving unexplored whether models distinguish missing facts from contradictions in statutory tax reasoning.Prior work examines abstention in other domains and legal knowledge retrieval, while tax reasoning applies legal rules to structured facts.Source: §1 · pp. 1–2

Study questions

  • The study evaluates six recent models on SARA-derived tax cases under clean, redacted, and contradicted conditions to test solving and verification behavior.Redaction removes a required numeric fact; contradiction adds a conflicting categorical fact, and verification is evaluated on contradicted cases.Source: §1 · p. 2

Contributions

  • On injected contradictions, the four high-accuracy models abstain on at most 15% while solving, but flag 75-83% when asked to verify the same inputs.They return the clean-input answer 63-76% of the time; a one-call gate recovers 70-83% contradiction abstention at a 0-5.1 percentage-point clean-accuracy cost across all six models.Source: §1 · p. 2

2 Related Work

Prior work spans legal reasoning, abstention, and contradiction detection, but the paper focuses on defects in facts supplied directly to a reasoning model. It compares behavior on missing and contradictory facts and applies self-verification to input validation.

2 Related Work

  • The paper situates its contribution across LLM legal applications, selective prediction, and language-model processing of contradictory information.Its central focus is the solve-versus-verify distinction on defective tax inputs.Source: §2 · p. 2, Figure 1 · p. 3

2.1 LLMs for Legal Reasoning

  • Legal-reasoning research ranges from tax-law expert systems and statutory benchmarks to LLM-based tax reasoning, multi-agent frameworks, and structured prompting.Prior work reports persistent reasoning gaps alongside accuracy sufficient to motivate deployment discussions.Source: §2.1 · p. 2

2.2 Abstention and Selective Prediction

  • Abstention research covers unanswerable questions across domains, while self-assessing answerability offers a simpler reliability signal that this study applies to input validation.Prior studies include benchmark evaluation, abstention training, and formal conformal, entropy-based, and ensemble methods.Source: §2.2 · p. 2

2.3 Contradiction Detection and Reasoning Under Conflict

  • Prior contradiction work examines conflicting retrieval passages or legal documents, whereas this study tests contradictions among facts given directly to a reasoning model.It compares behavior across missing and contradictory input defects.Source: §2.3 · p. 3

3 Methodology

The study perturbs SARA tax cases with controlled missing-fact and contradiction defects, then measures six models’ solving and verification responses. A separate verifier call can override the solver to abstention on detected contradictions.

3 Methodology

  • The evaluation applies controlled redaction and contradiction perturbations to clean tax cases, with the dataset, perturbation pipeline, scripts, and results made publicly available.Source: §3 · p. 3

3.1 Base Dataset

  • The base dataset comprises 91 standardized SARA tax cases covering statutory reasoning over the US Internal Revenue Code.Each describes a taxpayer’s situation and asks for a tax liability, with ground-truth answers computed from the applicable statute.Source: §3.1 · p. 3

3.2 Perturbation Design

  • Redaction removes a required non-zero numeric fact, while contradiction inserts a conflicting categorical statement into the case description.The retained sets include 78 redacted variants and 108 contradiction variants; cases were manually reviewed to exclude defects that did not make the answer undetermined or create a genuine conflict.Source: §3.2 · pp. 3–4

3.3 Models Evaluated

  • Six recent models from four providers were evaluated in their default configurations, including four chain-of-thought reasoners and two direct-answer models.Each model was run three times on the full dataset to assess result stability.Source: §3.3 · pp. 4–5

3.4 Evaluation Prompt and Response Classification

  • The solver receives the statute, case, and question, then returns a numeric liability or null; responses are classified as numeric, zero, or absent.Only an absent response counts as abstention, and the binary output design does not expose reasoning traces.Source: §3.4 · p. 5

3.5 Metrics

  • The evaluation measures clean-input accuracy, perturbed-input abstention, and answer preservation against the clean-input ground truth.Abstention is the primary metric; preservation diagnoses reproduction of the clean answer, while its meaning is less reliable for models whose clean answers are often wrong.Source: §3.5 · p. 5

3.6 Self-Verification and the Contradiction Gate

  • The verifier is the same model asked whether facts are sufficient and consistent, and the gate overrides the solver to null when it labels a contradiction.The gate adds one call, ignores missing labels, and is evaluated using detection rate, clean-input false-flag rate, and clean-accuracy cost; the verifier is run three times per model.Source: §3.6 · pp. 5–6, Table 2 · p. 6

4 Results

On clean cases, the four high-accuracy models perform far better than two direct-answer models, but solving does not reliably expose injected contradictions. A self-check gate raises contradiction abstention across all six models, while detection varies by contradiction type.

4 Results

  • Results average three independent runs; standard deviations are rough stability indications, and the solve-versus-verify diagnosis rests on the four high-accuracy models.The direct-answer models are included to test whether the gate transfers across different solving behavior.Source: §4 · p. 6, §4.1 · p. 6

4.1 Baseline Accuracy on Clean Inputs

  • At ±10%, clean accuracy was GPT-5 mini 96.7%, Qwen3.7-Plus 93.4%, Gemini 2.5 Flash 91.6%, and Kimi K2.5 89.4%, versus GPT-5.2 29.7% and Claude Sonnet 17.9%.The paper calls the first four high-accuracy models and the latter two direct-answer models.Source: §4.1 · p. 6

4.2 Contradiction: Models Do Not Report Inconsistency While Solving

  • On redacted inputs, no model preserves the clean-input answer above 1.3%; models abstain on 27–92%, showing the contradiction behavior is not general reluctance to answer.For Qwen3.7-Plus, preservation is 0.9% on redacted inputs versus 63.3% on contradictions.Source: §4.2 · p. 6, Table 3 · p. 7
  • On contradictions, the four high-accuracy models abstain at most 14.8% of the time but return the clean-input answer 63.3–76.2% of the time.Kimi K2.5 never abstains; the reported preservation rates are Gemini 76.2%, GPT-5 mini 74.7%, Kimi 67.6%, and Qwen3.7-Plus 63.3%.Source: §4.2 · pp. 6–7

4.3 The Contradiction Gate

  • 70–83%: the gate raises contradiction abstention across all six models; for high-accuracy models, clean-answer preservation falls from 63–76% to 11–15%.Figure 2 shows the P0-to-P1 change; Table 4 reports gate measures.Source: Figure 2 · p. 7, §4.3 · p. 7, Table 4 · p. 8
  • The gate’s false-flag rate on clean inputs is 0.0–5.5%; its accuracy cost is 5.1 points for GPT-5 mini and 4.0 for Kimi K2.5, but 0.4 for Gemini and Qwen.Across three runs, a solve-run disagreement baseline detects only 11–35% of contradictions, versus 75–83% for verification, with a 3–14% clean false-flag rate.Source: §4.3 · p. 8

4.4 Contradiction Type Analysis

  • Verifier detection is highest for filing-status conflicts (95–97%), lower for marital-status conflicts (78–83%), and lowest for dependent-count conflicts (30–59%).Dependent-count conflicts can be legally defeasible despite directly negating a stated fact; detection intervals for that category are wide.Source: §4.4 · pp. 8–9, Table 5 · p. 8

5 Discussion

Solving can miss contradictions that the same model detects when asked to verify; a single verification call can recover the missed abstention. The paper presents this gate as a cheap first check, not a replacement for stronger methods.

5 Discussion

  • When asked to solve, a model may compute through an injected contradiction; asked to check the same input, it reports the conflict.The authors do not settle why solving misses contradictions; defaulting to one of two competing values is offered as one possibility.Source: §5 · p. 9

5.1 Implications for Legal AI Deployment

  • High benchmark accuracy alone does not establish safe behavior on real-world legal inputs, where contradictory client information is common.A model can fail to recognize that a task should not be solved even when it could solve a clean version correctly.Source: §5.1 · p. 9
  • A single verification call can recover missed contradiction abstentions, using the same model without training or an external solver.The authors position the gate as a cheap first check, not a replacement for heavier checks that catch defeasible conflicts.Source: §5.1 · p. 9

6 Evaluation and Future Work

The evaluation identifies a solve-versus-verify gap, while limiting its conclusions to stylized SARA-derived US federal income-tax cases. The authors outline tests across domains and verifier designs to assess transfer and improve detection.

6 Evaluation and Future Work

  • Contradiction detection is a distinct failure mode from missing-premise abstention, and the paper’s addition is a same-model gate exploiting the solve-versus-verify gap.The authors describe the connection to reasoning-model abstention patterns as agreement, not confirmation, because reasoning mode cannot be separated from capability.Source: §6 · p. 9
  • The evaluation uses stylized SARA-derived cases from one benchmark and one legal domain, and pairs each defect type with one fact category.Whether the missing-versus-contradictory asymmetry generalizes to other legal settings remains open.Source: §6 · p. 9
  • The same-model verifier can share the solver’s blind spots, while reported gate recovery is a lower bound over verifier choices.The same-model setup tests whether detection is recoverable by changing the task without adding external knowledge.Source: §6 · pp. 9–10
  • Dependent-count contradictions have the weakest detection, at 6–59%; many are defeasible rather than strict conflicts.The authors suggest stronger consistency checks for settings that require strict treatment.Source: §6 · p. 10
  • Future work includes evaluation in other legal domains, reasoning-trace analysis, cross-model verification, and stronger verifiers for defeasible conflicts.Proposed stronger approaches include neuro-symbolic, conformal, and ensemble methods.Source: §6 · p. 10

7 Conclusion

Models almost never return the clean answer when required facts are missing, but the strongest compute through contradictions; asking them to verify instead exposes the conflict. A single-model contradiction gate turns that detection into abstention, though broader domains and stronger verifiers remain future work.

7 Conclusion

  • When facts are missing, models abstain or almost never reproduce the clean answer, but the strongest compute through contradictions without signaling the conflict.Asked to verify instead of solve, the same models report the conflict.Source: §7 · p. 10
  • A single self-verification call converts recovered contradiction detection into abstention across all six models, with little clean-accuracy cost and no training or external tooling.The paper describes the gate as an easy first check, not a complete solution.Source: §7 · p. 10
  • High accuracy on well-formed inputs does not predict safe behavior on contradictory inputs, making benchmark accuracy alone an incomplete reliability measure.Source: §7 · p. 10
  • Extending the gate to other legal domains and stronger verifiers that catch defeasible conflicts remains future work.Source: §7 · p. 10
Loading 2609.05928v1…