Source-linked AI summary
When Do LLMs Apply the Wrong Law? Diagnosing LLM Failures in Temporal Legal Reasoning
Yiqian Huang, Shuyuan Zheng, Qianying Liu, Shaowen Peng, Yuntao Kong, Kotaro Funakoshi, Chuan Xiao, Manabu Okumura, Yang Cao
TL;DR
LLMs must identify which historical version of a law governs facts occurring at a particular time, but their reliability in this temporal legal reasoning remains unclear. This paper benchmarks and diagnoses temporal applicable-law determination, finding a strong newest-law bias linked to reduced reasoning-path diversity and worse performance by stronger general reasoners.
Problem
Temporal legal reasoning requires identifying the statutory version governing conduct when it occurred, yet whether LLMs can reliably make this determination remains underexplored.
Method
The paper introduces a benchmark for temporal applicable-law determination and uses diagnostic probes to examine knowledge, temporal-scope understanding, and reasoning-policy mechanisms.
Results
LLMs show a strong newest-law bias, with failures concentrated on selecting the wrong statutory version rather than unrelated legal citations.
Takeaways & Limitations
Temporal legal reasoning requires reasoning policies aligned with domain-specific applicability rather than relying on general reasoning ability alone.
Takeaways & Limitations
The dataset covers Chinese civil verdicts, so the results may not directly generalize to other legal systems.
Abstract
from arXiv · showhide
Legal reasoning tasks such as legal judgment prediction (LJP) require identifying the temporally correct version of the law governing a case -- a capability we term temporal applicable-law determination. However, whether large language models (LLMs) can reliably perform this task remains unexplored. In this paper, we construct a benchmark to evaluate LLMs on temporal applicable-law determination, and systematically investigate why they fail at temporal legal reasoning. Our experiments reveal four key findings. First, LLMs exhibit a strong bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred. Second, this bias does not stem from an inability to understand that laws have temporal scope, nor from a lack of knowledge about historical statutes. Third, we provide behavioral evidence that reinforcement-learning-shaped explicit reasoning may be a key mechanism: while improving general reasoning ability, it reduces the diversity of reasoning paths, causing models to converge on applying the current law. Fourth, this produces a counterintuitive inverse relationship: models with stronger general reasoning ability tend to perform worse on temporal legal reasoning. Our findings offer concrete guidance for future work on improving LLM performance in temporally grounded legal reasoning.
1 Introduction
The paper studies why LLMs apply temporally incorrect laws, focusing on temporal applicable-law determination as a prerequisite for reliable downstream legal reasoning. It introduces a benchmark and finds that models systematically favor the most recently enacted law despite the timing of relevant facts.
- Motivation: Temporal legal reasoning requires applying the law in force when the relevant conduct occurred, because later revisions may not govern earlier events.Misapplying the temporal version can fundamentally alter legal conclusions.
- Task and benchmark: The paper introduces temporal applicable-law determination (TALD), which asks models to identify the statutory version governing a case from its facts and event date.TALD is presented as a necessary prerequisite for downstream tasks such as legal fact prediction and legal judgment prediction.
- Key findings: LLMs show a strong, systematic bias toward applying the most recently enacted law, regardless of when the legally relevant facts occurred.This behavior directly explains failures previously observed in LawShift.
- Key findings: The bias does not stem from ignorance of prior statutory versions, while reasoning-oriented reinforcement learning can reduce diversity in TALD-relevant reasoning paths.The paper frames this work as a systematic diagnosis of root causes in temporal legal reasoning.
2 The Temporal Applicable-Law Determination Task
Temporal applicable-law determination is the prerequisite of legal reasoning: models must identify the statute versions in force when legally relevant facts occurred, consistent with non-retroactivity. The TALD task formalizes this as predicting applicable statute-version citations and maximizing exact matching accuracy.
- Temporal Legal Structure: Non-retroactivity generally limits newly enacted laws to facts and legal relationships arising after their entry into force.Statutes and regulations have legally bounded effective periods, while completed matters generally remain governed by prior law.
- Role in Legal Reasoning: Temporal applicable-law determination is the necessary first step because applying a superseded or not-yet-effective statute produces legally erroneous results.This requirement precedes applying substantive law to reach a legal conclusion.
- Task Definition: Given facts F and legal question Q, TALD predicts a sequence y of citations identifying the legally applicable version of each statute.Each statute may have multiple versions resulting from temporal legal evolution.
- Evaluation: TALD accuracy measures matching between ground-truth versions y and model predictions ˆy, and the task objective is to maximize this accuracy.The metric counts correctly matched applicable statute versions relative to the relevant ground-truth or predicted statute positions.
3 Analysis Setup
This section frames temporal applicable-law determination (TALD) through three research questions on temporal cues, legal knowledge, and legal reasoning. It introduces a judgment-based benchmark with pre-Code and post-Code cases, diagnostic knowledge questions, and broad model evaluation.
- Research questions: The analysis asks whether LLMs use temporal cues correctly, possess temporal-applicability knowledge, and apply that knowledge through legal reasoning.These correspond to RQ1, RQ2, and RQ3.
- Benchmark construction: The TALD benchmark uses civil judgments to identify the temporally correct civil-law version from case facts and plaintiff claims, with cited law versions as ground truth.Queries and labels are extracted automatically from standardized Chinese civil judgments using regular-expression matching.
- Benchmark construction: The dataset contains 26,000 judgments split between post-Code cases governed by the 2021 Civil Code and pre-Code cases governed by prior standalone statutes.The Civil Code consolidates the former Property Law and Contract Law.
- Diagnostic evaluation: Two legal experts construct 16 either-or questions testing knowledge of Chinese Civil Code temporal-applicability provisions as a diagnostic for RQ2.The questions assess whether failures reflect missing knowledge of temporal applicability rules.
- Models and evaluation: Evaluation covers proprietary, open-source, and domain-specific legal reasoning models, using each model’s highest available reasoning effort by default.The models include GPT-5.4, Claude Opus 4.6, Gemini-3.1-Pro, DeepSeek-V3.2, GLM-4.7, Qwen3 variants, and LegalOne-8B.
4 Analysis on RQ1
Advanced reasoning LLMs show a pronounced directional failure in temporal applicable-law determination: they perform poorly when older law governs but competitively when newer law applies. Their errors are primarily version-centric, reflecting selection of the newer version rather than unrelated legal sources.
- Performance asymmetry: On the Old-version split, every model scores below 0.25, while seven out of eight score below 0.15; Claude-Opus-4.6 reaches 0.014.On the New-version split, scores range from 0.70 to 0.84 despite the same reasoning requirements and case facts.
- Performance asymmetry: Models systematically perform poorly when older law governs but perform competitively when newer law applies, indicating a default tendency toward the most recently enacted law.This directional asymmetry is identified as the first finding on advanced reasoning LLMs’ TALD performance.
- Fault analysis: Most errors are OLD-TO-NEW faults, whereas NEW-TO-OLD errors are rare and only a small portion of failures are version-irrelevant.OLD-TO-NEW means the model cites a new version when the previous version is legally applicable; NEW-TO-OLD is the reverse.
- Fault analysis: These failures are version-centric: models often identify the relevant statute family but select the newer version when the previous version should apply.The pattern suggests models recognize temporally versioned legal sources but choose the wrong temporal direction.
5 Experiments on RQ2
Experiments on RQ2 show that TALD failures are not primarily caused by missing statutory or temporal-effect knowledge. Instead, stronger general reasoning can worsen performance, while task-specific temporal-law hints redirect models toward the correct applicability policy.
- Legal knowledge: Models generally retain old and new statutory content and perform well on temporal-effect rules, indicating that knowledge deficiency is not the main source of TALD failure.DeepSeek-V3.2 achieves 0.963 character-level F1 on gold articles versus 0.901 on predicted articles; Qwen3-235B achieves 0.902 versus 0.776.
- General reasoning: The results contradict the hypothesis that TALD failure is merely caused by insufficient general reasoning ability.Best performances include Qwen3-30B-instruct (0.292), GPT-5.4high (0.237), and Qwen3-235B-thinki…
- General reasoning: Stronger general reasoning does not consistently improve old-version TALD and often makes performance worse.More than half of evaluated LLMs perform worse in reasoning-oriented configurations; Qwen3-80B drops from 0.080 in instruct mode to 0.028 in thinking mode.
- Reasoning policy: Temporal-law hints substantially improve old-version TALD for many reasoning-oriented configurations, including Qwen3-80B thinking from 0.028 to 0.512.Other gains include DeepSeek-V3.2 reasoner from 0.092 to 0.436 and GPT-5.4 under strong hint from 0.237 to 0.457.
- Reasoning policy: TALD failure is closely tied to task-specific reasoning-policy misalignment, which domain-expertise hints can redirect toward correct temporal applicability.The prompting effect is not uniform across configurations, so hints are not simply a general prompt bonus.
6 Experiments on RQ3
RQ3 tests whether policy-entropy collapse explains why explicit reasoning can harm temporal applicable-law determination (TALD). Across models, interventions that improve TALD generally reopen version-related reasoning paths, supporting task-directed diversity over temporal-law alternatives rather than merely more explicit reasoning.
- Measurement: RQ3 measures token-level policy entropy only in version-related reasoning spans as a behavioral proxy for exploration.These spans include law names, temporal markers, version descriptions, retroactivity, effective periods, and related legal-temporal expressions.
- Results: Stronger temporal-law hints generally increase both TALD accuracy and version-related entropy across Qwen3 and LegalOne models.Qwen3-30B rises from 0.020 accuracy and 0.315 entropy without hints to 0.450 accuracy and 0.462 entropy with strong hints; LegalOne-8B rises from 0.093/0.147 to 0.298/0.173.
- Results: Successful TALD correction is accompanied by reopened version-related reasoning paths, although higher entropy alone is not the optimization target.After a model reaches a better task-specific policy region, additional uncertainty around version-related tokens may not improve accuracy further.
- Conclusion: Finding 6 attributes explicit-reasoning failures to over-exploitation of salient newest-law trajectories on old-version cases.Interventions that improve TALD generally raise entropy in version-related chain-of-thought spans, indicating that success requires task-directed diversity over temporal-law alternatives.
7 Related Work
Prior Legal AI research largely treats legal labels as static downstream prediction targets, while emerging work examines temporal dynamics and reasoning-oriented LLMs. This paper complements these directions by focusing on whether models can determine which law version temporally governs a case.
- Downstream Legal Prediction: Legal AI has studied downstream prediction from case facts, including judgments, charges, law articles, and legal citations.CAIL2018 formulates legal judgment prediction as predicting applicable law articles, charges, and penalties from fact descriptions.
- Temporal Applicable-Law Determination: Existing article- and citation-prediction benchmarks typically treat legal labels as static, unlike reasoning that must distinguish statute versions with different effective periods.The paper therefore complements downstream prediction by diagnosing temporal applicable-law determination as a prerequisite for legal reasoning.
- Temporal Legal AI: Recent legal-AI studies examine temporal generalization, evolving legal language, and the ordering of legal events within narratives.ChronosLex uses chronological splits, LexTempus models legal-language evolution with a dynamic mixture-of-experts framework, and LexTime evaluates temporal event ordering.
- Reasoning-Oriented LLMs: Reasoning LLMs use reinforcement learning on verifiable tasks to generate explicit chain-of-thought, but such training is concentrated in mathematics, formal logic, and competitive programming.These training domains are structurally remote from legal reasoning, which involves long natural-language texts, domain-specific principles, and context.
8 Conclusion
The paper introduces TALD, a benchmarked task for identifying the temporally applicable statutory version, and finds that advanced LLMs strongly favor the newest law. The failures reflect a misaligned reasoning policy rather than primarily missing legal knowledge, while the study’s conclusions are limited in scope to Chinese civil judgments and the temporal axis of law version.
- Contribution: TALD requires models to identify the legally applicable version of statutory law, using a benchmark constructed from Chinese civil judgments.The study introduces TALD as a temporally grounded legal reasoning task.
- Main findings: Advanced LLMs exhibit a strong newest-law bias, performing much better when the newest version applies but failing severely when previous versions govern.Fault-type analysis indicates that these failures are usually version-centric rather than unrelated legal citations.
- Failure diagnosis: Stronger general reasoning does not naturally solve TALD and can worsen old-version performance, suggesting a misaligned task-specific reasoning policy rather than missing statutory knowledge.Diagnostic probes indicate that the failure is not primarily caused by ignorance of temporal-effect rules.
- Failure diagnosis: Entropy analysis suggests explicit reasoning over-exploits a narrow newest-law trajectory, whereas correction requires reopening exploration over temporal-law alternatives.The authors argue that legal LLMs should align reasoning with domain-specific applicability policies as well as general problem-solving ability.
- Limitations: The dataset covers Chinese civil verdicts, limiting direct generalization to other legal systems, and the study examines only the temporal axis of law version.Other applicability axes, such as jurisdictional version, also require selecting among related provisions.
9 Ethics Statement
The work is an evaluation of LLM behavior, not a source of legal advice, because incorrect law-version determinations can have serious real-world consequences. LLM outputs require professional verification, and judgment-document datasets should follow privacy and data-use rules.
- 9 Ethics Statement: LLM outputs should not be used as legal advice and must be verified by qualified legal professionals because incorrect law-version determinations can have serious consequences.Dataset construction should follow applicable privacy and data-use rules, including anonymization and removal of personally identifiable information where required.
A System Prompts with Hint Levels
The section defines a base law-article prediction prompt and two progressively stronger prompts that emphasize temporal applicability, intertemporal effects, and retroactivity. The strongest hint additionally states non-retroactivity as the civil-law default, identifies beneficial retroactivity as an exception, and directs examination of when facts occurred.
- A.1 Default Prompt (No Hint): The default-prompt subsection flags that the exact base prompt for law-article prediction remains to be inserted.The passage provides an implementation placeholder rather than the prompt text.
- A.2 Weak Temporal-Law Hint: The weak-hint prompt asks for factual, good-faith legal consultation responses.It supplements the default interaction framing with a temporal-law instruction.
- A.2 Weak Temporal-Law Hint: The weak hint directs attention to event timing and the law’s temporal applicability or intertemporal effect.It specifically highlights retroactivity as a relevant issue.
- A.3 Strong Temporal-Law Hint: The strong-hint prompt requests factual, faithful legal consultation responses.This wording differs from the weak hint’s good-faith response instruction.
- A.3 Strong Temporal-Law Hint: The strong hint intensifies temporal guidance by emphasizing event timing, intertemporal applicability, and retroactivity.It instructs the model to examine whether prior law governed when the relevant facts occurred.
- A.3 Strong Temporal-Law Hint: The strong hint presents non-retroactivity as civil law’s basic rule while acknowledging recognized exceptions such as beneficial retroactivity.This frames temporal-law analysis around a default rule with stated exceptions.