Source-linked AI summary
LAURA: Knowledge Distillation for Interpretable Ambiguous Clause Identification in Legal Contracts
Amrita Singh, Aditya Joshi, Jiaojiao Jiang, Hye-young Paik
TL;DR
Contract ambiguity ranges from flexible interpretation to significant legal conflict, so identifying ambiguous clauses alone does not show which need attention. LAURA distills teacher-generated legal rationales into an open-weight student trained for both classification and rationale generation. With Flan-T5, it achieves state-of-the-art interpretability over interpretable baselines while matching the best opaque baseline's identification performance.
Problem
Contract ambiguity can permit flexible interpretation or produce legal conflicts, while the available identification approach lacks interpretability and does not distinguish these consequences through rationales.
Source: §1 · p. 1Method
Results
LAURA with Flan-T5 matches RoBERTa under SFT’s best opaque-baseline performance (binary F1 0.70; accuracy 0.69) and achieves state-of-the-art interpretability over interpretable baselines.
Source: §4 · pp. 3–4, §5 · p. 4Takeaways & Limitations
The authors conclude that prompt-based approaches alone are insufficient and task-specific model training is essential for correct and reliable interpretation.
Source: §5 · p. 4Evaluation uses one relatively small, English-only dataset of legal contracts, so generalization to other legal text types may or may not hold.
Source: Limitations · p. 5
Abstract
from arXiv · showhide
Legal contracts contain ambiguities that expose enterprises to financial and legal risks. Some ambiguities allow flexible interpretation without triggering disputes, while others lead to significant legal conflicts. This makes identification alone insufficient, and interpretable rationale analysis essential. We propose LAURA, a post-training framework for interpretable ambiguous clause identification. LAURA leverages knowledge distillation with an IRAC-Unlearning prompting technique to transfer knowledge from a teacher LLM to an open-weight student model (<=1B parameters), which is then trained using a joint objective combining classification and rationale generation losses. The framework supports both legal and non-legal stakeholders in making informed decisions about which ambiguities require further attention. Extensive experiments across 7 baselines and 7 open-weight models demonstrate that LAURA with Flan-T5 (250M) delivers state-of-the-art interpretability over all interpretable baselines while matching the identification performance of the best-performing opaque baseline.
1 Introduction
Contract ambiguity can range from harmless interpretive flexibility to disputes, so classification alone does not capture which clauses warrant attention. LAURA pairs identification with textual rationales and uses knowledge distillation to support interpretable analysis with open-weight models.
1 Introduction
- The work addresses a gap in contract ambiguity research: the publicly available identification approach lacks interpretability, as its authors acknowledge.The paper describes its work as the first on interpretable ambiguous clause identification, to the authors' knowledge.Source: §1 · p. 1
- Ambiguity may permit flexible interpretation or lead to significant legal conflicts, making a binary label insufficient to distinguish consequential clauses.The requested rationale should identify ambiguous terms, explain their conflicting interpretations, and note legal consequences where applicable.Source: §1 · p. 1
- LAURA distills legally grounded teacher rationales using IRAC-Unlearning prompting and trains a student jointly for ambiguity classification and rationale generation.The student predicts labels and produces human-readable rationales.Source: §1 · p. 1
- LAURA with Flan-T5 matches the best-performing opaque baseline's identification performance while achieving state-of-the-art interpretability over interpretable baselines.The evaluation spans 7 baselines, 7 models, and 5 LAURA variants.Source: §1 · p. 1
- LAURA uses open-weight models without commercial APIs at inference, addressing stated cloud cost, scalability, and privacy concerns for sensitive contracts.The paper notes that API-based clause-by-clause processing across long documents can be costly and difficult to scale.Source: §1 · pp. 1–2
2 Framework of LAURA
LAURA uses teacher-generated, label-conditioned rationales to train a student model for joint clause classification and rationale generation. Its training objective combines classification and rationale losses, and the distilled model is evaluated on held-out clauses.
2 Framework of LAURA
- GPT-4o generates each training rationale conditioned on both the clause and its ground-truth label, supporting rationale correctness and dataset integrity.This addresses the limited annotated dataset, which contains no rationales.Source: §2 · p. 2
- IRAC-Unlearning prompts structure teacher rationales through Issue, Rule, Application, and Conclusion stages, augmented with an unlearning signal.The technique is intended to elicit domain-faithful rationales transferable to the student.Source: §2 · p. 2
- The student receives the prompt concatenated with the clause and produces a predicted label followed by a rationale token sequence.Training combines binary cross-entropy classification loss and sequence-level cross-entropy rationale loss, weighted by λ, set to 1 in all experiments.Source: §2 · p. 2
- After fine-tuning, the student becomes the distilled model evaluated on the held-out test set.LAURA produces the label and rationale in one forward pass, requiring a sequence-generating architecture.Source: §2 · p. 2
3 Experiment Setup
The experiments use the publicly available contract ambiguity dataset and compare prompt-based and fine-tuning baselines alongside LAURA. Evaluation measures classification performance and assesses rationale quality and classification errors.
3 Experiment Setup
- The sole publicly available contract ambiguity dataset contains 1,000 clauses across 25 contract types, with 524 labeled ambiguous and 476 not ambiguous.The experiments use an 80:20 train-test split; the dataset provides no rationales.Source: §3 · pp. 2–3
- Seven baselines include a majority classifier, three GPT-4o prompting methods, and three fine-tuning-based approaches: SFT, IFT, and PPI.PPI generates rationales from IFT predictions using IRAC-Unlearning prompting, without additional training.Source: §3 · p. 3
- Classification evaluation reports ambiguous-class precision, recall, and F1 alongside accuracy, emphasizing accuracy and binary F1 as primary measures.The study also conducts classification error analysis and qualitative rationale evaluation for correctness, completeness, and conciseness.Source: §3 · p. 3
4 Results and Analysis
LAURA with Flan-T5 balances identification performance while achieving the strongest rationale quality among the compared interpretable approaches. The analysis also shows that prompt-only approaches and clause-level error cases remain challenging.
Identification performance
- Prompt-based baselines perform poorly: Direct and CoT barely exceed the majority baseline, while Few-Shot reaches binary F1 of 0.44 and 0.40.Direct and CoT have high recall but low precision; Few-Shot has higher precision but lower recall.Source: §4 · p. 3
- Table 1 reports LAURA with Flan-T5 at binary F1-score 0.70 and accuracy 0.69, matching the best-performing opaque baseline, RoBERTa under SFT.RoBERTa under SFT also achieves binary F1 of 0.70 and accuracy of 0.69.Source: Table 1 · p. 3, §4 · pp. 3–4
- SFT and IFT/PPI achieve binary F1 ranging from 0.65 to 0.71, but no model in those approaches balances binary F1 and accuracy as well as LAURA and RoBERTa under SFT.IFT/PPI shows higher recall and lower precision, indicating over-prediction of ambiguous clauses.Source: §4 · pp. 3–4
Rationale quality
- LAURA achieves the highest correctness and conciseness scores across baselines, and outperforms all baselines and variants across correctness, completeness, and conciseness.Its completeness can be lower when minor details are missed.Source: Figure 2 · p. 4, §4 · p. 4
Error analysis
- LAURA’s classification errors include 19 vague, 2 incomplete, 4 referential, and 25 not-ambiguous clauses; context-dependent terms and standard legal phrases contribute to misclassification.The analysis also notes over-flagging of cross-references and redacted content.Source: §4 · p. 4
5 Conclusion
The paper presents LAURA as a post-training framework for joint ambiguous-clause identification and rationale generation, evaluated systematically with open-weight models up to 1B parameters. LAURA with Flan-T5 achieves state-of-the-art interpretability while maintaining identification performance, and the analysis identifies context-dependent terms and clause-level processing as continuing challenges.
Conclusion
- Across 7 baselines, 7 models, and 5 LAURA variants, Flan-T5 achieves state-of-the-art interpretability while maintaining identification performance.It outperforms all interpretable baselines in rationale correctness and conciseness.Source: §5 · p. 4
Limitations
The evaluation is limited by the available data, contract-only setting, model and task design, and clause-level context. LAURA can also omit rationale details and make identification errors.
Limitations
- Only one publicly available contract-ambiguity dataset exists; it is relatively small and exclusively English, limiting experimental scope.The dataset’s contract-type diversity does not establish generalization beyond this setting.Source: Limitations · p. 5
- LAURA is evaluated only on legal contracts, and its generalization to statutes, court cases, or legal opinions is uncertain.The paper notes that legal language and interpretation vary across genres.Source: Limitations · p. 5
- Clause-level processing limits access to broader document context, making cross-references, referential ambiguities, and context-dependent terms harder to resolve.The paper says LAURA nevertheless performs well at the clause level.Source: Limitations · p. 5
- The single-pass label-and-rationale output is a sequence-to-sequence task that may be harder to jointly optimize with decoder-only models at small parameter counts.The paper contrasts these models with Flan-T5’s encoder-decoder architecture, which it describes as well suited to the task.Source: Limitations · p. 5
- LAURA occasionally omits rationale details and makes identification errors.The paper treats these as limitations of joint legal interpretation and ambiguity identification with open-weight models up to 1B parameters.Source: Limitations · p. 5
Ethical Considerations
The discussion warns that ambiguity-identification systems can be misused or over-trusted, and that their outputs require oversight. It also describes six ambiguity categories and explains why rationales matter when clause ambiguity is uncertain.
Ethical Considerations
- Potential risks include deliberate insertion of ambiguity, harmful misinterpretations, and automation bias; model outputs are not substitutes for legal expertise and require caution and oversight.The dataset contains contract clauses without personal data, sourced from public U.S. SEC EDGAR filings.Source: Ethical Considerations · p. 5
A Types of ambiguity in legal text: definitions and examples
- The appendix distinguishes lexical, syntactic, semantic, vague, incomplete, and referential ambiguity; rationales help readers interpret clauses when ambiguity is uncertain.The paper notes that identifying ambiguity without a rationale is challenging, even when a clause is known to be ambiguous.Source: A Types of ambiguity in legal text:… · p. 7
B Rationale Evaluation Criteria
The section outlines rationale evaluation and teacher-prompting approaches, culminating in IRAC-Unlearning, which silently applies legal analysis before producing a concise rationale. It also describes prompts for direct classification and rationale generation.
B Rationale Evaluation Criteria
- Rationales are intended to identify ambiguity-driving terms, explain conflicting interpretations, and note legal consequences where applicable.The stated evaluation covers correctness, completeness, and conciseness.Source: B Rationale Evaluation Criteria · p. 7
C Prompt used to elicit the rationale from the teacher model (GPT-4o)
- The teacher-prompting section lists Chain-of-Thought, Few-Shot, Contrastive, SC-Unlearning, and IRAC-Unlearning approaches.The Contrastive prompt internally generates support for the given label and a counter-argument for the opposite label before weighing them.Source: C Prompt used to elicit the rationale… · p. 7, §C.3 · p. 8
C.1 Chain-of-Thought (CoT)
- The CoT teacher prompt requests a stepwise rationale that identifies relevant phrases, explains competing interpretations, and notes consequences when applicable.The output is a short rationale tied to the supplied clause and label.Source: §C.1 · p. 8
C.2 Few-Shot
- The Few-Shot prompt supplies an ambiguity definition, including vagueness, incompleteness, and referential ambiguity.The available prompt text also begins describing vagueness as phrases that allow borderline cases.Source: §C.2 · p. 8
C.4 Issue-Rule-Application-Conclusion (IRAC)-Unlearning
- IRAC-Unlearning asks the teacher to identify the interpretive issue, apply legal standards, test them against the clause and label, then output a concise rationale.Its prompt names contra proferentem, the plain-meaning rule, and the reasonable-person standard; when analysis conflicts with the label, it directs the model to re-examine through the label’s lens.Source: §C.4 · p. 8, I - ISSUE · p. 9, R - RULE · p. 9
C.5 Self Correction (SC)-Unlearning
- The paper’s prompting setup also includes SC-Unlearning and a separate phase of reasoning about a clause without first seeing its label.The latter phase asks the model to form an initial rationale and predicted label from the clause text alone.Source: §C.5 · p. 9, PHASE 1: UNBIASED ANALYSIS · p. 9
E Models and Hyperparameters
The experiments use seven open-weight models, with model-specific settings for fine-tuning and LAURA. Examples show predicted labels and rationales for clauses labeled both correctly and incorrectly.
E Models and Hyperparameters
- Seven open-weight models span encoder-only, encoder-decoder, decoder-only, and legal-domain-specific architectures.The set includes BERT, RoBERTa, Flan-T5, Qwen-2.5, Llama-3.2, Legal-BERT, and Contracts-BERT.Source: E Models and Hyperparameters · p. 10
- LAURA uses batch size 4 and learning rates 3e-4 for Flan-T5, 2e-4 for Qwen-2.5, and 3e-4 for Llama-3.2, with a maximum of 10 epochs.Runs use AdamW and a single seed, 2018, due to limited computational resources.Source: E Models and Hyperparameters · p. 11
F.1 Correct Predicted Labels with their Rationales
- Examples of correctly predicted clauses include clear arbitration venue wording and ambiguous terms such as “applicable regulatory requirements” and “sole discretion.”The rationales point to the stated location as unambiguous and to the latter terms as lacking specificity or a clear standard.Source: §F.1 · p. 11
F.2 Incorrect Predicted Labels with their Rationales
The examples show incorrect predictions in both directions: some clauses labeled not ambiguous are predicted ambiguous, while others labeled ambiguous are predicted not ambiguous.
F.2 Incorrect Predicted Labels with their Rationales
- A clause labeled not ambiguous is predicted ambiguous because “reasonable control” is treated as subjective and lacking a clear standard.The example is a force majeure clause listing circumstances beyond a party’s reasonable control.Source: §F.2 · p. 11
- A clause labeled not ambiguous is predicted ambiguous because redacted timing text, “[***] after the Effective Date,” is treated as insufficiently specific.The rationale says the redaction leaves uncertainty about when and under what conditions a forecast must be provided.Source: §F.2 · p. 11
- A clause labeled ambiguous is predicted not ambiguous because its warranty and liability terms are treated as clear to a reasonable person.The rationale interprets the terms as disclaiming liability for consequences arising from Internet access.Source: §F.2 · pp. 11–12
G Additional Analysis
The analysis reports that rationales for incorrectly labeled clauses are unconvincing and verifiably incorrect, with errors involving both false positives and false negatives.
G Additional Analysis
- False-positive rationales flag contextually clear “reasonable control” language and treat redacted “[***]” content as ambiguous despite not-ambiguous labels.For false negatives, LAURA selects random phrases to justify its predictions; annotators characterize rationales for incorrectly labeled clauses as unconvincing and verifiably incorrect.Source: G Additional Analysis · p. 12