Source-linked AI summary
How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?
Aniket Deroy, Kripabandhu Ghosh, Saptarshi Ghosh
TL;DR
Legal judgments are long and complex, and the paper asks whether pre-trained abstractive models and LLMs are ready to summarize them automatically. It evaluates these systems on Indian case judgments and finds that higher conventional scores do not prevent inconsistency and hallucination, so fully automatic deployment is not yet ready.
Problem
Because legal judgments are lengthy and complex, the paper asks whether pre-trained abstractive models and LLMs are ready for off-the-shelf case judgment summarization.
Source: §1 · pp. 1–2Method
The paper applies pre-trained abstractive models and LLMs to Indian court judgments, compares them with extractive models, and assesses summary quality and consistency.
Source: §1 · p. 2Results
Abstractive models often outperform extractive models on conventional metrics, but their summaries can contain inconsistencies and hallucinations.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
The authors conclude that these models are not ready for fully automatic legal case summarization and that a human-in-the-loop approach may be more suitable.
Source: §7 · p. 11Some abstractive-summary errors are subtle and much more difficult to detect automatically, including confusion between appellants and their lawyers.
Source: §7 · p. 11
Abstract
from arXiv · showhide
Automatic summarization of legal case judgements has traditionally been attempted by using extractive summarization methods. However, in recent years, abstractive summarization models are gaining popularity since they can generate more natural and coherent summaries. Legal domain-specific pre-trained abstractive summarization models are now available. Moreover, general-domain pre-trained Large Language Models (LLMs), such as ChatGPT, are known to generate high-quality text and have the capacity for text summarization. Hence it is natural to ask if these models are ready for off-the-shelf application to automatically generate abstractive summaries for case judgements. To explore this question, we apply several state-of-the-art domain-specific abstractive summarization models and general-domain LLMs on Indian court case judgements, and check the quality of the generated summaries. In addition to standard metrics for summary quality, we check for inconsistencies and hallucinations in the summaries. We see that abstractive summarization models generally achieve slightly higher scores than extractive models in terms of standard summary evaluation metrics such as ROUGE and BLEU. However, we often find inconsistent or hallucinated information in the generated abstractive summaries. Overall, our investigation indicates that the pre-trained abstractive summarization models and LLMs are not yet ready for fully automatic deployment for case judgement summarization; rather a human-in-the-loop approach including manual checks for inconsistencies is more suitable at present.
1 Introduction
Legal judgments are difficult for practitioners to read in full, and the paper asks whether pre-trained abstractive models and LLMs are ready to summarize them off the shelf. It evaluates these models against extractive systems and finds that higher conventional scores can coexist with inconsistent or hallucinated content, leading the authors to conclude that fully automatic use is not yet ready.
1 Introduction
- Legal judgments’ length and complexity make them difficult for even law practitioners to read fully, motivating automated summarization.Human-written summaries are traditionally used to reduce the effort of reading judgments.Source: §1 · p. 1
- The paper asks whether pre-trained abstractive models and LLMs are ready for off-the-shelf legal case judgment summarization.The authors note that these models can be applied without further training, while LLMs had seen little use for this task.Source: §1 · p. 2
- The study applies legal-domain pre-trained models and general-purpose LLMs to Indian Supreme Court judgments, with extractive models as comparators.The judgments have practitioner-written gold summaries, and evaluation includes conventional quality and source-consistency metrics.Source: §1 · p. 2
- Abstractive models often score slightly higher than extractive models on ROUGE, METEOR, and BLEU, yet their summaries can contain inconsistencies and hallucinations.Reported errors include wrong dates or names and confusion between people associated with a case.Source: §1 · p. 2
- The authors conclude that pre-trained abstractive models and LLMs need further improvement before legal experts can readily use them for case judgment summarization.They report this as the first analysis of consistency in legal abstractive summaries and present examples of errors from these models.Source: §1 · p. 2
2 Related work
Prior legal case summarization work mostly used extractive methods, while more recent work has explored abstractive models and long-document techniques. The paper identifies limited study of LLMs and hallucination in legal summarization as gaps its analysis addresses.
Summarization of legal case judgements
- Legal case judgment summarization has traditionally used extractive methods, including unsupervised, supervised, optimization, multi-task learning, and classification approaches.Examples include Bhattacharya et al. (2021), Agarwal et al. (2022), and Liu and Chen (2019).Source: §2 · p. 2
- Recent legal abstractive work includes models applied to Indian and UK judgments, long-document segmentation, and textual-entailment approaches.The cited work includes Shukla et al. (2022), Moro and Ragazzi (2022), and Feijo and Moreira (2023).Source: §2 · p. 2
Hallucinations in large language models
- Hallucination in NLP is text that is not true or accurate given the input, and prior summary research has tried to reduce it.The cited survey describes possible causes including limited or biased training data and model-architecture limitations (Ji et al., 2023).Source: §2 · pp. 2–3
- Prior discussion of LLMs notes risks of inaccuracies, biases, fabrications, and text inconsistent with established facts.The passage also cites examples of sexist and racist generated content (Stanczak and Augenstein, 2021).Source: §2 · p. 3
Novelty of this work
- The authors characterize analysis of abstractive methods and LLMs for legal judgments as limited, with hallucination in legal summarization not previously studied to their knowledge.They describe their work as a first step toward assessing readiness for automatic case judgment summarization.Source: §2 · p. 3
3 Dataset
The study uses IN-Abs, a dataset of Indian Supreme Court judgments paired with expert-written summaries. Models are trained or fine-tuned using its training split and evaluated on a held-out test set.
3 Dataset
- IN-Abs is reused from prior work (Shukla et al., 2022) and contains judgments from the Indian Supreme Court.The table caption describes statistics for the train and test sets and identifies their roles in training, fine-tuning, and evaluation.Source: §3 · p. 3, Table 1 · p. 3
- IN-Abs contains 7,130 Indian legal judgments, each paired with one abstractive summary written by law experts.The judgments are from the Legal Information Institute of India website.Source: §3 · p. 3
- The dataset divides 7,130 judgment-summary pairs into 7,030 training pairs and 100 test judgments; all models are evaluated on the test set.Some supervised models are trained or fine-tuned on the training set.Source: §3 · p. 3
- The dataset description identifies average word counts for judgments and gold summaries as part of its train/test statistics.The supplied table caption gives the scope of the statistics but no specific word-count values.Source: §3 · p. 3, Table 1 · p. 3
4 Methods for summarizing legal case judgements
The methods compare general-domain LLMs, legal-domain abstractive models, and extractive summarizers on Indian legal judgements. Long documents are handled in chunks, with model-specific prompts and training adaptations.
4.1 General-domain Large Language Models
- The study tests Text-Davinci-003 and Turbo-Gpt-3.5 with Tl;Dr variants; only Turbo-Gpt-3.5 is shown with an explicit prompt requesting a document summary in a specified number of words.Source: §4.1 · pp. 3–4
- Documents over 1,024 words are split into chunks of at most 1,024 words, summarized separately, then concatenated in source order.Shorter documents are summarized whole; chunk-size experiments led the authors to use 1,024 words consistently, while further exploration was left for future work.Source: §4.1 · pp. 4–5, §4.2 · p. 6
- Each LLM chunk receives a target summary length proportional to the document’s gold-summary compression ratio, allocating equal summary length despite uneven chunk importance.The authors note that relative chunk importance is not easy to determine.Source: §4.1 · p. 5
4.2 Legal domain-specific abstractive summarization models
- Legal-Pegasus and Legal-LED, initially fine-tuned on U.S. legal materials, were further fine-tuned on 7,030 Indian cases and summaries; long documents are chunked at 1,024 words.Legal-LED uses the Longformer architecture for long sequences.Source: §4.2 · p. 5
4.3 Extractive summarization models
- The extractive comparison uses CaseSummarizer, which scores sentences or phrases, plus supervised BertSum and SummaRunner/RNN_RNN sentence-selection models.CaseSummarizer adjusts scores using entities, dates, and proximity to headings; the supervised models were trained on the IN-Abs training set.Source: §4.3 · p. 6
5 Comparing performances of summarization models
The paper compares summarization models on their match with practitioner-written summaries and their consistency with the source judgements. Legal-domain abstractive models lead on standard metrics, but consistency analyses reveal potential hallucinations and inconsistencies.
5 Comparing performances of summarization models
- The evaluation compares generated summaries both with gold-standard summaries and with the input documents.The first comparison assesses match with the gold standard; the second assesses consistency with the source.Source: §5 · p. 6
5.1.1 Metrics
- ROUGE measures bigram and longest-common-subsequence matches, METEOR measures unigram overlap, and BLEU measures summary-to-gold-standard matching.The reported ROUGE measures include precision, recall, and F1; higher metric scores indicate a closer match with the gold-standard summary.Source: §5.1.1 · pp. 6–7
5.2 Consistency of summaries
- The consistency analysis checks whether generated summaries align with the original documents, with higher scores on its metrics indicating greater consistency.The authors focus on this check particularly for abstractive models and LLMs, which are known to hallucinate in text generation.Source: §5.2.1 · p. 7, §5.2 · p. 7
5.1.2 Comparative results
- Legal-domain abstractive models score best across the three model families, but their improvements over the best extractive model are statistically significant only for ROUGE-L metrics.LegPegasus-IN and LegLED-IN perform best among the domain-specific models; Indian legal-document fine-tuning improves on their US-legal-only counterparts.Source: §5.1.2 · p. 7, Table 3 · p. 8
5.2.1 Metrics
- SummaC estimates document-summary consistency using sentence-level natural-language-inference scores, while NumPrec and NEPrec measure the fractions of summary numbers and named entities found in the source.NumPrec and NEPrec depend on accurate detection; identifying all named entities in Indian legal documents is challenging, and the values depend on the Spacy toolkit.Source: §5.2.1 · p. 8
5.2.2 Comparative results
- LLM SummaC scores range from 0.5 to 0.65, while abstractive models still show inconsistent named entities and numbers despite often exceeding 0.9 on NEPrec and NumPrec.LegPegasus-IN and LegLED-IN score consistently higher across consistency metrics than LegPegasus and LegLED.Source: Table 4 · p. 9, §5.2.2 · p. 9
5.3 Takeaways from this section
- Across these comparisons, abstractive models outperform extractive models on ROUGE, METEOR, and BLEU, while supervised models outperform ChatGPT and Davinci on standard metrics.The authors also note that abstractive summaries can be inconsistent with their source documents.Source: §5.3 · p. 9
6 Inconsistencies in abstractive summaries
The authors manually inspect summary-document pairs to characterize errors, focusing on sentences with low SummaC scores and unmatched numbers or named entities. They report errors ranging from chunk-boundary sentence merges to hallucinated dates, legal entities, and mistaken identities.
6 Inconsistencies in abstractive summaries
- Manual inspection found multiple error types in abstractive summaries, including merged sentences, wrong numbers, irrelevant US legal references, and named-entity confusions.The authors inspected sentences with low SummaC scores or numbers and entities that did not match the source, then checked the relevant document passages.Source: §6 · p. 9
- Sentence merges that leave the first sentence incomplete appeared in summaries from almost all abstractive models and LLMs, mostly at chunk boundaries.The paper gives examples from Tables 5, 6, and 7.Source: §6 · p. 9
- A LegLED summary of a case heard in 1961 hallucinated the year 2019, which was absent from the source document.The example appears in Table 6, error 5.Source: §6 · p. 9
- LegLED summaries sometimes introduced unrelated US court and statute names into Indian case judgements; the authors attribute this tendency to its US legal training.They did not observe this error in LegLED-IN summaries or in ChatGPT and Davinci summaries.Source: §6 · pp. 9–10
- Named-entity errors included confusing a judge with a lawyer and mistaking appellants’ lawyers for the appellants, errors the authors say can mislead readers.The paper presents these as examples from LegLED and ChatGPT summaries, respectively.Source: §6 · p. 10
7 Concluding discussion
The experiments find that abstractive models can score better than extractive models on standard metrics, yet still produce inconsistencies and hallucinations. The authors conclude these models are not ready for fully automatic legal summarization and favor human oversight and better error detection.
7 Concluding discussion
- The evaluation applies a wide range of standard and consistency metrics to LLMs and domain-specific abstractive models on Indian Supreme Court judgments.The standard metrics include ROUGE, METEOR, and BLEU; the consistency metrics include Numprec, NERprec, and SummaC.Source: §7 · p. 10
- Abstractive models often outperform extractive models on ROUGE, METEOR, and BLEU, yet their summaries contain inconsistencies and hallucinations.The evaluation also uses Numprec, NERprec, and SummaC to assess consistency.Source: §7 · p. 10
- Fine-tuning LegLED on Indian legal data considerably reduces unrelated U.S. court and statute mentions, but subtle appellant–lawyer confusions remain difficult to detect automatically.Some errors may be detected and addressed through careful post-processing.Source: §7 · p. 11
- Pre-trained abstractive models and LLMs are not ready for fully automatic legal summarization; a human-in-the-loop approach and better detection of complex errors are needed.The authors identify these as conclusions from their experiments.Source: §7 · p. 11