Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation

1 Introduction

• The study develops three datasets from Indian and UK Supreme Court cases and benchmarks representative summarization methods across model families.The analysis includes domain-independent and legal-specific methods, as well as supervised and unsupervised extractive and abstractive approaches.Source: §1 · pp. 1–2
• Legal-corpus pretraining, including Legal-Pegasus, consistently improves performance in the reported experiments.The study also explores ways to generate legal training data and further fine-tune pretrained models.Source: §1 · p. 2
• Chunking-based summarization generally performs better for legal documents, especially with fine-tuning, while Longformer performs best on UK-Abs for some metrics.The paper compares long-document summarizers, chunked short-document summarizers, and an extractive-then-abstractive approach.Source: §1 · p. 2
• The evaluation combines document-wide and segment-wise automatic scores with assessments by law practitioners.Practitioner evaluation is included because legal summaries may need to represent rhetorical segments such as Facts and Final Judgement.Source: §1 · p. 2
• Law practitioners often prefer simpler unsupervised and extractive summaries, even when supervised methods score higher on ROUGE and extractive and abstractive models achieve similar best ROUGE scores.Document-wide metrics alone therefore give an incomplete picture of legal-summary quality.Source: §1 · p. 2

Extractive domain-independent methods

• General extractive methods include unsupervised approaches such as Reduction and LexRank, alongside matrix-factorization, reconstruction-based, and supervised neural methods.SummaRuNNer and BERTSum frame summarization as sentence-level in-summary versus out-of-summary classification.Source: §2 · p. 2

Extractive domain-specific methods

• Legal-specific extractive methods include sentence-ranking systems, template-based summaries, and Gist, which ranks sentences using handcrafted features and three alternative models.The described systems include CaseSummarizer, MMR, and Gist.Source: §2 · pp. 2–3

Abstractive methods

• Most abstractive models have input limits of at most 1,024 tokens, while Longformer architectures can handle documents up to 16 × 1,024 input tokens.This contrast motivates methods designed for long legal documents.Source: §2 · p. 3

Long-document methods

• Long-document approaches include compressing documents to salient sentences before generation and divide-and-conquer methods that split documents into smaller summarization problems.The paper reproduces a simplified extractive-abstractive approach and applies a method inspired by divide and conquer to fine-tune abstractive models.Source: §2 · p. 3

Abstractive methods

• LegalSumm generates summaries of at most 200 tokens, too short for the target summaries in this study, so the authors do not apply it.The method uses the Portuguese RulingBR dataset, whose documents and summaries are shorter than those in this work.Source: §2 · p. 3

Indian-Abstractive dataset (IN-Abs)

• IN-Abs contains 7,130 Indian Supreme Court judgment-summary pairs, with 7,030 for training and 100 reserved for evaluation.The summaries are headnotes collected from the Legal Information Institute of India.Source: §3 · p. 3

Indian-Extractive dataset (IN-Ext)

• IN-Ext contains expert-written extractive summaries for 50 Indian Supreme Court judgments, with sentences labeled by rhetorical segment.Two LLB graduates labeled 9,380 sentences and wrote summaries about one-third the length of the documents; Ratio and Precedent were summarized together.Source: §3 · p. 3

UK-Abstractive dataset (UK-Abs)

• The UK-Abs reference summaries are official press summaries divided into Background to the Appeal, Judgement, and Reasons for Judgement.They accompany judgments published by the UK Supreme Court.Source: §3 · p. 4
• UK-Abs judgments are approximately twice as long as IN-Abs and IN-Ext documents, and their 0.11 compression ratio makes UK-Abs the most challenging dataset for automatic summarization.The UK dataset contains 793 judgments and summaries from cases decided during 2009–2021, with 693 pairs for training and 100 for evaluation.Source: §3 · p. 4

Target length of summaries

• At inference, models generate summaries up to a target length L set from the reference summaries’ word counts.For IN-Ext, L is the average length of the two reference summaries; for the other datasets, it is the length of the single reference.Source: §4 · p. 4

Evaluation of summary quality

• Document-wide ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore compare each generated summary with its reference, averaging scores over evaluation documents.For IN-Ext, scores against two reference summaries are averaged for each document.Source: §4 · p. 4
• Segment-wise ROUGE evaluates the full generated summary against each rhetorical segment’s portion of the reference, and is available only for IN-Ext and UK-Abs.The reported scores are averaged across documents for each segment.Source: §4 · p. 4
• Segment-wise evaluation reports recall rather than precision or F-scores because systems do not label which generated sentences belong to each segment.The authors consider precision and F-scores misleading under this setup.Source: §4 · p. 4

Expert evaluation

• Three law experts evaluate summaries from selected methods that achieve high ROUGE scores.The expert assessment is a separate component of the evaluation.Source: §4 · p. 4

Consistency scores

• The authors conclude that SummaCCONV consistency scores are unreliable for legal summarization and do not report them.It assigned expert-written references average scores of 0.485 on IN-Abs and 0.367 on UK-Abs; the authors suggest lack of legal-domain fine-tuning as a possible reason.Source: §4 · p. 5

• Extractive methods span legal-specific and domain-independent models, with both supervised and unsupervised approaches represented in each group.The legal-specific groups include LetSum, KMM, CaseSummarizer, MMR, and Gist; the domain-independent groups include LexRank, LSA, DSDR, Luhn, Reduction, PacSum, SummaRuNNer, and BERTSum.Source: §5 · p. 5
• Because IN-Abs and UK-Abs contain abstractive summaries, they cannot directly provide sentence labels for training extractive models.The supervised models require each source sentence to be labeled for inclusion or exclusion.Source: §5 · p. 5
• Avr selects three source sentences per abstractive-summary sentence by average ROUGE-1, ROUGE-2, and ROUGE-L, then labels their union for inclusion.Among the tested conversions from abstractive to extractive summaries, Avr produced the best supervised-method performance.Source: §5 · p. 5

6 Abstractive Summarization Methods

• Legal documents often exceed abstractive models’ input limits, motivating experiments with long-document models, chunking, and hybrid summarization.The section applies pretrained and legally finetuned abstractive methods.Source: §6 · p. 5

6.1.1 Models meant for short documents

• BART and Legal-Pegasus use chunking for short-document inputs, dividing documents within model token limits and appending summaries of the chunks.BART accepts up to 1024 tokens, fewer than the words in a typical legal case document.Source: §6.1.1 · p. 5

6.1.2 Models meant for long documents

• Legal-LED accommodates most case documents using a 16,384-token input capacity, while longer UK-Abs documents are truncated at that limit.Longformer models use attention that scales linearly with sequence length.Source: §6.1.2 · pp. 5–6

6.1.3 Hybrid extractive-abstractive approach

• BERT_BART first selects salient sentences with a BERT-based extractive model, then generates the final summary with chunking-based BART.Chunking-based BART is used because the required summary length often exceeds 1024 tokens.Source: §6.1.3 · p. 6

6.2 Finetuning Abstractive Models

• Finetuning data is built by mapping reference-summary sentences to their most similar document sentences, then grouping them by source chunk as chunk summaries.This produces multiple document-chunk and summary pairs for finetuning.Source: §6.2 · p. 6
• Among four sentence-similarity measures, MCS performed best; it represents sentences with mean SBERT token embeddings and compares them by cosine similarity.BART_MCS performed best among the three finetuned BART versions, and MCS was used to generate data for other models.Source: §6.2 · p. 6

7.1 Evaluation of Extractive methods

• Luhn leads unsupervised extractive methods on IN-Ext, while DSDR leads on IN-Abs and UK-Abs; SummaRuNNer leads supervised methods on IN-Abs and UK-Abs, and BERT-Ext on IN-Ext.CaseSummarizer leads unsupervised legal-specific methods on IN-Abs and UK-Abs, while LetSum leads on IN-Ext.Source: §7.1 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

7.2 Evaluation of Abstractive methods

• No method performs well across all segments: LetSum leads in only one of five IN-Ext segments, while Legal-LED leads FAC and ARG on IN-Ext and two of three UK-Abs segments.The authors suggest Legal-LED may have an advantage over chunking-based methods on the longest UK-Abs documents.Source: §7.1 · p. 7, §7.2 · p. 8, Table 6 · p. 8
• Legal-Pegasus leads pretrained abstractive models on IN-Ext and IN-Abs, while BART_MCS leads on UK-Abs; Legal-Pegasus and chunked BART perform best document-wide, followed by BERT_BART.All models improve notably through fine-tuning; Legal-Pegasus with chunking outperforms long-document Legal-LED on document-wide scores.Source: §7.2 · pp. 7–8, Table 3 · p. 7, Table 4 · p. 7

7.3 Expert evaluation

• Law experts rated DSDR highest overall, followed by CaseSummarizer; DSDR also scored best for important-information coverage and readability.The evaluation covered 35 summaries from seven algorithms across five IN-Abs documents, rated by three experts.Source: §7.3 · pp. 8–9, Table 8 · p. 9
• Experts found no summary balanced across rhetorical segments; DSDR represented final judgements and statutes well but missed precedents and arguments.Experts also reported that summary quality often declined from the middle, and abstractive summaries could be disorganized or incomplete.Source: §7.3 · p. 9
• Expert Overall ratings correlated weakly with ROUGE-1 (0.212), ROUGE-2 (0.208), ROUGE-L (0.132), and BERTScore (0.067).The authors conclude that automatic metrics may be insufficient for judging summaries in specialized domains such as Law.Source: §7.3 · p. 9

8 Concluding discussion

• The study develops datasets and benchmark results for legal case judgment summarization.Source: §8 · p. 9
• For legal case summarization, DSDR and SummaRuNNer are promising extractive methods, while Legal-Pegasus is a good abstractive choice.Source: §8 · p. 9
• Fine-tuning models through chunking seems promising for long documents.Source: §8 · p. 9
• Document-wide evaluation is incomplete; the study recommends domain-specific evaluation, including evaluation by domain experts.Source: §8 · p. 9

A.1 Rhetorical Role Labels in a Legal Case Document

• The labels identify sentences’ semantic functions, and experts say a good summary should concisely represent each segment.Source: §A.1 · p. 12
• Facts (FAC) cover the chronology leading to filing, while Argument (ARG) covers the parties’ arguments.Source: §A.1 · p. 12
• Statute (STA) captures laws cited by the present court, and Precedent (PRE) captures prior cases it referred to.Source: §A.1 · p. 12
• Ratio denotes the present court’s reasoning for its judgment, while Ruling by Present Court (RPC) denotes its final judgment.Source: §A.1 · p. 12

A.2 Implementations details of Domain-Specific Extractive Summarization Methods

• Because Indian and UK training data for MMR’s two-stage classifier were unavailable, the study uses only its Maximum Margin Relevance module as a baseline.MMR balances sentence relevance to the case against similarity to the current summary, with λ = 0.5.Source: §A.2 · p. 12
• Gist ranks sentences using handcrafted features, omitting case-category features absent from the Indian and UK documents; its Gradient Boosted Decision Tree performed best.Features include sentence length and position, legal terms, word embeddings, and POS tags.Source: §A.2 · pp. 12–13
• CaseSummarizer combines normalized sentence TF-IDF with date, entity, and section-start features; for Indian cases, legal-term counts replace section headings.The adaptation also counts Act section numbers alongside dates and uses Stanford NER for entity identification, with the same setting on UK-Abs.Source: §A.2 · p. 13
• LetSum and KMM use sentence-ranking modules based on TF-IDF and K-mixture scores; rhetorical labels are only used after summarization for structured display.Source: §A.2 · p. 13

A.3 Implementation Details of Domain-Independent Extractive Summarization Methods

• Several domain-independent methods use public implementations; the study omits BERTSUM trigram blocking because it produced summaries that were too short.Source: §A.3 · p. 13

A.4 Methods for obtaining Training Data for Extractive Supervised Methods

• For extractive training labels, Maximal greedily adds sentences while ROUGE improves, whereas TF-IDF labels source sentences similar to reference-summary sentences above a 0.4 threshold.The appendix describes these as alternatives to the Avr method used in the experiments.Source: §A.4 · pp. 13–14

A.5 Implementation details of Abstractive Summarization Methods

• The abstractive implementations include BART, Legal-Pegasus, and Legal-LED, with fine-tuning hyperparameters reported in Table 9.Source: §A.3 · p. 13, §A.5 · p. 14

A.6 Methods for obtaining finetuning data for abstractive summarization models

• For abstractive fine-tuning data, MCS_RR partitions documents into seven rhetorical-role segments, summarizes each, and further chunks segments exceeding model input limits.It uses rhetorical-role sentence labels and MCS sentence similarity; other similarity measures tried are also described.Source: §A.6 · p. 14

A.7 Detailed Summarization Results

• Detailed tables report document-wide ROUGE and BERTScore for all methods on IN-Ext, IN-Abs, and UK-Abs, plus segment-wise ROUGE for IN-Ext and UK-Abs.Source: §A.7 · pp. 14–15, Table 10 · p. 15, Table 11 · p. 15

A.8 More Insights from Segment-wise Evaluation

• Many domain-independent models perform better on large opening segments than on short ending segments; performance is relatively poor on legally important rulings and final judgments.The passage identifies lead bias as a possible issue and describes strong performance on opening segments as a pattern, not a definitive cause.Source: §A.8 · pp. 15–16

A.9 Expert Evaluation Details

• The expert survey selected five IN-Abs documents with the best average ROUGE-L across seven methods, and expert availability limited evaluation; IAA uses averaged pairwise Pearson correlations.The survey’s selection excluded lower-ROUGE summaries that ideally would also have been evaluated.Source: §A.9 · pp. 16–17

A.10 Ethics and limitations statement

• The study covers Indian and UK legal documents, so whether its reported algorithm-performance trends generalize to other countries is uncertain; expert findings may also vary across practitioners.Source: §A.10 · p. 17