PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities

Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, Doug Downey

arXiv:2206.10883v3cs.CLcs.CY

TL;DR

CRLC summaries are written by human legal experts who must synthesize key events and outcomes across multiple documents, often totaling over two hundred pages per case. The paper introduces Multi-LexSum, a dataset of expert-authored summaries at multiple granularities for large-scale U.S. civil-rights lawsuits. Current models struggle to summarize long source documents; human assessments average 0.43 on a 0–3 scale, while multi-task approaches show promise for long-summary quality.

  • Problem

    CRLC summaries are written by human legal experts who must synthesize key events and outcomes across multiple documents, often totaling over two hundred pages per case.

    Source: §1 · p. 1

  • Method

    The paper introduces Multi-LexSum, a dataset of expert-authored summaries at multiple granularities for large-scale U.S. civil-rights lawsuits.

    Source: §1 · p. 2

  • Results

    Current models struggle to summarize long source documents; human assessments average 0.43 on a 0–3 scale, while multi-task approaches show promise for long-summary quality.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The dataset is intended to support summarization research and development of systems that assist the CRLC and similar efforts.

    Source: §1 · p. 2, §6 · p. 10

    CRLC cases overrepresent plaintiff wins, and the dataset is limited to federal cases with dockets available online, so performance may not generalize to underrepresented cases.

    Source: §6 · p. 10

Abstract

from arXiv · show

With the advent of large language models, methods for abstractive summarization have made great strides, creating potential for use in applications to aid knowledge workers processing unwieldy document collections. One such setting is the Civil Rights Litigation Clearinghouse (CRLC) (https://clearinghouse.net),which posts information about large-scale civil rights lawsuits, serving lawyers, scholars, and the general public. Today, summarization in the CRLC requires extensive training of lawyers and law students who spend hours per case understanding multiple relevant documents in order to produce high-quality summaries of key events and outcomes. Motivated by this ongoing real-world summarization effort, we introduce Multi-LexSum, a collection of 9,280 expert-authored summaries drawn from ongoing CRLC writing. Multi-LexSum presents a challenging multi-document summarization task given the length of the source documents, often exceeding two hundred pages per case. Furthermore, Multi-LexSum is distinct from other datasets in its multiple target summaries, each at a different granularity (ranging from one-sentence "extreme" summaries to multi-paragraph narrations of over five hundred words). We present extensive analysis demonstrating that despite the high-quality summaries in the training data (adhering to strict content and style guidelines), state-of-the-art summarization models perform poorly on this task. We release Multi-LexSum for further research in summarization methods as well as to facilitate development of applications to assist in the CRLC's mission at https://multilexsum.github.io.

1 Introduction

Multi-LexSum addresses the demanding CRLC task of summarizing long, changing civil-rights cases with a large expert-authored dataset offering multiple summary lengths. Experiments find current models perform poorly, though multi-task training shows promise for long summaries.

1 Introduction

  • The CRLC’s expert writers synthesize multiple document types, often exceeding 200 pages per case, into case timelines; a typical summary takes 1–4 hours and must be updated as litigation proceeds.Automation could expand coverage and enable near-real-time updates, increasing access to case descriptions and understanding of disputes and resolutions.Source: §1 · pp. 1–2
  • Multi-LexSum collects about 40,000 source documents and 9,000 expert-written summaries covering about half as many cases.The dataset concerns large-scale civil-rights lawsuits from the Civil Rights Litigation Clearinghouse.Source: §1 · p. 2
  • The dataset combines tiny, short, and long summaries averaging 25, 130, and 650 words, source collections averaging over 75,000 words, and expert-authored summaries reviewed for quality.Writers follow carefully created guidelines; expert review is intended to reduce unsupported facts in the summaries.Source: §1 · p. 2
  • Human assessments averaged 0.43/3, while multi-task approaches training across granularities showed promise for long-summary quality.The authors say substantial improvements are needed before model summaries can provide utility for the CRLC project.Source: §1 · p. 2

2 Related work

Prior legal NLP spans many document types and tasks, while court-summarization work has largely used extractive methods. Multi-LexSum adds expert-authored abstractive summaries at multiple granularities for exceptionally long, multi-source cases.

2.1 Natural language processing for legal documents

  • Legal lawsuits contain domain-specific language and complex structures; legal NLP research spans document classification, information extraction, question answering, and summarization across multiple document types.Systems in specialized domains often benefit from language models pretrained on legal text.Source: §2.1 · p. 2
  • Earlier court-summarization approaches mainly extract source sentences, whereas Multi-LexSum studies abstractive paraphrases and provides over 9,000 expert-written summaries from about 4,500 U.S. federal civil-rights lawsuits.Abstractive output offers the potential for more succinct summaries in accessible language.Source: §2.1 · pp. 2–3

2.2 Summarization datasets in other domains

  • Multi-LexSum provides up to three expert-written summary granularities for the same source, unlike BookSum’s summaries of different-length source units such as paragraphs, chapters, and whole books.This creates an opportunity to study learning from and generating summaries at varying granularities for a fixed source.Source: §2.2 · pp. 3–4
  • Experts write Multi-LexSum summaries under detailed instructions, and another expert reviews them for correctness and stylistic consistency.The paper contrasts this with automatically curated datasets that can vary in quality or contain information unsupported by the source.Source: §2.2 · p. 4
  • Multi-LexSum’s multi-source context averages over 75,000 words, while BookSum averages 127,000 words per book but has 403 samples versus Multi-LexSum’s 4,500.The paper describes Multi-LexSum’s sources as an order of magnitude longer than most other datasets.Source: §2.2 · p. 4

3 Multi-LexSum

Multi-LexSum frames civil-rights case summarization across multiple output lengths and task mappings, using expert-curated lawsuit materials that can span hundreds or thousands of pages. Its dataset characterization shows long source and target texts alongside summaries that are both source-grounded and abstractive.

3.1 Task definition

  • Cases may span hundreds or thousands of pages; tasks map their documents, existing summaries, or combinations to long, short, and tiny outputs.The task definitions include document-to-summary mapping, shortening a ground-truth summary, and mappings that combine source documents with summaries.Source: §3.1 · p. 4

3.2 Creating Multi-LexSum summaries

  • Writers select about eight major-event documents, draft long summaries before shorter versions, update them as cases progress, and have drafts reviewed for accuracy and readability.Legal scholars, attorneys, and trained law students manually curate the data; an unusually complex case may take an experienced attorney ten or more hours to understand and summarize.Source: §3.2 · p. 5

3.3 Dataset characterization

  • Multi-LexSum’s source and long-summary texts are longer than those of existing datasets, except BookSum, which has fewer samples and focuses on literature.Table 2 compares dataset size, source-document counts, text lengths, and source-target coverage, density, and compression ratio.Source: §3.3 · pp. 5–6, Table 2 · p. 6
  • Long summaries have 0.94 extractive-fragment coverage, while Multi-LexSum density ranges from 2–4, indicating references are source-grounded yet abstractive.The paper says high coverage suggests fewer unsupported entities and facts, while density indicates summaries are generally not verbatim extractions.Source: §3.3 · p. 6

4 Experiments

Experiments test direct multi-document summarization, generating shorter summaries from longer ones, and multitask training across target granularities. Results show advantages for longer input context and cross-granularity training, while a generated-summary pipeline can lose performance when it lacks gold intermediate summaries.

4.1 Experimental Setup

  • The experiments cover D→L, D→S, D→T, L→S, L→T, and S→T, comparing BART, PEGASUS, LED, and PRIMERA using Rouge and BERT Score.LED and PRIMERA are designed to handle longer inputs; PRIMERA also uses an MDS-specific pretraining objective.Source: §4 · p. 6, §4.1 · p. 7

4.2 Multi-doc legal case summarization

  • The best extractive models trail PEGASUS by 47%, 64%, and 84% in Rouge-2 across the long, short, and tiny tasks, respectively.BERT-EXT performs similarly to or worse than the simple extractive heuristics.Source: §4.2 · p. 7

4.3 Generating shorter summaries from the longer version

  • PEGASUS Rouge-2 improves by 79% on L→S, 104% on L→T, and 128% on S→T compared with the corresponding direct-source tasks.S→T performs better than L→T.Source: Table 4 · p. 8, §4.3 · p. 8

4.2 Multi-doc legal case summarization

  • Long-input LED and PRIMERA outperform BART and PEGASUS on all three tasks; longer LED input adds 4–10% Rouge-2 on D→T.PRIMERA beats LED-16384 on D→L and is similar to LED-4096 on the shorter-target tasks.Source: §4.2 · p. 8

4.3 Generating shorter summaries from the longer version

  • Without a gold intermediate summary, the pipeline using a BART-generated input can perform more than 20% worse than direct-source models when generating tiny summaries.With gold long or short summaries at test time, PRIMERA matches the corresponding summary-to-summary models.Source: §4.3 · p. 8

4.4 Multitask training for summaries of different lengths

  • Training on three tasks generally improves performance; long-summary metrics rise 11–17%, while short-summary results change little over single-task training and are slightly worse with two tasks.All cases have long summaries, so the three-task models also see short and tiny views of those cases.Source: Table 5 · p. 9, §4.4 · p. 9

5 Human evaluation

The usability study tested model-assisted long-summary writing with CRLC writers and found that substantial editing remained necessary. Writers continued to identify erroneous or missing key fields as a barrier to useful generations.

Study Design

  • Two CRLC writers edited BART-generated long summaries for 40 new cases, taking 180 hours in total.The study scoped evaluation to D→L, the most effort-intensive setting.Source: §5 · p. 9

System Design

  • Initial CRLC feedback said end-to-end long-summary outputs hallucinated key information and could take longer to correct than writing from scratch.The revised design used writer-selected snippets to guide paragraph-level generation.Source: §5 · p. 9

Results

  • System generations received a 0.43 user rating; writers made 87 token edits per paragraph and extended generation lengths by 65% on average.Rouge-1, Rouge-2, Rouge-L, and BERT-Scores were 45.6, 30.0, 35.4, and 38.0, respectively.Source: §5 · pp. 9–10
  • Follow-up interviews found that erroneous or missing key fields continued to prevent generations from being useful.The system let writers select source snippets and generated each paragraph separately from those selections.Source: §5 · pp. 9–10

6 Conclusion

Multi-LexSum provides expert-authored summaries of federal civil-rights lawsuits, but current models struggle to produce them, and the dataset’s case selection limits generalization. The authors hope it will support broader access to up-to-date case descriptions.

6 Conclusion

  • Current models averaged 0.43 on a 0–3 human-assessment scale; plaintiff-win and online-docket selection may limit generalization.The dataset covers federal cases with online dockets, and the authors note that under-represented cases include those where defendants win.Source: §6 · p. 10
  • The authors hope wider availability of up-to-date case descriptions will benefit reporters, advocates, and the public.They identify an even greater potential benefit for projects covering hundreds of thousands of lawsuits.Source: §6 · p. 10

A.1 Accessing Multi-LexSum

  • Multi-LexSum’s public loader downloads source files and loads them into an easily usable format.The files are stored in JSON format and hosted on Amazon S3.Source: §A.1 · p. 16

A.2 Multi-LexSum distribution and maintenance

  • The authors commit to maintaining the dataset, migrating its files to common repositories if they can no longer host them.They also plan to monitor usage and provide needed bug fixes.Source: §A.2 · p. 16

B.1 Reading source documents

  • Writers skim the docket and select a small subset of important entries, typically a dozen or fewer documents, for deeper reading.The docket lists filed documents chronologically with dates, titles, brief descriptions, and often links.Source: §B.1 · p. 17, Table 6 · p. 17

B.2 Writing summaries

  • Summaries generally follow the docket’s event order, narrating key developments and, when relevant, important document contents.The narrative progresses through the most important docket entries.Source: §B.2 · p. 17

B.3 Reviewing summaries

  • Reviewers check factual accuracy and improve clarity, style, legal terminology, and accessibility for non-lawyer readers.Guidelines include past tense, concise writing, and avoiding legal jargon.Source: §B.3 · pp. 17–19

C Usability study system design

The usability-study system breaks case summarization into stages that use human-selected documents and snippets to support model-generated drafts. Despite these aids, author-reported risks include hallucinations and dataset coverage that may not represent other cases or legal systems.

C Usability study system design

  • The system guides users through docket-entry selection, event grouping, source-text extraction, and summary selection, rating, and editing.Writers can select a preferred draft from BART and DistilBART.Source: C Usability study system design · pp. 19–20
  • Human-selected documents and text snippets improved generation quality, yet model outputs received a mean rating of 0.43.The system reduces generation and editing burden by producing one paragraph at a time; study participants verified the higher quality relative to end-to-end systems.Source: C Usability study system design · pp. 19–20

D Negative social impact

  • If deployed at scale before hallucinations are solved, automatic court summaries could mislead readers and undermine trust.The authors warn that factual errors may pose these risks even when summarization is used sparingly.Source: D Negative social impact · p. 21
  • Because the dataset draws from a non-representative subset of U.S. civil-rights cases, models may adapt to its cases and perform unfairly elsewhere.The authors identify other case types and legal systems as settings where this could be problematic.Source: D Negative social impact · p. 21

E Multi-LexSum train-test split

The authors use a random case split and evaluate split size and stability to support their train-dev-test design. Larger test splits produced more stable results, while cross-validation showed low sensitivity to random splits.

E Multi-LexSum train-test split

  • Cases were randomly assigned to 70% train, 10% development, and 20% test splits.The authors checked test-set size and sensitivity to the particular split.Source: E Multi-LexSum train-test split · p. 21
  • A 20% test split produced more stable results than smaller test splits, while retaining 70% of the data for training.The authors compared 10%, 15%, and 20% test splits using the same model trained on 70% of the data.Source: E Multi-LexSum train-test split · p. 22
  • Five-fold cross-validation showed low standard deviation, indicating that models were not sensitive to specific random splits.Training data was capped at 70%; hyperparameters differed slightly from the main results.Source: E Multi-LexSum train-test split · p. 22

F Multi-LexSum datasheet

Multi-LexSum was created as a benchmark for long civil-rights lawsuit documents, offering expert-written summaries at multiple levels of detail. Its documentation describes the dataset’s purpose, access, and provenance.

F Multi-LexSum datasheet

  • The dataset sheet follows a published template, and the dataset is available through the project website and GitHub repository.The construction was partly funded by NSF Convergence Accelerator Award ITE-2132318.Source: F Multi-LexSum datasheet · p. 23, MOTIVATION · p. 24

MOTIVATION

  • Multi-LexSum targets lawsuit inputs that can average up to 70,000 words and provides up to three summary granularities for each case.The authors present it as a benchmark for long-document summarization and note that expert legal writers create and review its summaries.Source: MOTIVATION · p. 23
  • The dataset was created through collaboration between the Civil Rights Litigation Clearinghouse and Allen Institute for AI.It builds on Clearinghouse data used to inform the public about civil-rights litigation.Source: MOTIVATION · p. 23

What do the instances that comprise the dataset represent (e.g., documents, photos, people,

Multi-LexSum instances pair public federal-court case documents with expert-written summaries and metadata, but the dataset is a selected, non-representative sample of civil-rights lawsuits. Coverage varies across summary granularities, and some dockets or extracted text may be incomplete or noisy.

What do the instances that comprise the dataset represent (e.g., documents, photos, people,

How many instances are there in total (of each type, if appropriate)?

  • The dataset contains 4,539 lawsuits, 40,119 source documents, and 9,280 summaries; some cases lack short or tiny summaries, largely due to when they were summarized.Every instance has a long summary, but some do not have short summaries, tiny summaries, or either.Source: How many instances are there in total… · pp. 25–26
  • The sample covers selected federal cases with computerized dockets and CRLC topic criteria, and overrepresents plaintiff wins, limiting generalization to under-represented cases.The authors caution that performance might not generalize to under-represented cases, such as cases where the defendant wins.Source: How many instances are there in total… · p. 25

PREPROCESSING/CLEANING/LABELING

Are relationships between individual instances made explicit (e.g., users' movie ratings, social

  • Cases may disclose sensitive details and identify people, including plaintiffs and lawyers named in court documents.Examples include information about discrimination, criminal history, or health-related issues in relevant cases.Source: Are relationships between individual… · p. 28

Any other comments?

  • Dockets are incomplete for 118 cases, and scanned PDFs of earlier, non-digitized dockets in a small number of cases are not yet tagged.Source: Any other comments? · p. 28

COLLECTION PROCESS

  • Legal experts manually curated the cases and wrote summaries from documents obtained through the CRLC, following summary-writing guidelines.The CRLC obtained the documents from federal courts and assigned legal scholars, lawyers, and law students to write summaries.Source: COLLECTION PROCESS · pp. 29–30

USE

  • The dataset supports training models to summarize cases from source documents, control summary granularity, and condense long summaries into short ones.Source: USE · p. 33

What (other) tasks could the dataset be used for?

Multi-LexSum may support legal-text pretraining and information extraction, but its selection bias limits some uses. The dataset is publicly downloadable, with access materials provided by the authors.

What (other) tasks could the dataset be used for?

  • Potential additional uses include pretraining legal-document-understanding models and extracting case metadata, such as causes of action and outcomes, from source documents.Source: What (other) tasks could the dataset… · p. 33
  • The dataset is a non-representative sample, and the authors caution that performance may not generalize to under-represented cases, such as defendant-win cases.They state that it should not be used to train models to predict lawsuit outcomes.Source: What (other) tasks could the dataset… · pp. 33–34

DISTRIBUTION

  • The dataset is publicly downloadable from an AWS S3 bucket, and the authors provide a Python loading script.Source: DISTRIBUTION · p. 35

When will the dataset be distributed?

The dataset has been available since June 16, 2022. Its components have different licensing terms.

When will the dataset be distributed?

Do any export controls or other regulatory restrictions apply to the dataset or to individual

The dataset's support and correction information is described, and the authors envision future additions and fixes without a concrete update plan. Older versions will remain available.

Who will be supporting/hosting/maintaining the dataset?

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete

Related papers

Loading 2206.10883v3…