Source-linked AI summary
BillSum: A Corpus for Automatic Summarization of US Legislation
Anastassia Kornilova, Vlad Eidelman
TL;DR
Legislative summarization lacked a dedicated corpus despite the scale and complexity of US bills. BillSum introduces such a corpus, benchmarks extractive models using document context and summary-like language, and evaluates transfer from Congressional to California bills. The models outperform unsupervised baselines on California bills, while the analysis identifies summary-like language patterns.
Problem
Legislation had not previously been studied for automatic summarization despite thousands of long, technical bills and limited availability of summaries outside Congress.
Method
The paper builds BillSum and benchmarks extractive sentence classifiers using document-context features, BERT-based summary-language modeling, ensemble scoring, and MMR selection.
Results
Models trained on US bills transfer to California bills, where all three supervised methods outperform unsupervised baselines; summary-like words tend to be verbs or department names.
Takeaways & Limitations
BillSum supports legislative summarization research and indicates that models developed on Congressional bills can be applied to legislatures lacking human-written summaries.
Takeaways & Limitations
Abstractive models trained on news and Wikipedia produced ungrammatical results, and domain-specific abstractive strategies were left for future work.
Abstract
from arXiv · showhide
Automatic summarization methods have been studied on a variety of domains, including news and scientific articles. Yet, legislation has not previously been considered for this task, despite US Congress and state governments releasing tens of thousands of bills every year. In this paper, we introduce BillSum, the first dataset for summarization of US Congressional and California state bills (https://github.com/FiscalNote/BillSum). We explain the properties of the dataset that make it more challenging to process than other domains. Then, we benchmark extractive methods that consider neural sentence representations and traditional contextual features. Finally, we demonstrate that models built on Congressional bills can be used to summarize California bills, thus, showing that methods developed on this dataset can transfer to states without human-written summaries.
1 Introduction
BillSum addresses the need to process large volumes of long, technical legislation by introducing a corpus for legislative summarization and benchmarking automatic methods.
- US Congress introduces over 10,000 bills annually, while state legislatures introduce tens of thousands more.Bills are long and technical, making key details difficult to identify quickly.
- BillSum contains 22,218 US Congressional bills with reference summaries, plus 1,237 California bills for testing.The US corpus is split into training and test sets, while California evaluates transfer to another legislature.
- The dataset establishes benchmarks and targets methods better suited to technical legislative language.
2 Background
Prior summarization research covered several domains but had not produced a corpus specifically designed for legislative text, motivating BillSum's legislative focus.
- Summarization research has covered news, emails, scientific papers, and court proceedings, but not legislative text.
- Earlier legislative-text studies treated bills largely as bags of words rather than modeling the importance of individual sentences.
- BillSum is presented as the first corpus specifically designed for summarization of legislation.
- State-government documents had been used for parsing and textual similarity, but not previously as a dedicated legislative summarization corpus.
3 Data
BillSum combines Congressional training and test bills with California test bills, focusing on mid-length, technically structured legislation and human-written summaries.
- BillSum contains 18,949 Congressional training bills, 3,269 Congressional test bills, and California bills from the 2015–2016 session.Congressional bills span the 103rd–115th sessions, while California summaries were written by the Legislative Counsel.
- The corpus focuses on bills between 5,000 and 20,000 characters because shorter and longer legislation creates different summarization conditions.Short bills may require little compression, whereas very long bills may contain several large sections.
- Summaries are capped at 2,000 characters because 90% are at or below that length.
- Duplicate bills are removed using a 96% cosine-similarity threshold, with remaining duplicate titles confined to training.
- Most summaries range from 1,000 to 2,000 characters, with little correlation between bill length and human summary length.
- Bills commonly use nested bullets and may contain technical edits to existing laws whose context is difficult to interpret.The dataset therefore combines straightforward descriptions of proposed actions with more context-dependent legislative language.
4 Benchmark Methods
The benchmarks score and select extractive sentences using document-context and summary-language signals, combine these models, and apply MMR to produce concise summaries.
- 4 Benchmark Methods: Extractive benchmarks score individual sentences and select a best subset for the summary.
- 4 Benchmark Methods: Abstractive models trained on news and Wikipedia produced ungrammatical results, so abstractive domain adaptation is left for future work.
- 4 Benchmark Methods: Sentence labels are binary, based on whether sentence-level Rouge-2 Precision exceeds 0.1 against the reference summary.
- 4.1 Document Context Model (DOC): The DOC models use contextual features such as sentence position, section-header proximity, and TF-IDF statistics.Bullets shorter than 10 words are joined to the preceding sentence; longer bullets are treated as full sentences.
- 4.2 Summary Language Model (SUM): The SUM model uses BERT to classify sentences according to summary-like language, emphasizing major actions over administrative edits.The model is pretrained on the US training corpus before sentence classification.
- 4.3 Ensemble and Sentence Selection: The ensemble averages DOC and SUM scores, then MMR balances sentence scores against cosine similarity to reduce redundancy until 2,000 characters.MMR uses experimentally chosen weights of 0.7 for score and 0.3 for similarity.
5 Results
The evaluated extractive classifiers outperform unsupervised baselines, while US-trained models transfer to California bills with lower overall performance. Summary-language analysis identifies lexical signals associated with summaries and text.
- Benchmark results: All three supervised classifiers outperform unsupervised baselines, indicating summary-like language signals across bills.
- Benchmark results: The SUM model outperforms DOC, while DOC+SUM performs similarly to DOC.
- Cross-state transfer: US-trained supervised methods outperform unsupervised baselines on California bills, although performance is lower than on US bills.
- Cross-state transfer: The California SUM and DOC models perform similarly, possibly because BERT overfit US bill language and California summaries include existing-law content.
- Summary language analysis: A large KL(S|T) value marks summary-like words, while KL(T|S) marks text-like words.
- Summary language analysis: Summary-like words tend to be verbs or department names, whereas text-like words refer mainly to edits or background content.
6 Conclusion
BillSum is introduced as the first corpus for legislative summarization, addressing technically challenging and structurally complex bills. Baselines reveal substantial room for improvement, while US-trained methods transfer to California bills.
- BillSum is the first corpus for legislative summarization and contains technically complex, structurally challenging bills.
- Baseline results show a large performance gap relative to the oracle, indicating ample room for further development.
- Methods trained on US bills transfer to California bills, supporting use in legislatures without human-written summaries.
A Duplicate Removal Procedure
The duplicate-removal procedure combines text vectorization, word filtering, cosine similarity, and a 96% similarity threshold. Remaining identical-title bills are assigned to the training partition.
- Near-duplicate bills arise from companion bills and reintroductions across legislative sessions.
- Bills are vectorized with CountVectorizer, excluding the top 15% most common words and generic stop words.
- Cosine similarity between bill texts and summaries is computed for each pair and averaged.
- Examples exceeding 96% similarity to any previously added bill are skipped.
- Remaining bills with identical titles are placed in the train partition, because titles may be generic or modified reintroductions.
B Additional ROUGE Scores
The paper reports additional ROUGE analyses for Congressional and California bills, separating precision and recall to assess comprehensiveness and conciseness. The methods generally favor recall because they generate fixed-length summaries.
- F-Scores balance comprehensiveness and conciseness, while Tables 4 and 5 separately report precision and recall for US and California bills.
- All tested methods favor recall because they consistently generate 2000-character summaries rather than stopping when a shorter summary may suffice.
- The Oracle and DOC+SUM summarizer differ less in Recall than in precision for both datasets, indicating that extractive methods capture substantial useful summary content.
C.1 Complex Structure Example
A complex Congressional bill example shows why legislative summarization must preserve nested structure and contextual relationships. Relevant details may be concentrated in specific line ranges, yet parsing the original organization remains nontrivial.
- Complex formatting: Nested bullet layers must be handled carefully because extracted snippets can become uninterpretable without their surrounding context.
- Complex formatting: Lines 22–26 simultaneously introduce new legal language and use a bulleted structure.
- Complex formatting: Line 27 removes and replaces existing subsection (f), while the subsequent text does not clearly show the change relative to current law.
- Human-written summary: The human-written summary begins by defining a women’s business center through several eligible entity categories.
- Complex formatting: The example concentrates most relevant details in lines 8–14 and 20–24, whose summary language is extracted almost directly from the bill.
C.2 Paraphrase Example
Some bills receive paraphrased technical summaries rather than summaries that reproduce source text. An Endangered Species Act example illustrates why purely extractive methods cannot always construct the required summary.
- For bills whose technical language is paraphrased by the CRS, extractive summarization methods are particularly limited.
- The example bill revises how the Interior or Commerce Department reviews petitions to list species as endangered or threatened and establishes a backlog process.
- Because the bill elaborates the process while the summary states that one was created, a purely extractive method would struggle to construct the summary.
C.3 California Example
California bills generally resemble US bills, but some California summaries use a different format that adds context about existing law. The paper identifies this context as useful for future summary generation.
- California bills follow the same general patterns as US bills, but some summaries use a different format.
- In the illustrated format, the summary first explains existing law and then explains the proposed change.
- The paper identifies references to existing law as useful additional context for creating better California summaries in future work.