PaperlayerView exact revision on arXiv

Source-linked AI summary

BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization

Eva Sharma, Chen Li, Lu Wang

arXiv:1906.03741v1cs.CLcs.LG

TL;DR

Existing summarization datasets provide limited support for modeling global structure because news summaries often flatten discourse, reuse input text, and concentrate salient content at the beginning. BIGPATENT addresses this gap with 1.3 million U.S. patent documents and human-written abstracts, yielding more abstractive summaries with richer entity coherence and more evenly distributed salient content. The dataset also exposes challenges for current models, including irrelevant repetition and fabricated information.

  • Problem

    Most news-based summarization datasets flatten discourse, reuse long input fragments, and concentrate salient content near article beginnings.

  • Method

    BIGPATENT pairs 1.3 million U.S. patent descriptions with their human-written abstracts as summaries and evaluates baselines and popular summarization models.

  • Results

    BIGPATENT summaries have richer discourse, more evenly distributed salient content, and fewer shorter extractive fragments, while current models often repeat irrelevant entities and fabricate information.

  • Takeaways & Limitations

    BIGPATENT supports research on global content modeling, entity and relation understanding, and discourse-aware planning for abstractive coherent summarization.

  • Takeaways & Limitations

    The BIGPATENT task differs from traditional patent summarization, which summarizes claims into a more readable format.

Abstract

from arXiv · show

Most existing text summarization datasets are compiled from the news domain, where summaries have a flattened discourse structure. In such datasets, summary-worthy content often appears in the beginning of input articles. Moreover, large segments from input articles are present verbatim in their respective summaries. These issues impede the learning and evaluation of systems that can understand an article's global content structure as well as produce abstractive summaries with high compression ratio. In this work, we present a novel dataset, BIGPATENT, consisting of 1.3 million records of U.S. patent documents along with human written abstractive summaries. Compared to existing summarization datasets, BIGPATENT has the following properties: i) summaries contain a richer discourse structure with more recurring entities, ii) salient content is evenly distributed in the input, and iii) lesser and shorter extractive fragments are present in the summaries. Finally, we train and evaluate baselines and popular learning models on BIGPATENT to shed light on new challenges and motivate future directions for summarization research.

1 Introduction

Existing news-based summarization datasets often flatten discourse, reuse long input fragments, and concentrate salient content near article beginnings. BIGPATENT introduces a large-scale patent dataset designed to support more abstractive summaries and global content understanding.

  • News-based datasets often contain flattened narratives, rare entity repetition, and long fragments directly extracted from inputs.
  • BIGPATENT contains 1.3 million patent documents paired with human-written abstractive summaries.
  • Its summaries feature recurring entities, evenly distributed salient content, and fewer and shorter reused phrases.
  • Benchmarking shows lower ROUGE scores on BIGPATENT than on existing news datasets, while generated summaries often repeat irrelevant entities and fabricate information.
  • The dataset directs research toward global content modeling, semantic understanding of entities and relations, and discourse-aware text planning.

2 Related Work

Prior summarization datasets provide either structurally limited news examples or smaller scientific-document collections. These limitations motivate datasets that better support global content understanding and abstractive summarization.

  • Existing abstractive models can generate fluent summaries but often produce fabricated and repeated content.
  • Existing content-selection models rely on positional information and can be fooled by adversarial input content.
  • News summaries have nonuniform salient-content distributions, while arXiv and PubMed are smaller and have longer, more extractive summaries.
  • BIGPATENT is proposed to guide research toward more abstractive summarization with global content understanding.

3 BIGPATENT Dataset

BIGPATENT uses U.S. patent descriptions as inputs and their abstracts as human-written summaries, providing a large corpus for studying highly compressed and abstractive summarization. Dataset statistics compare its size, compression, and extractive-fragment properties with established corpora.

  • BIGPATENT contains 1.3 million U.S. patent documents collected from Google Patents Public Datasets across nine technological areas.
  • Each patent description serves as input, while its abstract serves as the gold-standard summary.
  • The dataset includes patents filed after 1971 and provides preprocessing details in Appendix A.1.
  • Table 1 reports compression ratio and extractive fragment density for BIGPATENT and commonly used summarization corpora.
  • The BIGPATENT task summarizes patent abstracts rather than converting patent claims into more readable text.
  • BIGPATENT is significantly larger and has longer inputs and summaries than the compared datasets.

4 Dataset Characterization

BIGPATENT differs from news and other summarization corpora through more globally distributed salient content, greater abstractiveness, and richer entity-based discourse structure. These properties create challenges for models that must cover long inputs while maintaining coherent entity relations.

  • Salient Content Distribution: BIGPATENT distributes salient content more evenly across document segments than news datasets, where roughly 50% appears in the first segment.Only 6% more salient words occur in BIGPATENT’s first segment than in the others.
  • Salient Content Distribution: 80% of BIGPATENT’s input is needed to cover summary salient words, compared with 29–63% for the evaluated news datasets.The result indicates that effective BIGPATENT content selection requires reading broadly across the input.
  • Summary Abstractiveness: BIGPATENT has the second-highest percentage of novel 2-, 3-, and 4-grams, indicating fewer and shorter extractive fragments than most datasets.XSum is the exception among the compared datasets, while BIGPATENT also has the lowest extractive fragment density in Table 1.
  • Entity Distribution and Coherence: BIGPATENT summaries contain 18.5 unique entities on average, compared with 6.7 in Newsroom, 10.9 in NYT, and 12.4 in CNN/DM.PUBMED and ARXIV have more unique entities, but their summaries are considerably longer.
  • Entity Distribution and Coherence: 11.1% of BIGPATENT entities appear in two consecutive sentences, indicating longer entity chains and greater sentence-to-sentence relatedness than news summaries.The analysis uses entity-grid representations and coreference rules to characterize summary discourse structure.
  • Entity Distribution and Coherence: BIGPATENT entities recur in later sentences at an average frequency of 2.3, versus 0.4 for CNN/DM, suggesting combined local and global coherence.PUBMED and ARXIV show higher recurrence counts but different patterns, with recurrence often delayed by three sentences.

5 Experiments and Analyses

BIGPATENT is evaluated with extractive and abstractive baselines against established news datasets, revealing challenges from its longer-range content distribution and discourse structure. Models achieve weaker or flawed behavior on these dimensions, although SENTREWRITING performs best on BIGPATENT.

  • Evaluation setup: BIGPATENT is evaluated using LEAD-3, two oracle systems, unsupervised extractive models, and neural abstractive summarization systems.The evaluation compares popular systems with established datasets including CNN/DM and NYT.
  • ROUGE results: Almost all models outperform LEAD-3 on BIGPATENT, while TEXTRANK and LEXRANK outperform RNN-EXT RL trained only on the first 400 words.The result is attributed to BIGPATENT’s more uniform salient-content distribution and indicates that neural models must handle longer inputs efficiently.
  • ROUGE results: SENTREWRITING achieves the best performance on BIGPATENT.SENTREWRITING is a reinforcement learning model using ROUGE as its reward.
  • Generation quality: Generated summaries have novel unigram and bigram content comparable to GOLD but still contain fabricated or irrelevant repetitions.The cited example repeats related phrases about an upper portion and a sole portion, indicating inadequate semantic control.
  • Entity distribution: Neural abstractive models generally repeat entities more often than humans, whereas POINTGEN+COV produces significantly fewer entity repetitions.Entities mentioned at least three times occur in 6.7% and 22.6% of SEQ2SEQ outputs versus 5.2% and 4.0% for GOLD.

6 Conclusion

The conclusion presents BIGPATENT as a large-scale dataset with abstractive, structurally richer summaries and broadly distributed salient content. It positions the dataset as a basis for developing more robust abstractive and coherent summarization systems.

  • Dataset contribution: BIGPATENT contains human-written abstractive summaries with fewer and shorter extractive phrases and richer discourse structure than existing datasets.Its salient summary content is also more evenly distributed throughout the input.
  • Research direction: BIGPATENT can enable future research on robust systems that generate abstractive and coherent summaries.

A.1 Dataset Details

The appendix describes BIGPATENT’s construction from filtered English USPTO patent records, retaining patent metadata and text fields while using abstracts as gold summaries. It also analyzes salient-content distribution across the input.

  • Dataset construction: BIGPATENT contains 1.3 million U.S. patent documents collected from Google Patents Public Datasets.The collection is restricted to English USPTO documents filed after 1971 for more consistent writing and formatting.
  • Dataset construction: Each patent record is associated with a hierarchical Cooperative Patent Classification code spanning nine technology categories.The nine categories are summarized statistically in Table 6.
  • Dataset construction: The retained fields include title, authors, abstract, invention claims, and description text, with the patent abstract serving as the gold-standard summary.Description text includes background material, figure descriptions, and detailed invention content.
  • Preprocessing: Records are filtered by compression ratio, summary length, input length, and the extractive overlap of summary-worthy unigrams.The stated bounds retain summaries of 10–2,500 words and inputs of 150–80,000 words, while emphasizing abstractive pairs.
  • Salient content distribution: BIGPATENT’s salient content is relatively uniformly distributed across all parts of the input.Salience is measured using summary bigrams and longest common subsequences, corresponding to Figures 4 and 5.

A.2 Experiment details

The experiments use fixed train, validation, and test splits and implement extractive and abstractive baselines with specified preprocessing and model settings. Neural abstractive systems generally receive truncated inputs and outputs, while SENTREWRITING uses the authors’ implementation.

  • Data splits: BIGPATENT is randomly split into 1,207,222 training, 67,068 validation, and 67,072 test pairs.CNN/DM and NYT are also evaluated using separately specified preprocessing and splits.
  • Extractive systems: Extractive baselines include TEXTRANK, LEXRANK, SUMBASIC, and RNN-EXT RL using established implementations.TEXTRANK generates three-sentence summaries, while RNN-EXT RL uses the authors’ implementation.
  • Abstractive systems: Except for SENTREWRITING, neural abstractive models truncate inputs to 400 words and outputs to 100 words.These models are trained with OpenNMT-py following the authors’ instructions.
  • Abstractive systems: SEQ2SEQ with attention uses a 128-dimensional word embedding, a 512-dimensional one-layer LSTM, a bidirectional encoder, and beam size 5.POINTGEN and POINTGEN+COV use the same training settings, with POINTGEN+COV adding coverage and length penalties.
  • Abstractive systems: SENTREWRITING is trained with the authors’ implementation using its full reinforcement-learning model and default parameters.

A.3 Summaries for sample Input Document from BIGPATENT

Table 7 presents a sample BIGPATENT input alongside its gold-standard summary and summaries generated by three systems. The input is truncated to its first 400 words for brevity.

  • Table 7 compares a gold-standard summary with outputs from SEQ2SEQ, POINTGEN + COV, and SENTREWRITING.
  • The displayed sample uses a pre-processed BIGPATENT input document.
  • The input is truncated to 400 words in the table for brevity.
Loading 1906.03741v1…