Source-linked AI summary
TLDR: Extreme Summarization of Scientific Documents
Isabel Cachola, Kyle Lo, Arman Cohan, Daniel S. Weld
TL;DR
Scientific TLDR generation compresses entire papers into faithful, single-sentence summaries, but requires domain knowledge and has limited specialized training data. The paper introduces SciTLDR, a multi-target dataset, and CATTS, which uses titles as an auxiliary signal; CATTS improves over strong modeling baselines in automated metrics and human evaluation. The dataset’s experiments are limited to abstract-only and AIC input spaces, while discipline-specific notions of TLDRs remain open.
Problem
Scientific TLDR generation requires extreme compression, domain knowledge, and complex-language understanding to produce faithful and correct summaries.
Method
The paper releases SciTLDR and trains CATTS to generate TLDRs with titles as an auxiliary, control-coded training target.
Results
CATTS improves over strong modeling baselines under both automated metrics and human evaluation.
Takeaways & Limitations
SciTLDR provides multiple author-written and expert-derived targets for studying TLDR generation, training, and evaluation.
Takeaways & Limitations
Experiments use only abstract-only and AIC input spaces, and TLDR meanings across academic disciplines remain unexplored.
Abstract
from arXiv · showhide
We introduce TLDR generation, a new form of extreme summarization, for scientific papers. TLDR generation involves high source compression and requires expert background knowledge and understanding of complex domain-specific language. To facilitate study on this task, we introduce SciTLDR, a new multi-target dataset of 5.4K TLDRs over 3.2K papers. SciTLDR contains both author-written and expert-derived TLDRs, where the latter are collected using a novel annotation protocol that produces high-quality summaries while minimizing annotation burden. We propose CATTS, a simple yet effective learning strategy for generating TLDRs that exploits titles as an auxiliary training signal. CATTS improves upon strong baselines under both automated metrics and human evaluations. Data and code are publicly available at https://github.com/allenai/scitldr.
1 Introduction
The paper introduces TLDR generation as extreme, single-sentence summarization for scientific papers, where producing faithful and correct summaries requires domain expertise. It releases SciTLDR and proposes CATTS, while analyzing human-written TLDR variability and evaluating generated summaries.
- 1 Introduction: TLDR generation produces extreme, single-sentence summaries of entire scientific papers while preserving salient information and factual correctness.Unlike abstracts, TLDRs emphasize key contributions and omit nonessential background or methodological details.
- 1 Introduction: CATTS improves TLDR generation by using paper titles as an auxiliary training signal alongside strong Transformer-based baselines.The strategy trains models to generate titles and TLDRs using control codes.
- 1 Introduction: Author-written and peer-reviewer-derived TLDRs reveal substantial variability in human summaries.The analysis examines differences in information content and lexical realization across the two sources.
- 1 Introduction: 5,411 TLDRs over 3,229 papers form SciTLDR, a multi-target dataset combining author-written and expert-derived summaries.Expert-derived TLDRs are collected through a protocol that avoids requiring annotators to read the full paper.
- 1 Introduction: Human evaluation analyzes system-generated TLDRs for informativeness and factual correctness.The study complements automated evaluation with extensive qualitative and human assessment.
2 Dataset construction
SCITLDR is a multi-target dataset combining author-written and peer-reviewer-derived TLDRs for scientific papers. Its annotation protocol uses peer-review summaries to reduce the burden of reading full papers.
- SCITLDR contains 5,411 TLDRs over 3,229 computer science papers, with author-written and expert-derived summaries.
- The training set has 1,992 papers with one gold TLDR, while development and test sets contain 619 and 618 papers with 1,452 and 1,967 TLDRs.
- The dataset includes multiple gold summaries because human TLDRs vary, making single-reference automated evaluation potentially inaccurate.
- Peer-reviewer TLDRs are created by asking domain experts to rewrite faithful summaries from peer-review comments rather than read entire papers.
- Annotators were trained to produce 15–25-word TLDRs, preserve review wording when possible, omit excess details or criticism, and skip unclear reviews.
3 Dataset analysis
SCITLDR combines extreme compression with substantial abstraction and supports analysis of the information and linguistic variation in author- and reviewer-written TLDRs. The two TLDR perspectives often cover related content with very different wording and abstraction levels.
- 3.1 Compression and abstractiveness: SCITLDR has short summaries and long source documents, producing a much higher compression ratio than existing datasets.
- 3.1 Compression and abstractiveness: SCITLDR is more abstractive than other scientific-domain datasets but less abstractive than non-scientific-domain datasets.
- 3.2 Information content: TLDR-Auth and TLDR-PR typically contain two to four nuggets, with subject area and contributions appearing together in 63% and 71% of summaries, respectively.
- 3.3 Variability in TLDRs: TLDR-Auth average 18.9 words and TLDR-PR average 22.9 words, yet their 1-, 2-, and 3-gram mean Jaccard indices are only 15.0%, 2.5%, and 0.7%.
- 3.3 Variability in TLDRs: TLDR-PR has a 20.2% novelty score versus 9.6% for TLDR-Auth because reviewer summaries derive from already abstracted peer-review comments.
4 CATTS
CATTS addresses limited training data and the need for domain knowledge by training a model to generate both TLDRs and paper titles. Titles serve as an auxiliary scaffold for locating salient information.
- CATTS, or Controlled Abstraction for TLDRs with Title Scaffolding, is a learning strategy for generating scientific-paper TLDRs.
- CATTS uses scientific-paper titles as additional generation targets because titles often contain key information about the paper.
- The method combines scaffold tasks from multitask learning with control codes for conditional generation.
- Training shuffles SCITLDR with a title-generation dataset and appends ⟨|TLDR|⟩ or ⟨|TITLE|⟩ control codes to select the desired output.
- At generation time, the appropriate control code is appended to the source, while task up-sampling provides task-specific weighting.
5 Experiments
The experiments evaluate extractive and abstractive TLDR generation on SCITLDR using abstract-only and AIC input spaces, automated Rouge metrics, and human assessments of informativeness and correctness.
- Baselines: SCITLDR baselines include PACSUM, BERTSUMEXT, MatchSum, BART, and BARTXSUM for extractive and abstractive TLDR generation.
- Oracle: The study also reports extractive sentence-level oracle scores to estimate the upper bound for sentence-selection methods.The oracle selects the document sentence with the highest Rouge overlap for each gold TLDR and returns the best resulting sentence.
- Input space: The experiments compare abstract-only input with AIC context consisting of the abstract, introduction, and conclusion.AIC is used to reduce computational costs while retaining salient paper information.
- Automated evaluation: Rouge-1, Rouge-2, and Rouge-L are computed against each paper’s multiple gold TLDRs, using the maximum score across targets.The maximum operation rewards matching any available gold TLDR.
- Human evaluation: Human experts assess generated TLDRs for informativeness and correctness in addition to automated metrics.Informativeness uses length-normalized nugget counts, while original authors evaluate correctness.
6 Results
CATTS improves abstractive baselines across both input settings, while input-space effects differ across extractive models and human evaluation shows higher informativeness without a correctness difference.
- Extractive results: MatchSum has the highest extractive performance, while expanding input to AIC improves PACSUM but decreases BERTSUMEXT and MatchSum performance.The authors attribute the latter changes to increased difficulty learning model parameters with longer inputs.
- Abstractive results: +0.5 and +1.8 Rouge-1 are achieved by CATTS and CATTSXSUM over their BART baselines with abstract-only input.
- Abstractive results: +2.0 and +0.9 Rouge-1 are achieved by CATTS and CATTSXSUM over their BART baselines with AIC input.The comparisons use paired tests with Holm-Bonferroni correction for significance assessment.
- Human evaluation: CATTSXSUM is more informative than BARTXSUM and comparable to author-written gold TLDRs, but less informative than peer-review-derived TLDRs.The human comparison uses the AIC input space on 51 sampled papers.
- Human evaluation: BARTXSUM and CATTSXSUM show no correctness difference, with 42 ties and both models averaging 2.5 between partially accurate and mostly correct.
- Generation analysis: BART variants are less abstractive than CATTS variants, while CATTS maintains similar abstractiveness across abstract-only and AIC settings.
- Generation analysis: All systems generate TLDRs with lengths similar to the reported average TLDR length.
7 Related work
Prior scientific summarization work targets longer summaries or specialized settings, whereas SCITLDR introduces a dataset for extreme scientific summarization with short, highly compressed, abstracted outputs.
- Transformer summarization: Transformer-based models provide strong extractive and abstractive summarization foundations, including PACSUM, MatchSum, BERT-based systems, and sequence-to-sequence models.
- Scientific document summarization: Most scientific-paper summarization datasets target 150–200-word summaries, abstracts, extractive outputs, citation contexts, or related specialized formats.
- Scientific document summarization: Existing methods exploit citation contexts, survey generation, or structural properties such as document length and organization.
- Research gap: These methods had not been studied for extreme summarization involving short targets, high compression, and high abstraction.
- Contribution: SCITLDR is presented as the first dataset facilitating extreme summarization research focused on scientific papers.
8 Conclusion
The paper presents SCITLDR and CATTS for scientific TLDR generation, using paper titles as auxiliary training signals. It also identifies future directions involving scientific-document properties, longer inputs, multiple targets, and cross-disciplinary variation.
- SCITLDR is a multi-target dataset of TLDR-paper pairs, supporting summarization research with multiple summaries per paper.
- CATTS improves TLDR generation by exploiting auxiliary training signals from paper titles.
- Existing scientific summarization properties, including sections, citation contexts, and discourse roles, remain opportunities for improving TLDR-generation models.
- The dataset supports longer-input research and training or evaluation with diverse target summaries, while TLDR conventions may differ across academic disciplines.
A How many nuggets in TLDRs?
The supplied passages frame TLDR content as subjective and domain-dependent, while the referenced tables concern category and venue coverage. They also assume readers have sufficient background knowledge within the relevant scientific domain.
- Table 8 reports the number of categories represented in a TLDR, while Table 9 reports the venue breakdown of represented papers.
- TLDR content is subjective and follows community-specific commonsense rather than a formally defined procedure.
- Readers are assumed to have sufficient background knowledge to follow a general research topic in their scientific domain.
D Additional model training details
The additional training details specify model-specific optimization settings, checkpoint selection, tuning procedures, and training durations for baselines and CATTS.
- PACSUM: PACSUM used beta and lambda1 set to 0 after tuning produced no significant performance difference.Runtime was 12 minutes on abstracts and 6.5 hours on AIC.
- BERTSUMEXT: BERTSUMEXT used a batch size of one sentence, 5,000 training steps, learning rate 2e-3, and dropout 0.1.
- MatchSum: MatchSum used 15 epochs, selected checkpoints using a linear combination of Rouge-1, Rouge-2, and Rouge-L, and received manual hyperparameter tuning.
- BART: BART models were trained for 500 steps with 20% warm-up, corresponding to approximately five epochs, after longer training overfit the training set.
- CATTS: CATTS used the BART baseline parameters without training-hyperparameter tuning, with 11,000 abstract-only steps and 45,000 AIC steps.Training took 2.5 hours in the abstract-only setting and 10 hours for AIC.
E Mean ROUGE test results
The supplied results caption defines a mean-ROUGE evaluation table for abstractive methods and identifies statistically significant CATTS improvements over corresponding BART baselines.
- Table 10 reports test-set results for abstractive methods using mean Rouge scores instead of maximum Rouge scores.
- A dagger marks CATTS variants that significantly outperform their corresponding BART baselines.
- The significance threshold for CATTS-versus-BART comparisons is p < 0.05.