Source-linked AI summary
CTRLsum: Towards Generic Controllable Text Summarization
Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, Caiming Xiong
TL;DR
CTRLsum addresses the mismatch between generic summaries and user-dependent information needs with a unified controllable summarization framework. It uses keywords or prompts at inference time across multiple control aspects and domains, while achieving strong results in both controlled and uncontrolled settings.
Problem
Existing abstractive summarization systems produce generic summaries, although users may prefer different information selected according to their interests.
Method
CTRLsum trains a summarization model conditioned on source documents and keywords, then uses keywords or prompts as control tokens to represent user preferences at inference time without extra annotations or predefined control aspects.
Results
Experiments across three domains and five control aspects demonstrate CTRLsum’s efficacy, including around 95% success for lead-3 and full-article entities and further zero-shot reading-comprehension gains of 15.6 and 17.9 points.
Takeaways & Limitations
CTRLsum provides flexible multi-aspect summary manipulation through textual controls, with experiments demonstrating efficacy across entity, length, contribution, invention-purpose, and question-guided summarization.
Takeaways & Limitations
Non-expert human judgments of summarization may be unreliable and correlate poorly with expert judgments.
Abstract
from arXiv · showhide
Current summarization systems yield generic summaries that are disconnected from users' preferences and expectations. To address this limitation, we present CTRLsum, a novel framework for controllable summarization. Our approach enables users to control multiple aspects of generated summaries by interacting with the summarization system through textual input in the form of a set of keywords or descriptive prompts. Using a single unified model, CTRLsum is able to achieve a broad scope of summary manipulation at inference time without requiring additional human annotations or pre-defining a set of control aspects during training. We quantitatively demonstrate the effectiveness of our approach on three domains of summarization datasets and five control aspects: 1) entity-centric and 2) length-controllable summarization, 3) contribution summarization on scientific papers, 4) invention purpose summarization on patent filings, and 5) question-guided summarization on news articles in a reading comprehension setting. Moreover, when used in a standard, uncontrolled summarization setting, CTRLsum achieves state-of-the-art results on the CNN/DailyMail dataset. Code and model checkpoints are available at https://github.com/salesforce/ctrl-sum
1 INTRODUCTION
CTRLsum addresses the mismatch between generic abstractive summaries and user-specific information needs by conditioning generation on textual control tokens. The framework supports broad summary manipulation without extra annotations or predefined control aspects, and evaluates five applications across three domains.
- Motivation and approach: The framework targets content selection that reflects user interests rather than only the generic information selected by the model.The introduction illustrates this with basketball fans seeking summaries centered on particular players and their scores.
- Motivation and approach: CTRLsum lets users manipulate abstractive summaries with keywords or descriptive prompts that convey their preferences.Keywords condition training and inference, while prompts can additionally constrain decoding at inference time.
- Motivation and approach: The model is trained with keywords identified from training summaries, requiring neither extra human annotations nor predefined control aspects.This design aims to support control aspects not specified during training.
- Evaluation scope: CTRLsum evaluates entity-centric, length-controllable, contribution, invention-purpose, and question-guided summarization across CNN/DailyMail, arXiv, and BIGPATENT.The experiments cover three domains and five control aspects.
2 CTRLSUM
CTRLsum represents user preferences as keywords and optional prompts, using them as a flexible interface for controlling summary content and length. The framework applies this interface to entities, length, scientific contributions, invention purposes, and question-guided summarization, while keywords and prompts have complementary strengths.
- Control-token framework: CTRLsum models controlled generation as p(y|x, z), where control tokens z represent user preferences alongside the source document x.Keywords are used during training and inference, while prompts may additionally constrain decoding at test time.
- Control-token framework: Users can combine automatically extracted keywords, user-provided keywords, and optional prompts without changing model parameters for new use cases.This separates test-time user control from training and pretraining.
- Control applications: Keywords support entity and length control, while prompts are useful for open-ended formats such as listing scientific-paper contributions.Length control maps user-specified parameters to keyword counts derived from five training-data length buckets.
- Control applications: CTRLsum uses prompts to focus patent summaries on invention purpose and to guide news summaries toward answers to given questions.These applications use textual control sequences tailored to the desired summary behavior.
- Control-token framework: Prompts and keywords are complementary because prompt performance can be weak and sensitive to wording for entity and length control, where explicit keywords work better.The paper reports this limitation from empirical observations and preliminary experiments.
3 RELATED WORK
Prior controllable summarization methods commonly train on predefined control codes, while related keyword-guided generation uses textual inputs for different generation objectives. These approaches motivate CTRLsum’s broader test-time control through keywords and prompts.
- Controllable summarization: Earlier controllable summarization systems use supervised entity or length control codes and generalize poorly to aspects absent during training.Saito et al. instead control length using the number of word prototypes, similarly to CTRLsum’s keyword-based strategy.
- Keyword-guided generation: Keyword-guided generation has also been used to mask copying operations, improve uncontrolled summarization, and plan poetry, dialogue, or stories.These works differ in motivation and application from CTRLsum’s multi-aspect summarization control.
4 EXPERIMENTS
Experiments evaluate CTRLsum across multiple datasets, control aspects, and human judgments, showing effective control and strong uncontrolled summarization performance. Results also examine factuality, length adherence, question-guided reading comprehension, contribution and purpose summarization, and evaluation limitations.
- Experimental setup: CTRLsum is evaluated on three distinct-domain datasets using ROUGE, BERTScore, control adherence measures, and human evaluation where references are unavailable.The experiments target five control aspects and uncontrolled summarization performance.
- Entity control: Approximately 95% entity-control success is achieved for both lead-3 and full-article entities, while factual correctness remains comparable to unconstrained BART.The entity-control evaluation uses human judgments for factual correctness and shows that other systems struggle particularly with entities absent from the article’s beginning.
- Length control: CTRLsum with length-guided keywords achieves high positive PCC and reduces MAD relative to automatic baselines, whereas oracle lengths provide only relatively small gains.LengthCode fails to consistently improve over BART and nearly ignores its length signal, with PCC close to 0.
- Contribution and purpose summarization: CTRLsum improves over BART in most contribution and invention-purpose summarization cases, focusing more precisely on keyword-related content than BART’s over-generated summaries.The comparison reports precision, recall, and F1 for BERTScore in addition to overall summarization performance.
- Question-guided summarization: Question keywords further improve F1 by 15.6 points on NewsQA and 17.9 points on SQuAD after summarization fine-tuning already improves the scores by 24.4 and 25.9 points.CTRLsum approaches the supervised MatchLSTM score on NewsQA despite not training on question-answer pairs.
- Automatic summarization: In uncontrolled summarization, CTRLsum reaches new state-of-the-art performance on CNNDM, outperforms BART and PEGASUS on CNNDM and arXiv, and is comparable to BART in BIGPATENT BERTScore.BIGPATENT performance has an inferior ROUGE-2 score despite comparable BERTScore to BART.
- Human evaluation: Human evaluation finds CTRLsum significantly better in control relevance overall, while non-expert summarization judgments may be unreliable and poorly correlated with expert judgments.Control accuracy for important-entity and purpose control is comparable between CTRLsum and BART without significant differences.
5 CONCLUSION
CTRLsum is a generic framework for multi-aspect controllable summarization. Keywords and prompts let users interact with the model flexibly at inference time, and experiments across five control aspects demonstrate its efficacy.
- CTRLsum performs multi-aspect controllable summarization using keywords during training and keywords or prompts as inference-time control tokens.
A.1 GENERAL SETUP
The experiments fine-tune pretrained BARTLARGE with specified optimization settings and tune automatic keyword-extraction hyperparameters on validation performance. An additional experiment tests whether prompts cause source copying in invention-purpose summarization.
- All experiments fine-tune the pretrained BARTLARGE model with a 3e-5 learning rate, polynomial decay, Adam optimization, and batch size 64.
- Automatic keyword extraction uses the number of pre-selected sentences, selection threshold, and maximum keyword count as hyperparameters.These values are selected per dataset using validation ROUGE-2, with performance robust across a reasonable range.
- Only 3 of 763 invention-purpose test articles matched searched prompt strings, indicating models usually do not copy source sentences through exact prompt matching.
A.2 HUMAN EVALUATION SETUP
Human evaluations use sampled examples and present system summaries together for scoring, with different procedures for controlled and uncontrolled summarization. Contribution summarization is excluded from human evaluation, and unreliable Mechanical Turk judgments lead the authors to use independent annotators for controlled tasks.
- Controlled Summarization: Controlled evaluations sample 100 examples per task and present summaries together; contribution summarization is not evaluated because judging papers across domains is unrealistic.
- Controlled Summarization: Amazon Mechanical Turk workers often ignored user intent, so two independent human annotators were used for controlled summarization instead.
- Uncontrolled Summarization: Uncontrolled evaluations sample 100 examples per dataset, use three independent Mechanical Turk workers, and report the average of per-example median scores.
B ABLATION ANALYSIS OF ENTITY CONTROL
The entity-control ablation finds that CTRLsum achieves about 95% success, while removing pretraining causes a smaller decline than replacing the Transformer architecture with a weaker convolutional model.
- ∼95% of CTRLsum summaries mention the requested entity, compared with 61.2% for lead-3 entities and 33.8% for full-article entities in prior work.Success rate is the fraction of decoded summaries that mention the given entity.
- The convolutional seq2seq variant loses over 40 success-rate points, whereas the non-pretrained Transformer variant drops around 5 points.The ablation attributes keyword dependence primarily to the Transformer architecture, with pretraining providing an additional improvement.
C ABLATION ANALYSIS ON KEYWORDS AND PROMPTS
CTRLsum’s control tokens can be supplied as keywords, prompts, or both, with their roles evaluated through entity-control and related tasks.
- Control-token roles: Entity control uses prompts, keywords, or both to test how each control-token type affects success rate.The prompt variant uses “a summary focused on [entity] is:”.
- Reference performance: CTRLsum’s automatic-keyword Transformer Seq2Seq variant obtains ROUGE-1/2/L scores of 43.69/20.78/40.55.The Conv Seq2Seq variant obtains 41.19/18.71/38.05.
- Prompt design: Alternative QA-style prompts for entity control do not lead to meaningful entity control.The tested variants include prompts asking what happened to or is known about an entity.
D ROBUSTNESS ANALYSIS OF KEYWORDS EXTRACTION HYPERPARAMETERS
CTRLsum’s uncontrolled summarization performance remains relatively robust across reasonable keyword-extraction hyperparameter settings.
- Robustness result: CTRLsum’s automatic summarization performance is relatively robust to keyword-extraction hyperparameters in a reasonable range.The validation analysis reports ROUGE-2 scores across different extraction settings.
- Hyperparameter setting: CNNDM uses finer-grained stride sizes when iterating mmax because its source articles are usually shorter than arXiv and BIGPATENT.The stride choice is dataset-specific in the hyperparameter analysis.
E.1 ENTITY CONTROL
The appendix illustrates CTRLsum’s controllable outputs across entity, length, contribution, invention-purpose, and question-guided summarization examples.
- Entity control: Entity-control examples randomly sample three CNNDM articles and five entity keywords per article.The examples show model outputs conditioned on selected entities.
- Entity control: Paired-entity examples contrast one important entity with one unimportant entity as simultaneous keywords.Three CNNDM articles are sampled, with five repeated samplings per article.
- Length control: Length-control examples generate five different-length summaries for each of three randomly sampled CNNDM test articles.The control tokens are bolded in the corresponding examples.
- Scientific and patent summarization: Contribution and invention-purpose examples use keywords and prompts together as control tokens.The tables mark these combined controls with square brackets.
- Question-guided summarization: Question-guided examples show five questions and answers for each of three randomly sampled NewsQA articles alongside gold answers.The examples use control tokens that are bolded in the tables.