Source-linked AI summary
Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarization
Xianjun Yang, Yan Li, Xinlu Zhang, Haifeng Chen, Wei Cheng
TL;DR
Aspect- and query-based summarization is less explored than generic news summarization, despite its focus on user-specific information needs. This paper evaluates ChatGPT across four diverse benchmark tasks and finds performance comparable to traditional fine-tuning methods in Rouge scores. The authors conclude that ChatGPT shows promise while noting input-length and ethical-evaluation limitations.
Problem
Performance evidence for LLMs on practical aspect- and query-based summarization tasks remains limited compared with generic news summarization.
Method
The paper systematically evaluates ChatGPT on four aspect- and query-based summarization tasks using benchmark datasets and Rouge-based analysis.
Results
ChatGPT-generated diverse specific summaries are highly comparable to traditional fine-tuning methods in terms of Rouge scores.
Takeaways & Limitations
ChatGPT holds promise as a powerful tool for diverse controllable text summarization and motivates further research on its generated summaries.
Takeaways & Limitations
The experiments were limited by ChatGPT’s maximum input sequence length of around 5000 tokens, which may affect generalizability across documents and datasets.
Abstract
from arXiv · showhide
Text summarization has been a crucial problem in natural language processing (NLP) for several decades. It aims to condense lengthy documents into shorter versions while retaining the most critical information. Various methods have been proposed for text summarization, including extractive and abstractive summarization. The emergence of large language models (LLMs) like GPT3 and ChatGPT has recently created significant interest in using these models for text summarization tasks. Recent studies \cite{goyal2022news, zhang2023benchmarking} have shown that LLMs-generated news summaries are already on par with humans. However, the performance of LLMs for more practical applications like aspect or query-based summaries is underexplored. To fill this gap, we conducted an evaluation of ChatGPT's performance on four widely used benchmark datasets, encompassing diverse summaries from Reddit posts, news articles, dialogue meetings, and stories. Our experiments reveal that ChatGPT's performance is comparable to traditional fine-tuning methods in terms of Rouge scores. Moreover, we highlight some unique differences between ChatGPT-generated summaries and human references, providing valuable insights into the superpower of ChatGPT for diverse text summarization tasks. Our findings call for new directions in this area, and we plan to conduct further research to systematically examine the characteristics of ChatGPT-generated summaries through extensive human evaluation.
1 Introduction
This paper extends ChatGPT evaluation beyond generic news summarization to aspect- and query-based tasks across diverse domains. It systematically examines whether ChatGPT can produce specific summaries comparable to traditional fine-tuning methods.
- Existing LLM summarization research has focused mainly on generic news summaries rather than aspect- or query-based applications.
- Aspect- and query-based summarization tailors outputs to specific aspects or queries instead of producing one condensed summary of an entire document.
- The study presents a comprehensive evaluation of ChatGPT on four aspect- and query-based summarization tasks.
- ChatGPT-generated specific summaries are highly comparable to traditional fine-tuning methods in Rouge scores.
- The paper analyzes LLM-generated summaries and identifies potential future research directions leveraging their strengths.
2 Related Work
Prior work established the promise of large language models for generic summarization, but evidence for diverse aspect- and query-based tasks remained limited. This paper positions its evaluation as a response to that gap and distinguishes these tasks from generic summarization methods.
- Aspect- and query-based summaries are tailored to specific aspects or queries, unlike generic summaries produced for the entire document.
- These tasks cover content types including news articles, meetings, stories, and Wikipedia articles.
- End-to-end methods directly generate summaries, whereas extract-then-summarize methods first extract important sentences or phrases and then summarize the shortened document.
- GPT-3 and ChatGPT attracted substantial NLP interest after achieving strong results across tasks such as classification, question answering, and machine translation.
- Earlier studies found GPT-3 summaries slightly below traditional fine-tuning in Rouge scores, while human evaluators preferred them.
- News-domain LLM summaries were reported as comparable to human-written summaries, but the datasets were not explicitly designed for summarization.
- The paper therefore evaluates ChatGPT on diverse aspect- and query-based summarization tasks to examine its strengths and limitations.
3 Task Formulation
The study evaluates ChatGPT on aspect- and query-based benchmarks using manually sampled test examples, Rouge scoring, and dataset-specific prompts. Prompting is generally zero-shot, with a one-shot setup for CovidET.
- 3 Task Formulation: The tasks target summaries tailored to specific aspects or queries rather than generic summaries of complete documents.
- 3 Task Formulation: The evaluation uses publicly available datasets selected for consistency with previous fine-tuning methods.
- 3 Task Formulation: ChatGPT is evaluated on question-answer pairs with Rouge scores, using 100 randomly selected examples from each test set.
- 3 Task Formulation: The study could not use a ChatGPT API for large-scale input processing, so evaluation was performed manually on the ChatGPT platform.
- 3.1 Prompts: The experiments use prompts designed to generate summaries for the evaluated datasets.
- 3.1 Prompts: SQuALITY prompts request answers of around 200 words for specific questions and around 450 words for general questions.
- 3.1 Prompts: QMSum prompts provide either the original meeting or golden spans of sentences together with a query.
- 3.1 Prompts: CovidET uses a one-shot prompt requesting one short aspect-focused sentence, while NEWTS asks for an aspect-focused article summary; other datasets use zero-shot evaluation without further correction conversations.
4 Experiments and Analysis
Across four diverse aspect- and query-based summarization datasets, ChatGPT generally matches traditional fine-tuning on Rouge scores while producing systematically different summaries. Its performance varies with input characteristics, and its longer, more abstractive outputs expose limitations of Rouge-based evaluation.
- Overall results: ChatGPT achieves comparable performance to traditional fine-tuning across all evaluated datasets.The evaluation covers Reddit posts, news articles, meeting dialogues, and stories.
- Overall results: ChatGPT outperforms fine-tuning on all Rouge scores in the news domain.The authors associate this result with the large availability of news data for pre-training, but state this as a suspicion.
- QMSum: In QMSum, ChatGPT outperforms fine-tuning on Rouge-2 and matches it on Rouge-1 after splitting overlength inputs, but performs worse on Rouge-L.With golden meeting spans, it performs slightly better on Rouge-1 and Rouge-2, while Rouge-L remains substantially lower.
- SQuaLITY: In SQuaLITY, ChatGPT shows similar Rouge performance, lagging the baseline by only 1 point.Stories longer than 3000 words are truncated, and instances whose queries cannot be answered from truncated text are discarded.
- Summary characteristics: ChatGPT-generated summaries consistently have lower compression ratios, indicating more extended summaries, while coverage and density show no apparent difference.For long-input datasets, ChatGPT has higher unique 1- and 2-gram rates but lower unique 3- and 4-gram rates, suggesting greater short-word abstractive variation.
- Insights: ChatGPT requires no additional training-data labeling in the zero-shot setting and may improve with better prompts or multiple self-correcting conversations.The authors leave complete human evaluation for future work.
5 Conclusion
The paper concludes that ChatGPT shows strong promise for controllable aspect- and query-based summarization across diverse domains. It also identifies improved retrieval, prompting, and conversational refinement as directions for future investigation.
- Conclusion: ChatGPT demonstrates strong capabilities on aspect- and query-based summarization across diverse domains.The conclusion cautions that Rouge may not adequately evaluate ChatGPT-generated summaries and plans human evaluation.
- Future directions: Future work should investigate retrieval modules that address ChatGPT’s input-length constraints.The paper specifically suggests integrating a lighter model such as LED to retrieve salient sentences from lengthy inputs.
- Future directions: Improved prompting and multiple conversations for summary refinement remain promising research directions.The authors note that their preliminary experiments did not thoroughly explore the prompt space or multiple conversations.
6 Limitations
The experiments are constrained by ChatGPT’s maximum input sequence length and by unresolved ethical concerns surrounding fairness, bias, and factuality. These boundaries limit how broadly the results should be generalized or applied.
- Scope limitation: ChatGPT’s approximately 5000-token maximum input length limits the experiments’ generalizability across datasets with widely varying document lengths.Real-world summarization inputs may exceed the setting evaluated in this study.
- Ethical limitation: The study does not establish that ChatGPT summarization is suitable for decisions that could negatively affect individuals or groups.The authors call for further assessment of fairness, bias, and factuality.
A Generated examples
The appendix presents zero-shot examples of ChatGPT-generated query-based and aspect-based summaries. Table 4 focuses on query-based summaries, while Table 5 focuses on aspect-based summaries.
- Query-based summaries: Table 4 presents examples of query-based summaries generated under the zero-shot setting.
- Query-based summaries: The query example targets the aspect of money, including terms such as pay, card, credit, and spending.
- Aspect-based summaries: Table 5 presents examples of aspect-based summaries generated under the zero-shot setting.