Source-linked AI summary
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, Dragomir Radev
TL;DR
The paper addresses the difficulty of producing concise summaries for long, multi-topic meetings while serving users with different information needs. It introduces query-based multi-domain meeting summarization and QMSum, using a locate-then-summarize baseline; the benchmark exposes challenges involving long-context processing, generalization, query consistency, and factuality.
Problem
Long meetings contain multiple people, topics, decisions, and tasks, making a single concise summary difficult while users may seek different information.
Method
The paper defines query-based meeting summarization, constructs the multi-domain QMSum benchmark, and evaluates a locate-then-summarize pipeline with varied models and training settings.
Results
Human evaluation identifies challenges involving query types and factuality errors, while Locator-equipped truncated-input models achieve comparable results to gold-span inputs for PGNet and BART.
Takeaways & Limitations
QMSum leaves open research directions in processing long meeting discourse, improving generalization, maintaining transcript-query consistency, and reducing annotation cost.
Takeaways & Limitations
Current baseline models still generate ungrammatical and factually inconsistent summaries, limiting their reliability for direct organizational use.
Abstract
from arXiv · showhide
Meetings are a key component of human collaboration. As increasing numbers of meetings are recorded and transcribed, meeting summaries have become essential to remind those who may or may not have attended the meetings about the key decisions made and the tasks to be completed. However, it is hard to create a single short summary that covers all the content of a long meeting involving multiple people and topics. In order to satisfy the needs of different types of users, we define a new query-based multi-domain meeting summarization task, where models have to select and summarize relevant spans of meetings in response to a query, and we introduce QMSum, a new benchmark for this task. QMSum consists of 1,808 query-summary pairs over 232 meetings in multiple domains. Besides, we investigate a locate-then-summarize method and evaluate a set of strong summarization baselines on the task. Experimental results and manual analysis reveal that QMSum presents significant challenges in long meeting summarization for future research. Dataset is available at \url{https://github.com/Yale-LILY/QMSum}.
1 Introduction
Existing meeting summarization typically produces one summary for an entire, multi-topic meeting, but users often need different granularities and facets. QMSum addresses this with query-based summaries that locate relevant content and summarize it for specific intents.
- Existing meeting summarization usually generates a single summary for the whole meeting, despite meetings containing multiple topics, opinions, actions, and decisions.
- Different users may seek different meeting facets, making it difficult for one short summary to contain all salient information.
- QMSum defines query-based meeting summarization, where a model generates a summary from a meeting transcript and a user query.
- The dataset uses hierarchical annotations for queries, gold summaries, main topics, and relevant transcript spans across multiple granularities.
- The proposed locate-then-summarize approach first identifies query-relevant utterances and then summarizes the extracted spans.
- Human evaluation identifies query-type effects and factuality errors as challenges for the new task.
2 Related Work
Prior query-based summarization research focuses on news, debate, and Wikipedia rather than meetings. QMSum extends query-based summarization to multi-granularity meeting content for diverse user needs.
- Most prior text summarization research evaluates methods on news datasets such as CNN/DailyMail and Newsroom.
- Prior query-based summarization studies address news, debate, and Wikipedia, but not meeting discourse.
- Earlier meeting summarization work generally generates summaries for complete meetings using approaches such as entailment graphs, ranking, templates, compression, and multimodal information.
- QMSum instead summarizes multi-granularity meeting content to cater to different users’ needs across an entire meeting.
3 Data Construction
QMSum combines meetings from three domains with hierarchical annotations linking topics, transcript spans, queries, and summaries. Its 232 meetings and 1,808 query-summary pairs support multi-granularity, cross-domain evaluation.
- Data sources: QMSum collects meetings from product, academic, and committee domains.The sources include AMI product meetings, ICSI academic meetings, and parliamentary committee meetings from Wales and Canada.
- Annotation pipeline: The annotation pipeline has three stages: topic segmentation, query generation, and query-based summarization.
- Annotation pipeline: Topic segmentation records main topics and all relevant transcript spans, including spans that are separated across the meeting.For example, one topic can correspond to Turn 25–50 and Turn 73–89.
- Annotation pipeline: Annotators create general and specific queries from schema lists covering facets such as overall discussion content, opinions, and reasons for proposals.
- Annotation pipeline: Summaries are required to be faithful, informative, abstractive, fluent, and concise, with limits of 50–150 words for general queries and 20–100 words for specific queries.
- Dataset statistics: 1,808 query-summary pairs cover 232 meetings, with approximately 15% of meetings assigned to validation and another 15% to testing.
- Dataset statistics: QMSum spans multiple domains and has an average summary length of 69.6 words, supporting generalization evaluation while requiring precise compression.
4 Method
The method introduces query-based meeting summarization as a conditional generation task and implements it with a two-stage locate-then-summarize pipeline. A Locator extracts query-relevant spans, which a Summarizer uses to generate the final query-based summary.
- Task Definition: The task generates a summary Y conditioned on a query Q and all meeting utterances and speakers.This extends meeting summarization beyond generating one summary for an entire transcript.
- Locate-then-Summarize: The Locator extracts shorter, query-related text spans because most abstractive models cannot process long meeting transcripts.The extracted spans are passed to the subsequent Summarizer.
- Locator: Pointer Network locates spans by predicting start and end turns for each query, extracting three spans because a query may correspond to multiple spans.The model treats each query as an extractive span-selection problem over meeting turns.
- Locator: The hierarchical ranking-based Locator combines fixed BERT word embeddings, CNN turn-level features, and speaker embeddings to represent each turn.The turn representation is xi = [ui; si], where speaker embeddings represent participants’ speaking styles.
- Locator: A document-level Transformer contextualizes turn representations, after which query-aware MLP scores select the highest-ranked relevant turns.The Locator is trained with binary cross-entropy, and selected spans become input to the Summarizer.
- Summarizer: The Summarizer stage evaluates Pointer-Generator Network, BART, and HMNet as abstractive models over the selected spans.PGNet and BART receive the query and relevant spans, while HMNet receives the query as an additional initial turn.
5 Experiments
Experiments evaluate Locator effectiveness, query-based summarization performance, cross-domain robustness, and query-type difficulty on QMSum. Results show that hierarchical span selection improves recall and downstream summarization, while domain transfer and certain query types remain challenging.
- 5.2 Effectiveness of Locator: Similarity performs worse than Random, while Pointer Network is only slightly better because sparse span supervision is weak for meetings averaging more than 500 turns.ROUGE-L recall measures overlap between predicted and gold relevant spans under equal extraction budgets.
- 5.2 Effectiveness of Locator: 72.51 ROUGE-L recall is achieved by the hierarchical ranking-based Locator while extracting only 1/6 of the original meeting text.The method greatly exceeds Random and reduces the Summarizer's long-text processing burden while preserving information.
- 5.3 Experimental Results on QMSum: PGNet improves from 28.74 to 31.37 R-1 and BART from 29.20 to 31.74 R-1 when equipped with the Locator.Neural models outperform Random and TextRank, while HMNet achieves the best overall performance.
- 5.3 Experimental Results on QMSum: BART improves over PGNet by 1.13 ROUGE-L, indicating more fluent generated sentences, while HMNet achieves the best performance.HMNet's advantage may be attributed to cross-domain pretraining that better matches meeting-transcript style.
- 5.3 Experimental Results on QMSum: Locator-selected spans yield results comparable to gold spans for truncated PGNet and BART, but gold spans substantially improve HMNet.The two-stage pipeline is therefore described as simple but effective for models that must truncate long inputs.
- 5.3 Experimental Results on QMSum: Cross-domain performance is poor for Academic- and Committee-trained models tested on Product, with ROUGE-L scores of 24.09 and 22.17, respectively.Product-only training reaches 31.37 ROUGE-L, whereas multi-domain training remains comparable across domains and improves Academic ROUGE-2 from 4.32 to 5.05 and ROUGE-L from 22.58 to 23.01.
6 Analysis
QMSum analysis examines query-type distributions, human difficulty, and model errors. The results show that multi-person interaction is common, reason-focused queries are especially difficult, and generated summaries often lack factual consistency or query relevance.
- Query Types: 40% of sampled queries require summarizing multi-person interaction, while personal opinions and conclusion or decision queries each account for almost 20%.
- Query Difficulty: Reason-focused queries are hardest for humans to locate and organize, whereas multi-person interaction and overall-content queries are relatively easier.
- Query Difficulty: BART exceeds 30 R-L on multi-person interaction and overall-content queries but performs poorly on the remaining query types.
- Error Analysis: 74% of sampled generated summaries contain inconsistent facts, and 31% are completely unrelated to the given query.
7 Conclusion
The paper introduces QMSum as a benchmark for query-based meeting summarization and evaluates a locate-then-summarize baseline with model variants and training settings. The benchmark highlights open challenges in long-discourse processing, cross-domain generalization, and generating summaries consistent with both meetings and queries.
- Conclusion: QMSum is a benchmark for query-based meeting summarization, accompanying a new task and a high-quality dataset.
- Conclusion: The locate-then-summarize pipeline is evaluated with different Locators, Summarizers, and cross-domain or multi-domain training settings.
- Conclusion: The task leaves open challenges in processing long meeting discourses, generalizing across domains, and generating summaries consistent with meeting text and queries.
Ethics Consideration
The paper addresses intellectual-property, privacy, bias, annotator compensation, intended use, and risks associated with applying QMSum and its models.
- Intellectual Property and Privacy Rights: All collected meeting transcripts were public and open to use, and annotation procedures were intended to respect the intellectual-property and privacy rights of meeting authors and annotators.
- Compensation for Annotators: Annotators were paid approximately $14 per product or academic meeting and $28 per committee meeting, with additional bonuses for higher-volume annotation.
- Steps Taken to Avoid Potential Problems: The dataset construction addressed potential gender bias from pronoun use by requiring annotators to replace pronouns with speaker information.
- Failure Mode: Baseline models can produce ungrammatical and factually inconsistent summaries, so direct organizational deployment could spread misinformation and harm comprehension and decision making.
- Bias: The authors acknowledge that domain imbalance, distribution mismatch, and unaddressed political bias may bias models trained on QMSum.
- Misuse Potential: Generated summaries are not reliable enough for unsupervised use, and models trained on internal meeting data require strict supervision because of intellectual-property and privacy concerns.
- Collecting Data from Users: Researchers must consider privacy before training on meeting transcripts because some meetings are intended only for internal use.
A.1 Query Schema List
The query schema list organizes queries according to the generality or specificity of the meeting content they address. The detailed schemas are provided in Table 6.
- Schema Organization: The query schema list supports annotators in designing queries for QMSum.
- Schema Organization: Queries are divided into general and specific schema lists according to the meeting content they target.
- Schema Organization: The detailed query schemas are presented in Table 6.
A.2 Other Details of Annotation Instruction
The annotation instructions specify how to segment topics, diversify and disambiguate queries, and produce informative, faithful query-based summaries with consistent wording conventions.
- A.2.1 Topic Segmentation: Annotators represented main topics with noun phrases and selected approximately 3 to 8 important topics per meeting.
- A.2.1 Topic Segmentation: Relevant spans for a main topic could be scattered across the meeting, so annotators labeled all relevant noncontiguous spans.
- A.2.1 Topic Segmentation: Chatting counted as an independent main topic when it occupied substantial meeting time, even if it was unrelated to the meeting’s main theme.
- A.2.2 Query Generation: Annotators diversified specific queries by using different schemas to reduce bias toward particular query types.
- A.2.2 Query Generation: Each query-answer pair was written independently and specified the relevant agreement, topic, or time instead of relying on prior annotations.
- A.2.3 Query-based Summarization: Answers to opinion queries included brief relevant context alongside the speaker’s statement to make summaries more informative and challenging.
- A.2.3 Query-based Summarization: Annotators labeled all relevant spans, used past tense for completed meetings, and avoided gendered or abbreviated speaker references when unclear or inappropriate.
- A.2.3 Query-based Summarization: Transcript abbreviations containing underscores, such as ‘L_C_D_’ and ‘A_A_A_’, were rewritten as ‘LCD’ and ‘AAA’.
A.3 Annotation Review Standards
This appendix describes pre-annotation and expert review procedures, then presents examples and schema references illustrating the QMSum annotations.
- A.3 Annotation Review Standards: The pre-annotation stage required annotators to complete one meeting before continuing with the task.
- A.3 Annotation Review Standards: Experts reviewed initial work and continued checking annotations for faithfulness, informativeness, relevant-span length, and typographical errors.
- A.3 Annotation Review Standards: The appendix provides QMSum examples from product and committee meetings in Tables 7 and 8.
- A.3 Annotation Review Standards: Table 6 defines the general and specific query schema notation, where A and B are speakers, X is a main topic, and Y is a related subtopic.