PaperlayerView exact revision on arXiv

Source-linked AI summary

MeetingBank: A Benchmark Dataset for Meeting Summarization

Yebowen Hu, Tim Ganter, Hanieh Deilamsalehy, Franck Dernoncourt, Hassan Foroosh, Fei Liu

arXiv:2305.17529v1cs.CL

TL;DR

Recorded meetings need useful summaries, but annotated meeting corpora are scarce and difficult to create because meetings can be confidential and expert-written summaries are labor-intensive. MeetingBank constructs a city-council dataset with aligned segment-level references and evaluates summarizers; the experiments suggest that systems should capture main discussion points accurately.

  • Problem

    Annotated meeting datasets are scarce, while confidentiality and labor-intensive expert annotation make meeting-summary references difficult to create.

    Source: §1 · p. 1

  • Method

    MeetingBank collects city-council meetings and divides them into segments aligned with corresponding summaries from meeting minutes.

    Source: §1 · p. 2

  • Results

    Experiments with automatic metrics and expert annotators suggest meeting summarizers should prioritize capturing main discussion points and maintaining accuracy to the original.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    A meeting summarizer could help citizens navigate public meetings more efficiently and support citizen engagement.

    Source: §3 · p. 4

    The dataset is limited to U.S. city council meetings and excludes non-verbal cues such as gaze, gestures, and facial expressions.

    Source: §9 · p. 9

Abstract

from arXiv · show

As the number of recorded meetings increases, it becomes increasingly important to utilize summarization technology to create useful summaries of these recordings. However, there is a crucial lack of annotated meeting corpora for developing this technology, as it can be hard to collect meetings, especially when the topics discussed are confidential. Furthermore, meeting summaries written by experienced writers are scarce, making it hard for abstractive summarizers to produce sensible output without a reliable reference. This lack of annotated corpora has hindered the development of meeting summarization technology. In this paper, we present MeetingBank, a new benchmark dataset of city council meetings over the past decade. MeetingBank is unique among other meeting corpora due to its divide-and-conquer approach, which involves dividing professionally written meeting minutes into shorter passages and aligning them with specific segments of the meeting. This breaks down the process of summarizing a lengthy meeting into smaller, more manageable tasks. The dataset provides a new testbed of various meeting summarization systems and also allows the public to gain insight into how council decisions are made. We make the collection, including meeting video links, transcripts, reference summaries, agenda, and other metadata, publicly available to facilitate the development of better meeting summarization techniques. Our dataset can be accessed at: https://meetingbank.github.io

1 Introduction

Annotated meeting corpora are scarce, while confidentiality and labor-intensive summary writing complicate their creation. MeetingBank addresses this gap with a large city-council collection, segment-level references, and evaluations of summarizers.

  • Existing meeting datasets are scarce, small, or unrepresentative, while confidentiality and labor-intensive annotation complicate the creation of annotated meeting corpora.Source: §1 · p. 1
  • MeetingBank provides 1,366 meetings from six major U.S. cities, with over 3,579 hours of video, transcripts, minutes, agendas, and other metadata.Source: §1 · p. 2
  • Its divide-and-conquer strategy segments lengthy meetings and aligns each segment with a minutes-derived summary, yielding 6,892 segment-level instances.Source: §1 · p. 2
  • Experiments compare extractive, fine-tuned abstractive, and prompted GPT-3 summarizers; evaluations suggest prioritizing main discussion points and accuracy to the source.Source: §1 · pp. 2–3

2 Existing Datasets

Existing meeting corpora vary in size, setting, and annotation, and several adjacent conversational datasets do not capture meeting decision-making. MeetingBank complements them with naturally occurring city-council meetings and segment-level references.

  • ICSI contains 75 naturally occurring seminar meetings among speech researchers, while AMI contains 140 scenario-based product-design meetings with multiple annotation types.Source: §2 · p. 3
  • ELITR covers 120 English and 59 Czech technical project meetings across 180 hours, while QMSum supports query-focused summaries for limited parliamentary, AMI, and ICSI meeting sets.Source: §2 · p. 3
  • Podcast, interview, livestream, and TV-series datasets provide related summarization resources, but their conversations usually involve fewer participants and lack decision-making.Source: §2 · p. 3
  • Chat-based corpora include manually summarized short dialogs and other online conversations, whereas this paper focuses on naturally occurring meeting conversations.Source: §2 · pp. 3–4

3 Creation of MeetingBank

MeetingBank collects public city council meetings and aligns transcript segments with concise reference summaries derived from meeting minutes. Its release includes 1,366 meetings from six cities, with 6,892 segment-level instances.

3 Creation of MeetingBank

  • The dataset’s public-meeting focus is intended to support summarization research and make meetings more accessible for citizen engagement.The paper notes that only 11% of Americans attend public meetings to discuss local issues.Source: §3 · p. 4
  • The release contains 1,366 meetings from six cities spanning over a decade, with transcripts, minutes, agendas, and other metadata.The researchers selected cities that regularly publish meeting minutes and downloadable videos, and consulted their legal team or councils when needed about licensing and data policies.Source: §3 · p. 4
  • San Francisco was excluded because its council permitted attributed video reposting but prohibited reposting or editing official minutes and agendas.The dataset’s inclusion depended on whether source materials could be used under applicable licensing and data policies.Source: §3 · p. 4
  • The divide-and-conquer process extracts council bill numbers, uses bill descriptions as reference summaries, and aligns each bill to its meeting segment and transcript.After filtering noisy or too-short segments, the process yields 6,892 instances; segments must be at least 60 seconds and summaries at least 10 words.Source: §3 · p. 5
  • Speechmatics transcribed 3,579 hours of meetings; transcripts average 28,358 tokens, while segments average 2,892 transcript tokens and 87 summary tokens.The transcripts include word-level time alignment, casing, punctuation, and speaker diarization; the average compression rate is 97%.Source: §3 · p. 5
  • The dataset focuses on spoken language and does not capture non-verbal cues or persuasive argumentation.Agendas and supporting documents are gathered when available.Source: §3 · p. 5

4 Data Analysis

MeetingBank reference summaries show substantial reuse of transcript wording, but their abstraction levels vary across cities. Coverage and Density scores characterize this variation, which may reflect meeting and minute-taking differences.

4 Data Analysis

  • Coverage scores range from 0.7 to 0.9 for most cities, suggesting meeting summaries often reuse discussion points verbatim rather than abstracting them.The authors note this alongside a compression rate above 90%, so they argue that effective summarizers should accurately identify content to include.Source: §4 · pp. 5–6
  • Coverage measures the share of summary words appearing in the source, while Density measures how much the summary consists of extractive fragments.Figure 3 plots Coverage on the x-axis and Density on the y-axis.Source: §4 · pp. 5–6
  • Seattle and Boston have the highest Density scores, while Denver has the lowest, indicating more editing in Denver’s minutes.The authors note that plainly read resolutions and ordinances can raise summary Density relative to news documents.Source: §4 · p. 6
  • Coverage and Density can vary with meeting length and complexity and with minute-takers’ preferences, reflecting the task’s diversity.The paper presents these measures as influenced by multiple factors rather than as fixed properties of each city’s summaries.Source: §4 · p. 6

5 Performance of Existing Systems

The paper evaluates extractive, fine-tuned abstractive, and prompted GPT-3 summarizers on meeting segments. The Extractive Oracle performs strongly, DialogLM leads the abstractive systems, and fine-tuning BART on in-domain data improves its performance.

5 Performance of Existing Systems

  • Systems are evaluated on 862 test instances, with each summarizer generating a concise summary from a meeting-segment transcript.The dataset is split into 5,169 training, 861 validation, and 862 test instances; evaluation reports test-set results.Source: §5 · p. 6
  • The extractive baselines include an Oracle, LEAD, LexRank, and TextRank; the Oracle adds high-scoring sentences until combined R-1 and R-2 cannot improve.LEAD selects the first N sentences, while LexRank and TextRank use sentence centrality in a graph and each extract two sentences.Source: §5 · p. 6
  • The abstractive models include BART-large, Pegasus, Longformer, DialogLM, and HMNet, with BART-large tested both with and without fine-tuning.The evaluation also includes text-davinci-003 prompted to summarize the source in two sentences.Source: §5 · pp. 6–7
  • Automatic evaluation covers lexical-overlap, embedding-based semantic-similarity, and question-answering metrics.The paper uses evaluation toolkits and metrics including ROUGE, BLEU, METEOR, BertScore, MoverScore, and QAEval.Source: §5 · p. 7
  • The Extractive Oracle reaches 46.60% R-2 F-score; DialogLM performs best among abstractive systems, Pegasus is on par, and in-domain fine-tuning improves BART.LexRank and TextRank perform poorly despite their strong performance on news articles; GPT-3 prompting performs poorly on automatic metrics.Source: §5 · p. 7

6 City-by-City Analysis

The city-by-city analysis finds that metric scores vary with summary extractiveness and that performance depends on city-specific training data. Seattle and Denver show substantial drops when their own training instances are excluded.

6 City-by-City Analysis

  • Extractiveness-sensitive metrics, including MoverScore and ROUGE, vary across datasets, while BERTScore and QAEval produce scores in similar ranges.The analysis evaluates reference-summary abstraction using Coverage and Density, plotted in Figure 3.Source: §6 · p. 7
  • Removing Seattle training instances reduced the BART R-2 F-score to 40.05%, despite Seattle contributing a moderate number of instances.The authors suggest Seattle’s training instances are effective and its council may produce higher-quality minutes than other cities.Source: §6 · p. 8
  • Denver’s BART performance dropped 32.43% when same-city training instances were excluded, and Seattle, Boston, and Denver benefited more from same-city data.The authors connect this pattern to differences in meeting discussions and minute-writing styles across cities.Source: §6 · p. 8

7 Human Evaluation

Human evaluation on 200 randomly selected instances finds abstractive systems stronger than extractive systems, with Pegasus the best-performing abstractive system. GPT-3 earns the highest overall score, but its informativeness and factuality remain weaker than its fluency and coherence.

7 Human Evaluation

  • Evaluators watched a meeting video segment, read its transcript, and rated each system summary for informativeness, factuality, fluency, coherence, and redundancy.The evaluation recruited experienced workers from the United States and India, including pilot annotations.Source: §7 · p. 8
  • Abstractive systems performed stronger than extractive systems, and Pegasus was the best-performing abstractive system among the systems evaluated on 200 randomly selected instances.The authors attribute Pegasus’s effectiveness to pretraining that masks key sentences and regenerates them from the remaining sentences, helping identify important transcript content.Source: §7 · pp. 8–9
  • GPT-3 received the highest overall human-evaluation score, 4.21, across all criteria.Its fluency and coherence scores were 4.52 and 4.41, while informativeness and factuality scored 3.74 and 3.82.Source: §7 · p. 9

8 Conclusion

The paper presents MeetingBank as a city-council-meeting benchmark and evaluates extractive, fine-tuned abstractive, and prompted GPT-3 summarizers. The authors describe it as a testbed for meeting summarization and extracting structure from meeting videos.

8 Conclusion

  • MeetingBank is a benchmark dataset built from city council meetings.Source: §8 · p. 9
  • The authors tested extractive systems, fine-tuned abstractive systems, and GPT-3 with prompting; GPT-3 was well received by human evaluators but fell short on informativeness and factual consistency.Source: §8 · p. 9
  • The dataset is presented as a testbed for researchers developing meeting summarizers and extracting structure from meeting videos.Source: §8 · p. 9

9 Limitations

MeetingBank is limited in geographic, temporal, and modality scope, and licensing prevented inclusion of some meetings. Its data omit non-verbal cues that may matter for systems relying on them.

9 Limitations

  • The authors describe the dataset as having potential to improve meeting efficiency and effectiveness despite these limitations.Source: §9 · p. 9
  • The dataset covers U.S. city council meetings from the past decade, with licensing restrictions excluding some meetings.San Francisco did not permit redistribution of its meeting minutes.Source: §9 · p. 9
  • The dataset omits eye gazes, gestures, and facial expressions, potentially limiting its suitability for summarization systems that rely on these cues.Source: §9 · p. 9

10 Ethical Considerations

The authors collected publicly accessible city council videos, minutes, and metadata, consulting legal counsel and councils as needed to address licensing and data policies. They sought to exclude confidential information and released the dataset for research purposes.

10 Ethical Considerations

  • The team obtained videos, minutes, and metadata from public sources and consulted legal counsel and councils as needed about licensing and data policies.Source: §10 · p. 9
  • The authors made efforts to exclude confidential information and designated the dataset for research use only.Source: §10 · p. 9

A Experimental Settings

The experiments use GPT-3 text-davinci-003 with a two-sentence summarization prompt, alongside extractive and fine-tuned abstractive systems. Preliminary comparison suggests limited transfer between city-council meetings and the specialized structures of ICSI/AMI meetings.

A Experimental Settings

  • GPT-3 text-davinci-003 is prompted to summarize a transcript in two sentences.The prompt template asks for a summary in N sentences; implementation details and hyperparameters for extractive and fine-tuned abstractive systems are listed in Table 6.Source: A Experimental Settings · p. 13, Table 6 · p. 13

B Comparison to ICSI/AMI

  • A BART summarizer fine-tuned on MeetingBank does not perform optimally on ICSI/AMI in preliminary experiments.The authors characterize ICSI as research seminars by speech researchers, unlike MeetingBank's U.S. city council meetings.Source: B Comparison to ICSI/AMI · p. 13

Related papers

Loading 2305.17529v1…