Source-linked AI summary
Summarizing and Exploring Tabular Data in Conversational Search
Shuo Zhang, Zhuyun Dai, Krisztian Balog, Jamie Callan
TL;DR
Tables can be difficult to present in conversational search because reading them aloud is impractical and they may be too large for chat. The paper builds a crowdsourced collection of question-oriented, exploration-supporting table summaries and evaluates generation baselines; T5 performs best, though generated summaries often contain quantity and reference errors.
Problem
Tables are structurally rich, contain little natural-language content, and can be too large to present effectively in conversational search interfaces.
Source: §1 · p. 1Method
The authors crowdsource a collection of question–table pairs with manually written summaries and quality assessments, then compare neural language-generation baselines.
Source: §4 · p. 4Results
T5 is the best model, while both pre-trained models, GPT-2 and T5, substantially and significantly outperform CopyNet on all metrics.
Source: §3.2 · p. 4Takeaways & Limitations
A summary can answer the user’s question while supplying table information that supports follow-up exploration.
Source: §1 · p. 1Wrong Quantity errors occurred in 84% of summaries and Wrong Reference errors in 38%.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Tabular data provide answers to a significant portion of search queries. However, reciting an entire result table is impractical in conversational search systems. We propose to generate natural language summaries as answers to describe the complex information contained in a table. Through crowdsourcing experiments, we build a new conversation-oriented, open-domain table summarization dataset. It includes annotated table summaries, which not only answer questions but also help people explore other information in the table. We utilize this dataset to develop automatic table summarization systems as SOTA baselines. Based on the experimental results, we identify challenges and point out future research directions that this resource will support.
1 INTRODUCTION
Tables can be difficult to present in conversational search because they are structurally rich, low in natural-language content, and potentially too large to read aloud. The paper proposes summaries that answer the user’s question while helping them explore the table, and builds a crowdsourced collection to study the task.
1 INTRODUCTION
- Tables are challenging to present in conversational search because they are structurally rich, low in natural-language content, and may be too large for chat or voice interfaces.Reading a table line by line can create a poor user experience.Source: §1 · p. 1
- Existing structure- and content-based table summarization methods often assume decomposable tables, rely on predefined schemes, and do not use the conversation context that triggered the table.Source: §1 · p. 1
- A conversational table summary should answer the last-turn question and provide an overview or other table information that helps users explore further.The example suggests this context can support follow-up questions about other entries.Source: §1 · p. 1
- The authors create a crowdsourced, publicly available test collection and compare general-purpose abstractive summarization methods as baselines, analyzing results to identify future directions.Source: §1 · pp. 1–2
2 CREATING A TEST COLLECTION
The authors create a test collection of question–table pairs with crowdsourced summaries and sentence- and summary-level quality assessments. The collection process targets summaries that answer the question and offer clues for exploring the table.
2 CREATING A TEST COLLECTION
- The test collection pairs a conversational query with a result table to study summaries that help answer the query.Source: §2 · p. 2, §2.1 · p. 2
2.1 Queries and Tables
- The study randomly selected WikiTableQuestions tables with at least six rows and four columns, pairing them with 45 dataset-sampled questions and 155 questions written by two authors.Source: §2.1 · p. 2
2.2 Collecting Candidate Summaries
- Workers were asked to write concise, question-relevant summaries that offered exploration clues; five workers produced 1,000 candidate summaries across the 200 query–table pairs.Summaries were targeted at around 30–50 words.Source: §2.2 · p. 2
2.3 Quality Assessment
- The authors assessed the five candidate summaries for each table at both sentence and whole-summary levels to identify quality and relevance.Source: §2.3 · p. 2
2.3.1 Sentence Level.
- Three judges rated each sentence as highly relevant, relevant, or non-relevant, based on whether it addressed the question, described the table, or was unrelated.Source: §2.3.1 · p. 2
2.3.2 Summary Level.
- Seven judges selected the best of five summaries using criteria of concise, readable language, question relevance, and ability to invite follow-up questions; the most-voted summary became ground truth.Source: §2.3.2 · pp. 2–3
- 154/200 (77%) of tables had at least three judges agree on the top-voted summary; those summaries were longer, more vocabulary-diverse, and used more numbers than lower-voted summaries.The authors interpret the greater use of numbers as indicating more factual information.Source: §2.3.2 · p. 3
3 EXPERIMENTS
The experiments benchmark three neural generators on flattened table sequences. Pre-trained models outperform CopyNet, but generated summaries still make quantity and reference errors.
3.1 Methods for Comparison
- The models receive tables flattened row by row into key:value header-cell pairs, with row indexes, page and table context, and total-row count.The keys are intended to help models learn the meanings of the values when generating summaries.Source: §3.1 · p. 3
- CopyNet, GPT-2, and T5 serve as reproducible baselines; evaluation uses five-fold cross-validation and ROUGE and BLEU against the top-voted summary.Each fold trains on 160 tables and tests on 40.Source: §3.1 · p. 3
3.2 Experimental Results and Discussion
- T5 is the best model, while both T5 and GPT-2 significantly outperform CopyNet.CopyNet has the weakest performance on all metrics.Source: Table 1 · p. 4, §3.2 · p. 4
- Wrong Quantity errors occur in 84% of summaries, and Wrong Reference errors occur in 38%.The authors suggest combining semantic parsing with natural language generation to address these problems.Source: §3.2 · p. 4
4 CONCLUSION AND FUTURE DIRECTIONS
The paper introduces a crowdsourced benchmark for conversational table summarization and compares neural generation baselines. It frames the benchmark as a first step, with traditional methods and task-specific approaches left for future study.
4 CONCLUSION AND FUTURE DIRECTIONS
- The test collection contains 200 questions, each paired with a result table and five manually created candidate summaries with sentence- and summary-level quality assessments.Source: §4 · p. 4
- The Table 2 example shows that both the human-written and T5-generated summaries answer the question correctly and add information from the table.Source: Table 2 · p. 4
- The authors compare three neural language generation baselines and identify two main classes of errors.Source: §4 · p. 4
- The benchmark focuses on recent abstractive methods; the authors propose testing traditional summarizers and developing approaches that account for the task's distinctive characteristics.Source: §4 · p. 4