QAConv: Question Answering on Informative Conversations

1 Introduction

• Existing QA work emphasizes documents or short-paragraph conversational QA, while conversation-based QA must handle evidence scattered across speakers and turns, harder coreference, and missing training data.The paper distinguishes QA pairs grounded in conversations from sequential dialogue-like QA pairs grounded in short document paragraphs.Source: §1 · p. 1
• QAConv samples 10,259 conversations from emails, panels, and work channels and collects 34,608 question-answer pairs.The paper describes the dataset as a testbed for QA on informative conversations.Source: §1 · pp. 1–2
• Its collection uses segmented conversations, question-generation and summarization tools, crowdsourced QA, and quality verification.The paper also reports human evaluation of its question-generation approach.Source: §1 · pp. 1–2
• QAConv evaluates both chunk mode, where a conversational chunk is provided, and full mode, where retrieval precedes answering.The paper studies pretrained QA models in zero-shot and fine-tuning settings across both modes.Source: §1 · p. 2

2 QAConv Dataset

• Collection has four stages: selecting and segmenting informative conversations, generating question candidates, crowdsourcing QA pairs, and verifying quality before data splits.Source: §2 · p. 2

2.1.1 Selection and Segmentation

• Email, panel, and Slack conversations form the dataset; long dialogues are segmented below 512 tokens, while Slack threads are disentangled and then segmented or combined.Source: §2.1.1 · pp. 2–3

2.1.2 Question Generation

• A T5 question generator proposes questions from conversation chunks, while a BART summarizer supplies summaries on which the question generator also operates.Source: §2.1.2 · p. 3

2.1.3 Crowdsourcing QA Pairs

• Crowd workers either write QA pairs or answer recommended machine-generated questions, following constraints on question form, pronouns, answer length, and source wording.Source: §2.1.3 · p. 4

2.1.4 Quality Verification and Data Splits

• Uncertain QA pairs receive another crowd review using fuzzy answer matching; the final splits contain 27,287 training, 3,660 validation, and 3,661 test samples.The corresponding unanswerable shares are 4.7%, 5.1%, and 4.8%.Source: §2.1.4 · pp. 4–5

2.2.1 Question Analysis

• QAConv’s leading question types are what (29%) and which (27%), versus SQuAD 2.0’s 49% what; machine-generated questions were longer, more fluent, more complex, and drew lower answer confidence.Source: Table 3 · p. 4, §2.2.1 · p. 5, Table 4 · p. 5

2.2.2 Answer Analysis

• Noun phrases make up 30.4% of answers, followed by people, organizations, dates, other numeric answers, and countries; about 1% use multiple source spans.Source: §2.2.2 · p. 5, Table 5 · p. 6

2.3 Chunk Mode and Full Mode

• Chunk mode provides the collected conversational chunk, whereas full mode requires retrieval; email and panel searches stay within a known dialogue, while Slack searches cover the known channel.Source: §2.3 · pp. 5–6

3.1 State-of-the-art Baselines

• On Molweni, these baselines outperform DADgraph (Li et al., 2021a), which uses expensive discourse annotation and graph neural networks.The experiments cover zero-shot and QAConv-fine-tuned settings in both modes.Source: §3.1 · p. 6

3.1.1 Span-based Models

• Span-based baselines include DistilBERT, BERT-Base/Large, and RoBERTa-Base/Large; the BERT-Large baseline uses whole-word masking.The paper reports that this BERT-Large version performs better than the token-masking version.Source: §3.1.1 · p. 6

3.1.2 Free-form Models

• UnifiedQA baselines use T5-Base, T5-Large, and T5-3B when fine-tuned, with T5-11B evaluated zero-shot.UnifiedQA was further fine-tuned on eight QA corpora spanning four question formats.Source: §3.1.2 · p. 6

3.1.3 Retrieval Models

• The full-mode retrieval comparison uses BM25 and DPR-wiki, respectively a term-frequency-weighted bag-of-words retriever and a BERT-based dense retriever.DPR-wiki was trained for open-domain QA retrieval.Source: §3.1.3 · p. 6

3.1.4 Computational Details

• Most experiments use two V100 GPUs, while T5-3B uses four A100 GPUs; the authors release hyperparameters and trained models for reproducibility.BERT-based models train for five epochs and T5 models for ten.Source: §3.1.4 · p. 6

3.2 Evaluation Metrics

• Evaluation reports exact match, token-overlap F1, and FZ-R, with answer normalization including punctuation removal, lowercasing, and number-word conversion.EM requires an exact normalized answer match; FZ-R uses Levenshtein distance.Source: §3.2 · pp. 6–7

3.3.1 Chunk Mode

• In chunk mode, UnifiedQA T5 models generally outperform BERT/RoBERTa zero-shot, and fine-tuning improves all baselines; BERT-Base gains 22.6 EM.Performance generally increases with UnifiedQA model size in the zero-shot setting.Source: §3.3.1 · p. 7, Table 6 · p. 7

3.3.2 Full Mode

• BM25 outperforms DPR-wiki on recall@k, yet full-mode EM falls by 18.2% for zero-shot UnifiedQA T5-3B and 22.2% for fine-tuned RoBERTa-Large.The authors identify serious error propagation in full mode.Source: Table 8 · p. 7, Table 7 · p. 7, §3.3.2 · p. 8

Answerability Errors

• Among answerable questions, UnifiedQA outperforms span-based models, but predicts answers even for unanswerable questions; span models tend to predict the task as unanswerable.Span-based models show high recall but low F1 on unanswerable questions in the binary answerability setting.Source: §4 · p. 8

Fine-tuning Errors

• Of the RoBERTa-Large test samples improved most by fine-tuning, 75% had initially been incorrectly predicted as unanswerable.After fine-tuning, reported errors include 35.5% what-question, 18.2% which-question, 12.1% how-question, and 10.3% who-question errors.Source: §4 · p. 8

Evaluation Errors

• In a manual audit of 100 low-FZ-R errors, 20% seemed potentially correct, while 31% had the right entity type and 38% had a different entity type.The authors say current metrics cannot measure semantic distances between different answers; 11% were wrongly classified as unanswerable.Source: §4 · p. 8

QA Dataset Types

• QA datasets include cloze-style, reading-comprehension, span-based, and open-domain formats, exemplified by CBT and Who-did-What, RACE and DREAM, SQuAD and MS MARCO, and WikiQA and Natural Question.Source: §5 · pp. 8–9

Conversational QA

• Conversational QA asks sequential questions grounded in short passages; CoQA (Reddy et al., 2019) and QuAC (Choi et al., 2018) are representative examples.FriendsQA uses chit-chat transcripts, while Molweni (Li et al., 2020) focuses on short Ubuntu conversations.Source: §5 · p. 9

Conversations as Knowledge Sources

• Dialogue-state tracking can be reframed as QA, and query-based meeting summarization such as QMSum treats conversations as databases for abstractive answering.The paper also names extracting user attributes from open-domain conversations as a potential application.Source: §5 · p. 9

6 Conclusion

• QAConv evaluates question answering on informative conversations in chunk mode with oracle partial conversations and full mode with retrieval.The dataset is intended as a testbed for future QA-on-conversations research.Source: §6 · p. 9
• State-of-the-art QA models show limited dialogue understanding and tend to predict answerable QA pairs as unanswerable.Source: §6 · p. 9

Ethical Considerations

• QAConv covers only a few background-conversation domains, so it should not be treated as universal across domains and scenarios.Source: Ethical Considerations · p. 9
• Some incorrect labels and biases may persist from the original dialogue datasets despite efforts to ensure quality and accuracy.Source: Ethical Considerations · p. 9
• Transcripts and QA pairs may contain speakers’ biased views, toxicity, and subjective opinions that could affect model training.Source: Ethical Considerations · p. 9
• The dataset uses publicly available transcripts under their guidelines; Media data is for research purposes only, not commercial use.Source: Ethical Considerations · p. 9

A.1 Dataset documentation and intended uses

• The authors follow the Datasheets for Datasets guideline in documenting the dataset.Source: §A.1 · p. 13

A.1.1 Motivation

• QAConv is designed for QA on informative conversations such as business emails, panel discussions, and work channels.Source: §A.1.1 · p. 13
• The dataset is intended to fill the gap left by common Wikipedia-based QA tasks.Source: §A.1.1 · p. 13

A.1.2 Composition

• 34,608 QA pairs span 10,259 conversations, averaging 568.8 words each, with the longest conversation containing 19,917 words.Source: §A.1.2 · p. 13
• Conversations are randomly sampled from BC3, Enron, Court, Media, and Slack, with sample counts set based on related work and budget.Source: §A.1.2 · p. 13
• Each sample contains raw conversation text, speaker names, and QA pairs; answerable samples have at least one possible answer in list format.Source: §A.1.2 · p. 13
• Crowd-worker information is omitted because of potential privacy issues.Source: §A.1.2 · p. 13
• The dataset is self-contained, though question or answer annotations may contain potential noise.Source: §A.1.2 · p. 14

A.1.3 Collection Process

• Crowdworkers collected directly observable QA data on Mechanical Turk using instructions and partial-conversation context; payment was 10 per hour, calculated from an approximately four-minute average to read and write one QA pair.Source: §A.1.3 · pp. 14–15

A.1.4 Preprocessing/cleaning/labeling

• Before crowdworkers wrote QA pairs, the team removed code snippets; no further cleaning was done for release beyond format-conversion reading scripts.The raw data was reported as available at the same link as the dataset.Source: §A.1.4 · p. 15

A.1.5 Uses

• QAConv is proposed for conversational QA and can also support conversational retrieval and conversational machine reading.The authors use and release buffer-based chunking, noting that conversation disentanglement can affect performance.Source: §A.1.5 · p. 16

A.1.6 Distribution

• The dataset is released on GitHub under a BSD 3-Clause license, while Media conversations are restricted to research-only use.No DOI is specified.Source: §A.1.6 · p. 16

A.1.7 Maintenance

• Salesforce AI Research maintains the dataset; updates are not planned, but future versions would preserve the original, and contributions can be submitted by GitHub pull request.Users may contact the authors or open a GitHub issue.Source: §A.1.7 · pp. 16–17

A.2 Test Data Additional Verification

• For dev and test answers, FZR scores above 90 kept the original, scores between 75 and 90 kept both, and lower scores triggered a third answer and possible manual review.When the best two answers scored above 75 for similarity, they were selected as gold; otherwise, the QA pair was manually judged.Source: §A.2 · p. 17

A.3 License and Privacy

• Source terms vary: BC3 and Enron list Creative Commons licenses, Court includes an NSF funding disclaimer, and Media includes only publicly available transcripts.The appendix also describes public Slack channels focused on software-engineering discussions.Source: §A.3 · p. 17

A.4 Human evaluation description of human-written and machine-generated questions.

• Human evaluation rates question fluency, answer-finding complexity, and confidence in the answer.The rubric distinguishes fluent, somewhat fluent, and difficult-to-understand questions; answer search requiring repeated whole-conversation reading, several sentences, or one sentence; and confident versus not-confident answers.Source: §A.4 · p. 17

• Three screenshots are captioned as showing human-written QA collection, machine-generated QA collection, and QA verification.Source: Figure 4 · p. 22, Figure 5 · p. 22, Figure 6 · p. 23