PaperlayerView exact revision on arXiv

Source-linked AI summary

TopiOCQA: Open-domain Conversational Question Answering with Topic Switching

Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, Siva Reddy

arXiv:2110.00768v3cs.CL

TL;DR

Conversational QA datasets have generally assumed a known reference document and a single topic, despite real information-seeking conversations switching among related topics. TOPIOCQA addresses this gap with a Wikipedia-based open-domain dataset of free-form, multi-topic conversations and retriever-reader baselines. Its best model reaches 55.8 F1, 14.2 points below human performance, underscoring the benchmark's difficulty.

  • Problem

    Conversational QA datasets commonly assume a known reference document and omit topic switches, unlike real information-seeking conversations.

  • Method

    TOPIOCQA constructs open-domain Wikipedia conversations in which questioners shift among related documents while answerers provide free-form responses, evaluated with retriever-reader baselines.

  • Results

    55.8 F1 is achieved by the best model, which remains 14.2 points below human performance on the test set.

  • Takeaways & Limitations

    TOPIOCQA provides a challenging benchmark for conversational agents that must retrieve relevant documents across topic switches and use conversational history.

  • Takeaways & Limitations

    The reported analysis examines whether DPR Reader's extractive nature constrains performance, framing reader design as an evaluated boundary.

Abstract

from arXiv · show

In a conversational question answering scenario, a questioner seeks to extract information about a topic through a series of interdependent questions and answers. As the conversation progresses, they may switch to related topics, a phenomenon commonly observed in information-seeking search sessions. However, current datasets for conversational question answering are limiting in two ways: 1) they do not contain topic switches; and 2) they assume the reference text for the conversation is given, i.e., the setting is not open-domain. We introduce TopiOCQA (pronounced Tapioca), an open-domain conversational dataset with topic switches on Wikipedia. TopiOCQA contains 3,920 conversations with information-seeking questions and free-form answers. On average, a conversation in our dataset spans 13 question-answer turns and involves four topics (documents). TopiOCQA poses a challenging test-bed for models, where efficient retrieval is required on multiple turns of the same conversation, in conjunction with constructing valid responses using conversational history. We evaluate several baselines, by combining state-of-the-art document retrieval methods with neural reader models. Our best model achieves F1 of 55.8, falling short of human performance by 14.2 points, indicating the difficulty of our dataset. Our dataset and code is available at https://mcgill-nlp.github.io/topiocqa

1 Introduction

TOPIOCQA addresses the gap between single-topic, reference-document CQA and real information-seeking conversations that require open-domain retrieval and topic switching. It introduces a Wikipedia-based dataset with free-form answers, conversational history, and multiple related topics.

  • Motivation: Information-seeking sessions commonly switch topics, motivating datasets that measure conversational models under this behavior.Prior search-session data reports one to ten topics and a mean of 2.11 topic switches per session.
  • Motivation: TOPIOCQA targets CQA's two limitations: reference documents are not known beforehand, and conversations are not restricted to one topic.Existing open-domain adaptations still retain single-topic question sequences because their conversations were collected around given reference texts.
  • Dataset: TOPIOCQA starts from a Natural Questions seed topic and lets questioners shift to related Wikipedia documents as the conversation progresses.The dataset treats each Wikipedia document as a separate topic and simulates information seeking without exposing document content to the questioner.
  • Dataset: The conversations include free-form answers and complex coreference, requiring responses that may not be extractive spans from the visited documents.For example, later questions can depend on entities mentioned across several earlier answers and questions.
  • Dataset: TOPIOCQA contains 3,920 conversations and 50,574 QA pairs from 5.9 million Wikipedia documents, averaging 13 turns and 4 topics per conversation.Twenty-eight percent of turns require retrieving a document different from the previous turn.
  • Evaluation: The benchmark combines document retrieval with neural readers and tests both full conversation history and self-contained question rewrites.The strongest reported approach uses a Fusion-in-Decoder reader with concatenated conversation history.

2 Related Work

Related work frames open-domain CQA as a retriever-reader problem extended across conversational turns. Existing datasets and methods largely adapt single-topic CQA, motivating TOPIOCQA's explicit coverage of topical switches.

  • Open-Domain QA: Open-domain QA commonly retrieves relevant documents from a large corpus before a reader extracts or generates an answer.Retrievers may use sparse representations such as BM25 or learned dense representations.
  • Conversational QA: Conversational QA extends reading comprehension to multiple questions about a reference document, introducing anaphora, ellipsis, and pragmatic reasoning challenges.CQA systems have explored history concatenation, selected history turns, and question rewriting.
  • Conversational QA: Question rewriting converts context-dependent questions into self-contained questions that can be answered by open-domain QA systems.CANARD and QReCC provide rewrite data for this approach.
  • Open-Domain CQA: TOPIOCQA introduces open-domain CQA that explicitly covers topical switches, providing a benchmark for conversations spanning several topics.Its design combines the open-domain requirement with multi-turn conversational interaction.
  • Open-Domain CQA: Existing open-domain CQA datasets mainly repurpose conversations collected around a given reference document, so their question sequences remain centered on one topic.OR-QuAC and much of QReCC derive from QuAC, while QReCC's NQ-based conversations are created by a single annotator.

3 Dataset Collection

TOPIOCQA collects information-seeking conversations using Wikipedia documents, with separate questioner and answerer roles and hyperlink-enabled topic navigation. The interface withholds full document text from questioners while allowing answerers to locate rationales and generate free-form responses.

  • 3 Dataset Collection: Each conversation is an interaction between two annotators serving as the questioner and answerer.The collection process therefore separates question generation from document-based answering.
  • 3.1 Seed topics and document collection: Wikipedia provides the 5.9-million-document corpus, with hyperlinks retained so answerers can access documents requested during conversations.Seed topics are selected from around 35,000 Wikipedia Good articles, while later turns can use the full corpus.
  • 3.2 Data collection interface: Questioners see only document metadata, whereas answerers access full text, highlight rationales, generate free-form answers, or mark questions unanswerable.This setup is intended to simulate information-seeking without exposing the reference text to the questioner.
  • 3.2 Data collection interface: The first question comes from Natural Questions when answerable in the seed pool, while 28% of conversations instead begin from a questioner-written seed-topic question.This combines genuine search questions with additional diversity from seed-entity prompts.
  • 3.3 Enabling topic-switching: Hyperlinks in answerer documents enable switching to related documents, whose metadata and rationale-linked documents become available for subsequent question formation.For example, a question about London can move retrieval from Daniel Radcliffe to London through a hyperlink.

4 Dataset Analysis

TOPIOCQA conversations combine multiple Wikipedia topics, typically staying within one topic for several turns before switching, often to new or previously visited topics. The analysis also shows that specific-attribute questions dominate, while generic questions and longer answers are associated with new-topic transitions.

  • Number of topics vs conversation length: Longer conversations usually include more topics: most 10-turn conversations have 3, 14-turn conversations have 4, and 18-turn conversations have 5.Conversations shorter than 10 turns mostly contain 2 topics.
  • Topic flow in conversation: Topic clusters most often last 3 turns, with 2 or 4 consecutive questions also frequent and more than 10 turns rarely remaining on one topic.Clusters group consecutive turns whose answers come from the same document.
  • Topic flow in conversation: Most topic switches introduce new topics, but from the sixth turn onward questioners frequently return to topics seen one or two turns earlier.The later-turn flow therefore exhibits more complex interactions among topics.
  • Qualitative assessment of topic switching: Ask-specific questions are most common across turn types, while ask-generic questions are more prevalent when a turn switches to a new topic.The analysis manually labels 50 turns of each type; generic questions also tend to have longer answers.

5 Experimental Setup

The experiments frame open-domain conversational QA as answering the current question from conversation history, optionally using a document corpus. They compare closed-book and retriever-reader systems across question representations, retrieval methods, readers, and standard QA metrics.

  • Task formulation: The task gives previous questions and gold answers plus the current question, and asks the model to produce the current answer, optionally using a document corpus.Because previous answers are supplied, this is an oracle setting for conversational history.
  • Model categories: Closed-book models use only question-answer history, whereas open-book models additionally use a document corpus.The setup compares models according to whether external documents are available.
  • Retrieval and reading: Open-book systems use a two-step retriever-reader pipeline with BM25 or DPR retrieval and either a DPR Reader or Fusion-in-Decoder reader.BM25 uses bag-of-words scoring, DPR uses dense query-document vectors, and FiD jointly attends over retrieved passages during decoding.
  • Question representations: Models receive the current question as ORIGINAL, full conversational context as ALLHISTORY, or a decontextualized REWRITES query generated from the conversation.ALLHISTORY is truncated to fit encoder limits when necessary.
  • Evaluation: Evaluation uses exact match and F1, with human and system scores averaged over multiple reference-answer sets.TOPIOCQA supplies four annotations per answer: the original answer and three additional annotations.

6 Results and Discussion

TOPIOCQA remains difficult for neural retriever-reader systems: the best test-set model reaches 55.8 F1, while retrieval and reader quality both limit performance.

  • Closed-book versus open-book models: GPT-3 reaches 31.8 F1 on the test set, 24 points below the best open-book model, and often hallucinates answers.
  • Question representations: ALLHISTORY outperforms REWRITES with DPR, whereas REWRITES performs better with BM25, indicating retriever-dependent effects of question representation.
  • Reader limitations: With gold passages, FiD with ALLHISTORY still trails humans by 3.1 EM and 5.9 F1, showing that readers also need improvement beyond retrieval.
  • Reader limitations: The extractive upper bound is 47.3 EM and 81.0 F1, while DPR Reader falls short by 19.2 EM and 28.4 F1.

7 Conclusion

The paper introduces TOPIOCQA as an open-domain conversational QA dataset with topic switching and establishes neural baselines that remain below human performance.

  • TOPIOCQA combines open-domain conversational question answering with topic switching, providing a benchmark for information-seeking conversations.
  • The best DPR retriever + FiD model remains 6.9 EM and 14.2 F1 below human performance, suggesting that modeling advances are needed.

A Annotators Details

TOPIOCQA conversations were produced by trained questioner-answerer annotator pairs following guidelines designed to elicit contextual, multi-document exchanges and grounded answers.

  • Each conversation is an interaction between two annotators, a questioner and an answerer, selected and trained for both roles.
  • Questioners were instructed to begin with an unambiguous seed-entity question, contextualize follow-ups, and involve multiple documents.
  • Answerers had to identify the relevant document and section, ground answers in its contents, and avoid personal opinions.
  • Authors manually inspected early conversations, disqualified low-quality workers, and used 15 workers with ongoing checks and feedback.
  • Real-time interface checks encouraged topic switching and coreference while reducing lexical overlap with document metadata.

B Query Rewriting

The query-rewriting module converts conversational questions into decontextualized questions using conversation history, but its externally trained model struggles with complex later-turn references and reasoning.

  • QR takes the current question and conversation history as input and outputs a decontextualized rewritten question.
  • Because TOPIOCQA contains no rewrites, QR is trained on QReCC with T5 and used to generate rewrites for TOPIOCQA.
  • The QReCC-trained model achieves 62.74 BLEU on its test split before generating TOPIOCQA rewrites.
  • QR resolves simple coreferences but struggles with multiple entities and reasoning-dependent rewrites later in conversations.

C Hyperparameter Details

The section specifies training and retrieval settings, alongside examples illustrating multi-turn conversations, rewrites, topic continuity, and topic shifts.

  • C Hyperparameter Details: BM25 uses k1 = 0.9 and b = 0.4, while DPR and FiD use codebase-suggested hyperparameters and base model sizes.DPR Retriever uses four 40GB A100 GPUs, whereas the DPR Reader and FiD use eight 32GB V100 GPUs.
  • Conversation Examples: The examples include both answerable and unanswerable turns, including an unanswerable question about the first monument mentioned.Other turns provide the monument’s dedicatee, materials, and height.
  • Conversation Examples: The examples follow conversations that connect questions and answers across books, characters, empires, cities, and monuments.Examples include the Harry Potter series, the Byzantine Empire, Turkey, Ankara, Istanbul, and the Column of Constantine.
Loading 2110.00768v3…