Source-linked AI summary
A Graph-guided Multi-round Retrieval Method for Conversational Open-domain Question Answering
Yongqi Li, Wenjie Li, Liqiang Nie
TL;DR
Conversational open-domain QA must retrieve Web passages and extract exact answers, yet prior context modeling overlooks relations between history answers and current answers. This paper combines a hyperlink-based passage graph with multi-round relevance feedback to model those relations and refine questions; it reports 5% and 11% performance improvements over baselines with predicted and true history answers, respectively.
Problem
Conversational open-domain QA must retrieve relevant Web passages and extract exact answers, while prior approaches to historical context overlook relations between history answers and current answers.
Source: §1 · pp. 1–2Method
The method combines a graph-based Explorer that expands passages through hyperlinks with DHM, which uses retrieved passages as relevance feedback to refine question understanding across rounds.
Source: §1 · p. 2Results
5% and 11% performance improvement over baselines is reported with predicted and true history answers, respectively.
Source: §1 · p. 2Takeaways & Limitations
The graph-guided approach targets retrieval coverage by finding passages related to history-answer passages for subsequent answer extraction.
Source: §1 · p. 2The passage graph uses Wikipedia hyperlinks; the authors identify non-Wikipedia passages as a practical concern and plan to explore knowledge graphs.
Source: §5 · p. 10
Abstract
from arXiv · showhide
In recent years, conversational agents have provided a natural and convenient access to useful information in people's daily life, along with a broad and new research topic, conversational question answering (QA). Among the popular conversational QA tasks, conversational open-domain QA, which requires to retrieve relevant passages from the Web to extract exact answers, is more practical but less studied. The main challenge is how to well capture and fully explore the historical context in conversation to facilitate effective large-scale retrieval. The current work mainly utilizes history questions to refine the current question or to enhance its representation, yet the relations between history answers and the current answer in a conversation, which is also critical to the task, are totally neglected. To address this problem, we propose a novel graph-guided retrieval method to model the relations among answers across conversation turns. In particular, it utilizes a passage graph derived from the hyperlink-connected passages that contains history answers and potential current answers, to retrieve more relevant passages for subsequent answer extraction. Moreover, in order to collect more complementary information in the historical context, we also propose to incorporate the multi-round relevance feedback technique to explore the impact of the retrieval context on current question understanding. Experimental results on the public dataset verify the effectiveness of our proposed method. Notably, the F1 score is improved by 5% and 11% with predicted history answers and true history answers, respectively.
1 INTRODUCTION
Conversational open-domain QA must retrieve Web passages and extract exact answers while resolving questions dependent on dialogue history. The introduction proposes graph-guided retrieval and relevance feedback to address gaps in modeling cross-turn answer relations and using retrieval context.
1 INTRODUCTION
- Conversational open-domain QA retrieves relevant Web passages and extracts answer spans, making it practical but challenging for historical, elliptical, and unanswerable questions.The introduction situates it among conversational QA tasks with different emphases.Source: §1 · p. 1
- Explorer expands passages containing history answers and retrieved passages through hyperlinks, then uses a passage graph to retrieve candidates for answer extraction.The graph models relations among answers across conversation turns; the introduction reports that about 60% of QBLink answers can be found in two-hop connected passages containing history answers.Source: §1 · p. 2
- DHM uses retrieved passages as relevance feedback to select useful historical context and iteratively refine the current-question representation.The introduction notes that relevance feedback's impact on historical-context modeling in conversational QA had not yet been studied.Source: §1 · p. 2
- 5% and 11% performance improvement over baselines is reported with predicted and true history answers, respectively.The contribution summary reports both settings as experimental results on a public dataset.Source: §1 · p. 2
2 RELATED WORK
Related work spans conversational QA and open-domain QA. It reviews approaches to historical-context modeling, standard retrieval pipelines, and graph-based or iterative retrieval, distinguishing this work's focus on cross-turn answer relations and retrieval-informed history selection.
2.1 Conversational QA
- Prior conversational QA methods reformulate questions or refine their representations using history, but the reviewed approaches overlook relations among answers or answer passages across turns.The section contrasts this work's explicit passage-graph approach with methods using intermediate representations to capture conversational flow.Source: §2.1 · p. 3
2.2 Open-domain QA
- Conversational open-domain QA retrieves Web passages and extracts answer spans, commonly using a retriever–ranker–reader pipeline; this work applies dense and TF-IDF retrievers.The section describes DrQA's retriever and reader and a later addition of a ranker.Source: §2.2 · p. 3
- Prior graph retrievers address passage encoding or reasoning paths, while prior multi-round methods retrieve iteratively; this work instead explores conversation flow and uses returned passages to select useful history.The section describes feedback from a retriever or reader in earlier multi-round methods.Source: §2.2 · p. 3
3 OUR PROPOSED METHOD
The pipeline combines dynamic history modeling with dense retrieval, graph expansion, reranking, and answer-span extraction. Retrieved passages also feed back into question representations across retrieval rounds.
3.1 DHM and Retriever
- DHM first appends all history questions, then repeatedly uses retrieved passages to attend over history and refine the current-question vector for retrieval.The initial retrieval uses the current question with the appended history questions; later rounds incorporate retrieved passages into the history-question representations.Source: §3.1 · p. 4
- Passage vectors are encoded independently of questions, enabling precomputed representations and efficient maximum inner product search for retrieving top-ranked passages.The passage encoder is ALBERT-based, and passage and question vectors have equal dimensions.Source: §3.1 · p. 4
3.2 Explorer
- Explorer seeds a hyperlink graph with history-answer passages and dense- and TF-IDF-retrieved passages, expands linked nodes, applies graph attention, and returns top-n2 passages by question similarity.History answers may be gold or predicted, depending on whether true history answers are allowed.Source: §3 · p. 3, §3.2 · pp. 4–5
3.3 Ranker and Reader
- Ranker listwise reranks Explorer’s top-n2 passages, and Reader scores token start and end positions to extract the highest-scoring answer span.The reranker uses the question-passage sequence; the Reader predicts span boundaries in the Explorer passages.Source: §3.3 · p. 5
3.4 Training and Inference
- Training fixes passage-encoder parameters after offline encoding, manually adds missing gold passages to retriever results, and stages training with the retriever, ranker, and reader first.DHM and Explorer are trained separately afterward because of GPU-memory limitations.Source: §3.4 · p. 5
- At inference, the model considers only the top 20 spans, discards invalid predictions, and outputs the span with the maximum overall score.Invalid predictions include spans whose start follows their end or that overlap a conversation-context question.Source: §3.4 · pp. 5–6
4 EXPERIMENTS
Experiments on OR-QuAC compare the proposed system with conversational and single-turn baselines, then assess retrieval, component contributions, and case examples.
4.1 Dataset
- OR-QuAC contains 5,644 conversations and 40,527 questions, with about 11 million passages from an October 2019 English Wikipedia dump.Wikipedia internal links were crawled to construct the passage graph.Source: §4.1 · p. 6
4.2 Experimental Settings
- Evaluation uses word-level F1 and HEQ-Q/HEQ-D for QA, plus MRR and Recall for retrieval components.HEQ measures the percentage of examples where system F1 matches or exceeds human F1, at question and dialogue levels.Source: §4.2 · p. 6
4.3 Baselines
- The comparison includes DrQA, BERTserini, ORConvQA, and ORConvQA without history; the latter retains the first question in its reformulation.DrQA uses TF-IDF retrieval and an RNN reader, while ORConvQA uses a dense retriever, ranker, and reader.Source: §4.3 · p. 6
4.4 Overall Comparison
- 0.84 recall is reported for Ours-ta, which significantly outperforms ORConvQA-ta; overall, Ours achieves the best performance among baselines.Table 2 compares performance on development and test sets; the reported true-history comparison concerns recall.Source: §4.4 · pp. 6–7, Table 2 · p. 7
4.5 Component-wise Evaluation
- 0.3 F1 is lost without the Ranker and 0.8 F1 gained over w/o DHM; Explorer removal hurts most, while joint training can improve multiple components.The authors also report lower retriever performance without the Ranker and attribute cross-component gains to joint training.Source: §4.5 · pp. 7–8, Table 3 · p. 8
4.6 Additional Analyses
- Retriever seeds help until excessive seeds hurt; TF-IDF seeds generally raise F1 after an initial 0-to-1 decline, while more retrieval rounds do not monotonically improve performance.The authors attribute the round-count pattern partly to not tuning rounds during training and to possible noise or error propagation.Source: Figure 6 · p. 8, §4.6 · p. 8
4.7 Case study
- Case examples show answer-dependent references sometimes require history answers; attention weights are not strongly discriminative, while later retrieval rounds or the Explorer can recover missed gold passages.The Explorer sometimes retrieves a gold passage missed by both retrieval rounds, though its result may rank it less prominently than the Retriever does.Source: Table 4 · p. 9, §4.7 · p. 9
5 CONCLUSION AND FUTURE WORK
The paper combines dynamic history modeling and graph-guided multi-round retrieval to exploit answer relations and retrieval context in conversational open-domain QA. Experiments on a public dataset demonstrate effectiveness, while the authors identify data, hyperlink, and complex-question directions for future work.
5 CONCLUSION AND FUTURE WORK
- The method combines dynamic history modeling, Retriever, Explorer, Ranker, and Reader to exploit answer relations across turns and dynamically attend to useful historical information.The Explorer uses a passage graph, while relevance feedback informs multi-round retrieval based on dynamic question embeddings.Source: §5 · p. 9
- Experiments on a public dataset demonstrate the effectiveness of the proposed method.Source: §5 · p. 9
- The authors plan to extend passage-graph construction beyond Wikipedia hyperlinks by introducing knowledge graphs when hyperlinks are hard to obtain.Source: §5 · p. 10
- The authors call for a dataset better suited to real information needs in conversational open-domain QA, noting differences from conversational MCR.Source: §5 · p. 10
- The authors identify complex questions in conversational QA as an area for future work.Source: §5 · p. 10