PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Approaches to Conversational Information Retrieval

Jianfeng Gao, Chenyan Xiong, Paul Bennett, Nick Craswell

arXiv:2201.05176v1cs.IRcs.CL

TL;DR

CIR enables multi-turn natural-language information seeking, while commercial search engines still offer limited dialog. The passages describe hierarchical dialog management and report SC-GPT's state-of-the-art result on a public dialog-response-generation benchmark.

  • Problem

    Commercial search engines offer limited dialog, and coordination of CIR modules as an intelligent whole remains largely unexplored.

    Source: Introduction · p. 6, §1.5 · p. 14

  • Method

    CIR dialog management tracks the current dialog state and uses a hierarchical policy to select action modules and generate responses.

    Source: CIR Engine Layer · p. 12

  • Results

    SC-GPT achieves a new state of the art on the public dialog-response-generation benchmark, outperforming corpus-based methods including GPT-2 and SC-LSTM.

    Source: §6.6.1 · p. 98

  • Takeaways & Limitations

    Query-biased summaries significantly improve the accuracy and speed of users' relevance judgments compared with full texts and static summaries.

    Source: §1.6.3 · p. 18

    TREC CAsT's 100 topics with about ten queries each provide too few labels to reliably train large deep models from scratch using supervised learning.

    Source: §3.2.1 · p. 35

Abstract

from arXiv · show

A conversational information retrieval (CIR) system is an information retrieval (IR) system with a conversational interface which allows users to interact with the system to seek information via multi-turn conversations of natural language, in spoken or written form. Recent progress in deep learning has brought tremendous improvements in natural language processing (NLP) and conversational AI, leading to a plethora of commercial conversational services that allow naturally spoken and typed interaction, increasing the need for more human-centric interactions in IR. As a result, we have witnessed a resurgent interest in developing modern CIR systems in both research communities and industry. This book surveys recent advances in CIR, focusing on neural approaches that have been developed in the last few years. This book is based on the authors' tutorial at SIGIR'2020 (Gao et al., 2020b), with IR and NLP communities as the primary target audience. However, audiences with other background, such as machine learning and human-computer interaction, will also find it an accessible introduction to CIR. We hope that this book will prove a valuable resource for students, researchers, and software developers. This manuscript is a working draft. Comments are welcome.

Introduction

The introduction situates renewed interest in conversational information retrieval amid advances in deep learning, NLP, and conversational AI, and frames a survey of neural CIR approaches alongside CIR’s modular design, interactive-search foundations, and the challenge of coordinating modules around users’ changing goals and needs.

Introduction

  • The book surveys neural CIR advances from the last decade, building on the authors’ SIGIR 2020 tutorial and targeting IR and NLP readers as well as related fields.It is intended as an accessible resource for students, researchers, and software developers.Source: Introduction · p. 6

1.1 Book Organization

  • Chapter 2 discusses system-oriented evaluation on fixed datasets, user-oriented evaluation of real-user interactions, CIR simulation, and responsible CIR; Chapters 3–7 describe algorithms and methods for main CIR modules.Chapter 8 reviews commercial CIR systems and related tasks; Chapter 9 concludes with research trends and future work.Source: §1.1 · pp. 6–7

1.2.1 Information Seeking Tasks

  • Information seeking includes lookup and exploratory tasks; complex searches require sustained interaction, and CIR systems in the book mainly address these tasks.Exploratory searches involve iterative sensemaking, such as learning from multiple sources or investigating topics over time.Source: §1.2.1 · p. 7

1.2.2 Information Seeking Models

  • Radlinski and Craswell [2017] characterize conversational IR through user and system revealment, mixed initiative, memory, and set retrieval.These properties cover expressing information needs, revealing system capabilities, allowing either party to take initiative, retaining conversational context, and reasoning over complementary items.Source: §1.2.2 · p. 8

1.3 CIR as Task-Oriented Dialog

  • Most Web search engines leave users to iterate queries and inspect results, motivating CIR research that supports multi-turn interaction and a more active system role.The book focuses on neural methods for tracking intent, clarifying requests, recommending content, and handling natural-language input and output.Source: §1.3 · p. 10

1.4 CIR System Architecture

  • The reference architecture extends task-oriented dialog systems and Web search to open-domain information seeking, explicitly modeling multi-turn conversations across three layers.Because users can issue free-form queries, CIR systems cannot predefine every action and instead group actions into classes such as clarification and document search.Source: §1.4 · pp. 10–11

CIR Engine Layer

  • CIR engine modules combine contextual query rewriting, dialog-state tracking and hierarchical policies, and actions for retrieval, clarification, suggestions, topic shifts, and result generation.The dialog manager may personalize results and must address responsible AI concerns, including harms, toxicity, and bias.Source: CIR Engine Layer · pp. 12–13

User Experience Layer; Data Layer

  • The user-experience layer handles multimodal input and device-appropriate result presentation, while the data layer supplies indexed documents, knowledge bases, optional user profiles, and search-session logs.Interfaces may present concise answers or summaries on voice and mobile devices, or a search-results page on desktop.Source: User Experience Layer · p. 13, Data Layer · p. 13

1.4.1 An Example: Macaw

  • Macaw [Zamani and Craswell, 2020] is a modular, open-source CIR framework supporting multi-turn, multimodal, and mixed-initiative interaction across retrieval, question answering, recommendation, and structured-data exploration.Its modules run actions in parallel and select or combine outputs; researchers can add retrieval modules or interfaces and evaluate them in batch or interactive studies.Source: §1.4.1 · pp. 13–14, Figure 1.6 · p. 14

1.5 Remarks on Building an Intelligent CIR System

  • CIR’s central challenge is coordinating individually improved modules into a dialog manager that chooses actions for users’ open-domain goals and adapts efficiently as interests and world knowledge change.Optimizing modules in isolation is insufficient without knowledge of the user’s goal; adaptation must use few experiences because exploration can impose real-world costs.Source: §1.5 · pp. 14–15

1.6.1 System-Oriented and User-Oriented IR Research

  • System-oriented IR focuses on text representation, queries, and retrieval techniques, whereas user-oriented IR addresses information problems, interaction, interface design, and query reformulation.The book’s CIR methods aim to unify these complementary perspectives using neural approaches.Source: §1.6.1 · pp. 15–16

1.6.2 System Architecture

  • Early CIR designs drew on human search tactics and interaction studies, including modular intermediary functions, user-directed search facilities, and dialogs that help specify information needs.The early literature also considered user effort in interactive search and proposed a four-phase interface framework of formulation, action, review, and refinement.Source: §1.6.2 · pp. 16–17

1.6.3 Search Result Presentation

  • Query-biased summaries improve the accuracy and speed of relevance judgments, while visualizations and result clustering help users scan and navigate retrieved documents.Clustering must be fast, group topically similar documents, and provide concise, accurate descriptions; query-biased summaries reduce the need to consult full documents.Source: §1.6.3 · pp. 17–18

1.6.4 Relevance Feedback Interactions

  • Relevance feedback supports iterative retrieval revision through query refinement or ranking-model updates, and click data trains neural ranking models to favor clicked documents.Query refinement can track evolving information needs; DSSMs use contrastive learning with click data to rank clicked documents above unclicked ones.Source: §1.6.4 · pp. 19–20

1.6.5 Exploratory Search

  • Exploratory search requires intensive interaction that helps users learn and understand, with systems supporting query refinement, filtering, context, visualizations, collaboration, histories, progress, and task management.These design features are presented as fundamental to CIR systems.Source: §1.6.5 · pp. 20–21

Evaluating Conversational Information Retrieval

This section surveys how conversational information retrieval is evaluated, from component actions and fixed-dataset tests to users’ overall experience in real interactions, including logs, panels, and online experiments. It also covers user simulation and responsible-system assessment, considering task-aligned engagement measures, open-domain needs, and impacts on users and content.

2.1 Forms of Evaluation

  • System-oriented evaluation tests components against fixed datasets, while user-oriented evaluation observes real interactions through lab studies, field studies, or logs.The approaches inform each other: real-user observations can ground datasets and improve their external validity, while dataset-based evaluations can support system development.Source: §2.1 · p. 23

2.2 System-oriented Evaluation

  • System-oriented evaluation supports reproducible comparisons on reusable datasets, but results may have little real-world validity when content, queries, or annotations miss actual user preferences.Repeated test-set use can also cause overfitting; suggested safeguards include tracked leaderboards, hidden single-shot test annotations, and evaluation across multiple datasets.Source: §2.2 · p. 23, §2.2.1 · p. 24

2.2.1 Evaluating retrieval

  • System-oriented retrieval evaluation reuses fixed test collections and metrics such as NDCG or answer-span exact match and F1, but datasets may not reflect real applications or invite overfitting.NDCG rewards valuable results at higher ranks; answer extraction is evaluated against annotated spans, and systems can be compared on the same dataset.Source: §2.2.1 · p. 24

2.2.2 Evaluating Retrieval in Conversational Context

  • Conversational retrieval datasets evaluate queries with prior-turn context and context-aware judgments, but shared histories make their ranking tests off-policy rather than tests of a system’s own evolving behavior.TREC CAsT [Dalton et al., 2020] adds conversational context to an IR test collection; conversational datasets can also annotate shorter answers or passages suited to presentation.Source: §2.2.2 · p. 25

2.2.3 Evaluating Non-retrieval Components

  • Non-retrieval components include query augmentation, clarification, and summarization; their evaluation can use retrieval measures, human judgments, reference-text similarity, or learned metrics, each with limitations.Reference clarifications may unfairly penalize equally good alternatives, while reference-text and learned generation metrics can miss valid outputs or fail to simulate human judgment.Source: §2.2.3 · pp. 26–27

2.3 User-oriented Evaluation

  • User-oriented observation is needed to understand real-world needs and assess how a system performs across multi-turn interactions, beyond off-policy dataset evaluation.Even datasets built from real conversation traces may evaluate a new system using traces generated by a past system.Source: §2.3 · p. 27

2.3.1 Lab Studies

  • Lab studies found that CIR satisfaction depends on effort as well as task completion, and automatic systems struggled with complex information needs, context, and partially stated questions.In one study, self-reported satisfaction and success aligned for device control, but CIR satisfaction was more related to effort; another found automatic systems significantly worse than human alternatives on satisfaction, information finding, and quiz success. Users also combined feedback and questions in single utterances and could communicate presentation preferences.Source: §2.3.1 · p. 28

2.3.2 Scaling Up Evaluation

  • Large-scale panels and log analyses are proposed as important future steps for CIR evaluation, building on how search logs have revealed user intents and preferred responses.For example, Bing logs showed clicks on the Coca-Cola homepage were ten times more common than clicks on its Wikipedia page, suggesting stronger navigational intent.Source: §2.3.2 · pp. 28–29
  • Navigational, informational, and transactional queries account for roughly 20%, 50%, and 30% of queries, respectively, motivating log analysis to understand CIR usage and intents.Intent-aware evaluation can reflect these types of needs, as in the Navigational output category in the TREC Web Track.Source: §2.3.2 · p. 29
  • User panels can connect implicit behavior logs with explicit satisfaction, while requesting feedback only when a reward model is uncertain may reduce survey burden.The Web-search panel related dwell time to relevance, with short dwell sometimes indicating an irrelevant result.Source: §2.3.2 · p. 29
  • Online randomized trials compare system versions, and their overall evaluation criterion should align with long-term goals, cover the product, encourage diverse improvements, and remain sensitive and affordable.The passage describes establishing an overall evaluation criterion (OEC) as one important step in large-scale experimentation.Source: §2.3.2 · p. 29
  • CPS may reward longer chatbot sessions, whereas higher-quality Web search results are associated with shorter sessions, so long-term CIR experiments could test which short-term behaviors predict engagement [Zhou et al., 2020; Chapelle et al., 2012].Short-term CPS gains can undermine long-term liking, while task-completion functionality can reduce short-term CPS but potentially raise future engagement; Web search's task orientation may explain the metric contrast.Source: §2.3.2 · p. 29

2.4 Emerging Forms of Evaluation

  • CIR user simulation could enable multi-turn evaluation without real users; dialog simulators use scenario-based rule-driven agendas or trained sequence models, and recommendation simulators model user requests and preferences [Gao et al., 2019; Zhang and Balog, 2020].Real-user A/B testing may be unavailable to academics, while lab or field studies are expensive; rule-based simulators can encode domain constraints, and recommendation simulation uses action-labeled behavior and template generation.Source: §2.4 · p. 30
  • Open-domain CIR simulation remains challenging because information needs may range from structured booking or recommendation to free-form learning, beyond slot-filling or knowledge-graph paradigms.The passage calls for further progress in open-domain user simulation.Source: §2.4 · p. 30
  • Responsible CIR evaluation should address under-represented content and underserved user groups, harmful outputs, privacy, users' ability to refuse interaction, and transparency about system limitations.The passages identify inappropriate responses including hate speech and violence, and raise the broader challenge of evaluating and optimizing systems' impacts on people.Source: §2.4 · pp. 30–31

Conversational Search

This section surveys conversational search as document retrieval conditioned on dialog context, covering its task, benchmarks, and neural foundations. It also explains multi-head attention and how pre-trained language models transfer knowledge to downstream tasks through task-specific fine-tuning.

3.1 Task

  • Conversational search must infer intent from multi-turn context to retrieve relevant documents, while concise outputs and complex exploratory tasks create distinct demands and opportunities.Users can clarify intent through dialogue; systems may return up to three results or a short answer, rather than the “ten blue links” of ad hoc search.Source: §3.1 · pp. 32–34

3.2 Benchmarks

  • Conversational search benchmarks include TREC-style test collections and crowdsourced open-retrieval QA datasets, representing distinct construction approaches.TREC CAsT is built by the IR community; OR-QuAC extends QuAC for open retrieval using a passage collection and document-level relevance labels.Source: §3.2 · p. 34

3.2.1 TREC CAsT

  • TREC CAsT provides high-quality, reusable evaluation but has only 100 topics with about ten queries each, limiting supervised deep-learning training and potentially reflecting organizer bias.Its real-user-derived needs, expert-curated queries, and pooled relevance judgments support robust evaluation; its small label set makes it a few-shot benchmark.Source: §3.2.1 · pp. 35–36

3.2.2 OR-QuAC

  • OR-QuAC supplies tens of thousands of crowdsourced question-answer pairs and rewrites, but lower-quality conversations and answer-span relevance labels can misrepresent real search and relevance.Its constrained dialog patterns and same-section answer design may enable benchmark-specific shortcuts that do not generalize.Source: §3.2.2 · pp. 36–37

3.2.3 Other Related Resources

  • Other resources include open-domain QA and ad hoc search datasets used for weak supervision, plus QReCC, which adds conversations and Common Crawl passages to CAsT and QuAC.Examples include open-retrieval SQuAD, TriviaQA, Natural Questions, and MS MARCO.Source: conversational search · p. 38

3.3 Pre-Trained Language Models

  • Pre-trained language models have been adopted across nearly all conversational information retrieval components, and many recent advances are attributed to them.The chapter introduces BERT and the pre-training and fine-tuning framework for downstream applications.Source: §3.3 · p. 38

3.3.1 BERT: A Transformer-Based PLM

  • BERT uses a Transformer encoder that sums token, segment, and position embeddings, then applies stacked multi-head attention and feed-forward layers to contextualize input tokens.BERT uses absolute position embeddings, unlike the vanilla Transformer’s relative positions; DeBERTa uses disentangled position embeddings.Source: §3.3.1 · pp. 38–39
  • Multi-head attention applies parallel scaled dot-product attention to separately projected queries, keys, and values, then concatenates and projects the outputs [Vaswani et al., 2017].The learned projections produce multiple versions of the inputs before the parallel attention operations.Source: §3.3.1 · p. 39

3.3.2 The Pre-Training and Fine-Tuning Framework

  • BERT’s masked language modeling replaces 15% of input words with [MASK] and predicts them from context, enabling pre-training on raw text without manual labels.The text gives BERT base as a 12-layer, 110-million-parameter model pre-trained on Wikipedia and Google Book Corpora totaling 16 GB words [Devlin et al., 2018].Source: §3.3.2 · p. 40
  • Fine-tuning adapts a pre-trained model and task-specific head using task labels, with single prediction for one sequence, pair prediction for sequence pairs, and dual-encoding matching for similarity tasks.Examples include text classification, query-document relevance prediction, and document-search matching.Source: §3.3.2 · pp. 41–42
  • Dual encoding supports efficient retrieval by separately encoding queries and documents so documents can be pre-encoded and only the query encoded at retrieval time.The query-document relevance score uses their contextual representations and a similarity function such as cosine similarity.Source: §3.3.2 · p. 42

3.4 System Architecture

Conversational search uses a three-part pipeline: contextual query understanding, document retrieval, and document ranking. The chapter reviews approaches for making conversational queries standalone and retrieving documents with sparse lexical methods.

3.4 System Architecture

  • Conversational search ranks documents for each query using its dialog history, with contextual query understanding, retrieval, and ranking as the three components.Contextual understanding rewrites a query using its history; retrieval finds top-K candidates, and ranking refines that list.Source: §3.4 · pp. 42–43

3.5 Contextual Query Understanding

  • Contextual query understanding reformulates a conversational query using its history into a standalone query that can be retrieved without that history.Methods include query expansion and neural rewriting, or contextual understanding integrated into dense retrieval.Source: §3.5 · p. 43, §3.4 · p. 43

3.5.1 Heuristic Query Expansion Methods

  • Historical Query Expansion selects topic and subtopic keywords from the session, measures query ambiguity, then adds history terms according to the ambiguity category.It assumes the first query is well specified and that subtopics persist for a limited number of turns; whether these CAsT-derived assumptions generalize to real-world search is unclear.Source: §3.5.1 · p. 44

3.5.2 Machine Learning Based Query Expansion Methods

  • QuReTeC classifies each history term for expansion using a BERT-based binary classifier trained with distant-supervision term labels.A history term is labeled positive when it occurs in a relevant document but not in the current query; the procedure can be noisy, and its reported good performance on CAsT 2019 remains to be validated elsewhere.Source: §3.5.2 · p. 45, Generating Distance Supervision · p. 45

3.5.3 Neural Query Rewriting

  • Fine-tuned GPT-2 query rewriting became one of the best TREC CAsT 2019 performers and a standard solution for participants in 2020.The small number of manually generated CAsT rewrites motivates augmenting training with CANARD rewrites.Source: §3.5.3 · p. 46

Training Data Generation via Rules and Self-Supervised Learning

  • Machine-learning-generated weak-supervision data significantly boosts GPT-2 query rewriting on TREC CAsT benchmarks.The data are generated by simplifying ad hoc search sessions into conversation-like sessions.Source: Training Data Generation via Rules and… · p. 47

3.6 Sparse Document Retrieval

  • Sparse retrieval represents queries and documents as sparse vectors, scores their relevance, and returns the top-K documents, commonly using bag-of-words representations and BM25.BM25 uses term frequency, inverse document frequency, and document-length normalization; an inverted index supports efficient retrieval.Source: §3.6 · p. 47

3.7 Dense Document Retrieval

The chapter surveys neural dense retrieval, from dual encoders and approximate nearest-neighbor search to training and conversational retrieval. ConvDR significantly outperforms a pipeline approach on OR-QuAC when sufficient training data are available.

3.7 Dense Document Retrieval

  • Sparse lexical matching can miss semantically similar terms, while dense retrieval learns query and document representations from labeled pairs.Dense retrieval methods use dual encoders, approximate nearest-neighbor search, and search-log training to retrieve relevant documents.Source: §3.7 · p. 48

3.7.1 The Dual-Encoder Architecture

  • A dual encoder maps queries and documents to dense vectors and scores their similarity, allowing documents to be indexed in advance and queries encoded at runtime.Training optimizes query and document encoder parameters to increase relevance scores for relevant documents.Source: §3.7.1 · p. 48

3.7.2 Approximate Nearest Neighbor Search

  • Approximate nearest-neighbor search trades exactness for efficiency through vector compression or restricted index search, with larger search subsets increasing accuracy and latency.IVF probes nearby clusters, while HNSW traverses layered proximity graphs; adaptive termination can achieve the same accuracy with less total search than fixed configurations.Source: §3.7.2 · pp. 49–50, Figure 3.10 · p. 49

3.7.3 Model Training

  • Randomly sampled negatives are often too irrelevant to provide useful training information, and analysis finds this sampling strategy can yield inefficient learning or slow convergence.Sparse-model hard negatives offer an alternative but can push the dense model to mimic sparse retrieval; self-training instead samples negatives using the dense model being optimized.Source: §3.7.3 · p. 50

3.8 Conversational Dense Document Retrieval

  • ConvDR takes a conversational query and its history directly, using a query encoder that combines contextual understanding with dense retrieval and enables end-to-end retrieval training.This integrated approach avoids a separate, discrete query-rewriting stage.Source: §3.8 · p. 51
  • With sufficient query-context-answer training data, ConvDR significantly outperforms the pipeline approach on the OR-QuAC task.Source: §3.8 · p. 51

3.8.1 ConvDR Model Trainining

  • Few-shot ConvDR training distills a well-trained ad hoc dense retriever and combines knowledge-distillation and pairwise-ranking losses, relying on a few gold rewrites and labeled query-context-document pairs.Source: §3.8.1 · pp. 51–52

3.9 Document Ranking

  • Pair-prediction rankers can model query-document term matches but cannot score every document efficiently, whereas dual encoders support pre-encoding and efficient retrieval.The chapter presents a BERT ranker that predicts relevance from a concatenated query-document input.Source: §3.9 · p. 52

Query-Focused Multi-Document Summarization

Query-focused multi-document summarization (QMDS) generates concise summaries of retrieved documents that answer a user’s query and help users assess search results without reviewing each document. The section surveys extractive and abstractive approaches, query-relevant content selection and summary conditioning, datasets and computational challenges in conversational retrieval, and factuality evaluation using proxy tasks or extracted facts because overlap metrics do not detect factual inconsistencies.

4.1 Task and Datasets

  • QMDS summarizes retrieved documents to answer a specific query, helping users judge results and decide how to continue a conversational search.In the CIR formulation, the model receives the query, dialog history, and retrieved documents; the summary answers the query.Source: §4.1 · p. 53
  • The scarcity of large, high-quality QMDS data motivates simulated datasets; QMDS-IR uses real user queries and is described as the closest approximation to CIR.WikiSum and QMDS-CNN use simulated queries, with WikiSum’s queries and summaries noted as unnatural; QMDS-CNN uses manually written single-document summaries and retrieved paragraphs.Source: §4.1 · pp. 54–55

4.2.1 Extractive Summarizers

  • Extractive summarizers select source sentences, typically by representing sentences, scoring their importance, and selecting a nonredundant subset under a length limit.MMR balances sentence importance against overlap with the current summary to reduce redundancy.Source: §4.2 · p. 55, §4.2.1 · p. 55, Summary Sentence Selection · p. 56

4.2.2 Abstractive Summarizers

  • Abstractive summarizers rephrase source information; neural FFLMs and encoder-decoder RNNs generate summaries, and Nallapati et al. [2016] significantly outperform Rush et al. [2015] on DUC.The RNN approach combines linguistic features, a generator-pointer for copying out-of-vocabulary words, and hierarchical word- and sentence-level attention.Source: §4.2 · p. 55, §4.2.2 · p. 56, FFLMs · p. 56

Transformers

  • A hierarchical Transformer summarizer encodes within-document context locally and cross-document context globally before decoding a summary, and has been evaluated on WikiSum.Its global layer uses multi-head pooling and inter-paragraph attention to update token representations with document-level information.Source: Transformers · pp. 57–58

4.3.1 Extractive Methods

Coarse-to-Fine QMDS

  • Because retrieved document sets can be large, encoding every sentence for QMDS can delay responses and harm conversational user experience.Source: Coarse-to-Fine QMDS · p. 60
  • Coarse-to-fine QMDS progressively estimates query relevance, answer evidence, and summary centrality, discarding sentences at each stage to improve efficiency for large inputs and interactive CIR.Unlike classic methods that score all sentences simultaneously, its computational cost does not increase with the size of the original input D.Source: Coarse-to-Fine QMDS · p. 60, Figure 4.4 · p. 61

4.3.2 Abstractive Methods

  • Abstractive QMDS commonly selects a subset of documents extractively before an encoder-decoder generates a summary, addressing memory and latency constraints while conditioning generation on highlighted content.This two-stage process is described as simulating how people summarize multiple long documents: first highlighting pertinent information, then generating from those highlights.Source: §4.3.2 · pp. 60–61, Figure 4.5 · p. 61
  • HEROSumm extends the baseline [Liu and Lapata, 2019a] with query encoding and query-based document ordering to guide summary generation.Its query Transformer encodes query dependencies with document tokens and documents; document relevance scores are encoded as position embeddings, and query-context vectors are fed to the decoder with intra- and inter-document context.Source: §4.3.2 · pp. 61–63, Figure 4.6 · p. 62

4.4 Factuality Metrics for Summarization Evaluation

  • ROUGE measures n-gram overlap but not factuality; Kryściński et al. [2019] report that 30% of abstractive summaries contain factual inconsistencies from distorted or fabricated source facts.Factuality metrics are grouped into proxy-task methods, such as QA and masked-token prediction, and extracted-fact methods that assess consistency with source or reference facts.Source: §4.4 · p. 63, Figure 4.7 · p. 63
  • QA metrics assess how accurately a QA system answers questions using a generated summary, while BLANC scores masked-token reconstruction gains when a model has access to that summary.SummaQA answers questions derived from the source using the summary; FEQA answers summary-derived questions using the source. BLANC-help adds the summary to the input, whereas BLANC-tune compares a model tuned on it with the original model.Source: §4.4 · pp. 63–64, Figure 4.8 · p. 64
  • Extracted-fact metrics compare facts from generated summaries with source or reference facts, but evaluating machine-generated text quality remains an open research topic.Zhang [2019] uses OpenIE facts and semantic relevance, precision, recall, and F1; Chang et al. [2021] scores recall of manually extracted salient keywords.Source: §4.4 · p. 64

Conversational Machine Comprehension

This section surveys neural conversational machine comprehension readers that answer questions from retrieved passages using both the current question and dialog history. It covers extractive, generative, hybrid, and conversational approaches to modeling context, along with limitations of current CMC datasets and readers.

Conversational Machine Comprehension

  • The chapter frames a CMC reader as the result-generation component that produces a direct answer from query-relevant retrieved documents, enabling conversational open-domain question answering.It contrasts concise answers with document-link lists and outlines extractive, generative, hybrid, and conversational reader approaches.Source: Conversational Machine Comprehension · p. 65

5.1 Task and Datasets

  • CMC maps a question, retrieved passages, and dialog history to an answer; CoQA permits free-form, history-dependent answers, while CoQA and QuAC underrepresent topic returns, clarifications, definitions, and abstractive phenomena.Topic shifts and drill-down questions are represented in both datasets, but returns, clarifications, and definitional interactions are scarce; QuAC has few abstractive behaviors beyond yes/no, and CoQA has only a small number.Source: §5.1 · pp. 65–67

5.2 Extractive Readers

  • Extractive readers classify passage words as answer-span start, end, or outside, using question-passage representations to predict the most probable answer span.A typical architecture encodes question and passage into integrated contextual vectors, then predicts start and end probabilities for each passage word.Source: §5.2 · p. 68

5.2.1 BiDAF

  • BiDAF builds question-aware passage representations through bidirectional attention, then predicts answer-span boundaries from the integrated context.It contextualizes question and passage with bidirectional LSTMs, computes attention in both passage-to-question and question-to-passage directions, and combines the resulting vectors.Source: §5.2.1 · pp. 68–69

5.2.2 BERT-Based Readers

  • BERT-based readers encode concatenated question-passage inputs with Transformer attention and predict answer spans; training pre-trains the encoder, then fine-tunes it with the predictor on comprehension data.For domain mismatch, general-domain models can be continually pre-trained on in-domain data, or domain-specific models can be pre-trained from scratch when unlabeled text is abundant.Source: §5.2.2 · pp. 70–71

5.3 Generative Readers

  • Generative readers decode answers that need not appear in retrieved passages; Fusion-In-Decoder encodes passages independently and lets its decoder jointly attend to them.FID’s computational cost grows linearly with the number of passages, while joint decoding allows it to aggregate evidence across passages.Source: §5.3 · p. 71

5.4 Hybrid Readers

  • UnitedQA combines complementary extractive and generative predictions and set then-state-of-the-art on NaturalQuestions and TriviaQA.The approach is a simple ensemble of predictions from extractive and generative readers.Source: §5.4 · pp. 71–72

5.5 Conversational Machine Comprehension Readers

  • CMC readers on CoQA and QuAC answer the current question from its passage and dialog history, but do not select system actions through a dialog policy.Because topic returns, clarifications, and definitions are poorly represented, these readers do not recommend new topics or ask clarifying questions.Source: §5.5 · p. 72
  • CMC readers use passage and dialog history because answers may depend on earlier turns, extending single-turn readers with history selection and history-aware encoding while leaving the answer module unchanged.History can resolve dependencies such as ellipsis, where “she” in a question refers to “Jessica” in an earlier answer.Source: §5.5 · p. 72

5.5.1 History Selection

  • History selection limits encoding cost and filters irrelevant turns, since dialog histories can be long and conversation topics may shift.The passage notes that many Transformer-based readers accept sequences of up to 512 tokens.Source: §5.5.1 · p. 72
  • Heuristics truncate history to a preset length [Ju et al., 2019; Choi et al., 2018; Yatskar, 2018] or retain the last two turns [Zhu et al., 2018; Ohsugi et al., 2019].The last-two-turn choice is based on an empirical finding that including more turns does not improve performance.Source: §5.5.1 · p. 72
  • HAM weights each history turn by its usefulness to the input question and represents history as a weighted sum of turn vectors.This allows selection to account for relevance rather than simply turn position; a question may relate to a remote turn or follow a topic shift.Source: §5.5.1 · pp. 72–73, Figure 5.6 · p. 73

5.5.2 History Encoding

  • One history-encoding category appends selected turns to the input question; Choi et al. [2018] also encode turn numbers and previous answer-span locations as features.This converts CMC into a single-turn machine-comprehension problem.Source: §5.5.2 · p. 73
  • FLOW-based readers integrate intermediate representations from earlier questions; FlowQA [Huang et al., 2018] uses parallel Integration Flow layers to produce history-aware context vectors.The sequential alternative is computationally expensive to train on large dialog datasets; Integration Flow (IF) performs reasoning in parallel for each question and refines results across turns.Source: §5.5.2 · pp. 73–74
  • FlowDelta [Yeh and Chen, 2019] models information gain through dialog flow, while GraphFlow [Chen et al., 2019b] uses question-specific context graphs to model temporal dependencies.GraphFlow dynamically constructs a context graph from the passage at each turn.Source: §5.5.2 · p. 74
  • BERT-based readers encode dialog context for answer prediction, either concatenating turn-conditioned representations with a bidirectional RNN [Ohsugi et al., 2019] or selecting history via HAM [Qu et al., 2019].The HAM reader scores turn representations against the input question, incorporates weighted history into the question-passage representation, and predicts the answer span from the result.Source: §5.5.2 · pp. 75–76

Conversational QA over Knowledge Bases

C-KBQA is a special case of conversational information retrieval that answers queries by reasoning over structured knowledge bases rather than indexed documents. The chapter surveys its task, architecture, modular components, and a unified Transformer-based alternative.

Conversational QA over Knowledge Bases

  • C-KBQA generates answers by reasoning over structured knowledge bases, unlike document-based CIR, and can operate as a standalone system.Many task-oriented dialog systems are implemented as C-KBQA systems.Source: Conversational QA over Knowledge Bases · p. 77
  • The chapter covers semantic parsing, dialog-state tracking, dialog policy, and response generation, then discusses a Transformer-based system that unifies these modules.The unified system generates answers grounded in input queries, dialog states, and knowledge.Source: Conversational QA over Knowledge Bases · p. 77

6.1 Knowledge Bases and Questions

C-KBQA systems access relational databases or knowledge graphs, and multi-turn interaction lets users express complex information needs through simpler questions. The section describes task capabilities and surveys open benchmarks spanning conversational QA and database querying.

6.1 Knowledge Bases and Questions

  • Relational databases store entities with properties, whereas knowledge graphs represent subject-relation-object triples as linked entities.The chapter notes that relational databases operate faster on large record sets and use less storage, while knowledge graphs are commonly used for Web-scale, open-domain knowledge bases and relational databases for task-specific ones.Source: §6.1 · pp. 77–79
  • Multi-turn C-KBQA lets users query a knowledge base interactively instead of composing a complicated question in one turn.The chapter motivates this with exploratory search and reports that users rarely issue questions like “Which super heroes came from earth and first appeared after 2009?” [Iyyer et al., 2017]; the intent is expressed as a sequence of simpler questions.Source: §6.1 · p. 79
  • C-KBQA combines complex knowledge-base question answering with coherent conversational turns, requiring question parsing, context resolution, dialog policy, and knowledge-base path retrieval.Sessions can include simple triple-based questions, complex logical, comparative, or quantitative reasoning, co-reference, ellipses, clarification, and confirmation.Source: §6.1 · p. 79

6.1.1 Open Benchmarks

  • Open benchmarks include SQA, CSQA, ConvQuestions, CoSQL, and CLAQUA, covering sequential questions, complex reasoning, cross-domain text-to-SQL, and clarification.SQA contains 2,022 sessions averaging 2.9 questions; CSQA contains 200K sessions and 1.6M turns; ConvQuestions contains 11,200 conversations; CoSQL contains 30K turns and 10K annotated SQL queries; CLAQUA evaluates clarification-related tasks.Source: §6.1.1 · p. 80

6.2 System Overview

A typical C-KBQA system parses utterances and dialog context into formal queries, manages dialog state and actions, and generates natural-language responses. Its use of structured knowledge bases and semantic parsing distinguishes it from document-based CIR.

6.2 System Overview

  • The semantic parser maps a user utterance and dialog context to a formal query, which is executed on the knowledge base to retrieve an answer.The dialog manager tracks entities, predicates, and action sequences and selects system actions; the response generator turns the selected action into natural language.Source: §6.2 · p. 80
  • C-KBQA uses structured knowledge bases rather than document collections, and its semantic parser conditions formal-query generation on both the utterance and dialog state.The parsed query and retrieved result update the dialog state; when a result is too broad to present, the system may ask a clarifying question.Source: §6.2 · p. 81

6.3 Semantic Parsing

This section surveys semantic parsing for C-KBQA, from parser families and DynSP’s search-based approach to SCORE’s conversational pretraining and methods that avoid formal queries.

6.3.1 Overview

  • Weak supervision with question-answer pairs is common because question-parse pairs are expensive, but it risks spurious parses that return correct answers without capturing question meaning.Neural parsers are grouped into translation, classification, and ranking approaches.Source: §6.3.1 · pp. 81–82

6.3.2 A Dynamic Neural Semantic Parser

  • DynSP casts parsing as state-action search over partial or complete parses, scoring action sequences and using beam search to find a high-scoring end state.Its neural modules are constructed dynamically during search and can be trained end-to-end; weak supervision uses partial-parse answer overlap as a dense reward.Source: §6.3.2 · pp. 82–84
  • DynSP’s SUBSEQUENT statement handles follow-ups by adding conditions to a prior parse, such as narrowing a previous answer set.The example’s returned answer is Dragonwing, a subset of the previous answer.Source: §6.3.2 · pp. 82–83

6.3.3 Using Pre-Trained Language Models

  • SCORE starts from RoBERTa and continues pretraining on synthesized conversational parsing data with MLM, Column Contextual Semantics, and Turn Contextual Switch objectives.The synthesized samples combine questions, dialog history, database schemas, and formal queries; the authors report few-shot adaptation to new tasks with few manually generated labels.Source: §6.3.3 · p. 85, Figure 6.7 · p. 85
  • Column Contextual Semantics predicts operations for schema columns or tables, while Turn Contextual Switch predicts actions that modify a prior query or create a new one.Both tasks use labels derived from the target formal query.Source: §6.3.3 · p. 86

6.3.4 C-KBQA Approaches without Semantic Parsing

  • C-KBQA methods can omit semantic parsing: examples encode knowledge bases as graphs to select answers or infer missing pieces in incomplete follow-up questions.The section notes that applying pretrained language models to these non-parsing approaches remains an open question.Source: §6.3.4 · p. 86

6.4 Dialog State Tracking

This section describes dialog-state tracking for resolving context in follow-up questions, then presents contextual rewriting as an alternative way to make questions independently parseable.

6.4 Dialog State Tracking

  • DynSP’s SUBSEQUENT statement is limited to follow-ups whose answers are subsets of earlier answers, while real dialogs can contain entity, predicate, and pronoun ellipsis.Explicit dialog-state tracking is needed to retain contextual information for interpreting the current turn.Source: §6.4 · p. 87
  • Guo et al.’s dialog memory stores entities for coreference, predicates for predicate ellipsis, and prior action subsequences that can be reused in follow-up parses.The parser adds instantiated actions that access memory and replication actions that copy earlier actions.Source: §6.4 · p. 87

6.4.1 Contextual Question Rewriting

  • Contextual question rewriting fuses dialog history into a follow-up question, allowing a context-independent semantic parser to process the rewritten question.The cited dataset contains 1,000 examples spanning typical follow-up scenarios.Source: §6.4 · p. 87, §6.4.1 · p. 88
  • Ren et al.’s four sequence-to-sequence models differ in how they fuse history, and Model-4 combines Models 2 and 3 and is reported as the best performer.Model-2 uses separate question and history encoders with attention; Model-3 fuses history into the question during encoding.Source: §6.4.1 · p. 88
  • Liu et al.’s split-and-recombine method segments the question and history, then resolves conflicting spans to form a de-contextualized question.For example, it can replace “Smith” in the question with “Bill Collins” from the history.Source: §6.4.1 · pp. 89–90
  • XiaoIce rewrites questions against long conversation histories by identifying entities, resolving coreferences, and completing omitted predicates.Its history may span days or months, and the component tracks entity mentions and predicates in working memory.Source: §6.4.1 · p. 90

6.5 Dialog Policy

Dialog policy selects system actions from dialog states to support user satisfaction, and this section surveys clarifying-question policies, dialog acts, and reinforcement-learning approaches to policy optimization.

6.5.1 A Case Study

  • In movie search, the system presents a small result set or asks a clarifying question, choosing which attribute to ask about next.The example asks for release year because Bill Murray stars in many movies.Source: §6.5.1 · p. 91
  • The reinforcement-learning policy performs best among the four movie-attribute selection policies, combining strengths of entropy-minimization and frequency-based policies.It learns online from user-system interactions.Source: §6.5.1 · p. 92, Figure 6.15 · p. 92

6.5.2 Dialog Acts

  • A dialog act specifies an utterance’s intent and optional slots or slot-value pairs; intents are shared across tasks, while slots depend on the task’s knowledge-base schema.Policy optimization aims to select the best action for each dialog state to maximize user satisfaction.Source: §6.5.2 · p. 93

6.5.3 Reinforcement Learning for Policy Optimization

  • Because labeled state-action data are hard to collect and manual policies are difficult to specify for complex or changing tasks, the book recommends a rules- or supervised-learning warm start followed by online improvement.Reinforcement learning instead explores state-action choices and learns from interaction rewards, without a teacher.Source: §6.5.3 · pp. 93–94
  • REINFORCE estimates policy gradients from sampled dialogs and discounted rewards, but its trajectory-wide reward sum has high variance, motivating critic-based value estimates.The estimated value function approximates the discounted long-term reward from an action followed by the current policy.Source: §6.5.3 · p. 94
  • Learning from scratch with human users requires many interactions and can be prohibitively expensive or costly when failures occur; expert dialogs or teacher advice can speed policy search.These are among the approaches reviewed to reduce reinforcement-learning costs.Source: §6.5.3 · p. 94
  • For domain extension, BBQ maintains a posterior over network weights and samples weights when choosing actions; experiments report more efficient exploration than state-of-the-art baselines.The setting involves adapting to new tables or entity attributes in a knowledge base.Source: §6.5.3 · p. 95
  • Deep Dyna-Q combines human interaction, simulator refinement, and simulated planning; human-in-the-loop experiments report efficient policy improvement, while variants gate simulated samples or budget human interactions.Simulators are inexpensive but may lack human conversational complexity and introduce design biases.Source: §6.5.3 · pp. 96–97

6.6 Response Generation

Response generation must convey dialog-act meaning fluently. This section reviews template-based and corpus-based methods, then describes SC-GPT, which combines pre-trained language modeling with dialog-act control and task-specific fine-tuning.

6.6 Response Generation

  • Template-based methods fill handcrafted task-specific templates with slot values, often preserving required meaning but not always producing fluent responses.The section groups response-generation methods into template-based and corpus-based approaches.Source: §6.6 · p. 97
  • SC-LSTM uses dialog-act representations to guide generation, but requires extensive task-specific annotation and faces scalability issues as the number of dialog acts grows with slots.The section introduces pre-trained models such as GPT-2 as a response to this challenge.Source: §6.6 · p. 97

6.6.1 SC-GPT

  • SC-GPT generates responses conditioned on dialog acts and is trained through plain-text pre-training, dialog-act-controlled pre-training, and task-specific fine-tuning.Fine-tuning adapts it to unseen intents or slot-value pairs using a few dozen task-specific dialog-act-labeled samples.Source: §6.6.1 · pp. 97–98
  • SC-GPT achieves state-of-the-art results on a public response-generation benchmark, outperforming corpus-based methods including GPT-2 and SC-LSTM.The text reports that its responses are fluent and retain the intent and slot-value information specified by dialog acts.Source: §6.6.1 · p. 98

6.7 Grounded Text Generation for C-KBQA

GTG combines a Transformer with symbolic knowledge-base inference to build task-oriented conversational systems that are intended to be scalable, interpretable, and controllable. It tracks dialog state, retrieves task-specific knowledge, generates grounded responses, and can be trained and adapted through multiple stages.

6.7 Grounded Text Generation for C-KBQA

  • GTG combines a Transformer language model with symbolic knowledge-base inference to build robust C-KBQA systems at scale.The hybrid pipeline uses human-comprehensible symbolic module inputs and outputs, while GTG tracks dialog states and grounds responses in task-specific knowledge.Source: §6.7 · p. 98

6.7.1 GTG for Task-Oriented Dialog

  • At each turn, GTG derives a belief state from the query and dialog history, looks up matching entities, and generates a grounded response.The final response is lexicalized from a delexicalized response using the belief state and database results; delexicalization makes training more sample-efficient.Source: §6.7.1 · p. 99

6.7.2 GTG Training

  • GTG training combines language pre-training, task-grounded pre-training, and task-specific fine-tuning using supervised learning and reinforcement learning.Task-grounded pre-training teaches belief-state and intent tracking, knowledge-base lookup, and task-completion responses; fine-tuning uses machine teaching and task-specific knowledge.Source: §6.7.2 · p. 100
  • Combining policy and response generation reduces reliance on task-specific action labels, while GTG can be pre-trained on open-domain data and fine-tuned with fewer task labels.Peng et al. [2020a] argue the combined approach is more scalable; belief states are relatively task-independent and easier to collect in quantity than system-action labels.Source: §6.7.2 · p. 101

6.7.3 Remarks on Continual Learning for Conversational Systems

  • Continual machine teaching uses logged dialogs to correct system behavior, but selecting regression sessions and using them for teaching remain open research problems.Retraining with representative sessions can help retain previously learned skills; a generative model is proposed as an alternative to storing many old dialogs.Source: §6.7.3 · p. 101
  • Continual learning from user interactions can use reinforcement learning, but current methods require substantial interaction data that may be costly in real-world settings.Strategies discussed include exploring less-visited dialog states, using user simulators for model-based learning, and decomposing tasks into subtasks or skills.Source: §6.7.3 · pp. 101–102
  • A task bot can update shared memory with missing knowledge encountered during training, treating that memory as knowledge accumulated from experience.Shen et al. [2016] describe a shared memory that stores a compact version of the knowledge base and incorporates information when the system fails because it is missing.Source: §6.7.3 · p. 102

Proactive Human-Machine Conversations

This section surveys proactive conversational IR approaches that help users clarify information needs, identify useful next questions, and shift or extend conversation topics. It covers clarification selection and generation, question suggestions, topic-shifting systems and their data and methods, and measures of usefulness beyond relevance.

7.1 Asking Clarifying Questions

  • A large-scale user study found that users enjoy clarifying questions for their functional and emotional benefits, and that they can make a CIR system seem more intelligent and trustworthy.Zamani et al. [2020a] conducted the study using a Web search interface with clarifying questions.Source: §7.1 · p. 103

7.1.1 Question Selection Methods

  • Ou and Lin [2020] won ClariQ with ELECTRA response understanding, BM25 candidate retrieval, and ELECTRA question ranking; question-bank selection suits schema-bound tasks, not open-domain search.The ClariQ task includes identifying when clarification is needed and selecting a question from a bank.Source: §7.1.1 · pp. 104–105

7.1.2 Question Generation Methods

  • Open-domain clarification includes disambiguation, preference, topic, and comparison; Zamani et al. [2020a] use query-log taxonomy for template-based RTC and weakly supervised QLM generation.RTC often generates appropriate questions despite its simplicity.Source: §7.1.2 · pp. 105–106
  • QCM fine-tunes QLM to maximize clarification utility, and Zamani et al. [2020a] report questions that are as fluent but more useful and better at improving search results than QLM’s.The approach addresses QLM’s tendency to generate common questions by optimizing expected clarification utility with MIXER.Source: §7.1.2 · p. 107

7.2 Suggesting Useful Questions

  • Question suggestions for information seeking proactively lead users toward engaging experiences with interesting, informative, and useful follow-up questions.The section focuses on suggestions for information seeking rather than social-chatbot conversation.Source: §7.2 · p. 107

7.2.1 Question Suggestions in Commercial Search Engines

  • Google and Bing’s People Also Ask feature suggests natural-language questions for users’ next inquiry steps, extending the panel with more suggestions when users click a question.Its design assumes that organic results often address the current information need.Source: §7.2.1 · p. 108

7.2.2 From Related to Useful Suggestions

  • Rosset et al. [2020] propose a usefulness metric beyond relevance that assesses whether a suggestion adds value, advances a task, or helps users explore a topic productively.Their guidelines were used to label query-suggestion pairs sampled from Bing search logs.Source: §7.2.2 · p. 108
  • Rosset et al. [2020] identify five non-useful suggestion categories: missed intent, excessive specificity, prequels, duplicates of the query, and duplicates of answers.A useful suggestion adds valuable new information or reframes an information-seeking task along a coherent line of thought.Source: §7.2.2 · pp. 108–109

7.2.3 Question Suggestion Systems

  • PAA question suggestion uses a BERT ranker to select candidates or fine-tuned GPT-2 to generate them from a query and, when available, its history.The ranker learns from positive and negative query-suggestion pairs, while GPT-2 training uses positive suggestions.Source: §7.2.3 · pp. 109–110
  • Mined sessions are filtered for noise, coherent intent, and information-seeking intent, then support suggestion training and bank curation [Rosset et al., 2020].Filtering keeps sessions with at least three queries and a satisfied click, retains the largest query-embedding subgraph above cosine similarity 0.4, and uses an answer-based QA intent classifier.Source: §7.2.3 · p. 110, Figure 7.5 · p. 110
  • NQP training on mined conversational sessions significantly reduces overly specific or prequel suggestions, and an online A/B test validates its effectiveness [Rosset et al., 2020].The comparison table illustrates suggestions selected by the BERT-based ranker before and after NQP training.Source: Table 7.2 · p. 111, §7.2.3 · p. 111

7.3 Shifting Topics

  • Unlike clarification and question suggestion, topic shifting changes search intent; research on it is limited for CIR, but existing methods offer a foundation [Wu et al., 2019a; Tang et al., 2019; Xie et al., 2021; Zhou et al., 2020].Topic shifts can maintain engagement in social chatbots or guide goal-oriented dialogs toward a predefined target.Source: §7.3 · pp. 111–112

7.3.1 Topic Shifting in Social Chatbots

  • XiaoIce’s topic shifting significantly improves user engagement in an A/B test.Its topic manager detects when to shift and recommends a new topic; the system triggers shifts when it lacks knowledge or the user seems bored.Source: §7.3.1 · pp. 112–113
  • Xie et al. [2021] introduced TIAGE with turn-level topic-shift annotations; detection is relatively easy with a fine-tuned pretrained model, but generating topic recommendations is challenging.Topic shifts occur every 4–5 turns on average, and XiaoIce selects topics from a dataset rather than generating them.Source: §7.3.1 · pp. 113–114

7.3.2 Target-Guided Topic Shifting

  • Tang et al. [2019] model target-guided shifts as smooth transitions toward a designated topic using keyword prediction, target-progress filtering, and keyword-conditioned response retrieval.The target-progress module selects candidate keywords closer to the target, while the response module retrieves a suitable reply from dialog history and selected keywords.Source: §7.3.2 · pp. 114–115
  • Knowledge graphs represent topic shifts as transitions among entities and can be explainable and precise, but limited coverage confines their use to controlled settings or domains with high-quality graphs [Zhou et al., 2018; Wu et al., 2019a; Li et al., 2018].The target-guided dataset instead represents turn topics with keywords extracted using TF-IDF and part-of-speech features [Tang et al., 2019].Source: §7.3.2 · pp. 115–116

Case Study of Commercial Systems

This section reviews published accounts of commercial conversational systems, research platforms, and chatbots, surveying how conversational capabilities appear in toolkits, chatbot design, search interfaces, productivity-focused and device-based assistants, and hybrid systems combining automation with crowd support.

Case Study of Commercial Systems

  • Commercial-system coverage is limited to published material because many systems are proprietary; the chapter first reviews research platforms and toolkits, then application-area histories and trends.Source: Case Study of Commercial Systems · p. 117

8.1 Research Platforms and Toolkits

  • Toolkits abstract dialog state, make development accessible to less-experienced developers, and integrate APIs, channels, or devices; Conversation Learner learns from examples and uses reinforcement learning to predict next actions.Rules can override probabilistic action choices for more deterministic behavior in selected contexts.Source: §8.1 · pp. 117, 119

8.2.1 Chatbots

  • Chatbots increasingly blend retrieval, recommendation, chit-chat, and task completion, with personality and both emotional and intellectual qualities viewed as important to user acceptance.The chapter reviews chatbots as background for design decisions, including systems whose primary goal is companionship.Source: §8.2.1 · p. 120

Influential Historical Chatbots

  • ELIZA used canned phrases and rewrite rules; PARRY simulated mental states and featured early automated agent interaction; ALICE won the 2004 Loebner prize using manually curated response triples.ELIZA’s simple approach nonetheless supported long conversations, while ALICE’s triples were refined through past dialog failures. Works: ELIZA, PARRY, ALICE.Source: Influential Historical Chatbots · pp. 120–121

Modern Chatbots

  • 23 CPS was XiaoIce’s reported average conversational-turns-per-session engagement, with expected CPS also accounting for the number of active users (NAU).XiaoIce blends information seeking, retrieval, task completion, and chit-chat; its design prioritizes enjoyable, engaging conversation over minimizing turns.Source: Modern Chatbots · p. 122

8.2.2 Conversational Search Engine

  • Bing search interfaces add conversational affordances such as instant answers with supporting perspectives, entity-pane topic suggestions, People Also Ask follow-ups, and clarification questions.The examples include an answer to a query about Mustang speed and a programming-language clarification for “convert string to int.”Source: §8.2.2 · p. 124
  • Bing offers instant answers, entity exploration, and follow-up questions; specific, knowledge-adding clarifications can increase engagement, but contextuality across turns remains a challenge.Search engines experiment with conversational features, some of which do not last because they may work less well on websites than in chatbots or voice assistants.Source: Figure 8.8 · p. 125, §8.2.2 · p. 125

8.2.3 Productivity-Focused Agents

  • Productivity-focused personal assistants combine information seeking with task completion, such as checking calendar information before creating or changing a meeting.These assistants were often designed around private information stores and tasks such as calendar, task, and contact management.Source: §8.2.3 · pp. 125–126

From PAL to Device-Based Assistants

  • DARPA’s PAL projects RADAR and CALO influenced Siri; RADAR supported email, calendar, and task workflows, while CALO expanded to document authoring and personal-store question answering.RADAR addressed calendar management, uncertain resources in scheduling, and task management [Berry et al., 2011, 2006; Modi et al., 2004; Fink et al., 2006; Freed et al., 2008]. PAL learnings helped motivate an extensible architecture and a vision of a “do engine”.Source: From PAL to Device-Based Assistants · pp. 126–127
  • Device-based voice assistants face noisy-environment recognition and multimodal, multi-party interaction, while research increasingly customizes them for different users, situations, and scenarios.Situated speakers such as Alexa can serve multiple household members, increasing interest in speaker identification and authentication for personal productivity and other assistant functions [Ram et al., 2018b; Khatri et al., 2018b; Gopalakrishnan et al., 2019; Goel et al., 2018; Paul et al., 2019; Yavuz et al., 2019; Khatri et al., 2018a; Kollar et al., 2018].Source: From PAL to Device-Based Assistants · pp. 127–128

8.2.4 Hybrid-Intelligence Assistants

  • Hybrid-intelligence assistants combine automation with crowd workflows for uncertain or unsupported requests, producing training data that can improve more general models.Calendar.help used high-precision rules with crowd-powered fallback for uncertain meeting requests; its technology later influenced Microsoft’s Scheduler, then in early preview.Source: §8.2.4 · p. 128

Conclusions and Research Trends

CIR remains at a research-exploration stage: its proposed reference architecture is unverified, and its modules have yet to be integrated into a system. Open challenges span evaluation, conversational understanding, interaction design, responsible AI, and adaptation.

Conclusions and Research Trends

  • CIR remains at a research-exploration stage; its proposed reference architecture is unverified, and its modules have yet to be pieced together into a system.Researchers also face open questions about system architecture, real-world effectiveness, post-deployment improvement, and effective, responsible Human-AI interaction.Source: Conclusions and Research Trends · p. 129
  • Evaluation remains difficult because off-policy datasets measure individual modules but not system effectiveness in user interactions, while simulated-user and A/B tests are costly.Combining evaluation methods to balance cost and reliability remains an open challenge.Source: Conclusions and Research Trends · p. 129
  • Long-term engagement measures for chatbots need to extend beyond conversational turns per session and active users to include task completion and goal support.In goal-oriented settings, evaluation should consider both achieving users' goals and encouraging them to return.Source: Conclusions and Research Trends · p. 129
  • Continual learning is needed because conversational systems operate in changing settings, whereas many models assume static worlds and pre-collected data.The authors highlight adapting to new tasks with little supervision while avoiding sharp performance drops on old tasks.Source: Conclusions and Research Trends · pp. 129–130
  • Future research may draw on explicit and implicit knowledge, richer or less supervision-dependent learning, and human capabilities such as social and emotional intelligence.The authors cite teaching by demonstration, machine teaching through dialogue, and few-shot or zero-shot learning as research directions.Source: Conclusions and Research Trends · p. 130
  • Conversational understanding still needs grounded language generation and understanding that connects language to world state, from avoiding hallucinations to tracking process state.For device-based and physically situated assistants, integrating voice, vision, and text observations also remains challenging, including identifying who is interacting and responding to physical conditions.Source: Conclusions and Research Trends · pp. 130–131
  • Human-AI interaction design still faces challenges in flexible goal specification and helping users discover system capabilities as they evolve.Even when a system has new capabilities, users may not discover them and therefore may not learn the best ways to interact with it.Source: Conclusions and Research Trends · p. 131
  • Responsible conversational AI must address harms from toxicity and bias in data, feedback, or use, and protect personal information when other people are present.The authors describe Responsible AI as complex and evolving, requiring guidelines to evolve with new challenges.Source: Conclusions and Research Trends · p. 131

Related papers

Loading 2201.05176v1…