Source-linked AI summary
Survey on Evaluation Methods for Dialogue Systems
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset, Eneko Agirre, Mark Cieliebak
TL;DR
Dialogue-system evaluation must handle application-dependent notions of quality while reducing costly and time-intensive human assessment. This survey organizes evaluation concepts and methods by three dialogue-system classes, emphasizing automation and their relationship to system characteristics. It concludes that evaluation evolves with dialogue technology, while automation involves engineering effort or reduced correlation with human judgments.
Problem
Dialogue evaluation is difficult because quality and its measurement depend on the application, while human evaluations are costly and time-intensive.
Method
The survey reviews evaluation concepts and methods by distinguishing task-oriented systems, conversational agents, and question-answering systems.
Results
The survey presents the current state of research and shows that evaluation requirements evolve alongside dialogue-system technology.
Takeaways & Limitations
Automated evaluation can reduce human labour, but methods trade off engineering effort and correlation with human judgments.
Takeaways & Limitations
Some dialogue-system components remain constrained by manually created domain ontologies, static domains, and substantial training data requirements.
Abstract
from arXiv · showhide
In this paper we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost and time intensive. Thus, much work has been put into finding methods, which allow to reduce the involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented dialogue systems, conversational dialogue systems, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then by presenting the evaluation methods regarding this class.
1 Introduction
Dialogue systems support natural-language interaction across tasks such as virtual assistance, information seeking, and e-learning. Their evaluation is difficult because quality depends on the application, measurement is unclear, and human studies are costly and time-intensive.
- Applications: Dialogue systems let users interact with computers through natural language for tasks including assistance, information seeking, and e-learning.Examples include scheduling appointments, answering factual or complex questions, and training users for medical or military situations.
- Evaluation challenges: High-quality dialogue is difficult to define and measure, including whether a response is appropriate for a particular system.User feedback can help assess appropriateness, but the paper notes limitations to this approach.
- Evaluation challenges: Human evaluation is costly and time-intensive because user studies require preparation, recruitment, and compensation.These burdens motivate methods that reduce human labour.
- Evaluation challenges: Evaluation quality is defined by a system’s intended function, so correctness may suit question-answering systems but not conversational agents.The appropriate metric depends on the dialogue system’s purpose.
- Survey scope: Automated evaluation aims to produce efficient, repeatable procedures for comparing the quality of different dialogue strategies.The survey organizes evaluation methods around dialogue-system classes and the degree of automation they provide.
2 A General Overview
Dialogue systems are grouped into task-oriented, conversational, and interactive question-answering classes with different structures, domains, and goals. Evaluation methods therefore vary, while the survey emphasizes automation, repeatability, human correlation, discrimination, and explainability.
- System classes: The survey distinguishes task-oriented systems, conversational agents, and interactive question-answering systems.Interactive question answering cannot be completely placed in either task-oriented or conversational categories.
- System classes: Task-oriented systems solve specific tasks efficiently through structured, often closed-domain dialogues with mixed initiative.Conversational agents instead support unstructured open-domain social interaction, where longer dialogues are desired.
- System classes: Question-answering systems focus on answering questions, may be domain-specific or open-domain, and can support single or multiple turns.Their domain is often determined by underlying knowledge bases or text snippets.
- Evaluation requirements: An evaluation method should be automated, repeatable, correlated with human judgments, able to distinguish dialogue strategies, and explainable.Explainability includes identifying which system features affect dialogue quality.
- Human evaluation: Human evaluations include lab experiments, in-field studies, and crowdsourcing, with crowdsourced quality comparable to lab conditions when enough users participate.Lab settings are controlled but may not represent real-world conditions, whereas in-field studies collect feedback from operational users.
- Human evaluation: Human-based evaluation requires careful experimental preparation and must account for substantial variability in user behaviour, especially in crowdsourced environments.These factors make human evaluation difficult to set up and conduct.
3.1 Characteristics
Task-oriented dialogue systems perform clearly defined tasks in restricted domains using structured, efficient interactions. They retrieve information, perform actions, or collect user information through interfaces such as voice commands.
- Task-oriented systems: Task-oriented systems have measurable goals, structured dialogue behaviour, closed domains, and an emphasis on efficiency.Typical tasks include finding database information, performing actions, or retrieving information from users.
- Task-oriented systems: An in-car entertainment system can start music or let a driver query an agenda through voice commands while guiding the driver through available options.The example illustrates a task-oriented system acting as an interface to application functionality.
- Task-oriented systems: Task-oriented dialogue systems infer constraints through conversation and retrieve the requested information within a specific domain.Their dialogues are structured and tailored to the task.
3.2 Dialogue Structure
Task-oriented dialogue structure combines domain content with a strategy for filling required slots. Domain ontologies specify slot-value options, while dialogue acts represent the actions used to conduct the conversation.
- Dialogue structure: Task-oriented dialogue structure is defined by conversation content and the strategy used within the conversation.The content comes from the domain, while the strategy sequences actions.
- Content: A domain ontology represents conversational content as slot-value pairs, with each slot having a type and allowable values.The E2E restaurant ontology contains eight slots or attributes, each with a type and set of values.
- Strategy: Dialogue acts model the strategy for filling required slots as actions such as informing, querying, confirming, affirming, or negating.Each act has a type and arguments, and each utterance corresponds to an interlocutor action.
- Dialogue acts: An inform act communicates its arguments, such as a restaurant’s food type and area, while request acts seek slot values.For example, inform(food = “French”, area = “riverside”) conveys the specified restaurant attributes.
- Dialogue acts: Sample dialogues translate user utterances into dialogue acts and system acts back into natural-language utterances.The acts encode the user’s intended task and the parameters needed to complete it.
3.3 Technologies
Classic dialogue systems model interaction as a sequence of actions through interconnected modules, traditionally arranged as a pipeline. Task-oriented systems use NLU, dialogue-state tracking, dialogue management, and language generation, while newer approaches connect these functions with trainable end-to-end networks.
- A dialogue system architecture models dialogue as a sequence of actions across several interconnected components.
- Dialogue management: The dialogue manager uses the current conversation state to select a dialogue act representing the system’s next action.
- Pipelined modules: NLU extracts slots, values, and dialogue acts from user input, while DST infers the current dialogue state and NLG produces the system utterance.
- End-to-end architectures: End-to-end trainable architectures replace separately designed pipeline modules with a single trainable system whose neural networks interact.
- Pipelined modules: In the pipeline, user input may arrive as text or as an N-best list of ASR transcriptions before NLU processing.
- Dialogue-state tracking: DST represents uncertainty with a probability distribution over possible dialogue states because ASR and NLU errors affect state tracking.
- Dialogue management: Finite-state dialogue control is rigid but efficient for simple domains and tasks, whereas frame-based management supports greater flexibility and error handling.
- Dialogue-state tracking: POMDP-based systems model uncertainty with belief states, but still require a manually created domain ontology and operate on a static, predefined domain after training.
3.4 Evaluation
Dialogue-system evaluation combines human judgments with automated models tailored to system class and evaluation goal. The surveyed methods include task-success measures, predictive satisfaction models, interaction-quality metrics, and user simulation, with varying generalization and annotation limitations.
- User Satisfaction Modelling: PARADISE models subjective system performance as a linear combination of objective task-success and dialogue-cost measures.Multiple linear regressions found task-success and the number of repetitions significant.
- User Satisfaction Modelling: PARADISE generalized to other dialogue systems to a certain degree, but performance varied substantially across systems and user groups.Cross-system evaluation produced higher scores on TOOT than ANNIE, while novice-trained models failed to predict experienced-user scores.
- User Satisfaction Modelling: Automatic satisfaction modeling remains constrained by annotation and feature limitations, including low inter-rater agreement and incomplete dialogue-act representations.Some earlier models performed only marginally better than a random baseline, while later n-gram models reached 34.4% rating accuracy and 94.7% task-success accuracy.
- User Satisfaction Modelling: Interaction Quality uses median expert ratings at each exchange to support automatic detection of problematic dialogue situations.It is intended to approximate user satisfaction for settings where exchange-level user ratings are impractical.
- User Satisfaction Modelling: 0.776 correlation was achieved between the in-field Interaction Quality model and human judges using automatically extracted features.ASR features alone achieved ρ = 0.753, while the lab model reached ρ = 0.894 with user-specific features.
- User Simulation: User simulation methods include agenda-based and neural simulators, with neural user simulation outperforming agenda-based training in reported human and cross-model evaluations.ABUS achieved 90.6% average task success over 160 dialogues, whereas NUS-trained systems performed significantly better in both reported comparisons.
- User Simulation: Predicted judgments from simulated dialogues can reproduce relative user-study conclusions even when simulations do not match absolute values.The reported example preserved conclusions about younger versus older users and evaluations of dynamic help.
4 Conversational Dialogue Systems
Conversational dialogue systems target open-domain, engaging interactions and are commonly modeled with rules or data-driven methods. Their evaluation remains difficult because goals and relevant quality attributes are not clearly defined, while automated metrics show mixed validity and robustness.
- Conversational dialogue systems support unstructured, open-domain conversations, typically without a specific goal beyond maintaining engagement.
- Rule-based and corpus-based systems are the two main approaches for modeling conversational dialogue systems.
- Corpus-based systems exploit data through utterance selection, which ranks candidate responses, or generative models based on deep neural networks.
- Automated evaluation is difficult because conversational systems lack clearly defined goals and structured dialogues, making relevant quality attributes unclear.
- BLEU and ROUGE have no correlation with human judgments, whereas trained metrics can correlate more strongly but remain vulnerable to simple response manipulations.Reversing responses increased ADEM scores in 48.66% of cases, while dull dummy responses increased scores in 86.93% of cases.
- Evaluation results vary across methods: topic depth correlated with human judgments at ρ = 0.707, while topic breadth reached ρ = 0.512.
- Weak agreement measured data improvements, rising from 43% with raw Wizard-of-Oz data to 56% with paraphrases and 67% after manual annotation.
- Voted Appropriateness correlated with human judgments at 0.479 on the interaction level and 0.893 on the system level, but depends heavily on human annotations.
5 Question Answering Dialogue Systems
Question-answering dialogue systems focus on producing correct answers, ranging from single-turn factoid QA to context-dependent and interactive settings. Evaluation and modeling increasingly use pretrained transformers, while multi-turn QA and generative QA remain comparatively challenging research areas.
- QA systems are designed to find correct answers, unlike task-oriented systems that support multiple predefined tasks.
- QA systems may operate over broad or specific domains and can be categorized as single-turn QA, context QA, or Interactive QA.
- Single-turn QA handles one question without further interaction, performing well on factoid questions but struggling with complex inference or missing user information.
- Single-turn QA includes open QA across multiple sources and reading comprehension, where answers are gathered from a single document.
- Context QA: Context QA uses previous questions and answers to resolve ambiguity, track inference steps, and interpret context-dependent or elliptical questions.
- Interactive QA: Interactive QA combines context QA with task-oriented dialogue to relax or add constraints when a query returns too few or too many results.
- Pretrained transformer models such as BERT, XLNet, and ALBERT are fine-tuned for QA by predicting answer-span start and end tokens.
- Multi-turn QA systems incorporate dialogue history through augmented inputs, information-gain modeling, history embeddings, or history attention.
6 Evaluation Datasets and Challenges
The survey reviews datasets and challenges used to evaluate task-oriented, conversational, and question-answering dialogue systems. Coverage and reuse vary: datasets often target specific components, while shared environments and large corpora support broader comparisons.
- Datasets are important for dialogue-system evaluation, but coverage across dialogue components and evaluation methods is uneven.Datasets can feed more than one evaluation method or metric interchangeably.
- Task-oriented dialogue systems: Very few public datasets evaluate an entire task-oriented dialogue system, whose system-specific evaluation is difficult to reuse and requires substantial human effort.User involvement or external evaluators are usually needed.
- Task-oriented dialogue systems: MultiWOZ contains around 10K dialogues with belief-state and dialogue-action annotations for developing individual components and training end-to-end systems.Its size was at least one order of magnitude larger than earlier structured corpora.
- Task-oriented dialogue systems: Taskmaster-1 includes around 13K dialogues across six domains, with more than half created through self-dialogue methodology.Self-dialogues have richer and more diverse language than MultiWOZ because they are not restricted to a small knowledge base.
- Task-oriented dialogue systems: PyDial enables benchmarking by developing reinforcement-learning dialogue-management environments that compare task-oriented systems under the same conditions.LIDA instead addresses annotation by covering the dialogue annotation pipeline from segmentation through structured labelling.
- Conversational dialogue systems: Large conversational corpora include over 1.7 billion Reddit comments and roughly 1.3 million Twitter conversations, whose short, colloquial messages resemble spoken dialogue.These datasets support evaluation of non-task-oriented systems, although some methods may still require human judgments.
- Question-answering dialogue systems: CoQA contains 8K dialogues and 127K conversation turns, with free-form answers linked to highlighted evidence in multi-domain passages.The dataset is designed for conversational question answering.
7 Challenges and Future Trends
The survey identifies automation and quality definition as continuing challenges, alongside new demands from lifelong-learning systems. Evaluation methods trade off automation, engineering and annotation effort, human correlation, and coverage across domains and behaviors.
- Evaluation methods seek automation, repeatability, correlation with human judgments, strategy discrimination, and explanations of which system features contribute to quality.The survey presents these goals as an open research context.
- Automation: Fully automated word-overlap metrics do not correlate with human judgments at turn level, whereas ∆BLEU and ADEM improve correlation but require annotated data and substantial engineering.BLEU is more competitive at corpus or system level.
- Automation: Task-oriented systems support semi- or fully automatic evaluation through user-satisfaction models and simulations, but these approaches require engineering, annotation, and possible redevelopment for new domains.Models trained for one system are not always reusable for another.
- Automation: User simulations require data collection and simulation construction, and they poorly cover unseen behavior or transfer across tasks and domains.Their behavior is limited to what the corpus or rules represent.
- Automation: Automation remains incomplete because it brings significant engineering effort or reduced correlation with human judgments, making facilitation of evaluation automation an open challenge.
- High Quality Dialogues: Dialogue quality depends on the system’s purpose, target audience, and implementation, so evaluation must be adapted to those factors.Task success and dialogue turns are common task-oriented measures but do not apply universally.
- Lifelong Learning: Lifelong-learning systems need automatic or semi-automatic evaluation to detect critical dialogue states, while the monitoring module and adaptive behavior also require evaluation.One approach classifies the current state as satisfactory or unsatisfactory before requesting user feedback.
- Conclusion: Evaluation requirements evolve with dialogue-system technology, including the shift toward end-to-end trainable systems built from large amounts of data.
Funding
The work was supported by several European research and national funding agencies, universities, and grants.
- The paper acknowledges support from the LIHLITH project, European and national research agencies, the Swiss National Science Foundation, UPV/EHU, and Agence Nationale pour la Recherche.
Conflicts of Interest
The authors report no conflicts of interest.
- The authors disclose no conflicts of interest.