Source-linked AI summary
RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems
Chongyang Tao, Lili Mou, Dongyan Zhao, Rui Yan
TL;DR
Open-domain dialog lacks a standard automatic evaluation metric, and human annotation is labor-intensive. RUBER blends referenced groundtruth similarity with unreferenced query-reply relatedness without human satisfaction labels; experiments report strong human-annotation correlation and fair transferability across datasets.
Problem
Open-domain dialog systems lack a standard automatic evaluation metric, while manual annotation is expensive and time-consuming.
Method
RUBER blends an embedding-based groundtruth-reply scorer with a neural query-reply scorer trained without human satisfaction labels.
Results
RUBER significantly outperforms existing automatic metrics in Pearson and Spearman correlation with human judgments across retrieval and generative dialog systems.
Takeaways & Limitations
RUBER has strong correlation with human annotation and fair transferability across different open-domain datasets.
Takeaways & Limitations
The paper focuses on single-turn conversation; extending RUBER to context-aware dialog requires a neural network that incorporates context.
Abstract
from arXiv · showhide
Open-domain human-computer conversation has been attracting increasing attention over the past few years. However, there does not exist a standard automatic evaluation metric for open-domain dialog systems; researchers usually resort to human annotation for model evaluation, which is time- and labor-intensive. In this paper, we propose RUBER, a Referenced metric and Unreferenced metric Blended Evaluation Routine, which evaluates a reply by taking into consideration both a groundtruth reply and a query (previous user-issued utterance). Our metric is learnable, but its training does not require labels of human satisfaction. Hence, RUBER is flexible and extensible to different datasets and languages. Experiments on both retrieval and generative dialog systems show that RUBER has a high correlation with human annotation.
1 Introduction
Open-domain dialog evaluation lacks a standard automatic metric, while manual annotation is costly and existing metrics correlate weakly with human judgments. RUBER blends referenced and unreferenced scoring without requiring human satisfaction labels, and outperforms existing metrics across retrieval and generative systems.
- Manual annotation is expensive and time-consuming, motivating automatic evaluation for open-domain dialog systems.
- Traditional task-oriented metrics such as slot-filling accuracy and goal-completion rate do not transfer well to open-domain conversation.Open-domain utterances are diverse and uncertain, making accuracy and completion less meaningful.
- Existing generation metrics, including BLEU, METEOR, and ROUGE, show weak correlation with human annotation when applied to dialog evaluation.
- RUBER is a Referenced metric and Unreferenced metric Blended Evaluation Routine for open-domain dialog systems.
- Its referenced scorer measures embedding-based similarity between a generated reply and the groundtruth, while its unreferenced scorer measures query-reply relatedness.
- RUBER requires no human annotation labels for training and significantly outperforms existing automatic metrics in Pearson and Spearman correlation with human judgments.Experiments cover both retrieval and generative dialog systems and show fair transferability across open-domain datasets.
2 Empirical Observations
The paper motivates RUBER with three observations: groundtruth resemblance is useful but imperfect, query-reply relevance adds information, and word overlap is unreliable for casual dialog. These observations guide the referenced, unreferenced, and blended metric design.
- Observation 1: Groundtruth resemblance generally indicates a good reply, but word-overlap statistics have high variance for short, casual dialog utterances.The method therefore measures similarity using embeddings.
- Observation 2: A groundtruth reply represents only one possible response, so groundtruth similarity alone is insufficient for evaluating open-domain dialog systems.A semantically different candidate may still answer the query well, while generic replies such as “I don’t know” may be undesirable.
- RUBER designs referenced and unreferenced metrics from these observations and combines them into an overall metric.Figure 2 presents the overall design methodology.
- Observation 3: More query-relevant groundtruth replies receive higher human satisfaction scores, supporting query-reply information as an unreferenced evaluation signal.Figure 1 groups human scores into five equal-sized quantiles and plots the average query-reply relevance score for each group.
3 Methodology
RUBER combines a referenced similarity score with an unreferenced query-reply relatedness score, then normalizes and heuristically blends them for open-domain dialog evaluation.
- 3 Methodology: RUBER designs referenced and unreferenced metrics and combines them into a hybrid evaluation method.The referenced metric compares a generated reply with groundtruth, while the unreferenced metric compares the reply with its query.
- 3.1 Referenced Metric: The referenced metric measures generated-groundtruth similarity using max- and min-pooled word embeddings and cosine similarity.The pooling heuristic is used because groundtruth scores are unavailable for training a parametric model.
- 3.2 Unreferenced Metric: The unreferenced metric predicts query-reply relatedness with a neural network that produces a scalar score without using groundtruth.The model uses Bi-GRU sentence representations, concatenated query and reply vectors, a quadratic feature, and an MLP.
- 3.2 Unreferenced Metric: Negative sampling trains the unreferenced scorer without human-labeled data by requiring positive query-reply pairs to exceed randomly selected negative pairs by margin ∆.All parameters are trained with Adam and backpropagation.
- 3.3 Hybrid Approach: RUBER normalizes both component metrics to (0, 1) and combines them using minimum, maximum, geometric, or arithmetic averaging.The paper reports that these strategies yield similar results and consistently outperform baselines.
- 3.3 Hybrid Approach: The authors characterize RUBER as simple, general, and effective without sophisticated model designs.Its design methodology covers the referenced scorer, unreferenced scorer, and their combination.
4 Experiments
RUBER is evaluated against human judgments on retrieval and generative dialog systems using correlation analyses, qualitative cases, and transfer experiments. Across these settings, combining referenced and unreferenced scores improves correlation, remains near human performance, and transfers with only slight degradation.
- 4.1 Setup: RUBER is evaluated on both a feature-based retrieval-and-reranking system and a Seq2Seq generative dialog system.Human satisfaction is rated on a 0–2 scale, and Pearson and Spearman correlations are computed against automatic metrics.
- 4.2 Quantitative Analysis: Embedding-based referenced scoring correlates more strongly with human annotation than BLEU and ROUGE, whose exact word overlap is too strict for dialog.The referenced score uses groundtruth information while measuring sentence closeness through word embeddings.
- 4.2 Quantitative Analysis: The unreferenced score achieves higher correlation than the referenced score, indicating that query information is informative for reply evaluation.Its neural scorer is trained with negative sampling and outperforms an embedding-based cosine measure.
- 4.2 Quantitative Analysis: Combining referenced and unreferenced scores significantly outperforms either component and yields near-human correlation without human labels.Min and geometric or arithmetic means perform similarly, while the max operator is slightly worse because it is too lenient.
- 4.3 Qualitative Analysis: In qualitative analysis, BLEU and ROUGE are often zero, whereas RUBER scores are denser and balance complementary judgments of query relevance and groundtruth resemblance.The referenced score favors a reply correlated with the query, the unreferenced score favors one resembling the groundtruth, and RUBER balances both.
- 4.4 Transferability: Transfer to the Tieba dataset causes slight degradation because scorer parameters were not trained there, but RUBER still significantly outperforms baseline metrics.The transfer experiment therefore provides evidence of fair transferability across open-domain datasets.
5 Related Work
Related evaluation metrics range from heuristic overlap measures to supervised or learnable approaches. For open-domain dialog, prior metrics show weak human correlation or require extensive annotation, motivating RUBER’s unsupervised alternative.
- 5 Related Work: Automatic evaluation metrics include heuristic overlap measures such as BLEU, NIST, ROUGE, and METEOR, alongside source-informed approaches.These metrics differ in their use of precision, recall, n-gram overlap, or source-target distribution similarity.
- 5 Related Work: Learnable metrics can improve correlation with human judgments by integrating linguistic features through supervised learning.However, prior approaches commonly depend on manually annotated scores and handcrafted features.
- 5 Related Work: RUBER instead uses negative sampling to train a neural query-reply relatedness scorer without human-annotated scores.The paper presents this as a way to automatically extract features without supervision from human satisfaction labels.
- 5 Related Work: Existing dialog metrics such as BLEU, ROUGE, and METEOR show weak correlation with human judgments in empirical studies.This limitation has been reported for both adopted language-generation metrics and dialog evaluation more broadly.
6 Conclusion and Discussion
RUBER is an evaluation methodology for open-domain dialog systems that considers both groundtruth replies and their queries. Although unsupervised, it correlates strongly with human annotation and transfers fairly across datasets, while currently focusing on single-turn conversation.
- The methodology is presented for open-domain dialog systems, where existing metrics can be treated as features from the output and groundtruth.
- RUBER evaluates replies using both the groundtruth and the preceding user query.The framework blends referenced and unreferenced evaluation components.
- RUBER shows strong correlation with human annotation and fair transferability across open-domain datasets.
- The current study focuses on single-turn conversation, although the framework is described as naturally extensible to context-aware systems.Such an extension would require a neural network for the unreferenced metric that incorporates context.