Source-linked AI summary
deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, Bill Dolan
TL;DR
Evaluating generation is difficult when many outputs are acceptable, especially in conversation where possible responses are weakly constrained. The paper introduces deltaBLEU, which incorporates human judgments into multi-reference BLEU and outperforms IBM BLEU and sentence-level BLEU in conversational-response evaluation.
Problem
Automatic evaluation is difficult for generation tasks with intrinsically diverse targets because the space of plausible outputs can be enormous, particularly in data-driven conversation.
Method
deltaBLEU embeds human judgments about reference quality directly into corpus-level multiple-reference BLEU.
Results
deltaBLEU outperforms both IBM BLEU and sentence-level BLEU when evaluating generated conversational responses against human judgments.
Takeaways & Limitations
deltaBLEU can serve as a plausible intrinsic metric for evaluating and training data-driven conversation systems.
Takeaways & Limitations
Using larger observation units would make confidence intervals too wide to support a conclusive study.
Abstract
from arXiv · showhide
We introduce Discriminative BLEU (deltaBLEU), a novel metric for intrinsic evaluation of generated text in tasks that admit a diverse range of possible outputs. Reference strings are scored for quality by human raters on a scale of [-1, +1] to weight multi-reference BLEU. In tasks involving generation of conversational responses, deltaBLEU correlates reasonably with human judgments and outperforms sentence-level and IBM BLEU in terms of both Spearman's rho and Kendall's tau.
1 Introduction
Many generation tasks allow diverse valid outputs, making automated evaluation difficult because plausible responses can be enormous. deltaBLEU addresses this challenge by incorporating human judgments into multi-reference BLEU for conversational response evaluation.
- Diverse-output tasks include translation, summarization, compression, paraphrase, image-to-text, and conversational generation.
- BLEU correlates well with human judgments in statistical machine translation but transfers less successfully to generation tasks with larger spaces of plausible outputs.
- Prior adaptations such as iBLEU and tf·idf-weighted BLEU address specific tasks but introduce task-specificity or tuning requirements.
- deltaBLEU embeds human judgments about reference quality directly into corpus-level multiple-reference BLEU, reducing repeated evaluation effort.
- Conversational response evaluation is especially challenging because possible responses are weakly constrained by conversational inputs.
- deltaBLEU correlates with human evaluation scores and is presented as a basis for automated training and evaluation of data-driven conversation systems.
2 Evaluating Conversational Responses
The paper frames conversational evaluation as scoring generated hypotheses against multiple references, but naturally occurring dialogues rarely provide enough distinct responses. Its solution is to mine candidate responses and obtain quality judgments for metric construction.
- A response system generates a well-formed, pertinent hypothesis h from message m and prior conversation history c.
- BLEU evaluates system outputs against a set of J references available for each context-message pair.
- BLEU’s correlation with human judgment improves as the number of references increases, but conversational data typically supplies only one response per message.
- Exact matching of repeated context-message pairs is infeasible for finding diverse responses, especially as conversational context grows.
- The proposed solution mines candidate responses from conversational data and has judges rate their quality for use in the new metric.
3 Discriminative BLEU
deltaBLEU extends BLEU by weighting reference matches with human judgments, rewarding matches to good responses and penalizing matches to bad ones. It retains BLEU as a special case and is not restricted to conversation.
- deltaBLEU assigns each reference a human-quality weight wi,j ∈ [−1, +1] within multiple-reference BLEU.
- Positive weights reward matches with good reference responses, while negative weights penalize matches with bad reference responses.
- Each n-gram match uses the highest-scoring reference containing it, and that weight may be negative.
- When all reference weights equal 1, deltaBLEU is identical to BLEU; its maximum theoretical score is also 1.
- Although particularly appropriate for response generation, deltaBLEU makes no task-specific assumption and applies to machine translation and image captioning.
4 Data
The study constructs judged multi-reference conversational datasets by mining similar responses from a large Twitter corpus. It evaluates both references and generated outputs with independent crowdsourced judgments.
- Approximately 33K candidate triples were sampled from a 29M-triple Twitter corpus and rated by three crowdsourced annotators.
- The 4232 triples averaging at least 4 were randomly divided into seed development and test sets of 2118 and 2114 triples.
- BM25 retrieved the top 15 responses for each seed message-response pair, using similarity between original and candidate messages and responses.
- Five crowdsourced raters scored mined responses on a 5-point scale using both context and message, and all candidates were retained after scaling scores to [−1, +1].
- Table 1 illustrates mined references whose weights range from strongly negative, semantically unrelated responses to positive, relevant responses.
- Outputs from seven systems were evaluated by five raters through 12 system-pair comparisons, totaling about 126K pairwise ratings.
- Human judgments for multi-reference quality and system outputs were collected independently, and retrieval-based systems were excluded to avoid spurious correlation.
5 Setup
The study evaluates metric–human agreement using 100-response observation units and compares corpus-level BLEU, deltaBLEU, and averaged sentence-level BLEU. This design follows evidence that larger observation units yield stronger correlations.
- 5 Setup: The study uses Kendall’s tau and Spearman’s rho to measure agreement between metric-score differences and human-rating differences.Each observation pairs automatic-score and qualitative-rating differences for two systems on a test subset.
- 5 Setup: Prior work shows that metric correlations generally increase as observation units become larger, with corpus- or system-level units exceeding sentence-level units.The paper uses this evidence to motivate its observation-unit design.
- 5 Setup: The evaluation compares BLEU, deltaBLEU, and sentence-level BLEU averaged across 100-response observation units.Human ratings and metric scores are computed on the same sets of 100 responses.
6 Results
deltaBLEU correlates better with human judgments than BLEU in the reported best configurations, while reference weighting affects the metrics differently. The evaluation uses confidence intervals and examines single, high-scoring, and all-reference settings.
- 6 Results: deltaBLEU achieves better correlation with human judgments than BLEU when comparing the best configuration of each metric.The reported confidence intervals are (.415, .546) for deltaBLEU and (.265, .416) for BLEU.
- 6 Results: The analysis randomizes system-pair presentation order to raters and samples 1K sentence assignments for the correlation estimates.All metrics use the same sampled assignments, with sentences grouped by system pair.
- 6 Results: The results compare IBM BLEU, sentence-level BLEU, and deltaBLEU across single-only, high-scoring, and all-reference conditions.Table 2 reports human correlations with 95% confidence intervals.
- 6 Results: BLEU’s confidence interval is (.265, .416), whereas deltaBLEU’s is (.415, .546).The intervals barely overlap for the reported comparison.
- 6 Results: Adding lower-scoring references increases deltaBLEU’s correlation coefficients but degrades BLEU’s coefficients.BLEU treats all references equally, whereas deltaBLEU discriminates among them using their ratings.
- 6 Results: BLEU and sentence-level BLEU obtain roughly comparable correlation coefficients on this task and data.The paper associates the observed difference from prior work with observation-unit size rather than only micro- versus macro-averaging.
7 Conclusions
∆BLEU outperforms IBM BLEU and sentence-level BLEU for generated conversational responses, while Figure 2 examines metric behavior across thresholds, unit sizes, and BLEU orders. Its broader application requires an upfront human-evaluated reference set, and evaluation on other tasks remains future work.
- ∆BLEU outperforms both IBM BLEU and sentence-level BLEU for generated conversational responses.
- Figure 2 compares BLEU variants as reference-score thresholds decrease, correlation-study units increase, and BLEU order rises from 1 to 4.
- ∆BLEU requires an upfront human evaluation of the reference set before reducing the need for further evaluation during development.
- Evaluation of ∆BLEU on image-to-text, sentence compression, paraphrase generation, and statistical machine translation remains future work.