Source-linked AI summary
Towards a Unified Multi-Dimensional Evaluator for Text Generation
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, Jiawei Han
TL;DR
NLG automatic evaluation remains dominated by similarity metrics despite the need for comprehensive, explainable assessment. UNIEVAL reframes evaluation as Boolean QA and adds intermediate related-task learning, achieving stronger human-judgment correlation across tasks and zero-shot transfer. Its main limitations are black-box behavior and noise in synthetic training data.
Problem
Automatic NLG evaluation is dominated by similarity-based metrics, while reliable multi-dimensional evaluation of advanced models remains limited.
Method
UNIEVAL uses one Boolean QA evaluator for multiple dimensions, with intermediate multi-task learning followed by sequential training on synthetic evaluation data.
Results
UNIEVAL correlates better with human judgments across three typical NLG tasks, improving correlation by 23% on summarization and over 43% on dialogue response generation versus top unified evaluators.
Takeaways & Limitations
UNIEVAL supports unified multi-dimensional evaluation and transfers to unseen evaluation dimensions and NLG tasks in zero-shot settings.
Takeaways & Limitations
UNIEVAL remains a black-box model, and its synthetic pseudo-data contain noise that can mislabel negative examples.
Abstract
from arXiv · showhide
Multi-dimensional evaluation is the dominant paradigm for human evaluation in Natural Language Generation (NLG), i.e., evaluating the generated text from multiple explainable dimensions, such as coherence and fluency. However, automatic evaluation in NLG is still dominated by similarity-based metrics, and we lack a reliable framework for a more comprehensive evaluation of advanced models. In this paper, we propose a unified multi-dimensional evaluator UniEval for NLG. We re-frame NLG evaluation as a Boolean Question Answering (QA) task, and by guiding the model with different questions, we can use one evaluator to evaluate from multiple dimensions. Furthermore, thanks to the unified Boolean QA format, we are able to introduce an intermediate learning phase that enables UniEval to incorporate external knowledge from multiple related tasks and gain further improvement. Experiments on three typical NLG tasks show that UniEval correlates substantially better with human judgments than existing metrics. Specifically, compared to the top-performing unified evaluators, UniEval achieves a 23% higher correlation on text summarization, and over 43% on dialogue response generation. Also, UniEval demonstrates a strong zero-shot learning ability for unseen evaluation dimensions and tasks. Source code, data and all pre-trained evaluators are available on our GitHub repository (https://github.com/maszhongming/UniEval).
1 Introduction
NLG evaluation has moved beyond similarity scores toward explainable multi-dimensional assessment, motivating UNIEVAL, a single evaluator that uses Boolean QA and related-task training. Across three tasks, it correlates better with human judgments and transfers to unseen settings.
- Motivation: Similarity-based metrics such as ROUGE and BLEU can miss content quality, syntactic correctness, and overall generation quality.They rely on overlap or embedding similarity with reference text, which may not reflect human-likeness or task performance.
- Motivation: Multi-dimensional evaluation assesses generated text through explainable dimensions such as coherence, consistency, fluency, and relevance.This approach is dominant in human evaluation, but using separate evaluators for every dimension is difficult.
- UNIEVAL: UNIEVAL converts evaluation dimensions into Boolean QA questions, allowing one model to assess different perspectives by changing the question.For example, coherence can be evaluated by asking whether a summary is coherent to its document.
- UNIEVAL: A unified evaluator supports joint dimension training, external knowledge incorporation, and extension to additional dimensions through continual learning or prompting.These benefits address ease of use, internal complementarity, transferability, and extensibility.
- Results: 23% higher correlation with human judgments was achieved on text summarization, while dialogue response generation improved by over 43% against top-performing unified evaluators.Transfer experiments also showed stronger zero-shot performance on unseen dimensions and NLG tasks.
2 Related Work
Related work spans similarity-based metrics, single-dimensional evaluators, and unified evaluators, but each leaves gaps in comprehensive, explainable assessment. UNIEVAL is positioned as a unified framework aligned with human-evaluation dimensions.
- Similarity-based Metrics: Similarity-based metrics measure generated-reference similarity through lexical overlap or contextualized embeddings.ROUGE and BLEU remain widely used, although such metrics fail to measure content quality and syntactic correctness.
- Single-dimensional Evaluator: Single-dimensional evaluators provide fine-grained views such as summarization consistency or dialogue coherence.Because many dimensions lack standard evaluators, combining multiple specialized models is difficult.
- Unified Evaluator: Unified evaluators predict multiple evaluation scores using varied inputs, model variants, or formulas.Their scores often lack corresponding explanations or are reduced to generic precision, recall, and F1 categories.
3 Method
UNIEVAL formulates multi-dimensional NLG evaluation as Boolean QA, then trains a single evaluator with synthetic dimension data and intermediate related-task learning. Sequential training is used to address negative transfer across dimensions.
- Problem Formulation: Each evaluation dimension becomes a Boolean question, and the evaluator answers “Yes” or “No” for the candidate output, reference, and context.Changing the question lets one evaluator assess different dimensions.
- Problem Formulation: The score s_i is the model’s normalized probability of answering “Yes” rather than “No”.P(·) denotes the probability of generating a specific word.
- Unsupervised Learning: UNIEVAL uses a T5 backbone and constructs positive and negative pseudo-data with dimension-specific rules because large-scale human scores are unaffordable.Ground-truth outputs serve as positive samples, which are transformed into negative examples.
- Unsupervised Learning: For summarization, transformations target coherence, consistency, fluency, and relevance through sentence replacement, factual edits, and span operations.The reported operations include retrieval-based replacement, antonym substitution, numerical editing, entity replacement, syntactic pruning, repetition, deletion, and shuffling.
- Training Strategy: Sequential training is adopted because straightforward multi-task learning creates negative transfer in dimensions such as coherence and engagingness.The paper reports that sequential training can alleviate this problem.
- Intermediate Multi-task Learning: Intermediate multi-task learning adds external knowledge from NLI, opening sentence prediction, and linguistics-related data before evaluation-task training.These tasks are converted into the same Boolean QA format, with entailment mapped to “Yes” and other NLI labels to “No”.
4 Experiments
Experiments evaluate UNIEVAL across summarization, dialogue response generation, and data-to-text using meta-evaluation benchmarks and correlations with human judgments. UNIEVAL generally outperforms existing evaluators, while intermediate-task training supports performance and transfer.
- Experimental Setup: UNIEVAL is evaluated on summarization, dialogue response generation, and data-to-text, covering compression, creation, and transduction tasks.Summarization and dialogue generation also test transfer to unseen evaluation dimensions.
- Baselines: Similarity-based metrics correlate relatively well with selected dimensions but perform poorly on other summarization and dialogue evaluation dimensions.BERTScore and MoverScore measure similarity, while BARTScore and USR provide unified baselines.
- Summarization and Dialogue Results: UNIEVAL achieves the best correlation in all tested summarization dimensions among single-dimensional evaluators and surpasses CTC models.The results support the effectiveness of the Boolean QA formulation for strengthening the pretrained backbone.
- Summarization and Dialogue Results: 23% is the improvement over BARTScore after continual learning on summarization, exceeding the multi-task-learning improvement of 15% across dimensions.The main training-strategy gap is negative transfer on coherence; the unified version improves coherence and fluency relative to its single-dimensional version.
- Summarization and Dialogue Results: 48.9% and 43.2% are UNIEVAL (Continual)’s average Pearson and Spearman improvements over USR on dialogue response generation.The unified version improves the remaining dimensions over its single-dimensional version based on Spearman correlation, although naturalness declines.
- Transfer Experiments: UNIEVAL transfers to unseen dimensions and tasks, outperforming USR on understandability and BARTScore on most data-to-text dimensions.For unseen dialogue understandability, Pearson rises from 0.326 to 0.380 and Spearman from 0.327 to 0.468.
- Ablation Study: The combination of all four intermediate-task types produces the best NLG evaluation performance.NLI contributes most to consistency, opening-sentence prediction supports coherence, and generic QA enhances each dimension by focusing on the input question.
5 Conclusion
The paper proposes UNIEVAL, a unified multi-dimensional evaluator that correlates well with human judgments across three typical generation tasks and transfers to unseen settings.
- UNIEVAL provides unified multi-dimensional evaluation for various NLG tasks.
- UNIEVAL correlates well with human judgment on three typical generation tasks.
- UNIEVAL exhibits excellent transfer performance.
Limitations
The paper identifies limitations concerning interpretability, synthetic-data noise, computational scale, and language coverage.
- UNIEVAL is a black-box model, leaving its evaluation-score prediction process unclear.The authors identify interpretable multi-dimensional evaluation as a future direction.
- Synthetic pseudo-data contain noise, which can mislabel examples when deleting an unimportant span does not affect fluency.
- Experiments use only T5-large because of limited computational resources.The authors suggest smaller models and larger training datasets as future directions.
- UNIEVAL is limited to English tasks, leaving cross-language generation tasks for future work.
A Dimensions in Evaluation tasks
The evaluation dimensions cover summarization, dialogue response generation, and data-to-text, with task-specific criteria for judging generated outputs.
- Task coverage: The appendix provides detailed descriptions of dimensions for dialogue response generation and data-to-text tasks.
- Dialogue response generation: Dialogue response generation is evaluated on naturalness, coherence, engagingness, groundedness, and understandability.
- Data-to-text: Data-to-text is identified as a separate evaluation-task setting.
- Text summarization: Naturalness asks whether an utterance could plausibly have been produced by a human.
- Text summarization: Informativeness asks whether an utterance contains all information in the given content.
A.2 Pseudo Data Construction for Dialogue Response Generation
Pseudo-data for dialogue evaluation are constructed with dimension-specific positive and negative examples, using corpus-based transformations and generated responses.
- Naturalness pseudo-data modify the fluency construction by setting λ to 3.
- Coherence pseudo-data use gold responses from other dialogues as negative samples.
- Engagingness pseudo-data use DialogGPTsmall responses generated from one sentence as unattractive samples.
- Groundedness pseudo-data pair paraphrases of current knowledge-context sentences with sentences sampled from other contexts as negatives.
- Pseudo-data are constructed on the CNN/DailyMail and Topical-Chat corpora.
- Reference text is supplied only for summarization relevance; other dimensions use a reference-free evaluator with dimension-specific contexts.
B Examples for Intermediate Tasks
Intermediate tasks use the same Boolean QA format as evaluation tasks, representing inputs as context-question pairs and producing binary answers.
- Examples for each intermediate task are provided in Table 8.
- Intermediate-task inputs are defined as (c, q) pairs, and the model answers with “Yes” or “No”.
C Implementation Details
The implementation trains T5 on intermediate tasks before evaluation-task training, using pseudo-data and task-specific scoring adjustments. The paper also documents the input formats, intermediate-task examples, and correlation metrics used in its tables.
- Training: T5 is first trained on intermediate tasks for 2 epochs.
- Training: Pseudo-data contain 30k samples per evaluation dimension with equal positive and negative examples, using CNN/DailyMail and Topical-Chat.
- Training: The evaluator is trained for 1-3 epochs across NLG tasks on two A6000 GPUs for a total of 5 hours.
- Scoring: For fluency and consistency in summarization, the output is split into sentences, sentence scores are computed, and the final score averages them.
- Scoring: Engagingness in dialogue uses score summation, giving it a range of [0, +∞), while other dimensions range from [0, 1].
D Results on QAGS
The QAGS benchmark evaluates consistency across CNN/DailyMail and XSum, whose extractive and abstractive character differs. UNIEVAL correlates well across both subsets and outperforms prior consistency evaluators, especially on XSum.
- QAGS contains CNN/DailyMail and XSum summarization corpora for consistency meta-evaluation.
- UNIEVAL (Consistency) correlates well on both benchmark subsets, particularly on the more challenging XSum dataset.
- More than 30%: UNIEVAL (Consistency) outperforms the state-of-the-art CTC evaluator on average using Spearman and Kendall-Tau correlations.
- CNN/DailyMail references tend to copy the original text, whereas XSum summaries often contain words absent from the original text.