LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Models

• Human evaluation remains the gold standard but is labor-intensive, time-consuming, and costly, motivating automated assessment.Source: §1 · p. 1
• Existing automatic evaluators are often task- and attribute-specific, limiting their extension to uncommon NLG properties or inputs.Source: §1 · p. 1
• Prompt-scoring approaches often require massive 175B+ parameter LLMs, restricting applicability when access is limited to APIs.Source: §1 · p. 2
• Comparative assessment is more effective than prompt scoring for FlanT5 and Llama-style models, approaching or exceeding state-of-the-art performance across diverse NLG tasks and attributes.The method compares candidate outputs pairwise rather than scoring each independently.Source: §1 · p. 2
• LLM pairwise comparisons can exhibit strong positional bias, and the paper proposes debiasing that further improves performance, especially when bias is large.Source: §1 · p. 2

2.1 Reference-based Evaluation

• Reference-based evaluation compares generated outputs with gold references using n-gram metrics such as ROUGE and METEOR or embedding-based metrics such as BERTScore.Source: §2.1 · p. 2

2.2 Tailored NLG Evaluation Approaches

• Tailored evaluators target specific properties, including summary consistency and dialogue quality, but use task- or attribute-specific procedures.Source: §2.2 · p. 2

2.3 Zero-shot LLM Evaluation

• Zero-shot LLM evaluation has shown that systems such as ChatGPT can assess open-ended generation and score text quality along particular axes.Source: §2.3 · p. 2

2.4 LLM Pairwise Comparisons

• LLM pairwise comparison has been relatively underexplored for NLG, despite prior uses of pairwise judgement for exams, information retrieval, and chat-assistant assessment.Source: §2.4 · p. 2

3.1 Notation

• The framework predicts candidate ranks by modeling the probability that one response is better than another for a given context and attribute.Source: §3.1 · p. 3

3.2 Prompt Design

• Comparative prompts convert candidate responses and context into an LLM decision, while prompt design changes only the assessed attribute or response label.Source: §3.2 · p. 3, Figure 2 · p. 3

3.3 Comparative Decisions

• Comparative decisions can use label-token probabilities when available or sampled text responses when only limited API access is available.Source: §3.3 · pp. 3–4

3.4 Comparisons to Ranks

• Candidates are ranked by win-loss ratios from selected comparisons, with random, no-repeat, and symmetric strategies reducing the cost of exhaustive comparison.Source: §3.4 · p. 4

3.5 Debiased Comparative Assessment

• Positional bias is quantified through first- and second-position selection priors, then corrected by reweighting probabilities so symmetric comparisons yield P(A) = P(B) = 0.5.Source: §3.5 · p. 4

4.1 Datasets

• The study evaluates summarization, podcast-summary, dialogue, and data-to-text quality across SummEval, Podcast, TopicalChat, and WebNLG.The benchmarks cover attributes including coherence, consistency, fluency, relevance, engagingness, naturalness, grammar, and semantic equivalence.Source: §4.1 · p. 4

4.2 Base Large Language Models (LLMs)

• The experiments use instruction-tuned FlanT5 models from 220M to 11B parameters and Llama2-chat models at 3B and 13B parameters.FlanT5 is instruction-tuned on a diverse set of 1000 NLP tasks, while Llama2 is tuned on instruction datasets.Source: §4.2 · pp. 4–5

4.3 Baselines

• Baselines include reference-based and reference-free NLG metrics, bespoke supervised systems, prompt scoring, G-Eval, GPTScore, and prompt-based classifiers.Reference-based methods compare outputs with references, whereas reference-free methods compare generated text with the source or context.Source: §4.3 · p. 5, §4.3.1 · p. 5, §4.3.2 · p. 5

4.3.1 Bespoke Methods

• Bespoke methods rely on task-specific supervision or synthetic training data, including Longformer-SFT trained on human podcast-summary judgements with 5-fold cross-validation.UniEval converts assessment into Boolean question answering, while QuestEval and MQAG use question-answering approaches for summarization consistency.Source: §4.3.1 · p. 5

4.3.2 Zero-shot Methods

• Zero-shot methods apply without further training or fine-tuning, with prompt scoring asking models for 1–10 ratings and G-Eval estimating continuous scores from score probabilities.The scoring templates vary the evaluated attribute and response description across tasks.Source: §4.3.2 · p. 5, Figure 3 · p. 5

4.4 Methodology

• Comparative assessment evaluates all candidate pairs in both permutations, selects winners from Response A and Response B probabilities, ranks candidates by win-ratio, and measures Spearman correlation with human judgements.For Llama2, comparative prompts end with “Answer:” whereas scoring prompts end with “Score:”.Source: §4.4 · pp. 5–6

5.1 NLG Evaluation Results

• Prompt-scoring performs weakly for moderate-sized models, while comparative assessment yields more robust correlations, typically 30–50 for most 3B+ systems.G-Eval improves over prompt-scoring, but comparative assessment remains more effective in most settings.Source: §5.1 · p. 6
• Comparative assessment outperforms prompt-scoring across most settings, with FlanT5-3B competitive with zero-shot baselines and best on 3 of 4 SummEval aspects.It also achieves competitive performance with UniEval while remaining zero-shot and general, whereas UniEval degrades in out-of-domain settings.Source: §5.1 · p. 6
• Comparative assessment remains effective across domains: it outcompetes all baselines for long-input Podcast summarization and outperforms prompt-scoring on TopicalChat.For WebNLG, semantic understanding strongly favors 10B+ systems over 3B+ systems, while grammar and fluency are more similar across sizes.Source: §5.1 · p. 7

5.2 Positional Bias

• Comparative prompts show strong positional bias: although unbiased selection would yield P(A)=0.50, some systems select the first option 80% of the time.Larger systems appear more susceptible to this bias, potentially explaining similar performance between 3B and 11B/13B systems.Source: §5.2 · p. 7, Table 5 · p. 7

5.3 Debiasing

• Debiasing improves comparative assessment, especially for highly biased prompts and small comparison sets, with large TopicalChat gains enabling state-of-the-art performance for FlanT5-11B.At the comparison level, accuracies are 60–80%, and debiasing can substantially increase them.Source: §5.3 · p. 8, §5.4 · p. 8, Table 6 · p. 8

5.5 Self-Consistency

• Rankings remain highly self-consistent with few sampled outputs, while subset selection offers similar performance with fewer comparisons and symmetric selection differs by only 1 Spearman point from no-repeat.Full comparison requires O(N^2) comparisons; no-repeat marginally improves over random selection at small R, and debiasing helps most when comparisons are scarce.Source: §5.5 · p. 8, Table 8 · p. 8, §5.6 · pp. 8–9

• Comparative assessment outperforms absolute scoring for moderately sized LLMs and achieves near state-of-the-art performance across a range of NLG evaluation tasks.Source: §6 · p. 9

• N · (N − 1) comparisons can be computationally prohibitive as the number of candidates grows.Source: Limitations · p. 9
• The empirical findings cover LLMs up to 13B parameters and are limited to open-source models because of API costs and comparison complexity.Source: Limitations · p. 9
• The comparison-selection study considers only static schemes, leaving dynamic selection methods for future work.Source: Limitations · p. 9

• Comparative assessment may be ineffective for some tasks or datasets and generalize poorly across the task.Source: Ethics Statement · p. 9
• Deploying machine-learning classifiers in real-world classification settings carries associated risks requiring careful analysis.Source: Ethics Statement · p. 9
• Misuse or overconfidence in the approach may reduce users’ trust in LLM solutions.Source: Ethics Statement · p. 9

A.1 Partial Comparison Curves

• Figure 5 extends Figure 4 by examining assessment performance when only a subset of comparisons is considered across models, datasets, and scores.Source: Figure 5 · p. 12

A.2 Positional Bias

• Table 9 reports how often the first-position candidate is selected under full symmetric comparisons across prompts, datasets, and scores.Source: Table 9 · p. 12

A.3 Accuracy of Pairwise Comparisons

• Table 10 extends Table 6 with pairwise-comparison accuracy for candidates differing in true value across datasets and scores.Source: Table 10 · p. 13

B Alternate Ranking Strategies

• Beyond win-ratio ranking, ranks can be selected by maximizing the likelihood of observed comparisons or an approximation when only hard decisions are available.The likelihood-based formulation uses comparison probabilities; hard decisions permit an approximate-likelihood objective.Source: B Alternate Ranking Strategies · p. 13
• The comparative statistics can estimate the conditional comparison probabilities needed to define the likelihood when binary outcomes are observed.Source: B Alternate Ranking Strategies · p. 13

• Similar Spearman correlation was obtained with win-ratio and maximum-likelihood conversions for FlanT5-3B on SummEval.The comparison is reported in Table 11.Source: §B.1 · p. 13, Table 11 · p. 13
• Maximum-likelihood optimization poses intractability challenges and therefore requires an approximate greedy search.The main paper consequently focuses on the simpler win-ratio method.Source: §B.1 · p. 13