PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Judge for Evaluating Alignment

Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, Pengfei Liu

arXiv:2310.05470v2cs.CLcs.AI

TL;DR

LLM alignment tasks require evaluation that generalizes across scenarios, supports multiple protocols, and provides interpretable explanations. The paper introduces AUTO-J, a 13B generative judge trained on real-world queries and responses with a 58-scenario evaluation testbed. AUTO-J outperforms open-source and closed-source baselines across pairwise and single-response evaluation settings.

  • Problem

    Evaluation for human-oriented LLM tasks lacks sufficient generality across real-world scenarios, flexibility across protocols, and interpretability beyond numerical scores.

  • Method

    AUTO-J is a 13B generative judge trained on real-world queries and model responses, with 58-scenario data supporting pairwise comparison and single-response evaluation with critiques.

  • Results

    AUTO-J establishes state-of-the-art performance among open-source models across all 58 scenarios and surpasses ChatGPT and Claude-2 in pairwise evaluation.

  • Takeaways & Limitations

    The released model, judgment dataset, criteria, prompts, and testbeds provide resources for training and benchmarking LLM evaluation methods.

Abstract

from arXiv · show

The rapid development of Large Language Models (LLMs) has substantially expanded the range of tasks they can address. In the field of Natural Language Processing (NLP), researchers have shifted their focus from conventional NLP tasks (e.g., sequence tagging and parsing) towards tasks that revolve around aligning with human needs (e.g., brainstorming and email writing). This shift in task distribution imposes new requirements on evaluating these aligned models regarding generality (i.e., assessing performance across diverse scenarios), flexibility (i.e., examining under different protocols), and interpretability (i.e., scrutinizing models with explanations). In this paper, we propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios and accommodates diverse evaluation protocols (e.g., pairwise response comparison and single-response evaluation) with well-structured natural language critiques. To demonstrate the efficacy of our approach, we construct a new testbed covering 58 different scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models, by a large margin. We also provide detailed analysis and case studies to further reveal the potential of our method and make a variety of resources public at https://github.com/GAIR-NLP/auto-j.

1 INTRODUCTION

As LLMs increasingly address human-oriented tasks, evaluation must become more general, flexible, and interpretable. AUTO-J addresses these requirements with a 13B generative judge trained on real-world queries and responses, achieving strong performance across evaluation settings.

  • Motivation: Evaluation methods must support diverse real-world scenarios where gold references are often unavailable.Traditional metrics commonly rely on human references and constrained task-specific metrics.
  • Motivation: Evaluation methods must accommodate both pairwise comparison and single-response assessment.These protocols serve different uses, including reward-model training and final-system evaluation.
  • Motivation: Interpretability requires explanations alongside numerical evaluation scores.Natural-language explanations can improve reliability and support human involvement in evaluation.
  • Approach: AUTO-J is a 13B generative judge trained on user queries and model-generated responses from massive real-world scenarios.Its training data and design target generality, flexibility, and interpretability.
  • Results: AUTO-J improves pairwise evaluation by 8.9% over open-source models and by 12.1% and 12.4% over ChatGPT and Claude-2, respectively.The model supports both pairwise and single-response evaluation with structured natural-language critiques.
  • Resources: The paper contributes a 58-scenario judgment dataset containing numerical or pairwise judgments and critiques based on 332 curated criteria.The authors also release models, scenario resources, evaluation criteria, prompts, and formatted judgments.

2 RELATED WORK

Prior work uses human judgments or strong LLMs to evaluate model outputs, but human annotation is costly and proprietary evaluators create cost and reproducibility concerns. Existing meta-evaluation testbeds vary substantially in size and source coverage.

  • Evaluation of LLMs: Human evaluation is costly, time-consuming, and laborious, motivating automated proxies based on strong LLMs.Proprietary models can achieve promising agreement with human judgments under suitable prompting.
  • Evaluation of LLMs: Closed-source evaluators incur API costs and can reduce reproducibility when models behind the APIs change.Open-source alternatives include critique models trained from feedback or revised generations.
  • Meta-evaluation testbeds: Meta-evaluation testbeds assess the quality of LLM evaluators themselves.Prior resources include human-crafted, crowdsourced, and preference-annotated collections.
  • Meta-evaluation testbeds: Existing testbeds range from 80 human-crafted queries to more than 30K crowdsourced queries and other sets containing 352 to 2,553 samples.They differ in data sources, annotation types, and evaluation scale.

3 DATA CONSTRUCTION

The data-construction pipeline defines scenarios and criteria, collects real-world queries and model responses, and generates judgments for multiple evaluation protocols. It covers 58 scenarios and uses distinct procedures for pairwise and single-response judgments.

  • Pipeline: The pipeline has three steps: defining scenarios and criteria, collecting queries and responses, and generating evaluation judgments.The construction targets high-quality training and testing data from massive real-world scenarios.
  • Scenario and criteria definition: The dataset defines 58 scenarios grouped into Summarization, Exam Questions, Code, Creative Writing, Functional Writing, Rewriting, General Communication, and NLP Tasks.One additional “others” scenario is included.
  • Scenario and criteria definition: The authors craft 332 criteria, combining scenario-specific criteria with basic criteria shared across multiple scenarios.These criteria guide evaluation models through names and descriptions.
  • Judgment generation: Pairwise judgments are generated with GPT-4 using scenario-specific criteria, reformatted with heuristics, and filtered against human annotations or formatting failures.The final pairwise training set contains 3,436 samples.
  • Judgment generation: Single-response judgments use 960 balanced query-response pairs and combine critiques generated with and without scenario criteria before producing a final rating.This divide-and-conquer strategy avoids the performance impairment observed when criteria are directly imposed as the system message.

4 TRAINING AUTO-J

AUTO-J is trained to handle pairwise and single-response evaluation through prompt-controlled protocol switching. Pairwise data augmentation swaps response order and judgment labels to reduce positional bias.

  • Protocol flexibility: AUTO-J toggles between pairwise and single-response evaluation by applying the corresponding prompts.The model integrates data from both evaluation protocols during training.
  • Bias mitigation: Pairwise training samples are augmented by swapping response order and alternating the corresponding response labels.This augmentation is designed to lessen positional bias in pairwise comparison.
  • Implementation: AUTO-J is trained from LLaMA-2-13B-chat using DeepSpeed, ZeRO Stage 3, gradient checkpointing, and FlashAttention on 8 NVIDIA A100 GPUs.Training uses BF16 and TF32 mixed precision and runs for 5 epochs.

5 EVALUATION SETTING

The evaluation setting tests judges on pairwise comparison, critique generation, and overall single-response rating across diverse model outputs and scenarios.

  • Task I: Pairwise Response Comparison: Eval-P asks evaluators to choose the better response or a tie for each query-response pair.The test set contains 1,392 samples across 58 scenarios, with Win/Tie/Lose counts of 520/373/499.
  • Task II: Critique Generation for Single Response: Eval-C measures critique quality by comparing critiques of less-preferred responses using GPT-4 and expert human judgments.It contains 232 query-response pairs, sampling four queries per scenario from Eval-P.
  • Baselines: The evaluation compares AUTO-J with general-purpose models including LLaMA-2-Chat, Vicuna, WizardLM, ChatGPT, GPT-4, Claude-2, and LLaMA-2-Chat-70B.The model set varies by task and includes both open-source and proprietary systems.

6 EXPERIMENTS

Experiments evaluate AUTO-J’s consistency, critique quality, rating usefulness, system-level ranking, and explanation behavior across pairwise and single-response settings. AUTO-J generally matches or exceeds strong baselines, including GPT-4 on several evaluations, while producing structured critiques.

  • 6.1 Pairwise Response Comparison: Swapping response order tests positional-bias robustness, and AUTO-J’s consistency is similar to GPT-4 and higher than other baselines.Judgments count as agreeing with humans only when both orderings are consistent and match the human preference.
  • 6.2 Critique Generation for Single-Response: AUTO-J significantly outperforms all baselines, including GPT-4, when GPT-4 and humans compare critiques in Eval-C.GPT-4 gives few ties, whereas humans sometimes assign ties to more than 30% of comparisons.
  • 6.3 Overall Rating for Single Response: AUTO-J-selected responses generally receive higher GPT-4 ratings than baseline-selected responses across Best-of-N values of 8, 16, and 32.The comparison uses the average GPT-4 rating of responses selected by different rating models.
  • 6.3 Overall Rating for Single Response: AUTO-J’s ratings correlate better with GPT-4 ratings than all baselines on 1,993 query-response pairs.The analysis reports both Spearman and Pearson response-level correlations.
  • 6.4 Analysis and Case Studies: AUTO-J’s system-level rankings correlate with GPT-4’s AlpacaEval leaderboard rankings at 0.97 Spearman and 0.96 Pearson correlation.The authors use average AUTO-J ratings for open-source LLMs on the leaderboard.
  • Ablation Studies: An ablation shows that supporting multiple protocols and explanations does not reduce pairwise agreement, while AUTO-J matches a standard reward model and correlates better with GPT-4.The final-decision-only variant reaches 55.0 agreement versus 54.8 for AUTO-J; the standard reward model reaches 54.5.
  • 6.4 Analysis and Case Studies: Case studies show AUTO-J identifying tone and interactivity differences, novice-specific cooking concerns, and better Best-of-N responses.The Best-of-N case is evaluated using both GPT-4 ratings and human observation.

7 CONCLUSION

The paper develops AUTO-J, a 13B generative judge for alignment evaluation, addressing generality, flexibility, and interpretability. It builds a diverse judgment dataset and reports superior performance against open- and closed-source baselines.

  • AUTO-J is a 13B generative judge designed to address generality, flexibility, and interpretability in alignment evaluation.
  • The authors create a judgment dataset covering diverse evaluation protocols, user queries, model responses, real-world scenarios, and structured natural-language critiques.
  • Experiments show AUTO-J significantly outperforms both open-source and closed-source baseline models.
  • The authors release resources intended to facilitate future research.

A SCENARIO DESCRIPTION

This section documents the paper’s scenario definitions and the scenario-classifier dataset distribution. The accompanying tables provide detailed scenario descriptions and train/test distributions.

  • Table 6 provides a detailed description for each scenario.
  • Table 7 reports the scenario distribution in the training and test sets for the scenario classifier.
  • The scenario-classifier distribution notes that “rejecting” and “peer_review” were early-defined scenarios later removed.

B TRAINING DETAILS OF SCENARIO CLASSIFIER

The scenario classifier is trained as a generation task that outputs a scenario name from a query. Its data pipeline combines synthetic seeds, automatically classified conversations, manual validation, and final retraining.

  • Classifier formulation: The classifier generates only a scenario name from a user query, using “default” when uncertain.
  • Training pipeline: Training begins with roughly 10 ChatGPT-assisted seed queries per scenario and a conditional-generation model trained to output scenario names.
  • Training pipeline: The model initially classifies Chatbot Arena Conversations and ShareGPT 4 queries, then manually validates up to 50 queries per scenario.
  • Training pipeline: The final classifier combines synthetic and newly collected data, uses an 8:2 train/test split, and reaches accuracy 72.55 and F1 74.12.
  • Implementation: Queries longer than 1,998 tokens are truncated from the middle during classifier training and inference.
  • Supplementary materials: The section also documents scenario criteria, prompts, judgment formats, and supplemental training details in Tables 8–20.

E TRAINING DATA STATISTICS

This section presents training-data statistics as supplemental details of the paper’s data construction. The statistics are organized in Tables 20–21.

  • The section shows statistics for the training data.
  • The training-data statistics are presented in Tables 20–21.
  • The section is described as supplemental to the training-data construction discussion in §3.3.

F COMPLETE RESULTS AND CASES

This section provides detailed comparison results, prompt formats, training-data statistics, and complete case studies for the evaluations. It also documents rankings, critiques, and a Best-of-N selection example involving AUTO-J.

  • Evaluation formats: Tables 16-19 specify AUTO-J input formats, unified judgment formats, and complete single-response evaluation judgments.Table 19 includes queries, responses, GPT-4 critiques with and without scenario criteria, and a combined final judgment.
  • Detailed results: Table 22 provides detailed comparison results between AUTO-J and baseline critiques for single-response evaluation under GPT-4 and human judgments.The listed baselines include Vicuna-13B-v1.5, LLaMA-2-Chat-13B, and WizardLM-13B-v1.2.
  • Detailed results: Tables 23-27 report open-source-model rankings, complete pairwise judgments, critique comparisons, and a Best-of-N selection example.Table 23 compares AUTO-J and GPT-4 rankings on AlpacaEval, while Tables 24-27 expand the case studies and selection analysis.
Loading 2310.05470v2…