PaperlayerView exact revision on arXiv

Source-linked AI summary

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, Minjoon Seo

arXiv:2307.10928v4cs.CLcs.AI

TL;DR

LLM evaluation is difficult because instructions require varying combinations of abilities, while coarse overall scoring limits interpretability. FLASK decomposes evaluation into instruction-specific skill scores and finds improved reliability and high human–model evaluation correlation across LLM comparisons.

  • Problem

    Open-ended instructions require different combinations of skills, but existing benchmark and overall-scoring approaches provide limited interpretable alignment evaluation.

  • Method

    FLASK annotates each instruction with relevant metadata and scores its required skills using skill-specific or instance-specific rubrics with human or model evaluators.

  • Results

    Fine-grained evaluation improves interpretability and reliability, with highly correlated human-based and model-based results across evaluated open-source and proprietary LLMs.

  • Takeaways & Limitations

    FLASK provides skill-, domain-, and difficulty-level analysis for interpreting LLM capabilities and comparing models for specific situations.

  • Takeaways & Limitations

    Model-based evaluators can prefer particular response styles, such as GPT-3.5’s style over BARD’s, unlike human evaluators.

Abstract

from arXiv · show

Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations. We publicly release the evaluation data and code implementation at https://github.com/kaistAI/FLASK.

1 INTRODUCTION

FLASK addresses the limitations of coarse, overall-response evaluation by scoring the skills required for each instruction. It enables interpretable comparisons and improves evaluation reliability across human and model-based evaluators.

  • Open-ended instructions require multiple abilities, making a single evaluation metric insufficient for measuring alignment with human values.
  • Existing benchmark and overall-preference approaches limit scalability, interpretability, or reliability because they target separate skills or depend on surface-sensitive metrics.
  • FLASK defines 4 primary abilities divided into 12 skills and evaluates each instruction using its annotated skill set, domain, and difficulty.It collects 1,740 instances and assigns 1-to-5 scores using skill-specific rubrics.
  • Fine-grained evaluation improves interpretability and reliability, increasing correlation between human and model evaluations and mitigating model-based evaluation bias.
  • Open-source LLMs underperform proprietary models in Logical Thinking and Background Knowledge, while some skills require larger models and FLASK-HARD causes up to 50% degradation.
  • FLASK supports developers’ diagnosis of model alignment and practitioners’ selection of models for specific situations.

2 RELATED WORKS

Prior LLM evaluation combines independent benchmarks, automatic metrics, and holistic assessments to measure multiple capabilities. These approaches provide broader coverage but remain limited by scalability and interpretability concerns.

  • Holistic LLM evaluation is used to assess model strengths, weaknesses, and potential risks.
  • Multiple independent benchmarks commonly use automatic metrics such as accuracy for knowledge and reasoning tasks or ROUGE for long-form generation.
  • Multi-metric evaluation settings were proposed to assess multiple response aspects beyond accuracy.
  • LLM alignment research includes supervised instruction tuning and reinforcement learning from human feedback as fine-tuning approaches.

3 FLASK: FINE-GRAINED LANGUAGE MODEL EVALUATION PROTOCOL

FLASK evaluates instruction-following through instance-wise skill sets, metadata, and skill-specific scoring. Its protocol covers 12 skills across four abilities, uses human or model evaluators, and includes a challenging FLASK-HARD subset.

  • Protocol overview: FLASK defines four abilities divided into 12 skills and uses instance-wise multi-metric evaluation to enable task-agnostic assessment.Human evaluators and EVAL LM can perform the evaluations.
  • Skill set categorization: Logical Thinking covers correctness, robustness, and efficiency in reasoning processes and final answers.
  • Skill set categorization: Background Knowledge evaluates factuality and commonsense understanding when responses require general or domain-specific information.
  • Skill set categorization: Problem Handling covers comprehension, insightfulness, completeness, and metacognition when addressing instruction requirements and challenges.
  • Skill set categorization: User Alignment covers readability, conciseness, and harmlessness in relation to users’ intentions, preferences, expectations, and safety.
  • Dataset construction: The evaluation dataset contains 1,740 instances from 122 datasets, with annotations for necessary skills, target domains, and instruction difficulty.
  • Evaluation process: Evaluators score each selected skill from 1 to 5 using reference answers and skill-specific rubrics, then aggregate results by skill, domain, and difficulty.For model-based evaluation, EVAL LM generates a rationale before assigning scores.
  • FLASK-HARD: FLASK-HARD contains 89 expert-level instances and uses instance-specific skill rubrics instead of fixed rubrics for its challenging tasks.

4 RELIABILITY OF FLASK

FLASK’s fine-grained evaluation shows strong agreement between human-based and model-based judgments while improving robustness to stylistic changes. Skill-specific rubrics strengthen correlation and reduce some model-evaluation biases, although the most robust instance-specific rubric is more costly.

  • Human–model correlation: Human-based and model-based evaluations show similar skill-level patterns across GPT-3.5, BARD, VICUNA, and ALPACA.Both settings identify ALPACA as weakest on most skills and show major gaps for VICUNA and ALPACA on Logical Thinking and Background Knowledge.
  • Evaluation biases: Human evaluation is affected by central-tendency bias and fatigue, whereas model-based evaluation can prefer particular styles and longer responses.For some skills, the EVAL LM prefers GPT-3.5’s response style over BARD’s, unlike human evaluators.
  • Human–model correlation: Skill-specific fine-grained evaluation consistently produces stronger human–model correlation than skill-agnostic evaluation across EVAL LMs.The analysis reports Spearman, Kendall-Tau, and Pearson correlations and finds GPT-4 has the highest correlation among the tested EVAL LMs.
  • Human–model correlation: Removing the reference answer, rationale generation, or skill-specific score rubric significantly reduces model–human correlation.The drop is especially pronounced when the reference answer is removed.
  • Robustness to stylistic changes: Fine-grained evaluation becomes more robust to stylistic changes as evaluation moves from skill-agnostic to skill-specific and instance-specific rubrics.The instance-specific rubric is most robust, but it requires annotating subquestions and manually validating them.
  • Evaluation biases: FLASK uses instance-specific rubrics only for FLASK-HARD because their additional annotation and validation costs make whole-set deployment more expensive.The authors leave extending this rubric to the whole evaluation set and investigating other biases for future work.

5 ANALYSIS BASED ON AUTOMATIC EVALUATION OF FLASK

FLASK enables automatic, skill-level analysis of model performance across model types, scales, and instruction difficulties. The analysis reveals distinct capability gaps and scaling patterns, while harder evaluation settings expose substantial weaknesses even in proprietary models.

  • Evaluation setting: Because human evaluation is time-consuming and expensive, the analysis focuses on automatic model-based evaluation given its high correlation with human evaluation.The paper uses automatic evaluation for extensive analysis of the whole FLASK set.
  • Model comparison: Open-source models nearly match proprietary models on Problem Handling and User Alignment but show large gaps in Logical Thinking and Background Knowledge.VICUNA and WIZARDLM perform similarly across skills, while both lag GPT-3.5 particularly on reasoning and factual knowledge.
  • Model scaling: Larger models improve Logical Robustness, Logical Correctness, and Logical Efficiency faster than Readability, Harmlessness, and Metacognition.Some skills nearly saturate after 30B, 13B, or 7B, indicating different scaling requirements across skills.
  • Instruction difficulty: Scaling T¨ULU is more effective for easier instructions, reducing its gap with GPT-3.5 on simple lifestyle knowledge while leaving larger gaps at higher difficulties.Figure 6 compares GPT-3.5 with T¨ULU models across Logical Robustness, Logical Correctness, Factuality, and Completeness by instruction difficulty.
  • FLASK-HARD: Proprietary models also degrade substantially on FLASK-HARD, with large gaps from GPT-4 for Logical Correctness, Insightfulness, and Commonsense Understanding.On the standard FLASK set, CLAUDE leads Problem Handling while GPT-3.5 leads Logical Thinking and Background Knowledge.
  • FLASK-HARD: Instance-specific rubrics produce lower overall scores than skill-specific rubrics because they impose more specific requirements.The comparison is shown for proprietary models on FLASK-HARD.

6 APPLICATION OF FLASK

FLASK supports targeted model development and model selection by exposing skill-specific performance. Developers can compare models and checkpoints, while practitioners can match models to task demands using metadata about their own test sets.

  • FLASK for Developers: Developers can use FLASK to analyze their models more precisely and identify detailed action items for intermediate checkpoints.Open-source developers can compare against proprietary models, especially on Logical Thinking and Background Knowledge.
  • FLASK for Practitioners: Practitioners can annotate their own test sets with metadata and estimate which models suit different situations.The paper gives 7B fine-tuned open-source models as potentially sufficient for chit-chat, while proprietary APIs may be worthwhile for complex reasoning.

7 CONCLUSION

FLASK provides a fine-grained, interpretable framework for evaluating language-model skills across domains and difficulty levels. The authors report improved evaluation reliability, while noting limitations in both human and model-based evaluators.

  • Conclusion: FLASK categorizes 12 skills and annotates required skills, target domains, and difficulty for each instance.This supports analysis of language-model capabilities by skill, domain, and difficulty.
  • Conclusion: Fine-grained evaluation improves correlation between human-based and model-based evaluation and increases robustness to stylistic changes.The conclusion presents these as reliability benefits of the evaluation setting.
  • Limitations: Human evaluators face central-tendency bias and annotation fatigue, while model evaluators prefer longer responses and styles resembling their own writing.The paper also notes that human-based evaluation was not conducted extensively because of cost and time constraints.

A.2 SCOPE OF THE EVALUATION

The evaluation covers diverse LLMs and uses human- and model-based scoring, while restricting instances to English, single-turn, language-focused, zero-shot instructions. Agreement is moderate for human labels and higher across repeated model-based evaluations, with skill-dependent variation.

  • The evaluation is limited to monolingual English, single-turn, language-focused, zero-shot instances.
  • The study compares proprietary and open-source LLMs across model sizes, training techniques, and training datasets.
  • Human evaluation uses three labelers per instance, while model evaluation uses three nondeterministic GPT-4 runs.
  • Human-based inter-labeler agreement is 0.488, compared with 0.835 for model-based evaluation.
  • Agreement varies by skill: Logical Correctness and Commonsense Understanding are more consistent than Readability and Conciseness.

C.2 ANALYSIS OF DIFFERENT FINETUNING DATA

The analyses examine how fine-tuning data, teacher quality, RLHF, training duration, evaluator choice, and model scale affect skill-level performance. Results show that these factors produce uneven changes across skills rather than uniform gains.

  • Fine-tuning data: Fine-tuning datasets differ in their primary-ability, skill, domain, and difficulty compositions.
  • Fine-tuning data: On short-reference-answer instances, performance gaps between training datasets shrink, especially for Logical Thinking and User Alignment.
  • Teacher quality: GPT4-ALPACA 13B outperforms ALPACA 13B on all skills, but still underperforms GPT-3.5 across all skills.
  • RLHF: RLHF improves Logical Thinking but impairs performance on the remaining skills in the VICUNA comparison.
  • Fine-tuning steps: Most skills are acquired during the first epoch, while later training affects skills differently; Logical Correctness improves most consistently.
  • Evaluator choice: Using CLAUDE instead of GPT-4 as evaluator raises Logical Thinking scores and lowers User Alignment scores overall.
  • Model scale: Larger models improve skills overall, but open-source models retain a significant GPT-3.5 gap in Logical Thinking and Background Knowledge.
  • Self-revision: Self-feedback and revision improve Logical Robustness, Logical Correctness, Insightfulness, and Completeness without closing proprietary-model gaps in Logical Thinking and Background Knowledge.

D.1 EVALUATION OF LLMS

FLASK evaluates models through fine-grained skill-specific scoring rather than relying only on separate automatic benchmarks or overall preference scores. Its analyses address evaluator bias, response length, metadata structure, and evaluation workflow.

  • Motivation: Conventional evaluation uses separate benchmarks and automatic metrics, but these metrics capture surface features and differ in targeted skills, domains, and difficulties.
  • Evaluator bias: LLM evaluators can exhibit position, verbosity, and style biases, although FLASK eliminates position bias through its evaluation setup.
  • Evaluation process: FLASK’s evaluation process includes data construction, metadata annotation, scoring, and collection of the FLASK-HARD subset.
  • Evaluator bias: Fine-grained skill-level scoring reduces the correlation between response length and Logical Correctness performance, while Completeness remains strongly length-associated.

F METADATA STATISTICS OF EVALUATION SET OF FLASK

The FLASK evaluation set is characterized by distributions over abilities, skills, domains, and difficulty, with Comprehension especially prevalent. Human annotation validates domain and skill metadata while showing moderate agreement on difficulty levels.

  • Dataset composition: Comprehension is the largest skill category, while Harmlessness and Metacognition are the least represented.
  • Difficulty: Formal-education and major-level knowledge difficulty levels account for the largest proportions, while expert-level knowledge accounts for the least.
  • Human evaluation: Human evaluation assigns three labelers to each instance and scores skills from 1 to 5, with N/A for irrelevant skills.
  • Dataset composition: The FLASK-HARD set provides a separate skill distribution for analyzing more challenging evaluation instances.
  • Metadata reliability: Human acceptance rates are 81.32% for domain annotations and 95.22% for skill annotations.
  • Metadata reliability: Difficulty annotation correlates with human labels at Spearman 0.779, Kendall-Tau 0.653, and Pearson 0.774.

G.3 COST AND TIME COMPARISON BETWEEN MODEL-BASED AND HUMAN-BASED EVALUATION

Model-based evaluation is substantially cheaper and faster than human-based evaluation, but the paper recommends using both settings for reliable evaluation because each has limitations.

  • 22 times cheaper and 129 times faster, model-based evaluation outperforms human-based evaluation in cost and time.
  • The comparison is presented in Table 8 as a cost-and-time analysis of model-based versus human-based evaluation.
  • The authors recommend both evaluation settings for reliable evaluation because each has respective limitations.

H ADDITIONAL RESULTS

Additional results describe FLASK’s skill categorization, datasets, scoring rubrics, and analyses across models, domains, skills, and instruction difficulty.

  • Additional analyses: Additional analyses compare model performance across skills, domains, and instruction difficulty using model-based evaluation.
  • Skill categorization: FLASK defines skill categories and provides the same skill definitions to the EVAL LM and human labelers.
  • Source datasets: The evaluation set combines conventional benchmarks with datasets from diverse domains, including financial numerical reasoning and haiku generation.
  • Score rubrics: Separate predefined score criteria are written for each skill and each score from 1 to 5 to provide objective standards during evaluation.
  • Score rubrics: Skill-specific rubrics are used for the whole FLASK set, while FLASK-HARD additionally uses more fine-grained instance-specific rubrics.
Loading 2307.10928v4…