PaperlayerView exact revision on arXiv

Source-linked AI summary

FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, Minjoon Seo

arXiv:2307.10928v4cs.CLcs.AI

TL;DR

LLM evaluation is difficult because instructions can require combinations of abilities, while common evaluations use separate benchmark metrics or overall preference scores. FLASK scores instruction-specific skills and shows stronger human–model evaluation correlation with fine-grained scoring.

  • Problem

    Common LLM evaluations use separate task metrics or overall scores, while instructions can require multiple abilities, complicating comprehensive assessment of alignment.

    Source: §1 · p. 1

  • Method

    FLASK evaluates instructions through instance-wise skill scoring across four primary abilities and 12 fine-grained skills.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Skill-specific FLASK scoring yielded consistently stronger human–model correlation than skill-agnostic evaluation across EVAL LMs, and fine-grained evaluation improved robustness to stylistic changes.

    Source: §4 · p. 6, §7 · p. 9

  • Takeaways & Limitations

    FLASK provides interpretable analysis of LLM capabilities across skills, domains, and difficulty levels.

    Source: §7 · p. 9

    Human evaluation is time-consuming and expensive, so extensive analysis of the full FLASK set relies on automatic model-based evaluation.

    Source: §5 · p. 7

Abstract

from arXiv · show

Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations. We publicly release the evaluation data and code implementation at https://github.com/kaistAI/FLASK.

1 INTRODUCTION

FLASK evaluates instructions by scoring their annotated alignment skills rather than relying on a single overall score or a fixed metric set. The paper reports that fine-grained evaluation improves interpretability and reliability.

Introduction

  • Evaluating alignment is difficult because open-ended instructions combine abilities, required skills vary by instance, and existing overall scores or fixed metrics miss these distinctions.Multiple independent benchmarks are not scalable and offer limited interpretability and reliability, while task-specific fixed metric sets do not suit task-agnostic alignment evaluation.Source: §1 · p. 1
  • FLASK defines 4 primary abilities and 12 fine-grained skills, then evaluates each instance against its annotated skill set.It collects 1,740 instances annotated for skills, domain, and difficulty; evaluators score each skill from 1 to 5 using skill-specific rubrics, with instance-specific rubrics for the 89 hardest instances.Source: §1 · p. 2
  • Human- and model-based evaluations correlate highly; fine-grained evaluation improves interpretability and reliability while increasing correlation and mitigating model-based bias.The paper compares models across skill sets, target domains, and difficulty levels.Source: §1 · p. 2
  • Fine-grained model comparisons can guide developers’ alignment improvements and help practitioners choose models for specific situations.Source: §1 · p. 2

2 RELATED WORKS

Prior work evaluates LLMs through multiple benchmarks, metrics, and fine-grained task settings, while alignment research spans diverse training techniques; FLASK addresses the need for instance-wise evaluation and broader comparison of user-aligned models.

Holistic Evaluation of LLMs

Alignment of LLMs

  • Alignment research explores instruction tuning and RLHF techniques, but comprehensive comparisons across user-aligned models trained with different techniques remain insufficiently studied.Instruction-tuning approaches include scaling, distillation, instruction generation, model-generated-response augmentation, multilingual tuning, and in-context learning; RLHF work includes synthetic feedback and reinforcement learning during pretraining.Source: §2 · p. 3

3 FLASK: FINE-GRAINED LANGUAGE MODEL EVALUATION PROTOCOL

FLASK evaluates language-model alignment by scoring the skills required for each instruction, using human evaluators or EVAL LM to enable task-agnostic, fine-grained analysis.

3.1 SKILL SET CATEGORIZATION

  • FLASK defines four primary abilities comprising 12 skills for assessing language-model responses to instructions.The abilities are Logical Thinking (logical correctness, robustness, and efficiency), Background Knowledge (factuality and commonsense understanding), Problem Handling (comprehension, insightfulness, completeness, and metacognition), and User Alignment (readability, conciseness, and harmlessness).Source: §3 · p. 3, §3.1 · pp. 3–4

3.2 EVALUATION DATA CONSTRUCTION

  • The evaluation set contains 1,740 English instruction-reference instances from 122 datasets, with skill, domain, and difficulty metadata.Instances are collected, modified where instructions are absent, and filtered to exclude inputs longer than 2,048 tokens; single-task datasets contribute at most 20 instances each. Metadata annotation validation found 95.22% skill acceptance, 81.32% domain acceptance, and a 0.774 Pearson correlation for difficulty, supporting use of EVAL LM for annotation. Each instance is assigned up to three essential skills, one of ten domains, and one of five difficulty levels.Source: §3.2 · p. 4

3.3 EVALUATION PROCESS

  • Evaluators score each instance’s selected skills from 1 to 5 using skill-specific rubrics, then aggregate scores by skill, domain, and difficulty.For model-based evaluation, EVAL LM generates a rationale before assigning each score.Source: §3.3 · pp. 4–5

3.4 FLASK-HARD

  • FLASK-HARD comprises 89 expert-level instances and uses instance-specific skill rubrics based on checklists to assess responses.EVAL LM generates up to five skill-related subquestions per instance; duplicates and unrelated questions are manually removed, and evaluators score fulfillment from 1 to 5.Source: §3.4 · p. 5

4 RELIABILITY OF FLASK

FLASK’s skill-specific evaluation correlates more strongly with human judgments than skill-agnostic evaluation, and finer-grained rubrics improve robustness to verbosity changes. The evaluations also have complementary limitations: human ratings show central-tendency and fatigue issues, while model ratings can favor style and verbosity.

4 RELIABILITY OF FLASK

  • The correlation comparison used 200 sampled instances, 10 human labelers, four evaluated models, and GPT-4 as the default model evaluator.The evaluated models were GPT-3.5, BARD, VICUNA-13B, and ALPACA-13B.Source: §4 · p. 5
  • Human- and model-based evaluations show similar skill-level trends, including ALPACA’s generally weakest performance and gaps for VICUNA and ALPACA in logical thinking and background knowledge.The paper characterizes the settings as complementary rather than perfect substitutes.Source: §4 · p. 5, Figure 2 · p. 6
  • Human ratings exhibit central-tendency bias and fatigue, while model ratings can show style and verbosity bias, sometimes favoring GPT-3.5’s style over BARD’s contrary to human evaluators.The cited discussion associates human central-tendency bias with Goldfarb-Tarrant et al. (2020), fatigue with Casper et al. (2023) and Bowman et al. (2022), and model evaluation biases with Wang et al. (2023b), Dubois et al. (2023), and Zheng et al. (2023).Source: §4 · pp. 5–6
  • Skill-specific FLASK evaluation correlates more strongly with human judgments than skill-agnostic evaluation across EVAL LMs; GPT-4 correlates highest, while ROUGE-L is lowest.Removing the reference answer, rationale generation, or skill score rubric significantly reduces correlation, especially when the reference answer is removed.Source: §4 · p. 6, Table 1 · p. 6
  • For GPT-3.5 responses on FLASK-HARD made more verbose without changing content, score consistency increased with rubric granularity; instance-specific rubrics were most robust but costlier.Because they require annotating subquestions and manual validation, instance-specific rubrics are used in FLASK-HARD only.Source: §4 · pp. 6–7, Figure 3 · p. 6

5 ANALYSIS BASED ON AUTOMATIC EVALUATION OF FLASK

FLASK’s automatic evaluation reveals skill-specific gaps across open and proprietary models, uneven benefits from scaling, and persistent weaknesses on harder instructions—even among proprietary models.

5 ANALYSIS BASED ON AUTOMATIC EVALUATION OF FLASK

  • Whole-set analysis uses automatic model-based evaluation because human evaluation is costly and prior results showed high correlation with human evaluation; Completeness and Conciseness scores are excluded as length-dependent.Source: §5 · p. 7
  • VICUNA and WIZARDLM nearly match GPT-3.5 on Problem Handling and User Alignment, but lag in Logical Thinking and Background Knowledge, especially in Math and Coding.Open-source models also perform poorly across difficulty levels, particularly in Logical Thinking and Background Knowledge.Source: Figure 4 · p. 7, §5 · p. 7
  • Larger T ¨ULU models improve overall, especially on Logical Robustness, Logical Correctness, and Logical Efficiency, while Readability, Harmlessness, and Metacognition improve slowly.Performance nearly saturates for Logical Efficiency and Conciseness after 30B, Insightfulness after 13B, and Metacognition after 7B.Source: §5 · p. 7, Figure 5 · p. 8
  • Scaling is more effective on easier instructions: larger T ¨ULU models narrow the gap with GPT-3.5 on Level 1 lifestyle knowledge, while the gap widens at higher difficulties.Figure 6 compares GPT-3.5 and T ¨ULU models across difficulty levels for Logical Robustness, Logical Correctness, Factuality, and Completeness.Source: Figure 6 · p. 8, §5 · p. 8
  • On the full set, CLAUDE leads Problem Handling and GPT-3.5 leads Logical Thinking and Background Knowledge; on FLASK-HARD, even GPT-4 struggles with Logical Correctness and Factuality.Performance on FLASK-HARD degrades for Logical Thinking and Background Knowledge compared with the full set; instance-specific rubrics produce lower scores overall than skill-specific rubrics.Source: §5 · pp. 8–9, Figure 7 · p. 9

6 APPLICATION OF FLASK

FLASK supports model developers in diagnosing performance and planning improvements, while helping practitioners choose models suited to different use cases. Its dynamic evaluation setting also allows practitioners to assess model suitability on their own annotated test sets.

FLASK for Developers

  • FLASK helps developers diagnose their models, target capability gaps, and improve checkpoints, datasets, and training techniques.Open-source developers can compare with proprietary models, especially on Logical Thinking and Background Knowledge; proprietary developers can target the FLASK-HARD set.Source: §6 · p. 9

FLASK for Practitioners

  • Practitioners can use FLASK to estimate which LLMs suit different situations and potentially route instructions to suitable models.They can annotate their own test sets; the passage gives chit-chat with 7B fine-tuned open-source models and complex reasoning with proprietary APIs as examples.Source: §6 · p. 9

7 CONCLUSION · A LIMITATION AND FUTURE WORK · A.1 LIMITATION OF EVALUATORS

FLASK evaluates LLMs across 12 fine-grained skills, domains, and difficulty levels, enabling interpretable capability analysis and improving evaluation reliability. The paper also identifies limitations in human and model evaluators and calls for broader human evaluation and mitigation work.

7 CONCLUSION

  • FLASK categorizes 12 skills and annotates each instance with necessary skills, target domain, and difficulty level.This supports analysis of model performance by skill, domain, and difficulty.Source: §7 · p. 9
  • Fine-grained evaluation improves human–model evaluation correlation and makes model-based evaluation more robust to stylistic changes.The paper presents FLASK as an initial guideline for comprehensive, reliable evaluation and meaningful insights for developers and practitioners.Source: §7 · p. 9

A.1 LIMITATION OF EVALUATORS

  • Human evaluators face central-tendency bias and annotation fatigue, which may contribute to only moderate inter-labeler agreement.The wide scope of knowledge needed to evaluate each instance contributes to these difficulties.Source: §A.1 · p. 20
  • Document retrieval for fact verification (Min et al., 2023) and highlight hints (Krishna et al., 2023) are proposed as ways to mitigate human-evaluation challenges.The paper leaves mitigation of limitations in both evaluator settings for future work.Source: §A.1 · p. 20
  • Model-based evaluation favors longer responses and writing styles similar to the evaluator model, despite being more time- and cost-efficient.The paper emphasizes that both human- and model-based evaluation are crucial to reliably assess language-model capabilities.Source: §A.1 · p. 20
  • Human evaluation was limited by cost and time; future work could recruit more demographically diverse labelers, evaluate more instances, and include more than four models.The paper evaluated only four models with human-based evaluation.Source: §A.1 · p. 20

A.2 SCOPE OF THE EVALUATION … C.1 INTER-LABELER AGREEMENT BETWEEN SKILLS

FLASK’s evaluation is limited in scope and model coverage, while its agreement analysis finds higher overall agreement across model runs than among human labelers and skill-dependent variation in agreement. The supplementary sections also describe the evaluated models and the evaluation’s limits.

A.2 SCOPE OF THE EVALUATION

  • The evaluation covers English-only, single-turn, language-focused, zero-shot instructions, leaving multilingual, multi-turn, multimodal, and few-shot evaluation for future work.The authors note that metadata annotation is automatic and dynamic, so the evaluation axes could be expanded once instances are collected.Source: §A.2 · p. 20
  • FLASK-HARD contains only 89 instances, making outliers consequential in skill-, domain-, or difficulty-level analyses; instance-specific rubrics are applied only to this subset.Human evaluation of those rubrics across the full FLASK set was not conducted because of time and cost constraints, and future models may require recategorizing abilities and skills.Source: §A.2 · p. 20

B MODEL DETAILS

  • The comparison spans proprietary GPT-3.5, INSTRUCTGPT, BARD, and CLAUDE 1.0 and open models including ALPACA 13B, VICUNA 13B, WIZARDLM 13B, T¨ULU 13B, and LLAMA2 Chat 70B.The open models include fine-tuning on human-curated or machine-generated data, and the proprietary models’ details were hidden from end users.Source: B MODEL DETAILS · p. 20
  • The model comparisons also vary T¨ULU size from 7B to 65B and fine-tuning datasets across SHAREGPT, CODE-ALPACA, ALPACA, FLAN V2, and EVOL-INSTRUCT.Responses were generated with temperature 0.7 and a maximum of 1024 generation sequences; proprietary model versions were those current in May–June 2023.Source: B MODEL DETAILS · pp. 20–21

C ADDITIONAL ANALYSIS

  • Human-labeler agreement was 0.488, versus 0.835 across model-evaluation runs.Agreement was measured with Krippendorff’s alpha; the human setting used three labelers per instance, while the model setting used three nondeterministic runs with GPT-4 fixed as the evaluator.Source: §C.1 · p. 21

C.1 INTER-LABELER AGREEMENT BETWEEN SKILLS

  • Agreement and human-model correlation vary by skill: Logical Correctness and Commonsense Understanding are high, while Readability and Conciseness are not.The authors interpret this pattern as greater subjectivity in User Alignment than in Logical Thinking and Background Knowledge, and suggest disagreement on User Alignment could provide training signals or support personalization.Source: §C.1 · p. 21, Table 2 · p. 21

C.2 ANALYSIS OF DIFFERENT FINETUNING DATA … D BROADER RELATED WORK & BACKGROUND

FLASK analyses show that training-data composition, response quality, RLHF, training duration, and evaluator choice all affect measured skill performance, often unevenly across skills. Additional-model and evaluator-agreement analyses further reveal both gains and persistent skill-specific gaps.

C.2 ANALYSIS OF DIFFERENT FINETUNING DATA

  • Fine-tuning datasets differ in ability, skill, domain, and difficulty composition: EVOL-INSTRUCT emphasizes Logical Thinking and Problem Handling and is hardest, while ALPACA and FLAN V2 are relatively low-difficulty.SHAREGPT, CODE-ALPACA, and EVOL-INSTRUCT contain more Coding and Technology instructions; FLAN V2 and ALPACA contain more Language instructions.Source: §C.2 · p. 21, Figure 8 · p. 22, Figure 9 · p. 23

C.3 EFFECT OF DIFFERENT TRAINING DATA

  • On short-reference-answer instances, performance gaps between fine-tuning datasets shrink, especially for Logical Thinking and User Alignment; the paper attributes FLAN V2’s weaker results to long-form generation, while harder instructions do not significantly change outcomes.The effect of replacing FLAN V2 responses with longer responses is left for future work.Source: §C.2 · p. 21, §C.3 · p. 23, Figure 12 · p. 25

C.4 EFFECT OF TRAINING ON BETTER RESPONSES

  • GPT4-ALPACA 13B outperforms ALPACA 13B across all skills, but remains below GPT-3.5 across all skills, indicating better teacher responses help but do not close the gap.The authors suggest that using a stronger base model than LLAMA 13B could lead to better performance.Source: §C.4 · p. 23, Figure 15 · p. 26

C.5 EFFECT OF RLHF

  • RLHF improves Logical Thinking but impairs the remaining skills; the authors conjecture that lower-quality RLHF data than SHAREGPT may explain the declines.They suggest advanced RLHF techniques might reduce the Logical Thinking gap between open-source and proprietary LLMs.Source: §C.5 · pp. 23–24, Table 3 · p. 26

C.6 FINE-TUNING STEPS VARIATION

  • Most skills are acquired in the first epoch, but later training improves some skills and degrades others, indicating that skill-specific training durations may be needed.Logical Correctness improves most consistently, while Harmlessness, Readability, and Logical Robustness decline after the first epoch.Source: §C.6 · pp. 24–25, Figure 16 · p. 27

C.7 USING CLAUDE AS EVAL LM FOR EVALUATION

  • Using CLAUDE instead of GPT-4 as evaluator raises Logical Thinking scores and lowers User Alignment scores, while widening open-source models’ apparent gains in Logical Thinking and Factuality.Because CLAUDE’s results conflict with human-based evaluation and correlate less with human labels, the paper defaults to GPT-4, while suggesting multiple evaluators with similar abilities.Source: §C.7 · p. 25, Figure 17 · p. 28

C.8 EXPLORING AGREEMENT BETWEEN EVAL LMS

  • Inter-model agreement is 0.471 overall, but skill-level agreement differs from human inter-labeler patterns, with the lowest agreement on Logical Robustness.The authors conjecture that evaluator ability gaps, especially for Logical Robustness and Logical Efficiency, contribute to this difference.Source: §C.8 · p. 26, Table 4 · p. 27

C.9 ADDITIONAL MODELS

  • Larger models improve skills overall, while self-revision selectively improves Logical Robustness, Logical Correctness, Insightfulness, and Completeness without closing Logical Thinking and Background Knowledge gaps.LLAMA2 Chat shows significant gains in Insightfulness and Completeness versus VICUNA 13B, but worse Conciseness; the authors note differing fine-tuning data confounds base-model comparisons. SELFEE uses iterative feedback and revision.Source: §C.9 · pp. 27–28

D.1 EVALUATION OF LLMS · D.2 USING LLMS AS EVALUATORS · E DETAILS FOR METADATA ANNOTATION PROCESS

These appendices describe the evaluation context for FLASK, its approach to evaluator biases, and the rules used to annotate skills, domains, and difficulty.

D.1 EVALUATION OF LLMS

D.2 USING LLMS AS EVALUATORS

  • FLASK's absolute-score setup removes position bias, while skill-level scoring is intended to mitigate verbosity and style biases.LLM evaluators may favor the first option, longer responses, or responses resembling their own style (Zheng et al., 2023).Source: §D.2 · pp. 29–30

E DETAILS FOR METADATA ANNOTATION PROCESS

  • Annotating the top three relevant skills per instance balanced avoiding universal labels such as Readability with retaining skills genuinely required by an instruction.For example, logical problems may not require Readability, whereas writing a LinkedIn profile summary may.Source: E DETAILS FOR METADATA ANNOTATION… · p. 30

D.2 USING LLMS AS EVALUATORS

  • Completeness remained highly correlated with response length, while Logical Correctness showed lower correlation, suggesting fine-grained scoring mitigates verbosity bias.The comparison uses average response length and skill performance across the FLASK evaluation set.Source: §D.2 · p. 30, Figure 22 · p. 31, Table 5 · p. 32

E DETAILS FOR METADATA ANNOTATION PROCESS

  • Metacognition annotation uses GPT-4 capability as reference and targets modality-dependent or future-event questions; skill-description order is shuffled to counter EVAL LM's top-three selection position bias.The passage notes that metacognition depends on the model being evaluated.Source: E DETAILS FOR METADATA ANNOTATION… · pp. 30–31
  • FLASK uses 10 domains and 38 sub-domains, adds Coding as a separate domain, and provides the taxonomy to model evaluators and human labelers.The Coding domain was separated because of its significance for LLMs (Li et al., 2023a; Luo et al., 2023).Source: E DETAILS FOR METADATA ANNOTATION… · pp. 31–32, Table 6 · p. 32
  • Difficulty is defined by the domain knowledge needed, with level descriptions and examples plus separate Math and Coding guidelines for their distinct knowledge requirements.The guidelines include examples for Math (Figure 50), Coding (Figure 51), and other domains (Figure 49).Source: E DETAILS FOR METADATA ANNOTATION… · p. 32

F METADATA STATISTICS OF EVALUATION SET OF FLASK … G.2 RELIABILITY OF AUTOMATIC METADATA ANNOTATION BY GPT-4

The section reports FLASK evaluation-set metadata distributions and describes human evaluation procedures used to assess GPT-4 metadata annotation reliability. It gives skill and difficulty distributions alongside annotation acceptance rates and agreement measures.

F METADATA STATISTICS OF EVALUATION SET OF FLASK

G HUMAN EVALUATION SETTING

  • Human evaluation sampled 200 FLASK instances and recruited 10 KAIST student labelers; each instance was labeled by three people.Each labeler annotated 60 instances and evaluated responses from GPT-3.5, BARD, VICUNA, and ALPACA with model names hidden.Source: §G.1 · p. 33

G.1 HUMAN EVALUATION SETTING DETAILS

  • The human evaluation assessed domain acceptance, skill annotation and per-skill scores, and difficulty levels using a 1-to-5 scale.Skill relevance could be marked N/A when a skill annotated by GPT-4 was unnecessary or irrelevant to answering the instruction; Level 1 denotes simple lifestyle knowledge and Level 5 expert-level knowledge.Source: §G.1 · p. 34

G.2 RELIABILITY OF AUTOMATIC METADATA ANNOTATION BY GPT-4

  • Domain annotation acceptance was 81.32%, skill annotation acceptance was 95.22%, and difficulty-level correlations with GPT-4 were 0.779 Spearman, 0.653 Kendall-Tau, and 0.774 Pearson.Difficulty-level agreement among labelers measured by Krippendorff’s alpha was 0.540, indicating moderate agreement (Hughes, 2021).Source: §G.2 · p. 34

G.3 COST AND TIME COMPARISON BETWEEN MODEL-BASED AND HUMAN-BASED EVALUATION

Model-based evaluation is reported as 22 times cheaper and 129 times faster than human-based evaluation. The authors nevertheless recommend both settings for reliable evaluation because each has limitations.

G.3 COST AND TIME COMPARISON BETWEEN MODEL-BASED AND HUMAN-BASED EVALUATION

  • Model-based evaluation is 22 times cheaper and 129 times faster than human-based evaluation.The comparison is presented in Table 8.Source: §G.3 · pp. 34–35, Table 8 · p. 35
  • Both model-based and human-based evaluation are recommended for reliable evaluation because each setting has limitations.The limitations are discussed in Section 4.Source: §G.3 · p. 35

H ADDITIONAL RESULTS … K LIST OF PROMPTS

The supplementary sections document FLASK’s skill definitions, dataset construction, evaluation rubrics, and prompts, alongside additional model comparisons. They report that GPT-3.5 leads Claude on Math and Coding, while Claude leads on the other domains.

H ADDITIONAL RESULTS

  • GPT-3.5 outperforms Claude on Math and Coding, while Claude outperforms GPT-3.5 on the other domains; additional figures compare models by skill, instruction difficulty, and domain.The comparisons include GPT-3.5, Vicuna, WizardLM, Tülu sizes, and proprietary models.Source: H ADDITIONAL RESULTS · p. 35, Figure 30 · p. 40, Figure 31 · p. 41

I SKILL CATEGORIZATION OF FLASK

SKILL DEFINITION APPLICATION

  • The same skill definitions are provided to both the model-based evaluator and human labelers.This shared wording applies to FLASK’s model-based and human-based evaluation.Source: I SKILL CATEGORIZATION OF FLASK · p. 37

J SOURCE DATASET LIST

  • FLASK’s evaluation set contains 1,700 instances from 120 NLP datasets, including MMLU (Hendrycks et al., 2020), BBH, FinQA, and Haiku Generation.When reference answers are missing, the authors use EVAL LM responses to supply them, reporting that preliminary experiments found EVAL LM references rather than fully relies on those answers.Source: J SOURCE DATASET LIST · p. 37

PRIMARY ABILITY

  • Figure 29 compares GPT-3.5, Vicuna, and WizardLM across skills on FLASK-HARD and across domains on FLASK.The passage also lists Vicuna 13B, WizardLM 13B, and GPT-3.5 as the compared models.Source: Figure 29 · p. 39

K.1 SCORE RUBRIC FOR EACH SKILL

  • FLASK’s skill rubrics define separate score criteria from 1 to 5 for each skill and are intended to provide objective scoring standards.The listed criteria cover skills including logical robustness, factuality, completeness, readability, and harmlessness.Source: §K.1 · pp. 39–40

K LIST OF PROMPTS

Related papers

Loading 2307.10928v4…