PaperlayerView exact revision on arXiv

Source-linked AI summary

xFinder: Large Language Models as Automated Evaluators for Reliable Evaluation

Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, Ding Chen

arXiv:2405.11874v3cs.CL

TL;DR

LLM evaluation faces fairness and reliability concerns from contamination, prompt sensitivity, and unreliable answer extraction. This paper introduces xFinder and the KAF dataset for key-answer extraction and matching, achieving 97.61% judgment accuracy against existing frameworks and judge models.

  • Problem

    LLM evaluation is challenged by test-set contamination, prompt sensitivity, and unreliable extraction, which threaten fairness and reliability.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Method

    xFinder uses the KAF dataset and a specialized evaluator to extract and match key answers in deterministic LLM tasks.

    Source: §1 · p. 3, §3 · p. 4, §4 · p. 5

  • Results

    97.61% judgment accuracy was achieved by xFinder, outperforming PandaLM, JudgeLM, and GPT-4 on the generalization set.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Improving key-answer extraction with xFinder can enhance evaluation reliability and efficiency relative to RegEx-based methods and judge models.

    Source: §1 · p. 3, §6 · p. 10

    RegEx extraction accuracy varies with prompt restrictions, so comparisons differ between restricted test responses and unrestricted generalization responses.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. As evaluation frameworks commonly use Regular Expression (RegEx) for answer extraction, models may adjust their responses to fit formats easily handled by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. Furthermore, recent studies proposing fine-tuned LLMs as judge models for automated evaluation face challenges in terms of generalization ability and fairness. This paper comprehensively analyzes the entire LLM evaluation chain and demonstrates that optimizing the key answer extraction module improves extraction accuracy and enhances evaluation reliability. Our findings suggest that improving the key answer extraction module can lead to higher judgment accuracy and improved evaluation efficiency compared to the judge models. To address these issues, we propose xFinder, a novel evaluator for answer extraction and matching in LLM evaluation. As part of this process, we create a specialized dataset, the \textbf{K}ey \textbf{A}nswer \textbf{F}inder (KAF) dataset, to ensure effective model training and evaluation. Generalization tests and real-world evaluations show that the smallest xFinder model, with only 500 million parameters, achieves an average extraction accuracy of 93.42\%. In contrast, RegEx accuracy in the best evaluation framework is 74.38\%. The final judgment accuracy of xFinder reaches 97.61\%, outperforming existing evaluation frameworks and judge models.

1 INTRODUCTION

LLM evaluation faces fairness and reliability problems across data transformation, prompting, answer extraction, and judging. The paper proposes xFinder with the KAF dataset to improve answer extraction and matching, achieving more accurate and efficient evaluation.

Problems

  • Current evaluation frameworks can produce unreliable outcomes because multiple pipeline stages affect fairness and reliability.The paper identifies concerns involving benchmark leakage, data transformation, prompting, answer extraction, and automated judging.Source: §1 · pp. 1–3

Key Answer Extraction

  • RegEx extraction fails when responses are irrelevant or non-standard, producing false judgments even when an answer addresses the question.The extraction problem is illustrated for alphabet, short-text, categorical-label, and math-option tasks.Source: §1 · p. 2, Figure 2 · p. 2

Proposed Method

  • xFinder is an evaluator for answer extraction and matching that replaces RegEx-based evaluation with a more accurate and efficient approach.Its core process combines a specialized evaluator with the KAF dataset for training and testing.Source: §1 · p. 3

Results

  • 97.61% judgment accuracy enables xFinder to outperform PandaLM, JudgeLM, and GPT-4 on the generalization set.The reported baselines are 51.9%, 78.13% for JudgeLM-33B, and 84.2% for GPT-4.Source: §1 · p. 3

Contributions

  • The authors review LLM evaluation processes and identify critical factors that can lead to unreliable evaluation results.The review covers the evaluation chain rather than only the final scoring stage.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work uses unified frameworks with RegEx extraction or fine-tuned judge models, but both approaches have important limitations. RegEx can misextract objective answers, while judge models generalize poorly across models and benchmarks.

Unified Evaluation Frameworks

  • Unified frameworks transform objective-question datasets and use RegEx extraction or first-token probabilities, but each method can misrepresent intended answers.Text-response evaluation is vulnerable to extraction errors, while first-token evaluation overlooks the reasoning process.Source: §2 · p. 3

Judge Models

  • Fine-tuned judge models perform pairwise selection or pointwise grading, yet they show poor generalization across LLMs and benchmarks.These methods aim to automate evaluation by aligning model judgments with response quality or scores.Source: §2 · p. 4

Reliable Evaluation

  • Reliable evaluation seeks to assess the authenticity of LLM-generated content and risks to information systems, but systematic improvement efforts remain limited.The paper presents its study as the first systematic effort in this area to the authors’ knowledge.Source: §2 · p. 4

3 PROBLEM DEFINITION

The paper defines key answer extraction as identifying a final-answer substring in an LLM response and mapping it to the best corresponding option or a no-valid-answer label.

  • Key answer extraction aligns the answer extracted from an LLM output with the human-expected key answer, targeting deterministic tasks such as multiple-choice questions and mathematics problems.Source: §3 · p. 4
  • Each evaluation item contains a question, a set of options, and the correct answer, and the LLM produces an output response for that item.Source: §3 · p. 4
  • A key answer sentence is a unique substring in the response that directly provides, wraps, or converts the final answer.Source: §3 · p. 4
  • When multiple substrings qualify, the method uses a Chain of Thought process to select the unique substring appearing after reasoning; without such a process, no substring is selected.Source: §3 · pp. 4–5
  • If no key answer sentence exists, the extracted key answer is [No valid answer]; otherwise, it is the option in C that best corresponds to the sentence.Source: §3 · p. 5

4 METHODOLOGY

The methodology builds and trains xFinder using the KAF dataset, diverse LLM responses, human-checked answer labels, and augmentation designed to improve generalization.

4 METHODOLOGY

  • The KAF dataset contains 26,900 training, 4,961 test, and 4,482 generalization samples for fine-tuning, testing, and performance assessment.Source: §4 · p. 5
  • xFinder replaces RegEx in the evaluation pipeline, with Section 5.3 experiments assessing the resulting approach.Source: Figure 3 · p. 5

4.1 LLM RESPONSE GENERATION

  • The response-generation stage covers 19 evaluation datasets across alphabet-option, short-text-option, categorical-label, and math-option tasks.Source: §4.1 · pp. 5–6

4.3 TRAINING XFINDER

  • Restricted prompts standardize responses by requiring the final answer to end with “The answer is <answer>.”Source: §4.1 · p. 6, §4.3 · p. 6

4.1 LLM RESPONSE GENERATION

  • Training and test data use responses from 10 LLMs across 8 tasks, while generalization data use 8 LLMs on 11 entirely new tasks.Source: §4.1 · p. 6

4.2 AUTO LABELLING AND HUMAN RECHECK

  • The dataset uses GPT-4 semiautomated annotations for training, with discrepant items and mathematics questions manually re-annotated, while test and generalization sets receive two manual labeling rounds.Source: §4.2 · p. 6

4.3 TRAINING XFINDER

  • xFinder takes the question, response, key-answer type, and answer range as input and outputs the extracted key answer.Source: §4.3 · p. 6
  • Data augmentation alters alphabet-option choices and diversifies prompt-wrapped answer forms to improve xFinder’s generalization capability.Source: §4.3 · pp. 6–7

5 EXPERIMENTS

The experiments evaluate xFinder for extraction, judgment, efficiency, and practical LLM ranking, comparing it with RegEx-based frameworks and judge models. Across these settings, xFinder shows stronger extraction reliability, judgment accuracy, and evaluation efficiency, while existing frameworks produce inconsistent rankings.

5.1 EXTRACTION ACCURACY: XFINDER VS. REGEX

  • Extraction accuracy varied by no more than 0.5% across xFinder models despite a 16-fold difference in parameter size.Source: §5.1 · p. 7
  • On the KAF generalization set for unseen tasks, xFinder maintained high extraction accuracy, whereas mainstream RegEx methods performed poorly, especially on mathematical problems.Source: §5.1 · pp. 7–8

5.2 JUDGMENT ACCURACY: XFINDER VS. JUDGE MODELS

  • 88.42% was GPT-4’s highest judgment accuracy, compared with higher accuracy from xFinder on the KAF generalization set.PandaLM reached 51.9% and JudgeLM-33B reached 78.13% in the reported comparison.Source: §5.2 · p. 8, Table 3 · p. 8
  • 2.43% was xFinder-llama38it’s judgment-to-extraction accuracy gap, versus 14.32% for the strongest baseline gap reported for OpenCompass.The gap is used to reflect judgment discrepancies associated with extraction errors.Source: §5.2 · p. 8, Table 4 · p. 9
  • $0.02 was the reported cost of xFinder-qwen1505 under the experiment’s setup, equal to 0.4% of GPT-4’s expenses.The comparison used 200 sampled instances per question type and reports lower costs alongside higher accuracy and greater flexibility.Source: §5.2 · pp. 8–9, Table 6 · p. 9

5.3 EVALUATION IN REAL-WORLD SCENARIOS

  • InternLM2-1.8B-Chat ranked first with UltraEval but fifth with OpenCompass on CommonsenseQA, showing substantial framework-dependent ranking differences.The real-world evaluation covered 10 LLMs across 14 mainstream tasks; xFinder checkpoints produced relatively consistent rankings across base models.Source: §5.3 · pp. 9–10
  • Alphabet-option and short-text formats produced significantly different CommonsenseQA rankings, suggesting that alphabet options may be unreliable.The study therefore advocates reducing reliance on alphabet-option formatting in evaluation frameworks.Source: §5.3 · pp. 9–10

6 CONCLUSION

The paper analyzes reliability problems in LLM evaluation, focusing on answer extraction and matching, and introduces xFinder to address them. Experiments report more accurate and efficient extraction than RegEx-based methods, improving evaluation reliability within the studied scope.

6 CONCLUSION

  • The paper focuses on unreliable LLM evaluation outcomes caused by issues in answer extraction and matching.Source: §6 · p. 10
  • xFinder is designed as an evaluator for accurate key-answer extraction and matching in LLM responses.Source: §6 · p. 10
  • Extensive experiments found that xFinder extracted key answers more accurately and efficiently than existing RegEx-based methods.The conclusion frames this work as a first step toward constructing a reliable evaluation framework.Source: §6 · p. 10

A.1 DATASET DETAILS

The appendix describes the datasets, selected language models, and KAF dataset structure used in the experiments. Dataset samples were standardized and categorized by answer format, while KAF separates training, test, and generalization data.

A.1 DATASET DETAILS

  • The experiments sampled 2,000 training points and 1,000 test or development points after preprocessing datasets into a unified format.Questions were categorized as alphabet, short text, categorical label, or math type according to key-answer format.Source: §A.1 · p. 15, Table 7 · p. 15

A.2 LARGE LANGUAGE MODELS

  • The study selected a set of open-source LLMs for experimentation, with the models listed in Table 8.Table 8 sorts the listed chat or instruct models by release date.Source: §A.2 · p. 15, Table 8 · p. 15

B.1 DATASET STRUCTURE

  • The KAF dataset contains training, test, and generalization segments serving distinct xFinder development and evaluation purposes.Source: §B.1 · p. 16

B.1.1 TRAINING SET

The KAF data pipeline defines training, test, and generalization splits for key-answer extraction, with human annotation rules covering multiple answer types. xFinder is fine-tuned with standardized settings, while extraction and judgment accuracy are distinguished analytically.

B.1.1 TRAINING SET

B.1.2 TEST SET

B.2 HUMAN ANNOTATION DETAILS

  • Five NLP-trained annotators labeled each item independently, with engineering review and adjudication for disagreements.Source: §B.2 · p. 19

B.2.1 HUMAN ANNOTATION GUIDELINES

Your Task.

  • Non-mathematical outputs must match a candidate answer, while mathematical outputs preserve the final answer’s original format; invalid outputs receive [No valid answer].Source: Your Task · p. 19, Annotation Details · p. 20

Annotation Details.

  • Option, short-text, categorical-label, and mathematical annotations apply type-specific rules, including exact option letters, candidate text, explicit labels, and final calculated answers.Source: Annotation Details · pp. 19–20

B.3 EXAMPLES FROM THE KAF DATASET \b {

  • The KAF examples show xFinder reproducing gold labels for alphabet, short-text, categorical-label, and math responses.Source: §B.3 · p. 21

C FINE-TUNING HYPERPARAMETERS

D.2 EXTRACTION ACCURACY RESULTS

The extraction experiments evaluate 19 KAF-fine-tuned LLMs on test and generalization sets, then compare extraction-based evaluation methods across alphabet, categorical-label, and mathematical tasks.

D.2 EXTRACTION ACCURACY RESULTS

D.3.1 ALPHABET & SHORT TEXT OPTION TASKS

D.3.2 CATEGORICAL LABEL OPTION TASKS

D.3.3 MATH OPTION TASKS

E.1 ALPHABET & SHORT TEXT OPTION TASKS

The appendix provides bump charts for evaluation-task families, organizing comparisons across alphabet or short-text options, categorical labels, and mathematical options.

E.1 ALPHABET & SHORT TEXT OPTION TASKS

E.2 CATEGORICAL LABEL OPTION TASKS

E.3 MATH OPTION TASKS

F PROMPTS

The compact appendix documents prompts for generating LLM responses, automatically labeling data, evaluating responses with GPT-4 judges, and extracting key answers with xFinder.

F.1 PROMPTS FOR LLM RESPONSES GENERATION & REAL-WORLD SCENARIOS EVALUATION

F.2 PROMPTS FOR AUTO LABELING USING GPT-4

F.3 PROMPTS FOR JUDGE MODELS USING GPT-4

F.4 PROMPTS FOR EXTRACTING KEY ANSWERS USING XFINDER

  • The appendix includes a dedicated prompt for extracting key answers with xFinder.Source: Figure 32 · p. 43

Related papers

Loading 2405.11874v3…