Source-linked AI summary
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations
Md Tahmid Rahman Laskar, Sawsan Alqahtani, M Saiful Bari, Mizanur Rahman, Mohammad Abdullah Matin Khan, Haidar Khan, Israt Jahan, Amran Bhuiyan, Chee Wei Tan, Md Rizwan Parvez, Enamul Hoque, Shafiq Joty, Jimmy Huang
TL;DR
Reliable LLM evaluation is difficult because diverse evaluation setups and incomplete practices produce inconsistent findings. This paper systematically reviews challenges across the evaluation workflow and develops recommendations for more reproducible, reliable, and robust assessments. It concludes that standardized evaluation protocols are needed, while its scope excludes the training phase and does not solve secure evaluation of closed-source models.
Problem
LLM evaluation is complex and often inconsistently documented across benchmarks, prompts, models, parsing, and methodologies, limiting reproducibility and reliable interpretation.
Method
The paper systematically reviews challenges and limitations across the LLM evaluation workflow and proposes recommendations for evaluation setup, methodology, and practice.
Results
The review identifies inconsistencies and complexities throughout the evaluation pipeline that affect reproducibility, reliability, and robustness.
Takeaways & Limitations
The paper recommends standardized and systematic evaluation protocols to improve the consistency and fairness of LLM evaluations for reliable real-world use.
Takeaways & Limitations
The study covers only the evaluation phase, excluding training-phase challenges, and does not resolve secure assessment of closed-source models without access to weights.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have recently gained significant attention due to their remarkable capabilities in performing diverse tasks across various domains. However, a thorough evaluation of these models is crucial before deploying them in real-world applications to ensure they produce reliable performance. Despite the well-established importance of evaluating LLMs in the community, the complexity of the evaluation process has led to varied evaluation setups, causing inconsistencies in findings and interpretations. To address this, we systematically review the primary challenges and limitations causing these inconsistencies and unreliable evaluations in various steps of LLM evaluation. Based on our critical review, we present our perspectives and recommendations to ensure LLM evaluations are reproducible, reliable, and robust.
1 Introduction
Reliable LLM evaluation is essential for academic and business use, but its complexity and fragmented treatment create inconsistent findings. This paper organizes the evaluation workflow, examines its challenges, and proposes recommendations for reproducible, reliable, and robust evaluation.
- LLM evaluation is essential before real-world use, including both academic benchmarks and business applications.
- Existing reviews often emphasize benchmarks, datasets, and criteria while overlooking broader issues such as robustness and reproducibility.
- Prior studies address issues such as contamination or reproducibility separately, leaving reliability and robustness insufficiently integrated.
- The paper constructs a practical evaluation-pipeline workflow and examines each step for inconsistencies and decision-making complexities.
- Based on this review, the paper provides guidelines intended to improve the reproducibility, reliability, and robustness of LLM evaluations.
2 Overview of LLM Evaluation Process
The LLM evaluation process consists of selecting benchmarks and models, generating responses through prompt and decoding choices, parsing outputs, and applying evaluation approaches. These stages support automatic, human, and LLM-based assessment, while resource availability and comparison fairness remain relevant concerns.
- Evaluation Setup: Evaluation begins with benchmark selection, including general-capability, specialized, and diverse benchmark categories.
- Evaluation Setup: Model selection is part of the evaluation setup because it affects risks such as data contamination and unfair comparisons.
- Response Generation: Prompt design and decoding parameters, including zero-shot or few-shot prompting and temperature, determine response-generation settings.
- Evaluation Methodology: Parsing scripts extract target labels from verbose model outputs before metrics are applied, helping align responses with evaluation criteria.
- Evaluation Methodology: Evaluation approaches include automatic metrics, human assessment of qualities such as clarity and factuality, and LLMs serving as evaluators.
3 Challenges in Evaluating LLMs
LLM evaluation is challenged by reproducibility, reliability, and robustness problems spanning data, prompts, decoding, parsing, metrics, and benchmark coverage. These issues can make evaluations inconsistent, misleading, or poorly aligned with human judgments.
- 3.1 Reproducibility: Evaluation reproducibility is limited by missing documentation and unreleased resources across datasets, prompts, models, decoding, parsing, and methodology.The paper identifies reproducibility as the ability to replicate results under the same conditions and notes that comprehensive documentation is often absent.
- Data Integrity: Incorrect or outdated benchmark labels, continued use of superseded datasets, and untested contamination can undermine model comparisons and zero-shot conclusions.Examples include issues in MMLU and HumanEval, outdated open-domain question-answering references, and contamination concerns in commonly used datasets.
- Evaluation Setup: Prompt hacking, undisclosed decoding choices, and single-prompt evaluation create fairness and transparency concerns because small setup changes can alter model performance.The review highlights manipulation through prompts, underexplored decoding-parameter selection, and the need to compare results across multiple prompts.
- Evaluation Methodology: Parsing scripts can differ from human judgments by more than 10%, while automatic metrics may disagree with human preferences or fail on some tasks.Hybrid evaluation can change results in open-domain question answering and SQuAD-V2, and ROUGE may rank LLM summaries below responses preferred by humans.
- Generalized Evaluation: Performance findings do not generalize reliably across datasets, tasks, prompts, or context settings, making diverse benchmark coverage necessary.Qwen2 outperforms LLaMA-3 on most reported datasets but falls short on GPQA and MBPP; coding results also differ between HumanEval and MBPP.
- Evaluation Methodology: Automated rankings can be unstable across implementations and may disagree with human-oriented rankings, even when correlations with human evaluations are reported.The review reports disagreements between MMLU and LMSys Chatbot Arena rankings and variation among MMLU implementations.
4 Recommendations and Best Practices
The paper recommends a structured evaluation framework that accounts for randomness, computational demands, and insufficient documentation. Its guidance covers benchmark selection, reproducible methodology, and practical implementation.
- A comprehensive framework is difficult to implement because LLM evaluation involves complex, multi-stage decisions and varied practical setups.
- Current evaluation practices are shaped by generative randomness, substantial computational demands, and inadequate documentation across stages.
- Evaluation Setup: Align benchmarks with intended capabilities, varied tasks, language complexity, and unseen data, while documenting subsets and model versions.The paper also recommends periodically refreshing evaluations to maintain relevant standards.
- Recommendations and implementation suggestions are consolidated in a dedicated table.
- Evaluation Methodology: Publish parsing scripts and evaluation outputs, then sanity-check parsers with varied response types to improve reliability and reproducibility.Runtime latency measurement is also recommended for assessing real-world applicability.
5 Conclusions and Future Work
The paper surveys inconsistencies and complexities across the LLM evaluation pipeline and proposes recommendations for more consistent and fair evaluation. It calls for standardized protocols covering datasets, metrics, and documentation.
- The review identifies pipeline-wide inconsistencies affecting evaluation reproducibility, reliability, and robustness.
- Future work should develop standardized protocols, benchmark datasets, evaluation metrics, and documentation of evaluation settings.
Limitations
The review is limited to evaluation-phase challenges and mainly monolingual, text-based settings. It also does not address preventing closed-source models from accessing online benchmarks.
- The review excludes challenges arising during LLM training because it focuses on the evaluation phase.
- The study does not examine how to prevent closed-source LLMs from accessing online benchmarks without model-weight access.
- The analysis mainly covers monolingual, especially English, data and leaves multilingual and resource-constrained evaluation for future work.
- Multimodal benchmark evaluation is left for future work because the paper focuses mainly on text-based NLP tasks.
Ethics Statement
The paper reports no ethical concerns because it reviews existing LLM evaluation challenges and offers recommendations rather than introducing a new empirical system.
- The review presents an opinion piece and recommendations for reliable, robust, and reproducible LLM evaluation, rather than a new evaluated system.
A.1 Benchmarking Datasets
The survey organizes LLM evaluation datasets into general-capability, specialized, and diverse benchmarks spanning broad knowledge, reasoning, task-specific abilities, safety, and robustness.
- General Capability Benchmarks: MMLU evaluates overall knowledge and reasoning, while commonsense benchmarks include HellaSwag, PIQA, SIQA, WinoGrande, Open-BookQA, and ARC.
- Specialized Benchmarks: Specialized benchmarks assess capabilities such as conversation, instruction following, biomedicine, hallucination detection, code generation, software engineering, safety, and red teaming.Examples include MT-Bench, AlpacaEval, Open Medical-LLM Leaderboard, HHEM, BigCodeBench, LiveCodeBench, SWE-bench, and safety-oriented benchmarks.
- Benchmark Selection: Benchmark datasets are categorized as general capability, specialized, or other diverse benchmarks for evaluating different aspects of LLM performance.General benchmarks include widely used datasets, while specialized and diverse benchmarks target narrower capabilities or broader evaluation scenarios.
- Other Diverse Benchmarks: Diverse benchmarks such as HELM, PromptBench, OpenLLM, and MixEval combine broad evaluation scenarios and often incorporate existing datasets.
- Benchmark Selection: Benchmark coverage spans general NLP tasks, biomedicine, finance, languages, social science, coding, and information retrieval.
A.2 Prominent LLMs
The survey discusses prominent LLMs because hundreds have been released, focusing on models that recently achieved high rankings on public leaderboards and distinguishing closed- from open-source systems.
- A.2 Prominent LLMs: The survey covers selected prominent LLMs rather than all released models, focusing on systems with top rankings in public leaderboards.It notes that hundreds of LLMs have been released in recent years.
- A.2 Prominent LLMs: LLMs are categorized as closed-source when accessed through APIs or web interfaces and open-source when pretrained weights are available.
A.2.1 Closed Source LLMs
The survey reviews closed-source models by developer, covering Anthropic, OpenAI, and Google systems with capabilities spanning text, multimodal processing, reasoning, safety, and efficiency.
- OpenAI Models: GPT-4 Turbo extends GPT-4 with an April 2023 knowledge cutoff, a 128,000-token context window, and improvements in efficiency and output consistency.
- OpenAI Models: GPT-4o processes text and images, offering double GPT-4 Turbo’s text-generation speed and 50% lower operational costs.
- Google Models: Google’s PaLM 2 and Gemini target advanced general and multimodal processing, with Gemini handling text, images, audio, and video across data-center and mobile platforms.
- Anthropic Models: Anthropic’s Claude family progresses from the original Claude through Claude Instant, Claude 2.0, and Claude 3 models with improvements in speed, reasoning, and multimodal processing.
A.2.2 Open Source LLMs
The survey reviews open-source LLMs from major organizations, including Microsoft, TII, Cohere, Google, Meta, Mistral, and Alibaba, emphasizing varied model sizes, capabilities, and deployment settings.
- Meta Models: Meta’s Llama series progresses from Llama to Llama 2 and Llama 3, with later versions offering larger sizes, improved training, longer context, and enhanced multimodal functions.
- Mistral Models: Mistral develops open-source models including Mistral 7B, Mixtral 8x7B, and Mixtral 8x22B, known for high performance and mixture-of-experts architectures.
- Alibaba Models: Alibaba’s Qwen models range from 0.5 billion to 110 billion parameters and support long contexts with performance across Chinese and English reasoning, coding, and mathematics tasks.
- Microsoft Models: Microsoft’s Phi-3 family ranges from 3.8 billion to 14 billion parameters and targets high performance with lower computational requirements.The models are positioned for simpler tasks, local devices, and resource-limited environments.
- TII and Cohere Models: Falcon models include parameter sizes from 1.3B to 180B, while Cohere’s Command models support conversational tasks, code generation, and retrieval-augmented generation.
- Google Gemma Models: Google’s Gemma family provides lightweight open-source models with multimodal capabilities.
A.3 Prompting Techniques
Prompting techniques span structured reasoning, adaptive guidance, multimodal inputs, recursive prompt refinement, demonstrations, decomposition, role control, and chained subtasks. Evaluation design also depends on decoding settings and faces limitations in human and LLM-based assessment.
- A.3 Prompting Techniques: Zero-shot prompting supplies instructions without examples, while few-shot prompting adds a small set of demonstrations to help models adapt to a task.The distinction concerns whether the model receives task-specific examples within the prompt.
- A.3 Prompting Techniques: Chain-of-Thought prompting elicits intermediate reasoning steps, and role- or style-based prompting controls the persona, tone, or formality of generated responses.Chain-of-Thought can be combined with few-shot prompting for complex tasks.
- A.3 Prompting Techniques: Prompt chaining divides a complex task into sequential subtasks, using each response as input for the next prompt to improve control and refinement.The sequence makes intermediate outputs available for subsequent processing.
- A.3 Prompting Techniques: Tree of Thoughts explores multiple solution paths through breadth-first or depth-first search, while decomposition solves complex problems through sequential sub-problems.These methods structure multi-step reasoning by evaluating alternatives or integrating simpler component solutions.
- A.3 Prompting Techniques: Directional Stimulus Prompting uses a trainable secondary model and reinforcement learning to generate hints that guide a fixed LLM toward targeted outputs.The approach adapts prompts according to their observed performance.
- A.3 Prompting Techniques: Multimodal prompting combines text with images, audio, or video, whereas meta-prompting uses AI-generated prompts to refine prompt engineering.Both techniques extend prompting beyond a single manually written text instruction.
- A.4 Decoding Parameters: Temperature controls output randomness, while beam size specifies the number of beams considered during beam search.These decoding parameters influence response generation and should be configured as part of evaluation setup.
- A.6 Human and LLM Evaluation: Human rankings can be non-transitive and vulnerable to ownership hacking, while LLM judges vary across tasks and require bias and hallucination detection.BTL ratings are described as transitive but not correlated with empirical win rates, and closed-source judges incur costs.