A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review

Motivation

• Human evaluation remains necessary because automated metrics do not capture the clinical utility, accuracy, safety, and ethics required for healthcare deployment, while LLM-based evaluation is also problematic 9,12.Human evaluators are therefore expected to remain the near-term gold standard for healthcare LLM applications.Source: §1 · p. 2

Research gap

• Prior reviews examined selected healthcare LLM evaluations but did not systematically summarize human-evaluation dimensions, metrics, or reporting practices, and existing guidelines do not specifically address human evaluation of LLMs 13,14,15,17,18,19,20.The absence of healthcare-specific best practices risks inconsistent and unreliable assessments that may compromise patient safety and care quality.Source: §1 · pp. 2–3

Objectives

• The review analyzes human evaluations of healthcare LLMs across diverse medical domains, tasks, and specialties, including variation in evaluation dimensions and approaches.It aims to synthesize reliable, valid, and ethical best practices and provide actionable guidance for standardized evaluation frameworks.Source: §1 · p. 3

Contribution

• The study develops a framework for methodically assessing healthcare LLM safety, reliability, and effectiveness while establishing guidelines for consistent, high-quality human evaluation.The framework is intended to support further research in the underexplored intersection of generative AI and medicine.Source: §1 · p. 3

2.1 Data Sources and Search Strategies

• The review followed PRISMA guidelines and searched publications from January 1, 2018, to February 22, 2024, covering the rapid evolution of healthcare LLMs.The period spans models from GPT-1 through LLaMA-2 and GPT-4.Source: §2.1 · p. 4, Figure 1 · p. 4
• Searches targeted peer-reviewed English journal articles and conference proceedings in PubMed using terms for generative LLMs, human evaluation, and healthcare.Studies were selected for relevance to healthcare applications and explicit discussion of clinical evaluation methodologies.Source: §2.1 · pp. 4–5

2.2 Article Selection

• After staged filtering, 1191 keyword results were reduced to 911, 795, and 688 studies, with exclusions based on language, publication period, and publication type.Commentaries, summaries, and non-peer-reviewed preprints were excluded because they lacked the required experimental focus.Source: §2.2 · p. 5
• Two-stage screening excluded non-generative models, multimodal applications, and inaccessible articles, producing a final review set of 142 studies after title/abstract and full-text assessment.Full-text screening emphasized methodological detail and applicability to healthcare human evaluation.Source: §2.2 · p. 5

• The review examines diverse methodologies and questionnaires for human evaluation of LLMs in healthcare, identifying current practices, challenges, and areas for future research.Source: §3 · p. 5

Healthcare Applications of LLMs

• 31.9% of categorized tasks involved clinical decision support, followed by medical education and examinations at 24.8%, patient education at 19.6%, and patient-provider question answering at 15%.Administrative tasks and mental health support each represented less than 11.8%.Source: §3.1 · p. 6, Figure 2 · p. 6
• Clinical decision-support studies evaluated LLMs for differential diagnosis, management, treatment suggestions, and bloodstream-infection planning, with some findings indicating plausible diagnoses but remaining adjunctive potential.Lechien et al.23 found ChatGPT’s primary and differential diagnoses plausible in 90% of 40 otolaryngology cases; Milliard et al. compared generated plans with infectious-disease consultants’ plans. Seth et al.24 reported validated medical information for nonmedical individuals with carpal tunnel syndrome.Source: §3.1 · pp. 6–7
• Medical education and examination applications comprised 24.8% of tasks, with studies evaluating LLM performance on licensing examinations and higher-order medical biochemistry problems.Ghosh et al. and Wu et al.27 examined these educational uses, while Ghosh et al.28 used statistical analysis to assess higher-order problem solving.Source: §3.1 · p. 7
• Patient education accounted for 19.6% of tasks, including vaccination answers, insomnia inquiries, and chatbot responses that outperformed medical students in direct and scenario-based question answering.Baglivo et al.31 conducted the feasibility study, and Alapati et al.6 examined clinically accurate responses to insomnia-related questions.Source: §3.1 · p. 7
• Patient-provider question answering represented 15% of tasks and examined orthopedic information, informed-consent replacement, and accurate, empathetic responses to online patient questions.Hatia et al.32 proposed ChatGPT as a replacement for informed consent, while Ayers et al.3 used qualitative and quantitative methods to assess online-forum responses.Source: §3.1 · p. 7
• In translational and evidence-oriented research, LLMs supported medical evidence compilation and diagnostic proposals but showed shortcomings in expert-level performance, depth, accuracy, or reliability in specialized settings.Peng et al.34, Xie et al.35, Tang et al.36, Moramarco et al.37, Bernstein et al.38, and Hirosawa et al.39 evaluated cancer, surgical research, evidence compilation, clinical records, diagnostic suggestions, and differential-diagnosis accuracy using statistical comparisons.Source: §3.1 · pp. 7–8

• Radiology led the literature with 12 studies, followed by urology with 9 and general surgery with 8, while several specialties had 7 studies each.Plastic surgery, otolaryngology, ophthalmology, and orthopedic surgery each had 7 studies; psychiatry had 6, and other specialties had fewer than 5.Source: Figure 3 · p. 8, §3.2 · p. 8
• Radiology evaluations examined report quality, accuracy, clinical utility, and interpretability, while urology studies often measured patient satisfaction, feedback, and usability. [41]The specialty classification followed the 24 certifying boards of the American Board of Medical Specialties. [40]Source: §3.2 · pp. 8–9
• General surgery evaluations assessed pre- and postoperative applications and surgical training through realism, educational value, skill transfer, task time, errors, and skill scores.Plastic surgery and otolaryngology evaluations often focused on patient satisfaction and aesthetic outcomes in cosmetic or reconstructive planning.Source: §3.2 · p. 9
• Emergency medicine studies evaluated time-critical decision-making and triage using accuracy, timeliness, and resource utilization, while internal medicine used satisfaction, clinical outcomes, and diagnostic accuracy measures.Emergency evaluations may use simulation-based studies, with methods depending on the specific internal-medicine application.Source: §3.2 · p. 9

3.3 Evaluation Design

• Healthcare LLM evaluation requires a comprehensive, multifaceted approach because medical specialties and clinical tasks vary substantially.Reviewed studies therefore used diverse methodologies that often blended quantitative and qualitative measures.Source: §3.3 · p. 9

QUEST - Five Principles of Evaluation Dimensions

• QUEST groups 17 healthcare LLM evaluation dimensions into Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence.The framework provides definitions, related concepts, and evaluation strategies for each dimension.Source: QUEST - Five Principles of Evaluation… · pp. 9–10

3.3.1 Evaluation Principles and Dimensions

• Quality of Information assesses accuracy, relevance, currency, comprehensiveness, consistency, agreement, and usefulness, while Understanding and Reasoning assesses prompt interpretation and logical processing.These dimensions address both the quality of information supplied and the model’s ability to understand and reason through a query.Source: QUEST - Five Principles of Evaluation… · pp. 10, 12

QUEST - Five Principles of Evaluation Dimensions

• Expression Style and Persona evaluates clarity and empathy, including whether responses are understandable and reflect the user’s emotions or tone.Empathy was evaluated in studies including 46,66–68.Source: QUEST - Five Principles of Evaluation… · pp. 10, 12
• Safety and Harm covers bias, harmful outcomes, self-awareness of limitations, and fabrication, falsification, or plagiarism in LLM responses.The framework distinguishes fabrication as made-up information, falsification as distorted or omitted critical details, and plagiarism as uncredited text or ideas.Source: QUEST - Five Principles of Evaluation… · pp. 10, 12, 13
• Trust and Confidence measures users’ trust and satisfaction in LLM responses, using concepts such as confidence, similarity to expert responses, and assurance.Identified evaluation formats included presence-or-absence categories and Likert scales.Source: QUEST - Five Principles of Evaluation… · pp. 10, 13

3.3.2 Evaluation Checklist

• Evaluation checklists should align evaluators with the study design by defining response options and the meaning of each option, but training and example provision are poorly reported.For accuracy, checklists may specify a Likert scale from 1–5 and define each rating; reviewed studies rarely clarify whether evaluators received adequate training or examples.Source: §3.3.2 · p. 14

3.3.3 Evaluation Samples

• Evaluation samples are a key methodological consideration because the number and variability of LLM-generated responses determine the comprehensiveness and generalizability of human assessment.The review examined both how many samples studies evaluated and how broadly those samples represented patients or applications.Source: §3.3.3 · p. 15

Sample Size

• Most reviewed studies evaluated 100 or fewer LLM outputs, whereas Moramarco et al. (2021) evaluated 2,995 sentences seven times each using MTurk.The larger sample addressed variability in annotations associated with limited control over crowdsourced annotators’ reading age and language capabilities.Source: Sample Size · p. 16, Figure 4 · p. 16

Sample Variability

• Sample variability supports evaluation across patient subpopulations by using patient-specific information from EHR clinical notes or clinician-prepared vignettes.Most prompts were patient-agnostic, but subgroup variation enabled comparisons by symptoms, diagnoses, and demographic characteristics.Source: Sample Variability · p. 16

3.3.4 Selection and Recruitment of Human Evaluators

• Evaluator recruitment should reflect the intended LLM users, with studies reporting mostly 20 or fewer evaluators across expert and non-expert assessments.Evaluator selection is task-dependent, and Figure 4 summarizes the reported evaluator counts across reviewed articles.Source: §3.3.4 · pp. 16–17

Expert Evaluation

• Clinical and clinician-facing studies generally recruited institution-based experts whose education, specialty, experience, position, and sometimes country were reported; only three studies recruited more than 50 experts.Most studies had fewer than 20 expert evaluators, as summarized in Figure 5.Source: Expert Evaluation · p. 17

Non-expert Evaluation

• Non-expert evaluation is cheaper, more available, and feasible at scale for patient-facing tasks, but typically uses more evaluators across fewer dimensions than expert evaluation.Singhal et al. (2023) recruited five non-medical evaluators from India, while Moramarco et al. (2021) used MTurk without reporting the evaluator count in detail.Source: Non-expert Evaluation · p. 17

Human Evaluators and Sample Sizes for Specific Healthcare Applications

• CDS studies had the lowest median number of evaluators and second-lowest median sample size, whereas patient-facing applications used more of both, with high variability across studies.The review suggests qualified evaluator recruitment may be especially difficult for CDS applications.Source: Human Evaluators and Sample Sizes for… · p. 18, Table 3 · p. 18
• Figure 6 showed an inverse relationship between evaluation sample size and evaluator count, highlighting the challenge of recruiting evaluators able to assess many samples.Source: Figure 6 · p. 19, Human Evaluators and Sample Sizes for… · p. 19

Evaluation Tools

• Evaluation tools included binary assessments of narrative coherence, logical reasoning, information use, and response errors, alongside Likert scales measuring quality, empathy, bedside manner, accuracy, understandability, and harm.Likert formats ranged from binary to 5-point or 7-point scales, with 4-point scales distinguishing completely from partially accurate answers.Source: Evaluation Tools · pp. 19–20

Comparative Analysis

• Comparative analyses benchmarked LLM outputs against humans, other LLMs, or clinical guidelines, while studies examined both controlled queries and real-world clinical scenarios.Twenty percent of studies (n = 29) compared outputs from different LLMs; examples used repeated-measures ANOVA, ANOVA, and chi-square tests.Source: Comparative Analysis · p. 20

Blinded vs. Unblinded

• Only 41 of 142 studies (29%) explicitly reported blinded evaluations, 20 (14%) reported unblinded evaluations, and 80 (56%) provided no blinding information.The review recommends consistently implementing and documenting blinding to reduce bias and strengthen reliability and validity.Source: Blinded vs. Unblinded · p. 21

3.3.6 Statistical Analysis

• Statistical analysis served two purposes: measuring inter-evaluator agreement and comparing LLM performance with human benchmarks or expected clinical outcomes.Table 4 summarized 11 statistical analyses, and Figure 7 provided a decision tree for selecting tests.Source: §3.3.6 · p. 21, Table 4 · p. 21

Assessing Inter-Evaluator Agreement

• Inter-evaluator agreement was assessed with measures including t-tests, Cohen's Kappa, Intraclass Correlation Coefficient, and Krippendorff's Alpha to quantify consistency beyond chance.These measures support reliability and reproducibility when evaluations involve subjective or qualitative judgments.Source: Assessing Inter-Evaluator Agreement · pp. 22–23, §3.3.6 · p. 22

Comparing LLM Performance against Benchmarks

• Benchmark comparisons used parametric tests for normally distributed data, non-parametric tests when normality assumptions failed, and Chi-Square or Fisher's Exact tests for categorical outcomes.Test selection should reflect the data type, assumptions, and evaluation objective when comparing LLMs with human outputs or clinical guidelines.Source: Comparing LLM Performance against… · pp. 23–24, Figure 7 · p. 24

3.4 Specialized Frameworks

• PEMAT-P evaluated the understandability and actionability of patient education materials generated by LLMs, including content, word choice, organization, layout, and design [63, 95].Source: §3.4 · p. 25
• SOLO structure assessed the complexity and depth of LLM responses to medical queries across pre-structural, uni-structural, multi-structural, relational, and extended abstract levels [97].Source: §3.4 · p. 25
• Other frameworks assessed medical-information data quality through accuracy, believability, completeness, conciseness, timeliness, and relevancy, while specialized questionnaires addressed factual consistency, harmfulness, coherence, and trustworthiness [98, 99].The review also notes that DISCERN was adapted to judge LLM-generated consumer health information, but specialized frameworks do not cover all metrics or QUEST dimensions.Source: §3.4 · pp. 25–26
• SERVQUAL assessed ChatGPT’s medical-information service quality for kidney cancer patients through ratings from urologists and urological oncologists across five dimensions: reliability, responsiveness, assurance, empathy, and tangibles [62].The framework was used to examine ChatGPT’s potential and limitations in direct patient interactions and learning gains.Source: §3.4 · pp. 25–26

3.5 QUEST Human Evaluation Framework

• QUEST defines a comprehensive framework for healthcare LLM evaluation, organized into Planning, Implementation and Adjudication, and Scoring and Review phases.Figure 8 presents the framework’s multi-stage process for evaluating healthcare-related LLMs.Source: §3.5 · p. 26, Figure 8 · p. 29

Planning Phase

• Planning specifies model goals, tasks, stakeholders, success criteria, evaluation samples, metrics, checklists, and evaluator requirements for each healthcare use case.Recommended sample sizes are at least 130 for clinical decision support and patient-provider quality assurance, and 100 for medical or patient education; evaluator teams comprise six for clinical applications and four for education or research.Source: Planning Phase · p. 27
• Evaluator training standardizes use of the questionnaire and guidelines, while the framework also supports crowd-sourced evaluation settings such as Mechanical Turk.Training is intended to establish consensus on tasks and requirements and promote consistent, informed assessments.Source: Planning Phase · p. 28

Implementation and Adjudication Phase

• Implementation collects evaluator ratings with surveys, forms, or spreadsheets and applies multiple statistical tests to assess rating consistency and agreement.Cyclical expert adjudication updates guidelines and retrains reviewers until consensus is reached, such as Cohen’s kappa of 0.7 or above.Source: Implementation and Adjudication Phase · p. 28

Scoring and Review Phase

• After consensus, dimension scores use mean or median ratings and are compared with automatic metrics such as F1 and AUROC for a holistic performance assessment.This comparison benchmarks human evaluations against machine-generated outputs to characterize LLM strengths and limitations.Source: Scoring and Review Phase · p. 28

4. Case Studies

• The case studies apply QUEST to ED clinical note summarization and triage decision support, where rapid, accurate clinical information processing may improve workflow efficiency.Figure 9 summarizes both applications.Source: §4 · p. 30, Figure 9 · p. 31

4.1: Human Evaluation of LLMs for Clinical Note Summarization in Emergency Department

• For ED clinical note summarization, the evaluation targets accurate, concise, standardized summaries that reduce physician documentation burden and support physicians, nurses, allied professionals, and patient care.Success requires preserving key information without omitting critical content; the study plans at least 130 notes and recruits seven clinician evaluators.Source: §4.1 · pp. 31–32
• The summarization evaluation prioritizes Accuracy and Comprehensiveness, uses binary Bias and AHRQ Harm metrics, and proceeds through guideline development, evaluator training, pilot evaluation, adjudication, and final analysis.Seven evaluators—four physicians, two nurses, and one pharmacist—complete the final evaluation over two weeks before management review.Source: §4.1 · p. 32

4.2: Human Evaluation of LLMs for Patient Triage in Emergency Department

• For ED triage decision support, 130 representative cases are independently assessed by triage experts as a gold standard and evaluated by three physicians and three nurses across QUEST dimensions.The evaluation also rates management recommendations on a five-point Likert scale, adjudicates disagreements, and generates dimension-specific scores for implementation decisions.Source: §4.2 · p. 33

5. Discussion

• Healthcare LLM evaluations remain hindered by opaque model reasoning, untraceable sources, uncertain evidence reliability, and traditional NLP metrics’ difficulty handling novel generated text.These challenges strengthen the need for evaluation frameworks aligned with human values.Source: §5 · p. 34
• Human-evaluation guidelines are constrained by evaluation scale, sample sizes, measurement choices, proprietary models, and limited healthcare computational resources, requiring cross-sector collaboration.The proposed collaboration involves medical communities, computer scientists, and major technology companies to improve clinical LLM quality and accountability.Source: §5 · p. 34

Limitations

• The review may omit recent, non-English, insufficiently indexed, or non-healthcare studies, potentially biasing findings and excluding relevant evaluation insights.The search may have missed articles published after February 22, 2024.Source: Limitations · p. 34
• QUEST implementation may vary across specialties and institutions because policies, infrastructures, resources, and trained informatics personnel differ, limiting standardization.Its applicability is also affected by clinical, educational, and research setting constraints.Source: Limitations · pp. 34–35
• QUEST is a task-specific starting point rather than a definitive solution, and its human-only focus should be balanced with scalable automatic quantitative evaluation.Users may need to adapt QUEST and combine it with use-case-specific frameworks.Source: Limitations · p. 35

Future Research Directions

• Future work should unify human and automatic evaluation, extend assessment to multimodal models, connect evaluation with patient or scientific outcomes, and improve scalability.Emerging methods may require QUEST additions, while LLM-assisted human evaluation has so far achieved limited success.Source: Future Research Directions · p. 35

LLM distribution

• OpenAI’s GPT-family models were used in the majority of reviewed studies, whereas open-source models such as Meta’s Llama were not among the most represented.Figure A1 presents the distribution of LLMs, and the accompanying text attributes GPT’s prevalence to its popularity among the general public.Source: Figure A1 · p. 48, Appendix · p. 49

Case-study evaluation questions

• The patient-triage case study asked whether management outputs were empathetic and free of racial or gender bias.These questions concerned diagnostic plans and treatment recommendations.Source: Table A3 · p. 50, Appendix · p. 51
• The case study also assessed whether management outputs were harmful and trustworthy, including accuracy, safety, and fairness.Harm was defined in relation to misinformation and adverse effects on users.Source: Table A3 · p. 50, Appendix · p. 51