PaperlayerView exact revision on arXiv

Source-linked AI summary

A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform

Yongjin Yi, Jinyong Park, Sejoong Kim

arXiv:2610.29312v1cs.HC

TL;DR

Existing LLM-VSP evaluations often lack fully crossed multi-rater comparisons against utterance-aligned human-expert references and diagnostic stratification. This study evaluates four models across Korean OSCE cases using a stratified human-rating framework and deploys a practice platform. The taxonomy surfaces a Korean negative-form Fluency penalty, and pooled encounter-level ICC(2,k) is 0.91 [0.88–0.94].

  • Problem

    Fully crossed multi-rater comparisons of LLM-based virtual standardized patients against human-expert references on identical tasks are scarce, and aggregate scores obscure specific failure modes.

    Source: §1 · p. 2

  • Method

    The study compares four LLMs with a human expert across three OSCE cases using 17 raters, an 11-category history-taking taxonomy, and three linguistic-form labels.

    Source: §1 · p. 2, §4.3 · pp. 5–6

  • Results

    The taxonomy surfaced a Korean negative-form penalty on Fluency, and fully-pooled encounter-level ICC(2,k) was 0.91 [0.88–0.94].

    Source: §1 · p. 2, §5.4 · p. 8

  • Takeaways & Limitations

    The deployed VSP platform supports free-form OSCE-style practice and applies the same taxonomy to continuously collected dialogue.

    Source: §1 · p. 3

    The human-expert reference came from one OSCE-faculty author, limiting its representativeness of educators and trained standardized patients more broadly.

    Source: §7 · p. 10

Abstract

from arXiv · show

The Objective Structured Clinical Examination (OSCE) is widely used to assess clinical communication skills in medical education, requiring trainees to take histories and reason clinically with standardized patients (SPs). Large language model (LLM)-based virtual standardized patients (VSPs), typically built by persona-injected prompting, offer a practical alternative. However, existing evaluations rely on a single backbone LLM, small human rater panels, or LLM-as-a-judge scoring without an utterance-aligned human-expert reference, and so cannot identify the specific shortcomings of individual models. We evaluated four LLMs (GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X), spanning commercial, open-source and Korean-developed models, against a human-expert reference across three Korean OSCE cases (hematuria, fever in pregnancy, and chronic cough). In a double-blind, fully crossed design, 17 senior medical-student raters scored every model utterance on three sentence-level and six encounter-level Likert metrics, yielding 7,480 data points. We analyzed the scores using a stratified linear mixed-effects model based on an 11-category history-taking and 3-level linguistic-form taxonomy. At the encounter level, three of the four LLMs fell 0.08 to 0.47 Likert points below the human-expert reference on Comprehensibility, Consistency and Hallucination-freeness. The difference was not statistically significant for Hallucination-freeness in Claude 3.5 Sonnet and GPT-4o, Consistency in GPT-4o, and Engagement in HyperCLOVA X. Llama-3.1 70B fell significantly below the reference on all six metrics. We also introduce a web-based OSCE platform that supports virtual practice across diverse clinical cases and continuously collects student-VSP dialogues under research and privacy consent.

1 Introduction

OSCEs retain educational value but face substantial cost and accessibility barriers, while evidence on LLM-based virtual standardized patients remains methodologically limited. This study combines stratified multi-rater evaluation against a human-expert reference with a deployed practice platform.

1 Introduction

  • OSCE administration faces substantial costs and access barriers even though its educational rationale remains intact.The paper links those costs and concerns about validity and redundancy to the retirement or termination of national clinical-skills examinations.Source: §1 · p. 2
  • Persona-injected LLMs offer a practical route to virtual patients that answer turn by turn in role.The approach is described as technically simple and plausible for recovering pedagogically important human-SP behaviors.Source: §1 · p. 2
  • Prior LLM-VSP evaluations commonly use one model, one scenario, small rater panels, aggregate scores, or English-language settings.Fully crossed multi-rater comparisons against a human-expert baseline on identical tasks remain scarce, and the deployable utility of VSPs on specific communication dimensions remains open.Source: §1 · p. 2
  • The study evaluates four LLMs against a human medical-education expert across three OSCE cases using 17 raters and 7,480 rating actions.The evaluation includes sentence- and encounter-level ratings and formal inter-rater reliability analysis.Source: §1 · p. 2
  • An 11-category history-taking taxonomy crossed with three linguistic forms is used to localize failure modes, including a Korean negative-form penalty on Fluency.The work also deploys a real-time VSP platform for free-form practice and applies the same taxonomy to platform-collected dialogue.Source: §1 · pp. 2–3

2 Related Work

Prior work spans VSP feasibility, educational efficacy, and dialogue-agent capabilities, but evaluations often lack broad human-rater comparisons and diagnostic stratification. This study combines fully crossed human scoring with history-taking and linguistic-form stratification, extending the framework to platform dialogue.

LLM-based virtual standardized patients

  • Prior LLM-VSP work includes feasibility and usability studies, randomized trials of educational efficacy, and agent-level dialogue evaluations.Examples include a deployment-scale RCT of DocSimulator (n = 247) and AMIE comparisons with primary-care physicians.Source: §2 · p. 3

Multi-rater human evaluation of clinical dialogue agents

  • Subjective, context-bound dialogue lacks a single correct answer, motivating multi-rater human evaluation rather than relying on aggregate accuracy or a single LLM judge.DICES and HealthBench [21] illustrate large-scale human rating panels in adjacent evaluation settings.Source: §2 · p. 3

Stratified, capability-based evaluation

  • This work combines fully crossed multi-rater dialogue scoring, history-taking and linguistic stratification, and analysis of model-specific failure modes hidden by aggregate scores.The framework also transfers from static evaluation to dialogue collected on a deployed platform.Source: §2 · p. 3

3 VSP Construction and Response Generation

The study constructs VSPs from educator-authored OSCE scenarios, persona-injection prompts, and responses generated by four LLMs plus a human expert. A common prompt set and blinded response corpus support utterance-aligned comparisons across sources.

3 VSP Construction and Response Generation

  • Each VSP combines a clinical-educator-written scenario, a declarative persona-injection prompt, and responses from four LLMs and a human medical-education expert.The construction targets coherent personas, in-role real-time responses, and reproducibility across systems.Source: §3 · pp. 3–4

3.1 OSCE Scenarios and Input Prompts

  • Three Korean OSCE scenarios cover hematuria, fever in pregnancy, and chronic cough, with case-specific histories and expected SP behavior.A set of 70 prompts sampled from prior OSCE-style training transcripts covers common history-taking tasks; all scenarios and prompts received IRB review and approval.Source: §3.1 · p. 4

3.2 VSP Prompting via Persona Injection

  • The same six-field persona-injection template was instantiated for each scenario, with case-specific values and no other prompt text varied.Each instantiation is approximately 700–900 Korean characters and is released verbatim with the dataset.Source: §3.2 · p. 4
  • Inference settings were held constant across GPT-4o, Claude 3.5 Sonnet, Llama-3.1 70B Instruct, and HyperCLOVA X.Each query received one response, with temperature 0.7, top-p 0.95, and a 1,024-token response limit.Source: §3.2 · p. 4

3.3 Generating the Multi-Source Response Corpus

  • Four LLMs and one expert independently answered the same 70 prompts, with the expert writing blind to model outputs to form a 1:1 utterance-aligned reference.Responses were randomized within case and re-blinded for raters; the full corpus contains 350 responses.Source: §3.3 · p. 4

4 Stratified Multi-Rater Evaluation

Seventeen raters scored every response and encounter in a fully crossed design, with ratings stratified by history-taking function and linguistic form. Reliability statistics and mixed-effects models were used to compare sources and examine stratified contrasts.

4 Stratified Multi-Rater Evaluation

  • Seventeen raters scored 350 responses and 15 encounters, producing 7,480 rating actions stratified across an 11-category history-taking taxonomy and three linguistic forms.The evaluation includes 5,950 sentence-level rating events and 1,530 encounter-metric assessments.Source: §4 · p. 5

4.1 Rater Recruitment

  • Raters were senior medical students with prior OSCE training and completed a 60-minute calibration session before scoring the blinded corpus.They scored asynchronously over four weeks in randomized order.Source: §4.1 · p. 5

4.2 Sentence- and Encounter-Level Metrics

  • Sentence-level ratings cover Fluency, Accuracy, and Contextual Relevance, while six encounter-level metrics capture properties of the full conversational arc.The metrics are rated on five-point Likert scales; Table 1 presents the metrics and exact rater agreement statements.Source: §4.2 · p. 5, Table 1 · p. 5

4.3 Question Stratification: History-Taking Function and Linguistic Form

  • Prompts are crossed across 11 history-taking categories and Affirmative, Negative, and Open-ended linguistic forms to support cell-level and interaction analyses.The negative-form distinction is motivated by Korean-English polarity-response divergence; ambiguous rule-based tags were reviewed by two faculty members and resolved by consensus.Source: §4.3 · pp. 5–6, Table 2 · p. 6

4.4 Inter-Rater Reliability and Mixed-Effects Modeling

  • Inter-rater reliability was measured with ICC(2,k) and ordinal Krippendorff’s α, while a REML linear mixed-effects model estimated source contrasts and stratified interactions.The model included Source, History-taking category, and Linguistic form as fixed effects, with random intercepts for Rater and Utterance nested within Case.Source: §4.4 · p. 6
  • Static evaluation materials are provided in the appendices, while the platform-collected dialogue corpus is separately released under CC BY-NC-SA 4.0.The platform dataset includes Croissant Core and Responsible-AI metadata.Source: §4.4 · p. 6

5 Experimental Results

Across encounter-level measures, the human expert led the four LLMs, with Llama-3.1 70B significantly below the reference on all six metrics; sentence-level scores were higher and more compressed. Stratification exposed category- and question-form-specific weaknesses, while rater agreement was high by ICC.

5.1 Encounter-level performance

  • Llama-3.1 70B fell significantly below the human reference on all six encounter-level metrics, with its largest Coherence gap at -1.86 [-2.14, -1.59].Claude 3.5 Sonnet, GPT-4o, and HyperCLOVA X stayed within 0.5 points on Comprehensibility, Consistency, and Hallucination-freeness; four source-metric contrasts were not statistically significant.Source: Figure 1 · p. 7, §5.1 · p. 7

5.2 Sentence-level performance

  • Pooled Accuracy was within 0.09 Likert points of the human reference across all four LLMs, while Fluency most clearly separated sources.Fluency means ranged from 4.91 for the human reference to 3.69 for Llama-3.1 70B; their ordering anticipated encounter-level Coherence ordering.Source: §5.2 · p. 7

5.3 Stratified analysis by question typology

  • Stratification exposed weaknesses in Clinical reasoning and Patient education, plus a Korean negative-form Fluency penalty significant only for Llama-3.1 70B.The additional drop from affirmative to negative-polarity yes/no questions was 0.149 [-0.286, -0.011] for Llama-3.1 70B; 16 of 350 utterances scored below 3.0 on at least one sentence-level metric, and 14 were Fluency failures.Source: §5.3 · pp. 7–8

5.4 Inter-rater reliability

  • Encounter-level pooled ICC(2,k) ranged from 0.79 (Hallucination-free) to 0.94 (Coherence), with fully pooled ICC(2,k) of 0.91 [0.88–0.94].Krippendorff’s α was lower, at 0.08–0.41, but both measures produced the same source-level ordering.Source: §5.4 · p. 8

6 VSP-OSCE Training Platform

The VSP-OSCE platform combines persona-injected role-play with a scenario-generation pipeline and real-time student interaction. Pilot sessions produced a released dialogue corpus, and all eleven history-taking categories appeared in the platform utterances covered by the taxonomy.

6.1 Platform architecture and scenario pipeline

  • The platform builds scenarios from 80 core symptoms, expert-validated disease lists, and 240 patient profiles, with generated checklists and narratives subject to expert review and diagnostic verification.At session start, a verified scenario is loaded into the persona-injection prompt; GPT-4o runs patient role-play and automated checklist-based evaluation.Source: §6.1 · pp. 8–9
  • Figure 3 organizes mean sentence-level scores by metric, source, and prompt, with prompts ordered by history-taking category and category shown by a color bar.The platform routes free-form student questions to the VSP, stores anonymized turn records, and returns a checklist-based score at session close.Source: Figure 3 · p. 9, §6.2 · p. 9, §6.1 · p. 9

6.2 Real-time interaction and released dataset

  • The pilot generated 49 sessions across 27 clinical scenarios and approximately 1,763 student–VSP turns; all eleven taxonomy categories appeared among the 1,614 covered platform utterances.The dialogue corpus is publicly released under CC BY-NC-SA 4.0 with Croissant metadata including Responsible-AI fields.Source: §6.2 · pp. 9–10

7 Discussion

The discussion identifies limits in the reference, rater and model panels, modality, and case coverage, then outlines extensions and concludes that learning effects still require controlled study.

Limitations

  • The reference came from one OSCE-faculty author, the rating and model panels were limited to 17 students and four LLMs, and the text-only study covered three scenarios.The authors note that the Korean OSCE/CPX case banks include more than fifty case classes, including specialties such as psychiatry and pediatrics that were not represented.Source: §7 · p. 10

Future directions

  • The authors propose broadening the human-reference pool and model panel, validating LLM-as-a-judge pipelines against the released ratings, and extending the platform to speech, video, or VR.They expect richer modalities to yield engagement signals closer to those of a real OSCE.Source: §7 · p. 10

Conclusion

  • The framework, prompt set, and deployed platform span scenario authoring, evaluation, and continuous dialogue generation, but establishing educational effects requires a controlled comparison of learning outcomes.The framework is released to support medical-education practice and downstream LLM evaluation in OSCE-based training.Source: §7 · p. 10

NeurIPS Paper Checklist

The checklist affirms that the paper’s claims align with its demonstrated contributions and scope, and records its reproducibility materials, statistical reporting, ethics procedures, and dataset documentation.

Claims and scope

  • The checklist confirms that the paper’s stated contributions are demonstrated in its evaluation, platform, and appendices.It also notes that claims reflect the paper’s scope.Source: NeurIPS Paper Checklist · p. 13

Limitations

  • The checklist identifies a single-author human reference, a constrained rater and model panel, and a text-only simulator as study limitations.The checklist also records that the scenarios and rating protocol received IRB review.Source: NeurIPS Paper Checklist · p. 13

Reproducibility

  • The paper provides its static evaluation materials, platform-collected dialogue corpus, and platform code, with reproduction details for inference and statistical analysis.The materials include the 350-response corpus, 7,480 rater scores, annotated input prompts, and persona-injection template; the platform dataset is released under CC BY-NC-SA 4.0.Source: NeurIPS Paper Checklist · pp. 14–15

Statistical reporting

  • Effect estimates are accompanied by confidence intervals, with bootstrap intervals for descriptive means and Wald-style 95% intervals for mixed-model contrasts.The checklist also notes that the paper reports confidence intervals for ICC.Source: NeurIPS Paper Checklist · p. 15

Dataset and impact documentation

  • The platform dialogue corpus is documented with a Datasheet and Croissant metadata, while the checklist records societal-impact discussion and asset licensing.The dataset documentation describes intended use, limitations, and biases; the released dataset uses a CC BY-NC-SA 4.0 license.Source: NeurIPS Paper Checklist · pp. 17–18

Human subjects and ethics

  • The study reports voluntary participation, written informed consent, IRB approval, participant compensation, and reproduced rating instructions.The platform-collected dataset was released under research-participation and personal-information consent.Source: NeurIPS Paper Checklist · pp. 16, 18, 19

A Datasheet for Datasets

The datasheet describes a consent-based Korean OSCE dialogue corpus collected on the deployed VSP platform and distinguishes it from the paper’s static evaluation materials.

A.1 Motivation

  • The platform-collected corpus was created to support research on LLM-based VSPs in Korean OSCE-style education through real-time student–VSP dialogue.The static evaluation corpus, rater scores, and prompt template are provided separately in the appendices rather than in the hosted dataset.Source: §A.1 · p. 19, A Datasheet for Datasets · p. 19
  • The authors created the dataset; its ratings came from 17 senior medical-student raters recruited with informed consent, and its creation had IRB-approved consent.The datasheet lists funding from the Korean government’s Bio&Medical Technology Development Program.Source: §A.1 · pp. 19–20

A.2 Composition

The dataset consists of student–VSP dialogue turns collected through the platform, with participant identifiers anonymized during preprocessing. Its composition and collection are bounded by a volunteer student cohort, occasional incomplete sessions, and minor dialogue noise.

A.2 Composition

  • Each instance pairs a Korean student question with a GPT-4o VSP response, recorded with session, participant, scenario, turn, question, and response fields and no added labels.The accompanying persona-injection prompt is provided; no external resources are linked.Source: §A.2 · p. 20
  • A few sessions ended early, and the corpus contains occasional typing errors and rare role-inconsistent VSP responses; all collected turns are retained.The corpus has no structural fields missing and no redundancies.Source: §A.2 · p. 20

A.3 Collection process

  • The platform logs real-time text interactions, and participants provided written research and personal-information consent with a right to withdraw.Withdrawn participants are removed from subsequent dataset releases.Source: §A.3 · pp. 20–21
  • Collection used a convenience sample of 17 medical-student volunteers, and the research team authored 27 scenarios from an internal symptom-and-profile database with diagnostic-model verification and expert review.The VSP answers are generated under persona injection; student questions are human-typed.Source: §A.3 · pp. 20–21

A.4 Preprocessing, cleaning, and labeling

  • Preprocessing replaces identifiers with R001–R017, substitutes real names in text, and destroys the original mapping; raw data are not saved separately.Preprocessing software is released with the dataset, and no labels beyond structured-record fields are added.Source: §A.4 · p. 21

A.5 Uses

The corpus is intended for research and supervised OSCE-related uses, with explicit boundaries on clinical use and population scope. The appendices document its distribution and maintenance, and provide the shared persona-injection prompt and a worked hematuria example.

A.5 Uses

  • The corpus is proposed for LLM-VSP development, OSCE training research, clinical-dialogue benchmarking, and domain-specialized LLM post-training, but not for diagnosis or standalone unsupervised medical training.Its stated validity is limited to Korean-language research, OSCE training, and multi-rater methodology, not non-Korean contexts or clinical decision-making.Source: §A.5 · p. 21

A.6 Distribution

  • The dialogue dataset is publicly distributed on Hugging Face under CC BY-NC-SA 4.0, with restrictions against re-identification, clinical decision-making, diagnosis, and standalone medical training.Distribution includes Croissant metadata.Source: §A.6 · p. 22

A.7 Maintenance

  • The dataset is version-tagged and can be extended; new platform sessions may be appended as cohorts use it.The authors describe it as part of a construction–evaluation–training–data-generation cycle.Source: §A.7 · p. 22

B Persona-Injection Prompt

  • All three cases use one shared persona-injection template, varying the chief complaint, patient identifier, and scenario body, with an optional pregnancy-specific behavioral directive.A fully instantiated hematuria example illustrates the template.Source: B Persona-Injection Prompt · p. 22

B.1 Unified template (English translation)

  • The unified template frames the scenario as a prior medical record and asks for a conversation between the patient and a student-doctor.Its scenario body covers demographics, symptom course, history, and patient concerns, including explicit negative findings.Source: §B.1 · p. 22

B.2 Per-case variable assignments

  • Table B1 identifies the per-case variable assignments for the unified persona-injection template.The supplied caption describes the table's scope but not its individual entries.Source: Table B1 · p. 23

B.3 Worked example — hematuria case (English translation)

  • The hematuria worked example combines symptom chronology and negative findings with lifestyle, family history, and the patient's concerns about serious illness and testing.The scenario identifies a 28-year-old man with red urine following fever and muscle aches.Source: §B.3 · p. 23

C Translated Generated Dialogues

This section presents English translations of generated dialogue excerpts from the hematuria, fever-in-pregnancy, and chronic-cough cases, with Korean originals available alongside the evaluation materials. The excerpts pair prompts with replies from four LLMs and the human-expert reference, showing similarities and differences in response detail and symptom descriptions.

C Translated Generated Dialogues

  • 350 translated responses cover 70 prompts, five sources, and three OSCE cases, with Korean originals released alongside the static-evaluation materials.The five sources are four LLMs and the human-expert reference.Source: C Translated Generated Dialogues · p. 24
  • This part presents the hematuria case, translated from Korean, with each source responding to prompts about the patient’s presentation and history.The excerpt includes questions about urine color and onset, associated symptoms, medication, family history, smoking, alcohol use, and sexual history.Source: C Translated Generated Dialogues · p. 24, Case: Hematuria (red urine)… · pp. 24, 26, 27
  • Speaker labels abbreviate the human-expert reference as Human and identify Claude 3.5 Sonnet, Llama-3.1 70B, and HyperCLOVA X as Claude, Llama, and HyperCLOVA.GPT-4o is also shown by name in the responses.Source: C Translated Generated Dialogues · p. 24, Case: Hematuria (red urine)… · p. 24
  • In the hematuria case, GPT-4o restates the proposed tests, Claude summarizes them, HyperCLOVA asks to start testing immediately, and Llama only acknowledges.The physician had proposed urinalysis, microscopy, proteinuria testing, kidney-function blood tests, and possible further testing if abnormal.Source: Case: Hematuria (red urine)… · pp. 27–28
  • At the pregnancy-case opening, Claude, GPT-4o, and HyperCLOVA add symptoms to the reported fever, while Llama also asks the next question.The human reports high fever; model replies variously add muscle pain or right-flank pain, and Llama adds severe body aches and sharp flank pain before asking about fever severity.Source: Case: Fever (in pregnancy) (translated… · p. 28
  • On chills, the human reports feeling cold and shivery, whereas Llama denies cold sweats while Claude, GPT-4o, and HyperCLOVA acknowledge chills.The four models otherwise give differing levels of detail in this exchange, from Claude's shivery description to brief acknowledgments.Source: Case: Fever (in pregnancy) (translated… · p. 28

C Translated Generated Dialogues

  • The cough’s positional pattern differed across replies: the human reported worsening on waking and lying down, while HyperCLOVA said lying down did not particularly worsen it.Claude and GPT-4o affirmed worsening when lying down; Llama described multiple positions and added worsening at night after oily or spicy food.Source: Case: Chronic cough (translated from… · p. 33
  • After the interviewer identified GERD-related cough as the most likely cause, Claude, GPT-4o, HyperCLOVA, and Llama each acknowledged GERD or acid reflux as the suspected cause.Source: Case: Chronic cough (translated from… · p. 35
  • In response to advice to elevate the pillow and avoid overeating, irregular meals, coffee, soft drinks, and juices, all four LLMs agreed to try the lifestyle changes.Claude named an elevated pillow, regular meals, coffee, and soft drinks; GPT-4o and Llama referred generally to the lifestyle changes, and HyperCLOVA said it would comply.Source: Case: Chronic cough (translated from… · pp. 35–36

D Per-Case Encounter-Level Likert Scores

The appendix reports per-case encounter-level scores and confidence intervals across three OSCE cases, with profiles for comparing five response sources. It also provides inter-rater reliability statistics and sentence-level means by linguistic form and source.

D Per-Case Encounter-Level Likert Scores

E Inter-Rater Reliability and Linguistic-Form Means

F Platform Corpus Category Distribution

Table F1 compares the distribution of eleven history-taking categories in the 70-prompt evaluation set and the platform-collected corpus.

F Platform Corpus Category Distribution

  • Table F1 compares eleven history-taking categories in 70 prompts and a platform corpus of 49 sessions, 27 scenarios, and 1,763 utterances.Platform percentages use the 1,614 utterances covered by the scheme; non-question directives and Rapport-building are counts only, and one researcher corrected all platform tags.Source: Table F1 · p. 39, F Platform Corpus Category Distribution · p. 39
Loading 2610.29312v1…