Source-linked AI summary
Prompting is not enough: supervised baselines and leakage control for measuring shared decision-making with LLMs in pediatric encounters
Bernardo Modenesi, Jody Lin, Kimberly Kaphingst, Angela Zhu, Maya Wheeler, Peilu Zhang, Angela Fagerlin
TL;DR
Measuring shared decision-making in pediatric encounters is costly and slow, raising whether zero-shot LLM prompting is sufficient. In patient-grouped evaluation, a small supervised model outperformed zero-shot prompting, while leakage control remained essential.
Problem
Human coding makes SDM measurement costly, slow, and retrospective, motivating evaluation of whether LLM prompting can approximate ratings.
Method
The study compared zero-shot Qwen 2.5 32B, a supervised classifier over frozen embeddings, and their logistic stack on 21 pediatric surgical encounters.
Results
The supervised model improved macro κ over zero-shot prompting by 0.088 (95% paired patient-bootstrap interval 0.051–0.119).
Takeaways & Limitations
Zero-shot prompting alone was insufficient, supporting supervised learning as a baseline and extending leakage control to labeled exemplars and prediction caches.
Takeaways & Limitations
This single-site study involved 19 patients with no external test set, so external and prospective evaluation is required.
Abstract
from arXiv · showhide
Objectives: To determine whether zero-shot prompting of a large language model (LLM) is sufficient to detect shared decision-making (SDM) behaviors in real clinical encounters, and whether supervised learning adds value under patient-grouped, nested evaluation. Methods: We analyzed 21 audio-recorded outpatient surgical decision encounters (19 unique patients; 7,566 utterance segments; ~6.1 hours) between families of children with multiple long-term conditions and their surgical providers. Trained coders labeled segments for 12 SDM behaviors (human-human macro Cohen's kappa = 0.695). We compared a zero-shot local LLM (Qwen 2.5 32B), a supervised classifier over frozen sentence embeddings, and their logistic stack, under patient-grouped outer folds with inner cross-fitted thresholds and patient-resampled confidence intervals. Results: The zero-shot LLM reached macro kappa = 0.139 (95% CI 0.111-0.164). The supervised classifier reached kappa = 0.227 (0.186-0.262), a paired improvement of 0.088 (0.051-0.119). A logistic stack of the two reached kappa = 0.242 (0.198-0.284). We identified multiple corpus-specific leakage paths, including grouping sibling recordings separately and allowing labels from an outer held-out patient to enter few-shot exemplars used while fitting downstream models. Conclusion: Zero-shot prompting alone is not sufficient to measure SDM behavior as reliably as a small supervised model, and patient-level grouping alone does not prevent leakage when labeled prompt exemplars are precomputed outside the outer evaluation loop. Reported performance is sensitive to the unit of data splitting and to where labeled exemplars enter the pipeline. External validation is needed before these findings generalize beyond this population, model, prompt, and codebook.
1 Background and Significance
Children with medical complexity are disproportionately costly and difficult to study, while treatment decisions may favor high-intensity options amid limited evidence. Human coding makes SDM measurement costly and retrospective, motivating on-premises comparisons of LLM prompting, supervised learning, hybrid modeling, and leakage auditing in real encounters.
- Motivation: Fewer than 5% of pediatric patients, children with medical complexity account for over 30% of pediatric health expenditures and have heterogeneous, medically fragile conditions.These characteristics make randomized evidence difficult to generate and may leave clinicians choosing high-intensity, high-risk options when less intense options are equally reasonable.
- Measurement challenge: Human coding of recorded encounters remains expensive, slow, and retrospective, limiting routine SDM measurement and feedback despite efficacious interventions.Recent work suggests that large language models may approximate human SDM ratings from transcripts.
- Evaluation context: Real pediatric surgical consultations are fragmented, multi-party, and rich in backchannel, while identifiable clinical audio and transcripts require approved safeguards that motivate local inference.Small-n clinical corpora are also vulnerable to optimistic estimates involving recordings, patients, and labeled prompt exemplars.
- Paper contributions: The paper compares zero-shot prompting, supervised learning, and their hybrid on real encounters entirely on-premises, audits leakage paths, and describes a patient-grouped supervised learning curve.The leakage audit addresses repeated patients and labeled prompt exemplars.
2 Objectives
This section frames the study around whether zero-shot LLM prompting can detect shared decision-making behaviors in real clinical encounters. It also examines whether supervised learning adds value under patient-grouped, nested evaluation.
- Objective: The study tests whether zero-shot LLM prompting is sufficient to detect SDM behaviors in real clinical encounters.The objective focuses on detection performance in actual clinical encounters.
- Objective: It evaluates whether supervised learning adds value beyond zero-shot prompting.The comparison is conducted under patient-grouped, nested evaluation.
- Objective: The evaluation uses patient-grouped, nested procedures to assess these modeling approaches.The objective explicitly specifies patient-grouped, nested evaluation.
3 Materials and Methods
The study analyzes patient-linked pediatric surgical encounters using a 12-behavior SDM codebook and compares zero-shot, supervised, and hybrid models under patient-grouped nested evaluation. Leakage audits and patient-clustered uncertainty estimates are integrated into the methodology.
- Data: 21 encounters from 19 patients yielded 7,566 speaker-attributed utterance segments across approximately 6.1 hours of locally transcribed and diarized audio.Two patients contributed two recordings each.
- Annotation: 0.695 macro Cohen’s κ measured agreement between human coders across 18 original double-coded recordings using the 12-behavior SDM codebook.Per-behavior agreement ranged from 0.41 to 0.83, and model targets used the union of available labels.
- Models: Three systems used the same 7-utterance speaker-marked window: zero-shot Qwen 2.5 32B, a frozen Qwen3-Embedding-0.6B supervised classifier, and a logistic hybrid.The zero-shot prompt supplied the 12-behavior codebook and produced graded confidence scores; the hybrid combined model log-odds per behavior.
- Leakage control: A four-shot labeled-exemplar audit was excluded from primary results because its precomputed cache was not fully nested with respect to outer folds.Although each query excluded exemplars from its own patient, the cache could still violate outer-fold isolation.
- Evaluation: 19 patient-grouped outer folds and 5-fold inner cross-fitting fit supervised probabilities, combiner weights, and decision thresholds without using in-sample supervised predictions.Thresholds were selected from 0.05 to 0.95 in 0.05 increments by maximizing training-fold κ.
- Evaluation: Macro Cohen’s κ was computed across 12 behaviors, with 95% intervals from 1,000 patient-cluster bootstrap resamples and paired differences from 5,000 resamples.Whole patients, rather than utterances or recordings, were resampled because segments within a patient are not independent.
4 Results
Under patient-grouped nested evaluation, supervised modeling improved macro κ over zero-shot prompting, while performance varied substantially by behavior and patient-level data sufficiency. Several leakage pathways show that patient-aware splitting alone is insufficient unless labeled exemplars are generated within each outer fold.
- Overall performance: 0.088: The supervised model improved macro κ over zero-shot prompting, with a 95% paired patient-bootstrap interval of 0.051–0.119.The zero-shot logistic stack had a higher point estimate than the supervised model, but overlapping marginal intervals precluded claiming a paired improvement.
- Behavior-level performance: The zero-shot LLM was strongest for definition of option, whereas the supervised model was markedly stronger for patient outcome expectations, plan for follow-up, and provider preferences & values.These are descriptive per-behavior contrasts without uncertainty intervals.
- Behavior-level performance: patient self-efficacy was not reliably detected by any approach despite human agreement of κ = 0.764, reflecting only 10 contiguous spans across 6 patients among 126 labeled segments.The paper attributes this to insufficient independent positive events for stable assessment, not inherent undetectability, and calls for targeted data collection.
- Leakage control: 3 of 21 recordings triggered exemplar leakage because excluding a held-out recording still admitted a sibling recording from the same patient.Two patients contributed two recordings each, allowing recording-level folds to place one recording in training and another in evaluation.
- Leakage control: 3 of 19 outer folds contained outer-held-out-patient exemplars in cached predictions used for downstream fitting, invalidating the cache for nested downstream modeling.The cache remained valid for leave-one-patient-out standalone prompting but not for nested downstream fitting; affected few-shot stack and metadata-gate estimates were excluded.
- Data sufficiency: ρ = 0.70: Across 15 patient-group subsampled runs, patient count had a positive descriptive association with supervised-model macro κ.Mean macro κ was 0.113, 0.202, 0.178, 0.200, and 0.214 at 6, 9, 12, 15, and 18 patients, versus 0.227 for all 19; limited replicates and overlapping subsamples prevent learning-curve conclusions.
5 Discussion
The discussion argues that zero-shot prompting was insufficient in this dataset and configuration, while emphasizing leakage control and limits on generalization. It proposes interactional structure and clinic-specific realizations as hypotheses, not established mechanisms.
- Why zero-shot prompting may underperform here: Zero-shot prompting may underperform because some SDM behaviors depend on interactional structure rather than topic words alone.The study does not isolate this mechanism.
- Why zero-shot prompting may underperform here: Clinic-specific span realizations may also challenge a general model that has not observed them.This remains a hypothesis for future testing, not an isolated mechanism in the present study.
- What this does and does not say about LLMs: The findings do not show that LLMs are unsuitable, but that zero-shot prompting was insufficient and a small supervised model was a stronger baseline.The study tested one local LLM, one prompt family, one codebook, and one clinical population, so it does not establish a general ranking.
- Evaluation is part of the model: Patient grouping is necessary but insufficient when labeled prompt exemplars or cached LLM outputs enter feature creation.All labeled artifacts must be generated inside the appropriate outer fold, a requirement easily violated when expensive outputs are cached and reused.
- Limitations: The study was single-site, included 19 patients, and had no external test set, limiting generalization beyond the developed corpus.Additional limitations include non-adjudicated union references, absent per-behavior uncertainty intervals, and offline evaluation that does not establish streaming or clinical performance.
6 Conclusion
In real pediatric encounters, a small supervised model agreed more strongly with SDM reference labels than zero-shot Qwen 2.5 32B under paired, patient-grouped evaluation. The conclusion emphasizes supervised baselines, broader leakage control, and external validation before clinical or broader LLM claims.
- Model comparison: A small supervised model agreed more strongly with SDM reference labels than zero-shot Qwen 2.5 32B under paired, patient-grouped evaluation.The finding supports supervised learning as a necessary baseline for clinical-dialogue measurement.
- Evaluation safeguards: Leakage control must extend beyond patient grouping to labeled prompt exemplars and prediction caches.
- Generalizability: Larger, externally validated cohorts are needed before drawing conclusions about clinical use or LLMs beyond this model.