PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing When to Abstain: Medical LLMs Under Clinical Uncertainty

Sravanthi Machcha, Sushrita Yerra, Sahil Gupta, Aishwarya Sahoo, Sharmin Sultana, Hong Yu, Zonghai Yao

arXiv:2601.12471v2cs.CLcs.AI

TL;DR

Medical LLM benchmarks largely prioritize accuracy, leaving uncertainty-aware abstention insufficiently assessed despite its importance in high-stakes care. MedAbstain introduces a unified medical MCQA protocol combining conformal prediction, perturbations, and explicit abstention, and finds that abstention options more strongly increase uncertainty and safer abstention than perturbations alone. The framework supports uncertainty-aware evaluation, while remaining limited in language and task scope.

  • Problem

    Existing medical LLM evaluations focus on accuracy and lack unified, scalable methods for assessing justified abstention and uncertainty, especially for black-box models.

  • Method

    MedAbstain evaluates original, perturbed, and abstention-augmented medical MCQA with conformal prediction across open- and closed-source LLMs.

  • Results

    Explicit abstention options generally increase uncertainty and abstention more than input perturbations, while prediction-set size correlates negatively with accuracy and positively with abstention.

  • Takeaways & Limitations

    Abstention-aware mechanisms provide a promising basis for improving uncertainty assessment and reducing unreliable outputs in high-stakes LLM deployment.

  • Takeaways & Limitations

    The benchmark focuses on English-language medical MCQA, which may not capture multilingual contexts or the complexity of open-ended, generative, and multimodal clinical tasks.

Abstract

from arXiv · show

Current evaluation of large language models (LLMs) overwhelmingly prioritizes accuracy; however, in real-world and safety-critical applications, the ability to abstain when uncertain is equally vital for trustworthy deployment. We introduce MedAbstain, a unified benchmark and evaluation protocol for abstention in medical multiple-choice question answering (MCQA) -- a discrete-choice setting that generalizes to agentic action selection -- integrating conformal prediction, adversarial question perturbations, and explicit abstention options. Our systematic evaluation of both open- and closed-source LLMs reveals that even state-of-the-art, high-accuracy models often fail to abstain with uncertain. Notably, providing explicit abstention options consistently increases model uncertainty and safer abstention, far more than input perturbations, while scaling model size or advanced prompting brings little improvement. These findings highlight the central role of abstention mechanisms for trustworthy LLM deployment and offer practical guidance for improving safety in high-stakes applications.

1 Introduction

Medical LLM evaluation has largely emphasized accuracy despite the safety-critical need to recognize uncertainty and abstain. MedAbstain addresses this gap by unifying conformal prediction, explicit abstention, and perturbed medical questions, revealing that abstention options substantially affect uncertainty and safer behavior.

  • Motivation: In clinical decision support, ambiguous or incomplete questions make abstention important for preventing harmful, overconfident answers.Fabricated clinical content can jeopardize patient safety.
  • Motivation: Accuracy-focused medical benchmarks rarely assess whether answers should have been withheld or whether model confidence was justified.This leaves abstention and uncertainty insufficiently evaluated, especially for closed-source models.
  • Contribution: MedAbstain combines conformal prediction with adversarially perturbed and abstention-augmented medical MCQA for open- and closed-source models.The benchmark includes original questions and versions with missing key details or misleading distractors.
  • Findings: Explicit abstention options generally increase model uncertainty, although some larger or closed-source models are exceptions.The reported association appears across datasets but is not universal, including cases such as GPT-4.1.
  • Findings: Information perturbations affect uncertainty less than abstention mechanisms, while model scaling and instruction tuning do not consistently improve abstention.Chain-of-thought prompting can increase uncertainty without making abstention safer.

2 Related Work

Prior work has advanced uncertainty estimation, abstention, and reasoning in LLMs, but standardized abstention evaluation remains limited. Existing studies rarely combine clinical MCQA, adversarial perturbations, model scale, and proprietary-model assessment.

  • Uncertainty and conformal prediction: Conformal prediction offers model-agnostic uncertainty quantification, whereas classical uncertainty methods can be impractical or poorly generalized for black-box LLMs.This motivates conformal approaches for scalable evaluation.
  • Abstention and calibration: Standardized abstention evaluation remains rare for MCQA and proprietary models despite recent abstention benchmarks and prompting, finetuning, and post-processing approaches.Existing methods often have limited calibration or generalization.
  • Reasoning and hallucination: Accuracy-centric reasoning benchmarks often overlook overconfidence and the tendency to answer despite uncertainty.The connection between hallucination and abstention has been explored, but systematic evaluation remains limited.
  • Open gap: Prior benchmarks seldom examine how model scale, reasoning, and abstention interact in clinical MCQA with adversarial or perturbed questions.Most related work instead focuses on open-domain tasks.

3 Methodology

MedAbstain evaluates medical MCQA using original, abstention-augmented, perturbed, and combined variants. It extracts prediction probabilities and applies conformal prediction to quantify uncertainty alongside accuracy and abstention behavior.

  • Datasets: The study focuses on medical MCQA because its well-defined label space supports conformal prediction and abstention analysis.Evaluation uses MedQA and AMBOSS datasets.
  • Dataset variants: The NA variant preserves original questions as the baseline, while the A variant inserts an abstention option at a random position.These variants probe prediction and confidence with and without an explicit abstention choice.
  • Evaluation: For each variant, the evaluation extracts logits or logprobs and applies conformal prediction using a calibration-derived threshold to construct prediction sets.The pipeline is repeated for open- and closed-source LLM families.
  • Dataset variants: The NAP variant removes essential information identified by GPT-4.1-mini, creating more ambiguous questions without an abstention option.It serves as a reference for assessing the subsequent combined condition.
  • Dataset variants: The AP variant combines omitted information with an abstention option to test whether uncertainty and abstention increase together.This condition is designed to challenge model confidence most directly.
  • Metrics: Accuracy measures top-label correctness, while abstention rate measures the percentage of test instances assigned the abstention option.Abstention rate is reported for abstention and perturbed-abstention variants.

2) Least Ambiguous Classifier (LAC)

LAC uses conformal prediction-set size as an uncertainty signal, while APS evaluates confidence and ranking quality. Larger LAC sets generally correspond to lower accuracy, and both measures support abstention analysis.

  • Conformal construction: The conformal threshold is the (1 −α) quantile of calibration scores and determines which labels enter each prediction set.The model’s label probabilities provide the scores used for set construction.
  • LAC: LAC measures prediction-set size, with larger sets typically indicating lower accuracy and greater uncertainty.Set size is used as a confidence-related signal in the evaluation.
  • APS: APS measures confidence and ranking quality by assessing how effectively correct answers are prioritized within the prediction set.It complements LAC’s set-size-based uncertainty measure.
  • Abstention: Abstention rate records how often the model selects the abstention option on abstention-enabled variants.This metric links uncertainty assessment with observable refusal behavior.

4 Experiments

The experiments evaluate open- and closed-source LLMs across multiple settings and use conformal prediction to quantify uncertainty through prediction sets.

  • The evaluation spans open- and closed-source LLMs across multiple architectural families and model scales.
  • Models are compared in zero-shot, few-shot, chain-of-thought, and reasoning-model thinking settings.
  • Each condition produces one selected answer token, whose logits and competing-choice logits support accuracy and conformal-prediction score computation.
  • Conformal prediction uses a 30% calibration split, a 70% stratified test split, and a target coverage guarantee of 90%.The protocol evaluates both LAC and APS scoring functions and records prediction-set size.

5 Results and Discussion

Across AMBOSS and MedQA, explicit abstention generally increases uncertainty and abstention more than perturbation, while few-shot and chain-of-thought prompting provide limited gains. Prediction-set size is generally associated with lower accuracy, but human comparisons show substantial missed clinically warranted abstentions.

  • Uncertainty and abstention: Across both datasets, LAC and APS set sizes are negatively correlated with accuracy and positively correlated with abstention.Set size therefore serves as an uncertainty signal for assessing whether a model should abstain.
  • Benchmark variants: Explicit abstention options produce the largest increases in uncertainty and abstention, especially in AP, while perturbation has a comparatively smaller effect.This pattern appears in both AMBOSS and MedQA.
  • Few-shot prompting: Few-shot prompting yields negligible abstention gains and modest accuracy improvements across AMBOSS and MedQA.The gains are generally small, with APS changes often near zero or heterogeneous.
  • Chain-of-thought: Chain-of-thought produces negligible changes in accuracy and abstention across the datasets.In AMBOSS it slightly enlarges LAC, while MedQA shows negligible effects on both accuracy and set sizes.
  • Thinking mode: Thinking-enabled models show small accuracy gains and tighter sets but consistently lower abstention rates across both datasets.The MedQA-AP condition is an exception for LAC, which increases slightly.
  • Human evaluation: 77.55% of human-labeled perturbed cases were judged to require abstention, while model abstention achieved 71.43% precision and 13.16% recall.The low recall indicates frequent failure to abstain when clinicians judged abstention appropriate.

6 Conclusion

MedAbstain studies how explicit abstention affects uncertainty and abstention behavior in medical MCQA. Its results show that abstention options influence uncertainty and unreliable-output avoidance more strongly than perturbations alone.

  • MedAbstain evaluates the effects of abstention mechanisms on model uncertainty, abstention selection, and their relationship.
  • Explicit abstention options exert a greater influence on uncertainty calibration and avoidance of unreliable outputs than input perturbations alone.Combining abstention-awareness with perturbations produces an even stronger effect.

7 Limitations

MedAbstain’s evaluation is limited by language, model coverage, task format, perturbation coverage, and black-box uncertainty estimation. These boundaries constrain how broadly its findings can be generalized to clinical deployment.

  • Language coverage: The benchmark uses only English-language datasets, limiting representation of multilingual and non-English medical contexts.The authors call for extensions to additional languages and healthcare systems.
  • Model coverage: Its finite set of open- and closed-source models may not represent future or unreleased models as capabilities and training paradigms evolve.Reported performance and behavior may therefore not generalize to later systems.
  • Task scope: Because the methodology focuses primarily on multiple-choice QA, it may not capture uncertainty in real-world clinical reasoning or open-ended medical tasks.The authors identify generative, free-form, and multimodal evaluation as future directions.
  • Scenario coverage: The perturbation and abstention protocols may not exhaustively represent clinically relevant ambiguities or uncertainty scenarios.Some real-world cases warranting abstention may be absent from the current protocols.
  • Black-box uncertainty: For black-box models, uncertainty estimates depend on API-exposed confidence scores or log-probabilities that may reflect implementation artifacts or undocumented calibration.The authors therefore identify closed-source uncertainty quantification as an open technical challenge.

8 Ethics Statement

The study evaluates medical LLM abstention using public MedQA and private AMBOSS data, with synthetic or de-identified exam data and research-only release constraints. Its protocol constructs perturbed and abstention-enabled variants while avoiding test-set exposure in few-shot exemplars.

  • Data sources: The evaluation uses publicly available MedQA and a proprietary AMBOSS clinical QA dataset that cannot be publicly released.AMBOSS is used solely for internal benchmarking under research-agreement terms.
  • Ethical handling: No patient-identifiable or private clinical data are used, and protocols rely on synthetic or de-identified medical exam data.The study is intended to improve safety and reliability in high-stakes applications but is for research purposes only.
  • Deployment boundary: The benchmark should not substitute for rigorous clinical validation or human oversight because abstention does not eliminate bias or inaccuracy.The authors note that models may reflect biases in their training data or benchmarks.
  • Release policy: The codebase is released under CC-BY-NC 4.0, while the AMBOSS dataset is excluded from the release.This supports research transparency without publicly distributing the licensed clinical dataset.
  • Dataset construction: MedQA contributes 1007 test examples, while AMBOSS contributes 1000 sampled test instances across five difficulty levels.Few-shot validation examples are sampled from provided validation splits.
  • Perturbation protocol: Perturbed questions remove the gold context while preserving the label, using GPT-4.1-mini to identify and rewrite around the omitted clue.The procedure isolates the effect of missing information on abstention.
  • Few-shot conditions: Few-shot exemplar pools come exclusively from training data to avoid test-set exposure.Four conditions are derived from the base pool: No-Abstention, Abstention, Perturbed–No-Abstention, and Perturbed–Random-Abstention.
  • Few-shot tuning: Tuning experiments selected k = 4 dynamic few-shot examples for all few-shot experiments.The choice used accuracy, LAC set size, and APS set size on a validation sample.

E Experiment Models

The experiments benchmark diverse open- and closed-source model families across scales, using zero-shot, few-shot, and chain-of-thought settings. Dynamic few-shot examples are selected by embedding similarity, with k = 4 used throughout.

  • Model families: The benchmark evaluates models across varying scales and architectural families, including both open-source and closed-source systems.The listed families include LLaMA, Phi, Qwen, Gemma, and GPT.
  • Few-shot selection: Dynamic few-shot evaluation selects k semantically similar training examples using cosine similarity in embedding space.Test and training embeddings are generated with text-embedding-ada-002.
  • Few-shot configuration: k = 4 dynamic few-shot examples are used for all dataset variants.The value was selected through a tuning procedure described in the appendix.
  • Abstention balancing: Perturbed abstention variants randomly replace the correct answer with abstention for 25% of sampled questions to mitigate selection bias.The construction process is described in Appendix C.
  • Prompt settings: The evaluation combines zero-shot, few-shot, and chain-of-thought prompts, with each setting using corresponding instructions and examples.Few-shot CoT combines the few-shot prompt, examples, CoT instruction, and test instance.
  • Human evaluation: Human evaluation covers 50 medical questions in four variants, producing 200 evaluated instances for Qwen models.The variants cross original versus perturbed questions with abstention enabled versus disabled.
  • Annotation criteria: Annotators rate removed-context importance on a 1–3 scale, from irrelevant to essential.The scale distinguishes redundant information, useful information, and information critical for the original answer.
  • Abstention labels: Annotators judge abstention appropriateness by whether a clinician would defer, request information, or order further tests.A negative judgment means a clinician could reasonably answer with high confidence.

H.3 Extended Abstention Analysis

Human comparison shows conservative model abstention that often misses clinically warranted deferral, while uncertainty–accuracy relationships vary by conformal method and regime. CoT and NoCoT produce distinct slope patterns, with AP showing the steepest negative trend.

  • Human comparison: 77.55% of 49 perturbed instances were judged clinically appropriate for abstention, but model recall was 13.16%.The model abstained correctly in 5 cases and unnecessarily in 2, yielding 71.43% precision.
  • Human comparison: Model abstention is relatively conservative but often fails to abstain when clinicians judge abstention warranted.This summarizes the comparison between model decisions and human judgments.
  • Coherence: Coherence ratings cluster around 2 and 3, indicating partially coherent but incomplete reasoning in a limited descriptive subset.Sparse coverage and uneven annotation density prevent quantitative comparison.
  • Reading the results: Across consolidated tables, darker entries denote better performance: higher accuracy, smaller set sizes, and higher abstention rates.The same visual convention applies across the reported metrics.
  • Accuracy and set size: The negative association between accuracy and set size is stronger for LAC than APS and varies by regime.The relationship is shown across Figures 9–13.
  • Regime comparison: AP shows the steepest negative accuracy–set-size trend, A is milder, and NAP is typically weakest.These regime differences are reported for the accuracy–set-size relationship.
  • Reasoning-mode comparison: For APS, CoT has a more negative average slope than NoCoT, whereas LAC slopes are similarly negative across modes.LAC still exhibits small condition-specific shifts.

I.2 Performance across benchmark variants

Across benchmark variants, explicit abstention consistently increases uncertainty, while perturbations and prompting produce smaller or heterogeneous effects. Larger prediction sets generally accompany lower accuracy, with important model- and dataset-specific exceptions.

  • Abstention and perturbation: Prediction sets increase under abstention-aware conditions, with AP producing the largest inflation and NAP a substantially smaller effect.This pattern appears across both datasets and supports abstention as the primary driver of uncertainty amplification.
  • Abstention and perturbation: Accuracy remains stable or mildly degrades under most benchmark variants, with AP causing the greatest drop and MedQA more sensitive than AMBOSS.The AP–NA contrast is the most negative, although the degradation remains modest overall.
  • Zero shot vs Few shot: Few-shot prompting has negligible overall effects on abstention and uncertainty, while accuracy gains and set-size changes remain small and heterogeneous.On MedQA, the largest accuracy boost appears under AP with CoT; smaller models show more variable effects.
  • Performance across models: Larger LAC and APS sets generally align with lower accuracy, although GPT-4o variants maintain near-neutral or non-negative slopes under CoT–few-shot.GPT-4.1 and small–mid instruction models more consistently show negative coupling, while few-shot improves calibration more reliably than CoT for smaller models.
  • Performance across models: CoT often widens prediction sets without consistent accuracy benefits, especially on MedQA, whereas few-shot more reliably shifts smaller models toward improved calibration.The negative accuracy–set-size coupling is stronger on MedQA, particularly for LAC, and APS can become more negative under CoT.
  • Reasoning effects: Thinking mode has negligible overall impact, with small accuracy gains and tighter LAC sets but reduced abstention rates across both datasets.MedQA–NAP/AP shows higher APS, indicating a trade-off between coverage and caution in noisier regimes.
Loading 2601.12471v2…