Source-linked AI summary
Refusal Is Not Robustness: Auditing Confident Fabrication in Large Language Models on a Provably Uninformative Clinical Pain Speech Transcript
Sagnik De, Sreenija Pavuluri
TL;DR
The paper addresses whether abstention benchmarks can distinguish justified refusal from unsupported prediction when the answer is unknowable from a transcript. It evaluates language models on a speech corpus with empirically signal-free transcripts and finds that cooperative abstention can conceal prompt-fragile behavior and confident fabrication under pressure.
Problem
Existing hallucination and abstention benchmarks rarely establish that the correct answer was unavailable, limiting clean evaluation of appropriate refusal.
Method
The study compares seven language models on TAME Pain using provably signal-free transcripts, positive-control pain statements, cooperative prompts, and authority-framed pressure prompts.
Results
Confident fabrication is consistently exhibited only by Gemini 2.5 Flash and Llama 3.1 8B under pressure, while cooperative prompting makes six models appear exemplary.
Takeaways & Limitations
Trustworthiness evaluations should vary pressure-prompt phrasing and score confidence in forced answers rather than abstention alone.
Takeaways & Limitations
The study uses a single dataset without claimed cross-corpus transfer, and its small, demographically skewed cohort makes bias analysis exploratory rather than confirmatory.
Abstract
from arXiv · showhide
Hallucination and abstention benchmarks rarely establish that a model could not have known the correct answer, making it difficult to distinguish appropriate abstention from an unsupported prediction. Seven large language models were evaluated on the TAME Pain speech corpus. Participants read phonetically balanced Harvard Sentences while one hand was immersed in cold or warm water and reported pain only during periodic pain statements. This protocol generated 5,750 no signal Harvard Sentence utterances whose transcripts contained no lexical pain information and 1,294 signal pain statement utterances in which the pain rating was explicitly spoken. In the no signal arm, pain was recoverable from acoustic features (AUC 0.622, 95% CI 0.553 to 0.662), whereas transcript based prediction was near chance (AUC 0.489, 95% CI 0.418 to 0.504). Because automatic speech recognition removes the acoustic pain cues, any pain score inferred solely from the transcript is unsupported by the available evidence. Under cooperative prompting, six models abstained on nearly all no signal transcripts, correctly extracted spoken pain ratings in the positive control task with accuracies ranging from 0.939 to 1.00, and maintained an expected calibration error of at most 0.100. Under authority framed prompts, abstention became prompt dependent, with the same model ranging from 0.18 to 1.00 across equivalent prompt phrasings. Most models produced low confidence estimates when forced to answer, whereas Gemini 2.5 Flash and Llama 3.1 8B consistently generated confident pain scores with confident fabrication rates of 0.53 and 0.76, compared with at most 0.15 for all other models. No significant demographic effects were observed in forced responses, with all $p$ values greater than or equal to 0.20.
1 Introduction
The paper argues that transcript-only clinical models need to recognize when pain cannot be inferred from available evidence. It introduces a provably uninformative speech setting to test abstention, pressure robustness, and confident fabrication.
- ASR discards acoustic cues such as tone, effort, tremor, and vocal catch that may carry clinically relevant pain information.
- Existing medical benchmarks rarely test whether models recognize when an answer is unavailable, allowing unsupported pain scores to go undetected.
- The TAME Pain protocol uses fixed, pain-neutral Harvard Sentences and spoken pain statements, making transcript-level absence of pain information provable.
- A matched acoustic-versus-transcript control establishes whether pain is recoverable from the voice while testing the transcript representation on the same utterances.
- Under cooperative prompting, almost every model abstains correctly, but pressure reveals prompt-fragile abstention and robust confident fabrication by Gemini 2.5 Flash and Llama 3.1 8B.
- The study distinguishes itself through a provably signal-free testbed, two tailored reliability metrics, and pressure-robustness evaluation.
2 Related Work
The related work spans medical question answering, abstention and calibration, prompt-induced failures, and automated pain assessment. The paper combines these strands in a clinical-speech setting where transcript uninformativeness is empirically established.
- Medical benchmarks mainly evaluate text-in, text-out knowledge tasks rather than reliability over speech transcripts and self-reported pain.
- Selective-prediction research studies abstention and calibration, but typically assumes that the answer is knowable rather than proving when abstention is correct.
- Prior work documents answer shifts under social pressure and authority framing; this paper measures that sensitivity where abstention is empirically defined as correct.
- Automated pain assessment commonly uses acoustic or facial classifiers, whereas this work audits language-model reliability after speech has become a transcript.
3 Dataset
TAME Pain provides a clinical-speech corpus with separate no-signal and signal arms. The design makes transcript abstention testable while preserving a positive control for extracting explicitly spoken pain ratings.
- The corpus contains 7,044 utterances from 51 participants recorded during a cold-pressor task, with audio-quality annotations for every clip.
- The no-signal arm has 5,750 pain-neutral Harvard-sentence utterances, while the signal arm has 1,294 pain statements containing spoken ratings.
- Level one accounts for 46.9% of utterances, establishing the pain-level base rate that transcript-only guessing cannot legitimately surpass.
- A further 542 utterances are degenerate because of audio cut-out, missing sentences, or missing ratings, and abstention is correct for them as well.
- The cohort is small, young, and demographically skewed, making it a reliability testbed but limiting generalization and bias analysis.
4 A Provably Signal-Free Testbed
The paper tests whether pain information exists in no-signal transcripts by comparing acoustic and text representations under the same classifier and validation procedure. Acoustics carry predictive information, whereas transcripts are near chance, establishing a signal-free basis for evaluating abstention.
- The matched control uses 15 acoustic descriptors and a 795-word bag-of-words representation with identical logistic regression, cross-validation, and evaluation metrics.
- The transcript result is indistinguishable from chance and its confidence interval is disjoint from the acoustic interval.
- Because ASR preserves words but removes sound, transcript-only models lose the pain-bearing channel in the no-signal arm.
- Mean word-error-rate is 0.078, with a median of 0.000, supporting faithful transcription rather than transcription failure as the explanation for downstream errors.
5 Benchmark Design
The benchmark audits an end-to-end pipeline from speech waveform through transcription, prompting, language-model inference, and structured output. It separately evaluates no-signal and signal arms while tracking transcription errors and model behavior under different prompting conditions.
- Pipeline: The pipeline converts speech waveforms into Whisper transcripts before frozen prompts produce structured language-model outputs.Each prompt requests a pain score or null, a ten-level distribution, an abstain flag, confidence, and a cited transcript span.
- Evaluation: The evaluation runs end to end while localizing transcription errors separately with word-error-rate analysis.
- Outputs: The structured output records both predictions and uncertainty through scores, distributions, abstention, confidence, and transcript citations.
6 Performance Metrics
The benchmark combines standard measures with two metrics tailored to no-signal confidence and signal-arm reliability. Its primary pressure outcome is abstention on no-signal utterances, while controls assess extraction, calibration, and signal detectability.
- Standard metrics: The abstention rate is the fraction of no-signal utterances declined for scoring, with higher values treated as better and the rate carrying the main result.
- Standard metrics: Leakage-Control Accuracy extracts the spoken number on the signal arm, while Expected Calibration Error is computed on that arm.
- Standard metrics: AUC measures signal existence, and word-error-rate measures transcription quality.
- Tailored metrics: The Illusory Confidence Score evaluates no-signal predictions against a 2.61-bit population-prior entropy reference, with lower scores better.It combines confidence with divergence from the prior distribution and averages over non-abstaining predictions.
- Tailored metrics: The Reliability Dissociation Index evaluates signal-arm invariance and sensitivity using a harmonic mean, with higher values better.A constant predictor scores zero, and the principal pressure result relies on abstention rate rather than this proposed metric.
7 Models
The audit covers seven production language models from five providers and evaluates them with a shared verbatim prompt suite. Pressure prompts retain the same task instruction and JSON schema while varying only their framing.
- Models: Seven production language models across five providers are included in the audit.The models span Claude, GPT, Gemini, DeepSeek, and Llama families.
- Prompting: The models are evaluated with a shared prompt suite whose pressure variants differ only in the framing preceding the transcript.
- Prompting: All prompts use one task instruction and one JSON schema for the evaluation.
8 Results
Cooperative prompting makes nearly all models abstain appropriately on no-signal transcripts, but pressure reveals prompt-fragile abstention and robust confident fabrication in two models. Calibration and demographic probes indicate targeted, rather than general, unreliability.
- 8.1 Cooperative Prompting Makes Almost All Models Look Honest: Six of seven models abstain on nearly every no-signal transcript, while Llama-8B reaches an abstention rate of 0.907.Only Llama-8B shows cooperative illusory confidence, at 0.029.
- 8.1 Cooperative Prompting Makes Almost All Models Look Honest: All seven models extract spoken pain ratings with accuracies from 93.9% to 100% and maintain Expected Calibration Error of at most 0.100 on the signal arm.This supports distinguishing transcript non-information from inability to read spoken ratings.
- 8.2 Cooperative Honesty Is Prompt-Dependent for the Weakest Model: Llama-8B’s illusory confidence rises from 0.025 to 0.159 as cooperative prompts remove explicit abstention permission, while abstention falls from 0.91 to 0.58.The zero-shot prompt produces roughly sixfold higher fabrication than the abstention-licensed prompt.
- 8.3 Under Pressure, Abstention Is Prompt-Fragile and Confident Fabrication Isolates Two Models: Authority abstention ranges from 0.18 to 1.00 across equivalent phrasings for GPT-5.2, demonstrating that single-prompt pressure measurements can be unreliable.The pressure design averages three phrasings per framing to separate wording artifacts from genuine effects.
- 8.3 Under Pressure, Abstention Is Prompt-Fragile and Confident Fabrication Isolates Two Models: Confident-fabrication rates are 0.53 for Gemini-Flash and 0.76 for Llama-8B, compared with at most 0.15 for every other model.Only these two models robustly invent confident pain scores across all three authority phrasings; other models generally hedge at low confidence.
- 8.4 Demographic Bias in the Forced Scores Is Not Detected: No significant White–Black persona differences are detected among adequately sampled models, with all p ≥ 0.20, but the cohort is underpowered.The study describes this as absence of evidence rather than evidence of absence.
9 Discussion and Limitations
The results show that cooperative abstention can conceal failures revealed by pressure, while confidence-aware and prompt-varied evaluation better exposes fabricated clinical scores. These conclusions are bounded by dataset, cohort, measurement, model-coverage, pressure-taxonomy, and acoustic-control limitations.
- Discussion: Six of seven models appeared exemplary when asked politely, but pressure exposed prompt-fragile and coarse abstention behavior.The abstention flag alone does not distinguish low-confidence hedging from confident fabrication.
- Discussion: Trustworthiness evaluations should test several prompt phrasings and score forced-answer confidence alongside abstention.The recommendation relies on a transcript whose lack of pain signal was empirically established.
- Limitations: The study uses a single dataset and makes no cross-corpus transfer claim.Its participant cohort comprises 51 young, demographically skewed participants, making the bias analysis exploratory rather than confirmatory.
- Limitations: Confidence reports are self-reported rather than derived from token logits, and the audit covers seven models without exhaustively surveying frontier systems or newer releases.These choices may limit how directly reported uncertainty represents internal uncertainty and how broadly the findings generalize across models.
- Limitations: The three pressure framings do not exhaust adversarial pressure, and the acoustic control may partly decode cold condition rather than pain specifically.The latter limitation does not affect the reported audio-versus-text contrast.
10 Conclusion
On a clinical speech benchmark with empirically established absence of transcript pain information, abstention was fragile under adversarial pressure. Multiple prompt phrasings and forced-answer confidence are therefore needed for a more complete assessment.
- Conclusion: Abstention varied substantially with prompt wording, making single-prompt evaluation incomplete and potentially misleading.
- Conclusion: Abstention rates alone cannot distinguish low-confidence hedging from confident fabrication.