PaperlayerView exact revision on arXiv

Source-linked AI summary

Being judged through the record: A Rosenhan-inspired test of psychiatric label sensitivity in large language models

Shaked Rosenblum, Teddy Lazebnik, Ziv Ben-Zion

arXiv:2610.35326v1cs.CY

TL;DR

The paper asks whether psychiatric history changes LLM disposition recommendations when current evidence is identical, a question relevant to clinical decision-making. In a controlled factorial vignette experiment across five models, adding a prior schizophrenia diagnosis increased hospitalization recommendations most when evidence supported discharge, while confidence also increased.

  • Problem

    It was unknown whether documenting a prior psychiatric diagnosis changes an LLM’s disposition recommendation when the patient’s current presentation is held constant.

    Source: Introduction · p. 2, Discussion · p. 14

  • Method

    The authors factorially varied prior schizophrenia diagnosis and current evidence in one synthetic vignette, querying five LLMs 1,000 times per version.

    Source: Methods · p. 4

  • Results

    With voices ceased, hospitalization recommendations rose from 4.7% to 22.3% with a prior diagnosis, while reference concordance fell from 95.3% to 77.7%.

    Source: Discussion · pp. 14, 16

  • Takeaways & Limitations

    Diagnostic history altered responses to an otherwise identical presentation, but the findings do not establish that LLMs possess a human psychological bias.

    Source: Discussion · p. 14

    The four vignette versions and single standardized prompt cannot establish whether the pattern occurs with other diagnoses, presentations, prompts, or decisions.

    Source: Discussion · p. 16

Abstract

from arXiv · show

The integration of large language models (LLMs) into clinical decision-making raises unresolved questions regarding diagnostic bias. We investigate whether a documented psychiatric history alters LLM recommendations when the current clinical evidence is identical. Using a counterfactual experiment inspired by Rosenhan's 1973 field study, we factorially varied diagnostic-history condition (no prior diagnosis stated vs. prior schizophrenia diagnosis) and current-evidence condition (voices ceased vs. voices persisted) within a clinical vignette. Across five LLMs (GPT-5.6 Sol, Claude Sonnet 5, Gemini 3.6 Flash, Grok 4.6, and Mistral Large 2512), we generated 20,000 model responses across independent queries. When evidence supported discharge, adding a prior schizophrenia diagnosis increased recommendations for continued hospitalization nearly fivefold, from 4.7% to 22.3% (risk difference, 17.5 percentage points; 95% CI, 16.2-18.8). This shift occurred across all five models (risk-difference range, 10.4-34.4 percentage points). In the same condition, reference concordance decreased from 95.3% to 77.7%, while stated decision confidence increased. Across all model-by-evidence contrasts, confidence increases corresponded to Cohen's d values of 0.35-1.09. Signal-detection analyses showed that the prior diagnosis shifted the decision criterion toward hospitalization across all five models and reduced discrimination between the two current-evidence conditions in three of them. When current evidence supported continued hospitalization, the label increased the recommendation for hospitalization from 92.6% to 97.0%.

Introduction

Psychiatric histories may shape how current evidence is interpreted, making disposition decisions a useful test of label sensitivity in LLMs. The study asks whether a prior diagnosis changes recommendations when the current presentation is held constant.

Introduction

  • Psychiatric disposition is especially susceptible to framing because histories can influence interpretation of otherwise identical current evidence.The introduction connects this concern to anchoring, persistent diagnostic expectations, and diagnostic labels that can make difficulties appear more persistent and less controllable.Source: Introduction · p. 2
  • Diagnostic overshadowing describes attributing new or independent symptoms to an existing diagnosis, while anchoring may limit updating from that diagnostic frame.The paper distinguishes these related ideas when motivating its test of whether a prior diagnosis influences judgments despite a constant current presentation.Source: Introduction · p. 2
  • The study uses Rosenhan’s scenario as a recognizable test case, not as a protocol to reproduce or settled empirical evidence.The authors note methodological and evidentiary challenges to Rosenhan’s report and focus instead on its question about whether present evidence can revise judgment.Source: Introduction · p. 2
  • The paper defines label-sensitive inference as a measurable model-output shift caused by a diagnostic label, distinct from human psychological mechanisms.Counterfactual input manipulation and repeated queries allow the label’s effect to be estimated while other clinical information remains constant.Source: Introduction · p. 3
  • Existing LLM bias audits have focused mainly on sociodemographic attributes, while mental-health studies have also examined stigma and other clinical tasks.The introduction situates the disposition question alongside prior work on performance disparities, stigmatizing attitudes, screening, diagnosis, counseling, and treatment support.Source: Introduction · p. 3
  • The authors identify a gap: whether documenting a prior psychiatric diagnosis changes LLM disposition recommendations when current presentation is held constant.They use a 2 × 2 vignette experiment across five LLMs, varying prior schizophrenia diagnosis and whether voices ceased or persisted.Source: Introduction · p. 3

Methods

The methods use a controlled factorial evaluation to isolate the effect of psychiatric history from current evidence. Five LLMs repeatedly assessed four versions of one synthetic vignette, with disposition and confidence recorded for analysis.

Methods

  • Five LLMs each completed 1,000 calls in four factorial conditions, yielding 20,000 outputs for disposition and confidence analysis.The workflow compared responses across psychiatric-background conditions and supplemented the disposition and confidence analyses with risk-difference, odds-ratio, and signal-detection analyses.Source: Methods · p. 4, Figure 1 · p. 4

Study design and conceptual framework

  • The design crossed prior schizophrenia diagnosis stated versus unstated with voices ceased versus persisted, holding other information constant.It estimated label sensitivity within one reconstructed scenario rather than its prevalence across psychiatric presentations.Source: Study design and conceptual framework · p. 4

Vignette development

  • The vignette reconstructed a case inspired by Rosenhan’s account, but was neither a literal pseudopatient case nor a replication of the field experiment.It described a 39-year-old White man reporting unfamiliar voices, with additional relationship, ward-behavior, well-being, and discharge-request details.Source: Vignette development · p. 5
  • The psychiatric-background manipulation changed whether the record stated a prior schizophrenia diagnosis; the current-evidence manipulation changed whether voices ceased or persisted intermittently.The versions held all other text constant, and investigators prespecified discharge for ceased voices and continued hospitalization for persistent voices as reference categories.Source: Vignette development · p. 5

Large language models

  • The study evaluated five contemporary general-purpose LLMs selected to cover commercial systems and assess effects across model families and providers.The models were GPT-5.6 Sol, Claude Sonnet 5, Gemini 3.6 Flash, Grok 4.6, and Mistral Large 2512.Source: Large language models · p. 5

Outcomes and processing

  • Disposition was the primary outcome and confidence the secondary outcome; the 20,000 responses were complete, with no exclusions.Reference concordance compared each recommendation with the investigator-prespecified disposition for that evidence condition.Source: Outcomes and processing · p. 6

Statistical analysis

  • Analyses compared hospitalization recommendations and confidence across diagnosis conditions, using pooled and model-specific regression, intervals, and corrected tests.Signal-detection measures treated hospitalization recommendations as hits when voices persisted and false alarms when voices had ceased; repeated calls were draws from fixed model-prompt outputs, not independent patients or cases.Source: Statistical analysis · pp. 6–7

Ethics, transparency, and reproducibility

  • The experiment used synthetic material without patients, medical records, or identifiable patient information.The authors released the prompt, vignette versions, processing and analysis code, cell-level counts, and deidentified outputs where provider terms permitted.Source: Ethics, transparency, and… · p. 7

Results

Across five models, a prior schizophrenia diagnosis shifted recommendations toward hospitalization, most sharply when the voices had ceased and discharge was the prespecified reference. Confidence also rose, while exploratory analyses found criterion shifts in every model and sensitivity declines in three.

Analytical sample

  • The analysis included 20,000 classified outputs, and the primary comparison assessed hospitalization recommendations by label within each model and evidence condition.Each of the five models contributed 1,000 responses to each of four conditions.Source: Analytical sample · p. 7

Prior schizophrenia diagnosis shifted recommendations toward hospitalization.

  • With voices ceased, hospitalization recommendations rose from 4.7% to 22.3% after a prior schizophrenia diagnosis was stated, a 17.5-percentage-point increase.The pooled label effect was positive across all five models and both evidence conditions; the 95% CI for the discharge-condition risk difference was 16.2–18.8 percentage points.Source: Prior schizophrenia diagnosis shifted… · pp. 7–8
  • When voices persisted, hospitalization recommendations rose from 92.6% without the label to 97.0% with it, a 4.44-percentage-point increase.Without the label, hospitalization recommendations were already 89.4%–94.6%, leaving comparatively little scope for further increases.Source: Prior schizophrenia diagnosis shifted… · p. 8
  • In the discharge-supporting condition, reference concordance fell from 95.3% to 77.7% as hospitalization recommendations increased; in the hospitalization-supporting condition, concordance rose from 92.6% to 97%.The prespecified references were not treated as definitive clinical ground truth, so the authors call this reference concordance rather than clinical accuracy.Source: Label-associated change in reference… · p. 8

Confidence increased even when reference concordance declined.

  • Confidence increased in all ten model-by-evidence contrasts, with Cohen’s d values of 0.35–1.09, even as discharge-condition reference concordance declined.For GPT-5.6 Sol in the discharge-supporting condition, mean confidence increased from 87.13 to 91.80 while concordance declined across models.Source: Confidence increased even when… · p. 9

Exploratory signal-detection analysis

  • The diagnosis shifted decision criterion toward hospitalization in every model, while sensitivity declined for Claude, Grok, and Mistral but changed little for GPT and Gemini.Criterion changes ranged from −0.49 to −1.04; sensitivity changes were −0.60, −0.95, and −0.97 for Claude, Grok, and Mistral, respectively, versus −0.13 for GPT and +0.03 for Gemini. These exploratory indices came from two evidence states in one reconstructed case and do not estimate discrimination across the broader clinical domain.Source: Exploratory signal-detection analysis · pp. 9–10, Figure 2 · p. 10

Discussion

A prior schizophrenia diagnosis shifted disposition recommendations despite identical current clinical information, with the largest relative effect when voices had ceased. The findings describe label sensitivity, but do not establish its mechanism or clinical appropriateness.

Discussion

  • When voices had ceased, continued-hospitalization recommendations rose from 4.7% without a prior diagnosis to 22.3% with one, across all five models despite unchanged current information.This was the largest absolute shift across the two evidence conditions.Source: Discussion · p. 14
  • The label effect was approximately 2.2 times larger when voices had ceased, but the 92.6% baseline hospitalization rate when voices persisted limited further absolute increase.The interaction challenges a constant-additive account but cannot identify the process that generated it; odds ratios at extreme event rates may be sensitive to small changes in discordant responses.Source: Discussion · p. 15
  • Adding the diagnosis shifted the decision criterion toward hospitalization in every model, but these model-specific patterns are exploratory because they use two versions of one scenario.Lower criterion values indicate a greater tendency to recommend hospitalization.Source: Discussion · p. 15
  • Across all ten model-by-evidence contrasts, diagnosis increased stated confidence; in the discharge-supporting condition, reference concordance fell from 95.3% to 77.7%.Stated confidence captures how strongly a recommendation was expressed, not a calibrated probability or evidence that a model experiences certainty.Source: Discussion · pp. 15–16
  • The experiment cannot determine why the pattern occurred, and input manipulation of proprietary models cannot attribute it to RLHF or another particular mechanism.Training associations and post-training preferences are possibilities, not findings established by this experiment.Source: Discussion · p. 16
  • Label sensitivity describes an output shift, not a shared human cognitive process or automatic error; diagnostic history may still inform risk and illness course.The key concern is whether historical information remains revisable as person-specific evidence becomes available.Source: Discussion · pp. 15–16
  • The study used general-purpose commercial models and four versions of one synthetic vignette, so it does not establish safety in practice or generalization to other diagnoses, presentations, prompts, or decisions.Repeated calls represented variability under fixed conditions, not independent patients or clinical cases; reference dispositions were prespecified categories, not definitive clinical ground truth.Source: Discussion · pp. 16–17
  • Psychiatric AI evaluations should test whether systems revise the meaning and weight of history as symptoms and circumstances change, using richer longitudinal cases, outcome data, and expert judgment.The aim is responsiveness to change, not invariance to history.Source: Discussion · p. 17

Declarations

The experiment used synthetic clinical vignettes without patients or identifiable information, and received no specific grant funding.

Ethics statement

  • The experiment used entirely synthetic clinical vignettes and involved no patients, medical records, or identifiable patient information.Source: Ethics statement · p. 19

Funding

  • The research received no specific grant from any funding agency.Source: Funding · p. 19
Loading 2610.35326v1…