Language-model ratings of depression reflect the rater more than the patient

• Early evaluations reported clinically acceptable score error [9], intraclass correlations near 0.75 [10,11], and recovery of self-reported depression severity [12,13].Source: Introduction · p. 1
• LLM ratings promise scalable, inexpensive, repeatable assessment, but plausible choices about prompts, views, answer formats, and scoring can define different raters.Source: Introduction · p. 1
• The framework separates how well a rater orders people, how many it flags, and whether another defensible rater selects the same people.Source: Introduction · p. 2
• Rater choice outweighed patient differences; accurate rankings still yielded divergent decisions, and recalibration improved decisions without resolving individual disagreement.Source: Introduction · p. 2
• The study crossed 11 open models and multiple wording, conversation-view, answer-reading, and scoring choices to create 880 raters for 189 interviews.The criterion was the eight-item Patient Health Questionnaire (PHQ-8); the locked procedure was also applied to 86 new interviews.Source: Introduction · p. 2

The same interview, opposite verdicts

• 40% of participants received opposite screening decisions from two randomly drawn raters with AUC ≥0.70.Across all raters, disagreement was 44%; even the stricter set meeting AUC, sensitivity, and specificity thresholds disagreed for 53% of participants.Source: The same interview, opposite verdicts · p. 2, Figure 1 · p. 3

Accuracy does not decide who is flagged

• 0–100% of participants were flagged across raters, while AUC for PHQ-8 ≥10 ranged from 0.34 to 0.88 (median 0.75).Rater offset, or mean over-rating of PHQ-8, governed share flagged along a normal-ogive curve (R2 = 0.91).Source: Accuracy does not decide who is flagged · p. 2, Figure 2 · p. 4

The rater outweighs the patient

• 30.0% of summed-symptom score variance came from model choice, versus 10.5% from stable participant differences.Averaging 11 models, five wordings, and three formats raised dependability only to 0.53, or 0.55 with both conversation views.Source: The rater outweighs the patient · pp. 2–3, Figure 3 · p. 5

How many versus who

• At the screening threshold, rate differences drove 38 of 40 percentage points of disagreement; equal-capacity raters still chose differently for 18% of participants.With 40 labelled participants, prevalence matching raised accuracy from 61% to 75% and halved disagreement to 20%; residual disagreement persisted with other calibration conditions.Source: How many versus who · pp. 5–6

Small choices move the threshold

• 27 percentage points separated median flag rates when raters scored only explicitly reported symptoms; reversing answer letters shifted rates by 21 points.Larger models ranked better (Spearman correlation 0.82) without better calibration or specificity; deployment changes also altered decisions.Source: Small choices move the threshold · p. 3, Figure 4 · p. 6

What the raters read

• Below the screening threshold, PTSD symptoms and talkativeness predicted excess flags; interviewer speech alone yielded median model AUC 0.59.Because PHQ-8 and PTSD severity correlated strongly, the authors could not conclude that ratings separated the diagnoses.Source: What the raters read · p. 4, Figure 5 · p. 7

The regularities held in new interviews

• In 86 new interviews, model choice explained 26.2% of score variance versus 13.0% for participant differences, and qualified raters again disagreed on 40%.Five of six evaluable pre-registered predictions were supported in confirmation.Source: The regularities held in new interviews · pp. 7–8

Checks that need no labels

• Internal consistency was unrelated to validity across raters, whereas agreement across rewordings tracked validity; label-free selection found none better than a sensible default.The rewording-agreement rule beat the model median in 9 of 11 confirmation models but did not outperform a fixed neutral configuration.Source: Checks that need no labels · pp. 4–5, Figure 6 · p. 8

Discussion

• Model choice explained about three times more summed-symptom score variation than stable participant differences, making the rater configuration consequential to judgments about people.The discussion frames the configuration as a model, instruction, and answer-format choice that can become part of how the instrument judges a person.Source: Discussion · p. 8
• With equal-capacity selection, different configurations still chose different people for about one participant in five after calibration repaired much of the disagreement.Recalibration changes how many people are flagged, not the configuration’s ranking of them.Source: Discussion · p. 8, Figure 7 · p. 9
• Calibration and pooling do not by themselves settle clinical decisions: shared departures from PHQ-8 remain, and thresholds trade missed positive screens against unnecessary flags.The clinical objective should be specified alongside the calibration rule.Source: Discussion · p. 9
• Ratings tracked PTSD as closely as PHQ-8, and trauma symptoms and speech predicted additional flags, leaving specificity unresolved where depression and PTSD co-occur.The discussion highlights trauma-exposed and veteran populations, and clinician speech in ambient clinical recordings, as practical concerns.Source: Discussion · p. 9
• Report decision ranges across defensible configurations, fix and test a neutral setup, calibrate locally, measure residual disagreement, test neighboring conditions, and freeze the full stack.The discussion warns that changing model, precision, inference software, instructions, or transcription can change decisions and complicate longitudinal interpretation.Source: Discussion · p. 10
• The evidence is bounded by open models up to 21 billion parameters, PHQ-8 screening rather than diagnosis, no clinician ratings, and one laboratory’s English-language protocol.Confirmation was small and conducted in a related corpus rather than a different clinical setting; several analyses were exploratory.Source: Discussion · p. 10

Online Methods

• The study used 189 DAIC-WOZ discovery interviews and 86 confirmation interviews, with PHQ-8 ≥10 as the registered positive-screen criterion.Discovery transcripts were manual; confirmation interviews used an autonomous interviewer and automatic speech recognition.Source: Online Methods · pp. 10–11
• Eleven open-weight instruction-tuned models from six developers were selected because their weights were already available on the workstation.The models were run using the stated hardware and inference stacks; the reasoning model’s reasoning block was left empty.Source: Online Methods · p. 11
• Models scored response-option probabilities after a fixed answer cue without sampling, using expected-value or most-likely-option scores.For direct totals, a frozen-code termination-factor defect affected scoring; re-scoring 25 participants changed no screening decisions, and item sums were unaffected.Source: Online Methods · p. 11
• 880 raters combined 11 models with transcript views, prompt wordings, answer formats and scoring rules across item sums or direct totals.A positive screen was a score of 10 or more; interviewer-only speech and ASR transcripts served as additional checks.Source: Online Methods · p. 11
• The estimands covered AUC, score correlation, error and bias, screening rates, participant-level disagreement, variance components, and score reliability and validity.Pre-registered remedies included averaging across raters or models, isotonic calibration using 20 or 40 labelled participants, and prediction-powered prevalence estimation.Source: Online Methods · p. 12
• Exploratory analyses tested configuration effects, recalibration and deferral, while the pre-registered analysis was locked before one-time confirmation-set analysis.The analysis plan specified bootstrap and split procedures; split-to-split ranges describe repeated-split variation, not confidence intervals.Source: Online Methods · pp. 12–13

Data availability

• DAIC-WOZ and E-DAIC are available under data-use agreements, and only aggregate results are distributed with the code.All processing ran locally; no transcript text was sent to an external service.Source: Data availability · p. 13

Extended Data

• The regularities table reports discovery findings checked in new interviews and notes that shared participants and raters make the regularities non-independent tests.It also states that digit-answer item sums over-rated in 8 of 11 models in both sets.Source: Table 1 · p. 15
• Decision studies plot generalizability and dependability of expected item sums against the number of models and the wording and format averaged.Both views of the conversation are averaged; the dashed reference line is 0.80.Source: Figure 3 · p. 16
• The averaging and labels figure compares single-rater and averaged-score AUCs, screening shares before and after calibration, and prevalence-interval widths and coverage.Calibration uses 20 or 40 labelled participants; intervals compare prediction-powered estimates with label-only estimates.Source: Figure 4 · p. 17
• The discovery-confirmation figure displays AUC, screening shares, decision overlap and contestation, and variance shares for participant and rater-related factors.Discovery and confirmation are distinguished by dark-grey and white marks, respectively.Source: Figure 5 · p. 17
• The all-raters figure orders discovery raters by AUC and shows their screening shares and configuration choices.AUC intervals are bootstrap intervals, with chance and 0.70 reference lines.Source: Figure 6 · p. 18
• The disagreement figure compares expected discordance as the number of sampled accurate raters grows and pairwise disagreement across progressively stricter rater sets.It distinguishes discovery from confirmation and identifies these analyses as exploratory.Source: Figure 7 · p. 18
• The recalibration figure tracks held-out disagreement, prevalence error, and disagreement across rater sets as labelled-sample size and admission rules vary.Its comparisons include prevalence matching, retaining labelled positives, and logistic regression.Source: Figure 8 · p. 19
• The deferral figure compares accuracy and sensitivity at matched coverage for consensus decisions and individual raters in discovery and confirmation.Consensus thresholds set aside participants with less than 60%, 70%, 80%, or 90% agreement; the 80% rule is marked.Source: Figure 9 · p. 20

1.1 Prompts

• Prompts placed a transcript before the task instruction, response format, and PHQ-8 item; direct-total prompts requested a score from 0 to 24.The supplementary table lists prompt texts; the five wordings include instructions that can change assessment policy as well as wording.Source: §1.1 · p. 21, Table 1 · p. 21

1.2 Models

• The models used specified weights and inference implementations, with the reasoning model given an empty reasoning block and gpt-oss-20b set to low reasoning effort.A numerical-precision-check model was not part of the multiverse.Source: §1.2 · pp. 21–22, Table 2 · p. 22

1.3 Reading the answer

• Option probabilities summed tokenisation variants and were renormalised; expected-value and mode rules then produced scores, with mode not necessarily matching generated answers.Direct-total scoring had a frozen termination-rule defect; the supplementary table compares that rule with re-scoring and reports decision flips.Source: §1.3 · p. 22, Table 3 · p. 22

1.4 Tree-structured evaluation

• Tree-structured evaluation encoded each transcript prefix once and evaluated continuations with attention restricted to their prompt-tree ancestors.Checks against one-prompt-at-a-time evaluation measured numerical agreement on one participant and sampled continuations, not a guarantee for every prompt.Source: §1.4 · p. 23

1.5 Generalizability analysis

• Variance components were estimated from mean squares for all 31 effects in the fully crossed person-by-model-by-template-by-format-by-view design.Negative estimates were retained while solving for lower-order components, then set to zero for reporting; the highest-order interaction is confounded with residual error.Source: §1.5 · p. 23

1.6 Consistency indices

• Expected item sums yield an expected total and sampling variance, while consistency indices quantify agreement across independent runs and across templates.Cronbach’s α was also computed across the eight expected item scores.Source: §1.6 · p. 23

1.7 Remedies

• For calibration, isotonic regression mapped each configuration’s scores to PHQ-8 totals using n=20 or 40 randomly drawn participants, then applied the map to remaining participants to evaluate decisions and implied prevalence.This was repeated for 200 draws; finite-cohort prevalence estimates were also computed for coverage assessment.Source: §1.7 · p. 23

1.8 Decision analyses (exploratory)

• At a fixed threshold, pairwise disagreement is decomposed into the minimum forced by unequal positive rates and excess disagreement, without assigning causes.At equal capacity, each rater selects the same number of highest-scoring participants.Source: §1.8 · p. 23
• Recalibration rules set each rater’s score cut-off using labelled participants, then assess held-out decisions across repeated splits.The rules include mean-offset subtraction, linear and logistic regression, prevalence matching, and a threshold retaining at least 80% of labelled positives.Source: §1.8 · p. 24
• The clinically defensible core comprises 44 discovery and 22 confirmation raters, selected without outcome labels; disagreement is also described by self-reported PHQ-8 band.The core uses specified wording, digit answers, expected values, symptom sums, and every view; the band analysis does not identify which decision is clinically right.Source: §1.8 · p. 24
• Deferral analyses compare majority decisions and single-rater decisions at matched coverage, but do not evaluate clinical policy because review and missed-case costs are undefined.The supplementary table reports coverage and retained positive and negative counts.Source: §1.8 · p. 24, Table 7 · p. 26
• Participant-resampling intervals for three headline disagreement estimates hold the rater sets fixed, so they describe participant sampling rather than configuration sampling.Transfer applies discovery calibrations unchanged to the same configurations in a related confirmation corpus with different transcript and interviewer procedures.Source: §1.8 · p. 24
• Supplementary comparisons cover restrictions of the rater grid and one-choice-at-a-time configuration pairs, with table captions specifying their decision measures.The captions also describe 40-label recalibration outcomes and disagreement by PHQ-8 band.Source: Table 5 · p. 25, Table 4 · p. 25, Table 6 · p. 25

2 Supplementary Results

• Supplementary Table 9 reports discovery-set results by model across all 80 configurations, including median sensitivity, specificity, bias, and option mass.The caption defines sensitivity and specificity for PHQ-8 ≥10.Source: Table 9 · p. 27
• Supplementary Table 10 presents variance components for the expected item sum across person, model, template, format, and view.Source: Table 10 · p. 27
• For one view, averaging all models, templates, and formats gives Eρ2 = 0.82 and Φ = 0.53; a single configuration gives Φ = 0.11.Source: Table 11 · p. 28
• Supplementary Table 12 lists directional predictions specified before analysis alongside their outcomes.Source: Table 12 · p. 28

3 Labels and transcripts

• A positive screen was registered as PHQ-8 ≥10, differing from the corpus binary flag for one discovery and five confirmation participants.Item scores were available for 142 discovery participants and complete for 141; excluding six transcripts containing literal ‘nan’ tokens changed median AUC from 0.753 to 0.757.Source: §3 · p. 28

4 Deviations from the pre-registered plan

• Regenerating two Qwen2.5 1.5B outputs left primary results unchanged but changed interviewer-speech prediction from supported (median AUC 0.61) to not supported (0.59).The regenerated outputs had option mass 0.995 and 0.990; original files were retained.Source: §4 · p. 28
• The end-of-number factor was applied only to numbers beginning longer candidates; in a four-model, 25-participant sample, this changed expected totals by at most 0.008 points without changing decisions.The authors note that this sample does not bound the defect for other models or views.Source: §4 · p. 28
• Although the plan named Steiger’s test, the code used Meng–Rosenthal–Rubin; the paired bootstrap gave matching configuration shares, and n−3 degrees of freedom preserved every regularity.The reported shares were 3.9% and 4.3% in discovery, and 1.1% and 0.9% in confirmation; the confirmation PTSD regularity had P = 0.049.Source: §4 · pp. 28–29
• Discovery decision studies averaged across both transcript views, while two addenda were specified before confirmation comparison and analyses added after confirmation were exploratory.Single-view values were reported in Supplementary Table 11.Source: §4 · p. 29
• An interim code change was reverted before registered analyses; reacquired Qwen2.5 weights reproduced earlier outputs, and resumed scoring agreed across batch compositions within the stated tolerance.The interim change made incomplete-output runs record interviewer-speech prediction as missing rather than stop.Source: §4 · p. 29