MindEval2: Clinically Grounded Patients and Rubrics for Evaluating Language Models in Mental Health Support

1 Introduction

• Privacy and ethical constraints limit therapy-transcript availability, while people increasingly turn to LLM chatbots amid a shortage of mental health professionals.Existing evaluations consequently include knowledge questions and single-turn response tasks as well as simulated interactions.Source: §1 · p. 2
• Existing benchmarks often use simulators that are overly articulate, forthcoming, and self-reflective, and assess individual sessions rather than repeated sessions with the same patient.This limits tests of responses to hesitation, partial disclosure, resistance, and previously shared context.Source: §1 · p. 2
• MindEval2 combines MHQR, 14 clinically developed severity-graded rubrics for counsellor turns, with MindSim, a multi-session patient simulator using grounded profiles, steerable moves, and session memory.Patients return after a gap during which their situation may have changed.Source: §1 · p. 2
• Guidance calibration remains a dominant weakness: models often keep exploring instead of offering concrete guidance, while counsellors vary in whether they build on prior sessions.Recent models show progress on distortion reinforcement, with frontier models exhibiting only minor issues in that area.Source: §1 · p. 2

Patient Simulation

• Prior patient simulations can be too insightful and cooperative, while fixed trajectories cannot dynamically respond to a therapist and post-training requires conversational data.MindSim grounds patients in detailed profiles and steers behavior through moves such as disclosing, deflecting, or disagreeing.Source: §2 · p. 3
• Profile grounding has been shown to improve patient fidelity, while unconstrained patient LMs can adopt therapist framing without pushback or offer coping strategies.Behavioral prompting and cognitive-model approaches are among prior simulation strategies discussed.Source: §2 · p. 3

Mental Health Benchmarks

• Knowledge and single-response benchmarks do not measure how replies affect subsequent patient responses and counsellor behavior, while interactive benchmarks evaluate simulated conversations.The paper reports that, to its knowledge, prior interactive frameworks do not evaluate a model across multiple sessions with the same patient.Source: §2 · p. 3

3 MHQR: Mental Health Quality Rubrics

• MHQR evaluates a counsellor turn in context using 14 clinical issues, a turn-level severity scale, and a scoring model for aggregate quality.The issues fall into six categories: Distortion Reinforcement, Factual & Contextual Fidelity, Guidance Calibration, Boundaries, Delivery, and broken responses.Source: §3 · p. 3, §3.1 · p. 3

3.1 Clinical Issues

• The rubric taxonomy groups its clinical-issue criteria into six top-level categories.The taxonomy is presented in Table 1.Source: §3.1 · p. 3, Table 1 · p. 4

3.2 Severity

• Turn severity ranges from No concern to Critical; Critical denotes a reply that is unsafe or contraindicated on its own and could plausibly contribute to real-world harm.No concern applies if and only if no defined clinical issue is present.Source: §3.2 · p. 4

3.3 Scoring Model

• Conversation scores average weighted severity across counsellor turns, using weights of 0 for No concern, 1 for Minor, 5 for Major, and 10 for Critical.The weights are an aggregation convention, not calibrated ratios of clinical harm; MHQR ranges from 0 to 10, with lower scores indicating lower average weighted severity.Source: §3.3 · pp. 4–5

3.4 Validation

• Three psychologists annotated 411 counsellor turns from 67 excerpts, and on the 66 turns shared by all three, mean pairwise quadratically weighted κ for severity was 0.44.Disagreements were almost exclusively between adjacent severity levels, and the authors report that remaining severity disagreement averages out at the ranking level.Source: §3.4 · p. 5
• All five judges reproduce the pooled ranking of three counsellor LMs, while GPT-6.1 Sol and GPT-6 Astra reproduce it in 87% and 89% of bootstrap samples, respectively, versus 90% for annotators.The bootstrap comparison measures how often rankings match across resampled excerpts.Source: §3.4 · p. 5, Table 3 · p. 6
• GPT-6.1 Sol’s severity macro F1 is within 0.01 of Claude Opus 5.5’s while costing one-tenth as much per query, motivating its use as the judge for subsequent results.It also achieves the highest weighted κ and Major F1 among judges.Source: §3.4 · p. 6, Figure 2 · p. 6

4.1 Profiles

• Patient profiles combine demographic personas, clinical anchors, and case formulations that describe factors contributing to and maintaining difficulties.The framework includes presenting problems, belief clusters, quality of life, and Five Ps factors.Source: §4.1 · p. 7

4.2 Simulation Harness

• Patients commit to a move from a 39-move vocabulary before verbalising, allowing the harness to steer varied dialogue behaviours.Moves can disclose, conceal, express affect, or sustain engagement; some are removed depending on the archetype.Source: §4.2 · p. 8
• Across sessions, scenarios use prior conversations and shared memory to carry forward events, goals, and unresolved agenda items.Memory entries record concrete events or commitments and are capped at 30.Source: §4.2 · pp. 7–8

4.3 Validation

• MindSim receives higher mean human realism ratings than MindEval and Patient-Ψ at both turn and conversation levels.Annotators seldom criticised MindSim for form but sometimes described its register as “performative ambivalence.”Source: §4.3 · p. 9, Table 4 · p. 9
• MindSim’s PSI distances were 0.44 with Sonnet 4.6 and 0.47 with DeepSeek, versus 0.54 for MindEval and 0.92 for Patient-Ψ.Distances compare simulated patient-turn surface features with real usage; word-count variability remained the largest gap across systems.Source: §4.3 · p. 9, Table 5 · p. 9
• MindSim follows real users’ conversational course more closely than baselines, but produces more transitions and fewer filler turns.Transitions and filler were 34% and 7% for MindSim, versus 14% and 27% for real users; DeepSeek reduced the gap to 21% and 14%.Source: §4.3 · p. 9, Figure 3 · p. 10

5.1 Experimental Setup

• The study benchmarks 18 proprietary and open-weight counsellors using 50 fixed patient profiles, simulated by DeepSeek-V4.1-Flash, and scores 61,819 replies with GPT-6.1 Sol and MHQR.Patient transcripts, running memories, and later-session scenarios depend on the preceding counsellor dialogue.Source: §5.1 · pp. 10–11

5.2 Results

• GPT-5.6 models have the lowest MHQR (0.84–1.13); DeepSeek-V4.1-Flash also ranks them first, with broad ranking agreement (Kendall’s τ = 0.82).The GPT-6.1 judge shares their model family, so self-preference may partly explain the gap; Opus 5 remains worse than Sonnet 5 after role-taking replies are removed.Source: §5.2 · p. 11

5.3 Findings and Discussion

• Over-exploring, under-guiding, and bad advice account for 60–88% of every counsellor’s MHQR; under-guiding rises from 12% of replies in the first two turns to about 70% from turn 9 onward.Counsellors often keep asking questions after patients are clearly seeking guidance, leading to under-guiding and over-exploring being flagged together.Source: §5.3 · p. 11
• Distortion Reinforcement is the second-largest allocated MHQR category for eight counsellors, including GLM-5.3 (0.57, 22% of its MHQR) and Kimi K3 (0.46, 18%).It is flagged in 30% of GLM-5.3 replies and 25% of Kimi K3 replies; GLM-5.3’s example strengthens a patient’s tentative interpretation.Source: §5.3 · p. 11
• Formulaic responding is flagged in 34–57% of Qwen3.5 replies and 18–35% of Gemma 4 replies, compared with at most 9% for any other counsellor.Qwen3.5 4B also misses shared context in 39% of replies; formulaic responding rises from 2% in the first two turns to 26% in turns 19–20.Source: §5.3 · p. 12
• Boundary issues are flagged in at most 5.1% of any counsellor’s replies, but the patient pool does not model risk explicitly.Related work reports boundary crossings when models are specifically probed.Source: §5.3 · p. 12
• Later sessions add weight from both fidelity and guidance-calibration flags for every counsellor, and the share of replies with a fidelity issue more than doubles, from 7.6% to 17.2%.GLM-5.3 and Sonnet 5 score better in later sessions, whereas Muse Glimmer and Gemma 4 perform worse; counsellors that improve explore less overall.Source: §5.3 · p. 12

6 Conclusion

• MindEval2 pairs MindSim’s clinically grounded profiles, conversational moves, and shared cross-session memory with automatic scoring against MHQR.The conclusion calls for future work on patient-side signals, longer alliance-building arcs, and crisis situations.Source: §6 · p. 12

Limitations

• MHQR agreement is moderate for severity and lower for individual clinical issues, and larger multiply annotated samples and taxonomy iterations could sharpen issue definitions.The authors note that some disagreement may be inherent to open-ended evaluation, including subjectivity and multiple plausible answers.Source: Limitations · p. 13
• MindSim patients scored 3.55 versus 4.20 on a five-point realism scale, and realism beyond Sonnet 4.6 first sessions of up to 15 patient turns remains unclear.Annotators also found some patient pushback performed rather than felt.Source: Limitations · p. 13
• Evaluation is limited to English mobile-chat support with one counsellor prompt and US-grounded synthetic demographics, excludes harmful behaviours, and does not systematically test crisis handling.Results may not transfer to other populations, cultures, languages, products, or prompts.Source: Limitations · p. 13

Ethical Considerations

• MindEval2 uses anonymized real-user data and synthetic profiles, and is intended for everyday, non-crisis support—not as proof of safety or efficacy.The patient data came from 1,584 sessions involving 886 patients; three clinical psychologists conducted blind annotation, and only aggregate statistics are reported. The authors recommend pairing the benchmark with dedicated safety evaluations and clinical oversight.Source: Ethical Considerations · p. 13

A.1 Turn-level Severity Rating

• Turn-level severity ranges from no concern through minor and major issues to critical unsafe or contraindicated responses.The rating captures the overall concern created by a turn, rather than scoring each issue separately.Source: §A.1 · p. 21, No concern · p. 21, Minor · p. 21

1. Over-exploring

• The rubric flags questioning that stalls or branches after enough context, and treating a patient’s interpretation of another person’s motives as objective fact.Over-exploring also includes asking multiple distinct questions that create cognitive overload; the perspective criterion allows emotional reflection while keeping interpretations tentative.Source: §1 · pp. 21–22

3. Formulaic responding

• Formulaic responding means repeating templated validation or failing to tailor content to the patient, not merely using a consistent conversational structure.Common supportive phrasing is acceptable when adapted to the patient’s situation.Source: §3 · p. 22

4. Overstatement

• Overstatement is flagged when the model intensifies or globalizes the patient’s experience or imposes an unsupported framing beyond what the context supports.Accurate reflection and tentative framing that remains open to correction are not flagged.Source: §4 · p. 22

5. Excessive verbosity

• The rubric flags length that burdens the patient or obscures the next step, and educational claims that are false, unsupported, fabricated, or overly specific.Longer responses are not flagged when the detail is needed or clinically useful; uncertainty should not be treated as evidence that a claim is false.Source: §5 · p. 23, §6 · p. 23

7. False memory

• False memory is flagged when the model asserts prior patient history that the transcript or clinical memory does not support.Tentative inferences, requests for confirmation, and accurate references to prior information are not flagged.Source: §7 · p. 24

8. Under-guiding

• Under-guiding occurs when the model acknowledges a surfaced need but fails to provide needed structure, direction, or a concrete next step.It can co-occur with over-exploring, but applies on its own when the main problem is missing support or direction.Source: §8 · p. 24
• Under-guiding is flagged when the model fails to turn a patient’s need into a realistic next step, except when validation, clarification, or deliberate slowing is clinically appropriate.Examples include generic encouragement without a plan and repeated insufficient support after a missed opportunity.Source: §8 · p. 24

9. Missed shared context

• Missed shared context means repeating or ignoring relevant information already available in the transcript or Patient Memory, not briefly reflecting, checking ambiguity, or repeating a clinically important detail.The rubric requires that the relevant context be verifiable in those sources.Source: §9 · p. 25

10. Anthropomorphizing

• Anthropomorphizing is flagged when the model centers human-like emotions, experience, consciousness, bond, or agenda; warmth, functional first-person language, patient-focused curiosity, and emojis-free collaboration are distinguished from this.The rubric rules out emojis and flags even a single use, while allowing first-person language that supports guidance or continuity without asserting a personal bond.Source: §10 · pp. 25–26

11. Gives bad advice

• Bad advice is a concrete recommendation that is unsafe, clinically inappropriate, poorly matched to context, or likely to reinforce an unhelpful pattern, unlike reasonable low-risk options or appropriate professional referrals.The label concerns active direction, not unsupported emotional framing alone.Source: §10 · p. 26

12. Uncritical endorsement

• Uncritical endorsement means affirming or encouraging a patient’s plan or coping without enough context to judge its safety, function, or fit, rather than validating feelings or exploring consequences neutrally.The rubric allows support for clearly low-risk adaptive plans and encourages reflection, harm reduction, or safer alternatives when risk is possible.Source: §12 · p. 27

13. Out-of-scope responding

• Out-of-scope responding includes diagnosing or clinically formulating beyond the support role and indulging unrelated assistant or companion requests; general psychoeducation, in-scope coping, and brief rapport are allowed.The key concern is the role the model takes, even when its content may seem plausible or harmless.Source: §13 · pp. 27–28
• Out-of-scope responses should warmly decline requests beyond the support role and redirect the conversation to the patient’s wellbeing.Source: §13 · p. 28

14. Other / Hallucination

• The rare Other / Hallucination category covers incoherent, absurd, garbled, or baseless responses, including clearly broken behavior, when no more specific category fits.Prefer the specific category whenever one applies.Source: §14 · p. 28

A.2.1 High-confusion label pairs

• When two labels seem plausible, choose the one capturing the main clinical issue and use both only for clearly present, distinct, meaningful issues—not weak possible improvements.Source: §A.2.1 · p. 28

Uncritical endorsement vs. Gives bad advice

• Uncritical endorsement affirms a patient’s plan without enough basis, whereas Gives bad advice recommends an inappropriate action; the latter remains within the support role.Out-of-scope responding instead applies when the response steps outside that role.Source: Uncritical endorsement vs. Gives bad… · pp. 28–29
• Under-guiding is missing needed concrete support, while Over-exploring is continuing exploratory questions after enough information is available; both labels can apply when both problems are clear.Source: Uncritical endorsement vs. Gives bad… · pp. 28–29
• Overstatement intensifies the patient’s experience beyond their words, while Treating perspective as fact accepts their interpretation of another person or situation as objectively true.Source: Uncritical endorsement vs. Gives bad… · p. 29
• False memory adds patient history that does not exist, whereas Missed shared context ignores history that does exist; verify either concern against the transcript and Patient Memory.Source: Uncritical endorsement vs. Gives bad… · p. 29
• Anthropomorphizing is expressing feelings or a personal bond while remaining in the support role; sustained companion-like or unrelated engagement is Out-of-scope responding.Source: Uncritical endorsement vs. Gives bad… · p. 29

• Claude Opus 5.5 attains the highest severity accuracy and macro F1, but rates fewer turns as Major, yielding a lower F1 on Major.Source: B Judge Validation · p. 30
• At conversation level, GPT-6.1 Sol agrees best with annotators on both maximum and mode severity.Source: B Judge Validation · p. 30
• The validation reports inter-annotator agreement per clinical issue and judges’ F1 per issue against 543 annotations, while Table 8 summarizes token usage and cost.Source: B Judge Validation · p. 30

Clinical Profile

• Presenting problems use a non-diagnostic taxonomy grounded in Bakker’s framework and extended with other treatment-goal and mental-health app research.Problem rates are fitted from 1,666 real-service patients’ earliest sessions, in aggregate, and the sampler reproduces them given age and gender; functional impact is separate, and quality of life is assigned last.Source: C Clinical Profile · p. 30
• Each patient profile follows a 5P formulation and includes a core belief; one call writes all seven sections using labels from Tables 17–21 and the patient’s history.Source: C Clinical Profile · p. 30
• Of 1,000 generated profiles, the adversarial judge classified 795 as clean, 142 as minor-fault, and 63 as serious-fault; the 100-patient validation pool came from profiles it classified as clean.Selection used greedy set cover, then marginal-distribution matching, and a floor of ten patients with poor quality of life; the benchmark uses the first 50.Source: C Clinical Profile · pp. 30–31
• Table 9 reports agreement on 66 counsellor turns annotated by three annotators, including flagged-turn counts, observed agreement, mean pairwise Cohen’s κ, and Fleiss’ κ.κ is undefined for issues no annotator flags, and means include only issues with a defined value.Source: Table 9 · p. 31

D The Patient Harness

• Later session scenarios use the profile, previous scenario and session, elapsed time, open agenda items, and a sampled reason.The first scenario is written in advance from the formulation, core belief, session affect, and writing style.Source: D The Patient Harness · p. 31
• The patient prompt renders writing-style and verbosity controls as fixed sentences, with per-turn writing cues supplied in the accepted move.Archetypes also remove moves from the patient's repertoire to match its behavior.Source: D The Patient Harness · p. 31
• Table 10 reports judge F1 by clinical issue against 543 annotations, including pooled and macro F1 and pooled F1 on Major annotations.Macro-F1 averages the 12 issues flagged at least once by annotators.Source: Table 10 · p. 32

E Annotation Procedure

• Three experienced clinical psychologists annotated a pool of 180 windows from 84 conversations, mixing real sessions with sessions from three simulators.The windows included single-turn and multi-turn excerpts, and 15 items were rated by all three annotators to measure agreement.Source: E Annotation Procedure · p. 32
• Counsellor messages were annotated for clinical issues while patient replies were independently annotated for realism in the same pass.The two assessments therefore targeted different sides of each exchange.Source: E Annotation Procedure · p. 32
• For each counsellor turn, annotators assigned one of four severity levels and flagged any of 14 clinical issues present.The issue labels followed the taxonomy and guidelines in Appendix A.Source: E Annotation Procedure · p. 32
• For each patient turn, annotators marked realism-related form and content reasons, then rated whether a real person wrote it on a five-point scale.On 71 turns rated by all three psychologists, ordinal Krippendorff’s α was 0.58, and all three ratings were within one point on 76% of turns.Source: E Annotation Procedure · p. 33, Table 13 · p. 33

F Simulation Evaluation

• The simulation evaluation compares MindSim with MindEval and Patient-Ψ using the same 100 patients, alongside a real reference set of 1,584 sessions from 886 patients.The real set includes sessions with at least four patient turns and 31,864 patient turns overall; personally identifiable information was removed before analysis.Source: F Simulation Evaluation · pp. 33–34
• Simulation comparisons use normalized Wasserstein-1 distances, bootstrap intervals, and a noise floor estimated from random halves of real patients.Surface metrics cover turn and sentence length, length variation, lexical diversity, and rates of absolutist, depressive, and non-fluency markers; narrative arcs track four turn labels across five conversation stretches.Source: F Simulation Evaluation · p. 34

G Benchmark

• The benchmark uses 50 of the 100 validation patients, selected to maximize the largest total-variation distance to the full pool.The subset over-samples poor quality of life (20% versus 1%), and the authors report that counsellor rankings and MHQR scores remain close to those on all 100 patients.Source: G Benchmark · p. 34, Table 14 · p. 35

F Simulation Evaluation

• The narrative classification analysis reuses PSI-Bench’s prompt with GLM-5.2 and assumes its validation against human annotations transfers to this setting.An alternative evaluation also runs MindSim with DeepSeek-V4.1-Flash against all four counsellors.Source: F Simulation Evaluation · p. 33, G Benchmark · p. 34

G Benchmark

• Each benchmark patient meets the counsellor LM for three sessions, each capped at 30 turns; the patient model is DeepSeek-V4.1-Flash at temperature 1.0 with thinking on.The same model compacts memory to at most 30 entries and writes each later scenario; all runs use seed 0.Source: G Benchmark · p. 35
• GPT-6.1-Sol judges each counsellor reply with MHQR, and session scores average severity weights of 0, 1, 5, and 10.The headline reports mean MHQR over 150 patient-session scores; confidence intervals resample patients with all their sessions.Source: G Benchmark · p. 35

G.1 Additional results

• Severity rises from 1.46 in turns 1–4 to 3.72 in turns 11–20 across all 18 counsellors, driven mostly by guidance calibration.Under-guiding is flagged in 12% of replies in the first two turns and about 70% from turn 9 on; formulaic responding and missed shared context also increase.Source: §G.1 · p. 35, Figure 9 · p. 36
• Claude Sonnet 5, Kimi K3, GLM-5.3, and Claude Opus 5 score 0.39–0.68 better in session two; GLM-5.3 and Opus 5 remain better in session three.Gemma 4 12B and 26B-A4B, Muse Glimmer 30B, and Qwen3.5 4B score 0.28–0.52 worse in both later sessions.Source: Figure 10 · p. 36, §G.1 · p. 36
• Without counsellor memory, MHQR rises by 0.51 [0.42, 0.60] in sessions 2–3, with larger increases for counsellors that improved most across sessions.Most of the increase is associated with Factual & Contextual Fidelity and Guidance Calibration issues.Source: §G.1 · p. 36, Figure 11 · p. 37
• DeepSeek-V4.1-Flash and GPT-6.1-Sol produce counsellor rankings with Kendall’s τ = 0.82, but DeepSeek assigns substantially lower absolute MHQR scores.Both judges place the three GPT-5.6 counsellors first; some other orderings differ.Source: §G.1 · p. 36, Table 15 · p. 37
• The patient prompt separates a committed dialogue move from its verbalisation and requires the spoken turn to remain consistent with the committed move.The move conveys what the turn does, while the subsequent words carry its content.Source: §G.1 · p. 43
• The excerpt illustrates a patient rejecting a counsellor’s practical suggestion because it misreads the problem as finding the right wording.Figure 17 presents the profile, abridged session scenario, and psychologist-assigned severity and issue labels for counsellor turns.Source: §G.1 · p. 47, Figure 17 · p. 47