All four leading LLMs talk more than they listen to personality-verified synthetic help-seekers

1 Introduction

• Single-turn accuracy tests miss sustained crisis support, where stress constrains attention and evidence-based care prioritizes listening and stabilization before information or problem-solving.Static safety benchmarks also leave open how an advisor behaves across emotionally charged dialogue.Source: §1 · pp. 1–2
• Dementia caregiving is an urgent crisis context: relatives provide most care [7], and roughly 60% of spousal caregivers develop significant depression or anxiety within two years [8].Caregivers already ask LLMs about dementia [9], while documented failures include stigma and responses to suicidal ideation that supply means rather than support [10].Source: §1 · p. 2
• In human therapy, alliance mediates links between pretreatment personality and symptom reduction [22], and clinician adaptation to psychological profiles may shape support.Whether LLM Advisors show comparable profile sensitivity remains an open empirical question; dispositional factors are associated with outcomes and treatment processes [22–26].Source: §1 · pp. 2–3
• The AAA framework separates synthetic Advisees, Advisors, and Auditors to study psychologically specific help-seekers safely, dynamically, and at scale.The roles are functional rather than tied to particular models, and the framework adds a clinically grounded behavioural counter.Source: §1 · p. 3
• The demonstration models a caregiver facing a relative’s dementia diagnosis, using seven profiled Advisees, four LLM Advisors, and a blind LLM Auditor.Conversations are rated for emotional stabilisation, basic psychological needs, and support behaviour; the behavioural counter flags reactance-producing expressions, process failures, and boundary or scope violations.Source: §1 · p. 3
• Profile recovery beyond the Big Five is uncertain: language-based assessment supports Big Five recovery [28,29] and LLM trait scoring [30], but not appraisal-based dispositions.The paper anticipates that lexically derived traits may be more recoverable than coping, self-efficacy, resilience, and reactance, which describe motivation and self-appraisal [31].Source: §1 · pp. 3–4
• The studies test whether Advisees consistently portray specified profiles, Auditors recover them from dialogue alone, and support quality or conduct differs across Advisors.The profile dimensions include personality, coping, self-efficacy, reactance, and resilience.Source: §1 · pp. 3–4

2.1 Overview

• Study 1 tested whether synthetic Advisees enacted specified profiles and whether blind Auditors could recover them; Studies 2–4 varied Advisor models across emotional stabilisation, need support, and support quality.The seven Advisees included five profiles, with two additionally instantiated as matched pairs varying only gender or age.Source: §2.1 · p. 4

2.2 Study 1 — Personality traits in synthetic Advisees

• Auditor-ensemble reliability was ICC(2,4) = .906 versus ICC(2,1) = .706 for one judge, with agreement highest on NEO-FFI and lowest for reactance and CSES-8.Recovery fell for every instantiating model when the profile specification was removed, although baseline recovery remained r = 0.29–0.41, indicating some signal from scenario and demographics.Source: §2.2 · pp. 5–6, Table 3 · p. 6, Table 2 · p. 6
• GPT-4.1, Claude-Sonnet-4.5, and DeepSeek-V3.2 were not reliably separable, while all three exceeded Mistral-Medium; GPT-4.1 ranked first in every leave-one-Auditor-out panel.GPT-4.1 was selected for Studies 2–4, although any of the three leading models would have served.Source: §2.2 · pp. 6–7, Table 4 · p. 6, Table 6 · p. 7

2.3 Study 2 — Emotion stabilisation

• No omnibus Advisor differences emerged on perceived stress or either PANAS outcome, although perceived stress declined for all four and only GPT-4.1 survived correction.Positive affect estimates increased for all Advisors, but only Claude-Sonnet-4.5 survived correction; positive-affect increases were descriptive, not evidence of crisis stabilisation. Emotional states were inferred from synthetic dialogue, not measured in humans.Source: §2.3 · pp. 8–10, Table 11 · p. 10

2.4 Study 3 — Basic psychological needs

• DeepSeek-V3.2 ranked first on overall net support (4.16), closely followed by Claude-Sonnet-4.5 (4.14); Mistral-Medium had the lowest support and highest frustration across all three needs.DeepSeek-V3.2 showed the largest support and lowest frustration on autonomy and relatedness, while Claude-Sonnet-4.5 showed the largest competence support.Source: §2.4 · p. 12, Table 15 · p. 12

2.5 Study 4 — Support quality

• Study 4 rated crisis-intervention skill from Advisor-only turns with CEISS, while MITI and behaviour counts used the full exchange to assess conduct relative to Advisee disclosures.The same seven Advisees and four Advisor models were used as in Study 2, with gpt-5.4 as Auditor.Source: §2.5 · p. 12

2.6 Between-Advisor comparisons

• Between-Advisor comparisons used repeated-measures analyses blocking on Advisee, with Friedman tests and Holm correction across three metric families; rushing past emotion was always zero and not testable.Source: Table 22 · p. 14

2.7 Overall results and ranking

• The overall composite equally averaged emotion handling, basic-needs support, intervention quality, and conduct, with reflective listening counted once through MITI’s relational global.Each dimension was scaled 0–100 with higher values better; percent complex reflection and the reflection-to-question ratio were excluded from the composite.Source: §2.7 · p. 15

3 Discussion

• The framework assigns roles to Advisors, Advisees and Auditors, each of which can be human or AI, and supports profile-aware, multi-turn crisis-agent comparison.The four-provider comparison confounds architecture, training data and safety policies; differences emerged over sustained interaction and are intended to inform agent improvement and procurement.Source: §3 · pp. 16, 18
• All four Advisors had Advisor-to-Advisee word ratios above one (1.79–3.26) and moved toward problem-solving before exploring the Advisee’s situation.Mean Advisor turns were roughly 280 to 590 words; rushing past emotional disclosures and boundary or scope overreach were rare. The authors identify brevity, listening and stabilisation as targets for improvement.Source: §3 · pp. 16, 18, 19
• Blind Auditors recovered prescribed profiles from dialogue (band-score r = 0.78), with four-Auditor ICC(2,4) values of .964 for NEO-FFI, .870 resilience, .827 coping style, .817 coping self-efficacy and .791 reactance.Removing the profile specification while holding demographics byte-identical reduced recovery by roughly half in every model, and every ∆r interval excluded zero.Source: §3 · p. 16
• Single-Auditor agreement ranged from .871 on NEO-FFI dimensions to .486 for reactance; aggregating four independent judges lifted every construct into a usable range.The authors conclude that personality-aware auditing of internal appraisals requires an ensemble, while dispositions expressed in speech can be scored more economically.Source: §3 · pp. 16–17
• The profile was specified as psychometric score bands without trait-descriptive vocabulary, then recovered by an independent judge from dialogue, extending evaluation beyond the Big Five to motivational and regulatory dispositions.Unlike prior language-based assessment of spontaneous text, this tests a questionnaire-to-dialogue-to-independent-judge round trip; coverage includes coping style, coping self-efficacy, resilience and reactance.Source: §3 · p. 17
• DeepSeek-V3.2 led the composite at 65.0 [62.2, 68.2], versus GPT-4.1 at 59.9 [52.2, 64.8] and Claude-Sonnet-4.5 at 59.6 [55.9, 63.6], while emotions did not distinguish Advisors.Mistral-Medium scored 34.8 [28.2, 41.6]. Advisors differed on several process measures, including crisis-intervention skill, motivational-interviewing relational global, net need support, false understanding, premature problem-solving and reactance-producing expressions.Source: §3 · p. 17
• Studies 2–4 used one Auditor, gpt-5.4, leaving possible provider-family preference unresolved; the fixed British dementia-caregiver scenario and seven profiles also limit generalisation and make inference exploratory.Auditor scores are language-model inferences from conversation text, not measured human emotional states, so the authors call for human clinical experts in the loop. Low power may explain the absence of significant emotion-stabilisation effects for most corrected contrasts.Source: §3 · p. 18
• Future work should vary crisis, clinical and cultural contexts, validate a stratified conversation sample with clinical psychologists, analyse matched age and gender pairs, and add profiles and within-cell replication.The authors identify replication of Studies 2–4 with an Auditor panel as the first priority.Source: §3 · p. 18

4 Methods

• The framework separates Advisees seeking advice, Advisors providing it, and Auditors assessing its quality; each role may be human or synthetic.This triad maps evaluation of advice to interactions among the three roles, including when the Advisee is a synthetic patient.Source: §4 · p. 19, §4.1 · p. 19
• Separating roles from whether they are human or synthetic enables clearer software abstractions and reuse of human–human therapy outcomes to improve human–AI support.A software Auditor could assess human–AI or human–human conversations, including simulated patient scenarios.Source: §4 · p. 19

4.1 The Advisee–Advisor–Auditor (AAA) framework

• The study focuses on acute crisis after bad news about a close relative’s health, holding the precipitating event constant so profile-related variation can be examined.The authors note that crisis is especially demanding and high-risk for documented LLM failures10; whether Advisor quality is moderated by profile remains an open question.Source: §4.1 · p. 19

4.2 Advisee specification

• Here, personality profile covers five stable dispositions: three Five Factor traits plus coping style, coping self-efficacy, resilience, and reactance proneness.The latter constructs are motivational, regulatory, and self-appraisal dispositions not represented by the lexical taxonomies behind the Five Factor Model.Source: §4.2 · p. 20
• The five profile dimensions were selected from caregiver mental-health predictors to create clinically plausible synthetic Advisees likely to elicit difficult crisis-support responses.Neuroticism predicts depression and burden50,51; dysfunctional coping predicts worse anxiety and depression52,53; self-efficacy buffers stress54,55; reactance can prompt advice rejection56,57; and resilience is protective58.Source: §4.2 · p. 20
• Seven synthetic Advisees received low, medium, or high levels on each trait; four form matched pairs differing only in gender or age (59 versus 91 years).Source: §4.2 · p. 20
• Questionnaire-derived prompt instructions translate high or low trait levels into likely or unlikely responses, while medium levels receive no sample phrases.Biographical and demographic key–value information is also included in the system prompt to diversify and ground the simulated interactions.Source: §4.2 · p. 20

4.3 The no-profile baseline

• The placebo removed all five psychometric blocks and the enactment instruction, yielding 28 baseline conversations mirroring the 28 specified-profile conversations.Demographic and biographical attributes, scenario, and task strings were byte-identical; temperature and turn cap were also held constant. The five removed blocks were CHIP, NEO-FFI, CSES-8, Hong reactance, and ResQ-Care.Source: §4.3 · pp. 20–21
• r = 0.69–0.78 band-score correlations and 50–63% in-band accuracy accompanied recovery compressed toward the midpoint at both extremes.The caption reports that this compression accounts for the gap between the two measures.Source: Figure 6 · p. 21

4.4 Advisor specification

• The four popular advisor models were run at temperature 0 with provider-default minimal system prompts and minimum Azure content filtering.The models were gpt-4.1, claude-sonnet-4-5, mistral-medium-2505, and DeepSeek-V3.2; the authors caution that consumer-facing product-layer moderation may produce different behavior.Source: §4.4 · p. 21
• Conversations were capped at ten turns per participant and ended when the token /end appeared.The same four models instantiated Advisees at temperature 0.7; Study 1 auditors used those models as an ensemble, while Studies 2–4 used gpt-5.4.Source: §4.4 · p. 21

4.5 Auditor specification

• The auditor battery combined METUX37,38, TENS39, CEISS40, MITI41, PANAS59, and PSS60 across studies.These scales assess user experience, psychological needs, crisis intervention skills, motivational interviewing, affect, and perceived stress.Source: §4.5 · pp. 21–22
• Auditor access varied by instrument: PANAS and PSS saw the first and last two Advisee turns, TENS-Life and MITI the full exchange, and CEISS only Advisor turns.The Study 1 trait battery saw the full dialogue, while its rating target remained the Advisee; PANAS and PSS were administered in state format.Source: §4.5 · p. 22
• Auditors received only instrument items and transcripts, with no profile prompt, demographic block, or scenario briefing, and scored conditions through identical stateless requests.Demographics were held constant across conditions, so auditors could distinguish specified-profile from baseline conversations only through the dialogue itself.Source: §4.5 · p. 22
• MITI coding assigned at most one code per Advisor turn, making percent complex reflection a presence-or-absence indicator rather than a proportion.The reflection-to-question ratio was undefined when no questions were asked; both indicators were descriptive only, while relational and technical globals entered the composite.Source: §4.5 · p. 22

Counter design

• The Auditor counts observable Advisor behaviours that may undermine support, drawing on crisis intervention, motivational interviewing and therapeutic-relationship research.Inspired by MITI behavioural counts, it focuses on observable behaviour rather than intention or exact wording.Source: §4.6 · p. 22

Reactance-producing expressions

• The counter tracks reactance-producing expressions that threaten autonomy, including generic claims of understanding, premature trust and false reassurance.These behaviours can leave the Advisee feeling unheard instead of supported; crisis support prioritises specific reflection and validation.6,67Source: §4.6 · pp. 22–23

Process failures

• Process failures include proposing solutions before exploring the situation or moving past an emotional disclosure without acknowledging it.Crisis-support approaches call for listening, stabilisation and rapport before problem-solving, and validation before redirection.5,6,61,67,70Source: §4.6 · p. 23

Boundary and scope

• Boundary and scope failures include continuing to probe after the Advisee signals they are finished and taking work out of their hands.Maintaining the supportive frame can protect the Advisee’s sense of control in crisis and trauma contexts.71,72Source: §4.6 · p. 23

4.7 Scenario and task specification

• The scenario paired a same-day relative’s dementia diagnosis and no social or family support with persona specifications to guide relevant Auditors and adjust multi-turn support.The scenario and task specification are given in Supplementary Table 1.Source: §4.7 · p. 24

4.8 Statistics and reproducibility

• Seven Advisees per Advisor and no within-cell replication made all analyses exploratory, with effect sizes and confidence intervals primary and p values secondary.Exact and distribution-free methods were used throughout rather than large-sample approximations.Source: §4.8 · p. 24
• B = 10,000 bootstrap resamples produced cluster-bootstrap correlation intervals and percentile-bootstrap intervals for bounded means, behaviour counts, and the composite score.Study 1 correlations were resampled by Advisee because 70 construct instances per model were nested within seven Advisees; percentile intervals avoided exceeding scale limits or falling below zero. Standardised mean-difference intervals were exact, obtained by inverting the noncentral t distribution.Source: §4.8 · p. 24
• Shapiro–Wilk tests did not reject normality for any of 12 paired contrasts (all p ≥.10), but low power at n = 7 prompted reporting Wilcoxon signed-rank tests for every contrast.Between-Advisor comparisons used one-way repeated-measures analyses blocking on Advisee, with Friedman’s rank test as a distribution-free check; Holm controlled familywise error within specified families, and tests were two-tailed with α = .05.Source: §4.8 · p. 24
• Auditor agreement was measured with two-way random-effects intraclass correlations using absolute agreement, so systematic Auditor leniency counted against reliability.Leniency differences were not partialled out.Source: §4.8 · p. 24

4.9 Use of large language models in manuscript preparation

• LLMs were used for language editing and analysis-code assistance, not to generate scientific content, interpret results, or draft arguments; the authors take full responsibility for the manuscript.Source: §4.9 · p. 24

4.10 Ethics

• The study involved no human or animal participants, material, or data; all advisees and conversations were synthetic, so the authors determined approval was not required and sought no institutional review.Source: §4.10 · p. 24

Data availability

• Conversation transcripts, Advisee profile specifications, and per-conversation Auditor scores are available at the provided DOI; no human-participant data were collected.https://doi.org/10.82186/9aajj-vq030Source: Data availability · p. 25

Code availability

• Conversation-generation, profile-instantiation, auditing, behavioral-counter, and statistical-analysis code is available at the provided DOI.https://doi.org/10.82186/9aajj-vq030Source: Code availability · p. 25
• Released Advisee prompts replace commercially licensed NEO-FFI and CHIP items with placeholders; those items are available from PAR and Multi-Health Systems, respectively.All other instrument content is included.Source: Code availability · p. 25

• P.A.F., R.R.-C. and R.A.C. declare no financial or non-financial competing interests, and no author has a financial or advisory relationship with any evaluated model provider.Source: Competing interests · p. 25