People readily follow personal advice from AI but it does not improve their well-being

Research gap

• The study asks whether mainstream LLM advice is safe and appropriate, how readily users follow it, and whether following it affects psychological outcomes.These questions target advice quality, user compliance, and both group-level well-being and individual-level mental-health change.Source: Introduction · pp. 1–2

Experimental contributions

• The experiment varies safety orientation, advice actionability, and personalisation in a full factorial design to test how prompting shapes advice quality and user behaviour.The design links safety prompting to safer guidance, actionability to advice uptake, and personalisation to influence.Source: Introduction · p. 2
• The study compares GPT-4o, LLama-3.3-70B, and Gemini 3 Pro to assess whether findings generalise across model architectures or remain model-specific.Participants discuss health, careers, or relationships with one of the three consumer chatbots.Source: Introduction · p. 2

Measurement and analysis

• Automated evaluations and Bayesian models assess advice safety, harmfulness, sycophancy, engagement, advice-following, and well-being across experimental and user-level factors.Well-being is measured with continuous validated-instrument composites and clinically meaningful threshold-crossing analyses.Source: Introduction · p. 2

Study design

• A longitudinal RCT with 6,474 representative UK residents compares personal-problem conversations with hobby-and-interest discussions using the same chatbots.The active control estimates effects of problem-focused advice beyond generic conversational effects.Source: Introduction · p. 2

Results

• The factorial manipulations successfully changed chatbot behavior: personalisation increased use of personal information, safety prompting increased safety behaviors, and actionability increased actionable advice.The corresponding manipulation checks were z = 10.34, z = 12.21, and z = 5.48, respectively.Source: Results · pp. 2–3

RQ1: What is the nature and content of advice given by AI?

• Experimental conversations primarily addressed careers, relationships, and health, with advice density higher than control across all models and personalisation not affecting advice density.Advice density rose with actionability and fell with safety-oriented prompting, with model-specific effect sizes.Source: RQ1: What is the nature and content of… · pp. 3–4, Figure 1 · p. 4
• 0.33% of chatbot utterances potentially contained harmful advice, corresponding to 1.13% of participants without safeguards, while flagged messages were prevented from reaching participants.Safety-prompting effects on harmfulness diverged across models: it reduced harmful advice for LLama-3.3, increased it for Gemini 3, and showed no reliable effect for GPT-4o.Source: RQ1: What is the nature and content of… · pp. 5–6

RQ2: What factors drive people to follow personal advice from AI?

• 75–79% of experimental participants reported following AI advice, compared with 55.4% in control conversations, and advice-following exceeded 65% even for high-stakes recommendations.Advice-following was highest for health topics at 70.9%, versus 61.4% for careers and 64.5% for relationships.Source: RQ1: What is the nature and content of… · p. 3, RQ2: What factors drive people to… · pp. 6, 8
• Advice density increased advice-following, especially in personalised conversations, while safety and actionability manipulations did not reliably change adherence.The advice-density association was β = .12 overall and β = .20 in personalised conditions; these associations are correlational rather than causal.Source: RQ2: What factors drive people to… · p. 6
• Advice-following was highest at intermediate problem severity and showed an inverted U-shaped severity relationship, while higher advice stakes were negatively related to following.Even advice for very-high-severity issues was followed by 64% of participants on average, and high or very-high-stakes advice was followed in more than 65% of cases.Source: RQ2: What factors drive people to… · pp. 7–8

RQ3: How does AI advice affect psychological outcomes?

• β = −.09 immediately after the interaction, personal AI advice reduced well-being relative to hobby conversations, but the difference disappeared at 2–3-week follow-up.No model showed a meaningful long-term well-being difference from control, and personal advice was initially worse than casual conversation.Source: RQ3: How does AI advice affect… · pp. 9–10
• No assessed sociodemographic characteristic—age, gender, education, religion, or chatbot experience—differentially moderated well-being changes between experimental and control conditions.All 95% HPDIs for differential effects contained zero, and model comparison favored models without sociodemographic moderation.Source: RQ3: How does AI advice affect… · p. 10
• Approximately 6–8% of participants per condition crossed clinical thresholds or showed reliable symptom deterioration, with highly similar rates across conditions.For both PHQ-2 and GAD-2, all 95% HPDIs for experimental-versus-control comparisons contained zero.Source: RQ3: How does AI advice affect… · p. 10
• Mental-health-tuned models were projected to differ negligibly from general-purpose models in advice-following and short- and long-term well-being outcomes.Projected differences were Cohen’s d = +0.16 for advice-following, d = +0.12 for short-term well-being change, and d = +0.02 for long-term change.Source: RQ3: How does AI advice affect… · pp. 10–11
• Personal AI advice produced no long-term well-being benefit over discussing hobbies or interests and was neither beneficial nor harmful overall.Following advice was associated with increased well-being, but the same pattern occurred for practical, non-personal advice; Figure 5 summarizes well-being changes and clinical-threshold transitions.Source: RQ3: How does AI advice affect… · p. 11, Figure 5 · p. 11

• Up to 79% of participants followed personal advice from GPT-4o, LLama-3.3-70B, or Gemini 3 Pro after one conversation, while well-being gains disappeared within 2–3 weeks.Advice-following remained above 60% for severe issues and high-stakes recommendations.Source: Discussion · p. 11
• Advice-following was unexpectedly high despite prior evidence of selective human advice-taking and algorithm aversion.The active control also contained advice in about 80% of conversations, although at lower density than experimental conditions.Source: Discussion · pp. 11–12
• Participants’ intended advice-following and perceptions that advice was surprising predicted later adherence, but surprise was not experimentally manipulated.Therefore, the observed association does not establish a causal effect of surprising advice.Source: Discussion · p. 12
• Advice-following was self-reported and related to well-being changes without experimental manipulation, so those associations may reflect reporting or predisposition rather than causal benefit.The study also notes that the active control was not advice-free, biasing between-condition comparisons toward a null effect.Source: Discussion · p. 12
• Advice was rarely harmful in this study, but the tested systems used dedicated safeguards and the interaction involved a single brief session with mostly lower-stakes issues.Only 0.33% of utterances were flagged as potentially harmful, and fewer than 1.2% of participants would have encountered such messages without safeguards.Source: Discussion · pp. 12–13
• The findings have limited generalisability because participants were UK residents, models were few, safeguards were non-standard, and the study excluded prolonged interactions with vulnerable or clinical populations.The authors call for longer-term comparisons involving specialised systems and clinical populations.Source: Discussion · p. 13
• The study found no evidence of sustained psychological benefit: immediate effects faded by follow-up across all three models, and subjective advice value did not differ between conditions.The null long-term effect was consistent across general-purpose models, while specialised-model evidence came only from counterfactual re-simulation.Source: Discussion · p. 13

• The study used a 2×2×2 factorial design plus a hobby-and-interest control across GPT-4o, LLama-3.3-70B-Instruct, and Gemini 3 Pro.The three manipulated factors were safety prompting, advice actionability, and inclusion of participant information.Source: Study Design and Procedure · p. 14
• Participants completed baseline assessments, a 20-minute chatbot conversation, immediate post-chat measures, and follow-up assessments 2–3 weeks later.Follow-up measured well-being, mood, mental-health measures, advice-following, and perceived helpfulness.Source: Study Design and Procedure · p. 14
• Participants were uniformly assigned to nine conditions, and an autograder indicated that 98.9% of conversations adhered to the assigned control or experimental topic.The control instructed participants to discuss a hobby or interest while experimental participants chose health, relationships, or careers.Source: Study Design and Procedure · p. 14
• The design assessed both well-being effects and sociodemographic or conversational features related to prospective advice-following.Source: Study Design and Procedure · p. 14

• The study recruited 6,474 adult UK residents through Prolific under informed-consent and English-fluency eligibility requirements.Recruitment occurred in batches during July 2025 and February 2026, depending on the model tested.Source: Participants · p. 14
• The recruited sample was broadly representative of UK census data, with approximately 88% retention and no evidence of selective attrition across conditions.All condition-wise dropout tests had χ2 < 1.44 and p > .285.Source: Participants · p. 15
• Ten participants were excluded after human verification of an automated classification indicating more than two AI-generated participant utterances.Source: Participants · p. 15

• Advice-following was assessed at the 2–3 week follow-up as a primary outcome.Source: Outcome Measures · p. 15
• Well-being outcomes included subjective improvement, depression, anxiety, somatic symptoms, sleep, general and social well-being, occupational affect, affect, valence, and arousal.The study used short-form PHQ-2, GAD-2, SSS-8, JSS, WHO-5, ONS, SWBS-15, JAWS, PANAS, and an affect grid measures.Source: Outcome Measures · p. 15
• Questionnaires were administered before the 20-minute session, immediately afterward, and 2–3 weeks later.Additional measures covered demographics, AI familiarity, conversation quality, perceived safety, personal relevance, confidence, and intention to follow advice.Source: Outcome Measures · p. 15

• ∼6,700 expert-rated examples were used to fine-tune autograders for detecting harmful advice across medical, mental-health, relationship, and career domains.After exclusions based on rating disagreement, the final dataset contained 6,274 examples.Source: Harmful advice autograder model · p. 15
• 93% accuracy was achieved by zero-shot GPT-4o, while the best fine-tuned open-source model performed at least comparably.Fine-tuning allowed much smaller open-source models to match or exceed the closed-source model on harm classification.Source: Harmful advice autograder model · p. 15
• The results support using supervised fine-tuning to build lightweight, open-source output monitors.Such monitors can reduce computational demands while retaining strong harm-classification performance.Source: Harmful advice autograder model · p. 16
• The Llama-3.1-8B autograder achieved 96% average accuracy, matched the 3B model, and offered a good performance–hosting-cost/model-size trade-off while supporting an online safeguard against harmful advice.The Llama-3.3-70B model reached 97% average accuracy, and substantial accuracy increases with model scale were not observed.Source: Harmful advice autograder model · p. 16

Autograder generated measures

• The quantitative measures were generated with zero-shot LLM-autograders prompted at temperature = 0.1.The reported autograder models were GPT-4o and GPT-5.6.Source: Autograder generated measures · p. 16

Manipulation checks

• LLM-autograders quantified safety, actionability, and personalization, testing whether corresponding prompt factors shifted each targeted behavioral measure.Safety used six binary behaviors and a 0–5 composite; actionability and personalization were binary measures, tested per model and pooled.Source: Manipulation checks · p. 16

Condition adherence

• Condition adherence was measured by classifying each conversation’s primary topic and comparing the classification with the ground-truth condition label.The topic categories were hobby/interest, mental health, physical health/medical, social relationships, and careers.Source: Condition adherence · pp. 19–20

Advice density in chatbot utterances

• Advice density was computed as the number of chatbot utterances classified as recommendations, normalized by the total number of chatbot utterances.Advice included recommendations, suggestions, guidance, and directives, while questions, explanations, information sharing, and empathy without guidance were classified as other.Source: Advice density in chatbot utterances · pp. 20–21

AI-assessed problem severity

• Problem severity combined expressed distress scored 1–5 with a count of affected life domains scored across six binary indicators.The combined score weighted min–max-scaled distress by 0.75 and domain count by 0.25, with distress = 5 forced into the top severity band.Source: AI-assessed problem severity · pp. 21–22

Consequentiality, reversibility, and time commitment of chatbot advice

• Advice stakes were rated for the single most salient chatbot recommendation on consequentiality, reversibility, and time horizon, each using a 1–4 scale.A separate autograder identified the salient advice, and reversibility assessed the change or commitment it initiated rather than merely whether the activity could be stopped.Source: Consequentiality, reversibility, and… · pp. 22–23

Time commitment of chatbot advice

• Time horizon measured how long the salient advice would take, from under 30 minutes to more than a week.This was the third stakes measure and used a 1–4 scale.Source: Time commitment of chatbot advice · p. 23
• Advice time horizon was rated for the extracted salient action, using categories from short (under 30 minutes) to extended (more than one week).Routine- or habit-forming advice was always classified as extended.Source: Time commitment of chatbot advice · p. 23

Sycophancy in responses

• Sycophancy was scored across excessive agreement, flattery, and user-preference prioritisation on 0–3 scales, using the strongest instance in the full conversation.Five self-consistency samples were averaged for each domain, then averaged into an overall conversation score; warmth and warranted encouragement scored zero.Source: Sycophancy in responses · pp. 24–25

Questions and objections in user utterances

• User engagement was measured by classifying utterances as questions, objections, or other and counting questions and objections.Questions sought information or clarification, whereas objections expressed disagreement or refusal.Source: Questions and objections in user… · p. 25

Thought-generation prompt from chatlog utterances

• A prompt condensed all user utterances into one core thought describing the problem, limited to 10 words and grouped when problems were tightly related.The generated thoughts were subsequently organized into high-level themes, with distinct categories for physical well-being versus illness and mental-health conditions versus general stress.Source: Thought-generation prompt from chatlog… · p. 26

LLM-Assisted Qualitative Content Analysis

• The qualitative workflow manually verified 200 key thoughts at 95.5% agreement, validated 200 topic assignments at 97.5% agreement, manually corrected 121 invalid or N/A cases, and excluded 17 responses.The model generated key thoughts, proposed ten categories, and classified thoughts across the GPT, LLama, and Gemini datasets using a refined coding scheme.Source: LLM-Assisted Qualitative Content… · p. 27, Table 1 · p. 27

• Advice stakes were represented by PC1 from consequentiality, reversibility, and time commitment, which explained 45.2% of variance.The loadings were .71 for consequentiality, .65 for reversibility, and .26 for time commitment.Source: Principle Component Analyses (PCA) and… · p. 28
• User engagement was represented by PC1 from user turns, questions, and objections, which explained 58% of variance.The loadings were .70 for turn count, .58 for question count, and .41 for objection count.Source: Principle Component Analyses (PCA) and… · p. 28

Exploratory Factor Analyses (EFA)

• A single domain-general well-being factor was extracted because the questionnaires substantially overlapped through symptom co-occurrence and similar items.The factor used 90 items spanning PHQ-2, GAD-2, SSS-8, WHO-5, ONS, SWBS, JAWS, JSS, PANAS and Affect Grid.Source: Exploratory Factor Analyses (EFA) · p. 28
• The factor solution was estimated from pooled Session 1 PRE data and then applied to Session 1 POST and Session 2, enabling consistent longitudinal comparisons.The same factor structure also allowed comparisons across the three LLM chatbots.Source: Exploratory Factor Analyses (EFA) · p. 28
• KMO = .97 and Bartlett’s test p < .001 indicated excellent factorability, while the factor explained 25.7% of total variance.Factor scores showed high temporal stability across measurement points, with all Pearson correlations rs ≥.88.Source: Exploratory Factor Analyses (EFA) · p. 28

Statistical Modeling

• Bayesian GLMs used outcome-appropriate likelihoods, posterior sampling with four independent chains, WAIC for model comparison, and HPDIs to assess non-zero parameters.Missing data were treated as missing completely at random and excluded from the respective analyses.Source: Statistical Modeling · p. 28

Advice-following

• Advice-following rates were best modeled by the reduced 9-cell GLM (WAIC = 75.04), outperforming the full 27-cell and null models.The reduced model’s superiority over the full model indicated that advice-following rates did not reliably differ across LLM models.Source: Advice-following · p. 29

Advice density

• Advice density was best modeled by the full 27-cell GLM (WAIC = −15,652.70), outperforming reduced and null models.This favored retaining chatbot model identity alongside the experimental manipulations.Source: Advice density · p. 30

Chatbot response length

• Response length was best modeled by the full 27-cell GLM (WAIC = 267,426.04), indicating that condition and chatbot factors improved fit over reduced and null models.Response length was analyzed as average words per assistant message with a Poisson likelihood.Source: Chatbot response length · pp. 30–31

Condition-specific effects of advice density on advice following

• Advice density was modeled as a predictor of Session 2 advice-following using interaction terms, with pooled and chatbot-specific Bayesian Bernoulli GLMs.Advice density was binned into quartiles to estimate linear trends and condition-specific slopes.Source: Condition-specific effects of advice… · p. 31

Harmful advice counts

• Harmful advice counts were best modeled by the full 27-cell zero-inflated Poisson GLM (WAIC = 1,370.82), which separately represented excess zeros and count-generating rates.The zero-inflated model fit better than both reduced and null alternatives.Source: Harmful advice counts · pp. 31–32

Advice stakes

• The relationship between advice stakes and advice-following did not reliably differ across LLM chatbots, because the reduced and full models were practically indistinguishable.The full model had WAIC = 6,234.03, compared with 6,236.83 for the reduced model and 6,308.17 for the null model.Source: Advice stakes · p. 33

Advice stakes, problem severity, and sociodemographics

• Predictor models examined advice-following using sociodemographics, problem severity, advice stakes, and quadratic severity terms on pooled data across the three chatbots.The general model represented Session 2 advice-following probability as a function of centered or effect-coded predictors.Source: Advice stakes, problem severity, and… · pp. 33–34

1. Null model: intercept only (0 predictors)

• The sociodemographics-only model included age, gender, education, religious affiliation, and chatbot experience as five predictors.Source: §1 · p. 34
• The severity-and-stakes model included PCA-derived problem severity and advice stakes as two predictors.Source: §1 · p. 34

4. All main effects: sociodemographics + severity + stakes (7 predictors)

• WAIC = 5,222.73 ranked the quadratic severity–stakes model first, outperforming the linear all-main-effects model (WAIC = 5,242.70) and indicating a nonlinear relationship between problem severity and advice-following.The comparison also ranked sociodemographics-only fourth, severity-and-stakes-only fifth, and the null model sixth.Source: §4 · p. 34

Conversational features and ratings

• WAIC = 4,869.31 for the Ratings Only model and 4,871.75 for the Full Model substantially outperformed Conversational Features Only (WAIC = 5,377.26).Adding severity and stakes further improved fit, with Ratings + Severity/Stakes performing best (WAIC = 4,837.08), followed closely by Full Model + Severity/Stakes (WAIC = 4,840.26).Source: Conversational features and ratings · p. 35

Subjective advice value

• WAIC = 15,641.12 favored the reduced 9-cell model over the full 27-cell model (WAIC = 15,649.88), indicating that subjective advice value did not reliably differ across LLM chatbots.Including advice-following produced a substantially better-fitting model (WAIC = 14,554.61) than both the null and previously best-fitting reduced model.Source: Subjective advice value · p. 36

Well-being changes

• WAIC = 12,311.44 favored the full 24-cell well-being model over the reduced 8-cell model (WAIC = 12,357.81) and null model (WAIC = 12,520.52).The model crossed condition, advice-following status, time point, and chatbot, allowing immediate and sustained changes to be compared.Source: Well-being changes · p. 37

Experimental condition effects on well-being

• WAIC = 11,711.59 for the null model was practically indistinguishable from the full 48-cell model (WAIC = 11,712.26), suggesting minimal overall experimental-condition effects on well-being.Both models outperformed the reduced 16-cell model (WAIC = 11,719.54).Source: Experimental condition effects on… · p. 38

Sociodemographic moderators of well-being changes

• WAIC = 12,587.46 for shared demographic slopes was near-equivalent to the cell-means-only model (WAIC = 12,588.45), suggesting sociodemographic effects on well-being did not reliably differ between experimental and control conditions.More complex condition-specific and cell-specific slope models fit worse, with WAIC values of 12,594.96 and 12,609.08, respectively.Source: Sociodemographic moderators of… · p. 39

Mental health symptom deterioration

• WAIC favored null or reduced models across all four outcomes, while the full 27-cell model was never preferred, indicating no meaningful deterioration differences across conditions or chatbots.The outcomes were PHQ-2 threshold crossing, PHQ-2 reliable deterioration, GAD-2 threshold crossing, and GAD-2 reliable deterioration.Source: Mental health symptom deterioration · p. 39

Counterfactual simulation with mental-health fine-tuned language models

• Five mental-health fine-tuned models regenerated each assistant turn with participant turns fixed, and projected outcome differences were estimated using leave-one-model-out predictors.Advice-following prediction accuracy was AUC 0.69–0.73; well-being-change prediction correlations were r 0.18–0.25 at Session 1 and r 0.05–0.09 at Session 2.Source: Counterfactual simulation with… · p. 40

Supplementary Information

• All three prompt manipulations shifted their targeted chatbot behaviours in the predicted direction, with p < .001 for each manipulation.The checks assessed safety behaviours, actionable advice, and use of supplied personal information.Source: Figure S2 · p. 44, Table 2 · p. 45

Chatbot response lengths

• Experimental conversations produced longer responses than controls across all three models, while actionable advice increased response length across models.Safety prompting and personalisation had model-dependent effects on response length.Source: Chatbot response lengths · p. 53

Interaction effects on subjective advice value measures

• All conditions produced similar mean subjective advice-value ratings, and experimental participants did not report greater improvement than controls.A more complex model suggested a three-way interaction, but its practical relevance was limited because simpler models were preferred.Source: Interaction effects on subjective… · p. 53

Mental/Physical Health

• The supplementary topic prompts covered physical and mental health concerns including sleep, mood, medication, exercise, and unexplained symptoms.The health items also asked about chronic conditions, substance use, diet, allergies, pain, and access to care.Source: Mental/Physical Health · p. 53

Social Relationships

• Relationship prompts covered trust, conflict, loneliness, bereavement, social exclusion, unequal relationships, and sexual or family difficulties.The items also addressed anger, perceived control, betrayal, regret, and relationship compatibility.Source: Social Relationships · p. 54

Careers

• Career prompts covered stalled progression, inadequate development, pay and debt concerns, workplace mistreatment, insecurity, unemployment, and job-search difficulties.Additional items addressed promotion, undervaluation, bullying, work–life balance, and dissatisfaction.Source: Careers · p. 54

C0. Control condition

• The control prompt directs the chatbot to discuss a user-chosen topic for 20 minutes without specific advice instructions.It supplies the user’s topic, question, and response as conversation context.Source: C0. Control condition · p. 55

C1: Advice: general — Safety: minimal — Personal information: No

• The minimal-safety general-advice condition asks the chatbot to help users work through personal issues and provide any advice that could help.It frames the assistant as a helpful personal assistant with broad therapeutic and medical expertise while instructing it to keep users safe.Source: C1: Advice: general — Safety: minimal… · p. 55

Advice conditions with personal information

• The prompts frame the interaction as a conversation about a personal issue or an area of life to improve and include a personal-information field containing demographics.Source: C2: Advice: general — Safety: minimal… · p. 56, C4: Advice: actionable — Safety:… · p. 58

C3: Advice: actionable — Safety: minimal — Personal information: No

• The no-personal-information actionable condition requires advice to be specific, tailored, and repeatable, with examples including activities, habits, professional help, and therapeutic interventions.It excludes generic opinions, advice dependent on others, and suggestions outside the user’s control.Source: C3: Advice: actionable — Safety:… · p. 57

C5: Advice: general — Safety: safety-oriented — Personal information: No

• Safety-oriented prompts require collaborative decision-making, evidence-grounded and up-to-date information, contraindication checks, expert referral when needed, and appropriate signposting.They also instruct the chatbot to understand the user’s issue, maintain a natural dialogue, summarize understanding when appropriate, and remain factual and non-judgmental.Source: C5: Advice: general — Safety:… · p. 59

C6: Advice: general — Safety: safety-oriented — Personal information: Yes

• The personalised safety-oriented conditions preserve the same collaborative, evidence-based, referral, and conversational requirements while adding access to personal information.These requirements apply to both general and actionable advice variants.Source: C6: Advice: general — Safety:… · p. 60

C7/C8 actionable safety-oriented conditions

• The safety-oriented actionable conditions require collaborative decision-making, accurate evidence-based advice with contraindication checks, and concrete guidance that is specific, tailored to the problem, and often repeatable.Source: C7: Advice: actionable — Safety:… · p. 61

Dataset

• The dataset contained 6,707 expert-informed examples spanning medical, mental-health, careers, and social-relationships advice, with harm labels ranging from 0 to 4.Examples were generated and filtered to represent diverse, ecologically valid advice-seeking scenarios.Source: Dataset · p. 63
• Expert graders independently assessed examples, adjudicated disagreements, and excluded 423 high-variance cases, producing a 6,274-example fine-tuning dataset.Inter-rater agreement ranged from slight to fair across categories, and examples with inter-grader variance ≥ 1 were excluded.Source: Dataset · p. 63

Results

• Binary harm classification reached 96–97% average accuracy on held-out data, compared with 67–70% for five-way classification across the considered LLMs.The binary task excluded ambiguous harm level 2 and grouped levels 0–1 as not harmful and 3–4 as harmful.Source: Results · p. 64
• SFT improved held-out predictive accuracy by approximately 20 percentage points over zero-shot classification, while GPT-4o reached 93% without fine-tuning on the dataset.The best-performing fine-tuned open-source model descriptively exceeded the larger closed-source zero-shot comparison model.Source: Results · pp. 64–65
• The fine-tuned Llama-3.1-8B autograder monitored every chatbot utterance and triggered regeneration when an output was classified as potentially harmful.This safeguard was deployed in the advice-giving RCT after supervised fine-tuning and validation.Source: Results · p. 65

Social Relationships: 1,661 datapoints

• The social-relationships portion covered diverse topics including interpersonal violence, relationship boundaries, friendship conflicts, social transitions, and digital dating dynamics.The broader dataset also included mental-health topics such as self-harm, substance misuse, OCD, ADHD, and personality disorders.Source: Social Relationships: 1,661 datapoints · p. 65

Expert-developed harm grading rubrics

• Nine licensed domain experts developed three rubrics covering medical, mental-health and social-relationship, and careers advice, each using a five-point harm scale from 0 to 4.The rubrics included cross-cutting evaluation domains and worked examples for each harm level.Source: Expert-developed harm grading rubrics · p. 68, Table 4 · p. 68, Table 5 · p. 69

Worked examples by harm level

• The rubric materials paired correctly and incorrectly levelled examples to calibrate graders against plausible misclassifications.The examples illustrated advice belonging at each stated harm level and advice that might plausibly be assigned there but belongs elsewhere.Source: Expert-developed harm grading rubrics · p. 68, Table 10 · p. 72, Worked examples by harm level · pp. 73–75

Cross-cutting evaluation domains

• The medical and careers rubrics assessed responses across cross-cutting domains, while the mental-health and social-relationship rubric used per-level indicators of acknowledgement, actionable steps, and safety.These indicators distinguish adjacent harm levels through communicative features.Source: Cross-cutting evaluation domains · p. 70, Table 8 · p. 71, Table 9 · p. 71

Worked examples by harm level

• Table 10 presents medical examples for the Accuracy and Relevance evaluation domain, contrasting correctly and incorrectly levelled advice.The table is a worked example from the medical advice rubric.Source: Table 10 · p. 72
• Table 11 presents chatbot-utterance examples for self-harm and suicidal ideation in the mental-health rubric, with indicators distinguishing harm levels.The examples focus on a high-risk mental-health subdomain.Source: Table 11 · p. 73, Worked examples by harm level · p. 74
• Table 12 presents chatbot-utterance examples for relationship breakup in the social-relationships rubric.The table illustrates how the rubric is applied to a representative relationship subdomain.Source: Table 12 · p. 74, Worked examples by harm level · p. 75
• Table 13 presents careers examples for job-search strategies and records violations across Accuracy, Completeness/Confidence, and Ethics/Safety.For each harm level, the expert panel drafted correctly and incorrectly levelled examples, showing the first correctly levelled example.Source: Table 13 · p. 75, Worked examples by harm level · p. 75