PromptSense: Sentiment-Aware Mental Health Chatbot with Dynamic Prompting

• The motivation includes unmet mental-health support needs and barriers such as stigma, financial constraints, and limited access.The introduction reports that 20% of Australians experience mental illness annually and that 31% of Americans fear being judged when seeking support.Source: §1 · p. 1
• Existing mental-health chatbots can lack empathy and contextual relevance, potentially leading to ineffective or harmful interactions.The introduction also notes that LLM chatbots may feel superficial in extended conversations and may give misinformation or incorrect advice.Source: §1 · p. 2
• The study evaluates qualitative and quantitative prompting strategies across base, DSFT, and RLAIF models using AI judges.Evaluation criteria include empathy, helpfulness, clarity, relevance, and safety.Source: §1 · p. 2
• PromptSense integrates sentiment analysis to adapt chatbot interactions to users’ emotional and psychological states.The paper frames the framework as aiming for empathy, contextual relevance, and risk awareness.Source: §1 · p. 2

• The related-work discussion traces LLM development from transformers and GPT-1 to GPT-3 and ChatGPT.It also notes growing mental-health chatbot research and the use of real-world conversational and social-media data.Source: §2 · p. 2
• Prior work has explored fine-tuning general-purpose LLMs for mental-health applications, including conversational agents, diagnostic tools, and therapeutic support.The cited overview says Table 1 lists study objectives and training and evaluation datasets.Source: §2 · p. 2
• The paper states that fine-tuning models such as LLaMA2 and GPT-4 with diverse, well-selected datasets can improve mental-health disorder identification.Source: §2 · p. 3
• The paper characterizes existing mental-health chatbots as potentially lacking empathy, context awareness, and sensitivity to user risks.PromptSense addresses these concerns by incorporating sentiment and depression analysis into prompt engineering.Source: §2 · p. 3

3.1 Proposed model architecture

• The architecture processes user input through sentiment analysis, risk assessment, model inference, and song recommendation modules.A RoBERTa transformer has Mental, Emotion, and Valence-Arousal heads whose outputs inform chatbot responses; song suggestions are personalized using emotional insights.Source: §3.1 · p. 3

3.2 Sentiment Analysis Model (SAM)

• The sentiment model uses MentalRoBERTa with two residual blocks, layer normalization, and dropout to support multi-head outputs.MentalRoBERTa is described as a mental-health-specialized RoBERTa variant.Source: §3.2 · p. 3
• The Mental Head classifies suicidal ideation for potential crisis intervention and is trained with cross-entropy loss.The passage identifies a suicidal-ideation detection dataset [11]; the accompanying prose defines the ground-truth label and predicted probability terms.Source: Figure 1 · p. 4, §3.2 · p. 4

Emotion Head (EH)

• The Emotion Head performs multi-label classification and uses inverse effective-sample-number weights to address GoEmotions class imbalance.The normalized weights are used for positive classes in binary cross-entropy loss.Source: Emotion Head (EH) · p. 4

VA Head

• The VA Head uses regression to predict continuous scalar measures of valence (mood polarity) and arousal (emotional intensity).Its predictions are trained with mean squared error between the ground-truth and predicted continuous values.Source: VA Head · p. 5

Uncertainty Weigh Loss

• The Emotion and VA Heads are jointly trained with Kendall et al.’s loss, which adaptively balances emotion classification granularity against valence-arousal regression stability.The loss uses learnable uncertainty parameters for the two tasks.Source: Uncertainty Weigh Loss · p. 5

3.3 Mental Health LLM Model

• The study uses DeepSeek-R1-Distill-Llama-8B as its base, fine-tunes it on Counsel Chat as DSFT, then refines it through RLAIF judged by Mistral-7B-Instruct-v0.3.Judge metrics include empathy, helpfulness, relevance, clarity, and safety; Fig. 2 is identified as the fine-tuning process.Source: §3.3 · p. 5, Figure 2 · p. 5

3.4 Prompt Engineering

• Quantitative prompting incorporates numerical probability values from the Sentiment Analysis Model directly into prompt construction.The example includes emotional-state probabilities and valence and arousal values, and limits the response to five sentences without reasoning.Source: §3.4 · p. 6
• Qualitative prompting maps probability values to verbal levels from Very Low through Very High for a more intuitive depiction of emotional states.The listed intervals are 0.0–0.2 Very Low, 0.2–0.4 Low, 0.4–0.6 Moderate, 0.6–0.8 High, and 0.8–1.0 Very High.Source: §3.4 · p. 6
• The prompt examples ask the model to respond compassionately with emotional support and thoughtful guidance tailored to the user’s feelings and concerns.The qualitative example describes anxiety, nervousness, and low valence with high arousal; both examples limit output to five sentences without reasoning.Source: §3.4 · p. 7

3.5 Datasets

• The suicidal-ideation dataset contains 54,412 entries across five mental-health classifications, while GoEmotions contains Reddit comments labeled with 27 emotions and Neutral.The passages identify the datasets as sourced from mental-health-related subreddits and Reddit comments, respectively.Source: §3.5 · p. 7
• The NRC VAS Lexicon provides valence-arousal scores for over 55,000 English words, and Counsel Chat contains 2,612 cleaned therapist-conversation scenarios.The NRC scores are used to add quantified emotional dimensions to GoEmotions; Counsel Chat removes personal information.Source: §3.5 · p. 7
• Evaluation uses 418 randomly selected EmpatheticDialogues test conversations to assess contextual and ethical appropriateness.The table passage lists 2,612 Fine Tuning LLM examples and 418 Model Evaluation examples.Source: §3.5 · p. 7

3.6 Evaluation Metrics

• Mental-state classification is evaluated by accuracy, emotion classification by macro-averaged F1, and valence-arousal regression by RMSE.The LLM judges rate empathy, helpfulness, clarity, relevance, and safety; Perspective API measures toxicity, identity attacks, insults, and threats, alongside compassion, respect, nuance, and incoherence.Source: §3.6 · p. 7, Mental Health LLM · p. 8

Mental Health LLM

• Three LLM judges provide evaluations using the same five criteria, including one judge specialized for mental-health scenarios.The judges are Mistral-7B-Instruct-v0.3, Mixtral-8x7B-v0.1, and MentaLLaMA-chat-13B.Source: Mental Health LLM · p. 8

4.1 Sentiment Analysis Model

• 0.6803 mental accuracy (17.5% increase) and 0.4840 emotion F1 (504.2% improvement) are reported for the unfrozen model versus the frozen model.Its valence and arousal RMSE values are 0.2025 and 0.1835, respectively; the text reports improvement over the baseline across all metrics.Source: §4.1 · p. 9, Table 4 · p. 9

AI Judge Evaluation

• +0.12 average gains for PE-Q and PE-P support structured prompting; PE-Q beats PE-P in 6 of 9 evaluations.Base scores 8.73, DSFT 8.68, and RLAIF 8.74 show only modest differences without prompt enhancement.Source: AI Judge Evaluation · p. 9, Table 5 · p. 10
• The general-purpose Mistral judge outperformed MentalLLaMA in every configuration, including Base scores of 9.12 versus 8.44.The reported judge scores cluster narrowly from 8.36 to 9.28, and the authors caution that AI evaluation may miss clinical factors such as therapeutic empathy and emotional appropriateness.Source: AI Judge Evaluation · pp. 9–10, Table 5 · p. 10
• Prompt-enhanced models outperform others in empathy and the paired quality dimension, while PE models show slightly lower relevance scores.The authors also report that BQ, RQ, and DQ cluster closely in the empathy-versus-safety plot, indicating improved emotional tone and reduced safety risks.Source: AI Judge Evaluation · p. 10, Figure 3 · p. 11
• Qualitative prompting is reported as effective for improving empathetic, higher-quality responses, whereas DSFT and RLAIF provide only marginal improvements.The authors identify limitations in AI-based evaluation of emotional appropriateness and long-term psychological impact, and plan evaluations by mental-health professionals and prompt refinement using expert feedback.Source: AI Judge Evaluation · p. 10

Perspective API Evaluation

• Base + PE-Q achieves the lowest toxicity (0.0156) and highest compassion (0.9415), while PE-Q reduces toxicity and incoherence and raises compassion and respect across Base, DSFT, and RLAIF.PE-P also improves outcomes, but less consistently; DSFT and RLAIF offer minimal ethical improvements without prompt enhancement.Source: Perspective API Evaluation · p. 10, Table 6 · p. 11

5 Conclusion and Future Work

• Prompt engineering, particularly qualitative prompting, is reported to enhance empathy and overall response quality, while DSFT and RLAIF offer only marginal improvements.The authors state that current fine-tuning strategies may not capture the nuanced demands of emotionally sensitive dialogue.Source: §5 · pp. 11–12
• AI-based evaluation remains limited for assessing emotional appropriateness and long-term psychological impact; future work will add mental-health-professional evaluations.The authors also plan to refine prompt generation using reinforcement learning from expert feedback.Source: §5 · p. 12