Source-linked AI summary
PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support
Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, Minlie Huang
TL;DR
Chinese mental-health research lacks suitable counseling corpora, especially for long support responses. The paper introduces PsyQA, a large Chinese question–answer dataset with theory-based strategy annotations and evaluates strategy-aware generation. Strategies improve generated-answer quality, but models remain substantially below human professionals and require further research.
Problem
Research on Chinese text-based mental-health counseling is limited by a lack of relevant corpora.
Method
The paper collects PsyQA from a Chinese mental-health platform, annotates part of its answers with support strategies, analyzes strategy patterns, and evaluates strategy identification and answer generation.
Results
Using support strategies improves generated answers’ fluency, coherence, topicality, and helpfulness, while contextual information benefits strategy identification.
Takeaways & Limitations
PsyQA provides a foundation for studying long Chinese counseling responses and organized support-strategy use, while generation remains challenging.
Takeaways & Limitations
The dataset comes from an online mutual-help forum rather than professional counseling, and purely generative systems may produce fabricated or inaccurate information, so deployment requires great care.
Abstract
from arXiv · showhide
Great research interests have been attracted to devise AI services that are able to provide mental health support. However, the lack of corpora is a main obstacle to this research, particularly in Chinese language. In this paper, we propose PsyQA, a Chinese dataset of psychological health support in the form of question and answer pair. PsyQA is crawled from a Chinese mental health service platform, and contains 22K questions and 56K long and well-structured answers. Based on the psychological counseling theories, we annotate a portion of answer texts with typical strategies for providing support, and further present in-depth analysis of both lexical features and strategy patterns in the counseling answers. We also evaluate the performance of generating counseling answers with the generative pretrained models. Results show that utilizing strategies enhances the fluency and helpfulness of generated answers, but there is still a large space for future research.
1 Introduction
PsyQA addresses the shortage of Chinese mental-health counseling corpora by collecting long question–answer exchanges and annotating support strategies. Experiments show contextual information aids strategy identification and explicit strategies improve generated answers, although models remain below human professionals.
- 1 Introduction: PsyQA fills a Chinese-language corpus gap with long, structured counseling answers from trained volunteers or professional counselors.Questions include descriptions and keywords, while answers average 524 words and provide analysis plus guidance.
- 1 Introduction: The dataset covers nine mental-health topic categories and includes support-strategy annotations for a portion of answers.The annotations support analysis of lexical features and strategy organization patterns.
- 1 Introduction: Contextual information greatly benefits support-strategy identification, while using support strategies improves generated answers’ fluency, coherence, topicality, and helpfulness.The experiments evaluate both strategy identification and answer generation.
- 1 Introduction: Generated counseling answers still lag substantially behind answers written by well-trained volunteers or professional counselors.The paper characterizes answer generation as challenging and leaves substantial room for future research.
2 Related Work
Prior work studies mental-health language, detection, and counseling datasets, but PsyQA focuses on organized strategy development in long Chinese responses across broad topics.
- 2 Related Work: PsyQA represents counseling strategies as a skeleton for generating whole long responses, extending prior work on implicit counselor behaviors.Its strategy organization is informed by patterns observed in high-quality answers and by Hill’s Helping Skills System.
- 2 Related Work: Existing mental-health corpora often come from general social networks or focus on single domains, creating concerns about irrelevant content, unprofessional responses, and narrow coverage.Prior counseling datasets also include forum-based resources and large-scale multi-turn conversations.
- 2 Related Work: Unlike Crisis Text Line’s multi-turn counseling conversations, PsyQA targets Chinese long-text responses in single-turn asynchronous counseling.The distinction concerns language, response length, and conversational format.
- 2 Related Work: PsyQA spans nine topics, including self-growth, emotion, relationships, treatment, marriage, and career, rather than one disorder domain.The dataset is intended for general mental-health support topics.
3 Data Collection
PsyQA was crawled from Yixinli’s counseling Q&A service, filtered for quality, and partially annotated with theory-based support strategies through trained human labeling and review.
- 3.1 Data Source: Yixinli provides anonymous questions and detailed long-text answers from trained volunteers or professional counselors across abundant mental-health topics.The platform contains more than 0.25 million Q&A pairs before collection and filtering.
- 3.2 Data Cleaning: The collection retained answers longer than 100 words and removed personal information, duplicates, emojis, links, advertisements, and non-support questions.These rules were intended to improve data quality and relevance.
- 3.3 Strategy Annotation: The annotation framework assumes counseling answers follow organized strategy sequences consistent with the Helping Skills System.The selected strategy subset was adapted to online counseling and Yixinli’s supporter guidelines.
- 3.3 Strategy Annotation: Workers labeled one or more consecutive sentence spans with strategies, while unmatched sentences were assigned to Others.The study sampled 4,012 questions and selected their highest-voted answers for annotation.
- 3.4 Annotation Quality Control: Annotation quality was supported by worker training with revised practice examples and examiner checks on a random sample of 200 annotated Q&A pairs.Two Clinical Psychology graduate students reviewed sampled annotations for incorrect labels and consistency.
4 Corpus Analysis
PsyQA spans nine broad mental-health topic categories and exhibits varied, structured use of counseling strategies across long answers. Different strategies show distinct frequencies, lexical associations, positional distributions, and recurring transition patterns.
- 4.1 Statistics: PsyQA covers 9 broad mental-health topics, including self-growth and emotion, with diverse subtopics such as personality improvement and emotion regulation.
- 4.2 Textual Features of Different Strategies: Interpretation and Direct Guidance are the most common annotated strategies, whereas Information and Self-disclosure are relatively rare.
- 4.2 Textual Features of Different Strategies: Information, Self-disclosure, and Interpretation have longer average spans than other strategies, with Information and Self-disclosure requiring external knowledge and background.
- 4.2 Textual Features of Different Strategies: Lexical correlates distinguish strategies, while Information and Self-disclosure have less typical and unique words because their content depends strongly on topics and supporters’ experiences.
- 4.3 Strategy Sequence Analysis: Strategies occupy different answer stages: Restatement is mainly at the beginning, Direct Guidance generally at the end, and Appro. & Reass. at both ends.
- 4.3 Strategy Sequence Analysis: The sequence A&R→Intpn.→Guid.→Intpn.→Guid. is the most common first-five-strategy pattern, accounting for 5.6% of such sequences.
5 Strategy Identification
The study identifies support strategies at sentence level and finds that contextual information improves classification, while performance remains limited for Restatement and Information.
- 5 Strategy Identification: The RoBERTa classifier assigns a support-strategy label to each sentence in a long counseling answer.Experiments compare models that process individual sentences with models that use multiple consecutive sentences.
- 5.3 Experimental Results: Contextual information significantly improves sentence-level strategy classification and helps address sample imbalance.The contextual model outperforms the model without contextual information on macro F1.
- 5.3 Experimental Results: Restatement and Information are the main classification bottlenecks, with F1-scores of 49.38% and 54.68%, respectively.Restatement is difficult without the question, while Information requires additional psychological knowledge.
- 5.3 Experimental Results: The results suggest that contextual information captures connections in strategy sequences, but the gap from human performance leaves substantial room for research.The authors attribute the improvement to recognition of strategy patterns.
6 Answer Generation
The paper generates long counseling answers from question context and tests whether explicit strategy signals improve generation. Strategy signals improve several automatic and human-evaluated qualities, although generated answers remain below professional responses.
- 6 Answer Generation: The generation task produces multi-sentence counseling text from a question, description, and keyword set, with the goal of providing helpful comfort and advice.The input consists of question text, a detailed description, and up to four keywords.
- 6.3 Implementation Details: The models prepend strategy tokens to answer spans, using classifier-predicted strategies together with human-annotated data for training.The strategy-token condition is compared with models trained without explicit strategy labels.
- 6.4 Automatic Evaluation: Strategy signals improve perplexity for all models, improve BLEU and diversity for GPT-2 models, and yield approximately 80% strategy controllability.BLEU, Distinct-1, and Distinct-2 improve for GPT-2, while the corresponding scores slightly decline for Seq2Seq.
- 6.5 Human Evaluation: Human evaluation uses 15 psychology or counseling graduate students to rate fluency, coherence, relevance, and helpfulness on a three-star scale.The evaluation samples 100 test questions and compares two generated answers with the golden answer.
- 6.5 Human Evaluation: Explicit strategy signals improve generated answers’ fluency and coherence, while also improving helpfulness and slightly improving relevance in human evaluation.The authors report a substantial remaining gap between generated answers and those written by well-trained help-supporters.
7 Conclusion and Future Work
PsyQA contributes a Chinese psychological-support dataset with strategy annotations and analyses, while benchmark results show that long counseling-answer generation remains substantially below human performance.
- 7 Conclusion and Future Work: PsyQA is a Chinese psychological-support dataset with strategy annotations based on the Helping Skills System.The paper analyzes lexical features and organizational patterns of support strategies.
- 7 Conclusion and Future Work: Existing models substantially underperform human professionals when generating counseling answers, making the task challenging.The authors identify this gap as evidence of substantial room for future research.
- 7 Conclusion and Future Work: Future work should incorporate more professional knowledge into answer generation and conduct more sufficient evaluation of risks in generated answers.Both directions are identified as crucial by the authors.
Ethical Considerations
The paper treats mental-health counseling data and generated answers as ethically sensitive, using anonymization and risk annotation while warning against uncritical deployment of purely generative systems.
- Ethical Considerations: PsyQA is restricted to researchers who sign agreements requiring adherence to ethical guidelines, while the study protects user anonymity and has IRB approval.The source posts are public and anonymous, and annotators were instructed not to deanonymize or contact users.
- Ethical Implications: Risk annotation categorizes ethical concerns as Inappropriate Guidance, Offensiveness, Risk Ignorance, and Serious Crisis.Risk Ignorance concerns overlooking a crisis in the question, while Serious Crisis concerns potentially causing severe harm such as suicide.
- Ethical Implications: With a threshold of at least two annotators, risky answers included 2 GPTft outputs but none from human answers or GPTft+strategy outputs.The authors interpret strategy control as associated with fewer risks in generated answers.
- Ethical Implications: The dataset comes from an online mutual-help forum rather than professional counseling, and the models may generate fabricated or inaccurate information.The authors recommend caution and suggest strategy analysis or sentence recommendation as safer application modes.
- Ethical Implications: The study explicitly makes no treatment recommendations or diagnostic claims.The authors urge users to examine the ethical implications of generated outputs in real-world applications.
A Question Keywords
PsyQA organizes questions across nine relatively balanced topic categories, with keywords formed from one broad topic and one to three subtopics.
- PsyQA covers 9 relatively balanced topic categories, with Table 9 reporting answer totals and average answers per question for each topic.
- Help-seekers describe problems by choosing one broad topic and 1–3 subtopics as question keywords.
- The dataset’s topics include broad areas such as self-growth and emotion, alongside subtopics such as personality improvement and emotion regulation.
B Reproducibility
The appendix reports training, inference, and strategy-classification settings, plus an example comparing GPT-2 fine-tuned with and without strategy labels against a reference answer.
- GPT-2 is pretrained for 168 hours and fine-tuned on PsyQA for 5 hours, while the Seq2Seq model takes 5 hours to train.
- Inference uses temperature 1.0, top p 0.9, top k 50, repetition penalty 1.5, and maximum length 1024 for nucleus sampling.
- Strategy classification compares RoBERTa models with and without contextual information for each support strategy.
- Table 12 compares GPTft answers generated with and without strategy labels with a golden answer selected as the highest-vote reference.
D Guideline for Human Evaluation
Human evaluation compares generated and golden answers using fluency, coherence, relevance, helpfulness, and ethical risk, with detailed criteria provided in Table 13.
- Human evaluation assesses generated and golden answers on fluency, coherence, relevance, helpfulness, and ethical risk.
- Table 13 provides the detailed guideline used for the human evaluation.