Source-linked AI summary
Overview of the PsyDefDetect Shared Task at BioNLP 2026: Detecting Levels of Psychological Defense Mechanisms in Supportive Conversations
Hongbin Na, Zimu Wang, Zhaoming Chen, Yining Hua, Rena Gao, Kailai Yang, Ling Chen, Wei Wang, Shaoxiong Ji, John Torous, Sophia Ananiadou
TL;DR
Existing emotional-support systems largely overlook the defensive function of seeker utterances, motivating a DMRS-grounded classification task. The paper organizes PsyDefDetect over a newly annotated corpus and analyzes participating systems, finding a best macro F1-score of 0.420 while exposing persistent majority-class bias and remaining headroom.
Problem
The defensive function of seeker utterances remains largely unmodeled in emotional-support systems, despite progress on empathy, strategy selection, and affect understanding.
Method
The paper organizes a DMRS-grounded shared task classifying target seeker utterances from preceding dialogue context into seven DMRS levels plus two auxiliary labels.
Results
The best system achieved a macro F1-score of 0.420, substantially surpassing the strongest fine-tuned baseline at approximately 0.315 macro-F1.
Takeaways & Limitations
Leaderboard analysis identifies class-imbalance sensitivity and systematic High-Adaptive over-prediction as persistent challenges for fine-grained defensive-function classification.
Takeaways & Limitations
PSYDEFCONV is a relatively small English corpus, so results may not generalize to other languages, cultures, clinical settings, or naturally occurring therapy conversations.
Abstract
from arXiv · showhide
We present an overview of PsyDefDetect, the shared task on detecting levels of psychological defense mechanisms in emotional support dialogues, co-located with BioNLP@ACL 2026. Grounded in the clinically validated Defense Mechanism Rating Scales (DMRS) framework, the task asks systems to classify a target seeker utterance, given its preceding dialogue context, into one of nine categories: seven hierarchical DMRS levels plus two auxiliary labels. Participants worked on PsyDefConv, a newly released corpus of 200 dialogues and 2336 help-seeker utterances annotated under DMRS with substantial inter-annotator agreement. The task attracted 172 participants on CodaBench who produced 563 submissions, with 21 teams officially registering their results for the final ranking. The best system achieved a macro F1-score of 0.420, surpassing the strongest fine-tuned baseline reported in the dataset paper by a notable margin, yet leaving clear headroom. Our analysis highlights (i) a persistent tendency to over-predict the majority High-Adaptive class, (ii) a widening gap between accuracy and macro-F1 that reveals class-imbalance sensitivity, and (iii) the value of theory-aware and LLM-based approaches for fine-grained defensive-function classification. We release all task materials and invite the community to continue work on this novel intersection of clinical psychology and NLP.
1 Introduction
PsyDefDetect introduces a DMRS-grounded shared task for classifying defensive functioning in help-seeker utterances during emotional support conversations. The task attracted substantial participation, while results exposed persistent class-imbalance and High-Adaptive over-prediction problems.
- The task addresses an under-modeled aspect of emotional support conversations: the defensive function of help-seeker utterances.
- PsyDefDetect classifies each help-seeker utterance, using preceding dialogue context, into seven DMRS levels plus phatic and under-specified auxiliary labels.The task is grounded in DMRS and uses the PSYDEFCONV corpus, whose annotations achieved substantial agreement.
- The best system achieved a macro F1-score of 0.420, surpassing the strongest fine-tuned baseline at approximately 0.315 macro-F1.
- 172 participants produced 563 leaderboard submissions, with 21 teams officially ranked.
- Leaderboard analysis found severe class-imbalance sensitivity and systematic over-prediction of the High-Adaptive level.The paper presents a system taxonomy and benchmarks submissions against zero-shot and fine-tuned baselines.
2 Background and Related Work
The paper situates PsyDefDetect at the intersection of psychodynamic defense theory and emotional-support dialogue research. It adapts a clinically oriented framework to conversational classification, addressing a dimension largely absent from existing support systems.
- Defense mechanisms and DMRS: DMRS organizes roughly thirty defense mechanisms into seven hierarchical levels of defensive maturity.The hierarchy ranges from Action Defenses at Level 1 to High-Adaptive Defenses at Level 7.
- Defense mechanisms and DMRS: DMRS was designed for longitudinal clinical case formulation, whereas PSYDEFCONV adapts it to conversational text at the level rather than mechanism granularity.The adaptation is intended to improve identifiability and annotation reliability.
- Emotional support conversations: Emotional-support dialogue research has increasingly focused on supporter strategies, empathy, affect modeling, and multi-turn interaction.
- Research gap: The defensive function of seeker utterances remains largely unmodeled in current emotional-support systems.
- Mental health classification: Prior BioNLP and CLPsych shared tasks addressed several clinically motivated problems, but mental-health tasks remained notably underrepresented.
3 Task Description
The task predicts a defense label for a target help-seeker utterance from preceding and current dialogue context. Its nine-label inventory combines seven DMRS levels with No Defenses and Needs More Information, while excluding future turns.
- Given a multi-turn dialogue and target utterance, systems predict one label from seven DMRS levels and two auxiliary categories.
- Level 0 denotes phatic or functional utterances that do not engage with psychological conflict.
- DMRS levels: Levels 1–3 cover Action Defenses, Major Image-Distorting defenses, and Disavowal.Examples include acting out, splitting, denial, rationalization, and projection.
- DMRS levels: Levels 4–6 cover Minor Image-Distorting, Neurotic, and Obsessional defenses.Examples include devaluation, repression, dissociation, intellectualization, and undoing.
- DMRS levels: Level 7 represents High-Adaptive defenses, including affiliation, altruism, humor, self-observation, sublimation, and suppression.
- Level 8 marks cases where context is insufficient for confident labeling, and systems may use only context preceding and including the target utterance.The no-future-turn constraint mirrors the online clinical annotation setup.
4 The PSYDEFCONV Dataset
PSYDEFCONV is a 200-dialogue corpus derived from ESConv and annotated at the seeker-utterance level for DMRS defense functioning. Double-blind expert annotation produced substantial agreement, but the dataset is strongly skewed toward High-Adaptive utterances.
- Construction: PSYDEFCONV contains 200 dialogues sampled from ESConv over the joint distribution of problem types and emotions.The corpus is annotated at the seeker-utterance level for defense functioning.
- Construction: Two psychology- and NLP-trained annotators independently labeled each seeker turn, reaching Cohen’s κ = 0.639 before consensus adjudication.DMRS CO-PILOT reduced mean annotation time by 24.0%.
- Corpus statistics: The corpus contains 4,709 utterances, including 2,336 annotated seeker turns and 2,373 supporter turns, across dialogues averaging 23.5 turns.
- Label distribution: 51.8% of annotated seeker utterances belong to Level 7 High-Adaptive, while Levels 0 and 8 jointly account for 17.4%.The remaining six defense levels occupy roughly one third of the corpus.
5 Evaluation Setup
The evaluation reports standard classification metrics and uses macro F1 over positive classes 1–8 as the official ranking measure to penalize majority-class bias.
- Evaluation Procedure: Participants submitted predictions through CodaBench after downloading the dataset and starter baseline kits.
- Metrics: Macro F1 over positive classes 1–8 is the official ranking metric.Level 0 is excluded because it denotes phatic or functional turns without defensive content; Level 8 remains included because insufficient context is an explicit task decision.
- Metrics: The evaluation also reports Accuracy, Macro Precision, and Macro Recall.
- Baselines: Zero-shot baselines include Gemini 2.5 Pro at 0.5636 accuracy and 0.2599 macro-F1, and DeepSeek-V3.2 with thinking at 0.5572 accuracy and 0.2617 F1.
6 Participating Systems
The participating systems span ensembles, retrieval- and rubric-grounded LLMs, parameter-efficient fine-tuning, encoder models, and theory-informed synthetic augmentation, largely targeting class skew and ambiguous DMRS boundaries.
- Participation: 172 participants produced 563 submissions, with 21 teams officially registering results and 15 contributing system-description papers.
- Shared Challenges: Nearly every team targets the Level 7 High-Adaptive skew and ambiguous DMRS boundaries in short, context-dependent utterances.
- Ensembles and Agents: Top-ranked systems use multi-model ensembles or deliberative-agent architectures to engineer complementary errors rather than rely on a single stronger model.
- Theory-Grounded LLMs: Retrieval- and rubric-grounded systems inject DMRS clinical guidance into prompts to address label ambiguity and extreme-label polarization.
- Parameter-Efficient Fine-Tuning: Many mid-to-top-ranked systems apply QLoRA or related parameter-efficient fine-tuning to open LLMs, often with augmentation or calibration for minority classes.
- Alternative Approaches: Other teams explore encoder-only transformers, domain-specific encoders, synthetic data, and structured clinical features as alternatives or complements to LLM fine-tuning.
7 Results
Leaderboard results show strong but uneven progress over established baselines, while accuracy can obscure minority-class failures caused by the dominant High-Adaptive class.
- Official Leaderboard: 0.4200 F1 is the top leaderboard score, ahead of UTS at 0.4055, while scores span 0.063–0.420.
- Accuracy and Macro F1: Figure 1 compares accuracy with macro F1 across submitted systems, highlighting cases such as ZZUNLP at 0.676 accuracy and 0.330 F1.
- Baseline Comparison: 0.420 macro F1 surpasses the best fine-tuned baseline, Ministral-8B at F1 = 0.315, by approximately 10.5 absolute points.
- Performance Variation: Macro F1-scores range from 0.063 to 0.420, a 6.7× difference, with the top quartile at ≥0.37 and the median near 0.25.
- Class Imbalance: Approximately 52% of the dataset belongs to Level 7, allowing High-Adaptive over-prediction to support accuracy while reducing minority-class recall and macro F1.
8 Analysis and Discussion
Across submitted systems, High-Adaptive is the easiest class to identify but also a dominant source of error, while stronger approaches trade majority-class recall against minority-class recovery. The leaderboard analysis favors ensembles and task-specific, theory-aware supervision over zero-shot prompting alone.
- Per-Class Performance: Level 7 (High-Adaptive) has the highest median per-class F1 and the narrowest inter-team spread, while Levels 1–6 show wider variation.High-Adaptive comprises approximately 52% of the corpus, whereas overlapping cues make Levels 2–4 especially variable.
- Leaderboard: The official winning score is macro-F1 0.4200, exceeding the strongest fine-tuned baseline by ≈10.5 absolute points.The leaderboard ranks systems by macro-F1 over positive classes 1–8.
- Error Patterns: All four top-ranked systems systematically over-predict High-Adaptive, but their design choices trade C7 errors against minority-class recall.LINGUIUTICS improves Level 8 recovery with more conservative C7 predictions, while PERCEPTIONLAB reaches Level 1 recall of 19/28 through dynamic DMRS-Q retrieval.
- What Worked, What Did Not: The two highest-scoring systems explicitly engineer ensemble or deliberative-agent diversity, while no primarily zero-shot system exceeds F1 = 0.30.Top-performing systems combine prompting with fine-tuning, retrieval, or ensembling; clinically grounded prompts alone are insufficient.
9 Future Directions
The paper identifies four next steps that extend PSYDEFDETECT from utterance-level classification toward broader, multilingual, interactive, and generative clinical-dialogue applications.
- Future Directions: Future work should model dialogue-level defensive trajectories rather than treating each seeker utterance independently.This builds on emotion-level trajectory analysis in the dataset paper.
- Future Directions: Multilingual PSYDEFCONV could extend DMRS-based annotation to Chinese, Japanese, Spanish, and other languages.DMRS CO-PILOT is proposed to reduce the clinical-annotation cost of cross-lingual scaling.
- Future Directions: Jointly modeling seeker defenses and supporter strategies could connect defensive-function detection with adaptive emotional-support interaction.This direction closes the loop with the broader emotional-support-conversation research program.
- Future Directions: Theory-aware generation could condition supportive responses on predicted defense levels, turning the benchmark into a component of clinically informed dialogue systems.The paper specifically frames this as a seeker-model → supporter-response direction.
10 Conclusion
PSYDEFDETECT establishes a DMRS-grounded shared task for detecting psychological defense levels in emotional-support dialogue. It attracted broad participation, achieved a substantial benchmark gain, and leaves minority-class challenges for continued work.
- Conclusion: PSYDEFDETECT is the first shared task to operationalize DMRS-based defensive functioning as utterance-level classification in emotional-support dialogue.
- Conclusion: The task attracted 172 CodaBench participants, 21 ranked teams, and 15 system description papers spanning 12 countries.
- Conclusion: The winning system reached macro-F1 = 0.420, approximately 10.5 points above the strongest fine-tuned baseline.The paper reports clear remaining headroom on minority classes where the L7 attractor dominates.
- Conclusion: The released task materials support continued research at the intersection of clinical psychology and NLP.
Limitations
The task’s conclusions are bounded by its small English source corpus, DMRS-level labeling, contextual ambiguity, and incomplete methodological coverage of leaderboard entries.
- Limitations: PSYDEFCONV is a relatively small corpus derived from English ESConv dialogues, limiting generalization across languages, cultures, clinical settings, and naturally occurring therapy.
- Limitations: DMRS-level labels improve annotation reliability but collapse clinically meaningful distinctions between individual defense mechanisms.
- Limitations: Short seeker utterances can remain ambiguous without broader personal or longitudinal information, even after trained annotation and adjudication.
- Limitations: Methodological analysis covers the 15 teams with system description papers, while other leaderboard entries remain quantitatively ranked but less interpretable.
- Intended Use: The dataset and resulting models are intended for research rather than diagnosis or clinical, legal, or employment decisions.Their English ESConv-derived domain also limits generalization across cultures and clinical settings.