Source-linked AI summary
Assessing Suicide Risk in Arabic Crisis Helpline Calls: A Comparison of Arabic and English Large Language Models
Linhai Ma, Rita El Hachem, Mahatab El Hajj, Lilian Ghandour, Samah Fodeh
TL;DR
Suicide-risk NLP has limited evidence for Arabic crisis helpline calls and must respect the privacy of real callers. This study classified risk from de-identified Lebanese helpline transcripts using Arabic models and English translations, finding useful high-risk performance and greater difficulty with lower-severity ideation. The results support further evaluation as an operator-facing decision-support tool, while remaining bounded by single-helpline data and other sampling and transcription limitations.
Problem
Suicide-risk NLP has little evidence for Arabic-language helpline calls and real-service settings, despite the need to support time-pressured operator assessment while protecting sensitive recordings.
Method
The study kept audio inside Lebanon’s helpline, locally transcribed and de-identified Levantine Arabic speech, translated transcripts into English, and fine-tuned decoder and encoder models for two C-SSRS-derived binary tasks.
Results
High-risk calls separated more cleanly than at-risk calls in both languages, and translating Arabic transcripts into English did not reduce the best observed performance.
Takeaways & Limitations
High-risk classification warrants further evaluation as an operator-facing decision-support tool, while lower-severity ideation remains the harder case.
Takeaways & Limitations
The data came from one helpline in one country over a limited period, and the study could not establish performance on other services, dialect regions, or times.
Abstract
from arXiv · showhide
Crisis helplines assess suicide risk through structured interviews, a process that is slow and dependent on operator training and workload. Natural language processing could support risk assessment and call prioritization, but almost no work addresses Arabic-language helpline calls or operates within the privacy constraints of real helpline data. We analysed de-identified transcripts from Lebanon's National Lifeline for Emotional Support and Suicide Prevention. Audio never left the helpline: calls were transcribed on site with a speech recognition model for Levantine Arabic, and an Arabic named-entity recognition model removed identifying information locally. Only the de-identified transcripts were shared with the research team. Operators recorded the five suicidal ideation items of the Columbia Suicide Severity Rating Scale, which we combined into two binary outcomes: at-risk and high-risk. We also machine-translated the transcripts into English, giving a paired Arabic/English comparison. On each corpus, we fine-tuned five instruction-tuned large language models alongside six transformer encoder baselines (four Arabic, two English) and evaluated all models on a held-out test set. We included 383 calls: 373 for the at-risk task (52.3% positive) and 297 for the high-risk task (30.0% positive). The best Arabic model reached a macro-F1 of 81.19 and a ROC-AUC of 90.61 on high-risk; the best English model reached 85.00 and 92.59, identifying 88.9% of high-risk calls. In both languages, high-risk calls separated more cleanly than at-risk calls, and translation to English did not reduce the best observed performance. Suicide risk can be classified from de-identified Arabic transcripts without sending audio outside the helpline. The high-risk results support further testing as an operator-facing tool; lower-severity ideation proved the harder case.
1 Introduction
This study addresses the lack of evidence on Arabic-language suicide-risk assessment from real helpline calls while preserving caller privacy. It develops a within-service transcript pipeline and compares Arabic models with English models applied to translations of the same calls.
- Motivation: Helpline operators assess risk during calls using structured questionnaires under time pressure, and existing questionnaires are not accurate enough to use alone.The C-SSRS is commonly used for this assessment, but a systematic review found insufficient standalone accuracy.
- Research gap: Most suicide-risk NLP research uses social media posts or clinical notes rather than real helpline conversations.Prior helpline studies have focused on English- and Chinese-language services, while an Arabic-speaking service was analysed only through Hebrew chats.
- Study setting: Lebanon’s national helpline provides recorded speech matched with supervised C-SSRS assessments, creating data suitable for model development.The service is the country’s only suicide-prevention helpline and operates in a context of substantial mental-health need and social taboo.
- Privacy-preserving design: The pipeline keeps recordings inside the helpline and sends researchers only de-identified Arabic transcripts.Speech recognition and named-entity removal run within the service because voice recordings cannot be edited to conceal identity while preserving confidentiality.
- Deployment concern: The analysis examines model failures under imbalanced risk labels because missed positive cases affect whether operator-facing use is safe.The authors frame models as support for operators rather than autonomous risk decision-makers.
- Study aims: The study compares Arabic transcript classification with classification after translating the same calls into English.Both language corpora use the same calls, labels, and model-testing framework, enabling a paired comparison.
2 Methods
The study constructs privacy-preserving Arabic and translated English corpora from supervised helpline assessments, defines two severity-based binary tasks, and evaluates decoder and encoder models on matched held-out splits. Missing or incomplete records are excluded when outcomes cannot be determined, reducing false-negative risk in the analysis.
- Data collection: Calls lasted five to ten minutes, were answered by trained operators, and received standard C-SSRS assessments recorded in a service database.The analysis used routine service data without caller contact, with ethical approval obtained before access.
- Privacy-preserving preprocessing: Recordings remained inside the helpline while on-site speech recognition and Arabic named-entity recognition produced redacted Levantine Arabic transcripts.Only de-identified text was shared with the research team.
- Outcome definition: The five C-SSRS ideation items were grouped into at-risk and high-risk tasks, with the lower two items defining at-risk and any higher item defining high risk.The two categories overlap but are modelled as separate yes-or-no outcomes.
- Missing annotation: Unclear outcomes were excluded when all recorded items were absent and at least one item was unrecorded.This removed 32 at-risk and 16 high-risk calls rather than treating missing items as absent, which could introduce false negatives.
- Translation: Each de-identified Arabic transcript was machine-translated into English, and 29 incomplete translations were removed from both language corpora.Removing the same calls preserved comparability between Arabic and English datasets.
- Model training: Five instruction-tuned decoder LLMs and six transformer encoders were fine-tuned separately for each task and language using the same 4-bit QLoRA recipe.Decoders produced a single-word yes-or-no answer, while encoders processed transcripts in overlapping windows.
- Evaluation: Evaluation used call-level stratified 80:20 training-to-test splits shared across Arabic and English, with ROC-AUC, PR-AUC, macro-F1, recall, and accuracy reported.The matched splits ensured that the language comparison differed only in language.
3 Results
Across held-out tests, fine-tuned decoders generally outperformed encoder baselines, with strongest performance on high-risk classification. English translation preserved or improved best observed results, while lower-severity at-risk classification remained harder and model comparisons were limited by small test samples.
- Dataset composition: 383 calls were included overall: 373 for at-risk classification and 297 for high-risk classification.The at-risk and high-risk datasets were derived from different call subsets.
- Primary models’ performance: The best Arabic decoder reached 81.19 macro-F1 and 90.61 ROC-AUC on high risk, while the English decoder reached 85.00 and 92.59.These were primary 70B decoder results on held-out test splits.
- Operating characteristics: 88.9% of high-risk calls were identified by the fine-tuned English model, compared with 66.7% for the Arabic model.The English model found 16 of 18 high-risk calls; the Arabic model found 12.
- Better separation for more severe cases: The best model in each language scored higher on high risk than at risk, despite high risk being rarer.The best Arabic macro-F1 values were 81.19 versus 78.63, and the best English values were 85.00 versus 78.66.
- Primary models’ performance: The best fine-tuned decoder beat the best encoder baseline on every task and language.Arabic encoder versus decoder macro-F1 was 69.25 versus 78.63 on at risk and 72.82 versus 81.19 on high risk; English was 64.57 versus 78.66 and 71.29 versus 85.00.
- Model performance under class imbalance: Macro-F1 alone could obscure opposite failure modes under class imbalance, so positive-class recall and PR-AUC should also be reported.Some models predicted no positives, while others flagged almost every call.
4 Discussion
The study extends suicide-risk detection to Levantine Arabic crisis calls through a privacy-preserving pipeline and paired English comparison. High-risk classification was more separable than lower-severity ideation, while practical deployment requires prospective evaluation and workload assessment.
- Contribution: The study extends prior suicide-risk NLP work from social media and clinical notes to Levantine Arabic crisis calls using speech content.It also differs from helpline studies using text counselling, acoustic features, or structured operator fields.
- Contribution: A privacy-preserving pipeline kept recordings inside the helpline while enabling classification from de-identified transcripts.The paper frames this design as addressing real-world, non-English crisis research without transferring raw recordings.
- Cross-language comparison: English translation preserved classification performance, suggesting a possible route for applying English-language models to Arabic crisis calls, pending validation beyond this dataset.
- Risk categories: Lower-severity ideation was harder to distinguish than high-risk ideation, indicating that clinically meaningful but indirect wording remains a difficult detection case.The paper links this difficulty to overlap between lower-severity ideation and general distress language.
- Helpline practice: Before practice use, models should be evaluated on consecutive incoming calls and their expected additional weekly reviews should be estimated.These steps are intended to clarify prospective performance and workload implications before deployment.
- Evaluation: Macro-averaged metrics should be reported alongside positive-class recall and PR-AUC because macro-F1 alone can obscure positive-class performance.Trivial classifiers had similar macro-F1 and accuracy, while positive-class recall separated them.