Source-linked AI summary
Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages
Mohammadreza Ghaffarzadeh-Esfahani, Nahid Yousefian, Ebrahim Heidari-Farsani, Ali Akbar Omidvarian, Sepehr Ghahraei, Atena Farangi, AmirBahador Boroumand
TL;DR
Clinical extraction in low-resource languages is constrained by scarce annotated data, privacy concerns, and deployment costs. This study evaluates a translation-augmented, few-shot pipeline using open-source SLMs on Persian palliative-care transcripts, finding that larger models performed better and English translation improved sensitivity while reducing missing outputs.
Problem
Low-resource clinical NLP lacks sufficient annotated data and privacy-preserving, computationally accessible approaches for extracting information from medical transcripts.
Method
The study benchmarks five open-source SLMs in a translation-augmented, few-shot pipeline for binary extraction of 13 features from 1,221 annotated Persian transcripts.
Results
Larger models showed superior sensitivity and MCC, while English translation improved sensitivity and reduced missing outputs but modestly lowered specificity.
Takeaways & Limitations
Moderately sized open-source SLMs with careful prompting provide a practical privacy-preserving pathway for clinical extraction under infrastructure and data-governance constraints.
Takeaways & Limitations
Generalizability is limited by the small dataset, machine-translation noise, and evaluation within a single palliative-care oncology domain.
Abstract
from arXiv · showhide
Extracting clinical information from medical transcripts in low-resource languages remains a significant challenge in healthcare natural language processing (NLP). This study evaluates a two-step pipeline combining Aya-expanse-8B as a Persian-to-English translation model with five open-source small language models (SLMs) -- Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Qwen2.5-1.5B-Instruct, and Gemma-3-1B-it -- for binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts collected at a cancer palliative care call center. Using a few-shot prompting strategy without fine-tuning, models were assessed on macro-averaged F1-score, Matthews Correlation Coefficient (MCC), sensitivity, and specificity to account for class imbalance. Qwen2.5-7B-Instruct achieved the highest overall performance (median macro-F1: 0.899; MCC: 0.797), while Gemma-3-1B-it showed the weakest results. Larger models (7B--8B parameters) consistently outperformed smaller counterparts in sensitivity and MCC. A bilingual analysis of Aya-expanse-8B revealed that translating Persian transcripts to English improved sensitivity, reduced missing outputs, and boosted metrics robust to class imbalance, though at the cost of slightly lower specificity and precision. Feature-level results showed reliable extraction of physiological symptoms across most models, whereas psychological complaints, administrative requests, and complex somatic features remained challenging. These findings establish a practical, privacy-preserving blueprint for deploying open-source SLMs in multilingual clinical NLP settings with limited infrastructure and annotation resources, and highlight the importance of jointly optimizing model scale and input language strategy for sensitive healthcare applications.
1. Introduction
Clinical NLP converts unstructured medical text into structured information but faces data scarcity, linguistic challenges, and privacy concerns. This study addresses these gaps with a two-step translation-and-extraction pipeline using open-source models on anonymized Persian palliative-care transcripts.
- NLP transforms unstructured clinical descriptions into structured, quantifiable information for predictive modeling, evidence-based decision support, patient outcomes, and operational efficiency.
- Traditional rule-based and supervised NLP systems face training-data scarcity and difficulty capturing linguistic nuances, while proprietary LLMs raise data-privacy concerns through API dependence.
- The study designs a two-step pipeline combining Aya-expanse-8B with five open-source SLMs for few-shot binary extraction of 13 clinical features from 1,221 anonymized Persian transcripts.
2. Results
The results evaluate a two-step pipeline for extracting structured clinical information from Persian palliative oncology call transcripts. Analyses examine comparative model performance, feature-specific behavior, translation effects, robustness to class imbalance, and errors.
- Overall comparative performance: The two-step pipeline combines Aya-expanse-8B translation with five open-source SLMs to extract structured clinical information from Persian palliative oncology call transcripts.The evaluation used a manually annotated dataset of 1,221 calls.
- Overall comparative performance: Model capability varied across the evaluation, with parameter count and input language identified as influential factors.The results benchmarked models against manually annotated transcripts.
- Results overview: The results are organized around overall model comparisons, feature-specific analysis, translation effects, imbalance-aware robustness, and error analysis.This multifaceted presentation addresses model behavior and deployment implications.
- Study workflow: The study workflow begins with 1,221 Persian palliative care phone-call transcripts, translates them into English, applies prompted SLM inference, and post-processes structured outputs into tabular data.The framework compares multiple SLMs across several analytical dimensions.
2.1. Qwen2.5-7B-Instruct demonstrated the highest overall performance among the evaluated models
Qwen2.5-7B-Instruct achieved the strongest balanced performance across the evaluated models, leading on specificity, macro-averaged F1-score, precision, and accuracy. Gemma-3-1B-it had the weakest sensitivity and macro-F1, while Aya-expanse-8B performed similarly across English and Persian prompts.
- Overall model performance: Qwen2.5-7B-Instruct achieved the highest median specificity (0.987 [0.975, 0.992]), macro-averaged F1-score (0.899 [0.832, 0.908]), precision (0.814 [0.759, 0.878]), and accuracy (0.96 [0.947, 0.984]).These results indicate the strongest balanced performance across the 13 extracted clinical features.
- Overall model performance: Gemma-3-1B-it recorded the lowest median sensitivity (0.613 [0.294, 0.735]) and macro-averaged F1-score (0.74 [0.676, 0.79]), indicating difficulty detecting positive instances.The reported weakness specifically concerned positive-instance detection.
- Overall model performance: Aya-expanse-8B performed comparably across prompt languages, with macro-averaged F1-score of 0.855 [0.804, 0.87] in English and 0.842 [0.753, 0.868] in Persian.The comparison used different prompt variants for the multilingual model.
2.2. Extraction performance was highly dependent on the type of clinical feature
Extraction performance varied substantially by clinical feature: models were strongest on overt physiological symptoms but diverged on psychological complaints, administrative requests, and complex somatic features. Model scale and input language also shaped performance for difficult features.
- High-performing features: Pain was the top-performing feature, with Llama-3.1-8B-Instruct achieving a macro-averaged F1 score of 0.93.Fever and respiratory complaints also exceeded 0.91 for multiple models.
- High-performing features: Fever and respiratory complaints showed robust performance, with multiple models scoring above 0.91.The strongest models differed by feature: Qwen2.5 variants and Aya-expanse-8B (Persian) for Fever, and Llama models plus Aya-expanse-8B (Persian) for respiratory complaints.
- Variable and challenging features: Performance varied for weakness and fatigue, decreased level of consciousness, and psychological complaints, with Qwen2.5-7B-Instruct reaching 0.909 and Aya-expanse-8B (Persian) reaching 0.859.Smaller models such as Gemma-3-1B-it struggled more on these conceptually related symptoms.
- Variable and challenging features: The doctor’s visit request was notably difficult, ranging from 0.410 for Aya-expanse-8B (Persian) to 0.832 for Llama-3.1-8B-Instruct.Sleep disorders and insurance/treatment cost issues also showed lower and more inconsistent scores.
2.3. Translating Persian transcripts to English enhanced sensitivity and reduced missing outputs
Translating Persian transcripts into English modestly improved Aya-expanse-8B’s balanced performance and sensitivity across 13 clinical features while reducing missing outputs. English processing was more robust for imbalanced extraction, although direct Persian processing retained marginally higher accuracy and specificity.
- Overall comparison: 0.855 macro-averaged F1 and 0.724 MCC were achieved in English, versus 0.842 and 0.686 in Persian.English sensitivity was also higher: 0.901 [0.842–0.911] versus 0.893 [0.818–0.952].
- Feature-level comparison: English produced higher macro-averaged F1 for six features, including insurance/treatment cost issues (0.870 vs. 0.674) and urinary tract issues (0.888 vs. 0.842).Other English-favored features were doctor’s visit request, seizures, weakness and fatigue, and decreased level of consciousness.
- Overall comparison: 28 missing outputs occurred in English, compared with 38 [38, 39] in Persian.Persian missing outputs reached 40 for fever and insurance/treatment cost issues, potentially reflecting difficulty following the structured output template.
- Interpretation: English translation enhanced robustness for imbalanced medical extraction by improving sensitivity, reducing missing outputs, and boosting F1 and MCC, despite marginally higher Persian accuracy and specificity.The comparison covered 13 binary features extracted from 1,221 transcripts using the evaluation metrics described in the methods.
2.4. Larger models demonstrated superior performance under class imbalance
Qwen2.5-7B-Instruct showed the strongest overall robustness to class imbalance, achieving the highest median MCC and consistently high correlations for physiological features.
- Class-imbalance robustness: 0.797 [0.67, 0.819] was Qwen2.5-7B-Instruct’s highest median MCC across seven configurations, indicating superior prediction–label correlation under severe class imbalance.Its physiological-feature MCCs included 0.8502 for fever.
2.5. Larger models favored sensitivity while smaller models achieved comparable or superior specificity
Larger models generally favored sensitivity and more balanced feature coverage, whereas smaller models achieved comparable or higher specificity. Qwen2.5-7B-Instruct had the highest median specificity, while Llama-3.1-8B-Instruct led median sensitivity.
- Sensitivity: 0.909 median sensitivity made Llama-3.1-8B-Instruct the top performer, followed by Aya-expanse-8B (English) at 0.901 and Aya-expanse-8B (Persian) at 0.893.These values summarize performance across 13 binary clinical features.
- Specificity: 0.987 median specificity made Qwen2.5-7B-Instruct the strongest model on this metric, with 0.998 for seizures and 0.997 for insurance issues.High specificity was noted as important for avoiding unnecessary escalations in resource-limited environments.
- Specificity: 0.986 and 0.982 median specificities for Gemma-3-1B-it and Qwen2.5-1.5B-Instruct were comparable to or higher than those of the larger Llama models.Llama-3.1-8B-Instruct and Llama-3.2-3B-Instruct achieved median specificities of 0.958 and 0.951, respectively.
- Feature-level trade-offs: 0.902 specificity for pain was higher in Aya-expanse-8B (Persian), but doctor’s visit requests had reduced detection performance at 0.342.The Persian variant showed a feature-level trade-off between specificity and detection performance.
- Model-size trade-offs: Increasing model size enhanced sensitivity and supported more balanced feature coverage, while smaller models maintained comparable or even higher specificity.The study therefore identified a consistent sensitivity–specificity trade-off across model sizes.
3. Discussion
The discussion presents open-source SLMs in a translation-augmented, few-shot pipeline as a practical approach for privacy-preserving clinical information extraction in Persian palliative care transcripts. It emphasizes larger-model advantages, reliable physiological symptom extraction, deployment simplicity, and limitations involving dataset size, translation noise, domain specificity, and difficult feature categories.
- Core findings: Open-source SLMs extracted structured clinical information from Persian palliative care transcripts without fine-tuning, with larger instruction-tuned models showing higher MCC and sensitivity under class imbalance.Common physiological symptoms were extracted reliably, while complex somatic features remained more challenging.
- Relation to prior work: The study extends prior low-resource clinical extraction work by evaluating smaller open-source models with prompt engineering instead of massive proprietary models and many-shot prompting.This positions the approach as a lower-resource alternative to the cited Turkish mammography-report study.
- Methodological contribution: Out-of-the-box few-shot general-purpose SLMs offer a simpler deployment strategy than hybrid fine-tuned, retrieval-augmented, or multi-granularity architectures.The discussion characterizes this simplicity as strategically advantageous for rapid deployment.
- Limitations: The study is limited by its relatively small dataset, machine-translation noise, single-domain palliative oncology setting, and persistent underperformance on administrative and psychological features.Translation may alter subtle semantic meanings, while the domain restriction limits generalizability.
- Implications: The findings establish a proof of concept for privacy-preserving clinical NLP beyond high-resource languages and provide a practical blueprint for systems constrained by infrastructure and data governance.The blueprint combines translation-augmented few-shot prompting with structured extraction without model fine-tuning.
4. Methods
The study benchmarked five open-source SLMs for binary clinical information extraction from 1,221 anonymized Persian palliative-care transcripts. Persian-to-English translation, few-shot structured prompting, complementary metrics, and local inference formed the evaluation pipeline.
- Dataset and annotation: 1,221 anonymized Persian palliative-care calls were independently annotated by two authors and cross-reviewed to establish the evaluation dataset.The dataset came from a cancer-focused palliative care call center.
- Preprocessing and prompting: Aya-expanse-8B translated each Persian transcript into English before few-shot prompting generated structured inputs for five SLMs.The translator supported 23 languages; generation used temperature 0.3 and a 2,048-token limit.
- Preprocessing and prompting: The extraction template contained 13 binary complication and concern fields, with outputs restricted to the predefined field order and True-or-False values.The prompt instructed models to extract only explicitly stated complications without additional text.
- Models and evaluation: The benchmark evaluated Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.2-3B-Instruct, Gemma-3-1B-it, and Qwen2.5-1.5B-Instruct alongside Aya-expanse-8B.Experiments used HuggingFace Transformers with sampling disabled and a 512-token generation limit.
- Models and evaluation: Accuracy, specificity, sensitivity, precision, macro-averaged F1, and MCC measured overall performance, class-wise behavior, and robustness to class imbalance.MCC was used as a stable summary of global agreement under severe class imbalance.
- Privacy and ethics: All model inferences ran locally on a single L4 GPU without external API calls, while the study used pseudonymization, an IRB consent waiver, and Helsinki Declaration compliance.The local setup used a two-core CPU and 8 GB of RAM.
Data availability · Funding
The study reports no funding for either the research or its publication. No data-availability statement is provided in the supplied passage.
- Funding: No funding was received for the study.
- Funding: No funding was received for publication.