Source-linked AI summary
DocTalkBN: A Novel Dataset of Expert Telemedicine Conversations in Bengali
Anik Saha, Fahmida Sultana Naznin, Sadatul Islam Sadi, Ananya Shahrin Promi, Wahid Al Azad Navid, Rifat Shahriyar
TL;DR
Medical conversational AI lacks authentic expert–patient data, particularly for low-resource languages such as Bengali. The paper introduces and curates DocTalkBN from broadcast telemedicine conversations, then benchmarks three downstream tasks. The dataset preserves real conversational characteristics, while clinically grounded reasoning tasks such as triage remain challenging.
Problem
Authentic expert–patient conversational datasets remain scarce, especially for low-resource languages such as Bengali, limiting available evidence for reliable medical conversational AI.
Method
The paper constructs DocTalkBN from nationally broadcast Bangla telemedicine programs and uses LLM-assisted curation followed by multi-annotator human validation for three downstream tasks.
Results
Clinically grounded reasoning tasks such as medical triage remain challenging for current models, with the best reported triage F1 score reaching 0.463.
Takeaways & Limitations
DocTalkBN is a resource for advancing reliable medical NLP and safer healthcare technologies for Bengali and other low-resource languages.
Takeaways & Limitations
The current study focuses on text-based evaluation despite the corpus’s multimodal audio and text data.
Abstract
from arXiv · showhide
Reliable medical conversational AI requires authentic expert--patient interaction data, yet such datasets remain scarce, especially for low-resource languages such as Bengali. We present DocTalkBN, a large-scale multimodal dataset of real-world expert telemedicine conversations in Bengali, collected from nationally broadcast telemedicine programs featuring board-certified physicians. DocTalkBN contains 557.63 hours of paired audio and text, 1,515 multi-turn patient calls, 10,274 host--doctor question--answer exchanges, totaling 1.7M tokens, spanning 26 medical specialties. Unlike prior resources derived from medical forums, written health content, or synthetic data, our dataset preserves the spontaneity, contextual richness, and spoken characteristics of authentic medical interactions in a low-resource setting. To support benchmark-driven research, we further construct three downstream tasks from the corpus, medical triage classification, advice safety evaluation, and medical named entity recognition, and benchmark a diverse set of large language models and encoder-based baselines. Our results show that DocTalkBN is a practically useful resource, particularly for clinically grounded reasoning tasks. We release this resource to facilitate future research on reliable medical NLP and safer, more culturally grounded healthcare systems for low-resource languages. Our source codes and dataset are publicly available at https://anonymous.4open.science/r/doctalk.
1 Introduction
DocTalkBN addresses the scarcity of authentic, expert-grounded medical dialogue data in Bangla by preserving spontaneous multimodal telemedicine interactions and supporting clinically meaningful benchmarks.
- Dataset gap and contribution: Authentic conversational cues such as digressions, omissions, hesitations, and regional expressions are difficult to automate but inform clinicians’ safe recommendations under uncertainty.Preserving acoustic nuances and linguistic authenticity is therefore central to clinically grounded conversational AI.
- Dataset gap and contribution: Existing medical NLP resources often remove spontaneity, prosodic cues, and expert grounding by relying on structured text, forums, or de-identified narratives.Recent doctor–patient dialogue collections also tend to be limited in scale, text-only, or narrowly focused.
- Dataset gap and contribution: DocTalkBN is introduced as a large-scale multimodal dataset of expert-grounded Bangla doctor–patient interactions.It is sourced from nationally broadcast telemedicine programs featuring board-certified specialists.
- Dataset gap and contribution: The corpus contains authentic, unscripted clinical interactions with rich linguistic and acoustic characteristics.Its source programs feature pre-vetted physicians delivering unscripted, real-time advice.
- Benchmarks: Three downstream benchmarks cover medical triage classification, advice safety evaluation, and medical named entity recognition.The benchmarks are constructed through LLM-assisted processing and multi-annotator human validation.
2 Related Work
Related work spans clinical dialogue documentation, medical dialogue corpora, task-oriented systems, and Bangla biomedical resources, but the cited resources address different modalities and tasks.
- Clinical dialogue resources: Clinical NLP resources model patient–provider conversations for structured documentation and dialogue summarization.Examples include MTS-Dialog, MEDIQA-Chat, NoteChat, and related summarization work.
- Medical dialogue datasets: Large-scale medical dialogue datasets include MedDialog, RealMedDial, and HealthCareMagic, while task-oriented systems use resources such as PriMock57, ChiCCo, CLINIC150, and MedDG.The related work therefore covers both general dialogue corpora and task-oriented healthcare systems.
- Bangla resources: Bangla resources include biomedical NER datasets, conversational summarization data, healthcare chat corpora, paraphrases, and clinical symptom reports.Named resources include BanglaBioMed, Bangla-HealthNER, and BanglaCHQ-Summ.
3 Dataset Statistics
DocTalkBN combines real Bangla telemedicine audio and text with substantial dialogue coverage across patient calls, host–doctor exchanges, and medical departments.
- Dataset comparison: Most prior medical dialogue datasets derive from online text, whereas DocTalkBN contains audio and text from live Bangla doctor–patient consultations.This distinction preserves the natural complexity of real clinical interactions in a low-resource setting.
- Corpus scale: 1,934 videos provide 557.63 hours of cumulative duration in the dataset.The videos come from the telemedicine collection described in the dataset statistics.
- Corpus coverage: Dialogues span 2 to 14 turns and cover 26 distinct medical departments.These statistics characterize the conversational length and specialty breadth of the corpus.
4 Methodology
DocTalkBN is constructed from nationally broadcast Bangla telemedicine videos through automated filtering, transcription preprocessing, and human validation. The resulting paired audio-text corpus supports three curated downstream tasks: triage classification, advice safety evaluation, and medical named entity recognition.
- Data Collection: Candidate videos are retrieved from national television telemedicine programs, filtered for medical relevance, and assigned specialty tags with Qwen3-30B-Instruct.The collection covers specialties including cardiology, dermatology, and gastroenterology.
- Preprocessing: Automatically generated Bangla subtitles are segmented, stripped of non-conversational content, and normalized before five annotators validate transcription fidelity and medical content.Validation also checks speaker continuity and major transcription issues.
- Preprocessing: Gemini-3-Flash reconstructs structured conversations by identifying speakers, timestamps, turns, and interaction type, producing paired audio clips and expert-grounded medical conversations.Interactions are classified as doctor–patient dialogues or host–doctor question–answer exchanges.
- Downstream Task Curation: Three downstream datasets are generated with Gemini-2.5-Flash and human validation, including medical triage, advice safety, and named entity recognition.The curation process includes iterative prompt refinement, three-annotator review, revision of low-quality samples, and 17 human corrections.
- Medical Triage: The triage task maps final doctor recommendations to four care-level categories while evaluating models using only patient profiles inferred before the final recommendation.The categories are self-care, routine outpatient visit, investigation or specialist referral, and urgent emergency care.
- Advice Safety and Entity Recognition: Advice safety classifies individual actions as safe or harmful, and named entity recognition extracts seven medical entity categories from curated Bangla conversations.The seven categories include symptoms, diseases, drugs, tests, treatments, anatomy, and specialties.
5 Experiments on Downstream Tasks
The experiments evaluate three downstream tasks using LLMs and encoder baselines under the paper’s stated prompting and training settings. Results show that triage remains difficult, advice safety is comparatively strong, and medical NER remains challenging with limited few-shot gains.
- Experimental Setup: The experiments use an 80:10:10 train-validation-test split, standardized LLM decoding, few-shot prompts with 3–5 examples, and fine-tuned BERT-based baselines.Open LLMs run locally when feasible; other models use the OpenRouter API.
- Evaluation Metrics: Medical triage is evaluated as four-way classification with macro-F1, while advice safety uses binary SAFE/HARMFUL classification with the same metric.Macro-F1 averages class-wise F1 values equally and is used for imbalanced triage data.
- Evaluation Metrics: Strict medical NER evaluation requires exact span-boundary and entity-label matches, reporting macro-F1 across seven entity categories.Precision and recall use correctly predicted, predicted, and gold entity counts.
- Medical Triage: 0.463 is the best triage F1, achieved by Llama 3 70B Instruct zero-shot; few-shot prompting generally improves models, except for this model.GPT-4o is second-best at 0.448 few-shot, while DeepSeek V3.2 and Qwen 3.5 Flash reach 0.383 and 0.416 at their best few-shot scores.
- Advice Safety Evaluation: 0.976 is GPT-4o’s best advice-safety macro F1, with all models exceeding 0.92 and most gaining modestly from few-shot examples.Llama 3 70B Instruct is an exception, decreasing from 0.930 zero-shot to 0.924 few-shot.
- Medical Named Entity Recognition: 0.743 is Qwen 3.5 Flash’s best strict NER F1, narrowly above BanglaBERT’s 0.742, while few-shot prompting does not improve evaluated models.BanglaBERT substantially surpasses mmBERT’s 0.707 fine-tuned performance.
6 Findings and Error Analysis
Medical triage is the hardest benchmark task, while advice safety is easier; NER errors commonly reflect both overprediction and incomplete entity spans.
- Medical triage is the hardest task, with only moderate performance even from the strongest models.The difficulty likely reflects broader clinical reasoning over incomplete, conversational patient histories.
- Medical triage: Models frequently confuse routine outpatient visits with investigation or specialist referrals.The distinction depends on the implied escalation level in the doctor's recommendation.
- Advice safety is comparatively easier for current models because decisions often depend on direct local cues.
- Medical named entity recognition: NER models often over-predict generic medical words as entities, indicating reliance on surface-level medical associations.Examples include disease, treatment, and medicine, which are not necessarily entities under the annotation rules.
- Medical named entity recognition: NER models also frequently extract only part of an entity, making boundary errors a major challenge.
7 Conclusion
The paper introduces DocTalkBN, a large-scale Bengali dataset of authentic expert telemedicine conversations, and derives three benchmark tasks from it. Results indicate that current models handle some dialogue-understanding aspects strongly, but clinically grounded triage reasoning remains challenging.
- DocTalkBN is a large-scale multimodal dataset of real-world expert telemedicine conversations in Bengali.It preserves linguistic variability, conversational structure, and spoken characteristics from nationally broadcast programs.
- The corpus supports medical triage classification, advice safety evaluation, and medical named entity recognition benchmarks.The study evaluates diverse large language models and encoder-based baselines on these tasks.
- Clinically grounded reasoning tasks such as triage remain challenging, although current models perform strongly on some aspects of medical dialogue understanding.
8 Limitations and Future Work
The current study benchmarks text-based models, while the paired audio-text corpus supports future work in Bangla speech recognition, advice generation, and broader medical reasoning.
- Limitations: The current study focuses on text-based benchmarking with zero-shot, few-shot, and encoder fine-tuning evaluations.
- Future Work: Paired audio and text enable future research on medically grounded Bangla automatic speech recognition in low-resource settings.
- Future Work: DocTalkBN can support advice-generation evaluation and broader studies of medical reasoning in low-resource settings.The proposed advice-generation direction involves producing safe, appropriate recommendation units from patient profiles.
9 Ethical Considerations
The study reports institutional ethical approval for collecting publicly accessible YouTube data and anonymized identifying information from patients and doctors.
- Institutional ethics approval covered data collection from publicly accessible YouTube videos.
- Identifying information of patients and doctors was anonymized.
A.1 Hyperparameter details for finetuning
This section presents the hyperparameter details used for finetuning.
- The section documents hyperparameter details for finetuning.
- Finetuning hyperparameters are presented as implementation information.
- The passage identifies finetuning configuration as the section’s focus.
A.2 Prompts
The appendix describes prompts for healthcare content processing, medical conversation parsing, triage classification, medical named entity recognition, and advice-safety evaluation. These prompts specify structured inputs, constrained labels, and JSON outputs for reproducible dataset curation and evaluation.
- Healthcare content processing: Healthcare content prompts classify videos as healthcare-related or not and extract medical tags from metadata.
- Medical conversation parsing: Medical conversation parsing prompts convert raw Bangla transcripts into structured medical conversation data.
- Medical triage classification: Triage prompts assign exactly one disposition label based on the overall clinical picture and actionable severity.
- Medical triage classification: Inference triage prompts preserve each patient profile and classify it into one of four medical disposition categories.
- Medical named entity recognition: Medical NER prompts extract exact medical substrings and assign one label from a fixed entity set without normalization.
- Advice-safety evaluation: Advice-safety prompts construct patient profiles and label doctor-grounded recommendations as SAFE or HARMFUL according to clinical appropriateness.