Source-linked AI summary
IndicMedDialog: A Parallel Multi-Turn Medical Dialogue Dataset for Accessible Healthcare in Indic Languages
Shubham Kumar Nigam, Suparnojit Sarkar, Piyush Patel
TL;DR
Medical dialogue systems often lack realistic multi-turn interaction and broad multilingual coverage. This paper introduces a verified, post-processed parallel dataset across English and nine Indic languages and fine-tunes IndicMedLM for multilingual symptom elicitation. Semantic post-processing yields strong results in Hindi, Marathi, and Bengali, while several languages remain in an extreme failure tier, and the authors identify synthetic-to-real validation as a key limitation.
Problem
Existing medical dialogue systems are commonly single-turn, English-only, or template-driven, limiting realistic multilingual consultation modeling.
Method
The paper constructs IndicMedDialog from synthetic consultations translated into nine Indic languages, verified by native speakers, script-corrected, and used to fine-tune IndicMedLM with patient pre-context.
Results
72.76% in Hindi, 68.51% in Marathi, and 58.72% in Bengali are reported after semantic post-processing, while Assamese, Tamil, and Telugu remain in an extreme failure tier.
Takeaways & Limitations
The dataset broadens parallel multi-turn medical dialogue coverage for Indic languages, while error analysis supports semantic evaluation and highlights language-specific safety risks.
Takeaways & Limitations
The gap between synthetic and real patient dialogue distributions remains unquantified, so generalization to real clinical interactions requires validation.
Abstract
from arXiv · showhide
Most existing medical dialogue systems operate in a single-turn question--answering paradigm or rely on template-based datasets, limiting conversational realism and multilingual applicability. We introduce IndicMedDialog, a parallel multi-turn medical dialogue dataset spanning English and nine Indic languages: Assamese, Bengali, Gujarati, Hindi, Marathi, Punjabi, Tamil, Telugu, and Urdu. The dataset extends MDDial with LLM-generated synthetic consultations, translated using TranslateGemma, verified by native speakers, and refined through a script-aware post-processing pipeline to correct phonetic, lexical, and character-spacing errors. Building on this dataset, we fine-tune IndicMedLM via parameter-efficient adaptation of a quantized small language model, incorporating optional patient pre-context to personalise multi-turn symptom elicitation. We evaluate against zero-shot multilingual baselines, conduct systematic error analysis across ten languages, and validate clinical plausibility through medical expert evaluation.
1 Introduction
Existing medical dialogue systems often use single-turn answers or limited English/template-based resources, while IndicMedDialog addresses these gaps with a verified parallel multilingual dataset and IndicMedLM.
- Single-turn systems cannot replicate the follow-up questioning through which clinical diagnosis progressively narrows the differential.
- English-only and template-driven datasets constrain linguistic diversity, conversational realism, and multilingual medical dialogue access for Indic-language speakers.
- IndicMedLM incorporates patient pre-context and structured multi-turn symptom elicitation before diagnosis, contrasting with general-purpose LLM responses lacking follow-up questioning.
- IndicMedDialog spans English and nine Indic languages, extending MDDial with synthetic consultations, native-speaker verification, and script-aware post-processing.
- The work also develops a parameter-efficient medical dialogue model for modest hardware and evaluates failure modes and clinical plausibility.
2 Related Work
Prior work includes multi-turn and iterative medical dialogue systems, but multilingual coverage remains limited and many datasets rely on English or template-based construction. IndicMedDialog combines broader Indic coverage with verification, post-processing, and multidimensional evaluation.
- Medical Dialogue Datasets and Systems: MDDial provides an English differential-diagnosis corpus, but its template-based construction limits conversational diversity and multilingual deployment.
- Medical Dialogue Datasets and Systems: Iterative systems such as AMIE and BianQue frame diagnosis as history-taking, more closely reflecting clinical workflows than predominantly single-turn systems.
- Synthetic Data and Multilingual Coverage: Existing synthetic or bilingual resources generally remain single-language, template-constrained, or limited to narrower language coverage.
- Synthetic Data and Multilingual Coverage: IndicMedDialog provides parallel multi-turn dialogue across nine Indic languages with native-speaker verification and script-aware post-processing.
- Evaluation: The evaluation combines diagnostic accuracy, semantic post-processing, error-taxonomy analysis, and medical expert assessment rather than final-answer accuracy alone.
3 Task Definition
The task is parallel multi-turn medical dialogue generation across English and Indic languages, requiring sequential symptom collection, preliminary guidance, and consistent diagnostic reasoning across languages.
- The task models conversational agents that collect symptoms and provide preliminary diagnostic guidance through sequential physician-patient exchanges.
- Unlike independent multilingual response generation, the setting emphasizes parallel dialogue corpora with consistent diagnostic reasoning across target languages.
- IndicMedDialog uses English as the source and translates dialogues into Assamese, Bengali, Gujarati, Hindi, Marathi, Punjabi, Tamil, Telugu, and Urdu.
- Automatic translation produced phonetic, lexical, and character-spacing errors, addressed by mapping erroneous tokens to closest correct target-language forms.
- Optional patient context includes demographics, location, allergies, and pre-existing conditions, informing personalized questioning and diagnostic reasoning.
4 IndicMedDialog Dataset
IndicMedDialog expands MDDial with synthetic, parallel consultations across ten languages and broad disease coverage, using translation quality controls and native-speaker safety assessment.
- IndicMedDialog is designed to simulate realistic physician-patient interactions across English and nine Indic languages.
- Synthetic Dialogue Generation: Synthetic consultations are generated with disease categories, demographic attributes, and stylistic constraints to increase conversational diversity beyond templates.
- Synthetic Dialogue Generation: The pipeline covers 12 diseases and 118 symptoms, with dialogues typically spanning 4–8 turns before diagnosis.
- Synthetic Dialogue Generation: 1,101 synthetic consultations enrich the original MDDial corpus with longer and more varied conversational structures.
- Translation and Verification: Two native speakers per language rate translation quality and clinical safety on 10-point scales for sampled translated dialogues.
- Translation and Verification: 9.50 mean translation quality and 9.56 mean clinical safety scores support the corpus’s linguistic fidelity and clinical suitability.
- Dataset Summary: 2,980 parallel dialogues yield 29,800 language-specific instances across ten languages, with disease labels and optional patient pretext.
5 Methodology
The methodology combines language-partitioned fine-tuning of compact models with structured post-processing and comparison against zero-shot multilingual baselines. Optional patient pre-context personalizes multi-turn diagnostic questioning.
- Model Comparison: IndicMedLM is evaluated alongside zero-shot Gemma, TinyAya, and LLaMA-3.2-3B-Instruct as a pre-adaptation reference.TinyAya provides native Indic language support, while IndicMedLM is the fine-tuned model.
- Parameter-Efficient Adaptation: The model uses LoRA with 4-bit NF4 quantization, adapting attention and MLP projections of LLaMA-3.2-3B-Instruct.Adapters use rank r = 16, α = 16, dropout = 0, and no bias terms.
- Training: Each Indic language variant is trained on its own language-partitioned split using identical hyperparameters.Training uses AdamW-8bit for 300 steps with a learning rate of 2 × 10−4 and mixed precision.
- Dialogue Formatting: Optional patient pre-context includes age, gender, allergies, and pre-existing conditions to personalize the model’s questioning strategy.Dialogues are formatted in ShareGPT style with patient turns as human messages and doctor turns as gpt messages.
- Output Processing: A constrained semantic-mapping pipeline converts verbose outputs to one of 12 canonical disease labels or NULL.An LLM judge selects the most semantically equivalent label and cannot generate labels outside the closed set.
6 Evaluation Metrics
Evaluation combines automatic diagnostic accuracy with expert assessment of clinical reliability and conversational quality. Medical practitioners assess safety, symptom understanding, contextual reasoning, diagnostic plausibility, and conversational quality using specified scoring schemes.
- Evaluation Strategy: The evaluation uses automatic diagnostic accuracy and human expert assessment of clinical reliability and conversational quality.Accuracy compares the final predicted disease label with the gold label, while expert review addresses dimensions accuracy alone cannot capture.
- Automatic Metric: Diagnostic accuracy compares each model’s final predicted disease label against the gold label in IndicMedDialog.The metric is straightforward but does not capture safety, reasoning quality, or conversational coherence.
- Expert Evaluation: Three qualified medical practitioners independently review a randomly sampled subset of generated dialogues.Reviewers hold MBBS degrees and are currently in postgraduate training.
- Expert Evaluation: Expert criteria cover safety, symptom understanding, contextual reasoning, diagnostic plausibility, and conversational quality.Criteria are scored on a 1–5 Likert scale, except medical safety, which uses binary pass/fail assessment.
- Framework Overview: Figure 3 organizes the framework from MDDial augmentation and translation through quality control, fine-tuning, and multi-turn diagnosis.The pipeline forms a parallel corpus in nine Indic languages and supports optional patient pre-context.
7 Results and Analysis
IndicMedLM shows strong but uneven multilingual diagnostic performance: semantic post-processing substantially improves results in some languages, while script- and language-specific failures remain severe. Error analysis links these failures to drift, label collapse, cross-domain confusion, tokenization/truncation, and paraphrase-over-label generation.
- 80.85% English accuracy and 72.76% Hindi accuracy are among IndicMedLM’s strongest post-processed results.
- IndicMedLM achieves the best post-processed accuracy in 7 of 10 languages, but performs at or below GEMMA in Assamese, Tamil, and Telugu.
- Hindi accuracy rises from 19.15% to 72.76% and Marathi from 13.19% to 68.51% after semantic post-processing.These gains reflect correct diagnoses embedded in hedging or paraphrased outputs rather than bare labels.
- Traumatic Brain Injury reaches 94.7% in English and Hindi but 0% in Assamese, Tamil, Telugu, and Urdu.Other diseases also show large within-disease variation across languages, including 100% conjunctivitis accuracy in Punjabi and 0% dermatitis accuracy in Telugu, Punjabi, and Urdu.
- IndicMedLM attains a 95.3% medical safety pass rate and strong expert scores for symptom extraction, context memory, diagnostic correctness, conversational flow, and efficiency.The reported average scores are 4.20, 4.40, 4.10, 4.30, and 4.00 respectively.
- Error Analysis: Failure analysis identifies instruction drift, label collapse, cross-domain confusion, tokenization/truncation failure, and paraphrase-over-label generation.Punjabi and Telugu show truncation problems, while Bengali and Assamese exhibit semantic label collapse.
- Strict label matching can underestimate capability by up to 55pp for Marathi, motivating semantic-equivalence evaluation as the primary metric.
8 Conclusion and Future Work
IndicMedDialog introduces a parallel multi-turn medical dialogue dataset for English and nine Indic languages, and supports IndicMedLM evaluation across multilingual failure modes. The authors report strong post-processed performance in some languages while identifying tokenizer and pretraining limitations in others.
- IndicMedDialog spans English and nine Indic languages and is built from synthetic consultations with native-speaker verification and script-aware post-processing.
- IndicMedLM achieves strong post-processed performance in Hindi (72.76%), Marathi (68.51%), and Bengali (58.72%).
- Assamese, Tamil, and Telugu remain in an extreme failure tier attributed to base-model tokenizer gaps and insufficient pretraining coverage.
- Error analysis identifies five failure modes and finds that strict label matching underestimates capability for Devanagari-script languages.
- Future work prioritizes English translation at inference, real annotated clinical dialogues, and expansion to more Indic languages and disease categories.
Limitations
The paper’s main limitations concern the unquantified synthetic-to-real gap, uneven language and script coverage, restricted disease and training scope, and text-only input.
- The gap between synthetic and real patient dialogue distributions remains unquantified, leaving generalization to real clinical interactions to be validated.
- Assamese, Tamil, and Telugu face tokenizer gaps for their Unicode blocks rather than data quantity alone.
- The 12-disease controlled evaluation requires broader ICD-10 taxonomies and multi-label cases before clinical deployment consideration.
- Training uses only 300 supervised fine-tuning steps and a maximum of 128 output tokens, which may limit convergence in extreme-failure languages.
- The system is limited to text-based dialogue and excludes medical images, laboratory reports, and speech.
A LoRA Training Configuration
IndicMedLM uses parameter-efficient LoRA adaptation for compact-model training, with quantization supporting consumer-grade hardware. The section also documents translation post-processing errors and reports where diagnostic accuracy is presented.
- A LoRA Training Configuration: LoRA adapters are inserted into the query, key, value, and output projection matrices of each transformer block.
- A LoRA Training Configuration: Four-bit quantization combined with LoRA-based training accommodates limited GPU memory on Google Colab and Kaggle consumer-grade environments.
- A LoRA Training Configuration: The training hyperparameters are reported in Table 5, while Table 4 reports the LoRA configuration.
- C.1 Bengali Post-Processing Example: Automatic translation introduces phonetic, lexical, and spacing errors in Indic medical terms, which the post-processing pipeline maps to canonical forms.
- Table 8 reports post-processed diagnostic accuracy for IndicMedLM by disease and selected languages.
E Clinical Risk of Cross-Domain Errors
The paper examines clinical risk through cross-domain misclassifications and broader expert-evaluation criteria. Its supporting tables cover disease, language, risk, safety, and dialogue-quality assessment structures.
- E Clinical Risk of Cross-Domain Errors: Table 9 lists frequent cross-domain misclassifications with clinical risk stratification, including critical errors involving potentially irreversible harm.
- The dataset covers 12 diseases across 8 organ systems, enabling evaluation of cross-domain diagnostic confusion.
- Table 8 reports per-disease post-processed accuracy across selected languages, with zero entries indicating complete generation failure for a disease-language combination.
- Medical expert evaluation uses 1–5 Likert scores for multiple safety, reasoning, and dialogue-quality aspects, while medical safety is assessed pass/fail.