Source-linked AI summary
ArogyaSutra: A Multi-Agent Framework for Multimodal Medical Reasoning in Indic Languages
Tanmoy Kanti Halder, Akash Ghosh, Subhadip Baidya, Arijit Roy, Sriparna Saha
TL;DR
Multimodal medical reasoning remains underrepresented and unreliable in low-resource Indic languages, especially for open-ended image–text queries. ArogyaSutra combines tool-grounded actor–critic reasoning with dual memory, and experiments report consistent improvements across Indic languages and imaging modalities.
Problem
Open-ended multimodal medical reasoning in Indic languages remains underrepresented, with degraded logical fidelity, unstable language behavior, and limited benchmark coverage.
Method
ArogyaSutra uses an actor–critic multi-agent framework with visual grounding tools and dual-memory mechanisms for step-wise reasoning over medical image–text inputs.
Results
ArogyaSutra and ArogyaBodha demonstrate consistent improvements over general-purpose and medical baselines across languages and imaging modalities.
Takeaways & Limitations
The work supports more reliable and linguistically consistent multilingual medical reasoning for underserved Indic-language settings.
Takeaways & Limitations
Performance may degrade on rare clinical cases, low-resource dialects, code-mixed language, or when visual grounding tools and the multimodal backbone fail.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown promising reasoning capabilities in general domains, yet their performance remains limited in specialized settings such as healthcare, especially in multilingual and low-resource scenarios. This gap is critical in regions like rural India, where patients often express complex medical queries in native Indic languages and rely on multimodal inputs such as medical images. Existing English-centric MLLMs struggle to support such use cases, limiting equitable access to AI-driven healthcare assistance. To address this challenge, we introduce ArogyaBodha, a large-scale multilingual multimodal medical question-answer dataset constructed from eight heterogeneous sources, covering 31 body systems, six imaging modalities, and 21 clinical domains across English and seven major Indian languages. We further propose ArogyaSutra, an actor-critic-based multi-agent framework that integrates tool grounding with dual-memory mechanisms for step-wise, reasoning-aware decision making, and uses stored actor-critic simulation trajectories for distillation. Experiments show that our dataset and framework improve multilingual medical reasoning accuracy across all Indic languages, with ablations validating the contribution of each component. The source code and dataset are available at: https://iitp-cse.github.io/ArogyaSutra/
1 Introduction
ArogyaSutra addresses the underrepresentation and reliability challenges of multimodal medical reasoning in Indic languages through the ArogyaBodha benchmark and an actor–critic multi-agent framework. The approach combines expert-verified multilingual data, visual grounding tools, and dual-memory step-wise reasoning for clinically grounded answers.
- Motivation: Indic-language multimodal medical reasoning remains underrepresented, with persistent failures in logical fidelity and language stability.Research is largely English- and Chinese-centric, while cross-lingual transfer offers only partial relief.
- Dataset and Benchmark: ArogyaBodha is a large-scale benchmark built from eight medical sources, spanning English, seven Indian languages, 31 body systems, and 21 clinical domains.It includes multiple imaging modalities and expert-verified queries paired with single unambiguous ground-truth answers.
- Framework: ArogyaSutra uses an actor–critic multi-agent framework with tool-based visual grounding and dual-memory mechanisms for step-wise reasoning over image–text inputs.The framework is designed for reliable multimodal medical reasoning in Indic languages rather than direct final-answer prediction.
- Framework: At each reasoning step, the multimodal Actor processes medical images and Indic-language queries while invoking tools such as zoom/crop, edge detection, depth analysis, and region-level detection.The Actor predicts intermediate semantic reasoning steps and extracts clinically relevant evidence.
- Impact and Validation: The work targets equitable healthcare access by enabling clinically grounded and linguistically consistent multimodal reasoning in low-resource Indic languages.Certified medical practitioners validate curated questions and answers, while reverse translation uses cosine similarity and BLEU4 to assess translation quality.
2 Related Works
Recent work advances multimodal medical reasoning through specialized training paradigms, collaborative or reinforcement-learning-based agentic frameworks, and realistic evaluation environments. However, existing approaches remain constrained by high-resource languages or modality-specific settings, motivating ArogyaSutra’s foundation on these lines of work.
- Training paradigms: Prompt tuning, supervised fine-tuning, reinforcement learning, and agentic distillation have improved MLLMs on complex vision–language reasoning and been adapted to medical settings.The passage identifies these as recent training paradigms for improving multimodal reasoning.
- Agentic medical reasoning: MedAgents uses collaborative multi-agent discussion for zero-shot medical reasoning, while MMedAgent applies reinforcement learning to coordinate specialist and general-practitioner agents.These frameworks target complex clinical reasoning through multi-agent collaboration and learned coordination.
- Agentic medical reasoning: Med-R1 distills Chain-of-Thought reasoning through reinforcement learning, extending efforts to improve medical reasoning with learned trajectories.The supplied passage explicitly describes Med-R1 as using reinforcement learning for CoT distillation, though the sentence is truncated.
- Evaluation environments: MedAgentBench and related work emphasize realistic medical-agent evaluation through multi-step decision-making and tool interaction.These benchmarks focus on evaluating agents in interactive clinical environments rather than isolated question answering.
- Research gap: Despite these advances, most existing approaches remain limited to high-resource languages or modality-specific settings, whereas ArogyaSutra builds upon these foundations.The passage frames this limitation as the motivation for the present work.
3 Dataset: ArogyaBodha
ArogyaBodha is a 40,857-sample multilingual medical dataset assembled from eight heterogeneous sources and covering English plus seven major Indian languages. Its translations use Gemini-2.5-Pro and are assessed through reverse-translation metrics and practitioner review to preserve clinical meaning and linguistic fidelity.
- Dataset Composition: 40,857 samples span English and seven major Indian languages, curated and filtered from eight heterogeneous medical sources.The languages are Bengali, Hindi, Assamese, Tamil, Telugu, Punjabi, and Marathi, alongside English.
- Dataset Construction: The dataset combines existing medical reasoning benchmarks with NEET-PG and FMGE postgraduate medical entrance questions to support clinically relevant, broad medical coverage.Few-shot prompting with GPT-4o-mini incorporates patient details, reports, and contextual medical information from the source datasets.
- Multilingual Translation: Filtered questions and options are translated into seven Indian languages using Gemini-2.5-Pro while preserving clinical semantics and reasoning requirements.The translation step provides multilingual coverage across the curated medical questions.
- Translation Evaluation: Reverse translation evaluates semantic consistency with SentenceTransformer cosine similarity and English reconstruction with NLTK BLEU4 across seven Indic languages.CosineBack scores remain consistently around 0.93, indicating strong semantic preservation.
- Translation Evaluation: 4.27 is the average practitioner score from manual review of 20% of translated test samples using a five-point clinical and linguistic quality scale.The review assesses clinical meaning, terminology correctness, semantic consistency, fluency, and contextual relevance.
4 Framework : ArogyaSutra
ArogyaSutra is a multimodal agentic framework for low-resource Indic-language healthcare that combines tool-grounded perceptual reasoning with short- and long-term memory. Its Actor-Critic module proposes and evaluates actions, uses language-aware reflection and halting rules to correct failures, and distills approved trajectories into an inference-efficient Actor policy.
- Framework overview: ArogyaSutra integrates tool-grounded perceptual reasoning with short-term and long-term memory to track prior errors and contextual dependencies across reasoning.The framework targets performance degradation in real-world healthcare, particularly low-resource Indic-language settings.
- Perceptual grounding: Four visual tools—open-vocabulary object detection, zoom/crop, edge detection, and depth estimation—are integrated directly into the MLLM inference loop.Tool execution and result incorporation form a coherent multimodal rollout.
- Language Analysis Aware Reflection: Language-aware reflection diagnoses linguistic instability or logical errors, providing English feedback for language failures and query-language feedback for reasoning failures.This feedback guides the Actor toward a corrected reasoning chain.
- Halting and distillation: Within three or fewer reasoning iterations, correct traces are retained; otherwise, the process restarts from long-term error summaries, and failures after two consecutive restart cycles are removed from training.The retained Actor-Critic trajectories are used for code-switched reasoning-chain distillation, after which inference retains only the Actor policy and avoids Critic evaluation.
5 Experimental Setup
The experiments use a controlled multilingual multimodal setup with standardized training and a balanced seven-language test set. Evaluation compares ArogyaSutra across diverse medical and general-purpose multimodal language models.
- Implementation Details: Experiments ran on an NVIDIA A100 80GB PCIe GPU, training models for 3 epochs on an 18k balanced multilingual multimodal dataset.Each run required approximately 12–15 hours and used PyTorch, Hugging Face Transformers, and Unsloth.
- Evaluation Data: The test set contains 910 samples across seven languages, with 130 curated samples per language.The test set is balanced across data sources, and identical questions are used for every language to enable controlled cross-lingual comparison.
- Evaluation Data: Each language test set samples eight sources, including 42 MedXpertQA, 34 PMC-VQA, 16 MedPix-2.0, and 17 MedTrinity-25M examples.It also includes 9 BRATS24, 8 NEET-PG and FMGE, 3 GMAI-MMBench, and 1 MAMA-MIA example per language.
- Baselines: Evaluation covers Qwen3-VL-8B-Instruct, Qwen2.5-VL-3B-Instruct, Mistral-Small-3.2-24B-Instruct, and LLaVA-v1.6-34B.The Qwen models assess performance across model scales, while Mistral and LLaVA provide general-purpose MLLM comparisons.
6 Result and Analysis
ArogyaSutra improves multilingual multimodal medical reasoning over general-purpose, medical-domain, and trained baseline models. It also demonstrates strong out-of-distribution generalization, with gains supported by controlled component and robustness analyses.
- Baseline Comparison: Evaluation against general-purpose and medical-domain multimodal baselines shows that tool grounding with short- and long-term memory improves medical reasoning.The study also includes controlled component and robustness analyses on an out-of-distribution benchmark across multiple Indic languages.
- Baseline Comparison: 43.40 average accuracy: ArogyaSutra (Qwen2.5-VL-7B) surpasses GPT-4.0 at 39.30 by +4.1 points, with gains across all seven languages.It also exceeds BioMistral-7B (23.83 avg) and MedGemma-4B-it (36.11 avg).
- Trained Baselines: 35.65 average accuracy: ArogyaSutra (Qwen2.5-VL-3B) improves from 29.56 for Qwen2.5-VL-3B-Instruct, yielding a gain of +6.1 points.ArogyaSutra (Qwen2.5-VL-7B) reaches 43.40 average accuracy versus 34.21 for Qwen2.5-VL-7B-Instruct, a +9.2-point improvement.
- Out-of-Distribution Performance: 50.4 accuracy: ArogyaSutra (Qwen2.5-VL-7B) on the OOD dataset outperforms Qwen2.5-VL-7B-Instruct at 35.0.It also surpasses LLaVA-v1.6-34B at 23.3 on the out-of-distribution dataset.
7 Ablation Studies
Ablations show that the Critic agent, tool grounding, code-switching, and dual-memory mechanisms are important for ArogyaSutra’s multilingual multimodal reasoning. Enabling both grounding and memory yields the highest reported accuracy, while removing them substantially reduces performance.
- Impact of the Critic Agent and its Components: Removing the Critic reduces accuracy to 33.43%–33.71%, compared with 43.40% for the full ArogyaSutra framework.The grounded Actor-only variant also underperforms the Qwen2.5-VL-7B-Instruct baseline at 34.21%.
- Impact of the Critic Agent and its Components: Removing tool grounding or code-switching causes the largest performance drop of 16.54%, while disabling long- and short-term memory reduces accuracy by 12.83%.These ablations identify grounding, code-switching, and memory as major contributors to performance.
- Impact of Tool Grounding and Memory: 26.86% accuracy results when both tool grounding and memory are disabled, increasing to 30.57% with memory alone and 43.40% when both are enabled.The results demonstrate complementary contributions from image grounding and memory utilization within the Critic agent.
8 Qualitative Analysis
ArogyaSutra’s generated outputs are qualitatively compared with MedGemma-4B-it and Qwen2.5-VL-7B-Instruct. Unlike both baselines, it consistently responds in the target low-resource language and produces more structured, corrective reasoning.
- 8 Qualitative Analysis: ArogyaSutra is qualitatively compared with MedGemma-4B-it and Qwen2.5-VL-7B-Instruct, which exhibit shallow reasoning and often fail to respond in low-resource languages such as Hindi.The comparison is shown in Figure 2.
- 8 Qualitative Analysis: In contrast, ArogyaSutra consistently generates responses in the target language and demonstrates more structured and corrective reasoning.
9 Risk Analysis and Limitations
Despite strong benchmark performance, ArogyaSutra remains vulnerable to reasoning errors and visual misinterpretation, especially in rare or atypical clinical cases. Its language coverage is incomplete, and the framework has additional dependencies that limit robustness in real-world use.
- Model reliability: ArogyaSutra may produce reasoning errors or misinterpret visual evidence, particularly in rare or atypical clinical cases.Such failures pose risks if outputs are misused for real-world decision making.
- Language coverage: ArogyaBodha covers seven major Indian languages but omits low-resource dialects and code-mixed language used in clinical practice.Performance may degrade under these linguistic conditions.
- Framework limitations: The actor–critic framework depends on additional factors that remain unspecified in the provided limitation passage.The passage introduces this dependency as a further limitation but does not state what it entails.
10 Conclusion
The paper introduces ArogyaBodha and ArogyaSutra for multilingual multimodal medical reasoning in low-resource Indic-language settings. ArogyaSutra combines tool-grounded perception, actor–critic reasoning, memory-aware reflection, and adaptive code-switching to improve reliability and linguistic consistency across languages and imaging modalities.
- Contributions: ArogyaBodha and ArogyaSutra advance multilingual multimodal medical reasoning in low-resource Indic-language settings.The conclusion presents both as the paper’s central contributions.
- Empirical findings: ArogyaSutra demonstrates consistent improvements over strong general-purpose and medical baselines across languages and imaging modalities.The conclusion reports consistency across both linguistic and imaging dimensions.
- Framework design: Tool-grounded perception, actor–critic reasoning, memory-aware reflection, and adaptive code-switching support more reliable and linguistically consistent reasoning across diverse Indic languages.These integrated components are identified as the mechanisms enabling the framework’s reliability and linguistic consistency.
Ethical Statement · Contribution Statement
The paper states that all datasets are anonymized, publicly available under applicable licenses, and used solely for non-commercial research. Tanmoy Kanti Halder and Akash Ghosh contributed equally.
- Ethical Statement: All datasets used in ArogyaBodha are anonymized.The statement specifies that they contain no personally identifiable patient information.
- Ethical Statement: The datasets contain no personally identifiable patient information.
- Ethical Statement: ArogyaBodha uses publicly available datasets under their respective research, academic, or open-access licenses.
- Ethical Statement: The datasets are utilized solely for non-commercial research purposes.
- Ethical Statement: The listed sources include MedXpertQA, MedTrinity-25M, PMC-VQA, MedPix-2.0, and MAMA-MIA.
- Ethical Statement: The listed sources also include BraTS 2024, GMAI-MMBench, and NEET-PG/FMGE.
- Contribution Statement: Tanmoy Kanti Halder and Akash Ghosh contributed equally to this work.