Source-linked AI summary
EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval
Ivan Nasonov, Nikita Glazkov, Ivan Makovetskiy, Mikhail Mozikov, Daniil Sukhorukov, Andrey Savchenko, Ilya Makarov
TL;DR
EmoMed addresses the challenge of making medical AI both clinically reliable and responsive to users’ emotional states. It combines multimodal emotion-aware generation with dual clinical retrieval and evaluates the approach across models, benchmarks, and users. The reported results show improved empathy and clarity without compromising factual accuracy, safety, or relevance overall.
Problem
Medical LLMs face hallucinations, knowledge drift, limited clinically annotated data, and insufficiently realistic safety evaluation, while affective adaptation must preserve factuality and safety.
Method
EmoMed processes text and medical images, detects affect, adapts response style, and grounds clinical information through web search and a continuously updated medical knowledge base.
Results
Across seven models and text and image QA, emotion-aware prompting improved empathy and clarity while keeping factual accuracy, safety, and relevance broadly on par with or better than a neutral baseline; a blinded user study corroborated these benefits.
Takeaways & Limitations
Emotion-aware style control can support patient-centered medical responses while maintaining clinical rigor across multimodal evaluation settings.
Takeaways & Limitations
The study uses simplified single-label emotion detection, potentially biased LLM judging, and curated datasets that may limit generalizability to realistic clinical settings.
Abstract
from arXiv · showhide
We present EmoMed - a multimodal medical consultation agent that adapts its responses based on users' emotional states while maintaining clinical accuracy. The system processes text and medical images, detects affect indicators (anxiety, confusion, urgency) from user input, and adjusts response tone, structure, and detail level accordingly. To ensure factual reliability, the agent grounds clinical information through a dual retrieval mechanism: web-based fact-checking and an API-connected, continuously updated medical knowledge base. We evaluate our approach across seven state-of-the-art language models (GPT-4/5, Qwen3, Llama 4, Gemini 2.5, Grok4, Claude3) using comprehensive metrics including LLM-as-judge assessments, MedQA style accuracy tests, BERT Score, safety/helpfulness ratings, and multimodal medical benchmarks. The results demonstrate that emotionally adaptive responses consistently outperform neutral baseline across evaluation dimensions, without compromising clinical accuracy. A controlled user study validated these findings, with participants reporting improved perceived empathy and communication clarity, while maintaining trust in factual accuracy. Source code: https://github.com/NasonovIvan/EmoMed-Agent
1 Introduction
Medical LLMs show strong benchmark performance but still face hallucination, knowledge drift, limited clinical data, and insufficiently realistic safety evaluation. EmoMed addresses this gap by combining emotion-aware interaction with multimodal processing, dual retrieval, and broad evaluation.
- Clinical LLM deployment remains constrained by hallucinations, uneven domain coverage, and knowledge drift in dynamic medical contexts.
- Multimodal medical QA is necessary for image-grounded questions, but clinically annotated data, external-knowledge reliance, and safety-aware evaluation remain challenging.
- Emotionally adaptive responses can improve engagement, perceived quality, and clarity, but must preserve factuality and safety.
- EmoMed processes text and medical images, adapts responses to anxiety, confusion, and urgency, and uses web search plus a continuously updated medical knowledge base.
- Across seven frontier models, emotion-adaptive responses improved perceived empathy and communication clarity without compromising clinical accuracy, with controlled-study confirmation.
2 Related Work
Prior work combines medical language and vision–language models with retrieval, safety evaluation, and affective personalization. EmoMed brings these strands together through emotion detection, adaptive style control, dual retrieval, and multimodal evaluation.
- Healthcare agents use LLMs for education, decision support, summarization, and reference Q&A, while safe adoption requires transparent sourcing and robust error analysis.
- Affective personalization improves empathy and engagement, but persona steering can introduce bias and therefore motivates policy-level tone control in clinical settings.
- The system’s emotion detector is instructed to infer one primary emotion from text and images, such as anxious, concerned, hopeful, or frustrated.
- The medical-support prompt uses detected emotion to shape supportive responses while prioritizing accuracy, currency, safety, and professional care for serious concerns.
- EmoMed combines interpretable style parameters for warmth, hedging, and structure with dual retrieval and evaluation of empathy, clarity, factuality, and trust.
3 Architecture
EmoMed is a multimodal, tool-augmented consultation agent that detects affect, retrieves current clinical information, and generates empathetic responses under safety constraints. Its evaluation compares emotion-aware and neutral versions across text and image-grounded medical QA.
- Architecture: The VLM processes text with optional medical images, detects emotional state, retrieves current clinical information, and generates fact-grounded responses under safety constraints.
- Tool-augmented retrieval: The agent chooses between direct answering, Tavily web search for time-sensitive information, and MediSearch for evidence-based clinical content, with five-call limits.
- Emotion detection: GPT-4o predicts a single primary emotion from text and any accompanying image; invalid outputs trigger neutral emotional context.
- Response generation: Response generation avoids diagnoses, acknowledges uncertainty, recommends professional care for serious concerns, and follows a structured guidance format.
- Experimental design: Evaluation contrasts emotion-aware prompting with an identical baseline lacking emotion instructions across text-only and multimodal medical datasets.
- Evaluation and implementation: A blinded user study compared the two response versions for appropriateness and supportiveness, while the implementation used one low-temperature VLM call sequence.
4 Results
Across text QA and Med-VQA, emotion-aware prompting consistently improved empathy and clarity while leaving core correctness broadly stable. A blinded user study corroborated these gains, while failure analysis identified occasional over-hedging, affect misclassification, retrieval misses, and weaker image-based affect cues.
- Overall effects: Across seven vision-capable models, emotion-aware prompting improved affective dimensions while preserving core correctness; text QA showed stable safety and mixed BERTScore and relevance.
- Text QA: +0.20-0.35 in empathy and +0.01-0.03 in clarity were typical text-QA gains, while safety shifts were small and positive on average.
- Text QA: Four models matched or exceeded baseline accuracy, while Gemini 2.5 showed a modest decrement despite substantial affective gains.
- Med-VQA: Med-VQA showed robust empathy gains, modest clarity improvement, positive safety trends, and small mixed accuracy differences, including Llama-4 Scout +0.101 and Gemini 2.5 -0.098.
- Cross-task comparison: Visual summaries show affective gains with stable content metrics in both text QA and Med-VQA, although image-grounded accuracy is lower overall due to task difficulty.
- User study: In a blinded study, 34 of 44 participants, or 77%, preferred emotion-aware responses for empathy and clarity without perceived losses in factual accuracy or safety.
- Failure analysis: Rare failure modes included over-hedging, verbosity, emotional misclassification, stale retrieval, extraction misses, and weaker affect cues in images.
5 Conclusion and Future Work
EmoMed combines emotion-aware style control with dual retrieval to provide patient-centered medical responses while preserving rigor. Across text and image QA, it improved empathy and clarity without sacrificing core evaluation dimensions, while future work targets broader clinical robustness.
- EmoMed combines emotion-aware style control with dual retrieval to deliver patient-centered responses without sacrificing rigor.
- Future work will address richer affect dynamics, stronger safety and provenance, specialty-grounded retrieval, and larger human studies in clinical workflows.
6 Limitations
The study identifies limitations in emotion representation, automatic evaluation, and dataset realism, which constrain how broadly its findings should be interpreted.
- The emotion model uses a simplified single-label approach that may not capture the complexity and subtlety of human emotions.
- LLM-as-judge evaluation may introduce systematic bias because language-model judgments are subjective.
- Curated datasets may limit generalizability to broader, more realistic scenarios.
- Further research is needed with larger user groups and integration into clinical practice.
A Appendix A. Average metrics for method across all models
The appendix summarizes radar-chart comparisons between emotion-aware and baseline agents for text-based and multimodal medical question answering. The emotion-aware variant achieves higher empathy while matching or slightly exceeding baseline factual and safety performance.
- Text QA: Figure 5 compares emotion-aware and baseline agents across empathy, clarity, helpfulness, and safety for text QA.
- Med-VQA: Figure 6 compares emotion-aware and baseline agents for Med-VQA.
- Across text-based and multimodal QA, emotion-aware responses achieve substantially higher empathy while matching or slightly exceeding baseline factual and safety metrics.
B Appendix B. Detailed metrics for individual models
Appendix B reports detailed model-level results for text-only and VQA benchmarks with and without emotional prompting. Emotional prompting generally improves affective metrics, while VQA consistency can favor the non-emotional condition.
- Appendix B provides detailed results for individual models on text-only and VQA benchmarks with and without emotional prompting.
- Text-only benchmark: On the text-only benchmark, emotional prompting consistently improves metrics across all six evaluated models.
- VQA benchmark: On the VQA benchmark, emotional prompting particularly enhances empathy and helpfulness, while consistency metrics tend to be higher without emotional prompting.
- The appendix includes metric definitions for factual accuracy, empathy, consistency, helpfulness, safety, BERT Score, relevance, and clarity.