EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval

1 Introduction

• Clinical LLMs support education, triage, reference Q&A, decision support, and documentation, but hallucinations, uneven domain coverage, and knowledge drift complicate safe deployment.Medical knowledge and reasoning benchmarks include USMLE and MedQA/MultiMedQA, while safe adoption also calls for transparent sourcing, uncertainty communication, and error analysis.Source: §1 · pp. 2–3
• Tool-using agents and retrieval can improve recency, grounding, and traceability, but medical AI still faces robustness, privacy, bias, poisoning, and unsafe-input risks.The introduction also points to healthcare safety benchmarks, clinician-in-the-loop validation, and programmable guardrails as safety measures.Source: §1 · p. 2
• Image-grounded questions in areas such as dermatology, radiology, and pathology make multimodality necessary, while limited clinical annotations and safety-aware evaluation remain challenges.Medical VQA and report generation have advanced, alongside datasets including PMC-VQA, SLAKE, VQA-RAD, and PathVQA.Source: §1 · pp. 2–3
• Because anxiety, confusion, and urgency shape risk perception, adherence, and trust, affective personalization may improve engagement and communication but must preserve factuality and safety.The work aligns emotion-adaptive interaction findings with clinically oriented safety and accuracy metrics.Source: §1 · p. 3
• EmoMed combines text-and-image consultation, affect-driven response adaptation, and dual medical grounding, and is evaluated across seven models using clinical and communication metrics.The introduction reports that emotion-adaptive responses improve perceived empathy and clarity without compromising clinical accuracy, with findings confirmed in a controlled study.Source: §1 · p. 3

• Healthcare agents support education, decision support, and summarization, but face realism, transparency, hallucination, and safety concerns addressed through retrieval, red-teaming, guardrails, and abstention.Source: §2 · p. 3
• Affective personalization uses emotion recognition, multimodal cues, and style conditioning to improve empathy and engagement, while persona steering risks bias in clinical settings.Source: §2 · p. 3
• EmoMed adapts warmth, hedging, and structure through interpretable style parameters while preserving factual accuracy and adding uncertainty and escalation cues.Source: §2 · p. 4
• Medical VQA models face grounding and safety challenges; EmoMed uses web and knowledge-base retrieval and combines judge, factuality, semantic, and user-study evaluation.Source: §2 · p. 4

Architecture

• The VLM can answer directly or use Tavily for time-sensitive information and MediSearch for clinical evidence, with tool calls capped at five iterations per query.A complete tool-call trace for a typical query is shown in Figure 2.Source: §3 · p. 5
• GPT-4o infers one primary emotion from input text and any accompanying image; invalid or unparsable labels default to neutral context.The emotion detector is instructed to output a single term, such as anxious, concerned, hopeful, or frustrated.Source: §3 · p. 5
• The response prompt forbids diagnostic claims, acknowledges uncertainty, recommends professional care for serious concerns, and adapts tone when an emotion is detected.Responses acknowledge the concern, briefly restate it, provide non-alarmist clinical context and actionable guidance, and may ask a clarifying question.Source: §3 · p. 5
• Evaluation compares emotion-aware and emotion-free versions on text-only medical questions and pathology-image VQA tasks from PathVQA.GPT-4o judges factual accuracy, empathy, emotional consistency, helpfulness, safety, relevance, and clarity; Sentence-BERT measures semantic similarity. A blinded pairwise user study involved 48 participants.Source: §3 · pp. 5–6
• The system supports user-uploaded images, while its primary emotional context comes from text messages.Visual processing supports the consultation scenario without shifting the focus from text-based emotional understanding.Source: §3 · p. 5
• A single VLM endpoint handles emotion detection, tool selection, and response generation in one lightweight, reproducible call sequence at temperature 0.1.Source: §3 · p. 6

Text QA (aggregate)

• On text QA, emotion-aware prompting raised empathy by +0.20-0.35 and clarity by +0.01-0.03, while accuracy was unchanged or higher for most models.Accuracy examples were GPT-4o Mini +0.139 and Grok-4 Fast +0.087; Gemini 2.5 Flash decreased by -0.080 despite strong affect gains.Source: §4 · p. 6

Med-VQA (aggregate)

• On Med-VQA, empathy increased robustly and clarity improved modestly, while accuracy differences were small and mixed and emotional consistency sometimes dipped.Accuracy examples were Llama-4 Scout +0.101 and Gemini 2.5 -0.098; weaker image-based affect cues were identified as a likely source of reduced emotional consistency.Source: §4 · p. 6

User Study

• In blinded pairwise comparisons, 34 of 44 participants (77%) preferred the emotional response, citing empathy and communication clarity without perceived factual-accuracy or safety losses.Feedback described clearer structure for confusion, reassurance for anxiety, and calibrated hedging for urgent cases.Source: §4 · p. 7

Error Analysis and Failure Modes

• Failure modes were uncommon and did not change aggregate trends; fixed content plans, detail caps, neutral-warm defaults, retries, sourcing, and escalation language mitigated them.Brief or ambiguous inputs occasionally caused emotional misclassification, retrieval or extraction misses occurred infrequently, and affective gains persisted after low-confidence detections were filtered.Source: §4 · p. 7

5 Conclusion and Future Work

• Across seven models and text and image QA, emotion-aware prompting improved empathy and clarity while keeping factual accuracy, safety, and relevance on par with or better than a neutral baseline.A blinded user study corroborated these benefits.Source: §5 · p. 7
• EmoMed combines emotion-aware style control with dual retrieval to deliver patient-centered responses without sacrificing rigor.Source: §5 · p. 7
• Future work will address richer affect dynamics, stronger safety and provenance, specialty-grounded retrieval, and larger human studies in clinical workflows.Source: §5 · p. 7

6 Limitations

• The single-label emotion model may not capture the complexity and subtlety of human emotions.The authors describe this approach as simplified.Source: §6 · p. 7
• LLM-as-judge evaluation may introduce systematic bias because language-model judgments are subjective.Source: §6 · p. 7
• Experiments on curated datasets may not generalize to broader, more realistic scenarios.Source: §6 · p. 7
• The authors call for research with larger user groups and integration into clinical practices.Source: §6 · p. 7

A Appendix A. Average metrics for method across all models

• In both text-based QA and Med-VQA, the emotion-aware agent matches or slightly exceeds the baseline on factual and safety metrics and achieves substantially higher empathy scores.Figures 5 and 6 show the comparisons for text-based and multimodal medical QA, respectively.Source: Figure 5 · p. 8, Figure 6 · p. 8, A Appendix A. Average metrics for… · p. 8

Text-only benchmark

• Across all six models, emotional prompting consistently improved text-only benchmark metrics.Table 1 covers factual accuracy, empathy, consistency, helpfulness, safety, BERT Score, relevance, and clarity.Source: B Appendix B. Detailed metrics for… · p. 8, Table 1 · p. 9

VQA benchmark

• On VQA, emotional prompting particularly enhanced empathy and helpfulness, while consistency metrics tended to be higher without prompting.Table 2 reports the VQA comparison across factual accuracy, empathy, consistency, helpfulness, safety, BERT Score, relevance, and clarity.Source: B Appendix B. Detailed metrics for… · p. 8, Table 2 · p. 9