Source-linked AI summary
MSM-Mem: A Universal Medical Structured Multimodal Memory Framework for Medical AI Agents
Md Asaduzzaman Jabin, Khoa Le, Lin Zhao, Tianming Liu
TL;DR
Current medical AI agents often lack retained experiential knowledge and consistent multimodal, longitudinal grounding for clinical reasoning. MSM-Mem addresses this by organizing clinical experiences into semantic, episodic, and visual memories that are retrieved and incrementally updated during inference. Experiments on MoE-LLaVA backbones and the curated MIMIC-CXR benchmark report consistent performance gains, with further improvements as memory is updated.
Problem
Current medical AI agents often generate hallucinated or clinically irrelevant information because they lack knowledge-, evidence-, and experience-informed contextual grounding for longitudinal clinical workflows.
Method
MSM-Mem decomposes multimodal clinical evidence into semantic, episodic, and visual memories, then retrieves and incrementally writes back experience during inference.
Results
MSM-Mem consistently improves downstream task performance across model backbones, with further gains observed as memory is incrementally updated.
Takeaways & Limitations
MSM-Mem offers a pathway toward medical AI agents that refine reasoning through accumulated longitudinal interactions and perceptual evidence.
Takeaways & Limitations
Current memory paradigms remain constrained by discarded visual and episodic details, caption bias, weakened temporal consistency, and possible memory drift.
Abstract
from arXiv · showhide
Clinical decision-making is inherently experience-driven: physicians progressively refine their reasoning by synthesizing patient history, multimodal observations, and prior diagnostic experiences across interactions. In contrast, current multimodal large language model (MLLM)-based medical AI agents largely operate as stateless inference systems, generating decisions independently for each interaction without retaining or internalizing experiential knowledge. This discrepancy limits their ability to progressively improve reasoning reliability through usage and adapt to longitudinal patient contexts in real-world clinical workflows. In this study, we propose Medical Structured Multimodal Memory (MSM-Mem), an agentic memory framework that enables medical AI agents to evolve through accumulated clinical experiences. MSM-Mem organizes heterogeneous clinical experiences into semantic, episodic, and visual memory and incrementally updates them during inference, allowing the agent to retrieve prior experiences to inform current reasoning and progressively refine decision-making over time. Evaluations on MoE-LLaVA backbones demonstrate consistent performance improve- ments with further gains observed through continued usage. In general, MSM-Mem offers a viable pathway toward medical AI agents capable of evolving their reasoning competence in a manner analogous to the way clinicians learn from practice over time.
1 Introduction
Current medical AI agents often lack the evidence- and experience-informed grounding needed for reliable longitudinal clinical reasoning. MSM-Mem addresses this gap by organizing clinical experience into persistent memory structures that are retrieved and incrementally updated during inference.
- MLLMs support multimodal medical tasks, but current agents can generate hallucinated or clinically irrelevant information without knowledge-, evidence-, and experience-informed contextual grounding.This limitation affects real-world clinical workflows despite MLLMs' ability to reason over radiological images, clinical notes, laboratory results, and structured patient records.
- Existing memory paradigms discard or distort important clinical context through text-only compression, caption bias, missing temporal scaffolding, or uncontrolled abstraction and memory drift.These limitations weaken fine-grained visual evidence, temporal consistency, clinical interpretation, and longitudinal reasoning.
- MSM-Mem decomposes multimodal clinical evidence into semantic, episodic, and visual memories representing structured knowledge, temporally indexed observations, and fine-grained imaging evidence.New clinical evidence and decision-relevant outcomes are incrementally written back to the corresponding memory modules.
- The framework accumulates experience across longitudinal interactions to support progressive self-improvement and more experience-informed clinical reasoning.Its architecture includes offline encoding and memory construction alongside online retrieval, writeback, and adaptive routing stages.
- Medical agents also face computation cost, limited-term reasoning, knowledge inconsistency, constrained adaptability, and context-switching discontinuity when relying on expanded token windows or memoryless inference.These systems may process every cue and become inferior for queries requiring deep clinical reasoning and disease diagnosis.
2 Methodology
MSM-Mem builds a structured medical-agent memory system that separates semantic, episodic, and visual evidence, retrieves relevant context during inference, and incrementally updates memory across encounters.
- Framework Overview: MSM-Mem uses offline multimodal encoders and memory construction, then online retrieval, prompt building, adaptive routing, and query inference.The base memory is constructed once, while online processing incorporates new heterogeneous data through sequential writebacks.
- Memory Construction: The memory store M = {MS, ME, MV} separates semantic patient state, episodic interaction experience, and visual observation evidence.These components represent complementary clinical knowledge, encounter history, and perceptual information.
- Episodic Memory: Episodic records retain identifiers, timestamps, visual embeddings, queries, responses, outcomes, and linked visual references for experience-level retrieval.Stored visual embeddings are normalized before memory write-back to support similarity-based query matching.
- Semantic Memory: Semantic memory aggregates historical interaction outcomes into a compact, patient-specific diagnostic summary that is updated with current impressions and decision outcomes.The update operator integrates the current impression and outcome into the existing semantic state.
- Memory-Conditioned Inference: For a new observation, MSM-Mem retrieves episodic and visual evidence by cosine similarity, adds semantic patient history, and fuses these components with the current query into a structured prompt.Only the current observation is encoded by the visual backbone; retrieved memories are injected as textualized contextual evidence.
- Outcome-Driven Memory Update: Episodic and visual memories are updated selectively using novelty, response-length, and clinical-keyword criteria, whereas semantic memory updates after every interaction.Selective updates aim to prevent redundant traces from degrading retrieval quality over time.
3 Experiments
Experiments evaluate MSM-Mem on MIMIC-CXR with MoE-LLaVA backbone variants and progressively updated memory partitions. Structured memory consistently improves performance, with full memory and continued write-back producing the strongest results.
- Dataset: MSM-Mem is evaluated on manually curated MIMIC-CXR data using subject-level splits for memory construction, fine-tuning, and validation.The memory-construction split simulates continuous interaction accumulation.
- Dataset: The evaluation probes progressive memory updates after base-memory initialization and successive write-back partitions.The base memory uses 4% of the memory split, followed by 6%, 6%, 6%, and 8% partitions.
- Implementation Details: MoE-LLaVA base, fine-tuned, and MoE-layered fine-tuned variants are compared, with ImageBind used to extract visual embeddings.The experiments use structured memory configurations across multiple backbone capacities.
- Structured Memory Effectiveness: Across model variants, individual memory modules improve over the memory-free baseline, while E+V and especially E+V+S provide larger gains.Episodic memory gives the largest single-module gain, and the full configuration achieves the highest F1 and EM scores across settings.
- Memory Scaling Analysis: 55% maximum accuracy is achieved by the MoE-FT backbone with WB4 and full S+V+E memory, increasing 27.3% and 26.07% over its prior model.This case is identified as the best among the experimental conditions.
- Memory Scaling Analysis: Full E+V+S memory exhibits the steepest growth across Base and WB1–WB4 phases, while MoE-FT gains over regular FT reach 8%, 7.66%, 7.16%, and 7.25% across configurations.The reported gains correspond to E, V, E+V, and S+E+V, respectively, under the final WB4 phase.
4 Conclusion
MSM-Mem enables experience-informed clinical reasoning by organizing interaction-derived evidence into episodic, visual, and semantic memory. Experiments report consistent gains across backbones, with further improvements as memory is incrementally updated.
- MSM-Mem decomposes interaction-derived evidence into episodic, visual, and semantic memory for experience-informed clinical reasoning.
- Experiments on a curated MIMIC-CXR benchmark show consistent performance gains across model backbones.
- Performance further improves as memory is progressively updated through longitudinal interactions and perceptual evidence.
- Future work will extend the framework to additional clinical data sources to enhance generalizability.
A.1 MSM Architecture
The proposed Bio-Mem architecture targets multimodal semantic insights, temporal cues, and long-term reasoning through offline construction, online updating, retrieval, adaptive routing, and query inference.
- Bio-Mem is designed for multimodal semantic insights, temporal cues, and long-term reasoning in biomedical applications.
- Its pipeline includes offline memory construction, online enhancement, efficient retrieval, adaptive routing, and query inference.
- The offline loop builds semantic, episodic, and visual features, whereas the online loop updates world memory when new heterogeneous data occur.
- Retrieval searches relevant memory modalities and prior episodic embeddings, while adaptive routing selects mixture-of-experts computation using memory-feature ranking and cosine similarity.
A.1.1 Offline Loop
The offline loop constructs the initial structured memory from a heterogeneous clinical dataset. It builds aligned episodic, visual, and patient-specific semantic representations that support later retrieval and prompt fusion.
- The offline loop runs once to create base memory from preprocessed multimodal data items.
- The memory builder creates episodic, visual, and semantic representations with consistent identifiers and aligned metadata.
- Episodic memory links timestamped queries to decision metadata, while visual memory links normalized keys to reference-frame paths, source episodes, and optional imaging labels.
- Semantic memory maintains per-patient, temporally aligned impression summaries in bounded histories without decision-making or statistical aggregation.
- Retrieved semantic, episodic, and visual memories are re-ranked by cosine similarity, filtered for patient consistency, clipped to top-k entries, and fused into a bounded textual prompt.
- The final prompt orders semantic context, longitudinal episodic records, and historical visual embeddings before the question and constrained Findings and Impression outputs.
A.1.2 Online Loop
The online loop performs retrieval, prompt construction, inference, and evaluation during repeated training, testing, and inference cycles. MoE adaptation updates router gating weights to allocate computation according to clinical context under bounded decoding constraints.
- The online loop repeatedly executes memory retrieval, prompt building, and evaluation during training, testing, and inference.
- The router selects standard or MoE inference paths while preserving a unified retrieval and prompt-building pipeline.
- Inference compresses only retrieved context to preserve instructions and question content under a strict context budget.
- Deterministic bounded decoding terminates through dialogue-delimiter stopping criteria to produce stable, fixed-format clinical reports.
- MoE adaptation updates only router gating weights while keeping expert parameters fixed.
- The gating network routes hidden-token representations to a compact subset of experts according to clinical context.
A.2 Mixture of Expert (MoE) Gating
The MoE gating module defines an expert set and uses a gating controller to select experts according to routing scores computed from the input embedding.
- The framework represents its mixture-of-experts components as an expert set containing N experts.
- A gating controller selects experts from the expert set based on routing scores.
- The routing score uses a learned gating-weight matrix and the input embedding.
B Additional Figures
The additional figures depict MSM-Mem's three-part base memory and its writeback and MoE routing controls, including top k = 2 selection from eight experts.
- The base memory output contains semantic, episodic, and visual memory for each subject.
- The MoE-based MSM-Mem agent includes a writeback controlling parameter and an MoE router gating condition.
- top k = 2 out of 8 experts is used during MoE routing because of system limitations.
C Limitations and Potential Risks
The framework faces deployment risks from unreliable memory contents, writeback errors, changing clinical patterns, and retrieval or representation instability.
- Irrelevant, outdated, partially matched, or noisy memories may produce incorrect reasoning, fabricated continuity, and clinically unsafe recommendations.
- Incorrect memory writeback can propagate errors and create long-term cascading hallucinations.
- When current clinical evidence contradicts older evidence, substantial expert human intervention is required.
- Other risks include unpredictable clinical pattern changes, misaligned latent spaces, unstable retrieval, and a saturated writeback window.