MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

Chunzheng Zhu; Jiaqi Zeng; Junyu Jiang; Jianxin Lin; Yijun Wang

Source-linked AI summary

MedSynapse-V: Bridging Visual Perception and Clinical Intuition via Latent Memory Evolution

Chunzheng Zhu, Jiaqi Zeng, Junyu Jiang, Jianxin Lin, Yijun Wang

arXiv:2604.26283v4Apr 29, 2026cs.CVcs.AI

TL;DR

Medical VLMs face a mismatch between discrete token reasoning and the continuous, case-adaptive memory used in expert diagnosis. MedSynapse-V evolves latent diagnostic memory through anatomical prior retrieval, causal refinement, and autonomous internalization, and it outperforms existing medical VLMs and reasoning methods across benchmarks while maintaining efficient inference.

Problem
Discrete tokenization limits medical VLMs’ representation of continuous clinical expertise and can dissipate visual information across long reasoning chains.
Method
MedSynapse-V retrieves anatomical priors into compact latent memories, refines them with causal counterfactual rewards, and transfers them into autonomous intrinsic memory.
Results
MedSynapse-V consistently outperforms medical VLMs, general-purpose VLMs, and RL-based chain-of-thought methods in accuracy and efficiency across multiple medical benchmarks.
Takeaways & Limitations
The results support latent memory evolution as an alternative to verbose explicit reasoning for medical visual-language diagnosis.
Takeaways & Limitations
Borderline cases can fall below the memory’s discriminative granularity, motivating calibrated uncertainty and adaptive memory strategies.

Abstract

from arXiv · show

High-precision medical diagnosis relies not only on static imaging features but also on the implicit diagnostic memory experts instantly invoke during image interpretation. We pinpoint a fundamental cognitive misalignment in medical VLMs caused by discrete tokenization, leading to quantization loss, long-range information dissipation, and missing case-adaptive expertise. To bridge this gap, we propose ours, a framework for latent diagnostic memory evolution that simulates the experiential invocation of clinicians by dynamically synthesizing implicit diagnostic memories within the model's hidden stream. Specifically, it begins with a Meta Query for Prior Memorization mechanism, where learnable probes retrieve structured priors from an anatomical prior encoder to generate condensed implicit memories. To ensure clinical fidelity, we introduce Causal Counterfactual Refinement (CCR), which leverages reinforcement learning and counterfactual rewards derived from region-level feature masking to quantify the causal contribution of each memory, thereby pruning redundancies and aligning latent representations with diagnostic logic. This evolutionary process culminates in Intrinsic Memory Transition (IMT), a privileged-autonomous dual-branch paradigm that internalizes teacher-branch diagnostic patterns into the student-branch via full-vocabulary divergence alignment. Comprehensive empirical evaluations across multiple datasets demonstrate that ours, by transferring external expertise into endogenous parameters, significantly outperforms existing state-of-the-art methods, particularly chain-of-thought paradigms, in diagnostic accuracy. The code is available at https://github.com/zhcz328/MedSynapse-V.

1 Introduction

Medical VLMs struggle to represent continuous clinical expertise with discrete reasoning, motivating MedSynapse-V’s latent diagnostic memory evolution framework. The framework condenses anatomical priors, refines memories causally, and internalizes them autonomously to improve diagnosis.

Motivation: Experienced clinicians rapidly activate compressed diagnostic memory, whereas medical VLMs rely on discrete tokens that poorly represent continuous pathological features.The paper identifies coarse vocabulary granularity and long-range information dissipation as sources of cognitive misalignment.
Motivation: Retrieval and prompt-based knowledge injection remain static, causally unverified, and do not evolve into intrinsic model capability.These approaches may provide domain cues without validating their relevance to the current diagnostic decision.
Motivation: Latent computation avoids discrete-symbol bottlenecks but requires structured anatomical priors to preserve clinical semantics.Without such priors, latent representations can become abstract vectors decoupled from clinical meaning.
Framework: MedSynapse-V uses Meta Query for Prior Memorization, Causal Counterfactual Refinement, and Intrinsic Memory Transition in a progressive training paradigm.These mechanisms retrieve anatomical features, prune causally irrelevant memory components, and internalize diagnostic patterns into autonomous latent memory.
Results: Across seven medical multimodal benchmarks, MedSynapse-V outperforms state-of-the-art approaches in diagnostic accuracy and cross-domain generalization with negligible additional inference cost.The reported comparisons include medical VLMs, RL-enhanced chain-of-thought methods, and general-purpose latent reasoning approaches.

2 Methodology

MedSynapse-V evolves diagnostic implicit memory through anatomical-prior retrieval, causal refinement, and intrinsic distillation within a VLM’s hidden stream. The three-stage process progressively converts encoder-derived features into compact, causally aligned, and autonomous diagnostic representations.

Architecture Overview: MedSynapse-V dynamically generates diagnostic memory vectors from image and query inputs and injects them into the VLM hidden stream.The memory set M contains N elements in the VLM hidden-state dimensionality.
Stage I: Meta Query for Prior Memorization: Learnable meta-query probes selectively aggregate anatomical encoder features into compact memory representations aligned with the VLM hidden space.The probes target pathological patterns such as boundary irregularity, density heterogeneity, and vascular-tissue relationships.
Stage I: Meta Query for Prior Memorization: The synthesized memory is inserted between question encoding and answer generation, allowing self-attention to aggregate latent diagnostic evidence during prediction.A semantic-alignment warmup stabilizes the mapping before subsequent refinement.
Stage II: Causal Counterfactual Refinement: Causal Counterfactual Refinement optimizes memory-conditioned policy behavior using accuracy and intervention-based rewards derived from masked anatomical regions.The causal reward compares performance under original and region-masked memory conditions, retaining priors with diagnostic contribution.
Stage II: Causal Counterfactual Refinement: The counterfactual criterion treats positive causal reward as evidence that a memory contributes to diagnosis and nonpositive reward as causal irrelevance.This aligns retained memory with clinical decision logic rather than fixed reference trajectories.
Stage III: Intrinsic Memory Transition: Intrinsic Memory Transition distills encoder-conditioned teacher behavior into an autonomous student branch through divergence alignment, reducing inference dependence on the anatomical encoder.The teacher uses encoder-derived memory, while the student generates autonomous memory; full-vocabulary alignment is performed across generation positions.

3 Experiments

Experiments across seven medical multimodal benchmarks compare MedSynapse-V with general, medical-specific, RL-enhanced, and latent-space baselines. Results cover benchmark accuracy, ablations, qualitative diagnosis, efficiency, reward dynamics, and latent-space organization.

Experimental setup: Experiments use seven medical multimodal benchmarks and compare general VLMs, medical-specific VLMs, RL-enhanced reasoning models, and adapted latent-space methods.Training includes PubMedVision, OmniMedVQA, SLAKE, and PathVQA data; Qwen3-VL-8B is the base model unless otherwise noted.
Main results: 61.4% average is achieved with the anatomical encoder, while encoder-free MedSynapse-V (IMT) retains 59.6% and surpasses all baselines.Against MMedExpert-R1 at 55.7%, the IMT variant leads by +3.9 pp and shows its largest margins on VQA-RAD, SLAKE, and PathVQA.
Main results: MedSynapse-V reaches the top performance on every benchmark without task-specific tuning, whereas small RL models fall below 19% on MedXpert.Adapted latent baselines range from 44.5% to 50.5%, and latent compression without clinical grounding is described as encoding statistical shortcuts rather than diagnostic logic.
Ablation study: The full pipeline averages 67.7; removing CCR or MQPM lowers performance, while causal reward contributes +4.1 pp from 63.6 to 67.7.Skipping MQPM collapses Avg to 52.9, while replacing CCR with SFT reaches 59.2.
Efficiency: IMT removes the encoder with only 1.4 pp degradation, while latency drops 39% and memory decreases 6.3 GB.The encoder-free model reaches 59.6% at 2.6 s/sample; full-scale CoT baselines require 5.8 s because of 300–400 autoregressive reasoning tokens.
RL dynamics and latent space: With rcausal, training reward rises to approximately 0.88, whereas the ablation without it plateaus near 0.48; evolved memories form clinically coherent modality and disease clusters.The t-SNE visualization separates eight imaging modalities and further segregates disease subtypes within CT and pathology.

4 Conclusion and Future Work

MedSynapse-V performs clinical reasoning through compact latent tokens and reports stronger accuracy and efficiency across medical benchmarks than existing approaches. Future work extends latent memory evolution to longitudinal and multimodal clinical analysis.

MedSynapse-V uses compact latent tokens instead of explicit chain-of-thought generation for clinical reasoning.
Across multiple medical benchmarks, MedSynapse-V outperforms medical VLMs, general-purpose VLMs, and RL-based CoT methods in accuracy and efficiency.
Future work will extend latent memory evolution to longitudinal analysis and multimodal report generation using heterogeneous clinical evidence.

5 Implementation Details

The implementation specifies training, architectural, and evaluation configurations for MedSynapse-V, including memory sampling components, data augmentation, and benchmark-specific answer extraction. Qualitative comparisons also report shorter outputs than competing CoT methods.

Training Configuration: Training uses random rotation, horizontal flipping, brightness/contrast adjustment, and color jittering while preserving diagnostic features and anatomical orientations.
Architectural Details: The Diagnostic Memory Sampler is a 2-layer Transformer with 8 heads and 16 meta-query probes, projecting outputs into the 4096-dimensional hidden space.
Diagnostic Memory Sampler: The sampler uses frozen anatomical features as key–value pairs for learnable probes and produces compact implicit memories injected into the VLM hidden stream.
Evaluation Details: Closed-ended tasks use option-letter extraction, while SLAKE and PathVQA report accuracy aggregated across their closed-ended and open-ended subsets.
Qualitative Comparison: MedSynapse-V produces approximately 38–43-token diagnoses, compared with approximately 195–215-token reasoning chains from other methods.

6 Additional Qualitative Results

Qualitative evaluations show MedSynapse-V identifying correct findings with concise outputs, while competing RL-CoT systems produce longer chains that can amplify hallucinated observations. Additional analysis highlights borderline cases where memory discrimination is insufficient.

Qualitative Comparison: Med-R1 and MMedExpert-R1 generate extensive reasoning chains whose hallucinatory observations can propagate into erroneous diagnoses.
Qualitative Comparison: MedSynapse-V identifies correct findings in 38–43 tokens without explicit chain-of-thought generation.
Failure Case Analysis: A correctness plot shows high-confidence predictions are predominantly correct, while some low-confidence cases remain incorrect.
Failure Case Analysis: Borderline cases such as benign versus dysplastic nevi fall below the memory’s discriminative granularity.

7 Benchmark Dataset Statistics

The evaluation covers seven medical multimodal benchmarks spanning closed-ended, multi-choice, mixed-modality, pathology, and broad multi-granularity tasks. Training uses expert-curated image–text and reinforcement-learning data, while per-modality accuracy is reported for OmniMedVQA.

Benchmark Coverage: The benchmark suite includes VQA-RAD, SLAKE, PathVQA, PMC-VQA, MMMU Health & Medicine, MedXpertQA-MM, and GMAI-MMBench.
Task Formats: Closed-ended tasks include VQA-RAD, SLAKE, and PathVQA, while multi-choice tasks include PMC-VQA, MMMU Health & Medicine, and MedXpertQA-MM.
Training Data: Stage I uses 50K expert-curated image–text pairs, and Stage II uses a specialized mixed-modality reinforcement-learning set of 4K samples.
Per-Modality Results: Table 4 reports per-modality accuracy on OmniMedVQA and improvement over the Qwen3-VL-8B zero-shot baseline.

8 Additional Analysis

The additional analyses show that MedSynapse-V improves performance across imaging modalities, focuses memory attention on diagnostically relevant regions, and adds little inference overhead. Ablations further examine memory synthesis, masking robustness, and evaluation templates.

Cross-modal performance: MedSynapse-V gains most on radiology-centric modalities, improving CT by +14.4, MRI by +14.9, and X-ray by +13.1 in OmniMedVQA.The method achieves consistent gains across all eight evaluated imaging modalities.
Causal attention visualization: CCR concentrates memory attention on retinal lesions and dermoscopic lesion boundaries while suppressing healthy or background regions.Fundus attention aligns with microaneurysms and hard exudates; dermoscopy attention tracks asymmetry, border irregularity, and color heterogeneity.
Inference efficiency: MedSynapse-V has 102 ms prefill latency, matching the vanilla baseline, while its injected memory vectors enter the KV cache without extra decoding cost.The latent conditioning also steers outputs toward shorter responses of approximately 34–44 answer tokens.
Memory synthesis: Average-pooling MedSAM features into 16 tokens improves zero-shot performance by +4.2 percentage points, confirming the value of anatomical priors.The analysis compares alternative memory aggregation and question-token conditioning strategies.
Evaluation templates: Closed-ended prompts adapt to datasets with 2–5 answer options, while Mauto is generated autonomously and injected into the hidden stream without changing the text prompt.The prompt templates cover the evaluated multi-choice and open-ended settings.
Mask robustness: CCR accuracy remains stable across mask thresholds τ ∈[0.5, 0.8], with only a 1.1-point spread, and all thresholds beat the 63.6% no-mask baseline.The robustness analysis varies confidence thresholds and mask-rank selection.

9 Evaluation Prompt Templates

The evaluation uses minimal prompt templates and separates closed-ended and open-ended formats. MedSynapse-V generates autonomous memory internally, while the reported tables specify the experimental configurations and output budgets.

Prompt design: Evaluations use a brief system instruction with the clinical query and image, without few-shot examples or chain-of-thought elicitation.This design isolates model capabilities from elaborate prompt engineering.
Memory generation: For MedSynapse-V, Aψ generates 16 diagnostic memory vectors from the VLM’s visual features and injects them into the hidden stream.The memory is produced autonomously rather than added to the textual prompt.
Experimental configuration: Table 9 reports memory-injection position and GRPO group-size analyses, while Table 10 reports divergence and causal-reward sensitivity.The tables use the full three-stage pipeline with IMT inference and highlight default configurations.
Output formats: Closed-ended outputs typically contain 1–3 tokens, whereas open-ended outputs typically contain 20–40 tokens.These budgets distinguish the two evaluation formats.

10 Extended Ablation Studies

The extended ablations identify preferred settings for memory placement, GRPO sampling, divergence alignment, and causal reward weighting. Together, they show that performance depends on preserving question-conditioned memory use and balanced optimization signals.

Memory injection position: Injecting memory after question tokens and before answer generation yields the best average accuracy of 69.3%.Earlier injection before visual tokens reaches 65.5%, while interleaving memory with question tokens reaches 68.2%.
GRPO group size: GRPO group size G=4 provides the optimal accuracy–cost balance, while larger groups add only +0.1–0.3 percentage points at 1.4–2× higher GPU cost.G=2 produces noisier advantage estimates with 67.3% accuracy.
Causal reward weight: Causal reward weighting performs robustly for λcausal ∈[0.3, 0.7] and peaks at 0.5.Removing the causal reward yields 65.4%, while values at least 1.0 over-penalize trajectories and destabilize training at 67.7%.

11 Related Works

MedSynapse-V connects latent computation, reinforcement learning for vision-language models, and medical image understanding. Its distinctive contribution is to combine continuous latent memory with anatomical priors and causal counterfactual rewards.

Latent computation and memory: Latent-computation research uses continuous hidden states for generation, while MedSynapse-V applies latent memory evolution to medical diagnosis.The related work distinguishes native latent reasoning and latent-regulated generation.
Reinforcement learning for VLMs: Prior medical VLM work uses reinforcement learning and chain-of-thought methods to improve reasoning and diagnostic assistance.The broader RL literature also covers spatial, geometric, chart, and visual reasoning tasks.
Causal reinforcement learning: MedSynapse-V extends causal counterfactual reward design by using region-level interventions to distinguish diagnostic memory contributions.The approach builds on GRPO while targeting memory relevance and training stability.
Medical image understanding: Its anatomical encoder derives spatial priors from large-scale segmentation pretraining for medical image understanding and deployment.This connects the method to annotation-efficient segmentation, pathology detection, and multicenter diagnosis research.