PaperlayerView exact revision on arXiv

Source-linked AI summary

SeVeR: Selective Visual Exposure and Retrieval for 3D Medical Image Question Answering

Yaojun Hu, Danyang Tu, Yang Liu, Jiajin Zhang, Wei Fang, Zhiqiang Liu, Chunlai Dong, Yingda Xia, Haochao Ying, Jian Wu, Ling Zhang

arXiv:2608.25630v1cs.CV

TL;DR

Volumetric medical VQA must reason over complementary multi-sequence MRI evidence without exposing the decoder to excessive redundant tokens. The paper introduces BreMRIs-VQA and SeVeR, which compress visual inputs and retrieve complementary evidence during decoding; SeVeR improves discriminative and generative performance on the reported benchmarks.

  • Problem

    Multi-sequence MRI VQA requires integrating complementary modality evidence while avoiding redundant visual exposure, but existing methods predominantly rely on single-modal inputs.

  • Method

    SeVeR selects compact modality-wise prototypes, retrieves complementary multi-level evidence with change-aware gated attention, and regularizes retrieval using marginal utility.

  • Results

    SeVeR outperforms same-scale fine-tuned baselines on BreMRIs-VQA, with gains across discriminative and generative tasks and a favorable accuracy-latency tradeoff.

  • Takeaways & Limitations

    Selective visual exposure is supported as a broadly applicable principle for volumetric medical reasoning across multi-sequence and public single-volume benchmarks.

  • Takeaways & Limitations

    Free-text evaluation relies on automated language metrics, while radiologist-grounded human evaluation and real-world workflow integration remain future work.

Abstract

from arXiv · show

Volumetric medical VQA requires reasoning over long and redundant 3D visual token sequences, especially in multi-sequence MRI where complementary modalities provide diverse diagnostic cues but expose the decoder to many repeated anatomical regions. To investigate reasoning under multi-sequence visual redundancy, we first introduce BreMRIs-VQA, a clinically curated breast MRI benchmark with 1.19M QA pairs from 71.0K sequences and 12.9K patients, covering both free-text and multiple-choice questions. We further propose SeVeR, a selective visual exposure framework that compresses dense volumes into modality-wise prototypes and retrieves complementary multi-level evidence with change-aware gated attention during decoding, trained with a marginal-utility self-consistency objective that suppresses unhelpful retrieval. Experiments on BreMRIs-VQA and public benchmarks show that SeVeR improves both discriminative and generative performance while exposing substantially fewer visual tokens.

1 Introduction

Multi-sequence breast MRI VQA requires combining complementary modality evidence while limiting redundant visual exposure. The paper addresses these challenges with the BreMRIs-VQA benchmark and SeVeR, a selective retrieval framework evaluated across discriminative and generative tasks.

  • Motivation: Multi-sequence MRI provides complementary diagnostic evidence, but existing medical VQA methods predominantly use single-modal inputs.The paper highlights that findings such as enhancement, T2-weighted signal, and diffusion restriction may need joint interpretation.
  • Motivation: Redundant anatomical content across modalities can dilute salient findings during multi-modal visual reasoning.The paper identifies redundant visual exposure as a central challenge of multi-modal medical VQA.
  • BreMRIs-VQA: 1.19M VQA pairs from 71.0K breast MRI sequences and 12.9K patients form the clinically curated BreMRIs-VQA benchmark.The benchmark includes 671.6K free-text and 515.1K multiple-choice pairs across seven workflow-grounded task groups and six MRI modalities.
  • SeVeR: SeVeR combines Greedy Prototype Selection, Change-aware Gated Attention, and Self-Consistency Regularization for selective visual exposure.GPS supplies compact modality-wise visual memory, CaGA retrieves question-dependent complementary evidence, and SCR-MU penalizes unhelpful retrieval.
  • Evaluation: SeVeR is reported to improve performance across multiple-choice and free-text clinical tasks.The cited overview and contribution passages describe gains across discriminative and generative evaluation settings.

2 BreMRIs-VQA

BreMRIs-VQA is constructed as an evidence-constrained, clinically curated benchmark for multi-sequence breast MRI reasoning. Its pipeline structures expert-verified reports into workflow-grounded questions, then paraphrases and filters them without changing their clinical supervision.

  • Dataset scope: 71,041 breast MRI sequences from 12,891 patients cover six heterogeneous MRI modalities, with at least three sequences per patient.The studies are paired with radiology and pathology reports verified by clinical experts.
  • Dataset construction: The dataset pipeline uses expert-verified reports as evidence sources rather than asking the language model to invent clinical facts from images.The process is described as LLM-assisted but evidence-constrained.
  • Dataset construction: Structured extraction normalizes reports into a closed clinical vocabulary and discards schema-invalid or out-of-vocabulary outputs before question generation.Unmentioned fields are set to null during extraction.
  • Task design: Seven deterministic workflow-grounded task groups span background assessment, lesion characterization, functional reasoning, diagnosis, and pathology prediction.Each question is tied to a source key, answer space, and task label.
  • Quality control: Paraphrasing diversifies wording only after source keys, answer labels, and answer spaces are fixed, while filtering removes attribute mismatches and answer leakage.Candidates are re-bound to their templates and rejected when they violate the predefined supervision constraints.
  • Quality control: Radiologist auditing and automated factual checks retain QA pairs with an average overall score of 2.85/3 and a 97.5% expert pass rate.Candidates with weak report support are discarded or escalated for manual review.

3 Method

SeVeR processes multi-modal 3D volumes by first compressing each modality into representative prototypes, then retrieving question-conditioned evidence during decoding. Its gated retrieval and marginal-utility training objective aim to reduce redundant visual exposure while retaining useful diagnostic information.

  • Visual encoding: Each modality volume is independently encoded with a shared 3D Vision Transformer into a dense spatial token set.The token count L_m corresponds to the number of patches in modality m.
  • Greedy Prototype Selector: Greedy Prototype Selection compresses each modality into k representative tokens that maximize affinity coverage while suppressing repeated visual content.The method normalizes token features, computes cosine-similarity affinities, and greedily selects prototypes with maximal marginal coverage improvement.
  • Change-aware Gated Attention: GPS prepares a compact, question-agnostic visual memory, while Change-aware Gated Attention retrieves complementary evidence from a multi-level feature bank during decoding.The decoder begins with prototypes from all modalities and question tokens, then uses language-conditioned hidden states for retrieval.
  • Change-aware Gated Attention: The retrieval gate is computed from mean cosine similarity between consecutive decoder layers and controls when multi-level cues are injected.A threshold and sharpness parameter map layer-to-layer similarity to a sigmoid gate, followed by gated fusion with bank features.
  • Marginal-Utility Regularization: Marginal-utility regularization compares retrieval-enabled and retrieval-disabled passes, penalizing retrieval when it fails to improve task loss beyond a soft margin.The disabled branch serves as a stop-gradient baseline, discouraging trivial always-on or always-off gating behavior.

4 Experiment

Experiments evaluate SeVeR across BreMRIs-VQA, public 3D benchmarks, missing-modality settings, interpretability analyses, ablations, prototype budgets, and backbone families. SeVeR generally improves performance and efficiency, with gains tied to selective retrieval and compact visual exposure.

  • Main Results on BreMRIs-VQA: SeVeR at 4B scale achieves the best overall average on BreMRIs-VQA, with its largest improvements on cross-sequence integration tasks.On Morphological Characterization, its multiple-choice accuracy marginally trails fine-tuned Qwen3-VL, suggesting compact prototypes may not fully preserve fine-grained shape information.
  • Efficiency Analysis: 70.21% accuracy at k=512 in 644 ms exceeds the full-token baseline's 69.18% in 953 ms.This comparison uses the same Qwen2.5-VL backbone and budget conditions as the evaluated pruning methods.
  • Modality Robustness and Interpretability: SeVeR maintains a smaller accuracy drop than its no-prototype variant when key modalities are absent, especially in settings S3 and S5.CaGA redistributes retrieval weights across available sequences rather than relying on fixed token positions.
  • Modality Robustness and Interpretability: Early layers favor structural modalities, intermediate layers increasingly focus on DCE, DWI, and ADC, and deeper layers produce sparse attention over relevant regions.The observed progression moves from broad anatomical exploration toward selective semantic condensation.
  • Transfer Benchmarks: SeVeR ranks first overall on both multiple-choice and free-text metrics on DeepTumorVQA and leads on Visual Reasoning and Medical Reasoning.On 3D-RAD, it achieves the highest accuracy on all three classification and temporal tasks and leads on most generation metrics.
  • Ablation Study: Removing CaGA causes the largest ablation drop, while removing SCR-MU most degrades free-text quality and prevents late-layer attention from sparsifying.The question-conditioned GPS variant also underperforms greedy GPS, supporting question conditioning through CaGA.
  • Prototype Budget: Accuracy improves with prototype budget up to k=512, then plateaus or declines while latency continues to rise.The authors therefore use k=512 as the default because it balances accuracy gains with latency below the full-token baseline.
  • Backbone Compatibility: SeVeR achieves consistent gains with Qwen2.5-VL-3B and Qwen3-VL backbones without architecture-level modifications.Reported accuracies are 70.21% for Qwen2.5-VL-3B and 70.57% and 72.13% for Qwen3-VL at 4B and 8B.

5 Conclusion

The paper introduces BreMRIs-VQA and SeVeR to support multi-sequence volumetric medical VQA with compact visual exposure and dynamic evidence retrieval. SeVeR outperforms same-scale fine-tuned baselines, improves the accuracy-latency trade-off, and transfers to public 3D benchmarks.

  • Conclusion: BreMRIs-VQA contains 1.19M QA pairs from 12.9K multi-sequence breast MRI cases.The benchmark is clinically curated and supports workflow-grounded VQA.
  • Conclusion: SeVeR combines prototype selection for compact global coverage with dynamic fine-grained evidence retrieval during decoding.Its self-consistency objective suppresses redundant retrieval.
  • Conclusion: SeVeR outperforms same-scale fine-tuned baselines on BreMRIs-VQA, with largest gains on cross-sequence integration tasks.It also has a more favorable accuracy-latency trade-off than token-pruning methods.
  • Conclusion: Results on public 3D single-volume benchmarks support selective visual exposure as applicable beyond multi-sequence breast MRI.The conclusion frames this as a broadly applicable principle for volumetric medical reasoning.

Limitations

The paper identifies limitations in evaluation, adaptive compression, and clinical deployment. Its free-text assessment relies on automated metrics, prototype budgets remain fixed per sequence, and real-world radiology workflow integration is outside scope.

  • Evaluation: Free-text generation is evaluated with automated language metrics that may not fully capture clinical nuance.Radiologist-grounded human evaluation is left for future work.
  • Compression: GPS uses a fixed prototype budget per sequence, leaving adaptation to sequence-level complexity unexplored.The limitation concerns compression ratios rather than the benchmark split or QA construction process.
  • Clinical Deployment: SeVeR is validated on algorithmic benchmarks, while integration studies in real-world radiology reading workflows remain beyond the paper's scope.The authors identify workflow studies as a direction for strengthening clinical applicability and deployment.

A.3 Dataset Statistics and Task Taxonomy

BreMRIs-VQA organizes questions across imaging levels and clinically meaningful task groups, with breast-level annotations providing the largest volume and natural-language answers using standardized clinical categories.

  • Breast-level annotations generate the largest question volume because every case supplies broad contextual information, regardless of lesion presence.Lesion- and pathology-level questions provide narrower complementary coverage when corresponding findings are annotated.
  • Ribbon widths show that labels are distributed across multiple clinically meaningful attributes without one attribute dominating within task groups.This distribution supports broad evaluation coverage across the benchmark taxonomy.
  • Question word clouds emphasize anatomical and imaging descriptors, while answers concentrate on standardized clinical categories such as yes, no, minimal, moderate, and positive.The visualization is computed from natural-language question and answer text after removing common function words.
  • The taxonomy spans breast-level, lesion-level, pathology-level, and free-text diagnostic reasoning questions.These categories support coverage from structured imaging attributes to holistic clinical reasoning.

B.2 Clinical Workflow Task Organization

The benchmark organizes VQA around radiologists’ progressive workflow, moving from global breast context and lesion assessment through multimodal reasoning to holistic diagnosis and pathology prediction.

  • Clinical Workflow Task Organization: Clinical reasoning progresses from global context and lesion detection toward integrated diagnostic conclusions rather than isolated attribute predictions.The workflow is designed to reflect how radiologists progressively combine information across analysis levels.
  • Global Background Assessment: Global background assessment covers fibroglandular tissue, background enhancement, symmetry, and post-surgical changes that influence lesion visibility and diagnostic difficulty.
  • Lesion Analysis and Multimodal Reasoning: Lesion-focused tasks include detection, localization, morphology, multi-sequence signal, enhancement kinetics, non-mass enhancement, and non-enhancing lesions.
  • Integrated Diagnosis and Pathology: Later workflow tasks address local invasion, lymph nodes, BI-RADS assessment, management recommendations, and pathology outcomes.Pathology questions include tumor characteristics, margins, biomarkers, and pathological staging.
  • Quality Assurance: Quality control evaluates factual consistency, clinical validity, and overall report support, retaining fully grounded candidates while removing or reviewing weakly supported pairs.The scoring rubrics use 0–3 scales for each criterion.
  • Motivation and Scope: The benchmark addresses limited systematic evaluation of workflow-grounded multi-sequence MRI reasoning, while related medical VQA work often uses single-volume or dense-token inputs.

E Ethics, Privacy, and Data Governance

The benchmark uses report-grounded structured extraction and deterministic filtering within approved retrospective-data governance procedures.

  • Ethics and Governance: The retrospective study and clinical-data use were approved through institutional ethics and data-governance procedures.Data handling follows the institutional protocol for retrospective medical research.
  • Grounding Controls: Templates bind each question to a clinical task, source key, allowed answer space, and question form to prevent unsupported queries.
  • Grounding Controls: Structured extraction requires valid schema-conforming JSON, assigns null to unmentioned attributes, and excludes unsupported or invalid outputs.
  • Method Interface: GPS takes a modality token matrix and prototype budget as input, returning selected prototype indices and position-aware prototypes.
  • Final Filtering: Final filtering removes unsupported, missing-attribute, out-of-vocabulary, semantically drifted, duplicate, contradictory, empty, and malformed QA pairs.Multiple-choice questions also undergo answer-leakage filtering.

F Method Details

SeVeR selects compact modality-specific prototypes by greedy affinity coverage, then uses a differentiable surrogate for end-to-end training while preserving hard selection at inference.

  • Greedy Prototype Selection: GPS iteratively selects k modality tokens that maximize marginal affinity coverage after L2 normalization and cosine-similarity computation.
  • Straight-Through Estimation: The straight-through estimator combines hard selected prototypes with a soft affinity-weighted representation during optimization.The soft representation is a differentiable surrogate for discrete selection.
  • Straight-Through Estimation: During the forward pass, the model uses the discrete greedy prototypes, matching the behavior at training and inference.
  • Straight-Through Estimation: During backpropagation, gradients flow through the soft surrogate and its affinity-based dependence on upstream visual features.
  • Trainable Components: Training updates the upstream 3D ViT and projection layers shaping the affinity space, rather than the greedy index operation itself.Changed feature geometry can alter selected tokens on later forward passes.
  • Retrieval Regularization: Marginal-utility self-consistency regularization encourages retrieval gates to activate only when additional visual evidence reduces task loss.It is introduced after supervised warm-up to stabilize optimization.

G.2 Two-Stage Training Protocol

The paper uses a staged training protocol for SeVeR and evaluates it against general, medical, 3D-specific, and token-pruning baselines. Scaling the backbone improves performance across all seven tasks and both metrics, while subtype results show strong performance on reasoning-intensive DeepTumorVQA categories.

  • Two-Stage Training Protocol: Phase 1 freezes the LLM backbone while training the visual encoder, GPS module, and cross-modal projector for domain-relevant visual alignment.For BreMRIs-VQA, captioning full radiology reports from multi-sequence MRI volumes provides the Phase 1 objective.
  • Two-Stage Training Protocol: Phase 2 unfreezes all parameters and jointly fine-tunes SeVeR on supervised VQA data with a standard language-modeling objective.The staged activation is intended to prevent degenerate early-stage gate saturation.
  • Baseline Evaluation: Baseline comparisons include Qwen3-VL, Lingshu, HuLu-Med, OmniV, M3D, VisionZip, DivPrune, MMTok, Merlin, RadFM, and CT-CHAT under their respective input and evaluation protocols.Token-pruning methods are applied at inference without additional training, while 3D-specific models are evaluated zero-shot on CT benchmarks.
  • Scaling Study: Performance improves monotonically across all seven tasks and both metrics as the backbone scales from 3B to 8B.The 4B→8B step yields +1.56 pp average accuracy and +0.15 pp BERTScore.
  • DeepTumorVQA Results: SeVeR-4B achieves the highest overall DeepTumorVQA averages in multiple-choice accuracy (0.687) and free-text performance (0.599), surpassing 13B RadFM.The largest margins occur in Visual Reasoning and Medical Reasoning, which require cross-slice spatial integration.

H.4 Full 3D-RAD Ablation Results

The 3D-RAD ablations show that CaGA, prototype count, marginal-utility regularization, and staged representation alignment each affect performance or efficiency. The modality-robustness evaluation compares SeVeR with and without selective exposure across six modality settings and free-text metrics.

  • 3D-RAD Ablation Results: Removing CaGA reduces Longitudinal Temporal Diagnosis from 75.28 to 66.66 and Static Temporal Diagnosis from 51.77 to 41.70.These drops indicate that prototype compression alone is insufficient for temporally demanding tasks requiring multi-slice evidence retrieval.
  • Modality Robustness: Figure 9 compares SeVeR and the model without selective exposure across six modality settings using BERTScore, BLEU, and ROUGE for free-text quality.The corresponding settings vary modality availability, while Table 16 reports task-wise performance under S1–S6.
  • 3D-RAD Ablation Results: Removing marginal-utility regularization decreases Existence Detection accuracy by 0.82 pp and worsens free-text BLEU and ROUGE across all three generative task groups.The ablation is associated with degenerate always-on gate patterns that dilute generation quality.
  • Training Ablation: The Phase 1-only checkpoint produces non-trivial free-text quality with BERTScore approximately 86–91 but near-zero discriminative accuracy.Supervised fine-tuning subsequently converts the aligned representation into stronger task-specific performance.
Loading 2608.25630v1…