PaperlayerView exact revision on arXiv

Source-linked AI summary

MedThink: Explaining Medical Visual Question Answering via Multimodal Decision-Making Rationale

Xiaotang Gai, Chenyi Zhou, Jiaxiang Liu, Yang Feng, Jian Wu, Zuozhu Liu

arXiv:2404.12372v2cs.CV

TL;DR

MedVQA solutions often provide limited insight into how images and questions lead to answers, while direct multimodal large language models are costly and slow. The paper builds rationale-annotated benchmark datasets and fine-tunes a lightweight generative framework, MedThink, to produce answers with medical decision-making rationales. MedThink reports 83.5% accuracy on R-RAD, 86.3% on R-SLAKE and 87.2% on R-Path, exceeding comparable-parameter state-of-the-art models.

  • Problem

    Existing MedVQA datasets commonly lack intermediate decision-making rationales, limiting transparency, and direct MLLM deployment is impractical because of cost and latency.

  • Method

    The paper uses semi-automated MLLM-assisted annotation to create R-RAD, R-SLAKE and R-Path, then fine-tunes lightweight generative models with three rationale-generation strategies in MedThink.

  • Results

    83.5% accuracy on R-RAD, 86.3% on R-SLAKE and 87.2% on R-Path exceed existing state-of-the-art models with comparable parameters.

  • Takeaways & Limitations

    MedThink combines improved MedVQA performance with explicit medical decision-making rationales that clarify the model’s inference process.

Abstract

from arXiv · show

Medical Visual Question Answering (MedVQA), which offers language responses to image-based medical inquiries, represents a challenging task and significant advancement in healthcare. It assists medical experts to swiftly interpret medical images, thereby enabling faster and more accurate diagnoses. However, the model interpretability and transparency of existing MedVQA solutions are often limited, posing challenges in understanding their decision-making processes. To address this issue, we devise a semi-automated annotation process to streamline data preparation and build new benchmark MedVQA datasets R-RAD, R-SLAKE and R-Path. These datasets provide intermediate medical decision-making rationales generated by multimodal large language models and human annotations for question-answering pairs in existing MedVQA datasets, i.e., VQA-RAD, SLAKE and PathVQA. Moreover, we design a novel framework, MedThink, which finetunes lightweight pretrained generative models by incorporating medical decision-making rationales. MedThink includes three distinct strategies to generate decision outcomes and corresponding rationales, thereby clearly showcasing the medical decision-making process during reasoning. Our comprehensive experiments show that our method achieves an accuracy of 83.5% on R-RAD, 86.3% on R-SLAKE and 87.2% on R-Path. These results significantly exceed those of existing state-of-the-art models with comparable parameters. Datasets and code will be released.

I. INTRODUCTION

MedVQA systems often lack explicit decision-making rationales, limiting interpretability, while direct use of multimodal large language models is impractical because of cost and latency. MedThink addresses these gaps with rationale-annotated benchmark datasets and a lightweight generative framework.

  • Existing MedVQA datasets generally omit the reasoning linking medical questions to answers, leaving inference difficult to evaluate and clinically opaque.Manual rationale annotation is time-consuming and requires substantial medical knowledge.
  • Directly applying multimodal large language models to MedVQA is impractical in real medical scenarios because of high operational costs and significant latency.
  • MedThink introduces R-RAD, R-SLAKE and R-Path, benchmark datasets containing intermediate medical decision-making rationales for MedVQA question-answer pairs.The rationales include medical background knowledge and medical-image descriptions.
  • MedThink fine-tunes the practical 223M-parameter T5-base architecture to generate decision outcomes together with corresponding rationales.The framework uses three answering strategies to expose the medical decision-making process during inference.
  • 83.5% accuracy on R-RAD, 86.3% on R-SLAKE and 87.2% on R-Path significantly exceed existing state-of-the-art models with comparable parameters.The experiments and ablations also evaluate the usefulness of the three rationale-annotated datasets.

II. RELATED WORK

MedVQA has evolved from CNN/RNN and transformer-based feature extraction, but many systems still frame the task as classification despite open-ended clinical questions. The paper instead formulates MedVQA generatively so models can produce informed responses beyond predefined answer options.

  • A. MedVQA: MedVQA combines computer vision and natural language processing to extract, fuse and reason over medical-image and question features before generating textual answers.
  • A. MedVQA: Earlier systems commonly used CNNs for visual features and RNNs for text, while later work shifted toward transformer-based models for both modalities.
  • A. MedVQA: Classification-based MedVQA is misaligned with clinical practice because clinicians rarely face questions restricted to predefined answer options.
  • A. MedVQA: The paper redefines MedVQA as a generative task that can produce informed responses to open-ended queries using learned medical knowledge.
  • A. MedVQA: Reasoning rationales have been incorporated into language-model training or inference to guide final predictions, motivating rationale-aware multimodal reasoning for domain-specific VQA.

III. METHODOLOGY

The paper formulates MedVQA as mapping a medical image and question to both an answer and a medical decision-making rationale. Its architecture combines transformer-based textual and visual processing with cross-modal fusion to support this output.

  • A. Problem Formulation: The MedVQA mapping f(·) takes a medical image and natural-language question and generates textual outputs for the task.
  • A. Problem Formulation: The output consists of a predicted answer A and a medical decision-making rationale R explaining how the model interprets the image and question.
  • B. Model Architecture: Figure 1 summarizes dataset preparation, model architecture and rationale-generation strategies, including textual and visual encoding followed by gated cross-modal fusion.

B. Model Architecture

MedThink encodes questions and medical images separately, aligns visual features to the textual query through cross-attention, and fuses both modalities before generating an answer and rationale.

  • B. Model Architecture: The architecture uses Transformer-based TextualEncoder, VisualEncoder, and TextualDecoder components, alongside cross-attention and gated fusion.The encoders produce modality-specific representations, while the decoder generates the joint output.
  • B. Model Architecture: The TextualEncoder maps question T to FT, while the VisualEncoder maps image I to FI using token and image-patch representations.FT has n textual positions and FI has m image patches, both with hidden dimension d.
  • B. Model Architecture: Cross-attention computes attention-guided visual features HI_attn that capture image information relevant to the textual query.The query, key, and value are derived from textual and visual representations as Q, K, V = FT, FI, FI.
  • B. Model Architecture: Gated fusion dynamically combines FT and HI_attn using a sigmoid-derived coefficient λ to produce Ffuse.The fused representation is a weighted sum of textual and attention-guided visual features.
  • B. Model Architecture: The TextualDecoder receives Ffuse and generates the paired output {A, R}, comprising an answer and medical decision-making rationale.

C. Loss Function

MedThink trains the model to generate the target answer-and-rationale sequence by maximizing the likelihood of each next token given the input and preceding output tokens.

  • C. Loss Function: The model uses negative log-likelihood over the target sequence as its primary training loss.The target output is Y = {A, R}, given input X = {I, T}.

D. Three Generation Strategies

MedThink explores three output-generation strategies that vary the order and staging of medical decision-making rationale generation before producing answers.

  • D. Three Generation Strategies: The three strategies are Explanation, Reasoning, and Two-Stage Reasoning, each defining a different rationale-generation order.They are designed to study how MDMR ordering affects MedVQA output generation.
  • D. Three Generation Strategies: Explanation generates the answer A first and then the medical decision-making rationale R.
  • D. Three Generation Strategies: Reasoning reverses the order by generating the rationale R before the answer A.
  • D. Three Generation Strategies: Two-Stage Reasoning first trains a model to generate R from the image and question, then uses R with both inputs to derive A.The two stages use different models.

B. Dataset Cleaning

MedThink cleans MedVQA datasets by detecting answer inconsistencies with an MLLM, revising them with experts, and validating generated rationales against coherence, relevance, and accuracy criteria.

  • B. Dataset Cleaning: Raw datasets contain inconsistent answers to similar questions about the same image, motivating additional expert review.One chest X-ray example receives conflicting answers about a hemidiaphragm and overall image normality.
  • B. Dataset Cleaning: An MLLM systematically reviews question-answer pairs, identifies inconsistencies, and supports domain experts in revising answers.The integration is intended to expedite processing while revealing subtleties that manual cleaning may miss.
  • B. Dataset Cleaning: After cleaning, the MLLM generates medical decision-making rationales from each image, question, and correct answer, which experts check for validity and applicability.Rationales that fail review are regenerated.
  • B. Dataset Cleaning: Rationales are accepted only when they satisfy coherence, relevance to the question and clinical context, and freedom from common-sense or medical-knowledge errors.All three criteria must be met for inclusion.

A. Training Details

MedThink uses lightweight pretrained components and evaluates generative MedVQA separately for closed-end and open-end questions, with dataset-specific training schedules and reported computational costs.

  • MedThink integrates UnifiedQA encoders and decoders with DETR as the visual encoder for MedVQA.
  • Training uses a 5e-4 learning rate, dataset-specific epoch counts, and batch size 32; Two-Stage Reasoning adds a second 20-epoch phase at 5e-5.
  • Training takes approximately 2.5, 5.5, and 14 hours on R-RAD, R-SLAKE, and R-Path, respectively, while inference takes about 6 seconds per sample.
  • Closed-end questions are evaluated with accuracy, while open-end questions prioritize clinical language-generation evaluation rather than classification accuracy.

C. Main Results

MedThink is evaluated across closed-end and open-end MedVQA tasks, with the Explanation strategy achieving the strongest reported closed-end accuracies on R-RAD and R-SLAKE. Results also show that rationale strategies can improve performance over training without MDMRs, while strategy effectiveness varies by dataset and task.

  • Closed-end questions: 83.5% and 86.3% accuracy are achieved by MedThink’s Explanation strategy on R-RAD and R-SLAKE, respectively.These are the highest reported closed-end accuracies for those two datasets in the supplied results.
  • Closed-end questions: 79.5% and 82.5% accuracy are achieved by PubMedCLIP on R-RAD and R-SLAKE, below MedThink’s corresponding results.MedThink is also reported to outperform LLaVA, Med-Flamingo, and Med-Gemini, while matching the overall accuracies of the more parameter-heavy LLaVA-Med and Med-Mo.
  • Open-end questions: In open-end evaluation, Explanation leads R-RAD on five of seven metrics, including 50.2% Rouge-1 and 29.5% Rouge-L.It also records 38.3% BLEU-1, 22.9% BLEU-2, and 14.0% BLEU-3.
  • Open-end questions: Two-Stage Reasoning leads R-SLAKE on five of seven open-end metrics, including 23.1% Rouge-2 and 39.5% BLEU-1.Its other reported leading scores are 24.5% BLEU-2, 15.8% BLEU-3, and 10.3% BLEU-4.
  • Rationale visualization: The rationale illustrations align medically relevant and potentially misleading text with red-boxed anatomical evidence in the images.Green text denotes knowledge aiding the answer, red text denotes information that could lead to incorrect conclusions, and red image boxes mark the described anatomical features.
  • Ablation studies: Compared with MedThink without MDMRs, Explanation improves accuracy by 4.5% on R-RAD, 3.8% on R-SLAKE, and 1.0% on R-Path.Two-Stage Reasoning improves accuracy by 1.5%, -3.4%, and 1.2%, respectively, while Reasoning changes it by -5.1%, -1.7%, and -2.9%.

E. Case Study

The case study examines how MedThink’s generated medical decision-making rationales affect question answering and analyzes errors across anatomical regions. Accurate rationales support precise answers, whereas erroneous rationales can induce hallucinations.

  • E. Case Study: Accurate medical decision-making rationales enable MedThink to answer related medical questions effectively and precisely.
  • E. Case Study: Erroneous medical decision-making rationales can misguide MedThink and cause hallucinated answers.The analysis uses R-SLAKE because it spans questions involving six anatomical regions, creating a complex evaluation setting.
  • E. Case Study: The regional error analysis groups questions by anatomical region, counts incorrect predictions, and computes each region’s proportion of errors.These proportions are reported in Table IV, whose caption specifies that lower error rates are better.
  • E. Case Study: MedThink significantly aids medical question answering for chest and abdomen cases, although these regions still account for errors.
Loading 2404.12372v2…