PaperlayerView exact revision on arXiv

Source-linked AI summary

MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model

Manyu Li, Ruian He, Chenxi Ma, Weimin Tan, Bo Yan

arXiv:2511.11407v1cs.CV

TL;DR

High-quality multimodal training data for microscopy reasoning remains scarce, and MicroVQA is a small evaluation benchmark. MicroVQA++ constructs a microscopy VQA dataset from literature figure–caption pairs, filters samples with a cross-modal graph, and reports that a fine-tuned 4B model matches GPT-5 on MicroVQA while setting an open-source state of the art.

  • Problem

    Microscopy lacks large-scale, high-quality multimodal training data, while MicroVQA has only 1,042 evaluation samples and does not address that scarcity.

    Source: §1 · pp. 1–2

  • Method

    MicroVQA++ derives grounded QA and MCQ supervision from expert-validated literature figure–caption pairs and uses HiCQA-Graph to filter inconsistent image, caption, and QA samples.

    Source: §1 · p. 2

  • Results

    A 4B-parameter MLLM fine-tuned on MicroVQA++ matches GPT-5 on MicroVQA and achieves new state-of-the-art performance among open-source MLLMs in microscopy reasoning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that carefully constructing high-quality data can advance microscopy reasoning with small MLLMs, and that MicroVQA++ offers a larger, harder evaluation set.

    Source: §5 · p. 9

    On the MicroVQA++ test set, reported failure modes include visual mislocalization, distractor susceptibility, and unsupported caption-like biomedical priors.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

Multimodal Large Language Models are increasingly applied to biomedical imaging, yet scientific reasoning for microscopy remains limited by the scarcity of large-scale, high-quality training data. We introduce MicroVQA++, a three-stage, large-scale and high-quality microscopy VQA corpus derived from the BIOMEDICA archive. Stage one bootstraps supervision from expert-validated figure-caption pairs sourced from peer-reviewed articles. Stage two applies HiCQA-Graph, a novel heterogeneous graph over images, captions, and QAs that fuses NLI-based textual entailment, CLIP-based vision-language alignment, and agent signals to identify and filter inconsistent samples. Stage three uses a MultiModal Large Language Model (MLLM) agent to generate multiple-choice questions (MCQ) followed by human screening. The resulting release comprises a large training split and a human-checked test split whose Bloom's level hard-sample distribution exceeds the MicroVQA benchmark. Our work delivers (i) a quality-controlled dataset that couples expert literature with graph-based filtering and human refinement; (ii) HiCQA-Graph, the first graph that jointly models (image, caption, QA) for cross-modal consistency filtering; (iii) evidence that careful data construction enables 4B-scale MLLMs to reach competitive microscopy reasoning performance (e.g., GPT-5) and achieve state-of-the-art performance among open-source MLLMs. Code and dataset will be released after the review process concludes.

1. Introduction

Microscopy MLLM reasoning is constrained by scarce high-quality training data, while existing evaluation data is small. MicroVQA++ builds a microscopy training corpus with graph-based quality filtering and reports strong 4B-scale benchmark performance.

Motivation

  • MicroVQA++ addresses the shortage of high-quality microscopy training data; MicroVQA has only 1,042 evaluation samples and is not a large-scale supervised-finetuning dataset.The paper says the small benchmark is suitable for evaluation but unlikely to alleviate data scarcity for further progress.Source: §1 · pp. 1–2

Dataset construction

  • The pipeline derives grounded QA pairs from expert-validated literature figure–caption pairs, then produces MCQs with explicit rationales.Question types follow microscopy workflows spanning visual understanding, hypothesis generation, and experiment proposal.Source: §1 · p. 2

Dataset scale

  • Compared with MicroVQA, MicroVQA++ has 19.2× more training questions and 5.8× more test questions, plus 33.7× and 20.4× more images in those splits.These scale comparisons are reported in Table 1.Source: Table 1 · p. 2

Contributions

  • A 4B-parameter MLLM fine-tuned on MicroVQA++ matches GPT-5 on MicroVQA and sets a new open-source microscopy-reasoning state of the art.The paper identifies MicroVQA++ as the first large-scale multimodal VQA training dataset specifically tailored to microscopy.Source: §1 · p. 2

Quality control

  • HiCQA-Graph jointly models images, captions, and QAs, combining NLI entailment, CLIP alignment, and agent signals to filter unreliable samples.The graph targets errors and hallucinations in automatically generated supervision.Source: §1 · p. 2

2. Related work

Related work contrasts microscopy benchmarks and MCQ-generation strategies with MicroVQA++’s larger, graph-filtered dataset. It frames the graph as a way to combine weak consistency signals for data filtering.

2.1. MLLM benchmarks in microscopy

  • The pipeline figure depicts sampling BIOMEDICA figure–caption pairs, constructing QA pairs, checking cross-modal consistency, and generating MCQs with rationales after validation.The accompanying caption notes that human checks follow graph validation.Source: Figure 2 · p. 3
  • MicroVQA++ provides 20K training and 6K test questions, exceeding MicroVQA in scale and the proportion of depth-oriented items.MicroVQA is described as deeper in reasoning than µBench, but has only 255 images and 1,042 QA pairs; µBench covers 22 tasks and 17,235 images, with shallower scientific depth.Source: §2.1 · pp. 2–3

2.2. QA and MCQ construction

  • Unlike prior prompt-based and expert-heavy MCQ pipelines, HiCQA-Graph models image, caption, and QA relationships under weak supervision to filter data for QA/MCQ construction.The passage contrasts this with µBench’s GPT-4o conversion followed by expert verification and MicroVQA’s expert–agent strategy.Source: §2.2 · p. 3

2.3. Graph-based data generation and filtering

  • The paper uses message passing with GraphSAGE and GAT to amplify consistency signals such as image–text alignment and textual entailment for data filtering.It situates this approach among graph-based denoising and weak-supervision methods.Source: §2.3 · p. 3

3.1. Motivation

The motivation is to address microscopy’s shortage of high-quality multimodal datasets. BIOMEDICA offers expert-curated captions and a large microscopy subset as a foundation for dataset construction.

Motivation

  • Microscopy lacks high-quality multimodal datasets, a bottleneck that the paper says limits the field’s potential.Source: §3.1 · p. 3
  • BIOMEDICA’s microscopy subset contains roughly 2.5M image–caption pairs, and its publication-derived captions are written, checked, and curated by human experts.The paper presents these properties as a foundation for building a large-scale multimodal microscopy dataset.Source: §3.1 · p. 4

3.2. HiCQA-Graph

HiCQA-Graph represents microscopy samples with image, caption, and QA nodes, then passes cross-modal consistency evidence through typed relations to predict weak labels. Its message-passing network combines image links, caption entailment, and QA similarity.

3.2.1. Notations formulation

  • The graph formalizes each node as an embedded image or text, with image, caption, and QA node types.Image nodes link to their caption and associated QAs; caption-to-QA edges carry NLI entailment, and QA-to-QA edges carry nonnegative cosine similarity.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.2. Node definition

  • QA-node Keep labels fuse image–QA CLIP similarity with Caption→Answer NLI entailment into a soft confidence score.A separate three-way label represents capacity generated by the MLLM agent.Source: §3.2.2 · p. 4

3.2.3. Graph neural network

  • HiCQA-Graph defines four directed relations: describe by, asked about, supports, and similar.Its image and QA node features include an additional consistency score, while captions use CLIP text features.Source: §3.2.3 · p. 5
  • Two heterogeneous message-passing layers sum relation-specific outputs, using GraphSAGE for image links and edge-aware GATv2 for caption-support and QA-similarity links.The GATv2 attention uses NLI entailment and textual similarity as edge attributes.Source: §3.2.3 · p. 5
  • Two MLP heads supervise QA nodes, predicting soft KEEP logits and three-way EU/HG/EP capacity logits.Training uses cross-entropy for both heads.Source: §3.2.3 · p. 5

3.3. Data generation

The data-generation pipeline uses an MLLM agent to create QA pairs from figure-associated answers, then generates multiple-choice questions with distractors and explanations. The test set receives human review for significant errors.

3.3.1. QA generation pipeline

  • An MLLM agent generates QA samples spanning EU, HG, and EP capacity types from figure-associated answers; overlong pairs are summarized to fit text-token limits.The source answers are derived directly from BIOMEDICA captions.Source: §3.3.1 · p. 5

3.3.2. MCQs generation pipeline

  • For MCQs, an MLLM agent generates three distractors and an explanation, and humans review the MicroVQA++ test set for significant errors.The paper reports no information leakage between the MicroVQA++ train and test distributions.Source: §3.3.2 · p. 5
  • Figure 4 presents a two-dimensional t-SNE visualization of CLIP embeddings for images and questions; t-SNE was run with perplexity 30 for 1,000 iterations.Source: Figure 4 · p. 6

3.4. Data quality analysis

MicroVQA++ contains a larger share of high-difficulty questions than MicroVQA. Its image and question distributions show dataset-specific structure, while cross-dataset overlap and image-question co-location inform how to interpret transfer and filtering.

3.4.1. Bloom’s level

  • MicroVQA++ has a substantially higher proportion of high-difficulty samples and a larger overall scale than MicroVQA.Difficulty is assessed using Bloom’s levels.Source: §3.4.1 · p. 6

3.4.2. CLIP feature distribution

  • MicroVQA++ has compact, uniform image features and broader image coverage, while its question features differ substantially from MicroVQA’s.The CLIP feature comparison is visualized with t-SNE.Source: §3.4.2 · p. 6
  • Cross-dataset overlap marks transfer-friendly regions where content and language conventions align and cross-dataset generalization after SFT tends to be strongest.The overlap between MicroVQA++ and MicroVQA is non-negligible.Source: §3.4.2 · p. 6
  • Image and question points from the same dataset occupy nearby CLIP neighborhoods, supporting CLIP alignment as a prior and the combination of NLI with vision-language signals.The combined textual NLI and vision-language alignment is reported as superior to either signal alone.Source: §3.4.2 · p. 6

3.5. MLLMs training details

Training experiments compare free-form QA and MCQ supervision with SFT and GRPO. QA SFT targets full answers, MCQ SFT targets rationales and choices, and GRPO adds format and accuracy rewards, with reported incremental gains after SFT.

3.5. MLLMs training details

  • Training compared free-form QA with close-form MCQ, and SFT with GRPO, using an InternVL-style instruct model unless otherwise noted.Microscopy images were resized to a 448-pixel short side.Source: §3.5 · p. 6

3.5.1. Supervised fine-tuning

  • SFT used LoRA rank 16 across attention projections and MLP blocks, with AdamW, a 1e-4 learning rate, and three epochs.Training used cosine decay with 5% warm-up, bfloat16, and an effective batch size of 256 sequences.Source: §3.5.1 · p. 6
  • QA-based SFT trains on the full textual answer with next-token cross-entropy; QA-only supervision improved factual grounding and descriptive precision but did not fully train option discrimination.The input contained an instruction, image, and question, with a microscopy-focused instruction encouraging concise, evidence-grounded responses.Source: §3.5.1 · p. 6
  • MCQ-based SFT trains the model to produce a brief rationale and a structured final choice tag from the instruction, image, question, and four options.The MCQ input and output format distinguish this objective from full-answer QA SFT.Source: §3.5.1 · pp. 6–7

3.5.2. Group relative policy optimization

  • GRPO sampled six completions per prompt and shaped advantages relative to the group mean using a frozen SFT reference policy.Optimization ran for one epoch over the MCQ subset.Source: §3.5.2 · p. 7
  • GRPO sharpened option calibration and format adherence, providing reliable incremental gains when used after SFT.Its rewards combine output-format and answer-accuracy signals.Source: §3.5.2 · p. 7

4. Experiments

Experiments evaluate MicroVQA++-trained models on MicroVQA and MicroVQA++ and compare graph-based filtering and training choices. SFT improves small-model performance on MicroVQA, while the higher-difficulty, low-overlap MicroVQA++ test set presents a stronger generalization check.

4.1. Dataset and evaluation metric

  • Models are trained on the MicroVQA++ training set and evaluated on MicroVQA and MicroVQA++ test sets using a shared inference prompt and MCQ accuracy across EU, HG, and EP.Accuracy is computed by regex-based matching.Source: §4.1 · p. 7

4.2. Comparison experiments

  • 22.7% and 27.5% relative Avg improvements for InternVL3.5-2B/4B-Instruct after SFT; the 4B model matches the strongest commercial models on MicroVQA.The comparison is reported on the MicroVQA benchmark.Source: Table 2 · p. 7, §4.2 · p. 7
  • +0.6 absolute gain for InternVL3.5-4B-Instruct after SFT on the MicroVQA++ test set, whose harder Bloom’s-level distribution and intentionally low train-test overlap raise generalization requirements.The test set is described as having reduced dataset bias.Source: Table 3 · p. 7, §4.2 · p. 7
  • HiCQA-Graph achieves the best average performance at the top-75% keep ratio, while NLI-only and CLIP-only filtering underperform NCLIP.NCLIP combines NLI and CLIP scores.Source: Table 4 · p. 8, §4.2 · p. 8

4.3. Ablation experiments

  • Removing any weak supervision signal or the cross-modal consistency token lowers Avg; MCQ SFT outperforms QA SFT, while GRPO with MCQ is competitive.Removing capacity supervision causes a modest performance drop.Source: Table 5 · p. 8, §4.3 · p. 8, Table 6 · p. 8

4.5. Computational overheads

  • 129.42 ms training and 109.82 ms inference per image end-to-end on one RTX 3090; NLI inference and CLIP encoding dominate graph construction.Graph training and inference are lightweight by comparison.Source: Table 7 · p. 8, §4.5 · p. 8

4.4. Error analysis on MicroVQA++

  • Observed test-set failure modes include visual mislocalization, distractor susceptibility, and caption-like hallucinations unsupported by the image.Visual grounding errors are especially noted for small or overlapping compartments.Source: §4.4 · p. 8

5. Conclusion

MicroVQA++ turns images and captions from PubMed articles into QA and multiple-choice questions, applying HiCQA-Graph consistency checks before human review. Fine-tuning a 4B model on its training set yields competitive microscopy reasoning results, while its test set offers a larger, harder evaluation.

5. Conclusion

  • The pipeline converts PubMed article images and captions into QA and multiple-choice questions with an MLLM agent, then applies HiCQA-Graph consistency checks before human review.Source: §5 · pp. 8–9
  • After closed-form MCQ SFT on MicroVQA++ training data, InternVL3.5-4B-Instruct rivals top proprietary systems on MicroVQA and sets a new open-source MLLM state of the art.The paper concludes that careful dataset construction enables small MLLMs to advance microscopy reasoning.Source: §5 · p. 9
  • The resulting MicroVQA++ test set is larger and contains a higher proportion of harder Bloom-level items, providing stricter evaluation.Source: §5 · p. 9
Loading 2511.11407v1…