PaperlayerView exact revision on arXiv

Source-linked AI summary

Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs

Xiaoke Huang, Ningsen Wang, Hui Liu, Xianfeng Tang, Yuyin Zhou

arXiv:2510.25867v2cs.LG

TL;DR

Medical VQA training is constrained by the lack of large, open, high-quality corpora, despite the need to reason jointly over images and text. MedVLSynther synthesizes and verifies questions from open biomedical literature; models trained on MedSynVQA show gains across six benchmarks, while the authors caution that synthetic data cannot replace curated clinical datasets.

  • Problem

    Medical VQA lacks large, openly usable, high-quality training corpora, limiting the available basis for broad and transparent model training.

    Source: §1 · p. 1

  • Method

    MedVLSynther uses rubric-guided, context-aware generation and multi-stage verification to create VQA items from open biomedical articles.

    Source: §1 · pp. 2–3

  • Results

    MedSynVQA training yields consistent gains for open-weight LMMs across six medical VQA benchmarks, and ablations indicate both generation and verification are necessary.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The open-data and open-model pipeline offers a reproducible, privacy-preserving, and transparent path to supervision for medical VQA.

    Source: §1 · p. 2, §5 · p. 9

    Synthetic data cannot replace carefully curated clinical datasets.

    Source: §1 · p. 3

Abstract

from arXiv · show

Large Multimodal Models (LMMs) are increasingly capable of answering medical questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guided generator-verifier framework that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive options under a machine-checkable JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image-text consistency), awards fine-grained positive points, and penalizes common failure modes before acceptance. Applying this pipeline to PubMed Central yields MedSynVQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with reinforcement learning using verifiable rewards improves accuracy across six medical VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up to 77.57 on VQA-RAD and 67.76 on PathVQA, outperforming strong medical LMMs. A Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers an auditable, reproducible, and privacy-preserving path to scalable medical VQA training data.

1 INTRODUCTION

Medical VQA training lacks large, open, high-quality corpora despite the growing need to reason jointly over images and text. MedVLSynther addresses this gap with rubric-guided generation and verification from open biomedical literature, producing MedSynVQA and improving open-weight models across medical VQA benchmarks.

1 INTRODUCTION

  • MedVLSynther generates self-contained multiple-choice items from figures, captions, and available in-text references, then verifies them with a multi-stage rubric.Generation enforces parallel, mutually exclusive options and a machine-checkable schema; verification applies essential gates, positive criteria, and penalties.Source: §1 · p. 2
  • The framework and training data operate on open literature and open models, allowing prompts, scoring rules, and filtering decisions to be inspected and reproduced.The authors frame this as a privacy-preserving alternative that does not rely on private patient data.Source: §1 · pp. 2–3
  • MedSynVQA training improves open-weight LMM accuracy over strong baselines across MMMU-Med, PathVQA, and VQA-RAD.The paper also reports that ablations favor both generation and verification, more verified data consistently helps, and contamination analysis finds no detectable evaluation-set leakage.Source: §1 · p. 2
  • MedSynVQA contains 13,087 questions over 14,803 images, covering 13 modalities and 28 anatomical regions.The dataset is presented as open and reproducible.Source: Table 1 · p. 3
  • The authors caution that synthetic data cannot replace carefully curated clinical datasets.They present high-quality, auditable synthesis as feasible and useful for medical VQA, not as a substitute for clinical curation.Source: §1 · p. 3

2 RELATED WORKS

Prior medical VQA resources trade off curation, breadth, training availability, and openness. MedVLSynther uses context-aware generation and verification to turn open biomedical literature into training examples intended to address these gaps.

Multimodal medical VQA

  • Expert-curated medical VQA datasets remain small or restricted to narrow modalities and topics, while broad benchmarks offer little or no training data.Open literature-derived image–caption collections provide scalable raw material, but the paper positions its work as converting that substrate into VQA training data.Source: §2 · pp. 3–4

Synthetic data generation for multimodal medical VQA

  • The depicted pipeline combines document extraction and pre-filtering with context-aware generation and multi-stage verification before threshold-based retention.Its verification stages include essential, fine-grained, and penalty criteria.Source: Figure 3 · p. 4
  • Earlier synthetic medical VQA pipelines can omit visual cues, produce ambiguous stems, and yield visually ungrounded or medically dubious answers.Some large compilations are closed, and modality-specific datasets remain narrow; these limitations motivate a quality-first strategy.Source: §2 · p. 4

Multimodal models, medical adaptation, and reasoning

  • Prior medical adaptation and reasoning work includes SFT and RL, but the paper identifies a shortage of open, high-quality multimodal supervision for medical reasoning.MedVLSynther presents its rubric-verified corpus as supervision that pairs naturally with RLVR.Source: §2 · p. 4

3 MEDVLSYNTHER AND MEDSYNVQA

MedVLSynther constructs medical multiple-choice VQA from biomedical figures and text through a rubric-guided generator and a context-aware verifier. Accepted items form MedSynVQA, which the paper uses to train medical LMMs with supervised fine-tuning or RLVR.

3 MEDVLSYNTHER AND MEDSYNVQA

  • The pipeline casts medical multiple-choice VQA synthesis as rubric-guided generation from figures and text followed by multi-stage rubric-guided quality control.The verifier screens candidate items before they enter the final corpus.Source: §3 · p. 4

3.1 DATA SOURCE, EXTRACTION, AND PRE-FILTERING

  • The source is PMC-OA figure metadata, with images, captions, and corresponding in-text references ingested when present; samples without an image or caption are discarded.Pre-filtering retains clinical imaging and microscopy labels and yields 23,788 figure-caption-reference triplets.Source: §3.1 · p. 5

3.2 RUBRIC-BASED, CONTEXT-AWARE VQA GENERATION

  • The generator receives images, captions, and references and returns a five-option JSON item whose stem, options, and answer must meet visual-grounding and medical-quality requirements.The rubric requires self-contained stems, exactly one best answer, medical correctness, parallel distractors, and a focused question; archetypes include identification, localization, diagnosis, and lesion grading.Source: §3.2 · p. 5

3.3 MULTI-STAGE, RUBRIC-BASED, CONTEXT-AWARE VERIFICATION

  • The verifier sees the same image-text context and applies an essential pass/fail gate, weighted positive criteria, and penalties for identified pitfalls.Essential checks include self-containment, a single correct option, clinical validity, and image-text consistency; malformed JSON is removed before the essential filter.Source: §3.3 · pp. 5–6

3.4 AGGREGATION AND ACCEPTANCE RULE

  • Candidates passing Stage 1 are accepted at a normalized score of at least 0.9670, yielding 13,087 MC-VQA items in MedSynVQA.The stated high threshold emphasizes precision while retaining useful yield.Source: §3.4 · p. 6

3.5 TRAINING MEDICAL LMMS WITH MEDSYNVQA

  • The corpus is used for supervised fine-tuning on reasoning-trace and answer pairs or for GRPO-based RLVR on answers alone.The RLVR reward promotes exact-match accuracy and schema adherence without over-fitting to a single imaging modality.Source: §3.5 · p. 6

4 EXPERIMENTS

Experiments compare open-weight LMM training setups across six medical VQA benchmarks. MedSynVQA-trained models achieve the strongest reported averages, while ablations examine the effects of generation, verification, data scale, model choice, and training approach.

4.1 SETUP

  • The setup finetunes Qwen2.5-VL 3B and 7B with SFT or RLVR, usually using 5K synthesized items, and reports multiple-choice accuracy on six benchmarks.Baselines include general-purpose and medical LMMs, including text-only-trained models.Source: §4.1 · pp. 6–7

4.2 RESULTS

  • Adding rubric-aware verification yields the best averages: 55.85 for 3B and 57.56 for 7B, after text-only or rubric-guided generation.Rubric-guided generation already outperforms the PMC-style recipe overall; verification adds further gains.Source: Table 2 · p. 7, §4.2 · p. 7
  • With more MedSynVQA items, 3B accuracy peaks at 55.85 with 5K items, while 7B peaks at 58.15 with the full dataset.The authors use 5K items in subsequent experiments to reduce computational cost.Source: Table 3 · p. 7, §4.2 · p. 7
  • Higher-capacity generator and verifier choices improve final average accuracy; GLM-108B as both generator and verifier reaches 58.08 for 7B.The main results retain the Qwen2.5-VL-72B verifier to maximize reproducibility with open weights.Source: Table 4 · p. 8, §4.2 · p. 8
  • RL outperforms SFT across both model sizes and all three data sources, while RL with MedSynVQA gives the highest averages: 55.85 for 3B and 57.56 for 7B.The three sources are PMC-VQA, m23k, and MedSynVQA.Source: Table 5 · p. 8, §4.2 · p. 8
  • 55.85 (3B) and 58.15 (7B) are the best open-weight-model averages considered; the models exceed MedVLThinker-7B by +0.97 and +3.27, respectively.Gains are reported as consistent across datasets, with up to 77.57 on VQA-RAD.Source: §4.2 · p. 8, Table 6 · p. 9
  • The case study presents two examples described as showing context-aware comprehension and verifier rejection of leakage.Further case-study details are in the appendix.Source: §4.2 · p. 9
  • The contamination analysis reports no overlaps between MedSynVQA and evaluation suites under the authors’ protocol.Source: §4.2 · p. 9

5 CONCLUSIONS

The conclusion presents MedVLSynther as a way to synthesize auditable medical VQA data at scale from open biomedical literature. It also reports that the manuscript used GPT-5 only for minor language refinement.

5 CONCLUSIONS

  • MedVLSynther pairs rubric-guided, context-aware generation with multi-stage verification to synthesize auditable medical VQA data at scale from open biomedical literature.MedSynVQA training delivers consistent gains for open-weight LMMs across six benchmarks; ablations indicate both generation and verification are necessary.Source: §5 · p. 9

A THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • GPT-5 was used for minor language refinement and smoothing, not for original content, data analysis, or core scientific ideas.The authors state that conceptual development, experimentation, and interpretation were conducted independently.Source: A THE USE OF LARGE LANGUAGE MODELS… · p. 14

B REPRODUCIBILITY STATEMENT

The appendix describes reproducibility resources and computing details, alongside licensing and research-use boundaries for MedVLSynther.

B REPRODUCIBILITY STATEMENT

  • The authors describe the data, code, models, prompts, rubrics, and filtering logs as resources for reproduction and auditing.Training used eight A100 GPU machines with mixed precision and hyperparameters following MedVLThinker.Source: B REPRODUCIBILITY STATEMENT · p. 14

C ETHICAL CONSIDERATIONS AND LICENSING

  • MedVLSynther uses PMC-OA literature, records source metadata and licenses, and excludes items whose licenses are incompatible with research redistribution.The dataset is intended exclusively for research, with no clinical use authorized.Source: C ETHICAL CONSIDERATIONS AND LICENSING · p. 14

D MORE DATA STATISTICS

The appendix labels additional MedSynVQA statistics and provides generation and verification prompts. The prompts specify item-writing constraints, a staged rubric, and structured output requirements.

D MORE DATA STATISTICS

  • Table 7 is identified as reporting image secondary labels from original Biomedica statistics.Source: Table 7 · p. 14

E PROMPTS

  • The generation prompt requires a self-contained MCQ grounded in figure and caption/context, with one correct answer and plausible, parallel options in strict JSON.It directs the writer to choose an archetype and produce five options before checking the rubric.Source: E PROMPTS · pp. 15–16
  • The verifier assesses MCQs using the figure, caption, and context, requiring supported clinical facts and exactly one correct option.The prompt distinguishes objective essential checks from stricter bonus-criteria review.Source: E PROMPTS · pp. 16–17
  • Verification output is a strict JSON rubric with criterion titles, categories, weights, scores, and brief notes; detected failure modes can incur penalties.The prompt specifies category-dependent weights and limits notes to 12 words.Source: E PROMPTS · pp. 17–18
  • The rubric separates fixed essential criteria from bonus criteria, with essential failures discarding an item and bonus scoring defaulting to zero absent irrefutable evidence.The listed essentials include self-containment, a single correct option, clinical validity, and image–text consistency.Source: E PROMPTS · p. 17

F DATA CONTAMINATION ANALYSIS

The audits found minimal textual overlap and no detected image overlap between MedSynVQA and the held-out test set. The figure’s examples illustrate that text or image similarity can reflect shared options, source variants, or modality rather than duplicated questions or instances.

Contamination examples

  • Figure 6 shows similar text-neighbor options paired with different images, and an image neighbor reflecting modality similarity rather than instance identity.Its pHash example retrieves a same-source size variant with different questions and answers, not an exact duplicate.Source: Figure 6 · p. 18

Text decontamination

  • The text audit normalized questions and options before screening 13,087 training rows against 8,220 held-out rows.It lowercased text, cleaned whitespace, masked digits, and normalized option labels.Source: F DATA CONTAMINATION ANALYSIS · p. 18
  • 0.280% (23/8,220) of test items exceeded the text-embedding overlap threshold, while the audit found no text overlap beyond trivial lexical matches.The audit also found no near-duplicate strings using MinHash and Levenshtein.Source: F DATA CONTAMINATION ANALYSIS · pp. 18–19

Image decontamination

  • No image overlap was found using pixel- and layout-level hashing alongside semantic image embeddings.Statistics used images stored in parquet, with the first image per record by default.Source: F DATA CONTAMINATION ANALYSIS · p. 19

G EXPERT EVALUATION STATISTICS

The intrinsic-quality evaluation used five board-certified imaging experts to review MedSynVQA questions. Their clinical backgrounds covered several imaging specialties, and some items received joint review when they crossed organs or modalities.

G EXPERT EVALUATION STATISTICS

  • Five board-certified imaging experts participated in the human evaluation of MedSynVQA question quality and verifier behavior.Their backgrounds included oncologic CT/MR, abdominal and gastrointestinal CT/MR, brain CT/MR, diagnostic radiology, and gastrointestinal endoscopy.Source: G EXPERT EVALUATION STATISTICS · p. 19
  • Some questions were reviewed jointly when they spanned multiple organs or modalities.Source: G EXPERT EVALUATION STATISTICS · p. 19
  • The evaluation complemented downstream results and decontamination analysis by assessing intrinsic item quality and verifier behavior.Source: G EXPERT EVALUATION STATISTICS · p. 19

G.1 VERIFIER-ACCEPTED ITEMS

Experts assessed 15 verifier-accepted multiple-choice questions on four quality dimensions and overall usability. Fourteen were judged acceptable, while ratings also identified room to improve wording clarity.

G.1 VERIFIER-ACCEPTED ITEMS

  • In the aggregate review, 14 of 15 items (about 93%) were judged acceptable overall; one was rejected because poor contrast made the brain MR image non-evaluable, independent of the question text.Source: §G.1 · p. 19
  • Mean ratings were 3.35 for medical correctness, 3.13 for clarity and professional wording, 3.46 for image grounding, and 3.40 for option design.Each criterion used a 1-to-4 Likert scale, with 4 indicating excellent and 1 poor.Source: §G.1 · p. 19
  • Two items received a clarity-and-wording score of 1, and the authors identify prompt refinement as a direction for improving wording quality.The authors report that vocabulary-related verifier checks do not fully capture wording quality.Source: §G.1 · p. 19

G.2 VERIFIER-REJECTED ITEMS

The rejected-item audit found that clinicians agreed with the verifier on inspected medically consequential failures, while vocabulary checks showed systematic false negatives. The authors identify this vocabulary-related issue as a limitation of verification.

G.2 VERIFIER-REJECTED ITEMS

  • 20 rejected items were audited by selecting one essential rubric scored zero for each; the audited rubric frequencies are summarized in Table 9.Source: §G.2 · p. 19, Table 9 · p. 20
  • Among audited rejected items, failures included 5 non-self-contained stems, 4 vocabulary constraints, 1 diagnosis leak, 2 single-correct-option issues, 4 option-type inconsistencies, 2 clinical-validity issues, and 2 image–text inconsistencies.Source: §G.2 · p. 19
  • Inspected non-self-contained stems depended on external context, while option-type failures involved heterogeneous formats or an empty “none” option.The authors judged these rejections appropriate.Source: §G.2 · p. 20
  • Vocabulary checks sometimes falsely flagged caption- or context-supported sex, age, or clinical history as unsupported, making this rubric a weaker signal.The authors identify prompt and rubric refinement as a target because the check does not reliably capture professionalism or terminology use.Source: §G.2 · p. 20
  • Clinicians agreed with the verifier in all inspected medically heavy cases involving diagnosis leaks, multiple plausible keys, medical errors, or image-evidence mismatches.Source: §G.2 · p. 20

G.3 SUMMARY

  • The authors characterize the study as small, but report that accepted items were mostly judged medically sound and usable and that essential medical rejection rubrics aligned with clinicians.Tables 8–11 cover aggregate accepted-item ratings, audited rejection-rubric distribution, and per-item audits and ratings.Source: §G.3 · p. 20, Table 8 · p. 20, Table 9 · p. 20

H CASE STUDY OF SYNTHETIC DATA

The case study illustrates how verification rejects synthetic items when stems depend on context, options are ambiguous or inconsistent, or claims exceed what images support.

Essential Criteria: Stem Self-contained

  • The osteolysis question fails self-containment because its stem depends on external context and leaves “AM model” undefined.Without the context, the solver cannot infer that severe osteolysis means complete destruction of the ischial ramus.Source: H CASE STUDY OF SYNTHETIC DATA · p. 21

Essential Criteria: Vocabulary Constraint

  • The hip item introduces age and sex as image-visible facts, adding unsupported clinical information that can prime a diagnosis.The analysis notes that these demographics are not verifiable from the image and are not needed to answer.Source: H CASE STUDY OF SYNTHETIC DATA · p. 22

Essential Criteria: Diagnosis Leak

  • The breast item fails diagnosis-leak screening because its stem repeats caption diagnosis details that its answer choice also mirrors.The analysis identifies overlap from caption to stem and from stem to option.Source: H CASE STUDY OF SYNTHETIC DATA · pp. 22–23

Essential Criteria: Single Correct Option

  • The panoramic implant item leaves multiple options defensible because the image does not clearly show radiolucency and the choices lack objective thresholds.The verifier flags the proposed answer as unsupported by the image.Source: H CASE STUDY OF SYNTHETIC DATA · p. 23

Essential Criteria: Option Type Consistency

  • The scoliosis item mixes numeric measurements with descriptive phrases, violating the requirement for options of the same semantic type.The stem asks for a measurement, but only options A and B are numeric.Source: H CASE STUDY OF SYNTHETIC DATA · p. 24

Essential Criteria: Clinical Validity

  • The kyphosis item is clinically invalid because dynamic flexion–extension radiographs, not MRI, directly show angle change.The verifier rejects MRI as the best modality despite the case text reporting a flexibility percentage derived from an MRI angle.Source: H CASE STUDY OF SYNTHETIC DATA · pp. 24–25

I CASE STUDY OF MODEL OUTPUTS

Sample outputs show correct answers across neuroanatomy, ECG, radiology, and pathology, while accompanying analyses flag places where explanations lack image-specific detail.

I CASE STUDY OF MODEL OUTPUTS

  • The 7B examples correctly map ECG waves to their physiological events and identify lateral geniculate input to visual cortex, though one analysis calls for clearer wave-morphology links.The ECG answer gives the P/QRS/T sequence; the visual-cortex answer identifies the lateral geniculate nucleus, while the ECG analysis notes the explanation could connect wave morphology to function.Source: I CASE STUDY OF MODEL OUTPUTS · pp. 26, 32
  • The 3B examples select medial prefrontal cortex for the green-highlighted habenula connection and the anterior interosseous nerve for deep anterior cortical wire penetration.The habenula analysis notes that the explanation omits the figure legend’s distinction between functional association and direct anatomical connectivity; the olecranon analysis describes the nerve-risk directionality.Source: I CASE STUDY OF MODEL OUTPUTS · pp. 27, 29
  • The outputs correctly answer lung-presence and pulmonary-edema questions, but the analyses say the explanations could anchor findings to more precise anatomic or radiographic signs.For the lung-presence example, the analysis asks for clearer landmarks; for pulmonary edema, it lists interstitial and alveolar signs that would strengthen the rationale.Source: I CASE STUDY OF MODEL OUTPUTS · pp. 30, 37
  • Examples correctly identify small bowel obstruction and pneumatosis intestinalis, while their analyses call for more specific CT-feature descriptions and differential reasoning.The obstruction analysis mentions dilated loops, air–fluid levels, and a transition point; the pneumatosis analysis notes the need to describe intramural gas patterns more precisely.Source: I CASE STUDY OF MODEL OUTPUTS · pp. 31, 33
  • Pathology examples correctly recognize Mycobacterium avium infection and papillary thyroid carcinoma from described biopsy features, with analyses noting limited slide-specific detail.The MAC analysis flags “filamentous” as imprecise, while the thyroid-carcinoma analysis says the response relies more on textbook summary than slide-specific description.Source: I CASE STUDY OF MODEL OUTPUTS · pp. 28, 34
Loading 2510.25867v2…