PaperlayerView exact revision on arXiv

Source-linked AI summary

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

Guanghao Zhu, Zeyu Liu, Zhitian Hou, Pengkai Wang, Zhijie Sang, Shuo Cai, Yang Yu, Yuanyi Wang, Yanggan Gu, Congkai Xie, Jianmin Wu, Hongxia Yang

arXiv:2609.34798v2cs.CLcs.CV

TL;DR

Medical multimodal training still faces data-design challenges across continued pretraining and post-training, including heterogeneous evidence and short-answer supervision. InfiMed2 uses stage-aware corpus construction and stability-aware response regeneration, and its models achieve strong mean accuracy across five medical multimodal benchmarks.

  • Problem

    Medical sources vary in structure, granularity, and information density, while short VQA targets offer limited supervision for informative, answer-consistent explanations.

    Source: §1 · pp. 1–2

  • Method

    InfiMed2 combines source-specific processing of a 55.68B-token corpus and stage-aware CPT with stability-guided response regeneration, correctness-constrained selection, and RLVR for the 4B model.

    Source: §1 · p. 2

  • Results

    Across five medical multimodal benchmarks, InfiMed2-4B reaches 66.73% mean accuracy and surpasses the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, highest among evaluated open-weight models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support coordinated data design across continued pretraining and post-training as a practical route to stronger medical multimodal capability.

    Source: §6 · p. 10

    InfiMed2 is intended for research and is not approved for unverified clinical use.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Recent medical multimodal models have benefited from larger corpora, broader modality coverage, and stronger reasoning-oriented training, yet effective data design across continued pretraining (CPT) and post-training remains challenging. Medical sources vary substantially in structure, granularity, and information density, and their utility shifts as training progresses from broad knowledge acquisition to late-stage consolidation. Meanwhile, post-training is often dominated by short-form visual question answering, providing limited supervision for informative and answer-consistent explanations. We introduce InfiMed2, a family of 4B and 27B generalist medical multimodal foundation models built around stage-aware data design. We curate a 55.68B-token corpus that combines broad clinical knowledge with context-rich biomedical visual evidence through source-specific processing. Our CPT pipeline first adapts the vision encoder, then builds broad medical knowledge, and finally transitions to an evidence-focused data mixture during learning-rate decay. For supervised fine-tuning (SFT), we regenerate visual question-answering responses using answer stability, answer-masked reconstruction, and correctness-constrained selection to produce more informative and answer-consistent supervision. The 4B model is further optimized with reinforcement learning with verifiable rewards (RLVR). Across five medical multimodal benchmarks, InfiMed2-4B achieves 66.73% mean accuracy after RLVR, surpassing the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among the evaluated open-weight models.

1 INTRODUCTION

InfiMed2 addresses data-design challenges in continued pretraining and post-training by coordinating source-specific curation, training stages, and stability-aware response supervision. Across five medical multimodal benchmarks, its 4B model surpasses a larger baseline and its 27B model leads evaluated open-weight models in mean accuracy.

1 INTRODUCTION

  • CPT must reconcile medical sources with different structures, granularity, and information density, while their value shifts between broad knowledge acquisition and late-stage consolidation.Early updates benefit from diverse coverage, whereas the low-learning-rate regime is more sensitive to evidence quality and task relevance.Source: §1 · pp. 1–2
  • Short medical VQA targets provide limited supervision for explanations, and teacher-generated responses may be weakly grounded, inconsistent with verified answers, or unreliable across examples.Post-training data therefore needs to improve explanatory content while accounting for stability and correctness.Source: §1 · p. 2
  • InfiMed2's CPT uses a 55.68B-token corpus with source-specific processing and separates vision-encoder adaptation, broad medical knowledge acquisition, and evidence-focused learning-rate decay.The final decay stage uses a compact mixture enriched with high-quality medical evidence.Source: §1 · p. 2
  • 66.73% mean accuracy for InfiMed2-4B surpasses the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among evaluated open-weight models.Both results are reported under a common evaluation protocol on five medical multimodal benchmarks.Source: §1 · p. 2
  • Stability-aware regeneration guides response reconstruction, correctness-constrained selection, escalation to a stronger model, and filtering to produce informative supervision consistent with verified answers.The selected 4B model subsequently uses reinforcement learning with verifiable rewards.Source: §1 · p. 2

2 RELATED WORK

Prior medical multimodal work has broadened models, data, and reasoning supervision, while source-aware evidence curation and verification of synthetic explanations remain less systematically studied. InfiMed2 addresses these gaps across continued pretraining and post-training.

2.1 MEDICAL MULTIMODAL FOUNDATION MODELS

  • Recent generalist medical multimodal systems broaden data, modalities, and reasoning supervision, but source-aware evidence curation and synthetic-explanation verification remain less systematically studied.InfiMed2 addresses these data-design problems across continued pretraining and post-training.Source: §2.1 · pp. 2–3

2.2 MULTIMODAL CONTINUED PRETRAINING AND MEDICAL DATA CURATION

  • Biomedical figure collections can reduce literature evidence to isolated figure–caption pairs and remain vulnerable to extraction noise, while source diversity and stage-dependent data value are underexplored.PMC-InterCPT reconstructs associations among figures, captions, and related context; InfiMed2 combines complementary medical sources through source-aware processing.Source: §2.2 · p. 3

2.3 MEDICAL POST-TRAINING AND RELIABLE SYNTHETIC SUPERVISION

  • Generated medical explanations may be weakly grounded or inconsistent with verified answers, motivating regeneration that combines reliability signals with reference-answer constraints.Self-consistency and predictive uncertainty are identified as useful reliability signals.Source: §2.3 · p. 3

3 DATA CURATION

InfiMed2 curates data around complementary medical sources and training stage, then filters and regenerates post-training examples to improve evidence quality, answer consistency, and visual balance.

3.1 CPT DATA CURATION

  • The 55.68B-token CPT corpus combines medical e-books, biomedical literature, medical web documents, and general multimodal replay using source-specific processing.Its components are 29.78B, 9.63B, 11.27B, and 5B tokens, respectively.Source: §3.1 · p. 3

3.2 POST-TRAINING DATA CURATION

  • Post-training curation sequentially filters invalid samples, regenerates informative VQA responses while preserving answer correctness, and forms a mixture by visual content.The curation addresses semantic inconsistencies, short-form answers, and imbalanced visual content.Source: §3.2 · p. 5

3.2.1 FINE-GRAINED QUALITY FILTERING

  • A fine-grained annotation scheme assesses image–question alignment, information completeness, and logical validity; a trained evaluator removes samples with critical defects.The evaluator assigns three levels, with Score 0 marking a critical defect, Score 1 a limited non-fatal issue, and Score 2 a valid sample.Source: §3.2.1 · p. 5

3.2.2 STABILITY-AWARE RESPONSE REGENERATION

  • 94.10% of 704,046 answer-masked reconstructions recover the reference answer, and the procedure exports 106,183 final examples.Stable samples undergo answer-masked reconstruction and correctness-constrained selection; intermediate-stability samples receive escalated regeneration, while samples with very low stability are discarded.Source: §3.2.2 · pp. 5–6

3.2.3 VISUAL-MODALITY-AWARE MIXTURE CONSTRUCTION

  • Category-level sampling across 27 visual categories reduces domination by frequent modalities while preserving clinically relevant long-tail content.The resulting SFT mixture contains 296,725 examples and 241.85M tokens, including 106,183 stability-aware examples totaling 108.32M tokens.Source: §3.2.3 · p. 6

4 MODEL TRAINING

InfiMed2 trains through staged continued pretraining followed by supervised fine-tuning, with an additional RLVR stage for the 4B model.

4 MODEL TRAINING

  • The 4B and 27B runs start from Qwen3.5-4B-Base and the instruction-tuned Qwen3.5-27B checkpoint, respectively, and both proceed through CPT and SFT.The 30B-CPT 4B variant additionally undergoes RLVR after SFT.Source: §4 · pp. 6–7

4.1 CONTINUED PRETRAINING

  • CPT proceeds through vision-encoder adaptation, broad warmup–stable training, and learning-rate decay on a high-quality 2.50B-token mixture.The stages use a 1B-token image–caption subset and a 55.68B-token mixture before the final decay mixture.Source: §4 · p. 7, §4.1 · p. 7

4.2 SUPERVISED FINE-TUNING

  • SFT trains all model components for five epochs on a 241.85M-token mixture combining general multimodal instructions, medical VQA, reasoning supervision, and stability-aware responses.The run starts from the post-decay checkpoint.Source: §4.2 · p. 7

4.3 REINFORCEMENT LEARNING WITH VERIFIABLE REWARDS

  • RLVR uses answer accuracy and format compliance rewards, prioritizing prompts with 3–5 correct rollouts while downsampling near-unsolved and saturated examples.A smaller easy subset is retained to reduce forgetting after SFT.Source: §4.3 · p. 7

5 EXPERIMENTS

The experiments evaluate InfiMed2 against general-purpose and medical models under a common five-benchmark protocol, then test CPT checkpoint choice, RLVR, and response-target selection. Results show strong average performance alongside trade-offs across training stages and target formats.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers five benchmarks and reports each accuracy plus their unweighted mean, with models assessed on the same questions, prompts, and scoring procedure.The compared models include general-purpose and medical multimodal baselines, alongside SFT-only controls.Source: §5.1 · pp. 7–8

5.2 MAIN RESULTS

  • 66.73% mean accuracy lets InfiMed2-4B outperform the larger Qwen3.5-9B by 2.02 points, while InfiMed2-27B reaches 73.72%, the highest among evaluated open-weight models.The comparison uses the paper’s common evaluation protocol across five medical multimodal benchmarks.Source: §5.2 · p. 8, Table 1 · p. 8

5.3.1 CPT CHECKPOINT SELECTION AND RLVR

  • Under matched SFT, average performance peaks at 30B warmup–stable CPT tokens for 4B and 24B for 27B; RLVR improves all five benchmarks for the selected 4B checkpoint.The reported gains are largest on MedXpertQA-MM and PMC-VQA-clean; extending the 27B run improves some tasks but reduces others.Source: §5.3.1 · pp. 8–9, Table 2 · p. 8, Figure 2 · p. 9

5.3.2 STABILITY-AWARE RESPONSE REGENERATION

  • With the same 4B checkpoint and 20k PMC-VQA examples, stability-aware response selection performs best on average, while answer-only targets remain stronger on direct-answer VQA.Generated explanations improve reasoning-oriented benchmarks, and stability-aware selection exceeds random response selection.Source: §5.3.2 · p. 9, Table 3 · p. 9

5.4 DISCUSSION

  • Neither longer CPT nor more verbose supervision is uniformly beneficial; checkpoint choice should follow downstream capability profiles, and post-training should retain concise targets or adapt response format to the task.The discussion also identifies open-ended clinical communication and expert assessment of explanation quality as future evaluation areas.Source: §5.4 · p. 9

6 CONCLUSION

InfiMed2 combines coordinated data design across continued pretraining and post-training with source- and stage-specific curation. The 4B and 27B models achieve strong results among evaluated open-weight models, and the ablations support matching data construction to each training stage.

6 CONCLUSION

  • InfiMed2 coordinates source-specific curation and stage-specific mixtures across continued pretraining and post-training to build 4B and 27B generalist medical multimodal models.The conclusion describes the approach as a practical route to stronger medical multimodal capability, supported by the ablations.Source: §6 · p. 10

AI USE STATEMENT

The statement describes AI assistance in literature search, code, drafting, formatting, and post-training data construction, with author review and responsibility. It also states research-use and release boundaries.

AI USE STATEMENT

  • Generative AI assisted literature search, code development, early-draft polishing, formatting, and post-training data construction; authors reviewed and verified outputs and take responsibility.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses no new human-participant data or prospective clinical intervention; heterogeneous corpora may still contain errors, biases, or sensitive content, and the model is not approved for unverified clinical use.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Planned release covers trained models and selected data or metadata only where licenses and privacy conditions permit redistribution; restricted resources will instead have provenance, preprocessing descriptions, and reconstruction scripts.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL DATA CURATION DETAILS

The appendix details source-specific curation and sampling for medical books, biomedical literature, web data, and stage-specific CPT mixtures. These procedures rebalance medical content and prepare evidence-rich data for training.

A.1 MEDICAL E-BOOK PROCESSING AND SAMPLING

  • After rebalancing, the sampled book corpus contains 60,433 books and 22.41B text tokens, or 29.78B tokens including visual tokens.Clinical disciplines rise from approximately 55% to 75% of text tokens, while basic medicine falls from 21.3% to 9.2%.Source: §A.1 · p. 13

A.2 BIOMEDICAL SCIENTIFIC LITERATURE

  • PMC-InterCPT links figures to explicitly referencing context, repairs incoherent records, filters for medical relevance and text quality, and excludes PMC-VQA test examples.Its final token budget allocates 45% to biomedical visual evidence, 30% to quantitative and tabular evidence, 20% to mechanism and structure evidence, and 5% to auxiliary evidence.Source: §A.2 · p. 13

A.3 MEDICAL WEB CURATION DIAGNOSTICS

  • Web curation yields an 11.27B-token training corpus after retrieval, deduplication, quality filtering, image downloads, and format validation.The component diagnostics assess individual filters rather than the end-to-end quality of the resulting corpus.Source: §A.3 · pp. 13–14

A.4 STAGE-SPECIFIC CPT MIXTURES

  • Vision-encoder adaptation uses a 1B-token PMC-InterCPT subset, while learning-rate decay uses a compact mixture emphasizing curated medical sources and retaining controlled general replay.The decay mixture totals 2.50B tokens.Source: §A.4 · p. 14, Table 4 · p. 14

B.1 POST-TRAINING QUALITY ASSESSMENT

This section describes sample-quality scoring, the supervised fine-tuning mixture, and stability-aware response regeneration. The regeneration pipeline filters candidate answers and retains long-form responses whose masked reconstructions preserve correctness.

B.1 POST-TRAINING QUALITY ASSESSMENT

  • The quality evaluator uses three levels: Score 0 marks critical defects, Score 1 limited but non-fatal issues, and Score 2 valid samples without listed defects.The assessment protocol is provided in Table 5.Source: §B.1 · p. 14, Table 5 · p. 14

B.2 SUPERVISED FINE-TUNING MIXTURE COMPOSITION

  • The final SFT mixture contains 296,725 effective training examples and 241.848M tokens.Vision and text contribute 120.459M and 121.389M tokens, respectively.Source: §B.2 · p. 15, Table 6 · p. 15

B.3 STABILITY-AWARE ROUTING AND RETENTION

  • Of 176,948 questions, 106,183 examples enter SFT after candidate generation, correctness checks, selection, filtering, and cleaning.The selected supervision contributes 108.316M training tokens, including 45.382M visual tokens and 62.933M text tokens.Source: §B.3 · p. 15, Table 7 · p. 15
  • Answer-masked reconstructions average 11.62 tokens versus 418.31 tokens for correct candidates, and 662,503 of 704,046 reconstructed answers remain correct (94.10%).The reconstruction checks whether a long response preserves the reference answer without replacing the informative long-form SFT target.Source: §B.3 · p. 15

C TRAINING AND IMPLEMENTATION DETAILS

The appendix specifies shared optimization choices, RLVR settings, and the sources used for medical reasoning. It emphasizes verifiable answer accuracy while retaining response-format compliance.

C TRAINING AND IMPLEMENTATION DETAILS

C.1.1 OPTIMIZATION SETTINGS

  • RLVR combines format compliance and answer accuracy rewards at a 1:9 weight ratio, placing most optimization signal on verifiable correctness.Training runs for three epochs with learning rate 1×10−6, global batch size 128, and eight rollouts per prompt.Source: §C.1.1 · p. 16, Table 9 · p. 16

C.1.2 DATA COMPOSITION

  • The RLVR resources combine image-grounded clinical questions with text-only medical reasoning, with train and test partitions reported for each source.Table 10 summarizes the resource composition.Source: §C.1.2 · p. 16, Table 10 · p. 16

D.1 BENCHMARK OVERVIEW

The appendix describes five medical multimodal benchmarks, a common inference and scoring protocol, and qualitative comparisons of model responses. The case studies highlight differences in evidence integration, statistical reasoning, explanation consistency, and diagram grounding.

D.1 BENCHMARK OVERVIEW

  • MMMU-Medical-test, MMMU-Pro-Medical-10, MedXpertQA-MM, PMC-VQA-clean, and OmniMedVQA cover expert medical reasoning, visual questions, and diverse medical images.The benchmarks include Health and Medicine subsets, multi-step multimodal questions, and broad coverage across modalities and anatomical regions.Source: §D.1 · pp. 16–17

D.2 INFERENCE AND PROMPTING PROTOCOL

  • All models use the same benchmark examples, prompts, generation limits, and answer extraction, with exact-match accuracy computed from the tagged option letter.OmniMedVQA uses a direct-answer prompt; the other named benchmarks use the reasoning prompt.Source: §D.2 · p. 17

D.3 QUALITATIVE CASE STUDIES

  • The cases show RLVR responses integrating clinical and visual evidence, selecting risk ratio from observed attack rates, and aligning explanation mechanisms with answers; anatomical label grounding remains variant-dependent.In one diagram example, InfiMed2-27B maps the atrioventricular node to label C, while InfiMed2-4B describes the anatomy correctly but maps it to the wrong marker.Source: §D.3 · pp. 18–20

Related papers

Loading 2609.34798v2…