PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometry-Supervised Visual Representation Learning for Multi-Phenotype Lesion Interpretation in Medical VLMs

Hao Wang, Qiwei Zeng, Shuchang Ye, Jinghao Lin, Yuezhe Yang, Yige Peng, Jinman Kim, Lei Bi

arXiv:2610.10238v1cs.CV

TL;DR

Medical VLMs struggle to interpret lesions requiring joint assessment of multiple phenotypes, while existing semantic supervision does not preserve anatomical and phenotype relationships in visual space. PureVision combines geometry-supervised representations with selective lesion-evidence aggregation, and improves lesion grounding and phenotype characterization across datasets and tasks.

  • Problem

    Medical VLMs struggle with multi-phenotype lesions, while semantic supervision lacks geometric constraints for anatomical hierarchies and phenotype relationships and sparse lesion patches can be diluted by normal patches.

    Source: §1 · pp. 1–2

  • Method

    PureVision combines PureEyes, which supervises visual encoders with ideal spatial distributions, and PureNeurons, which selectively aggregates lesion-specific anatomical and phenotypic evidence as soft semantic tokens.

    Source: §1 · p. 2

  • Results

    Across three datasets, PureVision improves lesion grounding and phenotype characterization in visual question answering and radiology report generation.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    Experiments across three datasets and multiple imaging modalities support geometry-aware visual representation learning for fine-grained medical image interpretation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Medical vision-language models (VLMs) have shown increasing potential for clinical image interpretation. However, these models still struggle to interpret multi-phenotype lesions whose diagnosis requires the joint assessment of multiple pathological phenotypes. Existing vision-language alignment methods produce visual representations that fail to preserve anatomical hierarchies and relationships among phenotypic subclasses. This stems from their reliance on semantic supervision, which lacks geometric constraints to preserve these relationships in the visual embedding space. Moreover, the sparsity of lesion-related anatomical and phenotypic representations makes it difficult for medical VLMs to capture important diagnostic evidence. To address these limitations, we propose \textbf{PureVision}, a geometry-supervised visual representation learning framework for multi-phenotype lesion interpretation in medical VLMs. It combines a geometry-supervised representation learning module, \textbf{PureEyes}, and an anatomy-guided evidence aggregation module, \textbf{PureNeurons}. PureEyes provides geometric supervision through ideal spatial distributions that encode anatomical hierarchies and phenotypic subclass relationships. PureNeurons projects visual representations into the learned latent space, using their positions to selectively aggregate lesion-specific anatomical and phenotypic evidence. Experiments on \textit{LIDC-IDRI}, \textit{CBIS-DDSM}, and \textit{3DReasonKnee} demonstrate that PureVision improves lesion grounding and phenotype characterization in visual question answering and radiology report generation. Code is available at: https://anonymous.4open.science/r/purevision-06C2.

1 INTRODUCTION

Medical VLMs struggle with lesions whose interpretation depends on multiple phenotypes and anatomical structures. PureVision addresses gaps in visual-space geometry and sparse lesion evidence through PureEyes and PureNeurons.

1 INTRODUCTION

  • Medical VLMs can diagnose single-phenotype diseases but struggle to ground lesions and describe phenotypes when diagnoses require joint anatomical and multi-phenotype assessment.The introduction attributes this gap to visual representations that do not sufficiently distinguish anatomical structures and diverse pathological phenotypes.Source: §1 · p. 1
  • Semantic supervision lacks geometric constraints for encoding anatomical proximity and hierarchy or continuity across phenotype severity levels.Sparse pathological patches can also be overwhelmed by normal patches, complicating lesion localization and phenotype discrimination.Source: §1 · p. 2
  • PureVision uses ideal spatial distributions to constrain visual embeddings and translate anatomical and phenotypic representations into semantic space.Its two modules are PureEyes for geometric supervision and PureNeurons for selective delivery of lesion-relevant evidence.Source: §1 · p. 2
  • PureEyes supervises visual encoders with ideal distributions that preserve anatomical hierarchies and relationships among phenotypic subclasses.PureNeurons selectively aggregates lesion-specific anatomical and phenotypic evidence and delivers it as soft semantic tokens to maintain alignment with semantic embeddings.Source: §1 · p. 2

2 RELATED WORK

Radiology VLM research spans medical model adaptation, multimodal and volumetric tasks, and methods for aligning image regions with clinical concepts. The cited fine-grained approaches leave visual-space geometry and sparse lesion evidence as open challenges.

2.1 MEDICAL VLMS IN RADIOLOGY

  • Medical VLMs use domain-specific multimodal training, with approaches spanning image–text alignment, radiology instruction tuning, tool integration, volumetric tasks, and anatomy-centric reasoning.Examples include LLaVA-Med, LLaVA-Rad, MedGemma, MedRAX, M3D-LaMed, Argus, and AOR.Source: §2.1 · pp. 2–3
  • Figure 1 contrasts PureEyes, which structures the visual latent space, with PureNeurons, which captures and delivers lesion-relevant evidence.Source: Figure 1 · p. 3

2.2 FINE-GRAINED ALIGNMENT IN MEDICAL VLMS

  • Fine-grained alignment methods use region supervision, anatomy-aware features, and concept guidance, but may not impose anatomical hierarchies or continuity among ordered phenotypes.Lesion evidence concentrated in a few patches can also be diluted by normal-tissue representations.Source: §2.2 · p. 3

3 METHODOLOGY

PureVision structures anatomical and phenotypic visual spaces with ideal geometries, then uses those spaces to select and deliver lesion-specific evidence to a language model. Its method combines staged geometry supervision with aligned-space lesion grounding and weighted semantic fusion.

3.1 OVERVIEW

  • PureVision combines geometry-shaped anatomical and phenotypic spaces with selective fusion of lesion-relevant evidence for downstream reasoning.Visual encoders are trained first and then frozen during semantic alignment and inference.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2 PUREEYES: BUILDING IDEAL-GEOMETRY SUPERVISED VISUAL REPRESENTATION SPACE

  • PureEyes learns complementary anatomical and phenotypic representations through lesion-centric phenotype, context-aware phenotype, and anatomical geometry training stages.The stages first organize isolated lesion patches, then constrain lesion-overlapping patches in full-image context, and finally organize annotated anatomical regions hierarchically.Source: §3.2 · pp. 4–5
  • Phenotype geometry matches visual distances to normalized measurement differences for continuous attributes, while compactness, margins, and ranking constraints structure discrete and ordered categories.The objectives support continuous variation as well as categorical separation and relative ordering among grades.Source: §3.2 · p. 4

3.3 PURENEURONS: GROUNDING SPARSE LESION POSITION AND ALIGN PHENOTYPES TO SEMANTIC REPRESENTATION

  • PureNeurons aligns anatomical and phenotypic patch representations in a shared semantic space, then identifies lesion-related patches by comparing their strongest lesion-label and other-anatomy matches.A larger ρ_i indicates a patch is more likely to belong to a lesion region than normal anatomy.Source: §3.3 · p. 5
  • PureNeurons weights candidate subtype embeddings rather than making a hard subtype prediction, then inserts anatomical and phenotype soft tokens alongside lesion location before the task instruction.The visual tokens remain available so the frozen decoder can integrate lesion-specific evidence with full-image context.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate PureVision on lesion localization and phenotype characterization across three datasets and two downstream tasks. Results report improvements over pretrained medical VLMs and alignment methods, alongside analyses of representation structure and lesion-evidence selection.

4.1 DATASET AND EVALUATION

  • Evaluation covers LIDC-IDRI, CBIS-DDSM, and 3DReasonKnee with 2,600 VQA pairs and 583 RRG instances; localization uses a 4×4 grid and Grounding Accuracy and Phenotype Characterization Accuracy.Source: §4.1 · pp. 5–6

4.2 EXPERIMENTAL SETUP

  • The evaluation compares pretrained medical VLMs and alignment methods on VQA and RRG, then examines PureEyes representations, PureNeurons patch selection, ablations, and qualitative cases.Source: §4.2 · pp. 6–7

4.3 COMPARISON WITH EXISTING METHODS AND ALIGNMENT METHODS

  • PureVision improved VQA grounding and phenotype characterization by averages of 14.19 pp and 18.94 pp, respectively, and achieved the highest accuracies against alignment methods across three datasets; RRG results also favored PureVision.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · p. 7

4.4 PUREEYES SUPERVISED VISION ENCODER PERFORMANCE

  • Figure 4 visualizes pathological phenotype representations on LIDC-IDRI, comparing the backbone with PureEyes.Source: Figure 4 · p. 8
  • On LIDC-IDRI, PureEyes yields compact, separated anatomical clusters and raises left- and right-lung nodule similarities from 0.363/0.357 to 0.531/0.532.Source: Figure 3 · p. 7, §4.4 · p. 8

4.5 LESION EXTRACTION EFFICIENCY AND INFORMATION DELIVERY ABLATION IN PURENEURONS

  • Lesion patch recall rises from 60.11% at K = 4 to 68.84% at K = 16, while redundant patch rate increases from 45.23% to 81.23%.Recall gains become limited beyond K = 8, as larger selection budgets mainly add redundant visual information.Source: §4.5 · pp. 8–9, Table 3 · p. 8
  • Grounding and phenotype accuracies rise from 11.50/10.50 to 32.50/36.57 without PureNeurons, 40.00/38.43 with lesion grounding, and 45.50/49.93 with lesion grounding and soft semantic fusion.Source: Table 3 · p. 8, §4.5 · p. 9

4.6 CASE STUDY

  • The LIDC-IDRI case study shows PureEyes structuring visual representations, PureNeurons selecting nodule-overlapping lesion patches, and the decoder generating a report with the correct location and multiple phenotype descriptions.Source: §4.6 · p. 9, Figure 6 · p. 9

5 CONCLUSION

PureVision combines geometry-structured representations with lesion-specific evidence aggregation, and experiments across three datasets and imaging modalities show improvements in lesion grounding and phenotype characterization.

5 CONCLUSION

  • Across three datasets and multiple imaging modalities, PureVision improves lesion grounding and phenotype characterization.The conclusion attributes these results to PureEyes structuring anatomical and phenotypic representations and PureNeurons capturing and delivering sparse lesion-specific evidence.Source: §5 · p. 10
  • GPT6-Astra parsed generated reports into lesion locations and phenotype labels, while deterministic metrics—not the model—assigned performance scores.The authors also report using generative AI to draft portions of the manuscript and state that they reviewed and revised the assisted content.Source: AI USE STATEMENT · p. 10
  • The authors report no additional ethical concerns, no new human-subject recruitment or identifiable private-information collection, and use of datasets under their original access and usage requirements.They provide an anonymous code repository to support reproduction of the methods and experiments.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 BENCHMARK CONSTRUCTION

The benchmark uses patient-disjoint evaluation splits across LIDC-IDRI, CBIS-DDSM, and 3DReasonKnee, with VQA instances built around lesion grounding and dataset-specific phenotypes.

A.1 BENCHMARK CONSTRUCTION

  • For all three datasets, evaluation patients are excluded from representation learning and semantic alignment, and images have anatomical masks, a lesion mask, and phenotype labels.Grounding questions use a 4×4 image grid, with the reference location determined by the lesion mask.Source: §A.1 · p. 13
  • LIDC-IDRI contributes 200 grounding questions and 1,600 phenotype VQA pairs across seven nodule attributes, with sampling stratified by answer category.The selected nodule masks lie entirely within one grid cell; each question has one correct answer and three distractors.Source: §A.1 · p. 13
  • CBIS-DDSM contributes 200 grounding questions and 600 phenotype questions across mass shape, mass margins, and calcification distribution.Each VQA question has one correct answer and three distractors.Source: §A.1 · p. 13
  • 3DReasonKnee contributes 100 grounding questions and 100 questions about medial meniscus medial extrusion grades.Its 200 VQA pairs each have one correct answer and three distractors.Source: §A.1 · p. 13

A.2 EVALUATION METRICS

The evaluation measures lesion grounding and phenotype accuracy for VQA and radiology report generation, using dimension-balanced phenotype scoring and deterministic report parsing.

A.2 EVALUATION METRICS

  • VQA Grounding Accuracy is the proportion of correctly identified grid cells, while Phenotypes Accuracy averages accuracy across phenotype dimensions.Missing or invalid answers count as incorrect; the number of phenotype dimensions is 7 for LIDC-IDRI, 3 for CBIS-DDSM, and 1 for 3DReasonKnee.Source: §A.2 · p. 13
  • For RRG, GPT6-Astra maps generated reports to predefined location and phenotype labels, which are evaluated deterministically against reference annotations.The evaluation uses a fixed prompt and deterministic decoding; missing or invalid outputs count as incorrect.Source: §A.2 · p. 13
  • RRG Phenotypes Accuracy averages balanced accuracy across phenotype dimensions, weighting classes equally within each dimension and dimensions equally overall.The number of dimensions is 7 for LIDC-IDRI, 3 for CBIS-DDSM, and 1 for 3DReasonKnee.Source: §A.2 · p. 14

B.1 DETAILED PHENOTYPES ACCURACY

PureVision improves phenotype-specific VQA accuracy across the reported LIDC-IDRI and CBIS-DDSM attributes, while visualizations show organized pulmonary representations across anatomical and phenotype categories.

B.1 DETAILED PHENOTYPES ACCURACY

  • On LIDC-IDRI with MedGemma1.5, accuracy rose for calcification from 8.00 to 83.50, lobulation from 10.00 to 60.50, and spiculation from 4.00 to 70.00.PureVision achieved the highest accuracy among compared alignment methods in five of seven phenotype dimensions.Source: §B.1 · p. 14
  • The detailed phenotype-accuracy tables cover LIDC-IDRI and CBIS-DDSM; no additional detailed table is provided for 3DReasonKnee, which has only one phenotype dimension.That dimension is medial meniscus medial extrusion.Source: §B.1 · p. 14
  • On CBIS-DDSM with MedGemma1.5, mass shape accuracy rose from 10.50 to 45.00, mass margins from 12.50 to 43.50, and calcification distribution from 7.00 to 57.50.PureVision achieved the highest accuracy among compared alignment methods across all three attributes.Source: §B.1 · p. 14, Table 5 · p. 15

B.2 FINE-GRAINED DISTRIBUTION OF LATENT VISION REPRESENTATION OF PULMONARY PATCH REPRESENTATIONS

  • In both lungs, normal patches cluster compactly while lesion patches occupy different regions; vessel patches partly overlap lesions, but many form distinct vessel-dominated subclusters.The authors note that lesion–vessel overlap is expected because nodules and cross-sectional vessels can look similar within a CT slice.Source: Figure 8 · p. 15, §B.2 · p. 15, Figure 9 · p. 16

B.3 ADDITIONAL PHENOTYPES DISTRIBUTION

  • For sphericity, margin, and lobulation, PureEyes representations group similar phenotype scores and separate more distinct manifestations while preserving gradual transitions between adjacent levels.The original MedGemma1.5 vision encoder shows substantial intermixing of samples with different phenotype scores.Source: §B.3 · p. 16

C.1 ARCHITECTURE AND TRAINING STAGES

The appendix specifies separate visual encoders and staged optimization, then details geometry-specific objectives and lesion-evidence selection and fusion.

C.1 ARCHITECTURE AND TRAINING STAGES; C.2 OPTIMIZATION

  • PureEyes trains separate anatomical and phenotypic visual encoders before freezing them for shared semantic alignment; optimization uses distinct learning rates and stages.The encoders use full visual towers, and training masks identify supervised patches but are not supplied at inference.Source: §C.1 · p. 16, §C.2 · p. 16

C.3 DISTANCE DEFINITION AND ROBUST PENALTY; C.4 CONTINUOUS PHENOTYPE GEOMETRY

  • Normalized embeddings use a bounded Euclidean distance that preserves pretrained angular geometry, while continuous-size targets scale with normalized physical measurement differences.The continuous objective keeps lesions with similar measurements nearby and makes larger measurement differences correspond to proportionally larger visual distances.Source: §C.3 · p. 17, §C.4 · pp. 17–18

C.5 ORDINAL PHENOTYPE GEOMETRY

  • Ordinal phenotype geometry uses ordered-label separation for target distances and adds a ranking constraint requiring farther categories to have larger mean distances.This applies to the listed ordinal attributes, including meniscus extrusion grade.Source: §C.5 · p. 18

C.6 CATEGORICAL PHENOTYPE GEOMETRY

  • Categorical phenotypes assign zero target distance within a category and 1.5 between categories, with category-imbalance compensation in the objective.Calcification in LIDC-IDRI and specified CBIS-DDSM attributes are treated as categorical.Source: §C.6 · p. 18

C.7 ANATOMICAL GEOMETRY OBJECTIVES

  • The anatomical objective combines regular-simplex structure separation, parent-region geometry, normal-versus-lesion separation, and distillation on non-lesion patches.Distillation minimizes cosine distance to a frozen pretrained teacher for non-lesion patches.Source: §C.7 · p. 19

C.8 CONFOUNDING-TISSUE SEPARATION

  • Confounding-tissue separation reduces similarity relative to a pretrained teacher by a margin, with supervised contrastive regularization; pulmonary vessels are the LIDC-IDRI confounder.The relative margin avoids forcing unrelated anatomical representations arbitrarily far apart.Source: §C.8 · p. 19

C.9 SEMANTIC ALIGNMENT AND LESION SELECTION

  • After PureEyes training, frozen encoders are aligned to language-compatible text targets, while PureNeurons selects the lesion-consistent 5 × 5 neighborhood and its top eight patches.Anatomical and phenotypic representations are matched using shared spatial patch indices.Source: §C.9 · p. 19

C.10 SOFT SEMANTIC FUSION; C.11 COMPUTATIONAL RESOURCES

  • Soft fusion weights selected-patch similarities, standardizes them within phenotype groups, converts them to subtype probabilities, and inserts fused tokens before task instructions.Experiments used RTX PRO 6000 GPUs and BF16 mixed precision where applicable.Source: §C.10 · pp. 19–20, §C.11 · p. 20

Related papers

Loading 2610.10238v1…