PaperlayerView exact revision on arXiv

Source-linked AI summary

$\Delta$Representation: Geometry Supervised Representation Learning of Phenotypes via Counterfactual Reasoning for Medical VLMs

Hao Wang, Qiwei Zeng, Jinghao Lin, Shuchang Ye, Yuezhe Yang, Yige Peng, Haoyuan Che, Jinman Kim, Lei Bi

arXiv:2610.10286v1cs.CV

TL;DR

Medical VLMs lack explicit representations of pathological changes relative to normal anatomy, a gap semantic alignment does not address. ∆Representation builds geometry-supervised anatomical references and learns counterfactual phenotype increments; experiments report consistent improvements in lesion grounding and phenotype characterization across datasets and backbones.

  • Problem

    Paired image-text learning does not explicitly represent pathological phenotype changes relative to corresponding normal anatomy, and semantic supervision provides limited separation of visually similar phenotypes.

    Source: §1 · p. 2

  • Method

    ∆Representation combines BaseAnatomy, which structures normal anatomy with geometric supervision, and ∆Phenotype, which learns phenotype-specific increments against counterfactual normal references.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    VQA averages improved by 31.20% grounding accuracy and 33.51% phenotype accuracy, while RRG averages improved by 20.80% and 28.57%, respectively, across backbone–dataset combinations.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    Across two datasets and multiple medical VLM backbones, anatomy-referenced incremental representation learning consistently improves lesion grounding and phenotype characterization.

    Source: §5 · p. 10

Abstract

from arXiv · show

Medical vision-language models (VLMs) have shown increasing potential for radiological image interpretation. Medical VLMs encode radiological images into visual representations that capture both anatomical and phenotypic information for diagnosis. Existing approaches improve pathological phenotype representations through semantic-guided representation alignment. However, pathological phenotypes arise as lesion-specific visual changes superimposed on underlying normal anatomy. Such semantic alignment approaches fail to model the phenotype-specific increment relative to the corresponding normal anatomical representation. To address this gap, we propose \textbf{$\Delta$Representation}, a visual phenotype representation learning framework based on counterfactual reasoning for medical VLMs. It comprises \textbf{BaseAnatomy}, a geometry-supervised representation learning module, and \textbf{$\Delta$Phenotype}, a counterfactual incremental representation learning module. BaseAnatomy provides fine-grained geometric supervision through spatial relationships across and within anatomical structures. $\Delta$Phenotype computes the representation increment between lesion representations and their corresponding normal anatomical representations, and supervises increments associated with the same phenotype to cluster in the representation space. Experiments on \textit{ReXGroundingCT} and \textit{LIDC-IDRI} demonstrate that $\Delta$Representation effectively structures pathological phenotype representations and improves lesion grounding and phenotype characterization accuracy in medical VLMs. Code is available at https://anonymous.4open.science/r/deltarep-CF6D.

1 INTRODUCTION

Medical VLMs align visual representations with clinical semantics, but paired image-text learning does not explicitly model pathological changes relative to normal anatomy. ∆Representation combines geometry-supervised anatomical references with counterfactual phenotype learning to improve grounding and phenotype characterization.

1 INTRODUCTION

  • Medical VLMs align radiological images with clinical texts, but their vision encoders lack explicit modeling of anatomical structures and pathological visual phenotypes.Potentially inaccurate visual representations may consequently be aligned with semantics and used as diagnostic evidence.Source: §1 · p. 1
  • Paired image-text learning does not explicitly represent phenotypic changes relative to normal anatomy, and semantic supervision provides limited separation of visually similar phenotypes.The paper defines a phenotype representation by comparing a pathological patch with its normal counterpart.Source: §1 · p. 2
  • ∆Representation consistently improves lesion grounding and phenotype characterization across medical VLMs, with learned increments providing more discriminative representations for different phenotypes.The framework combines anatomical geometry supervision with counterfactual phenotype learning.Source: §1 · p. 2
  • BaseAnatomy organizes normal anatomical representations using ideal spatial distributions that preserve inter-structure geometry and intra-structure continuity.These representations provide stable anatomical references for subsequent phenotype learning.Source: §1 · p. 2
  • ∆Phenotype learns phenotype-specific increments through counterfactual reasoning and progressively refines phenotype clusters as additional samples become available.The refinement improves intra-phenotype consistency and inter-phenotype separation.Source: §1 · p. 2

2 RELATED WORK

Radiology VLMs support image interpretation across medical and volumetric settings, while fine-grained alignment methods connect image regions or concepts to clinical descriptions. The related work identifies geometry preservation and anatomy-relative phenotype increments as remaining gaps.

2.1 MEDICAL VLMS IN RADIOLOGY

  • Radiology VLMs span medical visual conversation, report generation, image-encoder adaptation, tool use, volumetric tasks, and anatomy-centric interpretation.Examples include M3D-LaMed for multiple 3D tasks and AOR for anatomy-centric region-level reasoning.Source: §2.1 · p. 3

2.2 FINE-GRAINED ALIGNMENT IN MEDICAL VLMS

  • Fine-grained alignment methods use regional features, explicit supervision, or concept-specific visual features to connect images with clinical descriptions.Examples include region–language correspondence and anatomical or pathological concept banks.Source: §2.2 · p. 3
  • Text-derived anatomical semantics do not explicitly preserve geometric relationships across structures or spatial continuity within structures, and phenotype-specific changes relative to normal anatomy remain unmodeled.The paper identifies these limitations as constraints on stable anatomical encoding and accurate pathological phenotype representation.Source: §2.2 · p. 3

3 METHODOLOGY

∆Representation first establishes a geometry-structured normal anatomical space, then learns pathological phenotype increments relative to counterfactual normal references. Its training combines geometric constraints, neighborhood-based reference estimation, phenotype clustering, and continual refinement.

3.1 OVERVIEW

  • ∆Representation trains BaseAnatomy first and freezes its anatomical encoder to provide stable normal references during ∆Phenotype learning.The two modules structure normal anatomy and learn phenotype-specific increments relative to that anatomy.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 BASEANATOMY: BUILDING GEOMETRY-SUPERVISED NORMAL ANATOMICAL REPRESENTATION SPACE

  • Anatomical annotations and lesion masks provide training supervision, while lesion patches are excluded from normal anatomical supervision.The resulting normal representations serve as references for phenotype learning.Source: §3.2 · pp. 4–5
  • BaseAnatomy uses annotation-defined ideal spatial distributions to preserve both relationships across anatomical structures and spatial continuity within each structure.Its training matches pairwise cosine distances to ideal distances, with compactness, neighborhood, transformed-view consistency, and balanced-class constraints.Source: §3.2 · p. 4

3.3 ∆PHENOTYPE: LEARNING COUNTERFACTUAL PHENOTYPE INCREMENTS FROM NORMAL ANATOMY

  • ∆Phenotype estimates a lesion patch’s counterfactual normal representation from nearby mask-negative patches in the corresponding anatomical region, then isolates the pathological visual increment.The normal reference uses anatomically matched neighbors and normalized inverse-distance weights.Source: §3.3 · p. 5
  • Phenotype-specific objectives encourage increments from the same phenotype to remain consistent and separate different phenotypes, while additional lesion samples progressively refine their distributions against a fixed normal reference space.Localization supervision distinguishes lesion increments from hard normal patches, and anatomical preservation constrains normal-patch representations.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate ΔRepresentation across two chest CT datasets, six medical VLM backbones, VQA and report-generation tasks, and analyses of anatomical and phenotype representations. The results report improved grounding and phenotype characterization, with supporting analyses of spatial organization, data scale, and example reports.

4.1 DATASETS AND EVALUATION

  • The patient-disjoint evaluation uses ReXGroundingCT and LIDC-IDRI images, with Grounding Accuracy and Phenotype Accuracy measured for VQA and RRG.VQA grounding uses exact 4×4 grid-cell localization; RRG grounding assesses lesion-location attributes, while phenotype accuracy measures identification.Source: §4.1 · pp. 5–6

4.2 EXPERIMENTAL SETUP

  • The experimental analyses compare six pretrained medical VLMs and alignment methods, examine BaseAnatomy and counterfactual representations, vary lesion-positive data, and include an end-to-end case study.The setup identifies Tables 1–3 and Figures 3–6 as the corresponding comparisons and analyses.Source: §4.2 · pp. 6–7

4.3 COMPARISON WITH DIFFERENT MEDICAL VLMS AND ALIGNMENT METHODS

  • Across six backbones and both datasets, ΔRepresentation consistently improves VQA and RRG grounding and phenotype accuracy, and achieves the highest accuracies against RadAlign, MCA-RG, and Reg2RG under MedGemma1.5.The reported comparisons cover both ReXGroundingCT and LIDC-IDRI; Tables 1 and 2 present the VQA and RRG comparisons, respectively.Source: Table 1 · p. 6, Table 2 · p. 6, §4.3 · pp. 7–8

4.4 SPATIAL ORGANIZATION AND SCALE CONSISTENCY OF VISUAL ANATOMICAL REPRESENTATIONS WITH BASEANATOMY

  • After BaseAnatomy training, anatomical patches form more coherent clusters, vary gradually with position, and show cross-scale consistency; inferred counterfactual normal embeddings lie close to real normal representations.The two reported cosine distances between inferred and corresponding real normal representations are 0.003 and 0.005.Source: Figure 3 · p. 7, Figure 4 · p. 7, §4.4 · pp. 8–9

4.5 CONTINUAL REPRESENTATION LEARNING PERFORMANCE OF VISUAL PHENOTYPES AND T-SNE VISUALIZATION IN DIFFERENT DATA SCALE

  • As lesion-positive training increases from 1% to 100%, phenotype accuracy rises across both datasets and tasks, while phenotype representations become more compact and distinct.The reported accuracy ranges are 25.80 to 43.11 and 15.55 to 52.65 on ReXGroundingCT VQA and RRG, and 54.55 to 77.69 and 50.41 to 75.21 on LIDC-IDRI.Source: Table 3 · p. 8, Figure 5 · p. 8, §4.5 · p. 9

4.6 CASE STUDY

  • In two case studies, the model identifies ground-glass opacity and atelectasis and describes their bilateral lower-lung locations, while the original backbone reports no focal pulmonary abnormality.The examples illustrate anatomical projection and phenotype-increment projection contributing to report generation.Source: §4.6 · p. 9, Figure 6 · p. 9

5 CONCLUSION

The conclusion presents ΔRepresentation as anatomy-referenced incremental representation learning for medical VLMs, reporting improvements across two datasets and multiple backbones. The paper also reports its use of generative AI, ethical considerations, and code availability.

5 CONCLUSION

  • Across two datasets and multiple medical VLM backbones, experiments show consistent improvements in lesion grounding and phenotype characterization with anatomy-referenced incremental representation learning.BaseAnatomy structures normal anatomical representations, while ΔPhenotype learns phenotype-specific increments through counterfactual reasoning.Source: §5 · p. 10
  • Generative AI was used to parse report predictions for evaluation and to draft parts of the paper; the authors report manually checking and revising AI-assisted content.GPT6-Astra extracted lesion locations and phenotype labels, which were compared with reference annotations using deterministic metrics; it did not assign scores directly.Source: AI USE STATEMENT · p. 10
  • The authors report no ethical concerns requiring additional disclosure and provide an anonymous code repository to support reproduction.They state that no new human-subject recruitment or identifiable private information collection was involved.Source: ETHICS STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10

A.1 BENCHMARK CONSTRUCTION

The benchmark comprises patient-disjoint axial chest CT images from ReXGroundingCT and LIDC-IDRI. It supports lesion-grounding and phenotype VQA, as well as radiology report-generation evaluation.

A.1 BENCHMARK CONSTRUCTION

  • The benchmark contains 4,199 images split into 3,401 training, 394 development, and 404 locked test images, with no patient overlap between splits.The images come from ReXGroundingCT and LIDC-IDRI; their respective train/development/test counts are 2,720/230/283 and 681/164/121.Source: §A.1 · p. 13
  • Each image contributes lesion-grounding and phenotype VQA questions; grounding uses a 4×4 grid and lesion-mask annotations to define the reference cell.Annotations construct reference answers offline and are not provided at test-time inference.Source: §A.1 · p. 13
  • Each image also contributes an RRG prompt, yielding 808 VQA questions and 404 report-generation instances in the locked test set.RRG requests the primary abnormality and, when discernible, its laterality and craniocaudal region; unlike VQA grounding, it does not expose a grid cell.Source: §A.1 · p. 13

A.2 EVALUATION METRICS

The metrics score generated answers against reference annotations separately for each dataset. VQA uses exact matches, while RRG first parses free-form findings into structured predictions.

A.2 EVALUATION METRICS

  • Each dataset’s metrics use all test images as the denominator: 283 for ReXGroundingCT and 121 for LIDC.The reference labels are the grid cell or No lesion and the phenotype or no lesion.Source: §A.2 · pp. 13–14
  • VQA Grounding Accuracy and Phenotype Accuracy are exact-match accuracies of parsed generated answers against the reference grid cell and phenotype.The references can also be No lesion or no lesion, respectively.Source: §A.2 · p. 13
  • For RRG, GPT6-Astra extracts phenotype and lesion-location attributes, which are compared deterministically with their reference annotations.Location attributes include laterality and craniocaudal region; GPT6-Astra only structures the findings, and scores are computed from extracted outputs.Source: §A.2 · p. 14
  • For no-lesion RRG cases, an explicit statement that no focal lesion is present is parsed as No lesion.Extracted predictions are compared with reference annotations.Source: §A.2 · p. 14

B DISEASE-WISE RRG&VQA RESULTS

Disease-wise results show improvements across phenotypes and language backbones, with ∆Representation outperforming alternative alignment methods on MedGemma1.5 VQA subsets. Gains vary in size but are not confined to one disease category.

B DISEASE-WISE RRG&VQA RESULTS

  • The ReXGroundingCT breakdown covers 35 atelectasis, 43 consolidation, 88 ground-glass opacity, and 117 pulmonary nodule cases.Each task reports composite accuracy by disease subset, pooling its Grounding and Phenotype decisions.Source: B DISEASE-WISE RRG&VQA RESULTS · p. 14

Backbone-level consistency

  • Across six language backbones, adding ∆Representation improves nearly every disease-metric pair, while the remaining cells are maintained.The backbones are RadFM, LLaVA-Med, Lingshu, Hulu-Med, MedGemma, and MedGemma1.5; native report behavior differs across models.Source: B DISEASE-WISE RRG&VQA RESULTS · p. 14

MedGemma1.5 comparison

  • Adding ∆Representation improves VQA across all four MedGemma1.5 phenotype subsets versus RadAlign, MCA-RG, and Reg2RG, and improves RRG over the native backbone.The comparison fixes the MedGemma1.5 text backbone.Source: B DISEASE-WISE RRG&VQA RESULTS · p. 14

Residual difficulty by phenotype

  • Improvement magnitudes vary by phenotype and backbone, but gains occur across all four phenotype subsets rather than one disease category.The disease-wise analysis complements the aggregate results and covers heterogeneous pathological appearances.Source: B DISEASE-WISE RRG&VQA RESULTS · p. 14

C HIGH-DIMENSIONAL REPRESENTATION DIAGNOSTICS

High-dimensional diagnostics assess anatomical stability, phenotype-increment organization, counterfactual reference quality, and sensitivity to spatial pooling. They support stable geometry and informative pooled representations while showing a modest pooling trade-off.

Overall diagnostics

  • Across these diagnostics, BaseAnatomy maintains stable geometry, ∆Phenotype produces separable increments, and representations remain informative after downstream spatial pooling.The diagnostics complement the paper’s qualitative observations.Source: C HIGH-DIMENSIONAL REPRESENTATION… · p. 15

Anatomical representation stability

  • Pearson correlations under rotation, scaling, and translation are 0.9742, 0.9751, and 0.9750, respectively, indicating stable anatomical representation organization.These measurements assess agreement under moderate spatial transformations.Source: C HIGH-DIMENSIONAL REPRESENTATION… · p. 15, Table 5 · p. 16

Phenotype organization and counterfactual normal references

  • Phenotype increments reach a 0.3607 cosine silhouette score and 75.46% nearest-prototype accuracy; normal-ring compactness is 0.9882.Mean and median cosine distances from abnormal representations to estimated normal references are 0.0233 and 0.0210, respectively.Source: C HIGH-DIMENSIONAL REPRESENTATION… · p. 15, Table 5 · p. 16

Sensitivity to spatial pooling

  • After exact 4×4 pooling, fixed 5-NN accuracy rises from 60.98% to 64.23%, and macro accuracy from 58.98% to 60.83%.Cosine silhouette and own-centroid cosine decrease modestly; pooled representations retain organization relevant to downstream alignment.Source: C HIGH-DIMENSIONAL REPRESENTATION… · p. 15, Table 5 · p. 16

D IMPLEMENTATION DETAILS

The implementation trains anatomy and phenotype representations in sequence for each VLM backbone, then aligns the retained phenotype encoder with the frozen text decoder. Training uses anatomical and lesion masks, but inference does not.

Backbone initialization and input

  • For each VLM, the anatomical encoder starts from its native vision tower and trains separately; the phenotype encoder is initialized from the BaseAnatomy checkpoint.After BaseAnatomy training, its encoder is frozen; only the final vision layers and normalization layer of the phenotype encoder are updated during phenotype learning.Source: D IMPLEMENTATION DETAILS · p. 16

BaseAnatomy training

  • BaseAnatomy uses balanced anatomical-class sampling, α = 0.65, and matched rotation and scale transformations; lesion masks exclude pathological patches from anatomical supervision.Rotations span [−10°, 10°] and isotropic scaling spans [0.92, 1.08].Source: D IMPLEMENTATION DETAILS · p. 16

∆Phenotype training

  • For each pathological patch, ∆Phenotype estimates its normal reference from 8–48 anatomically matched, mask-negative neighbors using normalized inverse-distance weighting.Neighbors are drawn within grid radii 4–10.Source: D IMPLEMENTATION DETAILS · p. 16

Optimization

  • Both representation-learning stages use AdamW in BF16 with weight decay 0.01, cosine learning-rate decay, and gradient clipping at 1.0.Checkpoints are selected only on the patient-disjoint development split.Source: D IMPLEMENTATION DETAILS · p. 16

Vision–language alignment and inference

  • Downstream alignment freezes the trained vision encoder and text decoder and optimizes only the multimodal projector; inference uses no anatomical masks, lesion masks, or counterfactual references.The MedGemma 1.5–4B configuration average-pools each 4×4 block of the 64×64 grid to produce 256 visual tokens.Source: D IMPLEMENTATION DETAILS · p. 17

Compute

  • Representation learning uses one or two RTX PRO 6000 Blackwell Server Edition GPUs; alignment and evaluation also use an RTX A6000 with 48 GB memory.Source: D IMPLEMENTATION DETAILS · p. 17

Related papers

Loading 2610.10286v1…