PaperlayerView exact revision on arXiv

Source-linked AI summary

SentZero: An Enhanced Sentence-Centric Vision-Language Pretraining for Multi-Task Zero-Shot Chest X-Ray Analysis

Hangyul Yoon, Hyungyung Lee, Edward Choi, Eunho Yang

arXiv:2609.34479v1cs.CVcs.AI

TL;DR

Radiology reports are long and clinically dense, making simple-prompt zero-shot CXR analysis difficult and leaving prior methods reliant on task-specific fine-tuning. SentZero structures sentence-level semantics, expands shared positive pairs, mitigates false negatives, and conditions visual features on each sentence; it outperforms prior multi-task CXR models across diverse zero-shot tasks.

  • Problem

    Radiology reports are lengthy and clinically dense, making them difficult to align with simple zero-shot prompts and leaving many CXR methods dependent on task-specific fine-tuning.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    SentZero combines LLM-based abstract-level sentence mapping, selective patch-level attraction for shared clinical statements, and sentence-conditioned residual modulation of visual features.

    Source: §1 · p. 2

  • Results

    SentZero outperforms prior multi-task CXR models across diverse zero-shot tasks, including a 7.6-percentage-point pointing game accuracy gain over the strongest baseline on ChestXDet10.

    Source: §4.2 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    The authors identify explicit modeling of semantic hierarchy and cross-report expression overlap as key to robust, generalizable image–sentence alignment.

    Source: §5 · p. 10

Abstract

from arXiv · show

Vision-language (VL) pretraining using paired chest X-ray (CXR) images and radiology reports has shown strong potential for medical image understanding. However, existing methods often remain dependent on task-specific finetuning because radiology reports are lengthy, clinically dense, and difficult to align with simple zero-shot prompts. Recent sentence-level approaches partially address this limitation using clinical phrases extracted by large language models (LLMs), but they largely overlook the intrinsic characteristics of radiology discourse. In particular, limited positive-pair diversity constrains further gains, while clinically equivalent sentences frequently recur across patients, creating false negatives in contrastive learning. To address these issues, we propose SentZero, an enhanced sentence-centric VL pretraining framework for zero-shot, multi-task CXR analysis. SentZero introduces LLM-based abstract-level sentence structuring and mapping to expand positive-pair diversity, together with an additional loss term to mitigate false negatives. We further introduce sentence-conditioned residual modulation of visual embeddings, enabling visual features to adapt to the semantic characteristics of each input sentence. Across diverse downstream tasks and datasets, SentZero improves zero-shot generalization and outperforms prior multi-task zero-shot methods.

1. Introduction

Radiology reports make simple-prompt zero-shot CXR analysis difficult, and existing sentence-level methods leave recurring clinical equivalences and report structure underused. SentZero uses structured sentence mapping and patch-level alignment to address these issues, and reports stronger zero-shot performance across most evaluated benchmarks.

1. Introduction

  • Radiology reports are long, clinically dense, and variable, making image–text alignment noisier and simple-prompt zero-shot CXR analysis more difficult than general-domain recognition.Many existing methods consequently still require labeled, task-specific fine-tuning.Source: §1 · p. 1
  • Prior LLM rephrasing standardizes heterogeneous findings into sentence-level supervision, improving interpretability and enabling zero-shot grounding within a unified framework.The cited prior work is described as establishing the current state of the art.Source: §1 · p. 1
  • Sentence-level methods overlook radiology discourse structure, while clinically equivalent statements recur across patients and become false negatives in contrastive training.Because clinical descriptions occupy a narrow space, better phrase extraction alone cannot avoid these false-negative pairs.Source: §1 · pp. 1–2
  • Abstract-level mapping links detailed phrases to concise topic–presence statements, expanding positive pairs while retaining both detailed findings and underlying clinical concepts.A selective auxiliary loss attracts relevant patches toward shared statements, while sentence-conditioned residual modulation adapts attended visual features to each sentence.Source: §1 · p. 2
  • SentZero outperforms prior zero-shot multi-task methods on most evaluated benchmarks and transfers across downstream tasks without task-specific fine-tuning.The authors report that masking or relabeling false-negative pairs can impair performance, whereas selective patch-level attraction improves it while preserving the contrastive objective and pair assignments.Source: §1 · p. 2

2. Related Works

CXR vision–language pretraining aligns images with radiology reports, but prior methods commonly rely on task-specific fine-tuning despite progress in zero-shot transfer. SentZero is presented against this background as a framework that retains contrastive pair assignments while adding local patch alignment and sentence-conditioned visual adaptation.

2. Related Works

  • Most existing CXR vision–language pretraining methods still require task-specific supervised fine-tuning because lengthy, detailed reports complicate alignment with simple zero-shot prompts.Image–report pretraining has nevertheless been reported to adapt better to downstream medical tasks than general-domain ImageNet pretraining.Source: §2 · p. 2
  • MedKLIP and KAD incorporate medical knowledge, CARZero uses cross-attention and LLM phrase extraction, and RadZero adds similarity-weighted patch aggregation with multi-positive contrastive learning.The cited methods are MedKLIP, KAD (Zhang et al., 2023), CARZero (Lai et al., 2024), and RadZero.Source: §2 · p. 2
  • CoNNs relabels or excludes cross-patient pairs with shared clinical concepts, whereas SentZero retains contrastive pair assignments and applies auxiliary attraction to their highest-similarity patches.The authors report that direct relabeling or masking can degrade performance.Source: §2 · p. 3
  • SentZero also applies sentence-conditioned residual modulation to aggregated visual features before contrastive comparison, enabling sentence-dependent visual adaptation.Figure 1 depicts multi-pair sentence–patch contrastive learning, augmented positive pairs, and the combined contrastive and false-negative mitigation losses.Source: §2 · p. 3, Figure 1 · p. 3

3. Method

SentZero uses LLM-based phrase extraction and abstract-level sentence mapping to expand positive image–sentence pairs, then trains with sentence-conditioned visual features and selective false-negative mitigation. Its trained model supports zero-shot classification and spatial grounding without task-specific fine-tuning.

3. Method

  • SentZero's training pipeline combines LLM sentence mapping, sentence-conditioned image–text contrastive learning, and false-negative mitigation for zero-shot classification and grounding.The model is applied to downstream tasks without task-specific fine-tuning.Source: §3 · p. 3

3.1. Abstract-Level Sentence Mapping

  • LLMs map detailed clinical phrases to concise topic–presence sentences, adding unique positive image–sentence pairs while preserving underlying clinical concepts.For example, “There is mild opacity in the bilateral lung base” becomes “There is opacity.”Source: §3.1 · p. 4

3.2. Feature Extraction

  • A ViT-based visual encoder produces global and local patch embeddings, fusing transformed intermediate-layer representations with the final-layer representation before further trainable transformer refinement.The backbone is LoRA-finetuned, and selected intermediate layers each use a two-layer MLP and LayerNorm.Source: §3.2 · p. 4

3.3. Sentence-Conditioned Feature Modulation and Contrastive Loss

  • Sentence-to-patch similarities weight local visual features, which are then modulated by sentence semantics as a learned residual correction before image–sentence contrastive comparison.The modulation retains the aggregated visual feature through a residual connection and LayerNorm.Source: §3.3 · p. 5

3.4. False Negative Loss

  • For cross-study pairs sharing a clinical sentence, the auxiliary loss attracts only the top-M% most similar image patches while preserving the global contrastive objective and pair assignments.This adds local alignment supervision rather than removing false negatives from the contrastive loss.Source: §3.4 · p. 6

3.5. Multi-Task Zero-Shot Inference

  • At inference, clinical prompts support zero-shot classification, while patch-attention weights are upsampled into maps whose peaks identify likely finding locations.The attention map represents relative spatial importance across the image.Source: §3.5 · p. 7

4. Experiments

SentZero is evaluated for zero-shot classification and grounding across public CXR benchmarks, with component and false-negative-handling ablations and attention-map visualizations. It outperforms strong baselines on multiple datasets and shows improved grounding accuracy.

4.1. Experimental Settings

  • Evaluation covers zero-shot classification on five public datasets and grounding on ChestXDet10 and MS-CXR, using AUROC and pointing-game accuracy, respectively.The baselines are zero-shot-transfer-capable CXR VL methods.Source: §4.1 · p. 7
  • The vision encoder uses pretrained RAD-DINO with a LoRA adapter and four trainable transformer blocks, while MPNet encodes text and intermediate ViT layers are fused with final-layer representations.Training runs for up to 20 epochs with AdamW and early stopping; the reported training time is approximately five hours.Source: §4.1 · pp. 7–8

4.2. Main Results

  • SentZero leads multiple classification datasets and beats the strongest grounding baseline by 7.6 and 3.6 percentage points on ChestXDet10 and MS-CXR, while CARZero leads CheXpert AUROC.The evaluation compares zero-shot performance across vision tasks and datasets.Source: Table 1 · p. 8, §4.2 · p. 8

4.3. Ablation Studies on Model Components

  • Multi-layer feature aggregation most notably improves grounding; the full model is best or near-best on most datasets, though text-conditioned residual connection alone scores higher on PadChest and CheXpert.The component ablation also reports that adding each remaining component individually to multi-layer feature aggregation further improves classification performance.Source: §4.3 · p. 8, Table 2 · p. 9

4.4. Comparison on False Negative Handling Strategy

  • Masking and relabeling false negatives underperform SentZero on most benchmarks and usually underperform training without false-negative handling.Relabeling false negatives as positives causes large performance drops across multiple datasets; SentZero instead attracts high-similarity patches while preserving contrastive pair assignments.Source: §4.4 · p. 9, Table 3 · p. 9

4.5. Visualization of Attention Map

  • Attention maps highlight relevant pathological regions, including small focal lesions, and suggest potential for zero-shot visual grounding.The examples show original ChestXDet10 images alongside prompt-specific attention maps and green ground-truth boxes.Source: §4.5 · p. 9, Figure 2 · p. 9

5. Conclusion

SentZero combines abstract-level mapping, selective patch-level attraction for shared clinical statements, and sentence-conditioned residual modulation of visual features. It outperforms prior multi-task CXR models across diverse zero-shot tasks.

5. Conclusion

  • SentZero combines abstract-level mapping, selective patch-level attraction for shared clinical statements, and sentence-conditioned residual modulation of visual features.The framework is presented as sentence-centric VL pretraining designed for the semantic and structural complexity of CXR reports.Source: §5 · pp. 9–10
  • Across diverse zero-shot tasks, SentZero outperforms prior multi-task CXR models, supporting explicit modeling of semantic hierarchy and cross-report expression overlap for image–sentence alignment.The conclusion describes SentZero as a scalable foundation for semantically aware medical VL pretraining.Source: §5 · p. 10

A.1. Dataset Profiles

The evaluation uses public CXR datasets for classification, localization, and phrase grounding, with dataset-specific annotation and test-set choices. The authors report that evaluation images were excluded from both SentZero training and RAD-DINO pretraining.

A.1. Dataset Profiles

  • The evaluation images were excluded from SentZero training and from the pretraining data of the RAD-DINO checkpoint used in the experiments.Source: §A.1 · p. 13
  • Classification profiles include OpenI, ChestXray14, PadChest, and CheXpert; PadChest evaluation uses the 39,053-sample subset annotated by board-certified radiologists.CheXpert evaluation uses five observations and images from 500 patients.Source: §A.1 · p. 13
  • Grounding uses ChestXDet10’s 542 official test images and MS-CXR’s released 167-image test set, with MS-CXR linking bounding boxes to report phrases.Source: §A.1 · p. 13

A.2. Instructions for LLM-Based Abstract-Level Mapping

The LLM instruction extracts a structured topic-and-presence tuple for abstract-level sentence mapping. Generated outputs are format-filtered, and a few nonconforming samples are revised.

A.2. Instructions for LLM-Based Abstract-Level Mapping

  • Qwen3-Next-80B is instructed to extract a topic-and-presence tuple for abstract-level sentence mapping.The figure presents the instruction prompt used for tuple extraction.Source: §A.2 · p. 13, Figure 3 · p. 13
  • Generated outputs are filtered by format, and a few samples that do not conform to Python dictionary format are revised.Source: §A.2 · p. 13

B. Additional Experiments

The text-conditioning ablation finds improved performance on most datasets, while additional experiments examine the false-negative loss coefficient and patch-sampling ratio.

B. Additional Experiments

C. Additional Visualization Examples

The visualization examples show zero-shot grounding on ChestXDet10 and MS-CXR, including longer, more detailed clinical descriptions in MS-CXR.

C. Additional Visualization Examples

Related papers

Loading 2609.34479v1…