Does Vision-Language Pretraining Granularity Matter? A Controlled Evaluation of Vision-Language Objectives Across Chest X-Ray Interpretation Tasks

Task granularity

• Chest X-ray tasks range from global disease classification to spatially grounded abnormality detection and phrase grounding, with report generation and question answering between these extremes.The passage describes intermediate tasks as requiring global and local representations in varying degrees.Source: Introduction · p. 1

Pretraining granularity

• Global objectives align image-level content but provide limited spatially precise supervision, whereas local objectives explicitly align regional captions with image regions.The introduction frames the effect of this difference across the downstream task spectrum as underexplored.Source: Introduction · p. 1

Study design

• The study varies the pretraining objective while holding the vision encoder architecture fixed and evaluates representations across five chest X-ray interpretation tasks.The tasks span classification, report generation, VQA, medical difference VQA, and abnormality detection.Source: Introduction · p. 1

Study findings

• Local objectives outperform global ones on abnormality detection, while global objectives edge out local ones on classification.The reported pattern is more nuanced for intermediate tasks: local objectives are competitive on generation and VQA, while global captioning retains an advantage on RadGraph-F1.Source: Introduction · p. 2

Vision-Language Pretraining

• Vision-language pretraining developed as a way to learn aligned image-text representations from image-text collections, following the cost of scaling manual annotation.The passage describes contrastive learning as an early approach and captioning as a later generative alternative.Source: Vision-Language Pretraining · p. 2
• Recent work explores combining pretraining objectives to capture information at different levels of semantic granularity within a shared vision encoder.The passage situates this direction after image-text contrastive and generative captioning objectives emerged as alternatives.Source: Vision-Language Pretraining · p. 2

Global and Localized Vision-Language Alignment

• Global alignment objectives pair entire images with reports or captions but do not explicitly model relationships between specific regions and text descriptions.Examples include contrastive learning and image captioning.Source: Global and Localized Vision-Language… · p. 2
• Local objectives align image regions with localized text through referring expressions, grounded captioning, and region-level contrastive learning.Prior work cited in the passage suggests combining global and local objectives improves multimodal-task performance.Source: Global and Localized Vision-Language… · pp. 2–3

• All models use the same CLIP ViT-L/141 architecture and pretraining dataset, varying only the alignment objective.Source: Method · p. 3

Contrastive Learning

• Global contrastive learning trains image and text encoders to align matching image-report embeddings and separate unmatched pairs.The passage identifies the image encoder as CLIP ViT-L/14 and describes a bidirectional Transformer text encoder trained from scratch.Source: Contrastive Learning · p. 3

Contrastive Learning with Sigmoid Loss

• The sigmoid-based objective treats each image-text pair independently as a binary classification example, labeling matches positive and nonmatches negative.The setup uses a learnable bias and temperature, initialized to -10 and 10, respectively.Source: Contrastive Learning with Sigmoid Loss · p. 3

Image Captioning

• Image captioning trains a model to generate a report autoregressively from an image, conditioning each token on the image and preceding tokens.Visual embeddings are supplied to a Transformer decoder through cross-attention layers.Source: Image Captioning · p. 3

Local Captioning

• Grounded Captioning generates a regional caption from an image and bounding box; Automatic Referring Expression predicts the box, while Conditional Automatic Referring Expression generates both from an image and region name.The three autoregressive tasks are trained jointly, with task-specific prompts and loss computed over generated tokens.Source: Local Captioning · p. 3

Local Contrastive

• Local contrastive learning aligns ViT patch tokens inside anatomical bounding boxes with corresponding phrases, using matching phrase-and-anatomy pairs as positives and other combinations as negatives.It retains the global contrastive formulation, encoder, and training setup, changing only the granularity of aligned pairs.Source: Local Contrastive · p. 3

Local Contrastive Learning and Global Captioning

• Local contrastive learning and global captioning use a shared vision encoder with separate regional-contrastive and global-captioning modules, optimized in alternating steps with equal updates.The regional module is a bidirectional transformer encoder; global captioning uses a Transformer decoder.Source: Local Contrastive Learning and Global… · pp. 3–4

Local Captioning and Global Captioning

• Local captioning and global captioning train four tasks jointly in a multitask autoregressive framework, sampling each equally to balance task contributions.The additional global task generates the full radiology report from the image.Source: Local Captioning and Global Captioning · p. 4

Local Contrastive and Global Contrastive

• Local and global contrastive objectives share vision and text encoders and are interleaved so each receives an equal number of updates.The passage describes the training balance but does not specify further architectural differences.Source: Local Contrastive and Global Contrastive · p. 4

Local Captioning and Global Contrastive

• Local captioning and global contrastive learning share a vision encoder but use separate task-specific heads, with training interleaved to balance their contributions.The global contrastive head is a bidirectional encoder, while local captioning uses a Transformer decoder.Source: Local Captioning and Global Contrastive · p. 4

Task Setup

• Pretraining uses 100,000 MIMIC-CXR studies, while downstream generation and VQA use a disjoint 99,341; Chest ImaGenome supplies region boxes and phrases for local supervision.The study retains PA and AP views and filters out studies missing findings or impression sections.Source: Task Setup · p. 4
• Freezing the vision encoder and training only task-specific downstream heads isolates the effect of the pretraining objective.The encoder is trained from scratch alongside a text module during pretraining.Source: Task Setup · p. 4
• Models use CLIP ViT-L/14 and 50-epoch AdamW pretraining; downstream performance is averaged over three seeds and pairwise comparisons use corrected bootstrap tests.Generative tasks use BLEU, ROUGE, RadGraph-F1, and BERTScore; classification uses AUC and detection uses mAP@0.4 and mAP@0.5.Source: Task Setup · p. 4

Multilabel Classification

• Classification uses linear probing on NIH CXR-14, a 14-class multilabel dataset, with AUC as the evaluation metric.The primary protocol trains a single linear layer using binary cross-entropy and follows the official train/test splits.Source: Multilabel Classification · p. 4, Figure 2 · p. 5

Abnormality Detection

• Abnormality detection trains a DETR head on frozen features using classification, L1 box-regression, and generalized IoU losses, evaluated on VinDr-CXR.VinDr-CXR contains 18,000 images and 14 pathology classes, with 15,000 training and 3,000 test images.Source: Abnormality Detection · p. 4

Report Generation

• Report generation trains a decoder on frozen vision features using MIMIC-CXR training images and evaluates on MIMIC-CXR, IU X-ray, and CheXpert Plus.IU X-ray and CheXpert Plus are zero-shot evaluations because the decoder is not trained on them.Source: Report Generation · pp. 4–5

Visual Question Answering

• Visual question answering uses the single-image Medical Difference VQA subset, retaining questions whose images were excluded from pretraining to avoid leakage.Its six question categories include location, presence, abnormality, type, severity, and view.Source: Visual Question Answering · p. 5

Medical Difference VQA

• Medical Difference VQA trains a decoder to describe longitudinal changes between two images, concatenating frozen features and adding learned embeddings for reference versus current frames.Test-pair images are excluded from the pretraining dataset to prevent leakage.Source: Medical Difference VQA · p. 5

Results

• The results show no universal objective winner; objective type, granularity, and task characteristics interact.The evaluation covers nine pretraining objectives and five chest X-ray interpretation tasks spanning two granularity levels.Source: Results · p. 5

Pretraining granularity aligns with task granularity at the extremes

• Local objectives lead on abnormality detection, while global objectives lead on classification, aligning pretraining and task granularity at the spectrum’s extremes.On VinDr-CXR, Local Contrastive scores 6.56 mAP@0.4 versus 5.02 for the best global single objective, Global Sigmoid; on NIH-CXR-14, Global Contrastive scores 65.3 AUC versus 63.5 for the best local objective, Local Contrastive.Source: Pretraining granularity aligns with… · p. 5

Local objectives are competitive on global tasks

• Across report generation and VQA, local objectives match or outperform global objectives on NLG metrics, though their advantage is not consistent on RadGraph.For single-image VQA, Local Contrastive leads Global Sigmoid by nearly 7 BLEU-4 points (28.55 vs. 21.57); RadGraph differences are generally small and inconsistent.Source: Local objectives are competitive on… · pp. 5–6, Table 1 · p. 6

Captioning and contrastive learning excel at different granularity levels

• Global captioning outperforms global contrastive learning on most tasks, while local contrastive learning outperforms local captioning on most tasks.The MIMIC-CXR significance results show Global Captioning significantly above Global Contrastive and Global Sigmoid across all metrics; the relative merits of captioning and contrastive learning therefore depend on spatial scale.Source: Captioning and contrastive learning… · p. 6

Mixing captioning and contrastive objectives yields the strongest combinations

• Mixing captioning and contrastive supervision produces the strongest combinations on classification and in-distribution generation, while two captioning objectives lead on zero-shot report generation.Local captioning plus global captioning is weakest on classification and MIMIC-CXR RadGraph, but strongest among combinations on zero-shot IU X-ray and CheXpert Plus report generation.Source: Mixing captioning and contrastive… · p. 7

Limitations

• Findings are limited to frozen-encoder transfer with one architecture and pretraining scale, and most intermediate tasks derive from MIMIC-CXR.The local objectives also require dense region-level bounding-box and phrase annotations, which are costly and unavailable for most datasets and imaging modalities.Source: Limitations · p. 7

• No single pretraining objective is universally optimal; downstream performance depends on the interaction of objective type, granularity, and task.Source: Conclusion · p. 7
• Localized objectives lead on abnormality detection, while global objectives lead on classification.Source: Conclusion · p. 7
• On report generation and VQA, local objectives are competitive, outperforming global objectives on NLG metrics while global captioning retains an edge on RadGraph-F1.Source: Conclusion · p. 7
• Global captioning beats global contrastive learning, whereas local contrastive learning beats local captioning across most tasks.Source: Conclusion · p. 7
• Mixing captioning and contrastive supervision is strongest on classification and in-distribution generation, while two captioning objectives generalize best on zero-shot report generation.Source: Conclusion · p. 7