PaperlayerView exact revision on arXiv

Source-linked AI summary

SCCL-DT: Semantic-Conflict-Aware Contrastive Learning with Dual-Rule Teacher Distillation for Zero-Shot Chest X-Ray Diagnosis

Ting Liu, Haoran Lai, HaobinChen, Ming Liu, S Kevin Zhou

arXiv:2610.22094v1cs.CV

TL;DR

Standard contrastive pre-training treats unmatched image–report pairs as negatives, even when reports from different patients share findings. SCCL uses modality-specific adaptive thresholds to hard-mask candidate conflicts and distills inclusive and consensus teachers into one student. Across five public benchmarks, SCCL-DT achieves a mean zero-shot AUC of 0.850 versus 0.837 for RadZero.

  • Problem

    Standard contrastive objectives treat every off-diagonal image–report pair as a negative, although different patients’ reports can share findings and be semantically compatible.

    Source: §1 · pp. 1–2

  • Method

    SCCL tracks modality-specific similarity distributions with EMA statistics, adaptively hard-masks flagged candidates, and distills inclusive and consensus teachers into one student.

    Source: §1 · p. 2

  • Results

    0.850 mean zero-shot AUC across five public benchmarks, versus 0.837 for RadZero.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The results support adaptive conflict removal as an effective alternative to indiscriminate negative supervision under the evaluated settings.

    Source: §5 · p. 9

    Candidate-pair analyses provide proxy evidence rather than clinical adjudication, and future work should clinically adjudicate pairs and study calibration across batch sizes and domains.

    Source: §4.1 · p. 6, §5 · p. 9

Abstract

from arXiv · show

Contrastive vision-language pre-training supports zero-shot chest X-ray diagnosis, but standard objectives treat every off-diagonal image-report pair in a batch as a negative. In radiology, recurring findings across patients mean that instance mismatch need not imply semantic incompatibility, creating false-negative supervision that we term a semantic conflict. A training-trace audit shows that pair-observations selected by the online mask used during training are enriched for report-derived finding agreement, although this proxy analysis does not establish clinical equivalence. Fixed thresholds can become miscalibrated as similarity distributions evolve, while soft reweighting leaves residual repulsive gradients. We therefore propose Semantic-Conflict-Aware Contrastive Learning (SCCL), which tracks modality-specific similarity distributions with exponential-moving-average (EMA) statistics, derives adaptive thresholds, and hard-masks flagged candidates before softmax normalization. Inclusive (OR) and consensus (AND) rules define two SCCL teachers, whose bidirectional cross-modal rankings are consolidated through standard distillation into a single deployable student, SCCL-DT. Across five public chest X-ray benchmarks, SCCL-DT achieves a mean zero-shot AUC of $0.850$, compared with $0.837$ for the strongest evaluated baseline in our comparison. Code will be made publicly available soon.

1. Introduction

Medical vision–language pre-training supports zero-shot chest X-ray diagnosis, but standard contrastive learning treats every unmatched image–report pair as a negative. SCCL addresses potential semantic conflicts with adaptive hard masking and dual-rule teacher distillation into a single student.

1. Introduction

  • In radiology, reports from different patients can describe overlapping findings, so an unmatched pair may still be semantically compatible and receive false-negative supervision.The paper calls this mismatch between semantic compatibility and the assigned contrastive target a semantic conflict.Source: §1 · p. 2
  • Online inclusive-mask pair-observations show 3.96% exact present-label-set agreement versus 1.83% among eligible observations, a 2.16× enrichment across five consecutive training windows.This label-based proxy does not establish clinical equivalence.Source: Figure 1 · p. 1, §1 · p. 2
  • Fixed or globally normalized thresholds can become miscalibrated as modality-specific similarity distributions shift, while soft reweighting leaves residual repulsive gradients unless weights reach zero.The paper identifies adaptive detection and preventing flagged candidates from being pushed apart as the two remaining needs.Source: §1 · p. 2
  • SCCL tracks image–image and report–report similarity distributions with separate EMA statistics, sets adaptive thresholds, and hard-masks flagged candidates before contrastive normalization.The approach removes flagged candidates from negative supervision rather than relabeling them as positives.Source: §1 · p. 2
  • Inclusive OR and consensus AND teachers provide rule-specific supervision, which distillation consolidates into the deployable SCCL-DT student.The OR rule accepts evidence from either modality; AND requires agreement between both modality-specific cues.Source: §1 · p. 2
  • SCCL-DT reaches a mean zero-shot AUC of 0.850 across five public chest X-ray benchmarks, the highest among compared methods.The comparison includes single-model inference cost.Source: §1 · p. 2

2. Related Work

Prior chest X-ray vision–language work advances data use, encoders, knowledge integration, and fine-grained alignment, while SCCL targets a distinct supervision error: semantically compatible pairs treated as negatives. Its rule-specific teachers are consolidated through distillation into one student.

2.1. Medical vision–language pre-training

  • SCCL addresses a distinct source of supervision error from prior medical vision–language advances: treating semantically compatible off-diagonal pairs as negatives.Earlier approaches include broader data utilization, stronger encoders, knowledge integration, and finer-grained alignment.Source: §2.1 · pp. 2–3

2.2. False negatives in contrastive learning

  • Prior false-negative methods use distribution correction, hard-negative mining, progressive removal, or hardness scheduling, while medical VLP approaches use clinical knowledge, report semantics, or embedding and semantic similarity.SCCL instead uses modality-specific adaptive thresholds, hard-masks selected pairs without relabeling them, and distills rule-specific teachers into one student.Source: §2.2 · p. 3

2.3. Knowledge distillation

  • Knowledge distillation transfers teacher predictive distributions to a student, and SCCL uses it to consolidate inclusive and consensus teachers into a single student for efficient inference.The related formulations include model compression, peer supervision, and momentum-teacher targets.Source: §2.3 · p. 3

3. Methodology

SCCL modifies batch-contrastive supervision by adaptively identifying similarity-flagged off-diagonal candidates and excluding them from negative normalization. Inclusive and consensus teachers train separately, then distillation transfers their bidirectional rankings to one inference student.

3.1. Preliminaries: Batch-Contrastive Alignment

  • Batch-contrastive alignment scores image–report pairs in both directions, with matched pairs on the diagonal and every off-diagonal entry initially supervised as a negative.SCCL changes score-matrix supervision without changing the aligner’s internal architecture.Source: §3.1 · pp. 3–4

3.2. Semantic-Conflict-Aware Contrastive Learning

  • Separate EMA statistics track off-diagonal image–image and report–report similarities, with modality-specific thresholds set to each EMA mean plus k times its standard deviation.Masking is disabled during the initial warm-up; validation freezes the EMA states and counters.Source: §3.2 · p. 4
  • The inclusive OR rule flags a candidate when either modality cue exceeds threshold, while consensus AND requires both cues; OR favors coverage and AND is more conservative.Without candidate-level ground truth, this is an expected coverage–precision trade-off, not measured recall or precision.Source: §3.2 · p. 4
  • Flagged pairs are removed before softmax normalization in both score matrices, giving their softmax weights and gradients zero without relabeling them as positives.The same symmetric pair-level mask excludes both cross-swapped image–report candidates.Source: §3.2 · p. 4

3.3. Dual-Rule Teacher Distillation

  • Two frozen teachers provide supervision to an independently initialized student, with their raw logits averaged as targets and row- and column-normalized rankings distilled in both directions.The student is named SCCL-DT; teacher logits are computed on the fly rather than precomputed or stored.Source: §3.3 · pp. 4–5
  • SCCL-DT retains only the student at inference, using the teachers’ contrastively learned bidirectional rankings as its complete distillation supervision.The student is trained solely with the KL distillation loss.Source: §3.3 · p. 5

4. Experiments

Across five chest X-ray benchmarks, SCCL-DT attains the highest mean zero-shot AUC among compared methods, and its controlled studies examine the effects of adaptive thresholds, teacher rules, masking, and similarity cues.

4.1. Setup

  • Evaluation uses five public datasets without fine-tuning; classification is measured primarily by AUC, and grounding by mean Pointing Game accuracy across ten ChestXDet10 findings.The implementation builds on CARZero and pre-trains on frontal MIMIC-CXR images with usable reports; the comparison includes publicly released baseline models.Source: §4.1 · p. 5
  • The online OR mask enriches report-derived exact label agreement from 1.83% of eligible pairs to 3.96% of flagged pairs, a 2.16× lift.This proxy analysis does not measure recall of clinically adjudicated false negatives.Source: §4.1 · pp. 5–6

4.2. Main Results

  • 0.850 mean AUC gives SCCL-DT the highest zero-shot classification result across five benchmarks, ahead of RadZero (0.837) and CARZero (0.836).It ranks first on Open-I, ChestX-ray14, and ChestXDet10, and second on PadChest and CheXpert.Source: Table 1 · p. 6, §4.2 · p. 6
  • SCCL-DT outperforms CARZero on four of five datasets and RadZero on all but PadChest, with a 0.013 mean-AUC gain over RadZero.On CheXpert, SCCL-DT and CARZero are nearly tied at 0.922 and 0.923, respectively; the comparison re-evaluates public models on the same official test sets.Source: §4.1 · p. 5, §4.2 · p. 6
  • 0.583 mean Pointing Game accuracy is SCCL-DT’s highest result on ChestXDet10, exceeding CARZero (0.543) and RadZero at 224 pixels (0.537).The metric counts a hit when the maximum-attention region overlaps an annotated lesion, indicating spatial correspondence rather than detection or segmentation performance.Source: §4.2 · p. 6, Table 2 · p. 7, Figure 3 · p. 7

4.3. Ablation and Diagnostic Studies

  • Using both rule-specific teachers raises cross-dataset mAUC to 0.850, versus 0.846 for the inclusive-teacher student and 0.844 for the consensus-teacher student.Both teachers improve PadChest AUC, MCC, and F1 over Base; the dual-teacher student has the highest PadChest AUC and ties the highest MCC, while the inclusive teacher is marginally higher in F1.Source: §4.3 · p. 7, Table 3 · p. 7
  • Adaptive k=2 achieves PadChest AUC/MCC/F1 of 0.826/0.179/0.165, versus 0.814/0.156/0.138 for the strongest non-degenerate fixed threshold, τ=0.95.Fixed thresholds τ ≤0.80 are omitted because early OR-mask saturation nearly removes all off-diagonal negatives and yields a degenerate near-zero loss; adaptive thresholds track evolving similarity and approach 0.55 late in training.Source: §4.3 · p. 8, Table 4 · p. 8, Figure 4 · p. 8
  • Inclusive selection reaches PadChest AUC/MCC/F1 of 0.826/0.179/0.165; hard masking scores 0.826/0.179/0.165 versus 0.820/0.161/0.139 for soft weighting, and joint image-text cues outperform either alone.Consensus nearly preserves AUC at 0.825 but lowers MCC/F1 to 0.173/0.159; Two-tier performs worse across all three metrics, and count-matched random masking scores 0.817/0.156/0.139.Source: §4.3 · p. 8, Table 5 · p. 8

5. Conclusion

The conclusion presents adaptive conflict removal as an alternative to indiscriminate negative supervision, with SCCL-DT performing best on the evaluated benchmarks. The authors also identify clinical adjudication and broader calibration studies as future work.

5. Conclusion

  • SCCL uses modality-specific adaptive thresholds to remove candidate conflicts from contrastive normalization without relabeling them as positives.The online mask is enriched for report-derived finding agreement, which the authors describe as proxy evidence rather than pair-level clinical adjudication.Source: §5 · p. 8
  • 0.850 vs. 0.837 mean zero-shot AUC, and 0.583 mean ChestXDet10 Pointing Game accuracy, are SCCL-DT’s highest results among evaluated methods.The authors say these results support adaptive conflict removal under the evaluated settings.Source: §5 · p. 9
  • Future work should clinically adjudicate candidate pairs, examine calibration across batch sizes and domains, and extend detection to local region–token alignment and other medical imaging modalities.Source: §5 · p. 9

Supplementary Material

The supplement documents evaluation and implementation details, including the pre-training data choices shared across model variants.

Supplementary Material

  • The supplement covers data and evaluation scope, student distillation, Pointing Game protocol, ablations, candidate-set analysis, and low-fixed-threshold failure behavior.Source: Supplementary Material · p. 11
  • The supplement distinguishes proposed SCCL components from dataset and evaluation choices fixed across model variants.Source: Supplementary Material · p. 11, §A.1 · p. 11

A.1. Pre-training data

  • All variants use the same frontal AP/PA MIMIC-CXR training and validation manifests, report filters, and dataloading configuration.SCCL changes candidate-negative treatment during pre-training, not dataset partitioning or report selection.Source: §A.1 · p. 11

A.2. Downstream evaluation scope

The appendix specifies shared zero-shot evaluation conditions and the Stage-II procedure for distilling rule-specific teachers into a student.

A.2. Downstream evaluation scope

  • All compared models are evaluated without fine-tuning on the same benchmark subsets, target-label mappings, and metrics, using released checkpoints and inference code.ChestXDet10 supports classification and grounding; the other four datasets are used for classification.Source: §A.2 · p. 11

B. Stage-II Student Distillation Objective

  • The student matches row- and column-normalized distributions derived from the element-wise average of frozen inclusive and consensus teacher logits.The Stage-II objective is pure distillation, and only the student is retained for inference.Source: B. Stage-II Student Distillation… · p. 11
  • The two teachers remain frozen throughout Stage II, while the student alone is retained for inference.Source: B. Stage-II Student Distillation… · p. 11

C. Grounding Evaluation Details

Grounding is evaluated on ChestXDet10 with disease-prompt cross-attention maps and lesion annotations.

C. Grounding Evaluation Details

D. Diagnostic-Control Definitions and Stage-I Training Objective

The section defines diagnostic controls and explains how their masks or weights enter the stage-I contrastive objective. It also distinguishes corpus-level collision estimates from analysis of the online masks applied during training.

D.1. Two-tier: modality-asymmetric two-level rule

  • Two-tier uses a high tier that accepts strong evidence in either modality and a low tier that requires agreement between modalities.Its thresholds use different image- and text-side multipliers.Source: §D.1 · p. 12

D.2. Count-matched Random

  • Count-matched Random uniformly samples exactly the reference inclusive run’s number of unordered pairs per training call, but does not match per-row or per-column counts.The selected pairs are mirrored into a symmetric mask.Source: §D.2 · p. 12

D.3. Soft weighting: continuous Noisy-OR retention

  • Soft weighting assigns continuous false-negative probabilities by modality and applies a Noisy-OR retention weight in logit space before cross-entropy.The weight is added after fusion logits are produced; larger γ makes the transition sharper and closer to hard masking.Source: §D.3 · p. 13

D. Diagnostic-Control Definitions and Stage-I Training Objective

  • All variants use the shared fusion module and bidirectional contrastive cross-entropy, with control effects entering through within-batch logits before the loss.Positive diagonal pairs are retained, and mask or weight computation is excluded from gradient propagation.Source: D. Diagnostic-Control Definitions and… · pp. 11–12, §D.4 · p. 13

D.4. Unified execution summary

  • The unified execution sequence updates modality-specific similarity EMAs, forms masks or weights after warm-up, then injects them before the bidirectional loss.Two-tier and Random replace selected logits; Soft adds log retention weights.Source: §D.4 · p. 13

E. Training-Time Candidate-Set Analysis

  • The candidate-set analysis separately estimates random-batch finding-set collisions and tests whether the evolving online OR masks select pairs enriched for report-derived agreement.The trace analysis follows training-time decisions rather than reconstructing a detector from the final checkpoint.Source: E. Training-Time Candidate-Set Analysis · p. 13

E.1. Corpus-level collision analysis

This section estimates how often random batches contain cross-patient items with matching report-derived finding sets. It defines the anchor population and reports the resulting collision frequency.

E.1. Corpus-level collision analysis

  • The filtered corpus contains 227,237 image–report items, including 134,937 pathology-positive anchors defined by at least one of 12 findings.Present sets use only raw CheXpert values of 1 and exclude No Finding and Support Devices for the pathology-positive definition.Source: §E.1 · p. 13
  • The calculation counts other-patient corpus items sharing an anchor’s complete 14-label present set, then estimates matching batch mates under uniform random batching.Figure 1(b) averages the collision probability and expected count equally over the prespecified anchor population.Source: §E.1 · pp. 13–14
  • At batch size 256, 66.78% of pathology-positive anchors encounter at least one exact cross-patient label-set match, with 2.78 such matches per anchor.The all-anchor expectation is dominated by No Finding anchors, which account for 91.3% of expected matches.Source: §E.1 · p. 14

E.2. Recorded online masks and analysis endpoint

The analysis records the online masks actually applied during training and defines a best-checkpoint-aligned trace prefix. It specifies eligible pair observations and the estimands used to assess agreement enrichment.

E.2. Recorded online masks and analysis endpoint

  • The stored training trace includes sample identities, batch and training-step metadata, EMA statistics, thresholds, and the exact masks applied.For every analyzed batch, the applied mask equals the union of the stored image-high and text-high masks.Source: §E.2 · p. 14
  • The best-checkpoint-aligned endpoint spans mask steps 500–27,504, covering 27,005 batches and 505,289,600 strict-upper-triangle pair observations.The trace covers logged epochs 0–20 and batch sizes 64, 128, and 256.Source: §E.2 · p. 14

E.3. Estimands and uncertainty

  • The primary pool requires pathology-positive samples from different patients, and the analysis defines flagged agreement, baseline, lift, trigger rate, and capture rate.Confidence intervals use an epoch-stratified batch-level bootstrap; the baseline includes flagged observations from the same batches and pool.Source: §E.2 · p. 14, §E.3 · p. 14

E.4. Exact-agreement results

The online OR mask is enriched for report-derived finding agreement across the primary trace and several sensitivity checks. The section also reports fixed-threshold saturation and qualifies what the proxy analysis establishes.

E.4. Exact-agreement results

  • 3.9556% of flagged CheXpert-pool observations agreed exactly versus 1.8340% baseline, a 2.1568× lift and 8.0071% capture rate.The batch-conditioned expectation was 121,400.07 agreements, compared with 261,620 observed.Source: §E.4 · p. 14, Table S3 · p. 15

E.6. Image, text, OR, and AND rules

  • Image-only, text-only, and both-source flags account for 19.03%, 79.61%, and 1.36%; their exact-agreement lifts are 3.7490, 1.7181, and 5.5483.This source decomposition is associative, not a causal modality ablation.Source: §E.6 · p. 15, Table S5 · p. 16

E.7. Partial finding overlap

  • Partial overlap also rises among flagged observations: 57.4% share at least one present finding versus 45.5% overall, with mean Jaccard similarity 0.252 versus 0.181.This indicates the exact-agreement result is not solely a consequence of the equality threshold.Source: §E.7 · p. 15, Table S6 · p. 16

E.8. Alternative pools and full-trace sensitivity

  • Across the full post-warm-up trace, the CheXpert-pool lift is 2.1774×; the analogous NegBio-defined analysis yields 2.1013×.These are sensitivity estimates; the best-checkpoint-aligned prefix remains the primary endpoint.Source: §E.8 · p. 15, Table S7 · p. 17

E.9. Interpretation

  • The evidence is an association within one recorded training trajectory; without manual clinical adjudication, it establishes neither pair-level clinical equivalence nor causal downstream benefit.The absolute exact-agreement rate is 3.9556% and the OR capture rate is 8.0071%.Source: §E.9 · p. 15

E.5. Temporal persistence

  • Lift ranged from 1.93 to 2.22 across five epoch windows, with every window-level bootstrap interval above one.The result supports persistence across windows, not a monotone learning trend, because batch size changes with epoch.Source: §E.5 · p. 15, Table S4 · p. 16

F. Fixed-Threshold Mask Saturation

  • Fixed OR thresholds τ=0.50, 0.55, and 0.80 saturated the mask; τ=0.95 was the best-performing non-degenerate setting, while adaptive thresholds track evolving similarity scales.At τ=0.80, the image branch selected every off-diagonal pair in the audited representation.Source: F. Fixed-Threshold Mask Saturation · p. 16

Related papers

Loading 2610.22094v1…