PaperlayerView exact revision on arXiv

Source-linked AI summary

Text-Guided Diffusion-Based Adversarial Attacks on Chest X-Ray Images

Basudha Pal, Arjun Narayanan, Neha Ajith, Vikas R Bhat, Muhammad Umair

arXiv:2608.29456v1cs.CV

TL;DR

CXR robustness studies often rely on pixel-space attacks that may not represent plausible radiographic variation, motivating a generative threat model. The paper optimizes learnable text conditioning through a frozen diffusion generator and classifier, finding stronger classifier degradation with higher image fidelity and frequent preservation of clinician interpretation.

  • Problem

    Existing CXR robustness evaluations are dominated by pixel-space attacks, whose small numerical perturbations may not represent clinically plausible radiographic variation.

  • Method

    The framework optimizes learnable text conditioning while keeping the diffusion generator and target CXR classifier frozen, generating adversarial images through a learned image prior.

  • Results

    The diffusion-guided attack produced the greatest overall AUROC reduction while generating the most perceptually faithful adversarial images across binary and multiclass CXR classification.

  • Takeaways & Limitations

    Clinician interpretation remained unchanged for most evaluated diffusion-generated images despite substantial changes in classifier behavior, exposing a machine-human interpretation discrepancy.

  • Takeaways & Limitations

    Clinician evaluation was preliminary, perceptual metrics did not directly measure clinically relevant anatomy or pathology, and experiments covered a limited set of classifiers and architectures.

Abstract

from arXiv · show

As artificial intelligence is increasingly integrated into chest X-ray (CXR) interpretation, triage, and clinical decision support, understanding its vulnerability to adversarial manipulation is critical for safe deployment. Existing robustness evaluations, however, predominantly rely on pixel-space attacks that introduce numerically constrained perturbations but may not represent plausible radiographic variation. This limitation is particularly important in multi-disease CXR classification, where models simultaneously evaluate multiple overlapping pathologies and adversarial failures may alter several diagnostic predictions. We propose a text-guided diffusion-based adversarial framework that optimizes learnable text conditioning while keeping the diffusion generator and target classifier frozen, enabling adversarial generation through a learned image prior rather than direct pixel manipulation. We evaluate the framework across multiple classifier architectures in both binary atelectasis and multi-disease CXR classification and compare it with FGSM, PGD, and Carlini-Wagner attacks. Our approach consistently produced the greatest degradation in classifier performance, reducing AUROC to 0.3885-0.5646 in binary classification and 0.4441-0.4878 in the multi-disease setting, while achieving superior image fidelity (SSIM 0.9080, LPIPS 0.1670, FID 51.23). Importantly, clinician interpretation remained unchanged for 95.9% of binary and 73.8% of multi-disease adversarial images despite substantial changes in model predictions. These findings reveal a clinically important discrepancy between human and machine interpretation and demonstrate the need to extend medical AI robustness evaluation beyond conventional pixel-space attacks toward generative threat models that can expose failures under visually and clinically plausible image variations.

1 Introduction

Existing CXR robustness evaluations largely use pixel-space perturbations that may be mathematically small yet radiographically implausible. This work introduces a text-conditioned diffusion attack to test classifier failures under coherent image variation while assessing image fidelity and clinical interpretation.

  • Motivation: Pixel-space attacks constrain perturbation magnitude but may introduce artifacts unrelated to anatomical, physiological, or acquisition variability.This limitation motivates evaluating adversarial robustness under more clinically plausible radiographic changes.
  • Motivation: Multi-disease CXR classification creates a broader threat because adversarial modifications may simultaneously affect predictions across co-occurring disease classes.
  • Clinical criterion: Clinician assessment is necessary because perceptual metrics alone cannot establish preservation of diagnostic content when machine predictions change.A clinically meaningful vulnerability requires changed machine output alongside stable clinical interpretation.
  • Approach: The proposed framework optimizes learnable text conditioning while keeping the diffusion generator and disease classifier frozen.The learned conditioning guides image generation, and gradients propagate through the text-conditioning pathway rather than directly through image pixels.
  • Evaluation: The study compares diffusion-guided attacks with FGSM, PGD, and C&W across multiple architectures and binary and multi-disease settings.Evaluation includes structural, perceptual, and distributional fidelity measures, clinician assessment, preprocessing defenses, and adversarial training.
  • Threat model: Diffusion-guided attacks probe classifier stability under learned generative variation rather than only sensitivity to additive image-space noise.The learned image prior can produce coherent structural or textural modifications that conventional defenses may not remove through smoothing or compression.

2 Methods

The method targets frozen CXR classifiers in binary and multi-label settings, seeking adversarial images that change predictions while preserving the original radiograph’s anatomical structure through a diffusion-model latent manifold.

  • Problem setup: The experiments cover binary disease prediction and multi-label prediction of multiple thoracic diseases.
  • Problem setup: The pretrained CXR classifier outputs probabilities for C pathologies and remains frozen during adversarial generation.
  • Objective: The optimization seeks an adversarial image that alters classifier predictions while preserving the original radiograph’s anatomical structure.
  • Objective: Unlike pixel-based attacks, the similarity constraint is enforced within the latent manifold of a pretrained diffusion model.

2.2 Architecture Overview

The architecture inserts a learnable placeholder token into a prompt, uses its encoded embedding to condition a frozen diffusion generator, and evaluates the reconstructed image with a frozen CXR classifier.

  • Components: The framework contains a trainable text encoder, a frozen Stable Diffusion generator, and a frozen chest X-ray disease classifier.
  • Conditioning: A learnable placeholder token is inserted into a prompt and mapped to an embedding that conditions the diffusion generator.
  • Evaluation: The generated image is evaluated by the downstream disease classifier.
  • Optimization: Gradients propagate only through the text encoder while the diffusion model and classifier remain frozen.Adversarial perturbations are therefore induced solely through prompt conditioning.

2.3 Diffusion-Based Image Reconstruction

Image reconstruction begins by encoding the input radiograph into latent space, then applies diffusion noise and denoising conditioned on text before decoding the adversarial image.

  • Latent reconstruction: The input image is encoded into latent space with a variational autoencoder rather than generated from random noise.This latent reconstruction is used to preserve anatomical realism.
  • Forward diffusion: Noise is added through the forward diffusion process, with the noise level controlled by timestep t.
  • Denoising: The frozen UNet predicts noise conditioned on the text embedding during denoising.
  • Image generation: The adversarial image is obtained by decoding the denoised latent estimate.
  • Design principle: The formulation keeps perturbations on the diffusion manifold to preserve radiographic realism.

2.4 Binary Disease Classification Attack

The binary attack targets a single disease prediction and generates an image that reverses the classifier’s disease-presence decision. Its loss encourages either presence-to-absence or absence-to-presence errors.

  • The binary setting predicts the presence or absence of a single disease such as atelectasis or pneumonia.
  • The attack generates an adversarial image that flips the classifier prediction relative to the ground-truth label.
  • Minimizing the classification loss encourages presence-to-absence or absence-to-presence prediction errors.

2.5 Multi-Class and Multi-Label Attack

The multi-label attack predicts multiple thoracic diseases simultaneously and uses an untargeted classification objective combined with perceptual similarity constraints. Among successful candidates, the method selects the perturbation with minimum similarity loss.

  • The multi-label classifier predicts probabilities for 15 thoracic diseases simultaneously.
  • The untargeted objective makes the classifier incorrect on all diseases originally positive in the ground-truth label.The classification loss is negative binary cross-entropy combined with a VGG19-based multi-level perceptual loss.
  • Among successful adversarial candidates, the perturbation with minimum similarity loss is selected.

2.7 Attack Success Criterion

Attack success is defined by incorrect classifier predictions relative to the original labels, with distinct binary and multi-label criteria. Successful samples are retained, and adversarial training augments the classifier’s training data with generated examples.

  • An adversarial attack succeeds when the classifier prediction for the target disease is flipped relative to the ground-truth label.
  • In binary classification, success occurs when the disease-presence prediction crosses its decision threshold.This corresponds to changing a prediction from disease present to absent, or vice versa.
  • In multi-label classification, success requires every originally positive disease to be predicted as absent using class-specific thresholds.
  • Successful adversarial samples are retained for downstream evaluation.
  • Adversarial training incorporates generated adversarial samples into the training dataset and retrains the classifier for robustness.

2.10 Multi-Class and Multi-Label Attack

The multi-label attack evaluates simultaneous disease predictions using an untargeted loss and perceptual regularization. Performance is assessed with AUROC and image fidelity using SSIM, LPIPS, and FID.

  • Multi-Class and Multi-Label Attack: The multi-label classifier predicts multiple thoracic diseases simultaneously.
  • Multi-Class and Multi-Label Attack: The untargeted objective penalizes incorrect predictions for originally positive diseases while a VGG19 perceptual loss encourages visual realism.
  • Multi-Class and Multi-Label Attack: Table 1 reports AUROC for binary atelectasis detection and multiclass disease classification under different adversarial attacks.Lower AUROC indicates greater degradation in diagnostic performance.
  • Multi-Class and Multi-Label Attack: Table 2 reports averaged adversarial-image fidelity using SSIM, LPIPS, and FID across tasks.Higher SSIM and lower LPIPS and FID indicate improved similarity or perceptual realism.

3 Results

Across binary atelectasis and multiclass classification, the diffusion-based attack most severely degraded classifier performance while preserving radiographic appearance and clinician-perceived content better than pixel-space attacks.

  • Classifier performance: The diffusion-based attack produced the greatest overall AUROC degradation across binary atelectasis and multiclass disease classification.All evaluated attacks reduced AUROC relative to clean images across architectures.
  • Classifier performance: AUROC fell to 0.3885–0.5646 in binary atelectasis classification under the diffusion-based attack.DenseNet, EfficientNet, and ResNet each reached their lowest AUROC under the proposed attack.
  • Classifier performance: AUROC fell to 0.4441–0.4878 in multiclass classification under the diffusion-based attack across DenseNet, ResNet, and EfficientNet.The attack consistently produced low AUROC across architectures, with several conventional attacks also approaching chance-level discrimination.
  • Image fidelity: SSIM = 0.9080, LPIPS = 0.1670, and FID = 51.23 indicated the diffusion-based method achieved the strongest visual fidelity among evaluated attacks.CW had the next-best perceptual-quality measurements, while PGD and especially FGSM produced greater distortion.
  • Image fidelity: Diffusion-generated images preserved global thoracic anatomy and major structures with substantially less apparent corruption than FGSM and PGD.The qualitative findings were consistent across binary and multiclass settings and with the quantitative fidelity measurements.
  • Prediction changes: The diffusion attack redirected disease profiles, including suppression of abnormalities, diagnostic substitution, and false-positive disease predictions in multiclass examples.These prediction changes occurred without correspondingly conspicuous changes in radiographic appearance.
  • Defenses: Adversarial training improved robustness more than Gaussian smoothing or JPEG compression, but no defense restored clean-image performance.Against the diffusion attack, AUROC increased from 0.4441 without defense to 0.5362 with 75% adversarial training, while the attack remained most challenging.
  • Clinician assessment: Clinician interpretation remained unchanged in 95.9% of diffusion-generated binary cases despite substantial classifier-performance degradation.Only 4.1% of cases were judged to exhibit a diagnostic change after perturbation.

4 Discussion

The discussion positions diffusion-guided attacks as clinically relevant robustness tests because they can alter automated predictions while preserving plausible radiographic content and human interpretation. It also emphasizes that conventional defenses and evaluations remain incomplete, while broader clinical validation is still needed.

  • Interpretation: The study found CXR classifiers vulnerable to diffusion-based manipulation even when adversarial images retained substantial visual and clinical similarity to originals.The diffusion-guided attack produced the largest AUROC reduction across most evaluated architectures and both classification settings.
  • Robustness evaluation: Pixel-space robustness evaluations may underestimate vulnerabilities to semantic or generative attacks that remain within the data manifold.The discussion therefore supports incorporating generative and clinically constrained attacks alongside established pixel-space methods.
  • Image realism: Diffusion-generated radiographs retained major thoracic anatomy while achieving the strongest classifier degradation and the best SSIM, LPIPS, and FID results.This links the qualitative appearance of the examples with the quantitative perceptual analysis.
  • Human–machine discrepancy: Clinician interpretation remained unchanged for most evaluated diffusion-generated images despite substantial changes in classifier behavior.The result indicates that machine interpretation can change without a corresponding change in human diagnostic assessment.
  • Clinical relevance: A realistic adversarial radiograph could alter automated diagnostic outputs while escaping routine visual inspection.The discussion identifies possible consequences including missed detections, false-positive alerts, and altered automated risk estimates.
  • Defenses: Gaussian smoothing and JPEG compression provided only modest protection, whereas adversarial training improved robustness without restoring clean-image performance.Even 75% adversarial training left diffusion-generated examples as the most difficult to mitigate.
  • Limitations: The clinician evaluation should expand to multiple radiologists with varied expertise and formal inter-reader agreement analysis.The current assessment is described as an initial evaluation of clinical realism.
  • Limitations: The study evaluates a limited set of disease classifiers and architectures, leaving transferability across models, institutions, devices, and datasets for future work.Query-based optimization and black-box transferability are also identified as unresolved practical settings.

5 Conclusion

The diffusion-guided framework generated realistic adversarial chest X-rays that substantially altered automated classifier predictions across binary and multiclass tasks. Its results support generative threat models and human oversight for evaluating and deploying medical imaging AI.

  • The framework generated realistic chest X-ray images capable of substantially altering automated disease-classifier predictions.
  • Across binary atelectasis and multiclass classification, the method produced the greatest overall AUROC reduction among evaluated attacks.
  • The diffusion-guided method achieved the highest SSIM and lowest LPIPS and FID among evaluated attacks, indicating the most perceptually faithful adversarial images.
  • Clinician interpretation remained unchanged for most diffusion-generated images despite substantial changes in model behavior.
  • The findings motivate robustness evaluation with realistic generative threat models and human-in-the-loop deployment for high-stakes or uncertain cases.
Loading 2608.29456v1…