PaperlayerView exact revision on arXiv

Source-linked AI summary

Revealing Hidden Failure Modes in Chest X-ray Classification via Spectral Domain Analysis

Samuel Halimi, Loic Themyr, Arnaud Abreu

arXiv:2610.29830v1cs.CV

TL;DR

Chest X-ray models can lose performance under distribution shifts, while metadata and embedding-based analysis may miss signal-level variation. The paper clusters Fourier-derived spectral profiles to find failure-prone groups, reporting meaningful differences in model behavior across clusters and highlighting the relative structure of spectral energy as relevant.

  • Problem

    Chest X-ray classifiers can degrade on data outside their training distribution, while metadata- and embedding-based approaches may miss subtle signal-level shifts.

    Source: §1 · p. 1

  • Method

    The method computes Radially Averaged Power Spectrum profiles from image Fourier transforms and clusters them post hoc, including with hierarchical clustering.

    Source: §3.1 · p. 3, §2.2 · p. 3, §3.4 · p. 4

  • Results

    Across experiments, K-medoids and hierarchical clustering revealed model-behavior differences between spectral groups, with HAC isolating small atypical-profile sets that also had atypical performance across evaluation domains.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Spectral analysis exposes signal-level strata that standard monitoring may overlook, and robustness strategies may benefit from accounting for relative spectral structure such as low- and high-frequency balance.

    Source: §6 · p. 12

Abstract

from arXiv · show

Deep learning models for chest X-ray anomaly detection remain vulnerable to subtle distributional shifts (e.g., acquisition technique, patient-related factors, and preprocessing). Traditional error analysis often relies on semantic metadata or model embeddings, which can mask low-level signal variations that degrade performance. In this work, we propose a data-centric framework for automated failure mode discovery using spectral analysis. We project images into the frequency domain and extract a compact profile summarizing the distribution of signal energy across frequency bands. By performing unsupervised clustering on these spectral profiles, we demonstrate that model failures are not randomly distributed, but are strongly concentrated within specific spectral clusters. This method effectively isolates "blind spots", enabling the prediction of model reliability and the discovery of performance-degrading data slices without requiring ground-truth failure annotations.

1. Introduction

Clinical deployment raises the need to anticipate failures when chest X-ray models encounter data outside their training distribution. The paper proposes a signal-centered framework that uses frequency profiles to identify concentrated failure patterns and model blind spots.

1. Introduction

  • Chest X-ray models can degrade on data that deviates from their training distribution, making failure anticipation important for clinical reliability.The introduction notes that controlled-setting performance does not ensure generalization in deployment.Source: §1 · p. 1
  • Metadata stratification and embedding-based analysis can miss subtle signal-level shifts associated with performance degradation.The paper identifies these high-level semantic approaches as limited for capturing low-level irregularities.Source: §1 · p. 1
  • The proposed data-centric framework prioritizes signal characteristics over semantic features or metadata labels.It hypothesizes that systematic model failures correlate with specific frequency-domain profiles.Source: §1 · p. 1
  • Unsupervised clustering of compact frequency-energy profiles identifies spectral groups where model failures concentrate, exposing blind spots without ground-truth failure annotations.The profiles are computed from image Fourier transforms, and the clusters are presented as potential predictors of model reliability.Source: §1 · p. 2

2. Related work

Prior slice-discovery approaches use categorical metadata or model feature spaces, while frequency-domain work has mainly altered spectra for robustness training. This paper instead uses spectral analysis as a post-hoc diagnostic for signal-driven failures.

2.1. Slice Discovery in medical image datasets

  • Categorical slicing depends on tabular attributes and can miss hidden stratification caused by signal characteristics absent from clinical logs.Prior work cited for this issue includes Seyyed-Kalantari et al. (2020), Larrazabal et al. (2020), Ahluwalia et al. (2023), and Oakden-Rayner et al. (2020).Source: §2.1 · p. 2
  • Feature-distance OOD methods can miss subtle acquisition variation that degrades performance without triggering distance-based alarms (Wiles et al., 2021).The passage distinguishes these subtle shifts from stark outliers.Source: §2.1 · p. 2
  • Latent-space clustering may suppress failure-causing irregularities because deep networks are optimized for nuisance-variable invariance (Achille and Soatto, 2018).The passage argues this can render such failure modes invisible to latent clustering.Source: §2.1 · p. 2

2.2. Frequency domain analysis in deep learning

  • Unlike frequency-based domain generalization and adaptation methods that manipulate spectra during training, this work uses spectral analysis post hoc to cluster data by image physics.The paper frames this as a way to expose signal-driven failures that standard monitoring misses.Source: §2.2 · p. 3

3. Methods

The methods represent images with radial frequency profiles and cluster those profiles to examine model performance across spectrally coherent groups. The workflow combines correlation-based similarity, exploratory K-medoids, and hierarchical clustering with explicit handling of small outlier groups.

3.1. Radially Averaged Power Spectrum (RAPS)

  • RAPS maps an image’s Fourier magnitude spectrum to a one-dimensional, orientation-invariant descriptor by averaging spectral magnitude over frequency coordinates at sampled radii.The figure illustrates contributions from three frequency bands to the RAPS representation.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2. Similarity metric

  • Correlation distance compares RAPS profiles across sampled radii and groups their spectral morphology rather than absolute magnitude.It is invariant to additive and multiplicative shifts, preserving features such as relative sharpness, noise patterns, and spectral slopes while discounting global energy or sensor-gain changes.Source: §3.2 · p. 4

3.3. K-medoids clustering

  • K-medoids serves as an exploratory test of whether RAPS descriptors relate to model failures, with cluster counts varied to assess patterns at different granularities.The method is not specifically optimized for slice discovery.Source: §3.3 · p. 4

3.4. Hierarchical Agglomerative Clustering (HAC)

  • HAC merges the closest profile clusters by average linkage and cuts the dendrogram at the 0.5-quantile of pairwise distances.This adapts partition scale to spectral variability without requiring a preset cluster count, supporting isolation of underperforming slices.Source: §3.4 · p. 4

3.5. Outliers

  • HAC groups with fewer than 10 samples are classified as outliers and aggregated into one composite category.The paper defines these as isolated observations or small terminal groups remaining separate until late in the hierarchy.Source: §3.5 · p. 5

4. Experimental setup

Experiments use four public chest X-ray datasets and train pathology-specific classifiers on CheXpert, with other datasets serving as unseen domains. The setup standardizes image dimensions and evaluates classifier discrimination with AUROC.

4.1. Datasets and preprocessing

  • The study uses CheXpert, MIMIC-CXR, PadChest, and NIH ChestX-ray14, with dataset characteristics spanning different views, labels, and acquisition settings.The evaluation includes three findings—Atelectasis, Consolidation, and Pleural Effusion—in separate mono-pathology models trained on CheXpert; MIMIC-CXR, PadChest, and NIH are unseen domains.Source: §4.1 · p. 5
  • Images are standardized to 264×224 pixels using isotropic resizing and minimal center-cropping, while incompatible aspect ratios are excluded.The procedure is intended to preserve spectral geometry for radial averaging, although interpolation may slightly attenuate high-frequency components.Source: §4.1 · p. 5

4.2. Models, training and performance assessment

  • DenseNet-121 classifiers are trained with binary cross-entropy and Adam, and AUROC measures discrimination across experiments.Training uses early stopping based on validation performance; AUROC is selected for threshold invariance and robustness to fluctuating prevalence in unsupervised cluster assignments.Source: §4.2 · p. 6

5. Experiments and results

Across experiments, clustering RAPS profiles by correlation distance reveals performance differences that exceed random baselines and persist across pathologies, architectures, and cluster counts. Hierarchical clustering also identifies performance-degraded spectral subgroups on unseen domains, while the seen-domain CheXpert outlier group is not markedly different from other clusters.

5.1. Relevance of RAPS descriptors for performance analysis

  • The dispersion effect holds across pathologies, ResNet, EfficientNet, and DenseNet architectures, and all tested K values; correlation distance associates performance differences with spectral shape, including relative frequency distributions and slopes.The paper links the result to meaningful performance variability captured by Fourier-derived spectral representations.Source: §5.1 · pp. 6–7
  • RAPS k-medoids partitions show AUROC dispersion consistently above fully random and pseudo-random partitions, sometimes nearly twice as high.The pseudo-random control preserves the k-medoids cluster-size distribution and class proportions while discarding spectral organization; results are averaged over 50 seeds.Source: §5.1 · pp. 6–7, Figure 2 · p. 7

5.2. Influence of the similarity metric on performance stratification

  • Across all K values, correlation distance yields the largest separation from both random baselines; Spearman is next, while Euclidean shows only limited gains over the baselines.The ordering suggests relative spectral-energy structure is more informative for performance stratification than absolute magnitude.Source: §5.2 · pp. 7–8, Figure 3 · p. 8
  • Frequency-domain adaptation methods may benefit from accounting for relative radial-spectrum structure, including low/high-frequency balance and spectral-energy decay, rather than matching absolute energy alone.The paper presents this as a suggested direction for adaptation and robustness.Source: §5.2 · p. 8

5.3. Slice discovery and outlier detection

  • Across almost all considered datasets and pathologies, HAC isolates a cluster with significantly degraded AUROC on unseen-domain test data.HAC clusters show distinct AUROC levels across pathologies and external validation centers, and the identified outlier set differs significantly from the other clusters.Source: §5.3 · pp. 8–10
  • The outlier super-group merges HAC clusters with fewer than 10 samples and often has lower AUROC than the main-cluster mean, though sometimes higher and never comparable to the average cluster performance.Examples of lower AUROC include Atelectasis and Pleural Effusion on MIMIC and NIH, and Consolidation on Padchest and MIMIC; higher AUROC occurs for Atelectasis and Pleural Effusion on Padchest.Source: §5.3 · p. 10
  • On seen-domain CheXpert, the outlier super-group is not markedly different in performance, although HAC still partitions images into subsets with distinct AUROC levels.The paper attributes the lack of a marked outlier-group difference to training exclusively on CheXpert.Source: §5.3 · p. 10, Figure 5 · p. 10

5.4. Comparison with OOD detection baselines

  • 0.103 mean absolute difference for HAC of RAPS, compared with 0.077 and 0.053 for the supervised OOD baselines, while HAC is fully unsupervised.The table reports AUROC differences between detected outliers and the global test-set baseline.Source: Table 2 · p. 11, §5.4 · p. 11

6. Conclusion

Fourier-domain profiles provide model-agnostic descriptors for examining chest X-ray classifier variability. Spectral clustering reveals differences in model behavior, while correlation distance highlights the relevance of relative spectral patterns.

6. Conclusion

  • K-medoids and hierarchical clustering revealed differences in model behavior across spectral-profile groups, and hierarchical clustering isolated small sets with atypical frequencies and performance across evaluation domains.Source: §6 · p. 12
  • Correlation distance produced the strongest separation between clusters, linking performance differences to relative spectral-energy distribution rather than absolute magnitude.The conclusion identifies spectral shape and rate of decay as relevant characteristics.Source: §6 · p. 12
  • The findings point to limitations of metadata- or embedding-based slice discovery and suggest targeting spectral structure, particularly low/high-frequency balance, in robustness and adaptation strategies.Source: §6 · p. 12

Appendix A. Ablation extension

The appendix evaluates how k-medoids distance choice relates to AUROC variation for Consolidation and Pleural Effusion; the supplied captions do not report which metric performs best.

Appendix A. Ablation extension

  • The Consolidation ablation evaluates how the k-medoids distance metric influences AUROC variation.Source: Figure 6 · p. 17
  • Both ablations compare Euclidean (L2), Spearman, and correlation distance, with correlation distance used in the proposed method.Source: Figure 6 · p. 17, Figure 7 · p. 17
  • The Pleural Effusion ablation evaluates how the k-medoids distance metric influences AUROC variation.Source: Figure 7 · p. 17

Appendix B. Results with other models

The appendix evaluates RAPS clustering with ResNet50 and EfficientNet, comparing AUROC variability across cluster counts against fully and pseudo-random clustering; results average 50 seeds.

Appendix B. Results with other models

  • RAPS is evaluated on ResNet50 (He et al., 2015), comparing AUROC variability across cluster counts for correlation-metric k-medoids and fully and pseudo-random clusters.Source: Figure 8 · p. 18
  • Both model evaluations average results over 50 seeds.Source: Figure 8 · p. 18, Figure 9 · p. 18
  • RAPS is evaluated on EfficientNet (Tan and Le, 2020), comparing AUROC variability across cluster counts for correlation-metric k-medoids and fully and pseudo-random clusters.Source: Figure 9 · p. 18

Appendix C. Comparison with semantic metadata

For Pleural Effusion in NIH, the comparison reports that metadata slicing can yield only minor AUROC variation, whereas the proposed framework isolates a highly vulnerable slice with a substantial performance drop.

Appendix C. Comparison with semantic metadata

  • For Pleural Effusion cases from NIH, metadata slicing by age, gender, and view sometimes identifies only minor AUROC variations.Source: Appendix C · p. 19
  • The proposed framework isolates a highly vulnerable Pleural Effusion slice in NIH with a substantial and critical performance drop.Source: Appendix C · p. 19
  • Table 3 compares conventional semantic-metadata slicing with the proposed HAC of RAPS method for Pleural Effusion in NIH.Source: Table 3 · p. 19

Appendix D. Qualitative analysis

The appendix captions identify qualitative result panels for Atelectasis, Consolidation, and Pleural Effusion across Padchest, MIMIC, and NIH.

Appendix D. Qualitative analysis

Related papers

Loading 2610.29830v1…