Source-linked AI summary
Visual Feature Attribution using Wasserstein GANs
Christian F. Baumgartner, Lisa M. Koch, Kerem Can Tezcan, Jia Xi Ang, Ender Konukoglu
TL;DR
Visual attribution methods based on neural-network classifiers may detect only a subset of category-specific features, limiting complete interpretation of images and disease effects. The paper introduces a WGAN-based image-dependent map generator that transforms category-specific images toward a baseline distribution. It reports state-of-the-art attribution results on synthetic data and neuroimaging, including realistic Alzheimer’s disease effect maps closely matching observed effects.
Problem
Classifier-based visual attribution may ignore weaker category-specific features when stronger redundant features are available, motivating methods that capture all category-specific effects.
Method
The method learns an image-dependent additive map with a WGAN that makes an input from one category indistinguishable from images in a baseline category.
Results
The method produces state-of-the-art disease-effect maps on synthetic and neuroimaging data, with Alzheimer’s predictions in very good agreement with observed effects.
Takeaways & Limitations
The approach can capture multiple disease-affected regions and generate subject-specific maps at high resolution without requiring registration.
Takeaways & Limitations
The method assumes test-time category labels are known and was evaluated only for two labels.
Abstract
from arXiv · showhide
Attributing the pixels of an input image to a certain category is an important and well-studied problem in computer vision, with applications ranging from weakly supervised localisation to understanding hidden effects in the data. In recent years, approaches based on interpreting a previously trained neural network classifier have become the de facto state-of-the-art and are commonly used on medical as well as natural image datasets. In this paper, we discuss a limitation of these approaches which may lead to only a subset of the category specific features being detected. To address this problem we develop a novel feature attribution technique based on Wasserstein Generative Adversarial Networks (WGAN), which does not suffer from this limitation. We show that our proposed method performs substantially better than the state-of-the-art for visual attribution on a synthetic dataset and on real 3D neuroimaging data from patients with mild cognitive impairment (MCI) and Alzheimer's disease (AD). For AD patients the method produces compellingly realistic disease effect maps which are very close to the observed effects.
1. Introduction
Visual attribution detects and visualises evidence of a category in an image, but classifier-based methods may miss weaker category-specific features. The paper proposes a WGAN-based alternative for subject-specific disease effect maps and evaluates it on synthetic data and neuroimaging.
- Visual attribution detects and visualises evidence of a particular category in an image, supporting localisation, segmentation, and analysis of medical image effects.
- Classifier-based attribution analyses gradients or feature-map activations, but classifiers may focus on salient regions and ignore weaker redundant features.
- The proposed method learns an image-dependent additive map with a WGAN so category-specific features can be transformed toward a baseline-category distribution.
- The method assumes test-image category labels are known and requires a baseline category, making it particularly suited to applications such as medical image analysis.
- The method is demonstrated on synthetic 2D data and 3D brain MR data to predict subject-specific Alzheimer’s disease effects relative to mild cognitive impairment.
- Subject-specific disease maps could support patient stratification, disease subtype analysis, and clinical assessment of disease status and grade.
2. Related work
Prior attribution and medical-image analysis methods include CAM, saliency maps, and statistical disease models, but they have resolution or registration limitations. The paper positions its method as a high-resolution, registration-free GAN approach demonstrated on large structural 3D data.
- CAM creates class-specific activation maps from late classifier feature maps but is limited by their spatial resolution and often requires post-processing.
- Backpropagation-based methods generate saliency maps by propagating information from a model output back to the input image.
- Statistical medical-image methods commonly produce population-average disease maps, while newer approaches estimate subject-specific maps.
- Registration-based disease-effect models require non-rigid alignment to a common template, introducing additional uncertainty into subject-specific maps.
- The paper addresses registration-related shortcomings by generating subject-specific disease-effect maps without registration.
- The paper demonstrates a limitation of neural-network attribution methods and proposes a more complete, high-resolution technique for class-specific regions.
- The authors describe the work as the first application of generative adversarial networks to large structural 3D data.
3. Visual attribution using WGANs
The method estimates an input-dependent additive map that transforms images from a category of interest into images indistinguishable from a baseline category, using a WGAN-trained map generator and critic. An L1 regularizer encourages the smallest realistic modification, while the architecture is designed for high-resolution 3D medical images.
- Problem formulation: The formulation estimates M(x_i) so that adding it to a category-1 image produces an image indistinguishable from the category-0 distribution.The resulting map contains features distinguishing the input category from the baseline and can represent disease effects in medical images.
- WGAN formulation: The map generator learns realistic modifications by transforming category-1 images, while a WGAN critic distinguishes generated images from real category-0 images.Unlike a standard GAN generator, M receives sampled images rather than random noise.
- Inference: During testing, the learned map generator predicts an attribution map in a single forward pass.The training overview uses category-1 images as inputs and targets indistinguishability from category-0 images.
- Constrained effect maps: The L1 regularization term encourages the smallest map that still produces a realistic transformed image.This limits changes that could alter subject identity instead of capturing category-related effects.
- Architecture and training: The networks are designed for high-resolution 3D medical images, including volumes of 128x160x112 voxels, under GPU-memory constraints.The design uses a reduced 3D U-Net map generator, a fully convolutional critic, and minibatch gradient aggregation.
4. Experiments
The experiments compare VA-GAN with classifier-based attribution and additive perturbation maps on synthetic data and 3D brain MRI. VA-GAN produced the most localized synthetic effect maps and realistic, closely matching disease maps on AD subjects, although observed-map correlations are difficult to interpret because the maps are noisy.
- Experimental setup: VA-GAN was compared with Guided Backpropagation, Integrated Gradients, CAM, and a non-GAN additive perturbation method.The comparison covered synthetic data and publicly available ADNI brain MRI data.
- Experimental setup: The alternative additive-map method minimizes a classifier’s prediction while regularization encourages small, smooth maps.The L1 term encourages small maps, whereas total variation encourages smoothness.
- Synthetic experiments: Backpropagation methods focused on the always-present central square and often emphasized box edges rather than whole objects.This behavior was consistent with classifiers emphasizing the most predictive features.
- Synthetic experiments: CAM usually captured both synthetic squares but had limited spatial resolution, while VA-GAN localized entire boxes, followed their edges, and consistently identified both effects.The CAM classifier’s final feature-map pixels had a 39x39-pixel receptive field, which may have affected its ability to represent both squares simultaneously.
- Neuroimaging experiments: VA-GAN predictions agreed well with observed AD effect maps, identifying hippocampal, periventricular, and later-stage temporal-lobe effects.Backpropagation methods and additive perturbations were noisy and tended to identify only hippocampal areas.
- Neuroimaging experiments: VA-GAN obtained the highest NCC scores, but correlations were difficult to interpret because observed effect maps were noisy and could include nondisease effects.The observed maps were treated as ground truth in the experiments despite these concerns.
- Neuroimaging experiments: VA-GAN produced realistic deformations that made generated images more similar to corresponding MCI images.A sample comparison showed realistic changes in the generated image.
5. Limitations and discussion
The method captures multiple disease-affected regions and produces realistic effect maps, but currently assumes known test labels and was evaluated only for two categories.
- VA-GAN captures multiple disease-affected regions and achieves state-of-the-art disease-effect-map results on neuroimaging and synthetic data.
- Figure 5 shows generated AD maps closely matching observed effects, including hippocampal and ventricular atrophy and later temporal-lobe involvement.
- Figure 6 reports realistic generated deformations that make an AD image more similar to the corresponding MCI image.
- The method assumes test-time category labels are known and was evaluated only for two labels, although multiple generators could extend it to more categories.
- The authors plan to investigate other effects, datasets, and applications such as weakly supervised localisation.
A. Network architectures
The 3D VA-GAN uses convolutional architectures for its critic and map generator, with global averaging, batch normalization, upsampling, and skip connections.
- Figure 7 presents the VA-GAN critic architecture.
- The critic architecture uses 3D convolutions without batch normalization and global average pooling over feature-map spatial dimensions.
- The map generator uses 3D convolutions with batch normalization, learned deconvolutional upsampling, and channel-wise skip connections across the bottleneck.
- The synthetic 2D experiments use the same architectures after replacing all 3D operations with their 2D equivalents.
B. Close-up analysis of VA-GAN
A subject-level close-up shows that VA-GAN produces attribution maps closely approximating observed deformations, including asymmetric hippocampal effects.
- VA-GAN closely approximates observed deformations across all three orthogonal planes for an additional subject.
- For this subject, VA-GAN correctly predicts a smaller disease effect in the left hippocampus than in the right hippocampus.
C. Details of MR brain data cohort
The study uses longitudinal T1-weighted MR data from ADNI cohorts, split by subject into training, validation, and test sets for model development and evaluation.
- The dataset contains 5,770 images from 1,291 subjects across ADNI1, ADNIGO, and ADNI2, acquired with 1.5T and 3T scanners.
- Subjects were scanned an average of 4.5 times, with visits typically spaced six months to one year apart over multiple years.
- The cohort includes 496 female and 795 male subjects.
- Data were randomly divided into training, testing, and validation sets on a subject basis rather than an image basis.
- Training learned the map generator and critic, validation supported monitoring and hyperparameter tuning, and testing provided final qualitative and quantitative evaluation.
D. Alternative classifier architecture
The alternative classifier replaces the final dense layers with global average pooling and a single dense layer to test whether saliency maps better capture multiple regions. Although some peripheral-box pixels become faintly attributable, classifier-based maps still show incomplete and unstable feature attribution, unlike VA-GAN maps.
- Architecture: The experiment replaced the synthetic classifier’s final two dense layers with global average pooling followed by a single dense layer.This architecture was evaluated after both networks had fully converged.
- Results: The alternative architecture enabled normal and guided backpropagation to identify some peripheral-box pixels, but only very faintly.These faint attributions were highlighted with white arrows in Fig. 11.
- Results: Regardless of architecture, classifier saliency focused on pixels from only one edge, capturing a subset of the features characterising the class.The attributed edge orientation depended on the network’s random initialisation.
- Comparison: VA-GAN attribution maps were judged higher quality than maps from backpropagation-based techniques, with additional examples shown in Fig. 12.The comparison includes the corresponding map for the first sample and two further random inputs.
- Interpretation: The two classifier designs behave similarly because their final predictions aggregate information from receptive fields covering the whole image and can compress redundant features.This compression can omit features such as one of the two boxes despite their relevance to the class.