Counterfactual Contrastive Analysis

Yunlong He; Pietro Gori

Source-linked AI summary

Counterfactual Contrastive Analysis

Yunlong He, Pietro Gori

arXiv:2608.19032v1Aug 19, 2026cs.CVcs.AI

TL;DR

Visual counterfactual explanations require realistic, minimal edits that change predictions, yet common generative approaches rely on classifier guidance. This paper introduces a classifier-free Contrastive Analysis framework that swaps disentangled salient factors, achieving superior generation quality across three medical imaging datasets with 0.25s/image editing.

Problem
Existing visual counterfactual explanations seek realistic, minimal prediction-changing edits, often using classifier-guided objectives in applications where explainability affects trust and acceptance.
Method
The method disentangles common and dataset-specific factors with StyleGAN2 and swaps salient factors using F-space refinement for classifier-free counterfactual generation.
Results
Across three medical imaging datasets, the method improves disentanglement over CA baselines, surpasses prior VCE approaches in generation quality, and edits images in 0.25s/image.
Takeaways & Limitations
The framework provides high-quality, controllable counterfactuals that preserve shared anatomy while transferring dataset-specific patterns.
Takeaways & Limitations
Without strong assumptions, the method is not identifiable and does not enable causal discovery from purely observational single-modal data.

Abstract

from arXiv · show

Visual Counterfactual Explanations (VCEs) aim to explain image classifiers by generating minimally edited and realistic versions of an input image that change the classifier's prediction. Existing VCE methods are inherently classifier-dependent and therefore susceptible to classifier biases and failure modes, such as sensitivity to shortcut features and calibration errors. In this paper, we propose a classifier-free approach for visual counterfactual generation based on Contrastive Analysis (CA). Given two datasets corresponding to different classes (e.g., healthy and patients), we disentangle the generative factors that are common across the two datasets from those that are salient to each dataset, and generate counterfactual images by swapping only the salient factors. By operating directly on data distributions rather than decision boundaries, our method provides model-agnostic VCEs that are less sensitive to classifier biases. Our approach leverages the high-quality synthesis and well-structured latent space of StyleGAN2. We use the feature space F, instead than the usual W-space, to improve detail preservation. Unlike conventional CA approaches, which typically assume salient factors in only one dataset, we introduce an adapted framework and loss functions for VCE that allow multiple salient factors in each dataset. We evaluate our method on three medical imaging datasets and demonstrate superior counterfactual generation quality compared to existing approaches.

1 Introduction

The paper motivates visual counterfactual explanations as realistic, minimal, and interpretable edits for trustworthy medical-imaging classifiers. It proposes a classifier-free Contrastive Analysis framework that swaps disentangled class-specific factors and refines StyleGAN2 F-space features for controllable counterfactual generation.

Motivation: Visual counterfactual explanations seek the smallest realistic image changes that alter a classifier’s prediction, supporting trust in medical-imaging applications.The paper frames explainability as critical for accepting black-box models in real-world use.
VCE requirements: Counterfactual images must change the classifier’s prediction, remain realistic on the image manifold, and express minimal, interpretable semantic changes.Existing methods typically optimize these objectives through VAEs, GANs, or diffusion models.
Method: The proposed classifier-free CA method operates on data distributions, disentangles common and class-specific generative factors, and swaps salient factors to generate counterfactuals.This avoids constructing counterfactuals from a classifier’s decision boundary.
Contributions: The framework combines StyleGAN-based latent disentanglement with F-space refinement for high-fidelity, semantically interpretable medical-imaging counterfactuals.It uses StyleGAN2’s synthesis and F-space rather than standard W-space to improve detail preservation.
Contributions: The CA framework supports controllable, classifier-agnostic generation under both background-target and multi-salient assumptions.The method is evaluated on three medical imaging datasets.

2 Method

The method generalizes contrastive analysis to multiple salient factors in both datasets, separating shared and dataset-specific generative information. It trains this separation with distributional, independence, reconstruction, and F-space refinement objectives.

Multiple-salient factorization: The method extends contrastive analysis from the background-target setting to a multiple-salient setting, where both datasets contain distinct patterns absent from the other.It learns common factors shared across X and Y alongside salient factors specific to each dataset.
Common consistency: A discriminator aligns common codes from X and Y, encouraging them to capture consistent information across datasets.The discriminator distinguishes dataset origins from common codes, while the separator is trained adversarially to make those codes indistinguishable.
Common–salient independence: Adversarial regressors promote common–salient independence by preventing common outputs from predicting the corresponding salient outputs.This discourages the common representation from absorbing dataset-specific patterns, particularly because the common code is typically larger and more informative.
F-space refinement: After learning factors in StyleGAN2’s W space, an F-space refinement module computes salient-swap feature shifts and adapts them using source-image features.The shifts replace one salient factor while keeping the source common factor fixed, then concatenate the shift with encoded reconstruction features before adaptation.

3 Experiments

Experiments evaluate the classifier-free method on three medical imaging datasets against CA, diffusion-based VCE, and adapted text-to-image baselines. Results show sharper reconstructions, effective salient-factor swaps, and competitive counterfactual quality, while diffusion methods can struggle with tumor editing.

Datasets: The method is evaluated on BloodMNIST, OCTMNIST, and BraTS2023 using balanced groups X and Y defined by dataset labels.The datasets cover blood-cell classification, two OCT comparisons, and healthy-versus-tumor brain imaging, with the supplied train/test sizes.
Baselines and Evaluation Metrics: Comparisons include CA baselines, diffusion-based VCE methods, and FLUX and SDXL adapted with LoRA as counterfactual alternatives.CA baselines are MM-cVAE, SepVAE, and Double InfoGAN; diffusion VCE baselines include ACE, DiME, FastDiME, and TIME.
Comparison with CA Baselines: Reconstructions are sharper and better preserve fine anatomical structures than CA baselines.The comparison considers reconstruction and salient swapping, using common and salient factors from the original images for reconstruction.
Comparison with CA Baselines: Salient swapping transfers tumors, staining differences, or CNV-related abnormalities while preserving shared anatomy across the evaluated datasets.The preserved structures include healthy brain tissues, global cell layout and nuclear morphology, and normal anatomy.
Results: The method achieves the lowest FID/sFID scores and is better or on par with ACE, DiME, and FastDiME for counterfactual effectiveness measured by FR/MAD/BKL.Counterfactual effectiveness is evaluated through FR, MAD, and BKL, while synthesis quality comparisons use metrics including L2, LPIPS, and MS-SSIM.
Results: Diffusion-based methods struggle to remove large tumors completely, often add small or misplaced tumors, and can drift from real images during text-conditioned or per-image editing.The observed failures concern both Y→X tumor removal and X→Y tumor addition.

4 Conclusion

The paper presents a classifier-free, contrastive-analysis framework that disentangles common and salient factors to generate high-quality visual counterfactuals. It also identifies non-identifiability and lack of causal discovery as limitations while suggesting VLM adaptation and additional priors as future directions.

Contributions: The proposed classifier-free, CA-based framework achieves strong common–salient disentanglement and high-quality counterfactual generations.The conclusion characterizes these as the framework’s central outcomes.
Contributions: Interpolating salient factors gradually changes class-specific evidence while preserving common content.The interpolation weight α varies from 0 to 1, and classifier predictions change consistently along the trajectory.
Limitations: The method is not identifiable without strong assumptions and does not enable causal discovery from purely observational single-modal data.The conclusion notes that these limitations are shared by recent VCE and CA methods and are generally unavoidable in this setting.
Future work: Future work includes adapting the method to recent VLMs, studying identifiability, and incorporating additional priors or longer-term information.The passage frames these as directions aligned with causal representation learning theory.

Supplementary Material · 5 Architecture Details

The architecture uses a multi-branch StyleGAN2-based separator to disentangle common and dataset-specific factors, with discriminator and regressor networks providing regularization. Remaining components support encoding, F-space refinement, generation, and evaluation of counterfactual attributes and factor separation.

5.1 CS Separator: The CS separator contains one common branch and one or two salient branches, adding an X-specific branch in the multiple-salient setting.In the background–target setting, the branches are Cϕc and Sy,ϕsy; the multiple-salient setting additionally uses Sx,ϕsx. All branches share an architecture but have independent parameters.
5.1 CS Separator: Each separator branch is an MLP adapted from StyleGAN2’s Z-to-W mapping network, with four equalized-learning-rate linear layers and LeakyReLU activations of slope 0.2.The separator differs from the original mapping network in its linear-layer parameterization, which uses weight tensors rather than single 512 × 512 matrices.
5.2 Regularization Networks D and R: The domain discriminator regularizes common representations by flattening c ∈ R^L×512 into a 7168-dimensional vector when L = 14 at 256 × 256 resolution.It is implemented as a binary linear classifier.
5.2 Regularization Networks D and R: The dependency regressor predicts salient representations from common representations using two fully connected layers with an intervening ReLU and no output activation.For c ∈ R^B×L×512, batch and style dimensions are merged before mapping from (BL) × 512 to (BL) × 512, then the output is reshaped.
5.3 Other Architectures and Implementations: The remaining components use pSp for E, StyleGAN2 for G, and SFE-based F-space refinement with the original encoder Ef and trainable adapter FAdpt.The implementation also refers to a provided StyleGAN2 Z-to-W mapping-network implementation and uses a PyTorch implementation for G.
5.3 Other Architectures and Implementations: Image-level evaluation applies a U-Net-based classifier trained on real training images to edited images, while latent-space evaluation uses logistic regression to predict dataset labels from common or salient factors.The logistic regression classifier is implemented in scikit-learn.

6 Additional Results

Additional results show that the proposed training components progressively improve salient-factor swapping, while reconstruction and swapping examples demonstrate bidirectional edits across medical imaging datasets under BT and MS assumptions.

Ablation study: Adding LD improves tumor removal in the y →x direction, while LR enables clearer bidirectional edits and better preserves source anatomy.F-space refinement further enhances local details and image fidelity.
Ablation study: With only base reconstruction objectives, salient-factor swaps produce limited class-specific changes.This ablation is illustrated on BraTS using healthy and tumor MRI inputs.
Additional examples: Additional reconstruction and salient-factor swapping results combine each source image’s common representation with the other image’s salient representation.The examples use BT for Figs. 7–9 and MS for Fig. 10.
Additional examples: The additional examples cover BloodMNIST, BraTS, and OCTMNIST, including BT-based swaps and an OCTMNIST comparison under the MS assumption.The displayed input pairs span eosinophil–neutrophil, healthy–tumor, normal–CNV, and DME–drusen datasets.