PaperlayerView exact revision on arXiv

Source-linked AI summary

Targeted Visual Counterfactual Explanations for Contrastive Vision-Language Model

Van Bach Nguyen, Jörg Schlötterer, Christin Seifer

arXiv:2609.37638v2cs.CV

TL;DR

CLIP explanations highlight relevant regions but do not show how to change an image to obtain a target prediction. MACE uses adaptive attribution masks and CLIP-guided diffusion inpainting; across four datasets, its mask variants trade validity against image preservation and realism.

  • Problem

    CLIP explanations mainly identify relevant regions without showing how to change an input to alter its prediction, while prompt alignment does not ensure a target top prediction across the full zero-shot class set.

    Source: §1 · p. 1

  • Method

    MACE builds an editable region from source or source–target difference attribution, expands the mask only when needed, and uses CLIP-guided diffusion to inpaint the selected area.

    Source: §1 · p. 1

  • Results

    Across ImageNet, Food-101, Oxford Pets, and CUB-200, the source-attribution variant achieves the highest validity, while the source–target difference variant produces smaller pixel and perceptual changes and better realism scores.

    Source: §8 · p. 8

  • Takeaways & Limitations

    The results show adaptive masking balancing counterfactual validity against preservation of the source image, with both variants improving proximity and realism over full-image Stable Diffusion editing.

    Source: §8 · p. 8

    MACE’s pairwise source–target score objective encourages the target to outrank the source but does not explicitly require it to outrank every class in the zero-shot label set.

    Source: §8 · p. 8

Abstract

from arXiv · show

Current explanation methods for contrastive vision-language models such as CLIP mainly identify important regions without showing how to change the input in order to get a target prediction. We introduce Mask-guided Adaptive Counterfactual Explanations (MACE), a targeted visual counterfactual method designed specifically for CLIP zero-shot classification. MACE constructs an editable region from either source attribution or source-target attribution differences and expands the mask only when needed to reach a specified target class. A latent diffusion inpainting model then modifies the selected region, while a frozen CLIP model provides modification guidance and anchors the remaining image content to the original input. We evaluate MACE on ImageNet, Food-101, Oxford Pets, and CUB-200. The source-mask variant achieves the highest target top-1 success rate across all four datasets, while the difference-mask variant produces the smallest pixel level and perceptual changes and the best realism scores. Both variants improve proximity and realism over a Stable Diffusion-only baseline using the same generative backbone. These results show that adaptive mask-guided editing produces effective CLIP counterfactuals. They further reveal a tradeoff between counterfactual validity and source-image preservation.

1. Introduction

The introduction identifies a gap between CLIP explanations that highlight regions and counterfactuals that show how to change an image for a target prediction. MACE addresses it with adaptive attribution masks and localized, CLIP-guided diffusion, with different mask variants favoring validity or image preservation.

Motivation

  • CLIP explanations mainly highlight relevant regions without showing how to change an input to alter its prediction.Counterfactual explanations instead identify a small, meaningful change that produces a different model outcome.Source: §1 · p. 1
  • Existing visual counterfactual methods target conventional classifiers, while CLIP-guided editing can align with a target prompt without ensuring a top prediction across the full zero-shot class set.These editing methods also do not necessarily find the smallest localized change responsible for the decision change.Source: §1 · p. 1

Approach

  • MACE uses source-attribution or source–target difference masks, expands the editable region only when needed, and selects the smallest mask that changes the zero-shot prediction.CLIP-guided diffusion inpaints the selected region while preserving the rest of the image.Source: §1 · p. 1

Main results

  • Across ImageNet, Food-101, Oxford Pets, and CUB-200, the source-mask variant achieves the highest validity, while the difference-mask variant produces the smallest image changes and best realism scores.Both variants improve proximity and realism over a Stable Diffusion-only baseline using the same generative backbone.Source: §1 · p. 1

Scope

  • The paper evaluates two mask variants on four datasets and examines how CLIP guidance strength, mask fraction, and adaptive expansion affect counterfactual generation.It presents these examinations as practical insights for other diffusion-based editing methods.Source: §1 · p. 1

2. Related Work

The related work spans visual counterfactuals, CLIP interpretation and editing, and counterfactual learning for vision–language models. The paper positions MACE as an image-level method for targeted counterfactuals of CLIP zero-shot predictions.

2.1. Visual Counterfactual Explanations

  • Visual counterfactual methods use localized replacement, optimization, or generative priors to seek sparse, semantically consistent, realistic prediction-changing edits.Diffusion-based approaches constrain edits to the natural-image manifold, but are primarily designed for conventional CNN classifiers.Source: §2.1 · p. 2

2.2. CLIP Explainability

  • CLIP explainability methods identify regions or concepts contributing to image–text similarity, but do not establish whether modifying that evidence changes the zero-shot prediction.The literature includes attribution, localization, concept-based, and multimodal approaches.Source: §2.2 · p. 2

2.3. CLIP-Guided Image Editing

  • CLIP-guided editing methods align images with target text prompts, but do not explicitly seek the smallest change that changes a CLIP classifier’s prediction.CF-CLIP [40] uses contrastive objectives to encourage more localized and semantically accurate edits.Source: §2.3 · p. 2

2.4. Counterfactual Learning for Vision–Language Models

  • Counterfactual learning methods modify training or representation spaces, rather than providing image-level explanations for individual predictions of an already trained model.Examples include Counterfactual Prompt Learning [11], COMO, and CF-VLM.Source: §2.4 · p. 2

2.5. Positioning of Our Work

  • MACE operates in CLIP’s joint image–text embedding space and generates an explicit counterfactual image for a target class.Its stated question is what minimal, semantically meaningful visual change suffices for CLIP to predict the target rather than the source class.Source: §2.5 · p. 2

3. Problem Formulation

The formulation considers targeted visual counterfactuals for CLIP zero-shot classification: a desired target label, a changed image, and constraints on prediction, closeness, and realism. MACE uses attribution to choose an editable region and expands it when a localized intervention fails.

3. Problem Formulation

  • CLIP is described as an image encoder and a text encoder, with class-specific text prompts used for zero-shot classification.The supplied passages introduce normalized embeddings and cosine-similarity class scores but do not include their full expressions.Source: §3 · p. 2
  • For an input image, the source class is its current prediction and the target class is a desired alternative.The objective is a target-class counterfactual that remains close to the original and visually realistic.Source: §3 · pp. 2–3
  • MACE forms an attribution mask, expands it only when a localized intervention fails, and uses CLIP-guided latent diffusion to inpaint the selected region.The remaining image content is preserved; the overview starts with the top 25% of attribution values and expands in 5% steps until a candidate reaches the target class.Source: Figure 1 · p. 3
  • The formulation defines realistic targeted counterfactuals as a feasible set and expresses the desired counterfactual as a distance-minimizing solution.The equation’s feasible set is described in the surrounding prose as realistic images classified as the target class.Source: §3 · p. 3

4. Method

MACE localizes editable regions with CLIP attribution, expands them when a localized edit does not reach the target, and uses CLIP-guided latent diffusion inpainting while preserving the remaining image.

4. Method

  • MACE first locates where to edit with CLIP attribution, then inpaints the selected region under CLIP guidance while preserving the remaining image.The mask expands only when a strongly localized intervention fails.Source: §4 · p. 3

4.1. Adaptive Mask Construction

  • The source score identifies evidence supporting the current prediction, while the difference score selects evidence supporting the source more strongly than the target.The selected relevance score is converted to a binary mask whose support defines the editable region.Source: §4.1 · p. 3
  • The mask selects the top r_k fraction of relevance values, and increasing r_k, optionally with dilation, makes the mask less restrictive.A small editable region favors preservation but may not permit the target transition.Source: §4.1 · p. 3
  • MACE tests candidates along an expanding mask sequence, accepts the first target-reaching candidate, and selects the smallest successful mask.If all attempts fail, it chooses the candidate with the largest target-versus-strongest-competitor margin.Source: §4.1 · p. 4

4.2. CLIP-Guided Diffusion Inpainting

  • The latent diffusion inpainting model conditions denoising on the target prompt, projected mask, and masked-image latent, starting with independent noise inside the mask and the noised input outside.The standard Gaussian noise is sampled once and reused across timesteps.Source: §4.2 · p. 4
  • CLIP guidance minimizes the source–target score difference using gradients restricted to editable latent locations; protected locations follow the noised input trajectory.The guidance is evaluated on a composite using the clean-latent estimate inside the mask and the original image elsewhere.Source: §4.2 · p. 4

5. Experimental Setup

The evaluation uses four datasets and source–target pairs selected from images that CLIP initially classifies correctly. It assesses validity, proximity, and realism, with a controlled Stable Diffusion-only baseline and specified implementation settings.

5.1. Datasets

  • The evaluation covers ImageNet-1k, Food-101, Oxford-IIIT Pet, and CUB-200-2011, selecting images CLIP ranks correctly and assigning each the highest-scoring nonsource class as target.This target is rank 2 because the source class is ranked first.Source: §5.1 · p. 4
  • The main evaluation contains 4,726 counterfactual tasks across the four datasets.The datasets contribute 934 ImageNet, 1,003 Food-101, 1,194 Oxford-IIIT Pet, and 1,595 CUB-200-2011 pairs.Source: §5.1 · p. 5

5.2. Evaluation Metrics

  • Validity is target top-1 success rate; proximity includes normalized ℓ1 and ℓ2 distances, LPIPS, and SSIM, while realism uses FID and KID.Lower ℓp distances and LPIPS indicate smaller changes, higher SSIM indicates greater structural similarity, and lower FID and KID are better.Source: §5.2 · p. 5

5.3. Implementation and Configuration

  • The implementation uses OpenAI CLIP ViT-B/16 and CAV attribution, starting with the top 25% of attribution values and increasing the fraction by 5% for up to 20 attempts.It retains the smallest successful mask or, if none succeeds, the candidate with the largest target-versus-strongest-alternative margin.Source: §5.3 · p. 5

5.4. Baseline

  • The Stable Diffusion-only baseline uses the same inpainting checkpoint and generation settings but edits the full image using diffusion text conditioning alone, without an attribution mask or external CLIP guidance.It uses the same source images, targets, prompts, and evaluation pipeline as MACE.Source: §5.4 · p. 5

6. Results

Across four datasets, the source-mask variant leads in validity, while the difference-mask variant leads in proximity and realism. Qualitative examples show localized preservation by MACE, alongside failures from restrictive masks, persistent source evidence, and missing fine-grained cues.

6.1. Quantitative Comparison

  • Across all four datasets, MACEs achieves the highest validity, while MACEd leads on every reported proximity and realism metric; both beat Stable Diffusion-only on proximity and realism.Proximity is computed on the same set of successful counterfactuals across methods.Source: §6.1 · pp. 5–6, Table 1 · p. 6
  • MACEd's smaller changes and better realism come with lower validity than MACEs, exposing a validity–proximity tradeoff.Its source–target difference mask focuses on regions where source evidence exceeds target evidence.Source: §6.1 · p. 6
  • MACE performs across general, food, pet, and bird classification, while Stable Diffusion-only is weaker on ImageNet and especially CUB-200.The passage suggests full-image generation may capture broad target characteristics more readily than localized cues distinguishing similar bird species.Source: §6.1 · p. 6

6.2. Qualitative Results

  • MACE generally preserves source-image identity, pose, background, and composition while adding target features; Stable Diffusion-only often regenerates the whole scene.Examples include changing sandwich filling while retaining surrounding objects, and adding terrier features while maintaining pose and background.Source: §6.2 · p. 6
  • MACEd often edits smaller source-specific cues than MACEs, but can be too restrictive; the Cedar Waxwing example links this pattern to their validity–proximity tradeoff.MACEd mainly changes crown color, whereas MACEs edits a larger part of the head.Source: §6.2 · p. 6
  • Different masks provide different spatial context to independent inpainting processes, so their shared region can produce different generated content.Source: §6.2 · p. 7
  • Failures arise from masks too small to add target evidence, source evidence that remains after editing, or plausible edits that miss fine-grained target cues.Stable Diffusion-only also fails in examples by generating the wrong object or class, including fries instead of fish and an off-target bird.Source: §6.2 · p. 7

7. Ablation studies

Ablations show that stronger CLIP guidance generally improves validity, while fixed-mask effects vary by dataset. Adaptive expansion improves validity while keeping normalized ℓ1 changes below those of fixed masks at the same maximum fractions.

7. Ablation studies

  • Stronger CLIP guidance generally improves validity across datasets and mask types, with the largest gains for the difference mask.Larger fixed masks strongly improve validity on Food-101, but source-mask validity on CUB-200 does not improve monotonically; each fraction yields an independent candidate.Source: §7 · p. 7
  • Increasing guidance generally lowers normalized ℓ1 distance, with most improvement around scale 0.4 and little thereafter; larger fixed masks cause larger pixel changes.Stronger guidance can introduce visible artifacts that ℓ1 does not capture, and the difference mask remains more conservative as fixed mask fraction increases.Source: §7 · p. 7
  • Adaptive expansion improves validity for both mask types across datasets while maintaining lower normalized ℓ1 distances than fixed masks at the same maximum fractions.It evaluates progressively larger masks and returns the first successful candidate; higher maximum fractions yield larger validity gains.Source: §7 · pp. 7–8, Figure 3 · p. 8, Figure 4 · p. 8

8. Conclusion

MACE targets CLIP zero-shot predictions through localized adaptive editing, and across four datasets its mask variants trade validity against source-image preservation. Its objective does not explicitly require the target to outrank every label.

8. Conclusion

  • MACE combines class-conditioned attribution, adaptive mask expansion, and CLIP-guided diffusion inpainting to change a CLIP prediction while preserving content outside the editable region.Experiments span ImageNet, Food-101, Oxford Pets, and CUB-200.Source: §8 · p. 8
  • The source-attribution variant has the highest validity across four datasets, while the source–target difference variant makes smaller pixel and perceptual changes and earns better realism scores.Both variants improve proximity and realism over full-image Stable Diffusion editing.Source: §8 · p. 8
  • MACE optimizes the pairwise source–target score difference, which does not explicitly require the target to outrank every class in the zero-shot label set.The authors suggest multiclass or margin-based objectives as future alternatives.Source: §8 · p. 8

Related papers

Loading 2609.37638v2…