PaperlayerView exact revision on arXiv

Source-linked AI summary

CLEAR: Cooperative Latent Flow for Visual Counterfactual Explanations with Accurate Region Preservation

Yan Zeng, Changlu Guo, Oskar Kristoffersen, Anders Nymark Christensen, Morten Rieger Hannemose, Anders Bjorholm Dahl

arXiv:2610.23577v1cs.CV

TL;DR

Visual counterfactual methods face a trade-off between image-space fidelity and latent-space efficiency, alongside challenges from conflicting guidance and reconstruction changes. CLEAR uses latent Flow Matching to regulate guidance and preserve factual content outside editable regions. Across CelebA-HQ and ImageNet, it achieves high counterfactual quality while retaining latent-generation efficiency.

  • Problem

    Image-space visual counterfactual methods are computationally costly, while latent-space methods are more efficient but often less faithful; guidance conflicts and VAE reconstruction can also affect generation and unchanged regions.

    Source: §1 · pp. 1–2

  • Method

    CLEAR uses Flow Matching’s instantaneous velocity to preserve cooperative guidance and attenuate conflicting guidance, while constraining edits to dynamically identified regions and preserving factual content outside them.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Across CelebA-HQ and ImageNet, CLEAR achieves high counterfactual quality while retaining the efficiency of latent generation.

    Source: §5 · p. 9

  • Takeaways & Limitations

    CLEAR combines conflict-aware guidance with spatial preservation to address guidance conflicts and reconstruction leakage in latent visual counterfactual generation.

    Source: §5 · pp. 8–9

    The evaluation covers a limited range of counterfactual tasks, and quantitative ablations focus on CelebA-HQ Smile.

    Source: §5 · p. 9

Abstract

from arXiv · show

Visual counterfactual explanations (VCEs) should alter a classifier's prediction while remaining realistic and preserving decision-irrelevant content. Image-space methods provide high-quality counterfactuals but are computationally costly, whereas latent-space methods are more efficient but often less faithful. We identify two sources responsible for this gap. First, latent generation recovers a natural sample from a perturbed representation, whereas classifier guidance pushes it toward a target class, so their update directions may oppose. Second, VAE reconstruction can alter regions that should remain unchanged. We introduce CLEAR, a latent Flow Matching framework for VCEs. CLEAR uses Flow Matching's explicit instantaneous velocity as a reference for the generative direction to decompose classifier guidance into cooperative and conflicting components. It preserves the cooperative component and adaptively attenuates only the conflicting one according to conflict strength. We further analyze conflict frequency and strength throughout generation to reveal how classifier guidance interacts with the generative process. CLEAR also dynamically identifies editable regions, constrains latent modifications, and directly preserves factual content outside them to reduce VAE-induced changes. Across five bidirectional tasks on two datasets, CLEAR retains latent-space efficiency while achieving counterfactual quality comparable to or better than strong image-space baselines, with a 2.5x speedup over the strongest baseline. Project page: https://yan-zen9.github.io/CLEAR/

1. Introduction

Visual counterfactual explanations seek changes that alter a classifier’s prediction while remaining realistic and preserving unrelated content. CLEAR addresses latent guidance conflicts and reconstruction changes while aiming to retain latent-space efficiency and counterfactual fidelity.

1. Introduction

  • Visual counterfactual explanations show what image changes would alter a classifier’s prediction, unlike attribution methods that highlight potentially important regions.The cited framing of the question is “what meaningful change to an input image would alter a classifier’s prediction?”Source: §1 · p. 1
  • Image-space methods can produce high-quality counterfactuals but incur substantial computational and memory costs, while latent methods are efficient but often less faithful.A reliable counterfactual should change the prediction while remaining realistic and preserving content unrelated to the target decision.Source: §1 · p. 1
  • Latent generation seeks a natural sample while classifier guidance pushes toward a target class, so their update directions can oppose during counterfactual generation.The guidance signal can contain both cooperative and conflicting components, so treating it uniformly may retain harmful conflict or discard useful information.Source: §1 · p. 1
  • CLEAR uses Flow Matching’s instantaneous velocity to separate cooperative from conflicting classifier guidance, preserving the former and adaptively attenuating the latter according to conflict strength.The paper also analyzes conflict frequency, strength, and effects throughout generation.Source: §1 · p. 2
  • CLEAR improves latent visual counterfactual generation by addressing guidance conflicts and reconstruction leakage; across two CelebA-HQ attributes and three ImageNet class pairs, it improves FID and sFID over evaluated latent-space baselines.On CelebA-HQ Smile, it demonstrates a favorable quality–efficiency trade-off against image-space methods.Source: §1 · p. 2
  • CLEAR dynamically identifies editable regions, constrains latent modifications, and preserves factual image content outside the editable support to prevent reconstruction and editing leakage there.This addresses the possibility that VAE reconstruction changes regions that should remain unchanged.Source: §1 · p. 2

2. Related Work

Prior work spans image-space diffusion methods and latent visual counterfactual approaches, alongside work on classifier guidance and flow matching. CLEAR differentiates itself by relating external classifier guidance to the current latent generative direction while preserving decision-irrelevant regions.

Visual counterfactual explanations and image-space generation

  • Earlier visual counterfactual methods used region-based, generative, and semantic formulations, with later work investigating fine-grained localization and classifier-faithful explanations.Diffusion models later became a foundation for high-quality visual counterfactual explanations.Source: §2 · p. 2
  • Image-space diffusion methods generate high-quality counterfactuals, but generation and classifier guidance remain primarily in high-dimensional image space, motivating latent-space alternatives.DiME uses iterative image-space sampling and gradient computation; FastDiME reduces that burden, while MaskDiME adds adaptive masking.Source: §2 · p. 2

Latent visual counterfactual explanations

  • LDCE filters classifier gradients by agreement with the diffusion model’s implicit classifier, while ECED localizes edits and reduces reconstruction changes through background-latent optimization and per-input VAE fine-tuning.The passage distinguishes these approaches from CLEAR’s joint treatment of guidance conflict and efficient factual-content preservation.Source: §2 · p. 2

Classifier guidance and flow matching

  • Flow Matching provides an explicit instantaneous velocity field, while prior adaptive guidance schemes mainly target general image generation or classifier-free guidance rather than component-wise external guidance in latent VCEs.LeapFactual applies conditional flow matching to model-agnostic counterfactual generation; CLEAR uses the latent flow direction to selectively modulate conflicting guidance while preserving decision-irrelevant regions.Source: §2 · pp. 2–3

3. Method

CLEAR evolves a partially noised factual latent toward a counterfactual, using Flow Matching velocity to reconcile classifier guidance and adaptive support to constrain edits. It preserves factual image content outside the final editable support.

3.1. Latent Flow Setup and Classifier View

  • CLEAR encodes the factual image with a frozen VAE, starts from a partially noised latent at an intermediate time, and follows a Flow Matching trajectory toward the clean endpoint.At each timestep, the model predicts the generative velocity and decodes a one-step image estimate for classifier feedback.Source: §3.1 · p. 3
  • The current editable support defines the classifier view used to obtain feedback focused on the editable image region.The image-space support is mapped to latent resolution to constrain classifier-driven modifications.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Cooperative Flow Guidance

  • CLEAR attenuates only the classifier force opposing the generative flow, preserving cooperative and non-conflicting guidance.The attenuation increases with directional conflict, and the reconciled guidance is bounded to prevent it from overwhelming the flow.Source: §3.2 · p. 3

3.3. Adaptive Support Preservation

  • The editable support is dynamically updated from target-class Grad-CAM and mapped to a soft latent gate that constrains classifier-driven changes.The initial support comes from the factual image; subsequent supports use the current classifier view.Source: §3.3 · p. 4
  • CLEAR composes the final output to preserve factual pixels outside the final support, preventing reconstruction changes there.Within the support, spatially constrained latent editing is combined with the decoded final latent.Source: §3.3 · p. 4

4. Experiments

Experiments compare CLEAR’s inference efficiency and counterfactual quality across facial edits and ImageNet class pairs, then analyze directional conflict and ablate its main components. Results show strong quality with latent-space efficiency, while conflict frequency and strength vary across tasks and generation stages.

4.1. Experimental Setup

  • The evaluation uses CelebA-HQ Smile and Age with DenseNet-121 and three ImageNet class pairs with ResNet-50, assessing efficiency, validity, realism, preservation, and semantic similarity.Both datasets use 256 × 256 images; pretrained models remain frozen, and CLEAR requires no additional training or fine-tuning.Source: §4.1 · pp. 4–5

4.2. Computational Efficiency

  • 2.5× and 58.1× speedups over MaskDiME and DiME, respectively; CLEAR also achieves the lowest runtime and FID among compared methods.The comparison uses inference runtime, peak GPU memory, and counterfactual quality on CelebA-HQ Smile.Source: §4.2 · p. 5, Figure 4 · p. 6

4.3. Counterfactual Quality

  • CLEAR achieves the best FID and sFID on CelebA-HQ Smile and Age, and outperforms latent methods in FID and sFID across all three ImageNet class pairs while remaining competitive with image-space baselines.On CelebA-HQ, it maintains near-perfect validity and strong identity preservation; on ImageNet, it achieves best or tied-best S3 with near-perfect flip rates.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 7

4.4. Directional Conflict Analysis

  • Conflict strength follows task-specific trajectories: ImageNet transformations have strong early conflicts that rapidly decrease, while CelebA-HQ Smile and Age edits peak at intermediate stages; individual conflicts generally weaken toward generation’s end.Hard Projection retains 0% and Additive Guidance retains 100% of the conflicting component, whereas CLEAR adaptively retains 83.3–88.8% across five tasks.Source: §4.4 · p. 7

4.5. Ablation Studies

  • Ablations show that fully retaining conflicting guidance degrades realism and non-target preservation, while removing it entirely reduces validity; removing any adaptive-support component degrades overall performance.Without pixel reuse, background MSE outside the editable support is non-zero; factual-pixel composition guarantees zero MSE there but does not independently validate support accuracy or boundary continuity.Source: §4.5 · pp. 7–8, Table 3 · p. 8

5. Conclusion and Limitations

CLEAR addresses conflicts between classifier guidance and the generative flow, as well as reconstruction leakage beyond editable regions. The evaluation is limited in task range, with quantitative ablations focused on CelebA-HQ Smile.

Conclusion

  • CLEAR combines Cooperative Flow Guidance with Adaptive Support Preservation to address directional conflict and reconstruction leakage while retaining latent-generation efficiency.Its experiments cover CelebA-HQ and ImageNet.Source: §5 · pp. 8–9

Limitations

  • The evaluation covers a limited range of counterfactual tasks, and quantitative ablation studies focus on CelebA-HQ Smile.The authors identify more diverse task evaluations and ablations as future work.Source: §5 · p. 9

A.1. Additional Qualitative Results under Different Conflict Regimes

Across different conflict regimes, CLEAR adaptively attenuates conflicting guidance according to its strength, while additional examples illustrate exact preservation outside the editable support.

Conflict regimes

  • Stronger conflicts produce more visible differences between guidance strategies, while weak conflicts yield relatively small differences.Additive Guidance can overmodify images, whereas Hard Projection may suppress useful classifier guidance; CLEAR adaptively attenuates conflict strength.Source: §A.1 · p. 12

Preservation examples

  • Pixels outside the editable support remain unchanged in the illustrated CLEAR counterfactuals.Figure S3 shows the factual image, counterfactual, and pixel-wise difference map.Source: Figure S3 · p. 12

A.2. Adaptive Support Preservation

Adaptive Support Preservation restricts classifier-driven latent edits to a dynamically estimated support and reuses factual pixels outside the final support to preserve them exactly.

Adaptive Support Preservation

  • ASP restricts classifier-driven updates to a dynamically estimated editable support and reuses factual pixels outside the final support.This image-level reuse prevents VAE reconstruction and latent editing leakage from changing pixels outside the estimated support.Source: §A.2 · p. 12

Editable support

  • The illustrated modifications closely follow identified target-relevant regions, localizing counterfactual edits.Restricting the latent trajectory alone does not necessarily prevent VAE decoder reconstruction changes outside the intended region.Source: §A.2 · pp. 12–13

Exact preservation

  • Factual pixels outside the final editable support are exactly preserved, though this does not establish that the support correctly separates relevant from irrelevant content.The default configuration uses τ = 0.5, ts = 0.6, N = 20, s = 8, and ρ = 5%.Source: §A.2 · p. 13, Table S1 · p. 13

A.3. Hyperparameter Sensitivity

Hyperparameter sensitivity on CelebA-HQ Smile shows trade-offs among validity, realism, and preservation, while the default configuration provides a balance across these objectives.

Conflict threshold τ

  • τ = 0.25 reduces flip rate from 99.6% to 95.2%, while τ = 1.0 reaches 99.7% flip rate but raises FID and MNAC to 4.28 and 1.71.The authors favor moderate conflict-dependent attenuation over either aggressive suppression or largely retaining conflicting guidance.Source: §A.3 · p. 13

Starting time ts

  • Starting at ts = 0.4 yields FR = 99.9% and COUT = 0.74 but worsens FID and MNAC, whereas ts = 0.8 reduces flip rate to 86.4%.The default ts = 0.6 balances editing freedom and factual preservation.Source: §A.3 · p. 13

Sampling steps N

  • N = 10 reduces flip rate to 90.1%, while N = 40 restores high validity but does not improve realism or preservation over the default N = 20.Additional steps provide limited benefit once sufficient guidance has been applied.Source: §A.3 · p. 13

Guidance scale s

  • With s = 4, COUT = 0.44 and FR = 93.5%, while s = 12 gives COUT = 0.72 and FR = 99.7% but degrades FID and MNAC.The guidance scale controls maximum classifier-guidance strength.Source: §A.3 · p. 13

Support ratio ρ

  • ρ = 3% reduces FID and MNAC but lowers flip rate to 88.0%, whereas ρ = 8% restores high validity with slightly higher FID and MNAC.The default ρ = 5% balances localized editing and successful counterfactual generation.Source: §A.3 · p. 13

Overall sensitivity

  • The default configuration provides a consistent balance between validity, realism, and preservation.The observed trends align with Cooperative Flow Guidance controlling how strongly classifier information modifies the trajectory and ASP controlling where edits are allowed.Source: §A.3 · p. 13

Related papers

Loading 2610.23577v1…