Source-linked AI summary
PolypSteer: Counterfactual Endoscopic Synthesis via Training-Free Activation Steering
Trong-Thang Pham, Loc Nguyen, Anh Nguyen, Hien V. Nguyen, Ngan Le
TL;DR
Medical diffusion augmentation lacks causal, anatomy-matched counterfactual pairs because re-prompting changes the full trajectory and inversion introduces drift. PolypSteer estimates pathology directions in cross-attention and steers shared-seed generations, outperforming baselines across counterfactual generation, dye disentanglement, and downstream detection.
Problem
Endoscopic pathology detectors need anatomy-matched diseased and healthy pairs, but re-prompting alters non-targeted structure and inversion-based editing introduces reconstruction error and drift.
Method
PolypSteer is a training-free framework that estimates pathology vectors from contrastive prompts and steers cross-attention activations during shared-seed generation.
Results
PolypSteer outperforms baselines on counterfactual generation, dye disentanglement, and downstream polyp detection across Kvasir v3 and HyperKvasir.
Takeaways & Limitations
Shared-noise counterfactual synthesis preserves non-targeted structure and provides utility for downstream clinical detection evaluation.
Abstract
from arXiv · showhide
Generative diffusion models are increasingly used for medical imaging data augmentation, but text prompting cannot produce causal training data. Re-prompting rerolls the entire generation trajectory, altering anatomy, texture, and background. Inversion-based editing methods introduce reconstruction error that causes structural drift. We propose PolypSteer, a training-free activation-steering framework for endoscopic synthesis. PolypSteer identifies a pathology vector for each contrastive prompt pair in the cross-attention layers of a diffusion transformer. At inference time, it steers image activations along this vector, generating counterfactual pairs from scratch where the only difference is the steered concept. All other structure is preserved by construction. We evaluate PolypSteer across three experiments on Kvasir v3 and HyperKvasir. On counterfactual generation across three clinical concept pairs, PolypSteer achieves flip rates of 0.800, 0.925, and 0.950, outperforming the best inversion-based baseline in both concept flip rate and structural preservation. On dye disentanglement, PolypSteer achieves 75% dye removal against 20% (PnP) and 10% (h-Edit). On downstream polyp detection, augmenting with PolypSteer counterfactual pairs achieves ViT AUC of 0.9755 versus 0.9083 for quantity-matched re-prompting, confirming that counterfactual structure drives the gain. Code is at link https://github.com/UARK-AICV/PolypSteer
1 Introduction
PolypSteer addresses the need for anatomy-matched diseased and healthy endoscopic pairs by steering only the targeted pathology concept while preserving non-targeted structure.
- Pathology detectors need anatomy-matched diseased and healthy images to learn disease-specific features rather than confounding anatomy.
- Re-prompting rerolls the entire diffusion trajectory, changing anatomy, texture, and background between outputs.
- DDIM inversion introduces reconstruction error and structural drift, while mask-based editing additionally requires region annotations.
- PolypSteer estimates pathology vectors from contrastive prompts in frozen cross-attention layers and steers activations without fine-tuning, source images, or annotations.
- Shared noise trajectories make non-targeted concept structure identical by construction, and per-token steering provides spatial interpretability.
2 Related Work
Prior medical diffusion editing methods generally generate, invert, and edit, whereas PolypSteer avoids inversion and applies activation steering to preserve non-targeted structure.
- Medical counterfactual methods commonly follow a generate → invert → edit pipeline, inheriting approximate inversion and residual background drift.
- PolypSteer generates both images from the same noise seed, guaranteeing preservation of non-targeted structure.
- Concept-control methods use semantically meaningful activation directions, but several alternatives require extensive training, weight fine-tuning, or LoRA adapters.
- PolypSteer’s pipeline estimates pathology vectors offline and applies spatially selective cross-attention steering during inference.
- The medical setting remains challenging because class labels entangle visual attributes and utility requires downstream clinical validation beyond perceptual metrics.
3 Methodology
PolypSteer estimates concept-specific pathology directions from contrastive prompts and uses spatially selective cross-attention intervention to generate minimal-edit counterfactuals.
- 3.1 Backbone and Intervention Point: The method intervenes on cross-attention outputs because they convey textual semantics to image tokens, while anatomy is encoded elsewhere.
- 3.2 Pathology Vector Estimation: Contrastive prompt pairs differ only in the target pathology, while varied contexts and seeds marginalize texture and viewpoint variation.
- 3.2 Pathology Vector Estimation: Averaged positive-negative cross-attention features are L2-normalized into a unit pathology vector for each layer and denoising timestep.
- 3.2 Pathology Vector Estimation: Per-timestep vectors capture changing semantic representations from global structure in early denoising to fine details later.
- 3.3 Spatially Selective Pathology Steering (SSPS): Shared-seed branches generate counterfactual pairs from scratch, so structural differences arise solely from the SSPS intervention.
- 3.3 Spatially Selective Pathology Steering (SSPS): The cosine-similarity gate selectively modifies positively aligned tokens, subtracting the pathology component while leaving orthogonal anatomy and texture components unchanged.
4 Experiments
PolypSteer is evaluated for downstream detection, counterfactual generation, dye disentanglement, and steering behavior across Kvasir v3 and HyperKvasir. It consistently supports concept changes while preserving non-targeted structure, with performance depending on the chosen layer window and steering strength.
- Downstream Polyp Detection: PolypSteer achieves the best AUC under both ConvNeXt and ViT backbones when each augmentation method adds 1,000 synthetic images.Evaluation removes exact and near-duplicate HyperKvasir matches before testing.
- Counterfactual Generation Across Clinical Concepts: 0.800 flip rate is achieved for Polyp →Normal Cecum, alongside ∆p=0.721 and Bg-LPIPS=0.1449, with the best Bg-SSIM and Bg-PSNR.Pairs Ulcerative Colitis →Normal Cecum and Esophagitis →Normal Z-line reach flip rates of 0.925 and 0.950.
- Ablation Studies: Per-token cosine similarity maps show steering footprints that are broad early in diffusion and contract to sparse patches near the final steps.The maps provide spatial interpretability for where steering occurs at each diffusion step.
5 Conclusion
PolypSteer is a training-free activation-steering framework that extracts pathology vectors from contrastive prompts and applies them in a frozen diffusion transformer. Across downstream detection, counterfactual generation, and dye disentanglement, it outperforms baselines while providing per-token spatial interpretability.
- Conclusion: PolypSteer extracts pathology vectors from contrastive prompt pairs in cross-attention space and steers a frozen diffusion transformer without source images, inversion, masks, or retraining.Counterfactuals are generated from scratch through cosine-similarity-based activation steering.
- Conclusion: PolypSteer outperforms baselines on downstream polyp detection, counterfactual generation, and dye disentanglement across Kvasir v3 and HyperKvasir.Per-token cosine similarity also provides built-in spatial interpretability at each diffusion step.