Source-linked AI summary
When the Edit Changes the Patient: Measuring Identity Preservation in Counterfactual Retinal Images
Andrea Posada, Wenke Karbole, Bach Ngoc Doan, Alexander Weers, Solmaz Abdolrahimzadeh, Maria Patsiamanidi, Kahkashan Haider, Vaishali Khare, Daniel Rueckert, Andrew Lotery, Sobha Sivaprasad, Martin J. Menten
TL;DR
Counterfactual medical image editing must alter specified attributes without changing patient identity, but existing methods usually assume identity preservation rather than measuring it directly. This paper evaluates three text-conditioned OCT editing groups with referee classifiers, embedding alignment, and blinded readers, finding comparable realism and editing success but markedly different identity preservation. It concludes that identity preservation should be explicitly measured and reported, especially for modalities with subtle biometric cues.
Problem
Existing counterfactual editing methods rarely measure identity preservation directly, despite retinal OCT encoding identity in subtle biometric features.
Method
The study compares source-anchored, structured-prompt, and paired-training OCT editors using referee classifiers, embedding alignment metrics, and a blind reader study.
Results
All three methods produce high-quality OCT images with comparable editing success, but identity preservation differs markedly and can be lost despite realistic, successful edits.
Takeaways & Limitations
Identity preservation should be explicitly measured and reported alongside realism and editing success in medical counterfactual generation, particularly for retinal OCT.
Takeaways & Limitations
The referee classifiers were trained, validated, and tested on the same data splits as the generative editing models to avoid data leakage.
Abstract
from arXiv · showhide
Counterfactual medical image generation aims to modify an existing image to reflect a hypothetical scenario in which certain characteristics of the imaged subject are altered, while keeping their identity fixed. Most existing works repurpose established image editing methods, which do not directly supervise identity preservation. Instead, they assume that identity is implicitly preserved by anchoring generation to the source image. This assumption is rarely tested and may fail in domains where biometric cues are subtle, such as retinal optical coherence tomography (OCT). In this work, we explicitly measure identity preservation for three groups of text-conditioned editing methods - source-anchored, structured-prompt, and paired-training - using referee classifiers, embedding alignment scores, and a blind reader study. We find that all methods produce high-quality OCT images with comparable editing success, yet their identity preservation differs markedly. Source-anchored editing frequently alters the depicted subject, while paired-training preserves it best. We argue that future work on medical counterfactual generation must explicitly measure and report identity preservation alongside image realism and editing success.
1 Introduction
Counterfactual OCT generation must change disease-related attributes while preserving the subject’s identity. The paper tests whether existing editing approaches actually preserve identity when retinal biometric cues are subtle.
- Counterfactual medical images must be realistic, anatomically accurate, and identity-preserving to support disease-progression visualization and treatment-effect simulation.
- Existing methods generally assume identity is preserved implicitly through source-image anchoring or paired-image training rather than directly supervising it.
- Retinal OCT poses a harder identity-preservation problem than chest X-rays because its biometric cues are more subtle.
- The study compares source-anchored, structured-prompt, and paired-training editing groups on a longitudinal OCT dataset of AMD subjects.
- Identity preservation is evaluated with referee classifiers, embedding alignment between generated edits and real longitudinal changes, and a blind reader study.
- Identity preservation cannot be assumed: realistic edits that achieve the intended change may still alter the depicted subject.
2 Related Work
Related work largely adapts text-conditioned diffusion editing methods and evaluates image change without directly measuring whether the patient remains the same. The paper distinguishes editing paradigms and identifies identity preservation as an overlooked evaluation concern.
- Text-conditioned counterfactual methods include source-anchored, structured-prompt, text-and-mask-conditioned, and paired-training approaches.
- Source-anchored methods preserve identity through proximity to the source image, while SDEdit trades control for training-free, backbone-agnostic editing.
- Structured-prompt methods manipulate word-level attention maps between closely aligned source and target prompts, relying on implicit identity preservation.
- Text-and-mask-conditioned methods preserve structures outside a mask but require accurate masks and cannot model global edits such as aging, sex, or diffuse pathology.
- Paired-training methods learn counterfactual edits from same-subject image pairs, but paired medical imaging data is not readily available.
- Pixel, structural-similarity, and perceptual-distance metrics conflate identity with edit minimality, making them insensitive to identity-changing small edits and over-penalizing large identity-consistent edits.
3 Method
The method evaluates three OCT editing strategies with complementary identity, alignment, realism, and editing-success measures, then compares edit-focused and identity-aware model selection. Balanced selection incorporates identity metrics alongside realism and editing success.
- Evaluation design: The study compares source-anchored, structured-prompt, and paired-training strategies using referee networks, embedding-alignment metrics, and a blind reader study.
- Dataset: The dataset contains 43 165 OCT images from 6 157 AMD patient eyes, with longitudinal scans spanning up to seven years.
- Counterfactual setup: Generated counterfactual x_g is produced from real image x_s under text prompt p_t, with real target image x_t optionally serving as one valid counterfactual.
- Identity evaluation: Identity preservation is assessed by comparing real/edited or real/real image pairs using two referee classifiers and a blind reader study.
- Referee classifiers: The same-eye Siamese verifier predicts subject and laterality agreement, while the sex classifier provides a weaker complementary identity proxy.
- Evaluation controls: The classifiers use matched generative-model data splits to avoid data leakage, and the same-eye verifier has an F1-score of 0.97.
- Reader and embedding measures: Four clinically trained readers assess 240 blinded image pairs each, while embedding alignment measures semantic agreement with text prompts and real longitudinal transitions.
- Model selection: Edit-focused selection ranks realism and editing success, whereas balanced selection additionally includes sex agreement, eye agreement, and image change alignment.
4 Results and Discussion
Identity preservation declines as editing strength increases, and selecting models using balanced criteria can improve preservation with only marginal changes to realism and editing success. The strongest identity losses occur for edit-focused SDEdit, while InstructPix2Pix and P2P are more robust to editing-strength variation.
- For two of three methods, balanced selection favors lower editing strength than edit-focused selection, with only marginal effects on realism and editing success.The balanced criterion improves eye-identity and sex preservation according to referee classifiers.
- The edit-focused SDEdit configuration shows the most severe identity loss and is more often judged to depict different eyes than the same eyes.Reader performance for P2P and InstructPix2Pix is comparable to performance on real image pairs.
- Identity preservation metrics show a clear negative correlation with editing strength across all methods.The F1-score remains roughly constant overall because most transitions are subclinical, while restricted AMD-stage changes show the expected upward F1 trend.
- Lower editing strength keeps generated images closer to the real-data distribution, consistent with the observed FID trend.Editing-strength values are method-specific and therefore are not directly comparable across methods.
- InstructPix2Pix and P2P show high robustness to editing-strength variations within their respective ranges.This aligns with their strongest implicit identity signal in the preceding analysis.
5 Conclusion
The study finds that high-quality retinal OCT counterfactuals with comparable editing success can still alter subtle eye biometrics. It concludes that identity preservation should be explicitly evaluated and incorporated into model selection alongside realism and editing success.
- All three methods produce high-quality OCT images with comparable editing success, but some alter subtle biometric features of the eye.
- Evaluating models only by realism and editing success does not reliably capture identity changes, yielding invalid counterfactuals.
- Explicitly accounting for identity preservation during model selection and tuning can substantially improve it with minimal deductions in realism and editing success.
- Future medical counterfactual-generation work should measure and report identity preservation alongside realism and editing success, particularly for retinal OCT with subtle identity cues.