Source-linked AI summary
Distribution Matching Losses Can Hallucinate Features in Medical Image Translation
Joseph Paul Cohen, Margaux Luck, Sina Honari
TL;DR
The paper examines whether distribution-matching losses preserve medically relevant features during image translation when domain compositions differ. Through MRI translation experiments and objective-function analysis, it finds that translated images can add or remove tumor features, motivating caution when they are used for interpretation.
Problem
Distribution matching may bias medical image translation when target-domain classes are over- or under-represented, potentially altering known or unknown labels relevant to diagnosis.
Method
The paper analyzes GAN, CycleGAN, conditional GAN, and L1 translation objectives and tests Flair-to-T1 MRI translation under varied tumor proportions in paired and unpaired training.
Results
The experiments show that CycleGAN produces the most dramatic class-label changes, while CondGAN remains composition-sensitive and L1 is generally more consistent but fails under some train-test label mismatches.
Takeaways & Limitations
Translated medical images should not be used for interpretation without proper tools to verify the translation process.
Takeaways & Limitations
The paper’s warning concerns images presented for interpretation and does not aim to discourage data augmentation for improving downstream models.
Abstract
from arXiv · showhide
This paper discusses how distribution matching losses, such as those used in CycleGAN, when used to synthesize medical images can lead to mis-diagnosis of medical conditions. It seems appealing to use these new image synthesis methods for translating images from a source to a target domain because they can produce high quality images and some even do not require paired data. However, the basis of how these image translation models work is through matching the translation output to the distribution of the target domain. This can cause an issue when the data provided in the target domain has an over or under representation of some classes (e.g. healthy or sick). When the output of an algorithm is a transformed image there are uncertainties whether all known and unknown class labels have been preserved or changed. Therefore, we recommend that these translated images should not be used for direct interpretation (e.g. by doctors) because they may lead to misdiagnosis of patients based on hallucinated image features by an algorithm that matches a distribution. However there are many recent papers that seem as though this is the goal.
1 Introduction
Distribution-matching image translation can hallucinate or remove medical features when the target-domain composition differs from the source or test distribution. The paper illustrates this risk with CycleGAN translations that alter tumor presence in MRI images and cautions against using such outputs for direct interpretation.
- 1 Introduction: Adversarial translation methods match outputs to a target-domain distribution, which can add or remove image features when known or unknown labels are unevenly represented.The concern applies to paired and unpaired translation approaches used across several medical imaging tasks.
- 1 Introduction: A CycleGAN trained on a target distribution containing only tumors added tumors to translated images, showing that distribution composition can change apparent class labels.The paired examples contrast tumor removal under a tumor-free target with tumor addition under an all-tumor target.
- 1 Introduction: The paper focuses on translated images presented for interpretation, where hallucinated features can contribute to misdiagnosis of medical conditions.The stated concern includes clinical use such as interpretation by doctors.
- 1 Introduction: CycleGAN trained only on healthy T1 samples removed tumors from translated images despite tumors being present in the source Flair images.This caricature example demonstrates matching the target distribution without necessarily preserving source features.
- 1 Introduction: The authors do not aim to discourage using these losses for data augmentation to improve classification, segmentation, or other model performance.The warning is scoped specifically to presenting translated images for interpretation.
2 Problem Statement
The paper argues that source and target-domain composition can bias translation toward unwanted feature hallucination. It examines how distribution matching, cycle consistency, conditional discrimination, and L1 objectives can fail to guarantee correct feature preservation under distribution mismatch or target imbalance.
- 2 Problem Statement: Source and target-domain composition can bias image transformations toward unwanted feature hallucination.The paper frames this as a consequence of the domains’ composition rather than only the model architecture.
- 2 Problem Statement: A standard GAN trains a transformation from source samples toward target samples through a discriminator that guides the output to match the target distribution.The formulation does not itself constrain the mapping to preserve a correct correspondence between domains.
- 2 Problem Statement: CycleGAN adds cycle-consistency reconstruction terms, but hallucinated features or encoded information can still satisfy target matching without guaranteeing a correct mapping.Unpaired target data may represent a distribution unrelated to the source, increasing the risk of hallucination.
- 2 Problem Statement: Conditional adversarial training models the joint source-target distribution and requires paired examples, yet target-feature over- or under-representation can still bias feature translation.Even paired data can cause the discriminator to allocate unequal capacity to features such as tumors.
- 2 Problem Statement: L1 predicts pixels locally without global distribution matching, but train-test distribution differences can still cause bias when test-time labels were absent during training.The paper gives missing tumor pixels during training as an example involving new known or unknown labels at test time.
3 Bias Impact
The experiments vary tumor prevalence in target-domain training data while testing Flair-to-T1 MRI translation. Bias changes with target composition, most dramatically for CycleGAN, whereas L1 is generally more consistent but fails under certain train-test label mismatches.
- 3 Bias Impact: The study uses 1,700 paired BRATS2013 MRI slices, with 50% healthy and 50% tumor images; 1,400 train models and 300 form the holdout test set.The holdout set tests whether translations add or remove tumors.
- 3 Bias Impact: CycleGAN uses unpaired data with a balanced source and target tumor ratios varying from 0% to 100%, while CondGAN and L1 use paired domains with matching proportions.The scenarios isolate the effect of target-domain composition for unpaired and paired training.
- 3 Bias Impact: 80% accuracy on the test set was obtained by the CNN classifier used to assess tumor presence in generated T1 images.The study also measures mean absolute pixel reconstruction error against the ground-truth target images.
- 3 Bias Impact: CycleGAN produced the most dramatic class-label changes as target tumor composition shifted, encouraging tumors to be added or removed from translated images.The result reflects mapping a balanced source domain to an unbalanced target composition.
- 3 Bias Impact: CondGAN pixel error changed with tumor/healthy composition, while L1 appeared least biased but failed when tumor samples were tested after training with 0% tumors.L1 could translate healthy images after training with 100% tumors, which the authors attribute to healthy and tumor regions co-occurring within images.
4 Conclusion
The paper concludes that distribution-matching image translation can alter known and unknown class labels, including adding or removing tumors, and therefore may cause diagnostic errors. Translated images should not be interpreted directly without tools that verify the translation process.
- Experimental evidence shows that distribution matching can fail to preserve known and unknown class labels in translated medical images.The paper illustrates tumors being added and removed in MRI translations.
- Fig. 2 varies target-domain tumor composition from 0% to 100% across CycleGAN, CondGAN, and L1 using 33 trained models.The experiment evaluates classifier predictions on 300 unseen holdout samples, comprising 53% tumor cases.
- For CycleGAN, the percentage of translated images diagnosed with tumors increases as the target distribution contains more tumor images.CondGAN has more stable classification results, while pixel error varies with target-domain composition.
- Future methods should ensure that translation bias does not influence medical-diagnosis outcomes.
Supplementary Information
The supplementary figures visualize translation bias under different target-domain compositions and classifier outcomes. They show that class-preserving predictions can coexist with image artifacts, while incorrect predictions also reveal bias that differs across losses.
- CycleGAN translations can introduce hard-to-spot tumors into healthy images as the target-domain tumor ratio changes.Fig. S1 compares Flair source images with corresponding T1 ground truth images across 11 CycleGAN models.
- Even when a classifier predicts classes correctly across all 33 translations, translated images differ from ground truth and tumors may disappear or enlarge.L1 suffers less from these artifacts, while adversarial losses produce more erroneous results.
- When the classifier predicts classes incorrectly, translation bias remains visible and varies across loss functions.The examples include healthy sources predicted as tumors and tumor sources predicted as healthy; L1 suffers less from artifacts.