Source-linked AI summary
BCI: Breast Cancer Immunohistochemical Image Generation through Pyramid Pix2pix
Shengjie Liu, Chuang Zhu, Feng Xu, Xinyu Jia, Zhongyue Shi, Mulan Jin
TL;DR
HER2 evaluation is clinically important but expensive because it requires additional IHC staining, motivating HE-to-IHC synthesis. The paper introduces the paired BCI benchmark and a pyramid pix2pix method, which performs better than other pix2pix-like methods on BCI while the mapping remains challenging.
Problem
HER2 expression must be evaluated for precise breast-cancer treatment, but routine IHC assessment is expensive and no public HE-to-IHC HER2 translation dataset was available.
Method
The paper introduces BCI, a paired HE-to-IHC dataset, and pyramid pix2pix, which applies multi-scale constraints to generate IHC images from HE images.
Results
Pyramid pix2pix achieves better HE-to-IHC translation results than other pix2pix-like methods on BCI, especially for low HER2 expression, while high-expression areas remain difficult to identify.
Takeaways & Limitations
BCI establishes a benchmark for HER2-focused pathology image translation and supports future study of HER2 evaluation using synthesized IHC images.
Takeaways & Limitations
Accurate HE-to-IHC mapping remains challenging, and the paper leaves comparison with real-IHC HER2 evaluation and clinical treatment planning for future work.
Abstract
from arXiv · showhide
The evaluation of human epidermal growth factor receptor 2 (HER2) expression is essential to formulate a precise treatment for breast cancer. The routine evaluation of HER2 is conducted with immunohistochemical techniques (IHC), which is very expensive. Therefore, for the first time, we propose a breast cancer immunohistochemical (BCI) benchmark attempting to synthesize IHC data directly with the paired hematoxylin and eosin (HE) stained images. The dataset contains 4870 registered image pairs, covering a variety of HER2 expression levels. Based on BCI, as a minor contribution, we further build a pyramid pix2pix image generation method, which achieves better HE to IHC translation results than the other current popular algorithms. Extensive experiments demonstrate that BCI poses new challenges to the existing image translation research. Besides, BCI also opens the door for future pathology studies in HER2 expression evaluation based on the synthesized IHC images. BCI dataset can be downloaded from https://bupt-ai-cz.github.io/BCI.
1. Introduction
The paper frames HER2 evaluation as clinically important but costly because it requires additional IHC-stained slices, and introduces BCI to support HE-to-IHC synthesis with a pyramid pix2pix method.
- Clinical motivation: HER2 expression guides breast-cancer treatment, but routine IHC evaluation is expensive because it requires preparing an additional stained slice.The paper distinguishes four HER2 expression categories from IHC 0 through IHC 3+.
- Task formulation: The proposed task is to synthesize IHC-stained images directly from paired HE-stained images using image-to-image translation.This could support HER2 evaluation from synthesized IHC slices, within the scope proposed by the paper.
- Dataset and task: BCI addresses the lack of public, structurally aligned HE-IHC datasets for HER2 detection by providing paired breast-tissue images for translation research.The paper identifies aligned HE-IHC acquisition as difficult and says no public HER2 image-translation dataset previously existed.
- Proposed method: Pyramid pix2pix constrains generated images at multiple scales and achieves better BCI results than other pix2pix-like methods.The paper also reports experiments exploring the gains associated with different scales and multi-scale constraints.
2. Related Work
Related work covers supervised and unsupervised image translation, medical applications, and paired or unpaired datasets across domains. It positions aligned datasets and supervised methods as important when accurate domain mappings are needed.
- Unsupervised translation: Unsupervised translation avoids aligned training pairs and supports broader application, but may fail to establish accurate mappings when paired data are available.The paper notes that additional patch labels can improve paired-data mapping but require substantial human effort.
- Supervised translation: Supervised translation uses paired images; pix2pix combines adversarial and pixel-level losses, while pix2pixHD improves the generator for high-resolution synthesis.The related work also describes EPDN as using multi-resolution generation, multi-scale discrimination, and an enhancer.
- Medical applications: Medical image translation includes methods such as RegGAN, which adds a registration network to support both paired and unpaired data.The paper also mentions pathological-image translation from non-standard stained sections to standard staining.
- Datasets: Image-translation datasets are commonly paired or unpaired and span semantic synthesis, dehazing, face generation, multimodal medical imaging, and visible-infrared translation.Examples include Cityscapes, LLVIP, BraTS, CelebAMask-HQ, FFHQ-Aging, Selfie2anime, AFHQ, and RaFD.
3. BCI Dataset
BCI is introduced as a structural-aligned HE-to-IHC dataset for HER2 image translation. Its construction combines projection alignment, blockwise elastix registration, patch extraction, and filtering of unsuitable regions.
- 3. BCI Dataset: The dataset addresses the lack of pathological HE-to-IHC translation data by providing paired, structurally aligned images for HER2 expression research.
- 3.2. Registration: Projection transformation first aligns HE and IHC tissue contours using manually selected corresponding points.
- 3.2. Registration: Elastix then performs fine-grained regional non-rigid registration on 16 blocks per whole-slide image before re-splicing the registered regions.
- 3.2. Registration: The registration pipeline removes block-edge artifacts, fills borders, cuts registered slides into 1024×1024 patches, and filters blank or poorly aligned areas.
- 3. BCI Dataset: BCI contains 4870 paired 1024×1024 pathological image patches from 51 patients, spanning IHC categories 0, 1+, 2+, and 3+.
4. Proposed Method
The proposed pyramid pix2pix method relaxes pixel-level constraints by matching generated and target images across multiple Gaussian-smoothed scales. Its objective combines conditional adversarial, original-resolution L1, and multi-scale losses.
- 4.1. Architecture: Pyramid pix2pix applies Gaussian filtering and downsampling across octaves so generated images can match targets at multiple resolutions.Each octave contains multiple Gaussian-convolution layers, and successive octaves are formed by downsampling.
- 4.2. Objective Function: The method retains the original-resolution L1 loss while adding higher-scale constraints to reduce its restrictiveness for BCI image pairs.
- 4.1. Architecture: The multi-scale loss compares Gaussian-filtered generated and ground-truth images, with λ_i weighting each scale.
- 4.2. Objective Function: The overall objective combines conditional GAN loss, λ_1-weighted L1 loss, and the multi-scale loss.
5. Experiments
Experiments compare image-translation methods on BCI and LLVIP using PSNR, SSIM, visual assessment, and multi-scale analyses. Pyramid pix2pix improves authenticity over pix2pix variants and generalizes across datasets, but high HER2 expression remains difficult to identify.
- 5.2. Metrics: Evaluation uses PSNR for pixel error and SSIM for brightness, contrast, and structural similarity, with SSIM intended to better reflect human visual assessment.
- 5.3. Benchmark Results: CycleGAN cannot establish accurate HE-to-IHC mappings on registered pairs, while supervised pix2pix variants better stain cancerous regions but remain visually inaccurate.
- 5.3. Benchmark Results: Pix2pixHD slightly improves overall high-resolution quality and PSNR/SSIM over pix2pix but can generate incorrect dark-brown staining in low-HER2 regions.
- 5.3. Benchmark Results: Pyramid pix2pix produces more authentic BCI translations than pix2pix and pix2pixHD, although it still cannot identify high HER2 expression areas reliably.
- 5.3. Benchmark Results: On LLVIP, pyramid pix2pix achieves the best PSNR and SSIM, indicating versatility beyond pathological image translation.
- 5.4. Multi-scale Analysis: On BCI, the two-layer pyramid model pix2pix+S1 achieves the highest PSNR and SSIM among tested pyramid levels.
6. Conclusion
BCI introduces a new HE-to-IHC pathology translation dataset, while pyramid pix2pix targets the task’s structural and HER2-expression requirements. Accurate HE-to-IHC mapping remains challenging, motivating further method development and clinical validation.
- BCI introduces a dataset for translating HE-stained breast tissue sections into their IHC results.
- Pyramid pix2pix is designed for registered image pairs and targets accurate HER2 expression-area and expression-level identification while preserving generated-image authenticity.
- Accurate mapping from HE to IHC remains challenging, and more effective translation methods are still needed.