PaperlayerView exact revision on arXiv

Source-linked AI summary

Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis

Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, Ling Shao

arXiv:2002.05000v1cs.CVeess.IV

TL;DR

Missing MR modalities are difficult to collect, motivating synthesis from available modalities. Hi-Net combines modality-specific learning with layer-wise multimodal fusion and adaptive fusion weighting, and experiments report better quantitative and qualitative performance than other state-of-the-art methods across multiple synthesis tasks.

  • Problem

    Collecting every MR modality for each patient is challenging, motivating synthesis of missing modalities from existing scans.

  • Method

    Hi-Net learns modality-specific representations, fuses multimodal correlations across feature layers, and uses an MFB to adaptively weight summation, product, and maximization.

  • Results

    Hi-Net outperforms other state-of-the-art synthesis methods in quantitative and qualitative measures across multiple synthesis tasks.

  • Takeaways & Limitations

    The framework can synthesize missing modality images for subsequent multimodal medical-image analysis tasks such as segmentation and survival prediction.

Abstract

from arXiv · show

Magnetic resonance imaging (MRI) is a widely used neuroimaging technique that can provide images of different contrasts (i.e., modalities). Fusing this multi-modal data has proven particularly effective for boosting model performance in many tasks. However, due to poor data quality and frequent patient dropout, collecting all modalities for every patient remains a challenge. Medical image synthesis has been proposed as an effective solution to this, where any missing modalities are synthesized from the existing ones. In this paper, we propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis, which learns a mapping from multi-modal source images (i.e., existing modalities) to target images (i.e., missing modalities). In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality, and a fusion network is employed to learn the common latent representation of multi-modal data. Then, a multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality, acting as a generator to synthesize the target images. Moreover, a layer-wise multi-modal fusion strategy is presented to effectively exploit the correlations among multiple modalities, in which a Mixed Fusion Block (MFB) is proposed to adaptively weight different fusion strategies (i.e., element-wise summation, product, and maximization). Extensive experiments demonstrate that the proposed model outperforms other state-of-the-art medical image synthesis methods.

I. INTRODUCTION

Hi-Net addresses missing MR modalities by synthesizing target images from existing modalities, combining modality-specific representations with cross-modal correlations. Its layer-wise fusion and Mixed Fusion Block adaptively integrate multimodal features, and experiments report superior synthesis performance.

  • Motivation: Missing MR modalities arise from acquisition challenges, while discarding incomplete samples wastes information and worsens small-sample-size issues.Cross-modal synthesis is presented as a way to produce missing modalities without requiring actual scans.
  • Proposed approach: The model learns modality-specific properties, exploits correlations across modalities and feature layers, and generates target images from a learned latent representation.Its fusion strategy integrates multimodal multi-level representations.
  • Proposed approach: Hi-Net synthesizes target modalities by fusing existing multimodal images rather than relying on single-modality input.The framework is explicitly designed for multi-modal medical image synthesis.
  • Proposed approach: The Mixed Fusion Block adaptively weights element-wise summation, product, and maximization to improve fusion performance.These operations are combined within the proposed fusion design.

II. RELATED WORKS

Prior work spans GAN-based cross-modal synthesis, patch-based regression, sparse representation, atlas-based methods, and deep learning. Hi-Net differs by integrating multimodal representations across feature levels rather than focusing only on single-input or high-level fusion.

  • Cross-modal synthesis: GAN-based cross-modal synthesis uses adversarial learning between a generator and discriminator to produce realistic images.Pix2pix is described as reinforcing pixel-to-pixel similarity for paired data.
  • Medical image synthesis: Traditional medical image synthesis includes patch-based regression, regression forests, sparse representation, and atlas-based transformations.These approaches predict target-modality information from source-modality patches or aligned atlases.
  • Medical image synthesis: Deep learning methods have been applied to image-resolution mapping, missing PET estimation from MRI, and GAN-based synthesis.The related work describes these as developments in medical image synthesis.
  • Multimodal learning: Multimodal learning commonly seeks robust common components by exploiting complementarity among features from multiple modalities.This motivates learning shared representations across modalities.
  • Position of Hi-Net: Hi-Net integrates multimodal multi-level representations, distinguishing it from approaches that fuse features only in high-level layers.The framework also contains modality-specific, fusion, generator, and discriminator components.

III. METHODOLOGY

Hi-Net uses modality-specific autoencoder-like networks to learn representations for individual MR modalities, while a layer-wise fusion network combines their complementary information. Reconstruction supervision helps preserve discriminative modality-specific features.

  • Framework: Hi-Net comprises modality-specific networks, a multimodal fusion network, and a synthesis network with generator and discriminator.These are the three main methodological components.
  • Modality-specific network: Each modality-specific network captures individual-modality information and produces a high-level feature representation.The network is constructed separately for each modality.
  • Modality-specific network: An autoencoder-like structure reconstructs each original image from its learned representation.The reconstruction objective is used to learn meaningful high-level features.
  • Modality-specific network: The reconstruction loss uses an ℓ1-norm and provides side-output supervision for discriminative modality-specific representations.This supervision encourages the network to preserve properties of each individual modality.
  • Network implementation: The implementation uses 3 × 3 stride-1 padded convolutions, batch normalization, LeakyReLU and ReLU activations, and 2 × 2 pooling and upsampling filters.LeakyReLU is used in the encoder and ReLU in the decoder.

B. Multi-modal Fusion Network

The multi-modal fusion network combines modality features across layers using three fusion operations and adaptively weights their contributions with MFB modules.

  • Multi-modal Fusion Network: The layer-wise fusion network exploits correlations between low-level and high-level representations from different modalities.It is designed to reduce modality diversity while combining multi-level features.
  • Multi-modal Fusion Network: Each MFB applies element-wise summation, product, and maximization to feature maps from multiple modalities.The resulting representations are concatenated before convolutional processing.
  • Multi-modal Fusion Network: An adaptive weighting network uses convolutional layers to combine the three fusion representations and previous MFB outputs.For the first MFB, no previous output is available, so the first convolutional output feeds directly into the second convolutional layer.
  • Multi-modal Fusion Network: The fusion network contains three MFB modules with filter counts of 32 and 64, 64 and 128, and 128 and 128, respectively.All filters use 3×3 kernels with batch normalization and ReLU activation after each convolutional layer.

C. Multi-modal Synthesis Network

The multi-modal synthesis network uses a GAN generator to transform the fused latent representation into a target-modality image while combining hierarchical modality features through skip-connected MFB modules.

  • Multi-modal Synthesis Network: The generator maps the final multi-modal latent representation F_n to a synthesized target-modality image.The discriminator distinguishes real target images from synthesized images.
  • Multi-modal Synthesis Network: The generator objective combines adversarial realism with an ℓ1 difference between synthesized and corresponding real images.The ℓ1 term is weighted by the nonnegative trade-off parameter λ1.
  • Multi-modal Synthesis Network: The complete end-to-end framework combines the generator and discriminator objectives with a regularization term.The coefficient λ2 controls the trade-off associated with that regularization term.
  • Multi-modal Synthesis Network: The generator densely combines the latent representation with encoding-layer features from each modality using MFB-based skip connections.It processes the latent representation through convolutional layers, three MFB modules, upsampling, and final convolutional layers.
  • Multi-modal Synthesis Network: The discriminator receives either a real or synthesized target-modality image and predicts whether the input is real.Its input has the same size as the generator output.

IV. EXPERIMENTS AND RESULTS

The experiments compare Hi-Net qualitatively with state-of-the-art methods on Flair and T2 synthesis tasks using the BraTs2018 dataset.

  • Experiments and Results: The experimental section includes comparison results, ablation studies, and related discussion.The stated evaluation uses dataset, comparison-method, metric, and implementation settings.
  • Experiments and Results: Figure 3 compares qualitative Flair synthesis results on BraTs2018 between Hi-Net and other state-of-the-art methods.Hi-Net and MM-Syns use both T1 and T2 inputs, whereas the other methods use either T1 or T2.
  • Experiments and Results: Figure 4 compares qualitative T2 synthesis results on BraTs2018 between Hi-Net and other state-of-the-art synthesis methods.Hi-Net and MM-Syns use both T1 and Flair inputs, whereas the other methods use either modality.

A. Dataset

The study evaluates Hi-Net on BraTs2018, a multi-institutional dataset containing four co-registered MR modalities from 285 patients.

  • Dataset: BraTs2018 contains scans from 285 patients across 19 institutions, including glioblastoma and lower grade glioma cohorts.The dataset provides multiple MR scans for each patient.
  • Dataset: Each patient scan includes co-registered T1, T1c, T2, and Flair volumes sized 240 × 240 × 155.The architecture uses 2D axial-plane slices from these volumes.
  • Dataset: The data are randomly split into 80% training and 20% testing subjects.Each cropped slice is divided into four overlapping 128 × 128 patches, whose overlaps are averaged for final estimation.
  • Dataset: The quantitative Flair evaluation compares methods using T1, T2, or their combination on BraTs2018.Results are reported as mean±standard deviation, with bold indicating the best result.

B. Comparison Methods and Evaluation Metrics

The study compares Hi-Net with Pix2pix, cycleGAN, and MM-Syns, using PSNR, NMSE, and SSIM to quantify synthesis quality.

  • Pix2pix preserves pixel-wise intensity similarity, cycleGAN uses cycle consistency for unpaired training, and MM-Syns is included as a comparison method.
  • Synthesis quality is evaluated with Peak Signal-to-Noise Ratio, Normalized Mean Squared Error, and Structural Similarity Index Measurement.
  • Higher PSNR and SSIM values indicate better synthesized-image quality, whereas lower NMSE values indicate better quality.

C. Implementation Details

Hi-Net is trained with Adam for 300 epochs and evaluated across three BraTS2018 synthesis tasks, with additional qualitative and dataset-based evaluation.

  • The model is trained for 300 epochs with Adam, using a learning rate of 0.0002 for 100 epochs followed by linear decay to zero.The trade-off parameters λ1 and λ2 are set to 100 and 20, respectively.
  • BraTS2018 evaluation covers T1 + T2 → Flair, T1 + Flair → T2, and T2 + Flair → T1 synthesis tasks.
  • For Flair synthesis on BraTS2018, Hi-Net outperforms all comparison methods on PSNR, NMSE, and SSIM and better synthesizes tumor regions qualitatively.
  • For T2 and T1 synthesis on BraTS2018, Hi-Net obtains better comparison results, including the best performance across all metrics for the T1 task.
  • Sagittal and coronal evaluations on BraTS2018 show that Hi-Net continues to outperform other methods and synthesizes high-quality target images.
  • An additional ISLES2015 evaluation synthesizes T2 images from T1 and Flair images in the SISS cohort.

E. Ablation Study

Ablation studies examine the Mixed Fusion Block, hybrid fusion, and modality-fusion placement to identify components contributing to Hi-Net’s synthesis performance.

  • The ablation design compares full Hi-Net with concatenation-based variants, MFB placement variants, and early- or late-fusion models.
  • Qualitative results in sagittal and coronal planes are reported for three BraTS2018 synthesis tasks, with separate input-modality pairings.
  • The full model improves synthesis performance over the concatenation-only variant, indicating a benefit from the MFB’s adaptive weighting of fusion strategies.
  • Using MFB modules in the fusion network performs better than using them only in the generator network.
  • The hybrid fusion network performs better than both early and late fusion by combining cross-modality correlations with modality-specific properties.

F. Discussion

Hi-Net addresses missing-modality synthesis by combining modality-specific properties with shared cross-modal information, and its experiments support improved synthesis quality.

  • Unlike single-input methods, Hi-Net fuses multiple modalities to synthesize missing modality images.
  • Its modality-specific networks preserve modality properties while its fusion network exploits correlations among modalities through a common latent representation.
  • Synthesized missing modalities could be used before multimodal segmentation or overall survival prediction when clinical scans are incomplete.
  • Across multiple synthesis tasks, Hi-Net outperforms other state-of-the-art methods in quantitative and qualitative measures.The authors identify synthetic-image data augmentation for multimodal learning as future work.
  • The paper illustrates direct, early, and late fusion strategies as alternatives relevant to the hybrid-fusion design.
Loading 2002.05000v1…