Source-linked AI summary
Simultaneous Super-Resolution and Cross-Modality Synthesis of 3D Medical Images using Weakly-Supervised Joint Convolutional Sparse Coding
Yawen Huang, Ling Shao, Alejandro F. Frangi
TL;DR
High-resolution multimodal MRI is costly and time-consuming to acquire, motivating reconstruction of high-resolution target-modality images from low-resolution single-modality inputs. The paper proposes WEENIE, a weakly supervised joint convolutional sparse coding framework using few registered pairs, unpaired-data alignment, and distribution matching. Experiments report better SR and simultaneous SR-CMS performance than state-of-the-art methods, while the alignment can remain imperfect for highly heterogeneous domains.
Problem
Acquiring complete high-resolution multimodal MRI datasets is constrained by scanning costs, time, scanner availability, and patient comfort, while SR and CMS require different image representations.
Method
WEENIE jointly learns convolutional sparse representations for SR and CMS from a small registered-pair set augmented with aligned unpaired images and a maximum mean discrepancy term.
Results
WEENIE outperforms state-of-the-art methods in SR reconstruction and simultaneous SR-CMS on reconstruction-error and visual-quality measures.
Takeaways & Limitations
The framework provides a whole-image, weakly supervised approach for learning heterogeneous-resolution and cross-modality mappings with limited registered data.
Takeaways & Limitations
Feature alignment may be insufficient for highly heterogeneous domain adaptation, producing suboptimal filter pairs and inaccurate results.
Abstract
from arXiv · showhide
Magnetic Resonance Imaging (MRI) offers high-resolution \emph{in vivo} imaging and rich functional and anatomical multimodality tissue contrast. In practice, however, there are challenges associated with considerations of scanning costs, patient comfort, and scanning time that constrain how much data can be acquired in clinical or research studies. In this paper, we explore the possibility of generating high-resolution and multimodal images from low-resolution single-modality imagery. We propose the weakly-supervised joint convolutional sparse coding to simultaneously solve the problems of super-resolution (SR) and cross-modality image synthesis. The learning process requires only a few registered multimodal image pairs as the training set. Additionally, the quality of the joint dictionary learning can be improved using a larger set of unpaired images. To combine unpaired data from different image resolutions/modalities, a hetero-domain image alignment term is proposed. Local image neighborhoods are naturally preserved by operating on the whole image domain (as opposed to image patches) and using joint convolutional sparse coding. The paired images are enhanced in the joint learning process with unpaired data and an additional maximum mean discrepancy term, which minimizes the dissimilarity between their feature distributions. Experiments show that the proposed method outperforms state-of-the-art techniques on both SR reconstruction and simultaneous SR and cross-modality synthesis.
1. Introduction
High-resolution multimodal MRI acquisition is constrained by cost, time, scanner availability, and patient comfort, while SR and CMS remain challenging learning problems. The paper proposes WEENIE, a weakly supervised whole-image framework that jointly addresses both tasks using limited registered pairs and additional unpaired data.
- Motivation: Complete high-resolution multimodal MRI acquisition is limited by scanning costs, scanner availability, scanning time, and patient comfort.High-resolution 3D imaging can require long breath-hold and repetition times, making routine clinical acquisition challenging or infeasible.
- Challenges: SR reconstructs higher-resolution images from lower-resolution data, but its ill-posed nature makes the solution non-unique and dependent on strong priors.Existing priors include smoothness assumptions and external or internal data-driven learning.
- Challenges: CMS must address modality-specific image representations and reveal relationships between different representations of shared underlying information.A model learned from one modality may not be optimal for another modality.
- Limitations of prior work: Most existing methods rely on supervised training pairs or patch-based synthesis, creating practical data requirements and overlap-fusion inconsistencies.Supervised approaches require many training image pairs, while patch-based methods can produce inconsistencies where patches overlap.
- Proposed approach: WEENIE jointly solves SR and CMS using a small set of registered multimodal pairs, while learning a unified representation for heterogeneous data in a common feature space.The framework operates on whole-image representations to preserve local consistency and uses co-occurring texture features across domains.
- Proposed approach: A hetero-domain alignment term incorporates unpaired images, and a maximum mean discrepancy term supports joint learning of LR-HR cross-modality pairs.The paper reports improved reconstruction error and visual quality over state-of-the-art methods.
2. Related Work
Related work spans super-resolution methods that reconstruct HR images from LR inputs and cross-modality synthesis methods that generate unavailable modalities from available images. Prior approaches include interpolation, example-based learning, sparse coding, correlation analysis, and deep networks, but commonly rely on patch correspondences or modality-specific representations.
- Overview: The broader literature separates SR reconstruction from single or multiple modalities and CMS, reflecting distinct resolution-recovery and modality-transfer objectives.These categories motivate methods that can address both problems together.
- Image Super-Resolution: Super-resolution methods reconstruct HR images from LR counterparts using interpolation-based, external-data-driven, or internal-data-driven priors.Interpolation methods create denser HR grids through weighted averages of local neighbors, whereas learning-based methods learn compact dictionaries or models.
- Cross-Modality Synthesis: Cross-modality synthesis generates unavailable modality data from available source images, particularly in medical imaging.The literature includes example-based learning, sparse coding, canonical correlation analysis, location-sensitive deep networks, and CNN-based approaches.
- Cross-Modality Synthesis: Example-based synthesis uses nearest-neighbor patches and Markov random fields to reconstruct target patches from similar source-image patches.This approach is described as an established modality-transformation strategy.
- Cross-Modality Synthesis: Sparse-coding CMS methods assume cross-modality patch pairs share representations and use paired dictionaries to estimate the target contrast.Other approaches learn common structures from co-registered data or combine intensities with spatial coordinates.
3. Weakly-Supervised Joint Convolutional Sparse Coding
The proposed formulation learns whole-image convolutional sparse representations for heterogeneous LR and HR modality data, using a small paired set alongside mostly unpaired training images. Alignment establishes cross-domain correspondences before joint filter learning, while optimization alternates among non jointly convex variables.
- 3.1. Preliminaries: Convolutional sparse coding represents an entire image as a sum of convolutions between dictionary filters and sparse feature maps.It removes shift variations associated with patch-based sparse coding and supports consistent approximation of local neighborhoods.
- 3.1. Preliminaries: In CSC, filters and sparse feature maps are learned by minimizing reconstruction error with an l1 sparsity penalty and constrained filter norms.The feature map dimensions extend the image according to the filter size, and the constraint prevents filters from absorbing all system energy.
- 3.1. Preliminaries: CSC can be solved by alternately optimizing one variable while fixing the other, with fast implementations using ADMM in the Fourier domain.These procedures address the filter and feature-map variables separately rather than optimizing them simultaneously.
- 3.2. Problem Formulation: The SRCMS task infers an HR target-modality 3D image from an LR source-modality 3D image using training sets containing few paired and mostly unpaired samples.The formulation explicitly accommodates heterogeneous resolutions, modalities, and domain fit.
- 3.2. Problem Formulation: The method first establishes one-to-one correspondences between source and target data, then jointly learns paired filters from the aligned images.The source modality is M1 and the target modality is M2.
3.3. Hetero-Domain Image Alignment
The method aligns low-resolution source images with high-resolution target images across modalities by matching high-frequency features. A subject-specific binary correspondence retains the most relevant cross-domain match for each source image.
- Feature construction: Low-resolution images use first- and second-order horizontal and vertical gradients as high-frequency features for alignment.High-resolution images instead derive high-frequency features by subtracting their mean intensity.
- Feature construction: High-resolution image features are obtained by directly subtracting each image’s mean value.
- Cross-domain matching: For each source image, the alignment retains the target with maximum similarity and discards the remaining candidates.
- Cross-domain matching: The resulting subject-specific alignment matrix is binary, connecting each source subject to one target with the most similar brain structures.
3.4. Objective Function
The objective function jointly learns convolutional filter pairs, sparse feature maps, and a linear association between source and target features. It combines whole-image sparse coding with alignment and distribution-matching constraints to use paired and unpaired data.
- Joint representation: Convolutional sparse coding represents aligned source and target images using filter pairs and corresponding sparse feature maps over the whole image.This avoids shifted or inconsistent representations associated with independently aligned patches.
- Feature association: The model learns a mapping from source sparse feature maps to target sparse feature maps using a linear projection W_k.
- Data integration: The alignment term A(X,Y) enforces correspondence for auxiliary unpaired subjects after cross-domain matching.
- Objective design: The parameters λ, β, and γ balance sparsity, feature representation, and association mapping.
- Distribution regularization: An empirical maximum mean discrepancy term addresses mismatched hetero-domain pairs by reducing differences between their feature distributions in RKHS.
- Distribution regularization: Regularizing the objective with the MMD term refines filter pairs and draws the distributions of real aligned subject pairs closer under the learned feature maps.
3.5. Optimization
WEENIE optimizes its unified objective through alternating updates because the problem is not jointly convex, although it is convex in each variable when the others are fixed. The subproblems update feature maps, filters, and the mapping function in turn.
- Alternating optimization: The unified objective is not jointly convex in F, Z, and W, so the method uses a three-step alternating optimization strategy.Each subproblem is convex when the remaining variables are fixed.
- Feature-map updates: Sparse feature maps are optimized after initializing the filters and mapping function, with additional alignment and divergence-reduction terms beyond the original CSC formulation.
- Block updates: The convolutional least-squares update for filter pairs is followed by an l1-regularized least-squares update for sparse feature maps.
- Block updates: Filter pairs are updated with the corresponding sparse feature maps fixed, and the mapping matrix W is then learned separately.
- Filter updates: The filter update subject to Eq. (10) uses a one-by-one strategy with an augmented Lagrangian method.
- Mapping update: The W update is a ridge-regression problem with a regularization term and an analytically derived solution.
1 Initialize Fx
After initialization, WEENIE repeatedly updates sparse feature maps, filters, and the mapping until convergence, then outputs the learned filter pairs and mapping. The trained model synthesizes a target high-resolution modality from a test image’s sparse representation.
- Initialization and iteration: The algorithm iterates while unconverged, updating the source feature maps, source filters, and mapping while holding other variables fixed.
- Output: After convergence, the algorithm outputs the learned filter pairs F_x, F_y and mapping W.
- Synthesis: For a test image X_t, source filters produce sparse feature maps that W converts into expected target feature maps.
- Synthesis: The synthesized high-resolution M2 image is reconstructed by convolving the converted feature maps with target filters and summing the K components.
1 Initialize Zt
The procedure initializes transformed variables, iterates until convergence, and updates an intermediate quantity using equation (9).
- The optimization repeats its update steps while convergence has not been reached.
- An intermediate quantity is updated using equation (9) with Yt.
6 end
The procedure outputs the synthesized image Yt. Figure 2 presents example super-resolution results with corresponding PSNR and SSIM values.
- The procedure synthesizes and outputs image Yt.
- Figure 2 shows example super-resolution results alongside their PSNR and SSIM values.
- The figure invites detailed inspection of the reported super-resolution examples.
4. Experimental Results
Experiments evaluate WEENIE for brain-image super-resolution and simultaneous super-resolution with cross-modality synthesis on IXI and NAMIC datasets. The reported results show strong performance across the evaluated tasks and datasets.
- Experimental setup: Experiments use IXI and NAMIC brain multimodality datasets, with low-resolution images produced by bicubic downsampling at rate 1/2.The study also pads boundaries by eight pixels and empirically sets σ, λ, β, and γ to 1, 0.05, 0.1, and 0.15.
- Experimental setup: Training uses 30 registered IXI subject pairs and 3 registered NAMIC subject pairs, with remaining training data processed as unpaired.The synthesis comparisons also assess preprocessing involving skull stripping and bias correction.
- Brain MRI Super-Resolution: The super-resolution study compares WEENIE with ScSR, ANR, NE+LLE, Zeyde’s method, CSC-SR, and A+ on IXI PD-w images.Image super-resolution uses scaling factor 2 and evaluates average PSNR and SSIM over 95 test subjects.
- Brain MRI Super-Resolution: The reported super-resolution evaluation gives WEENIE the best PSNR and SSIM values among the compared methods.The authors attribute the improvement to MMD-regularized joint learning improving the representation power of learned filter pairs.
- Simultaneous Super-Resolution and Cross-Modality Synthesis: SRCMS experiments cover six synthesis directions across IXI and NAMIC, including LR-to-SR cross-modality translation between PD-w, T2-w, and T1-w images.Comparisons include MIMECS and, where applicable, Vemulapalli’s supervised and unsupervised methods.
- Simultaneous Super-Resolution and Cross-Modality Synthesis: Across the two datasets, WEENIE reaches the best PSNR and SSIM for almost all subjects in the reported SRCMS experiments.Visual comparisons show advantages over MIMECS in white-matter structures and overall intensity profile.
5. Conclusion
The conclusion presents WEENIE as a weakly supervised joint convolutional sparse coding method for simultaneous super-resolution and cross-modality synthesis in 3D MRI. It combines paired and unpaired data in a common feature space and reports superior performance against state-of-the-art methods.
- WEENIE jointly addresses 3D MRI super-resolution and cross-modality synthesis using weakly supervised joint convolutional sparse coding.
- The method uses a small set of registered pairs while aligning auxiliary unpaired images through a hetero-domain alignment term.
- Joint optimization learns filter pairs and a mapping function in a common feature space, with divergence minimization enhancing robustness.
- Whole-image processing uses a consistency prior to capture correlations between local neighborhoods.
- Extensive experiments report superior performance against state-of-the-art methods for both brain-image SR and SRCMS.