Source-linked AI summary
Task Transformer Network for Joint MRI Reconstruction and Super-Resolution
Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, Yong Xu
TL;DR
MRI reconstruction and super-resolution are important but commonly separate tasks, leaving their shared information underused. T2Net jointly learns both tasks with task-specific branches and a transformer-based feature-transfer module, and it outperforms sequential baselines quantitatively and qualitatively. The approach targets higher-quality, super-resolved reconstructions from highly undersampled and degenerated MRI data.
Problem
MRI reconstruction and super-resolution are commonly designed as separate tasks, although related tasks can share features and dependencies.
Method
T2Net uses two task-specific branches and a task transformer module to learn, transfer, and synthesize shared features for joint MRI reconstruction and super-resolution.
Results
T2Net outperforms sequential combinations of state-of-the-art reconstruction and super-resolution models, achieving up to 29.397 dB PSNR under 2× enlargement and 28.659 dB versus 21.895 dB under 4× enlargement on IXI.
Takeaways & Limitations
Sharing task-specific and anatomical-structure information enables higher-quality, super-resolved reconstructions with less blurring and artifacts within the evaluated MRI settings.
Takeaways & Limitations
The formulation assumes a specific MRI acquisition and degradation process based on undersampling k-space with a binary Cartesian mask and subsequent resolution downgrading.
Abstract
from arXiv · showhide
The core problem of Magnetic Resonance Imaging (MRI) is the trade off between acceleration and image quality. Image reconstruction and super-resolution are two crucial techniques in Magnetic Resonance Imaging (MRI). Current methods are designed to perform these tasks separately, ignoring the correlations between them. In this work, we propose an end-to-end task transformer network (T$^2$Net) for joint MRI reconstruction and super-resolution, which allows representations and feature transmission to be shared between multiple task to achieve higher-quality, super-resolved and motion-artifacts-free images from highly undersampled and degenerated MRI data. Our framework combines both reconstruction and super-resolution, divided into two sub-branches, whose features are expressed as queries and keys. Specifically, we encourage joint feature learning between the two tasks, thereby transferring accurate task information. We first use two separate CNN branches to extract task-specific features. Then, a task transformer module is designed to embed and synthesize the relevance between the two tasks. Experimental results show that our multi-task model significantly outperforms advanced sequential methods, both quantitatively and qualitatively.
1 Introduction
MRI reconstruction and super-resolution improve image quality but are commonly treated as separate tasks, despite their shared features and dependencies. T2Net addresses this gap by transferring complementary information between task-specific branches through a task transformer.
- Motivation: MRI acquisition is inherently slower than CT or X-rays, motivating efforts to accelerate data collection while preserving image quality.Reconstruction accelerates MRI by reducing k-space sampling, while super-resolution restores a high-resolution image from degenerated low-resolution data.
- Research gap: Existing reconstruction and super-resolution methods typically train separate models for one specific function, leaving potentially useful cross-task information underused.Related tasks often share features and dependencies, motivating multi-task learning.
- Proposed approach: T2Net is an end-to-end task transformer network that jointly integrates MRI reconstruction and super-resolution.The framework is designed for multi-task learning across both MRI tasks.
- Proposed approach: The framework uses two task-specific branches and a task transformer module to transfer shared visual features between them.The module includes relevance embedding, transfer attention, and soft attention for sharing complementary features.
- Reported outcome: T2Net produces superior results compared with sequential combinations of state-of-the-art MRI reconstruction and super-resolution models.The comparison concerns multi-task integration versus sequential task pipelines.
2 Method
T2Net jointly reconstructs and super-resolves undersampled, degraded MRI through task-specific branches connected by a transformer that transfers shared anatomical information. The reconstruction branch removes artifacts while the SR branch produces the final high-resolution, motion-artifact-free image.
- Input Formation: The input is formed by masking fully sampled k-space and then reducing resolution, producing undersampled, degraded MRI data for joint reconstruction and super-resolution.The undersampled k-space is obtained with a Cartesian binary mask; resolution reduction yields the multi-task input.
- Overall Framework: T2Net uses an SR branch, a reconstruction branch, and a task transformer module to learn task-specific and shared features end to end.The branches receive the same degraded input, while the transformer encourages a shared representation.
- SR Branch: The SR branch extracts features with convolutional layers and an EDSR backbone, applies task-transformer outputs through residual blocks, and upsamples with sub-pixel convolution.The branch is supervised by the fully sampled high-resolution image.
- Reconstruction Branch: The reconstruction branch extracts deep motion-corrected features and is supervised by the low-resolution motion-artifact-free image to remove artifacts from the input.Its output is fused into the SR branch to produce the final super-resolved, motion-artifact-free image.
- Task Transformer Module: The task transformer estimates relevance between SR and reconstruction features, selects corresponding reconstruction features, and synthesizes them with SR features using soft attention.SR and reconstruction features serve as query and key, while the transferred representation is sent to the SR branch.
3 Experiments
Experiments evaluate T2Net on two datasets under 6× Cartesian acceleration and multiple enlargement scales, comparing it with sequential reconstruction–super-resolution baselines. T2Net achieves stronger quantitative and visual results, especially at higher enlargement scales, while ablations assess its reconstruction branch and task transformer.
- Experimental setup: The experiments use the IXI and clinical brain MRI datasets, with patient-wise 7:1:2 training, validation, and testing splits.The clinical dataset contains fully sampled scans from 155 patients.
- Experimental setup: Four sequential baselines combine ADMMNet or MICCAN reconstruction with MGLRL or Lyu et al. super-resolution.The reconstruction model is applied first to remove artifacts, followed by the super-resolution model.
- Quantitative results: Under 6× acceleration, T2Net reaches 29.397 dB PSNR on IXI with 2× enlargement and improves PSNR from 21.895 to 28.659 dB at 4× enlargement versus the best sequential combination.The reported comparison is specifically on the IXI dataset.
- Quantitative results: On the clinical dataset, T2Net significantly outperforms all sequential combinations at both enlargement scales.The paper reports average PSNR, SSIM, and NMSE scores for the two datasets.
- Visual results: Visual comparisons show that sequential methods improve image quality but are less effective than T2Net, whose multi-task methods are more robust to aliasing and structural loss.The figure compares restored images and error maps from IXI at 2× enlargement and the clinical dataset at 4× enlargement.
- Ablation study: Ablations evaluate removing the reconstruction branch or task transformer module to assess the contribution of these components.The w/o Rec variant uses only the super-resolution branch, while w/o Htt removes the task transformer while retaining both branches.
4 Conclusion
The paper concludes that T2Net jointly learns MRI reconstruction and super-resolution through task-specific branches and transformer-based transfer of shared anatomical structure information. This design targets high-quality, super-resolved reconstructions with correct anatomy, fine details, and fewer blurring artifacts.
- Conclusion: T2Net uses target and auxiliary branches for super-resolution and reconstruction, respectively, plus a task transformer that transfers shared structure information.The transformer transfers anatomical information to the target super-resolution branch.
- Conclusion: The task transformer combines feature embedding, hard attention, and soft attention to transfer and synthesize the final reconstructions.The stated goal is to preserve fine details while reducing blurring and artifacts.
- Future work: The authors identify automatically learning the loss weights as future work.This is the paper's stated future direction.