Source-linked AI summary
Quadratic Autoencoder (Q-AE) for Low-dose CT Denoising
Fenglei Fan, Hongming Shan, Mannudeep K. Kalra, Ramandeep Singh, Guhan Qian, Matthew Getzin, Yueyang Teng, Juergen Hahn, Ge Wang
TL;DR
Low-dose CT denoising must remove dose-induced noise and artifacts while preserving image information. The paper constructs a quadratic autoencoder from quadratic neurons and evaluates it for this task. Experiments on the Mayo low-dose CT dataset demonstrate image-denoising utility and model efficiency, while the paper positions quadratic networks as trainable high-order representations.
Problem
Low-dose CT denoising must address noise and artifacts introduced when radiation dose is reduced, while existing traditional post-processing methods can produce over-smoothness and structural distortion.
Method
The paper constructs an encoder-decoder quadratic autoencoder whose neurons replace conventional inner products with quadratic input processing.
Results
Experiments on the Mayo low-dose CT dataset demonstrate the quadratic autoencoder’s utility for image denoising and model efficiency.
Takeaways & Limitations
Quadratic neurons provide a trainable high-order nonlinear representation for a deep autoencoder applied to medical image denoising.
Abstract
from arXiv · showhide
Inspired by complexity and diversity of biological neurons, our group proposed quadratic neurons by replacing the inner product in current artificial neurons with a quadratic operation on input data, thereby enhancing the capability of an individual neuron. Along this direction, we are motivated to evaluate the power of quadratic neurons in popular network architectures, simulating human-like learning in the form of quadratic-neuron-based deep learning. Our prior theoretical studies have shown important merits of quadratic neurons and networks in representation, efficiency, and interpretability. In this paper, we use quadratic neurons to construct an encoder-decoder structure, referred as the quadratic autoencoder, and apply it to low-dose CT denoising. The experimental results on the Mayo low-dose CT dataset demonstrate the utility of quadratic autoencoder in terms of image denoising and model efficiency. To our best knowledge, this is the first time that the deep learning approach is implemented with a new type of neurons and demonstrates a significant potential in the medical imaging field.
I. INTRODUCTION
The paper introduces quadratic neurons and a quadratic autoencoder for low-dose CT denoising, motivated by biological neuron complexity and the need to reduce dose-related image degradation.
- Biological neurons are more diverse and complex than the simplified homogeneous neurons typically used in neural networks.
- Quadratic neurons replace the inner product in conventional artificial neurons with quadratic input processing.
- The proposed quadratic autoencoder uses quadratic feature representations to test whether complex data can be represented more effectively and efficiently.
- Prior high-order approaches faced training difficulties, whereas the proposed quadratic network is described as trainable with reasonable model complexity.
- Low-dose CT denoising is needed because reducing radiation introduces noise and artifacts into reconstructed images.
- Deep learning post-processing methods have improved low-dose CT denoising compared with traditional methods that can cause over-smoothness and structural distortion.
A. Quadratic Neuron
A quadratic neuron computes a sparse quadratic transformation before activation, and quadratic convolution applies this operation across image fields to form nonlinear feature maps.
- A. Quadratic Neuron: A quadratic neuron processes an input vector through a quadratic operation using three parameter groups, requiring 3n parameters.
- A. Quadratic Neuron: The neuron applies an activation function to its quadratic response, with ReLU given as an example.
- A. Quadratic Neuron: A general quadratic filter with O(n^2) complexity is harder to use in deep networks because of training-cost growth and overfitting risk.
- B. “Quadratic Convolution”: Quadratic convolution slides a quadratic neuron over an input image to produce nonlinear feature maps.
- A. Quadratic Neuron: The quadratic neuron is characterized by quadratic processing before activation rather than merely using a quadratic activation function.
C. Algebraic Structure and Model Efficiency
The paper presents theoretical results linking quadratic networks to polynomial representation and potential efficiency advantages over conventional networks.
- C. Algebraic Structure and Model Efficiency: A quadratic network of depth O(log2(N)) and width no more than N can represent any univariate polynomial of degree N.
- C. Algebraic Structure and Model Efficiency: For some functions, a one-hidden-layer quadratic network achieves a given approximation with polynomially many neurons, while a conventional network may require exponentially more.
D. Training Quadratic Networks
Quadratic networks are trained by backpropagation, using gradients of the loss with respect to quadratic-neuron parameters to update weights and biases.
- D. Training Quadratic Networks: Training a quadratic deep network follows the conventional layer-wise chain-rule backpropagation procedure until convergence.
- D. Training Quadratic Networks: Gradients with respect to quadratic-neuron weights and biases are used to update parameters and minimize the loss.
E. General Autoencoder Model
The general autoencoder maps inputs to latent features and decodes them, with optional regularization on features and parameters. This study removes both regularization terms to isolate the quadratic autoencoder’s intrinsic potential.
- E. General Autoencoder Model: A general autoencoder encodes input X into latent features and decodes them, while supervised tasks use labels Y and unsupervised learning reconstructs X itself.H_E and H_D denote the encoding and decoding functions; W_E and W_D are their parameters.
- E. General Autoencoder Model: Regularization functions can impose feature constraints such as sparsity or invariance and weight constraints such as l1-norm penalties that help prevent overfitting.
- E. General Autoencoder Model: The study sets both regularization terms to zero, focusing on the generic quadratic autoencoder so gains cannot be attributed to regularization.
A. Network Design
The network design treats architecture, loss, optimization, and dataset as determinants of neural-network performance. The proposed quadratic autoencoder is evaluated with mean squared error against representative denoising models, while other loss functions remain possible.
- A. Network Design: Network performance is framed as depending on architecture, loss function, optimization strategy, and dataset, with architecture controlling intrinsic representation ability.The study uses Adam for optimization and contrasts the proposed method with CNN10, RED-CNN, WGAN-VGG, and CPCE-2D.
- A. Network Design: Mean squared error is used to evaluate the quadratic autoencoder’s intrinsic representation ability against competing denoising models.The passage notes that similarity loss can preserve structural and textual information, but MSE is used for the comparison here.
- A. Network Design: Table I presents a comparison of representative denoising models.
1) Quadratic Autoencoder Using Residual Shortcuts:
The quadratic autoencoder combines a symmetric convolutional encoder-decoder with quadratic neurons for low-dose CT denoising. Across image, reader-study, convergence, and efficiency evaluations, Q-AE delivered strong denoising and structure preservation while using substantially fewer parameters than RED-CNN.
- Architecture: The model uses stacked convolutional layers without pooling in the encoder, a bottleneck layer, and deconvolutional layers in a symmetric decoder.Zero padding in the first four convolutional layers preserves local correlation, while the decoder is designed to preserve structural details.
- Experimental setting: Experiments used 74,000 normalized 64*64 patches from five Mayo low-dose CT patient datasets, with 64,000 for training and 10,000 for validation.All methods were trained for 20 epochs with Adam and mean squared error under matched conditions.
- Convergence behavior: Q-AE achieved the lowest loss among the compared models, and after six epochs its validation performance surpassed CPCE-2D’s final performance.Its loss decreased monotonically and continued descending later in training, whereas CNN10’s trajectory fluctuated.
- Denoising performance: Q-AE produced lower image noise, preserved small structures, and maintained conspicuity of small organisms relative to the other denoising methods.The qualitative comparisons used a fixed display window of [-160, 240].
- Model efficiency: Q-AE used no more than one fourth of RED-CNN’s parameters, while RED-CNN required four times its GFLOPs for a forward pass.Q-AE had the lowest parameter count but incurred additional computational cost compared with some competing models.
6). Over-fitting
Q-AE is evaluated for fitting behavior, initialization robustness, intrinsic denoising ability, and comparison with quadratic-activation autoencoders. The supplied results indicate stable fitting, lower RMSE in compact models, and stronger performance than the compared alternatives.
- 6). Over-fitting: Q-AE’s training and validation losses decrease together, with similar trajectories later in training, indicating appropriate fitting of the denoising task.The authors use this behavior to assess over-fitting risk.
- 7). Training by Transfer Learning: Weight transfer initializes corresponding quadratic-network parameters from a trained conventional network while leaving quadratic-specific terms partially or randomly initialized.The authors state that conventional weights cannot initialize the quadratic terms and that optimal quadratic-network training remains open.
- 8). Robustness: Q-AE consistently outperformed CNN10, RED-CNN, and CPCE-2D in PSNR and SSIM across all slices from five patients.The four models had comparable numbers of trainable parameters in this anatomical-slice robustness test.
- 9). Intrinsic Denoising Ability of Quadratic Networks: Quadratic neurons reduced RMSE in denoising, but the reduction diminished as network complexity increased.This result supports a compact-model advantage rather than a uniform gain at every network size.
- 9). Intrinsic Denoising Ability of Quadratic Networks: Q-AE achieved 1~2% lower RMSE than conventional autoencoders in compact networks and retained a 1~2% advantage with 14,475 versus 16,470 trainable parameters.The RMSE reduction decreased to 0.5~1% at 32 channels and had no significant effect at 48 channels.
- 10). Comparison with Networks Using Quadratic Activation: The autoencoder with rectified quadratic activation did not converge well, whereas the autoencoder with quadratic activation converged to a level higher than Q-AE.The comparison used the same other experimental conditions as Q-AE, with empirically selected learning rates.
IV. DISCUSSIONS AND CONCLUSIONS
The paper concludes that quadratic deep learning performs well for low-dose CT denoising and may extend to applications such as radiomics. It also notes that the approach can achieve this performance with compact models, although some alternative quadratic-activation models remain noisy.
- IV. DISCUSSIONS AND CONCLUSIONS: Quadratic deep learning is reported to show merits in low-dose CT denoising, with the authors hypothesizing advantages in other applications such as radiomics.The conclusion connects quadratic neurons with more complex feature extraction and piecewise-quadratic decision boundaries.
- IV. DISCUSSIONS AND CONCLUSIONS: Q-AE’s established superiority over state-of-the-art low-dose CT denoising networks is presented as encouraging for further quadratic-model applications.The authors identify additional real-world applications as follow-up work.
- IV. DISCUSSIONS AND CONCLUSIONS: Autoencoders using quadratic activation produced noisy outputs with dotty artifacts.This observation accompanies the convergence comparison against Q-AE.