Source-linked AI summary
Image Super-Resolution via Dual-State Recurrent Networks
Wei Han, Shiyu Chang, Ding Liu, Mo Yu, Michael Witbrock, Thomas S. Huang
TL;DR
Deep SR models improve reconstruction but can require more parameters and storage. The paper recasts such models as compact recurrent systems and proposes DSRN, whose LR and HR states communicate bidirectionally; experiments report favorable efficiency and accuracy against state-of-the-art methods.
Problem
Increasing SR depth raises parameters, overfitting risk, and storage demands, motivating more compact architectures.
Method
DSRN extends the compact single-state RNN view with LR and HR recurrent states connected by LR-to-HR transformations and delayed HR-to-LR feedback.
Results
DSRN performs favorably against state-of-the-art SR models in efficiency and accuracy across benchmark datasets and DIV2K evaluations.
Takeaways & Limitations
Joint use of LR and HR features, with delayed feedback, supports a parameter-efficient SR design that retains competitive restoration performance.
Takeaways & Limitations
The formulation includes an assumption allowing xt to be chosen arbitrarily, including xt = 0 for all t.
Abstract
from arXiv · showhide
Advances in image super-resolution (SR) have recently benefited significantly from rapid developments in deep neural networks. Inspired by these recent discoveries, we note that many state-of-the-art deep SR architectures can be reformulated as a single-state recurrent neural network (RNN) with finite unfoldings. In this paper, we explore new structures for SR based on this compact RNN view, leading us to a dual-state design, the Dual-State Recurrent Network (DSRN). Compared to its single state counterparts that operate at a fixed spatial resolution, DSRN exploits both low-resolution (LR) and high-resolution (HR) signals jointly. Recurrent signals are exchanged between these states in both directions (both LR to HR and HR to LR) via delayed feedback. Extensive quantitative and qualitative evaluations on benchmark datasets and on a recent challenge demonstrate that the proposed DSRN performs favorably against state-of-the-art algorithms in terms of both memory consumption and predictive accuracy.
1. Introduction
Deep neural networks have improved single-image super-resolution, but deeper models increase parameters, overfitting risk, and storage demands. The paper reframes SR architectures as compact recurrent systems and proposes DSRN, which jointly exchanges LR and HR information.
- Single-image SR reconstructs a high-resolution image from one low-resolution image, with recent progress driven by deep neural networks.
- Greater CNN depth improves representational capacity and receptive fields, while residual connections facilitate training by preventing vanishing and exploding gradients.
- Deeper SR models add parameters, increasing overfitting risk and storage requirements; DRRN addresses this by sharing weights across residual units.
- Many recursive, weight-sharing SR models can be viewed as compact recurrent neural networks, motivating new SR architectures from this formulation.
- DSRN uses LR and HR states, connects LR to HR through deconvolution, and feeds predicted HR features back to LR through delayed feedback.
- Across four benchmarks and DIV2K, DSRN is reported to provide higher parameter efficiency, low memory consumption, and high restoration accuracy.
2. Related Work
Related SR work developed increasingly deep, recursive, compact, and perceptually oriented networks. These approaches trade model capacity, computational cost, reconstruction fidelity, and subjective quality in different ways.
- SRCNN introduced an end-to-end fully convolutional approach for nonlinear LR-to-HR mapping, conceptually separating representation, mapping, and reconstruction stages.
- Subsequent methods added more complex building blocks, optimization techniques, sparse-coding priors, and external or self-example information.
- VDSR increased depth with 20 convolutional layers and residual learning, while DRCN used 16 recursive layers with shared parameters to control model size.
- Direct LR processing and learned upscaling filters reduce computation, but relatively small networks may struggle with complicated mappings because of limited capacity.
- MSE-based SR methods can produce blurry predictions through regression-to-the-mean, motivating GANs and perceptual losses that improve subjective evaluation scores.
- Viewing residual networks as unrolled RNNs provides a framework for understanding deep structures and relates the paper to recurrent architectural design.
3. Single-State Recurrent Networks
The paper formulates recurrent and residual SR architectures as finite unfoldings of RNNs with shared or architecture-specific recurrent functions. This view unifies ResNet, DRCN, and DRRN as single-state systems.
- A recurrent neural network is characterized by states and transition functions, and is termed single-state or dual-state according to its number of recurrent states.
- The canonical single-state RNN is unfolded for a fixed length T, producing recurrent states and outputs through input, recurrent, and output functions.
- Time-independent parameters reused at every unfolding step allow ResNet, DRCN, and DRRN to share a recurrent structure while differing in recurrent functions and parameter-sharing rules.
- ResNet: A ResNet corresponds to a single-state recurrence when the input is suppressed after initialization and the recurrent function is a residual block.
- DRCN: DRCN recursively applies one shared convolutional layer for 16 unfoldings, generates outputs at every unfolding, and combines them through a weighted sum for final prediction.
- DRRN: DRRN differs from ResNet because its skip connection always originates from the initial unrolled state s0 rather than the previous residual block.
4. Dual-State Recurrent Networks
DSRN extends the compact RNN view of SR with recurrent states at LR and HR resolutions, enabling information exchange across resolutions. Its unrolled design uses shared transition functions, delayed feedback, and deep supervision for HR prediction.
- Dual-state design: DSRN uses two recurrent states: an LR state and an HR state, unlike single-state models operating at one spatial resolution.The LR state captures low-resolution information, while the HR state operates in the high-resolution regime.
- Dual-state design: Delayed feedback adds an HR-to-LR connection, complementing LR-to-LR, HR-to-HR, and LR-to-HR transitions.This creates two-way information flow between the LR and HR states.
- Unrolling: The unrolled DSRN represents recurrent computation across time steps, with shared parameters on edges having the same color.Figure 2 depicts the HR state in the top row and the LR state in the bottom row.
- Transition functions: DSRN uses residual blocks for self-transitions, convolution for down-sampling, and transposed convolution for up-sampling.The strides of both inter-state layers equal the SR upscaling factor.
- Deep supervision: The unrolled network makes predictions at every time step and averages them rather than using only the final unfolding.Each unrolled layer connects directly to the loss layer, and training minimizes error on the predicted residual image.
5. Experiments
Experiments evaluate DSRN’s implementation choices, recurrent design components, benchmark accuracy, visual quality, and parameter efficiency. Results show benefits from longer unrolling, shared parameters, dual-state feedback, and competitive performance across datasets and challenge comparisons.
- Implementation Details: DSRN is trained on 91 images and evaluated on Set5, Set14, B100, Urban100, and DIV2K using separate ×2, ×3, and ×4 models.Training uses geometric augmentation, and evaluation includes standard SR benchmarks plus the DIV2K challenge dataset.
- Implementation Details: The model uses luminance-only super-resolution, bicubic upscaling for chrominance, 64 filters initially, 128 thereafter, and 3×3 convolutions.The dual-state feature maps match LR and HR spatial dimensions.
- Model Analysis: Increasing unrolling length improves test performance, but gains diminish after T = 7; unrolling controls computation and capacity while parameter count stays fixed.For T unrollings, maximum depth is 2T + 4 convolution layers, while shared parameters remain independent of T.
- Model Analysis: Around 0.2dB performance is lost across all three upscaling scales when shared weights are replaced with untied weights.The authors associate untied weights with greater overfitting risk and slower convergence.
- Model Analysis: Both dual-state processing and HR-to-LR delayed feedback improve performance, with feedback gains observed consistently across ×2, ×3, and ×4 scales.The comparison uses a single-state RNN baseline and a dual-state model without feedback under matched unrolling steps.
- Comparison with the State-of-the-Art: DSRN achieves competitive benchmark performance, particularly for ×2 and ×3, while offering a favorable model-size/PSNR trade-off and 0.4s inference for ×4 output at 288×288.On DIV2K, it is competitive with EDSR+ and outperforms the other compared algorithms by a large margin; qualitative examples show sharper structures and fewer artifacts.
6. Conclusion
The paper formulates many state-of-the-art SR models as finite unfoldings of single-state RNNs and extends this view with DSRN, whose LR and HR states communicate through delayed feedback. Experiments show favorable efficiency and accuracy compared with state-of-the-art SR models.
- DSRN uses two hidden states operating at different spatial resolutions, with one capturing LR information and the other targeting HR domains.
- Delayed feedback enables two-way communication between the LR and HR states, allowing both predicted feature types to contribute jointly to final predictions.
- The paper expresses many state-of-the-art SR models as finite unfoldings of single-state RNNs with various recurrent functions.
- Extensive benchmark experiments report that DSRN performs favorably against state-of-the-art SR models in efficiency and accuracy.
- Future work will investigate DSRN for capturing temporal dependencies in video SR.