Source-linked AI summary
Pixel to Gaussian: Ultra-Fast Continuous Super-Resolution with 2D Gaussian Modeling
Long Peng, Anran Wu, Wenbo Li, Peizhe Xia, Xueyuan Dai, Xinjie Zhang, Xin Di, Haoze Sun, Renjing Pei, Yang Wang, Yang Cao, Zheng-Jun Zha
TL;DR
ASSR seeks arbitrary-scale HR reconstruction from LR images with one model, while INR approaches are limited by repeated decoding and coordinate-based representations. ContinuousSR reconstructs a continuous HR Gaussian field directly, using DGP-based covariance weighting and adaptive position drifting. It reports state-of-the-art quality and speed, including 19.5× speed improvement and 0.90dB PSNR improvement, while noting limitations from fixed kernel allocation and reliance on input image and model knowledge for detail generation.
Problem
INR-based ASSR methods face limited continuous-signal reconstruction quality and low efficiency because they rely on coordinate mappings with repeated upsampling and decoding.
Method
ContinuousSR reconstructs continuous HR signals from LR images through a Pixel-to-Gaussian framework with Gaussian modeling, DGP-driven covariance weighting, and Adaptive Position Drifting.
Results
19.5× speed improvement and 0.90dB PSNR improvement are reported over state-of-the-art methods across seven benchmarks.
Takeaways & Limitations
ContinuousSR provides fast, high-quality arbitrary-scale super-resolution by rendering from a reconstructed continuous Gaussian field.
Takeaways & Limitations
Assigning one or four Gaussian kernels per LR pixel can waste resources in low-frequency regions and increase optimization difficulty, while detail generation may require additional visual knowledge.
Abstract
from arXiv · showhide
Arbitrary-scale super-resolution (ASSR) aims to reconstruct high-resolution (HR) images from low-resolution (LR) inputs with arbitrary upsampling factors using a single model, addressing the limitations of traditional SR methods constrained to fixed-scale factors (\textit{e.g.}, $\times$ 2). Recent advances leveraging implicit neural representation (INR) have achieved great progress by modeling coordinate-to-pixel mappings. However, the efficiency of these methods may suffer from repeated upsampling and decoding, while their reconstruction fidelity and quality are constrained by the intrinsic representational limitations of coordinate-based functions. To address these challenges, we propose a novel ContinuousSR framework with a Pixel-to-Gaussian paradigm, which explicitly reconstructs 2D continuous HR signals from LR images using Gaussian Splatting. This approach eliminates the need for time-consuming upsampling and decoding, enabling extremely fast arbitrary-scale super-resolution. Once the Gaussian field is built in a single pass, ContinuousSR can perform arbitrary-scale rendering in just 1ms per scale. Our method introduces several key innovations. Through statistical ana
1. Introdution
ASSR uses one model to reconstruct HR images at arbitrary scales, but INR methods face both representational and efficiency limitations. ContinuousSR addresses these challenges by directly modeling continuous HR signals with a Gaussian field, enabling fast arbitrary-scale rendering.
- Motivation: ASSR uses a single model to reconstruct HR images from LR inputs at arbitrary upsampling factors, unlike fixed-scale SR methods requiring separate models.Fixed-scale methods target factors such as ×2, ×3, and ×4, while ASSR supports flexible scaling.
- Limitations of Existing Methods: INR-based ASSR learns coordinate-to-pixel mappings, but repeated upsampling and decoding reduce efficiency and coordinate-based functions constrain continuous-signal reconstruction quality.Different scales require multiple time-consuming upsampling and decoding processes.
- ContinuousSR: ContinuousSR directly reconstructs continuous HR signals from LR images using a Pixel-to-Gaussian paradigm and Gaussian modeling.The method first reconstructs a continuous HR Gaussian field and samples from that representation.
- ContinuousSR: ContinuousSR uses Gaussian modeling to replace repeated upsampling and decoding with direct sampling from a continuous representation.This design targets both improved continuity and greater efficiency in arbitrary-scale super-resolution.
- Method Innovations: The framework introduces Deep Gaussian Prior analysis, DGP-Driven Covariance Weighting, and Adaptive Position Drifting to improve Gaussian-field reconstruction.The DGP is identified from natural images, while covariance weighting and position adaptation address Gaussian parameter optimization and spatial representation.
- Method Innovations: Gaussian Splatting provides the continuous representation used by ContinuousSR, building on 2D Gaussian methods developed for image processing.Gaussian functions are selected as continuous basis functions because combinations of Gaussians can represent complex continuous functions.
4. Proposed Method
ContinuousSR models continuous HR signals with 2D Gaussian functions, using priors and adaptive modules to make Gaussian-field reconstruction tractable from LR images. Its framework combines DGP-driven covariance weighting, adaptive position drifting, and color mapping.
- Continuous Basis Function: Gaussian functions are chosen as continuous basis functions because Gaussian mixtures can represent complex continuous functions and benefit from efficient Gaussian-splatting engineering.Each Gaussian kernel uses RGB, position, and covariance parameters.
- Direct End-to-End and Deep Gaussian Prior: Direct end-to-end learning from LR images to Gaussian parameters is difficult because Gaussian space contains numerous parameters, broad solution ranges, and local traps.The direct approach can fall into a local optimum, with PSNR remaining as low as 10 dB.
- Direct End-to-End and Deep Gaussian Prior: Analysis of approximately 40,000 natural images finds that about 99% of three covariance quantities fall within bounded ranges and generally follow Gaussian distributions, defining the Deep Gaussian Prior.The reported ranges are 0–2.4 for σ2_x, 0–2.2 for σ2_y, and −0.9–1.5 for ρσ_xσ_y.
- DGP-Driven Covariance Weighting: DGP-Driven Covariance Weighting samples predefined Gaussian kernels from the prior and learns weights to combine them into target covariance kernels.The candidate kernels cover most covariance types and ranges observed in natural images, providing prior information for network convergence.
- Adaptive Position Drifting: Adaptive Position Drifting initializes kernels at LR-pixel centers, then learns content-dependent offsets to improve optimization and representational capacity.Offsets are constrained to −1–1 using Tanh, and richer textures receive denser kernel placement.
- Color Gaussian Mapping: Color Gaussian Mapping learns RGB parameters from LR features, completing the three-component ContinuousSR framework with covariance weighting and position drifting.The mapping uses five multilayer perceptron layers.
5. Experiment and Analysis
Experiments evaluate ContinuousSR across seven benchmarks, multiple image-quality metrics, runtime, memory, and ablations. The method reports strong quality, speed, and module-level improvements under the tested settings.
- Experiment Setting: Seven benchmarks are used for evaluation, including Set5, Set14, B100, Urban100, Manga109, DIV2K validation, and LSDIR.DF2K images are used for training, with bicubic degradation generating LR inputs.
- Experiment Setting: PSNR, SSIM, FID, and DISTS are reported, with PSNR/SSIM computed on RGB for DIV2K and luminance for the other test sets.The evaluation protocol follows previous arbitrary-scale super-resolution work.
- Quantitative and Qualitative Results: 0.80 dB higher PSNR, 0.0172 higher SSIM, and 0.83 lower FID are reported on Urban100 at ×4 versus CiaoSR.The authors state that their method achieves the best performance across the evaluated metrics and benchmarks.
- Quantitative and Qualitative Results: Nearly 280 times faster runtime than CiaoSR is reported on LSDIR across the evaluated scale range.Runtime is averaged over 45 scales from ×4 to ×48; memory usage is also compared with an input size of 48×48 without tiling.
- Ablation Studies: Removing APD or DDCW significantly worsens optimization and PSNR, while using both Pinit and Poff gives the best reported representational capacity and performance.The ablations are conducted on Urban100 ×4 and also evaluate DGP-related components.
6. Future Work
The paper evaluates ContinuousSR beyond standard degradation by considering rainy low-resolution images and reports additional analyses and limitations in the appendix.
- Future Work: Rain200H is used to evaluate super-resolution on low-resolution rainy images generated with bicubic downsampling.The method is compared with three existing state-of-the-art approaches in this setting.
- Future Work: Additional comparisons, analyses, limitations, future work, and visual examples are provided in the appendix.The supplied passage does not specify particular future directions or limitations.
7. Conclusion
ContinuousSR uses a Pixel-to-Gaussian paradigm to reconstruct continuous HR signals for arbitrary-scale super-resolution. The paper reports improvements in reconstruction quality and speed across seven benchmarks.
- Conclusion: ContinuousSR reconstructs 2D continuous HR signals from LR images using Gaussian Splatting and combines DGP-driven covariance weighting with Adaptive Position Drifting.These components are presented as improving the quality and fidelity of reconstructed Gaussian fields.
- Conclusion: 19.5× speed improvement and 0.90 dB PSNR improvement are reported across seven benchmarks against state-of-the-art methods.The conclusion presents these figures as evidence of gains in both speed and quality for ASSR.
8. Details of Compared Methods
The compared-methods evaluation uses seven state-of-the-art or widely adopted models, selecting their best-performing networks from official repositories for fairness.
- Compared Methods: Seven comparison models are listed: MetaSR, LIIF, LTE, SRNO, CiaoSR, MambaSR, and GaussianSR.The methods span established arbitrary-scale super-resolution approaches and Gaussian-based super-resolution.
- Compared Methods: The best-performing network for each method is selected from its official GitHub repository for comparison.The supplied passage gives examples including SwinIR-based MetaSR and LTE, and RDN-based LIIF and SRNO.
9. User Study
The additional comparisons evaluate ContinuousSR across benchmarks, backbones, real data, complexity measures, and qualitative user assessment. Across these settings, the method is reported to retain strong reconstruction quality while improving runtime efficiency.
- ContinuousSR achieves state-of-the-art performance across Set5, Set14, B100, Urban100, and Manga109 when using the same SwinIR backbone.
- ContinuousSR requires the lowest FLOPs and inference time at a single scale on Manga109 while achieving the best PSNR.
- ContinuousSR consistently outperforms existing approaches on the real COZ dataset at ×5, supporting its reported generalization to real-world data.
- GaussianImage fails to learn the LR-to-HR mapping effectively and requires nearly 1 minute of optimization per scene on a V100 GPU.
- ContinuousSR reaches 27.14 dB at ×4 on LSDIR, exceeding GSASR’s best reported 26.73 dB, while generating HR images across scales in 1 ms.
11. Additional Ablation Study
The ablations examine latent covariance representation, covariance count, Gaussian-kernel count, positional-offset range, and visual reconstruction. They identify settings that improve performance while showing that excessive Gaussian allocation increases optimization complexity without further gains.
- Ablation Study on K: A latent covariance representation of dimension 512 performs better than the original three-dimensional representation.The covariance parameters are mapped into a latent space through a convolutional neural network to facilitate convergence.
- Ablation Study on number of K: Performance improves as the number of Gaussian covariances increases to 730, with no further gains beyond 730.
- Ablation Study on N: Using four Gaussian kernels per pixel achieves the best performance, whereas introducing too many kernels increases optimization complexity without further gains.
- Position Distribution: The proposed Adaptive Position Drifting adaptively adjusts Gaussian position distributions according to image structure, with richer textures receiving more adaptive positional modeling.
- Algorithm Workflow: The workflow includes feature encoding, color prediction, offset prediction, covariance estimation, and final image reconstruction.
15. More Exploration and Results
On low-resolution rainy images, ContinuousSR is evaluated across multiple scaling factors against two-stage and all-in-one methods. It consistently achieves higher reported reconstruction scores across these comparisons.
- At ×4, ContinuousSR achieves a PSNR of 24.51, exceeding DRSformer+LIIF at 20.14.
- At ×8, ContinuousSR reaches 22.76 PSNR, outperforming DRSformer+LIIF at 18.93.
- At ×5, ContinuousSR achieves 23.95 PSNR, exceeding GaussianSR at 23.51 and CiaoSR at 23.45.
- Across the evaluated scaling factors, the method is reported to outperform both two-stage and all-in-one alternatives for super-resolution and deraining.
16. Limitation and Future Work
The limitations concern Gaussian-kernel allocation and the difficulty of generating high-quality details at large upscaling factors. Future work proposes adaptive kernel allocation and additional visual or semantic knowledge for finer detail generation.
- Position Distribution: Assigning one or four Gaussian kernels per LR pixel can waste resources in low-frequency regions and significantly increase optimization difficulty.
- Position Distribution: Future work will explore allocating Gaussian kernels according to image texture complexity to mitigate resource waste and optimization difficulty.
- Introduce Generation Knowledge: Future work plans to incorporate diffusion-model visual knowledge or large vision-language-model semantic knowledge for finer details at high magnification.