PaperlayerView exact revision on arXiv

Source-linked AI summary

Local Texture Estimator for Implicit Representation Function

Jaewon Lee, Kyong Hwan Jin

arXiv:2111.08918v6cs.CVeess.IV

TL;DR

Standalone MLPs struggle with high-frequency image content because of spectral bias, limiting implicit representations for super-resolution. The paper proposes LTE, which estimates data-driven dominant frequencies and Fourier coefficients within a deep SR architecture; the resulting method characterizes textures in 2D Fourier space, supports arbitrary-scale reconstruction, and is reported as faster than prior works. Its frequency estimation becomes limited under severe aliasing unless the encoder receptive field is enlarged, increasing computation and memory costs.

  • Problem

    Standalone MLPs are biased toward low-frequency components and have limited ability to capture fine image details in arbitrary-scale super-resolution.

  • Method

    LTE jointly uses deep SR features to estimate image-specific dominant frequencies, Fourier coefficients, and phase information for an implicit decoder.

  • Results

    LTE characterizes natural-image textures in 2D Fourier space and enables arbitrary-scale reconstruction, while its implementation is reported faster than previous works across memory settings.

  • Takeaways & Limitations

    Estimating essential Fourier information helps an implicit representation prioritize high-frequency details during continuous image reconstruction.

  • Takeaways & Limitations

    Under severe aliasing, LTE's dominant-frequency estimates can become inconsistent with the ground-truth spectrum; enlarging the encoder receptive field raises computation and memory costs.

Abstract

from arXiv · show

Recent works with an implicit neural function shed light on representing images in arbitrary resolution. However, a standalone multi-layer perceptron shows limited performance in learning high-frequency components. In this paper, we propose a Local Texture Estimator (LTE), a dominant-frequency estimator for natural images, enabling an implicit function to capture fine details while reconstructing images in a continuous manner. When jointly trained with a deep super-resolution (SR) architecture, LTE is capable of characterizing image textures in 2D Fourier space. We show that an LTE-based neural function achieves favorable performance against existing deep SR methods within an arbitrary-scale factor. Furthermore, we demonstrate that our implementation takes the shortest running time compared to previous works.

1. Introduction

Arbitrary-scale super-resolution avoids separate models for each scale, but standalone MLPs exhibit spectral bias and struggle with fine details. LTE addresses this by estimating image-specific Fourier information, enabling implicit reconstruction of high-frequency textures while reducing computation time.

  • Arbitrary-scale SR restores images continuously with one network, avoiding separate models for each scale factor.
  • Standalone MLPs are biased toward low-frequency components and fail to capture fine details.This limitation is known as spectral bias.
  • LTE estimates dominant frequencies and corresponding Fourier coefficients so implicit functions can learn fine details at arbitrary resolution.
  • When jointly trained with deep SR architectures, LTE characterizes natural-image textures in 2D Fourier space.The paper reports this capability with EDSR, RDN, and SwinIR.
  • LTE-based implicit representation prioritizes learning high-frequency details when essential Fourier information is estimated.
  • LTE is reported to run faster than previous works regardless of memory setting.The study evaluates computation time under several memory conditions.

2. Related Work

Prior work addresses continuous and arbitrary-scale super-resolution through implicit functions, Fourier mappings, sinusoidal activations, and increasingly sophisticated deep SR architectures. LTE builds on this direction by integrating Fourier-domain texture estimation into an arbitrary-scale SR pipeline.

  • Implicit neural representation: Coordinate-based MLPs provide memory-efficient representations of continuous-domain signals, but pretrained MLPs require training for each unseen signal.
  • Spectral bias: Spectral bias limits standard ReLU MLPs on high-frequency textures, motivating sine activations and Fourier-feature mappings.
  • Deep SR architecture: Deep SR methods improve representation through residual blocks, dense residual blocks, channel attention, and non-local architectures.
  • Arbitrary-scale SR: Arbitrary-scale SR uses a single model across scale factors, while LIIF uses an MLP with continuous coordinates and latent variables.The passage also notes structural distortion at extreme scales for LIIF.
  • LTE architecture: The LTE pipeline combines an encoder, Fourier-domain LTE, decoder, and low-resolution skip connection for arbitrary-scale SR.

3. Problem Formulation

The paper formulates arbitrary-scale image super-resolution with a local implicit representation that maps latent features and continuous coordinates to RGB values. LTE addresses the MLP’s spectral bias by estimating local Fourier information, while scale-dependent phase estimation and an LR skip connection support reconstruction across resolutions.

  • Local implicit neural representation: Local implicit representation decodes RGB values from latent tensors and 2D continuous coordinates using an image-shared MLP.The latent tensor is produced by an encoder, and local latent codes around each query coordinate are combined through local ensemble weights.
  • Local implicit neural representation: The representation uses relative local-grid coordinates x − x_j to represent local areas in the HR image.The same local-grid coordinate is used for arbitrary output scale factors r_x × r_y.
  • Learning dominant frequency component: LTE transforms local coordinates into Fourier features so an implicit neural function can learn dominant frequency components and fine image details.LTE is introduced to overcome the low-frequency bias of standalone ReLU MLPs.
  • Learning dominant frequency component: LTE is shift-invariant and comprises amplitude, frequency, and phase estimators operating on latent codes and local-grid coordinates.The amplitude estimator outputs R2K values, the frequency estimator outputs RK×2 values, and the phase estimator maps 2D coordinates to RK values.
  • Learning dominant frequency component: The amplitude vector and frequency matrix are extracted from receptive-field latent codes to estimate dominant frequencies and Fourier coefficients.The encoder determines the receptive-field size, and unfolding concatenates the 3 × 3 nearest latent variables before estimation.
  • Scale-dependent phase estimation: Scale-dependent phase estimation uses the cell size, replacing it with max(c, c_tr) to handle edge-location changes across SR scales.The minimum training cell size c_tr is used because ReLU MLPs cannot extrapolate unseen nonlinear spaces effectively.
  • LR skip connection: An upscaled LR skip connection is added because LTE tends to predict frequencies near the low-frequency DC region.The connection is intended to prevent LTE from learning only the DC component and to enrich high-frequency residuals.

4. Method

The LTE-based arbitrary-scale SR network combines an encoder, LTE, a shared MLP decoder, and an LR skip connection. LTE estimates local amplitude, frequency, and phase information to reconstruct fine details continuously across scales.

  • Network architecture: The network combines an encoder, LTE, a shared four-layer MLP decoder, and an LR skip connection.EDSR-baseline, RDN, and SwinIR provide encoder features without upsampling layers.
  • LTE: LTE estimates amplitude, frequency, and phase using convolutional and fully connected estimators with sinusoidal activations.Amplitude and frequency estimators use 3×3 convolutions with 256 output channels, while the phase estimator receives the cell as input.
  • Training strategy: Training samples scales uniformly from ×1 to ×4 and evaluates both seen and unseen scales, including ×6–×30.The training procedure uses randomly sampled HR pixels and bicubic-downsampled LR counterparts.
  • Qualitative evaluation: The qualitative comparisons use RDN as the encoder for arbitrary-scale methods and compare LTE with LIIF at non-integer scales.Another comparison evaluates various encoders with LTE for ×6 SR.

5. Experiment

Experiments evaluate LTE quantitatively, qualitatively, and in Fourier space across datasets, scales, encoders, and ablations. LTE performs strongly across arbitrary-scale settings, while its components and encoder choice affect frequency capture and visual quality.

  • Quantitative results: LTE achieves the best performance for all tested scale factors regardless of whether EDSR-baseline, RDN, or SwinIR is used as encoder.The comparison is conducted on the DIV2K validation set using PSNR.
  • Quantitative results: 0.15dB is the maximum PSNR gain on Urban100 for ×3 within the RDN comparison.The benchmark comparison includes RDN, SwinIR, MetaSR, LIIF, and LTE.
  • Qualitative results: LTE captures high-frequency details without discontinuities, while MetaSR produces blocky artifacts and LIIF exhibits structural distortion.These qualitative comparisons use RDN as the encoder for all methods.
  • Qualitative results: LTE restores clearer printed-text edges than LIIF across non-integer scale factors.The reported improvements are especially visible in several letters across the compared text images.
  • Ablation study: Removing phase estimation causes a significant performance drop, and removing the LR skip connection consistently reduces LTE quality.These effects are reported in the component ablation comparisons.
  • Fourier analysis: LTE follows the ground-truth dominant frequencies of horizontal, vertical, and diagonal textures using local receptive-field information.The corresponding Fourier coefficients are obtained from pixels inside the encoder-defined receptive field.
  • Fourier analysis: SwinIR-LTE captures diagonal dominant frequencies, whereas EDSR-baseline-LTE estimates only low-frequency components.The authors associate the stronger Fourier-space estimate with the more powerful encoder.
  • Ablation study: Removing amplitude estimation equalizes frequency coefficients and shifts learning toward low-frequency content, while reducing estimated frequencies harms high-frequency detail capture.The ablations also examine phase estimation and the LR skip connection through Fourier-space visualizations.

6. Discussion

LTE estimates Fourier information for natural images, but severe aliasing limits frequency accuracy and large-scale reconstruction can introduce ringing. Its LTE+ variant addresses computation and memory costs when querying all output pixels at once.

  • DFT and LTE: LTE with a deep neural encoder estimates accurate Fourier information for HR images, unlike DFT applied to aliased LR images.DFT uses a linear combination of image intensity, whereas LTE uses a multi-chain encoder with linear combinations and nonlinear activations.
  • Aliasing: Under severe aliasing, SwinIR-LTE estimates dominant frequencies inconsistent with the ground-truth spectrum.Extending the encoder’s receptive field can resolve this limitation but increases computation and memory costs.
  • Gibbs phenomenon: LTE may cause overshoot at large scale factors, such as ×12, due to the Gibbs phenomenon.The paper identifies smoothing algorithms as a future direction for alleviating ringing artifacts.
  • Computation time: LTE takes the shortest computation time under memory-limited evaluation while increasing memory usage.The comparison uses 96 × 96 output pixels per query on an NVIDIA RTX 3090 24GB.
  • Computation time: LTE+ takes shorter computation time and consumes less memory than previous works when all output pixels are queried at once.LTE+ replaces the shared MLP decoder with a GPU-friendly 1 × 1 convolution.

7. Conclusion

The paper proposes LTE-based arbitrary-scale super-resolution to overcome spectral bias in implicit neural functions. The resulting method combines deep feature extraction, Fourier estimation, and implicit reconstruction, with reported gains in performance, visual quality, and computation time.

  • 7. Conclusion: LTE-based arbitrary-scale SR combines a deep SR encoder, LTE, and an implicit representation function.The encoder extracts LR-sized feature maps, LTE estimates dominant frequencies and Fourier coefficients, and the implicit function reconstructs arbitrary-resolution images.
  • 7. Conclusion: LTE estimates dominant frequencies and corresponding Fourier coefficients for natural images from deep SR network features.Scale-dependent phase and an LR skip connection bias LTE toward high-frequency textures.
  • 7. Conclusion: The LTE-based neural function outperforms other arbitrary-scale SR methods in performance and visual quality with the shortest computation time.
Loading 2111.08918v6…