Source-linked AI summary
Learning Continuous Image Representation with Local Implicit Image Function
Yinbo Chen, Sifei Liu, Xiaolong Wang
TL;DR
The paper asks how to represent images continuously despite machines storing them as resolution-limited pixel arrays. It proposes LIIF, which decodes RGB values from coordinates and local latent features, and reports arbitrary-resolution representation, extrapolation beyond training scales, and better handling of varied-resolution ground truths.
Problem
Pixel-based image representations are limited by resolution, while resizing images or ground truths to fixed sizes can sacrifice fidelity.
Method
LIIF represents each image with a 2D feature map and shared decoder that predicts RGB values from a coordinate and neighboring feature vectors, trained with self-supervised super-resolution.
Results
LIIF generalizes to much higher precision than training scales, including ×30 higher resolution, and significantly outperforms resizing-based methods for size-varied ground truths.
Takeaways & Limitations
LIIF bridges discrete and continuous 2D representations and naturally exploits image ground truths available at different resolutions.
Takeaways & Limitations
At out-of-distribution high resolutions, cell decoding can hurt PSNR, raising uncertainty about whether learned cells generalize across scales.
Abstract
from arXiv · showhide
How to represent an image? While the visual world is presented in a continuous manner, machines store and see the images in a discrete way with 2D arrays of pixels. In this paper, we seek to learn a continuous representation for images. Inspired by the recent progress in 3D reconstruction with implicit neural representation, we propose Local Implicit Image Function (LIIF), which takes an image coordinate and the 2D deep features around the coordinate as inputs, predicts the RGB value at a given coordinate as an output. Since the coordinates are continuous, LIIF can be presented in arbitrary resolution. To generate the continuous representation for images, we train an encoder with LIIF representation via a self-supervised task with super-resolution. The learned continuous representation can be presented in arbitrary resolution even extrapolate to x30 higher resolution, where the training tasks are not provided. We further show that LIIF representation builds a bridge between discrete and continuous representation in 2D, it naturally supports the learning tasks with size-varied image ground-truths and significantly outperforms the method with resizing the ground-truths.
1. Introduction
LIIF addresses the mismatch between continuous visual content and resolution-limited pixel representations by modeling images as continuous functions. It uses local latent features and coordinate queries, trained through super-resolution, to support arbitrary resolutions and size-varied ground truths.
- 1. Introduction: Pixel-based image representations are constrained by resolution, and resizing varied-resolution ground truths can sacrifice fidelity.The paper motivates replacing fixed-resolution representations with continuous image functions.
- 1. Introduction: LIIF represents natural and complex images continuously using latent codes distributed spatially and a decoder that predicts RGB values from coordinates and local codes.The representation connects each queried coordinate with nearby learned features.
- 1. Introduction: LIIF can be presented at arbitrary resolution and extrapolates to ×30 higher resolution beyond the training tasks.This supports representing images at precision levels not provided during training.
- 1. Introduction: The encoder learns LIIF representations through self-supervised super-resolution with continuously changing up-sampling scales.Pixel-based inputs are used to predict higher-resolution counterparts.
- 1. Introduction: For size-varied image ground truths, LIIF avoids resizing during end-to-end training and achieves significantly better results than resizing-based methods.The representation can exploit information from different resolutions directly.
2. Related Work
The related work situates LIIF among implicit neural representations, learned function spaces, image generation, and super-resolution. It emphasizes that continuous image representation remains less explored and differs from fixed-scale super-resolution.
- Implicit neural representation: Implicit neural representations commonly use multilayer perceptrons to map coordinates to signals in 3D shapes, scenes, and appearances.The paper places LIIF within this coordinate-to-signal modeling paradigm.
- Learning implicit function space: Recent methods share implicit-function spaces across objects by associating latent codes with instances rather than optimizing an independent representation for each object.Auto-decoder approaches are identified as one way to obtain these latent codes.
- Learning implicit function space: Applications of implicit neural representation to natural images are relatively underexplored, and ReLU-based MLPs have been reported as incapable of representing fine image details.This motivates specialized approaches for continuous image representation.
- Image generation and super-resolution: LIIF is related to image-to-image translation and super-resolution but supports realistic high-resolution output through a continuous representation rather than deconvolution-based approaches.The paper distinguishes its focus from traditional super-resolution models designed for specific up-sampling scales.
3. Local Implicit Image Function
LIIF represents an image with spatially distributed latent codes and a shared decoder that maps coordinates and local features to RGB values. Local ensemble prediction, feature unfolding, and cell decoding address continuity, local context, and resolution-aware rendering.
- Representation: LIIF represents each continuous image as a 2D feature map of evenly distributed latent codes decoded by a shared MLP.Each latent code represents a local image piece and predicts signals for nearby coordinates.
- Representation: The decoder uses the nearest latent code and its coordinate to predict the RGB value at a queried continuous coordinate.The coordinate-to-signal mapping enables the representation to be evaluated across the continuous image domain.
- Feature unfolding: Feature unfolding enriches each latent code by concatenating its 3×3 neighborhood before subsequent LIIF computations.The unfolded feature map replaces the original feature map for computation, with zero-padding outside borders.
- Local ensemble: Local ensemble blends four neighboring latent-code predictions with normalized area-based weights to produce smooth transitions across local regions.The ensemble addresses discontinuities caused by abruptly switching the nearest latent code as the query coordinate moves.
- Cell decoding: Cell decoding adds the query pixel’s height and width to the decoder input, allowing RGB prediction to depend on pixel shape as well as position.This retains information from the queried pixel area when rendering the continuous representation at a specified resolution.
4. Learning Continuous Image Representation
The method trains an encoder and shared decoder to generate a continuous image representation from pixel-based inputs. Random-scale downsampling, coordinate queries, and reconstruction loss teach the representation to preserve fidelity at higher resolutions.
- Method: An encoder maps an image to a 2D feature map, while a shared function decodes the representation for continuous image reconstruction.The framework jointly trains the encoder and shared function to generate a continuous representation for unseen images.
- Training procedure: Training uses randomly downsampled inputs and high-resolution pixel samples as ground truth for coordinate-based prediction.The encoder produces the LIIF representation, which is queried at ground-truth pixel coordinates to predict RGB values.
- Training procedure: The reconstruction objective compares predicted RGB signals with ground-truth pixel values using L1 loss.Batch training averages the loss across sampled training instances.
- Evaluation: Figure 5 evaluates continuous representations by comparing 48 × 48 inputs with ×30 presentations after training on random scales.The comparison includes independently fitted 1-SIREN and encoder-based MetaSR and LIIF representations.
- Evaluation: Figure 6 demonstrates a generated LIIF representation presented at ×20 higher resolution than its input.The figure provides a visual example of high-resolution presentation from the learned representation.
5. Experiments
Experiments evaluate LIIF on continuous image representation, arbitrary-scale super-resolution, and size-varied ground truths. LIIF remains effective at unseen scales, although some design choices improve visual quality without improving PSNR at extreme scales.
- Continuous image representation: LIIF uses one model across continuously sampled ×1–×4 scales, whereas EDSR-baseline and RDN train separate models for different scales.MetaSR and LIIF use continuous random scales, while EDSR-baseline and RDN rely on scale-specific models.
- Continuous image representation: For out-of-distribution scales, LIIF outperforms MetaSR, with the advantage becoming more apparent as the scale increases.EDSR-baseline and RDN cannot be directly applied to out-of-distribution scales because they are trained for specific scales.
- Continuous image representation: At ×30, LIIF produces visually pleasing, high-fidelity results despite training only on ×1–×4, while MetaSR shows discontinuity and 1-SIREN lacks cross-image knowledge.The comparison evaluates MetaSR and LIIF at ×30 and independently fits 1-SIREN to each test image.
- Ablation study: Cell decoding improves visual results at in-distribution and large out-of-distribution scales, but its benefit may not appear in PSNR for some extreme scales.At ×30, cell-1/30 produces clearer edges than cell-1/1, cell-1/2, and no-cell.
- Ablation study: Training with random scales generally transfers better across in-distribution scales than training with a single scale, while ×4-only can achieve slightly higher out-of-distribution PSNR.The authors hypothesize that ×4-only is biased toward high scales and that cell decoding can improve visual quality without improving PSNR.
- Ablation study: Feature unfolding mainly helps moderate scales, local ensemble consistently improves quality, and deeper decoding benefits both in-distribution and out-of-distribution scales.These findings come from ablations removing feature unfolding, local ensemble, or decoder depth.
- Size-varied ground truths: For size-varied ground truths, LIIF significantly outperforms resizing targets to one size and training with classical up-sampling modules.LIIF uses information from different ground-truth resolutions without resizing, preserving their fidelity.
6. Conclusion
LIIF provides a continuous image representation in which a shared decoder predicts RGB values from coordinates and neighboring feature vectors. Self-supervised super-resolution training enables high-resolution generalization and use of differently sized ground-truths.
- LIIF represents each image as a 2D feature map and uses a shared decoder to output RGB values from coordinates and neighboring feature vectors.
- Self-supervised super-resolution training generates continuous LIIF representations from pixel-based images.
- The continuous representation generalizes to much higher precision than the training scales while maintaining high fidelity.
- LIIF builds a bridge between discrete and continuous 2D representations and exploits image ground-truths available at different resolutions.
- Better decoder architectures and applications to other image-to-image tasks remain directions for future work.