Source-linked AI summary
Implicit neural representation of textures
Albert Kwok, Zheyuan Hu, Dounia Hammou
TL;DR
Texture INRs address the challenge of representing textures continuously while balancing image quality, memory usage, and rendering time. The paper evaluates neural architectures and related rendering applications, finding that INRs can outperform classical compression methods across metrics at similar bitrates, while requiring careful hyperparameter selection.
Problem
Textures consume substantial GPU memory and processor energy, motivating compact continuous representations that preserve rendering quality.
Method
The paper evaluates texture INRs using MLPs, SIRENs, and positional encodings, integrates them into a renderer, and explores mipmap fitting and INR weight-space generation.
Results
INRs can outperform classical compression methods across all metrics for similar bitrates, although they can struggle with exact details.
Takeaways & Limitations
Fourier encoding and SIRENs significantly improve simple MLP performance, while hyperparameter tuning can substantially alter results across image types.
Takeaways & Limitations
Frequency values for SIRENs and Fourier MLPs were not tuned because of time and computation constraints, potentially affecting performance.
Abstract
from arXiv · showhide
Implicit neural representation (INR) has proven to be accurate and efficient in various domains. In this work, we explore how different neural networks can be designed as a new texture INR, which operates in a continuous manner rather than a discrete one over the input UV coordinate space. Through thorough experiments, we demonstrate that these INRs perform well in terms of image quality, with considerable memory usage and rendering inference time. We analyze the balance between these objectives. In addition, we investigate various related applications in real-time rendering and down-stream tasks, e.g. mipmap fitting and INR-space generation.
1. Introduction
The paper presents texture INRs that map continuous UV coordinates to RGB values, replacing discrete texture representations with compressed neural weights. It evaluates multiple INR designs and integrates them into rendering and downstream texture tasks.
- INRs represent continuous coordinate-based signals with neural networks rather than discrete grids or explicit parametrizations.
- Textures motivate INRs because they consume substantial GPU memory and processor energy, while compact representations support downstream material generation and asset baking.
- Tiny MLPs or related graphics models take UV coordinates as input and output RGB color, with optimized weights forming a compressed texture representation.
- The work implements four texture INRs, evaluates three for performance, efficiency, and memory usage, and integrates INR reconstruction into Mitsuba 3 rendering.
- The study also explores downstream applications including mipmap fitting and INR weight-space generation.
2. Related work
Prior texture compression uses methods such as JPEG and ASTC, while INR research uses neural networks to represent continuous signals. This work examines periodic activations and positional encoding schemes for texture INRs.
- Texture compression: JPEG applies discrete cosine transform-inspired image compression, while ASTC provides lossy block-based compression from 1 to 8 bits per pixel.
- INR design: INR design treats an overfit MLP as an implicit representation of images sampled from discrete data.
- Mitsuba 3 is a customizable Python / C++ renderer used as the rendering context for the paper’s INR integration.
- INR design: Periodic activation functions have been proposed for MLPs as efficient signal representations.
- INR design: Positional encoding maps low-dimensional inputs to higher-dimensional features; the paper considers Fourier and multiresolution hash encoding for texture INRs.
3. Method
The method builds texture INRs by training neural networks that map UV coordinates to RGB values, then extends them to mipmap fitting and INR-space generation. It evaluates multiple MLP designs, positional encodings, model sizes, and generative approaches for texture representations.
- 3.1. Dataset analysis and sample selection: The dataset uses 5,640 images across 47 Describable Textures Dataset categories, selecting diverse samples with Laplacian Variance as a sharpness measure.The selection criterion is FocusMeasure := LAPV(I) = Var(∆I(u, v)), and 25 images are sampled at regular histogram intervals.
- 3.2. INR architectures: SIREN replaces ReLU with sinusoidal activation functions, while Fourier encoding concatenates UV coordinates with frequency features before MLP processing.Frequency bands are treated as hyperparameters and adjusted using validation results.
- 3.2. INR architectures: Multiresolution hash encoding is implemented but its performance benefit is considered limited at the available texture resolution, with higher-resolution assessment deferred.The paper explicitly treats this higher-resolution evaluation as future work.
- 3.2. INR architectures: An MLP maps UV coordinates to RGB values, with optimized weights serving as a compressed texture representation.The evaluated networks include a pure MLP, SIREN, and an MLP using Fourier positional encoding.
- 3.2. INR architectures: The experiments vary hidden-layer counts, widths, and optimizers to compare architecture performance against bitrate, with Adam generally producing better results.Models use 1, 2, or 3 hidden layers of sizes 128, 256, or 512, except for 3 layers of 512.
- 3.3. Mipmap fitting: Mipmap fitting adds a normalized level-of-detail input t ∈ [0, 1] and trains on six downscaled mipmap levels paired with UV coordinates.The augmented training removes hallucinations at points not explicitly seen during training; bilinear downsampling creates the mipmap images.
- 3.4. INR-space generation: INR-space generation applies diffusion or VAE-based generative methods to MLP weights, treating generated weights as novel textures after inference.Evaluation focuses on fidelity and coverage between synthetic and reference texture sets, with RGB permutation used for data augmentation.
4. Evaluation
The evaluation compares texture INRs using pixel-wise, perceptual, and structural metrics, while examining architecture artefacts, optimizer behavior, compression quality, efficiency, rendering, mipmapping, and INR-space generation.
- Metrics: The evaluation measures reconstructed texture quality against ground truth using MAE, MSE, PSNR, SSIM, LPIPS, and VMAF.These cover pixel-wise errors alongside perceptual and structural similarity.
- Quantitative performance: Fourier MLPs performed best overall on SSIM, VMAF, and PSNR, while Fourier MLPs and SIRENs performed very well on LPIPS.SIRENs showed a noticeable LPIPS drop at very high compression, and all models had broadly similar curves for some metrics.
- Qualitative architecture evaluation: Pure MLPs consistently produced blurry results, Fourier MLPs often produced line artefacts and grainy noise, and SIRENs produced persistent lumpy curve artefacts.The artefacts affected the models differently, with SIRENs struggling particularly on geometric textures.
- Optimizer comparison: Adam produced better and more consistent results than Rprop, whose outputs were blurrier and sometimes contained additional learning artefacts.The authors note that the pure-MLP Adam–Rprop difference shown is unusually large because neither optimizer usually learned those images accurately.
- Compression comparison: Compared with ASTC, the best Fourier MLPs achieved consistently better results, while ASTC offered good compression ratios but sacrificed quality, especially in LPIPS.The INR advantage in VMAF, SSIM, and PSNR was noisier and less significant than its LPIPS gain.
- Efficiency: Training took 50–200 seconds for 50 iterations at roughly 0.5–2 iterations per second, while teaser rendering took 4.7 seconds for spp = 1 and seed = 1 on an Apple M1 CPU.Rendering time scales proportionally with samples per pixel and aggregated seeds.
- Downstream tasks: Models fit mipmapped textures well with similar results to non-mipmapped textures, whereas INR-space generation had imperfect quality despite convergence.The generative limitation may reflect the scarce training set of around 600 INR weights.
5. Conclusions
The paper evaluates texture INRs across performance, efficiency, memory usage, and complexity, finding strong perceptual compression with trade-offs in exact detail and hyperparameter sensitivity.
- INRs can be effective for perceptual image compression but may struggle with exact details.
- INRs can outperform classical compression methods across all metrics at similar bitrates.
- Fourier encoding and SIRENs provide significant improvement over simple MLPs, while hyperparameter tuning can substantially alter performance across image types.
6. Future directions
Future work targets more reliable hyperparameter selection, broader sampling and multi-image compression, and richer mipmap, filtering, and material representations.
- A consistent hyperparameter-selection process is needed for reliable INR compression, especially because SIREN and Fourier frequency values were not tuned.
- Training on more randomly sampled points may prevent grid-related artefacts.
- Encoding multiple images through mipmaps can preserve quality and may enable higher compression ratios for image libraries or animations.
- Future mipmap work could reduce GPU RAM use for anisotropic filtering and model filtering using viewing angle and level of detail.
- INRs could efficiently represent SVBSDFs, enabling more expressive materials to be stored in memory.
8. Selected dataset
The selected texture set is visualized in Figure 18.
- Figure 18 visualizes the selected 25 textures.
10. Addition results
Additional results include visualizations of sampled textures, training and reconstruction behavior, mipmap rendering, and real-time INR rendering from MLP weights.
- For bubbly 0122, Figure 19 shows the training loss, reconstructed texture, and residual error, with the error metric reported in Table 1.
- Figure 20a demonstrates a three-colour-channel mipmap pyramid for bubbly 0122.
- Integration with Mitsuba 3 enables real-time material rendering from MLP weights, illustrated by the INR plugin result in Figure 20b.
- Figure 18 visualizes selected textures together with their Laplacian responses.
- Figure 20a presents the input image mipmap used for the rendering comparison.
11. Proposal
The proposal investigates continuous neural representations for textures, focusing on architecture choices, high-frequency fidelity, efficiency, and downstream rendering applications. It includes implementation, evaluation, and extensions covering mipmaps, filtering, compression, and representation-space generation.
- The project studies how architecture, activation, initialization, and input encoding affect spectral capacity, smoothness, and derivative accuracy in coordinate-based networks.
- SIREN-style multilayer perceptrons use sinusoidal activations and specialized initialization to represent high-frequency signals and well-behaved derivatives.
- Evaluation uses PSNR/SSIM, spectral error, differential-error L2 norms, and ablations of activation, initialization, network size, and encoding bandwidth.
- Extensions consider anisotropic texture filtering, compressing multiple similar textures into one representation, and comparing compression efficiency with quality.