Source-linked AI summary
Deep Decoder: Concise Image Representations from Untrained Non-convolutional Networks
Reinhard Heckel, Paul Hand
TL;DR
Existing image-generating neural networks are typically overparameterized, convolutional, and trained on large datasets, motivating a concise architecture whose structure alone models natural images. The paper proposes the untrained, non-convolutional deep decoder, whose underparameterization supports image compression and inverse problems. Its compression is on par with wavelet thresholding, while its capacity constraint provides a barrier to overfitting and supports denoising.
Problem
The paper asks whether a neural network can model natural images with few parameters using architecture alone, without training data or algorithmic assistance.
Method
The deep decoder is an untrained, underparameterized, non-convolutional image model whose network weights generate the image while the input remains fixed.
Results
The deep decoder achieves compression on par with wavelet thresholding and supports inverse problems including super-resolution, where it outperforms bicubic interpolation and is on-par with the deep image prior.
Takeaways & Limitations
Underparameterization provides a barrier to overfitting, while the architecture itself supplies the data model without critical reliance on regularization or training.
Takeaways & Limitations
Nearest-neighbor upsampling can impose locality too strongly, making nearby pixel squares identical and unable to fit local variation within natural images.
Abstract
from arXiv · showhide
Deep neural networks, in particular convolutional neural networks, have become highly effective tools for compressing images and solving inverse problems including denoising, inpainting, and reconstruction from few and noisy measurements. This success can be attributed in part to their ability to represent and generate natural images well. Contrary to classical tools such as wavelets, image-generating deep neural networks have a large number of parameters---typically a multiple of their output dimension---and need to be trained on large datasets. In this paper, we propose an untrained simple image model, called the deep decoder, which is a deep neural network that can generate natural images from very few weight parameters. The deep decoder has a simple architecture with no convolutions and fewer weight parameters than the output dimensionality. This underparameterization enables the deep decoder to compress images into a concise set of network weights, which we show is on par with wavelet-based thresholding. Further, underparameterization provides a barrier to overfitting, allowing the deep decoder to have state-of-the-art performance for denoising. The deep decoder is simple in the sense that each layer has an identical structure that consists of only one upsampling unit, pixel-wise linear combination of channels, ReLU activation, and channelwise normalization. This simplicity makes the network amenable to theoretical analysis, and it sheds light on the aspects of neural networks that enable them to form effective signal representations.
1 Introduction
The paper asks whether an underparameterized neural network can model natural images effectively without convolutions, training data, or algorithmic regularization. It proposes the deep decoder as a simple image model for compression and inverse problems.
- Motivation: Deep neural networks for imaging commonly use over-parameterization, convolutions, and large training datasets.These networks often outperform traditional image models for compression, denoising, and related tasks.
- Research question: The deep decoder addresses whether architecture alone can provide an underparameterized model of natural images.The motivating question concerns efficient natural-image representations without training or algorithmic assistance.
- Deep decoder: The proposed network represents natural images with very few parameters and requires no training, while incorporating the data assumptions in its architecture.It is designed for compression, denoising, and other inverse problems.
- Key contributions: Underparameterization maps a lower-dimensional parameter space to higher-dimensional images, enabling compression and creating a barrier to overfitting.The paper reports compression on par with wavelet thresholding and uses the resulting capacity constraint to regularize inverse problems.
- Relation to prior work: Unlike the deep image prior, the deep decoder does not critically rely on early-stopping regularization, and its shared network and code can serve multiple applications.The paper also notes reduced sensitivity to training-test distribution misfit because no learning is involved.
- Architecture and analysis: The deep decoder uses no convolutions, instead imposing nearby-pixel relationships through upsampling and sharing pixelwise channel-combination weights across spatial positions.Its few repeated building blocks also make the architecture amenable to theoretical analysis.
2 Concise image representations with a deep image model
This section evaluates whether an untrained, underparameterized deep decoder can represent natural images concisely. Across 100 ImageNet validation images, its compression is comparable to wavelet compression, with the relative advantage depending on compression factor.
- Concise representations: The deep decoder is an untrained, non-convolutional image model whose concise representations are reported to be on par with state-of-the-art wavelet thresholding.The section uses concise approximation with few parameters as evidence that the model describes natural images effectively.
- Model: The model maps N parameters to an image of dimension n with n ≫ N, making its network weights a lower-dimensional representation of the image.The input tensor is fixed, so the generated image depends on the optimized network parameters C.
- Experimental setup: 100 randomly sampled 512x512 color ImageNet validation images were fitted with the deep decoder and evaluated using peak signal-to-noise ratio.The comparison used the N-largest wavelet coefficients for each image and an equal parameter count.
- Results: At compression factor 32.3, the deep decoder representation is slightly better for most images, whereas wavelets are slightly better at compression factor 8.The figure identifies deep-decoder-favored cases as crosses above the red comparison line.
- Implications: The experiment supports using the deep decoder for lossy compression by quantizing and storing its optimized coefficients.The paper reports that coefficient perturbations do not substantially affect image quality and contrasts this with learned encoder-decoder approaches.
3 The deep decoder
The deep decoder uses fixed random inputs, shared pixelwise channel combinations, upsampling, ReLUs, and channel normalization to represent images with far fewer parameters than pixels. Its non-convolutional, underparameterized design supports concise representations, and matched-parameter simulations favor 1 × 1 channel combinations over spatial convolutions.
- Architecture: The decoder transforms a fixed random tensor into an image through pixelwise channel combinations, upsampling, ReLUs, and channel normalization.The coefficient matrices contain the trainable network weights, while combinations are shared consistently across spatial positions.
- Architecture: The deep decoder is underparameterized, with output dimension n much larger than parameter count N.The model is defined as G: R^N→R^n with n ≫ N, and the fixed input is not counted among its parameters.
- Architecture: The default six-layer architecture uses 25,536 or 100,224 parameters for RGB images with 786,432 dimensions, depending on whether k equals 64 or 128.The parameter count is N = dk^2 + 2dk + 3k for the stated default settings.
- Non-convolutional design: Pixelwise channel combinations are not proper convolutions because they share weights across positions without spatially coupling pixels.Most comparison networks instead use convolutional filters with nontrivial spatial extent.
- Non-convolutional design: With matched parameter counts, the p = 1 deep decoder represents images about 1 dB better than the p = 3 variant, depending on the image.The comparison uses k = 64 for p = 1 and k = 22 for p = 3, keeping parameter counts essentially equal.
4 Solving inverse problems with the deep decoder
The deep decoder solves denoising, super-resolution, and inpainting by fitting its image model to observations through a forward operator. It achieves performance on par with strong untrained methods in denoising and super-resolution, while the deep image prior performs slightly better on average for inpainting.
- General inverse-problem procedure: The method models inverse problems as recovering x from y = f(x) + η, then fits the deep decoder parameters to the observation.The estimated image is G(Ĉ), where Ĉ minimizes the observation loss.
- General inverse-problem procedure: The fitting problem is non-convex and uses least-squares loss with Adam, although gradient descent gives comparable results.The loss can be adapted to the structure of the noise.
- Denoising: Denoising performance is on par with the untrained DIP method and BM3D algorithm.The comparison is restricted to other state-of-the-art untrained methods rather than learned methods.
- Denoising: Underparameterization filters out a significant proportion of noise while retaining most of the signal.The paper attributes this behavior to the decoder representing natural images well but only a small proportion of noise.
- Super-resolution: For fourfold downsampling, the deep decoder outperforms bicubic interpolation and is on par with the deep image prior.The super-resolution experiment uses Lanczos downsampling and k = 128.
- Inpainting: In inpainting, the deep decoder performs well, but the deep image prior performs slightly better on average over the examples considered.The inpainting experiment uses a more expressive prior with k = 320.
5 Related work
The deep decoder is positioned among untrained and trained image-generating methods, especially DIP, while differing in parameterization, regularization, and convolutional structure. Experiments show both models fit images well, but only the deep decoder inherently limits noise fitting.
- Positioning: The deep decoder is related to untrained image models such as wavelets, curvelets, and the Deep Image Prior, as well as trained generative networks.Its weights are optimized for the target inverse problem, unlike methods whose weights are fixed after training.
- Deep Image Prior: DIP is over-parameterized and critically relies on early stopping and input noise, whereas the deep decoder is under-parameterized and does not require regularization.Regularization can nevertheless enhance deep-decoder performance.
- Architectural contrast: The DIP uses convolutional layers, while the deep decoder uses no convolutions and obtains spatial coupling only through upsampling.Its pixelwise channel combinations share weights spatially but do not couple neighboring pixels.
- Denoising comparison: With sufficiently many iterations, both DIP and the deep decoder fit the image well, but the deep decoder cannot fit noise well even after extensive optimization.The paper formally attributes this noise-fitting limit to underparameterization, while DIP relies on fitting structured images faster than noise.
- Denoising comparison: Deep-decoder denoising follows from retaining most of the signal while filtering out a significant proportion of noise.The deep decoder’s underparameterization supplies this filtering behavior without DIP’s critical dependence on early stopping.
6 Discussion on what makes the decoder work
The discussion attributes the deep decoder’s behavior to underparameterization, upsampling, and simple signal-building operations. These components jointly support natural-image representation while limiting noise fitting and shaping locality and smoothness.
- Noise fitting: The deep decoder is shown empirically and theoretically to fit only a small proportion of noise relative to its degree of underparameterization.The discussion connects this property to the network’s ability to represent natural images well.
- Noise fitting: The one-layer analysis treats the network as mapping a low-dimensional parameter space into a higher-dimensional output space.The argument ignores batch normalization and does not require a particular upsampling matrix.
- Upsampling: Upsampling is the deep decoder’s only explicit source of locality, so its choice strongly affects the character of the resulting signal estimates.Without upsampling, pixels are decoupled and random permutations of a fit image remain equally fit-able.
- Upsampling: Linear upsampling acts as an indirect signal prior that promotes piecewise smoothness.Figure 5 compares linear and convex nonlinear upsampling on a one-dimensional piecewise smooth signal.
- Upsampling: Nearest-neighbor upsampling produces piecewise-constant patches that impose locality too strongly for local variation in natural images.When dimensions double at each layer, the constant patches span 2^d × 2^d pixels.
A Proof of Proposition 1
The proof bounds how much Gaussian noise a one-layer deep decoder can fit by partitioning its range into low-dimensional subspaces and controlling noise projections onto them.
- Subspace construction: For fixed activation patterns, the one-layer deep decoder output lies in a subspace of dimension at most k^2.The proof embeds the range in a k^2-dimensional subspace before bounding projected noise energy.
- Counting subspaces: The number of possible subspaces is bounded by n^(k^2), using a bound on the number of attainable sign patterns.Each sign-pattern collection determines one subspace in the proof.
- Noise projection bound: A Gaussian projection bound controls the noise energy in each fixed subspace before a union bound extends the control across all possible subspaces.The proof invokes Lemma 2 for a subspace of dimension k^2.
- Conclusion: The proposition’s conclusion is established by combining the subspace count with the Gaussian projection inequality under the stated parameter-size assumption.The final step chooses β = 2 log(n0) and uses k^2 < n/(32 log n0).
A.1 Proof of Lemma 1
The proof counts the activation sign patterns induced by homogeneous hyperplanes, then applies a standard upper bound on the number of resulting partitions.
- Sign-pattern counting: The number of sign patterns (Av > 0) equals the maximum number of regions formed by n hyperplanes through the origin in a k-dimensional space.The hyperplanes are perpendicular to the rows of A.
- Sign-pattern counting: For k ≥ 5, the proof applies the stated upper bound on the number of hyperplane-induced partitions.This bound supplies the combinatorial estimate used in the preceding subspace count.
B Sensitivity to parameter perturbations and distribution of parameters
The deep decoder’s image representation is relatively robust to perturbations of weights in individual layers, while fitted weights are approximately Gaussian distributed.
- Sensitivity to parameter perturbations: Perturbing weights in individual layers while keeping other weights and the input fixed allows layer-wise sensitivity to be measured in the image domain.The experiment fits Barbara with a six-layer deep decoder, adds Gaussian noise to one layer’s weight matrix at a time, and measures image PSNR.
- Sensitivity to parameter perturbations: The deep decoder’s representation is relatively insensitive to perturbations of its coefficients.The reported sensitivity curves are obtained by varying the perturbation signal-to-noise ratio for each layer.
- Distribution of parameters: The fitted network weights for the Barbara test image are approximately Gaussian distributed.Figure 8 shows the weight distribution together with a Gaussian fit.