Source-linked AI summary
Recurrent Inference Machines for Solving Inverse Problems
Patrick Putzky, Max Welling
TL;DR
Existing inverse-problem methods balance prior expressiveness against optimization difficulty, while learned iterative RNNs remain shaped by hand-chosen model and inference components. RIMs train an RNN to learn an inference algorithm from the data and task, without explicitly specifying those components. The authors report state-of-the-art image-restoration results and across-task generalization.
Problem
MAP inference must balance prior expressiveness against optimization difficulty, and learned iterative methods still depend on choices of model and inference components.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
A RIM trained on deconvolution can be applied to inpainting by swapping likelihood gradients while keeping the rest of its parameters fixed.
Source: §1 · p. 2Experiments focus on linear image-reconstruction tasks, although the authors say the framework can also apply to nonlinear inverse problems.
Source: §4 · p. 8
Abstract
from arXiv · showhide
Much of the recent research on solving iterative inference problems focuses on moving away from hand-chosen inference algorithms and towards learned inference. In the latter, the inference process is unrolled in time and interpreted as a recurrent neural network (RNN) which allows for joint learning of model and inference parameters with back-propagation through time. In this framework, the RNN architecture is directly derived from a hand-chosen inference algorithm, effectively limiting its capabilities. We propose a learning framework, called Recurrent Inference Machines (RIM), in which we turn algorithm construction the other way round: Given data and a task, train an RNN to learn an inference algorithm. Because RNNs are Turing complete [1, 2] they are capable to implement any inference algorithm. The framework allows for an abstraction which removes the need for domain knowledge. We demonstrate in several image restoration experiments that this abstraction is effective, allowing us to achieve state-of-the-art performance on image denoising and super-resolution tasks and superior across-task generalization.
1 Introduction
MAP reconstruction balances prior expressiveness against optimization difficulty, while learned iterative RNNs inherit the design choices of their hand-selected prior and inference procedure. RIMs instead learn an inference algorithm from data and the task, without explicitly specifying those components, and are reported to outperform hand-engineered methods and generalize across tasks.
1 Introduction
- Inverse problems reconstruct signals from observations corrupted by a known or inferred forward model; the paper assumes that model is known and focuses on linear measurements.The paper gives y = Ax + n, with A as the corruption matrix and n as additive noise.Source: §1 · p. 1
- More expressive priors can better represent signals but typically make optimization harder, and only a few trivial prior-likelihood pairs yield convex inference.Scalable inference often requires objective approximations and approximate double-loop algorithms.Source: §1 · p. 1
- In learned iterative inference, the chosen prior and inference model determine the RNN architecture, burden practitioners with expert choices, and trade off against each other on complex tasks.The framework unrolls the inference procedure and trains model and inference parameters jointly with back-propagation through time.Source: §1 · p. 2
- RIMs learn an iterative inference algorithm without explicitly specifying a prior or inference procedure; their update uses the current reconstruction, hidden memory, and likelihood gradient.The likelihood gradient carries information about the known generative process and how well the reconstruction reproduces measurements.Source: §1 · p. 2
- With little domain knowledge, RIMs outperform state-of-the-art hand-engineered models and generalize across tasks by swapping likelihood gradients while keeping other parameters fixed.The introduction describes applying a RIM trained on deconvolution to inpainting.Source: §1 · p. 2
2 Recurrent Inference Machines
The section recasts iterative MAP inference as a learned recurrent update, replacing explicit prior and step-size components with trainable updates while retaining likelihood-gradient input. It adds constrained parameterization, memory, and training losses over successive predictions.
2 Recurrent Inference Machines
- Because a closed-form observation-to-signal map is usually intractable, inverse problems are typically solved iteratively, and RNNs can implement any algorithm.The passage attributes the RNN capability to Turing completeness.Source: §2 · p. 2
- The RIM generalizes iterative MAP updates by learning an update function that combines prior and step-size behavior while taking the likelihood gradient as input.This keeps the known forward-model information explicit in the update.Source: §2 · p. 3
- A differentiable link function constrains reconstructed x to a desired range while RIM iterations proceed in unconstrained η-space.Source: §2 · pp. 3–4
- A hidden memory state lets the inference procedure track progression or curvature, approximate a preconditioning matrix, and determine a stopping criterion.These are potential uses of the memory state, not guarantees for every RIM.Source: §2 · p. 4
- Training simulates T inference steps and applies losses to predictions at each step so they improve over time.The simplest total loss is a weighted sum of per-step losses; this work sets all weights to 1.Source: §2 · p. 4
3 Experimental Results
Experiments evaluate RIMs on random-projection reconstruction, denoising, and super-resolution, including transfer to inpainting and reconstruction of larger images.
3.1 Multi-task learning with Random Projections
- RIM clearly outperforms FFN and GDN on random-projection tasks; FFN degrades on harder tasks over time, while GDN degrades after the 20 training steps.The authors suggest recurrent state information contributes to convergence and performance.Source: §3.1 · p. 5, Figure 3 · p. 5
- On unseen inpainting, RIM and FFN transfer while GDN fails; RIM also leads qualitatively on 128×128 reconstructions despite training on 32×32 patches.The results support transfer without retraining, and Figure 4 shows the restoration steps toward the ground truth.Source: §3.1 · pp. 5–6, Figure 4 · p. 6
3.2 Image Denoising
- RIM-3task is competitive with other methods at all noise levels; RIM-denoise outperforms most methods at all noise levels, with performance slightly deteriorating after 8-bit quantization.RIM-denoise uses the same model across noise levels without hand tuning after training.Source: §3.2 · p. 6, Table 2 · p. 7
- For RGB denoising, RIM exploits correlations across color channels, providing an additional boost in reconstruction performance.Table 1 reports color denoising on 68 BSD-300 test images at σ = 25 after 8-bit quantization.Source: Table 1 · p. 6, §3.2 · p. 7
3.3 Image Super-resolution
- RIM outperforms SRCNN at all super-resolution scales using one model, while SRCNN was trained separately for each scale.The model has about 500,000 parameters; Figure 6 gives a qualitative example.Source: §3.3 · p. 8, Figure 6 · p. 8
4 Discussion
RIM jointly learns model and inference without specifying their explicit functional forms, and transfers across tasks without retraining. The discussion reports state-of-the-art denoising and super-resolution results while noting that this work focused on linear image reconstruction.
- The framework reports state-of-the-art results on image denoising and super-resolution.Source: §4 · p. 8
- RIMs can transfer across tasks without retraining.Source: §4 · p. 8
- RIM jointly learns model and inference without requiring their explicit functional forms.Source: §4 · p. 8
- This work focuses on linear image reconstruction, although the framework can be applied to nonlinear inverse problems.Source: §4 · p. 8