Exploiting the Potential of Standard Convolutional Autoencoders for Image Restoration by Evolutionary Search

• Prior image-restoration research has explored increasingly complex architectures, specialized losses, and adversarial training to address the ill-posed inverse problem.Examples include skip-connected encoder-decoder networks, memory networks, perceptual losses, and adversarial frameworks.Source: §1 · p. 1
• Adversarial training can introduce difficulties including mode collapse, evaluation challenges, and theoretical limitations.Source: §1 · p. 2
• Simple CAEs using convolutional layers and skip connections achieve state-of-the-art image-restoration performance with standard ℓ2 training and no adversarial or novel hand-designed loss.The paper uses ADAM to minimize the standard ℓ2 loss.Source: §1 · p. 2
• An evolutionary algorithm searches CAE architectures by optimizing layer count, filter sizes and numbers, and layer connections.Source: §1 · p. 2
• The study presents automatic architecture search for image restoration as a first contribution in this application area, whereas prior architecture-search studies tested classification tasks.Source: §1 · p. 2

2.1. Deep Learning for Image Restoration

• Deep-learning restoration methods span denoising, super-resolution, deblurring, compressive sensing, and inpainting, with recent work often using GAN-based training.Source: §2.1 · p. 2

2.2. Automatic Design of Network Architectures

• Neural-network architectures have traditionally been designed manually, often through substantial trial and error, motivating automated design methods.Source: §2.2 · p. 2
• Automatic architecture search methods use evolutionary algorithms or reinforcement learning, but reinforcement-learning approaches can require 10-800 GPUs.Source: §2.2 · p. 2
• The paper adapts Suganuma et al.’s computationally efficient evolutionary method to symmetric CAEs, enabling searches with 1 to 4 GPUs over a few days.Source: §2.2 · p. 2

2.3. Evaluation Methods for Image Restoration

• The evaluation emphasizes pixel-to-pixel accuracy because pure restoration targets ground-truth intensities, whereas perceptual quality favors realism and may not improve PSNR.Human-rated measures such as MOS are also difficult to reproduce for small image differences.Source: §2.3 · pp. 2–3

3.1. Search Space of Network Architectures

• Symmetric CAEs use convolutional layers, ReLU activations, optional skip connections, and task-dependent downsampling, with the encoder determining the decoder.Skip connections connect encoder layers to mirrored decoder layers, while the decoder is symmetric to the encoder.Source: §3.1 · p. 3

3.2. Representation of CAE Architectures

• A phenotype is a directed acyclic graph on an M-by-N grid whose unique input-to-output path specifies the encoder architecture.The maximum encoder depth is N, and the level-back hyperparameter controls allowable predecessor columns.Source: §3.2 · p. 3
• A genotype encodes each graph node’s filter configuration, skip-connection status, and input connection, including a separate output-node connection.The filter counts and sizes are represented by integer-coded node attributes.Source: §3.2 · pp. 3–4

3.3. Evolutionary Strategy

• The evolutionary strategy generates λ mutated children from a parent, trains them, evaluates validation PSNR, and retains the best parent-or-child architecture.The process repeats for G generations, with mutations applied to node types and connections.Source: §3.3 · p. 4
• Mutations that alter only non-functioning nodes are skipped, while unsuccessful generations modify non-functioning nodes without changing the realized CAE.This preserves the current architecture while continuing genotype exploration.Source: §3.3 · p. 4

• The experiments assess the proposed approach on image inpainting and image denoising.Source: §4 · p. 4

4.1.1. INPAINTING

• Inpainting uses CelebA, Cars, and SVHN images resized to 64 × 64 pixels, with separate training, validation, and test handling.CelebA contributes 100,000 training, 1,000 validation, and 2,000 test images; Cars and SVHN use provided splits.Source: §4.1.1 · p. 5
• Inpainting evaluates central-square, 80%-random-pixel, and randomly selected half-image masks.Random masks are regenerated for each training minibatch and test image for the latter two mask types.Source: §4.1.1 · p. 5
• Inpainting CAEs use symmetric hourglass downsampling and upsampling with stride = 2, while each encoder layer chooses either downsampling or a skip connection.Source: §4.1.1 · p. 5

4.1.2. DENOISING

• Denoising uses 300 grayscale BSD500 training images and 200 test images, with random 64×64 patches corrupted by Gaussian noise at σ = 30, 50, or 70.A separate model is trained for each noise level.Source: §4.1.2 · p. 5
• Denoising uses CAEs without downsampling and zero-padding so convolutional layers preserve feature-map spatial dimensions.Source: §4.1.2 · p. 5

• The search uses mutation probability r = 0.1, λ = 4 children, G = 250 generations, and a phenotype grid with M = 3, N = 20, and L = 5.Source: §4.2 · p. 5
• Candidate filters use F ∈ {64, 128, 256} and k ∈ {1 × 1, 3 × 3, 5 × 5}, with I = 20k training iterations per evolutionary evaluation and batch size b = 16.Source: §4.2 · p. 5
• After evolution, the best CAE is fine-tuned for 500k iterations with learning-rate reductions at 200k and 400k iterations before test evaluation.The experiments use ADAM with initial learning rate 0.001 and four P100 GPUs.Source: §4.2 · p. 5

4.3.1. INPAINTING

• E-CAE outperforms CE and SII on every reported inpainting dataset-mask combination despite evaluating complete outputs against ground truth.CE and SII use masks and retain unmasked ground-truth pixels, whereas E-CAE reconstructs complete images without masks.Source: §4.3.1 · p. 5, Table 1 · p. 6
• E-CAE performs stably across mask types, producing especially realistic and sharp results for random pixel masks.For filled missing regions, outputs tend to be less sharp but can still infer contents accurately in number images.Source: §4.3.1 · pp. 5–6, Figure 2 · p. 6
• E-CAE also yields blurry reconstructions in several difficult inpainting cases.These examples identify a concrete failure mode alongside the generally stable results.Source: §4.3.1 · p. 6, Figure 3 · p. 6

4.3.2. DENOISING

• E-CAE achieves the best denoising performance for all BSD200 noise levels except one case, where MemNet performs better at σ = 30.The comparison covers σ = 30, 50, and 70; E-CAE uses a 15-layer architecture versus 30 layers for RED and 80 for MemNet.Source: §4.3.2 · pp. 6–7, Table 2 · p. 6
• E-CAE correctly removes noise and produces an image as sharp as the ground truth in the shown denoising example.Figure 4 shows an input with noise level σ = 50.Source: §4.3.2 · p. 7, Figure 4 · p. 7

4.4. Analysis of Optimized Architectures

• Most optimized networks use a skip connection at the first layer, suggesting input information is important for accurate outputs across tasks.The observation holds for both inpainting and denoising architectures.Source: §4.4 · p. 7
• 1 × 1 convolutions are common in both tasks, dominating denoising networks while appearing twice in every inpainting network.The optimized filter sizes are difficult to determine manually despite otherwise non-unique overall structures.Source: §4.4 · p. 7
• Inpainting architectures use larger 5 × 5 filters more often than denoising architectures, averaging 2.8 versus 0.8 layers.They use 1 × 1 filters less often, averaging 2.0 versus 3.2 layers.Source: §4.4 · p. 7

4.5. Effects of Parameters of Evolutionary Search

• The best result is obtained with mutation probability r = 0.1.Smaller mutation probabilities start less accurately but eventually achieve higher accuracy after sufficient generations; larger values make search less stable while reducing local-optimum risk.Source: §4.5 · p. 7
• The best performance is obtained with λ = 4 children, although final PSNR differs little across population sizes after sufficient generations.Using λ = 1 remains competitive while requiring the minimum computational cost per generation.Source: §4.5 · p. 7
• With λ = 1, evolution takes 1.68 days on one P100 GPU and reaches PSNR = 29.80 after model fine-tuning.This demonstrates a competitive result at the lowest tested child count.Source: §4.5 · p. 7

5. Conclusion

• Simple E-CAEs outperform state-of-the-art inpainting and denoising networks despite using standard convolutional layers, optional skip connections, and standard ℓ2 training.The competing networks use deeper architectures, hand-designed losses, and adversarial training, whereas E-CAEs are trained with ADAM.Source: §5 · p. 7
• Table 3 reports the five best-performing E-CAE architectures, showing encoder configurations and averaged denoising PSNR and SSIM across three noise levels.The table’s C(F, k) and CS notation distinguishes filter configurations with and without skip connections.Source: Table 3 · p. 8
• Evolutionary search finds effective CAE architectures within a high-dimensional space of layer counts, filter configurations, and skip-connection choices.These architectural parameters are difficult to determine manually.Source: §5 · pp. 7–8
• The search strategy suggests that classical convolutional architecture spaces may still offer room for exploration in colorization, depth estimation, and optical-flow estimation.Source: §5 · p. 8

A. Supplementary Material

• E-CAE inpainting examples cover center-mask, random-pixel-mask, and half-mask distortions alongside masked inputs and ground truths.Source: A. Supplementary Material · p. 10, Figure 7 · p. 11
• E-CAE denoising examples cover noise levels σ = 30, σ = 50, and σ = 70.Source: A. Supplementary Material · p. 10, Figure 8 · p. 11
• The authors make their implementation publicly available through the Evolutionary-Autoencoders GitHub repository.Source: A. Supplementary Material · p. 10