Source-linked AI summary
Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search
Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, Qingyuan Li
TL;DR
Super-resolution research must balance restoration quality against model simplicity for constrained resources, a trade-off that is difficult to optimize manually. This paper uses elastic neural architecture search with micro- and macro-level search, multi-objective optimization, and a hybrid evolutionary–reinforcement-learning controller. Its generated models dominate most state-of-the-art super-resolution methods at comparable individual FLOPS and span fast lightweight to larger more accurate designs.
Problem
Human-designed super-resolution models make the restoration-capacity versus simplicity trade-off difficult to optimize, especially for constrained resources.
Method
The paper uses an elastic micro- and macro-level search space, constrained multi-objective optimization, and a hybrid controller combining evolutionary computation with reinforcement learning.
Results
Generated models dominate most state-of-the-art super-resolution methods with respect to individual FLOPS.
Takeaways & Limitations
One search run can generate models ranging from fast and lightweight to relatively large and more accurate for constrained devices.
Abstract
from arXiv · showhide
Deep convolutional neural networks demonstrate impressive results in the super-resolution domain. A series of studies concentrate on improving peak signal noise ratio (PSNR) by using much deeper layers, which are not friendly to constrained resources. Pursuing a trade-off between the restoration capacity and the simplicity of models is still non-trivial. Recent contributions are struggling to manually maximize this balance, while our work achieves the same goal automatically with neural architecture search. Specifically, we handle super-resolution with a multi-objective approach. We also propose an elastic search tactic at both micro and macro level, based on a hybrid controller that profits from evolutionary computation and reinforcement learning. Quantitative experiments help us to draw a conclusion that our generated models dominate most of the state-of-the-art methods with respect to the individual FLOPS.
I. INTRODUCTION AND RELATED WORK
Single-image super-resolution restores high-resolution images from degraded low-resolution inputs, but human-designed models are difficult to fine-tune or compress. The paper addresses this resource–restoration trade-off with elastic neural architecture search and constrained multi-objective optimization.
- Single-image super-resolution is an ill-posed inverse problem that reconstructs a high-resolution image from a degraded low-resolution one.
- Human-designed super-resolution models are difficult to fine-tune or compress, motivating automated architecture search.
- The proposed search combines micro- and macro-level exploration within a dense, elastic search space to improve visual results.
- Super-resolution is formulated as a constrained multi-objective optimization problem using PSNR, mult-adds, and parameter count.
- A hybrid model-generation method balances exploration and exploitation while producing models for different requirements under one set of constraints.
III. ELASTIC SEARCH SPACE
The search space supports elastic super-resolution architecture design at both micro and macro levels, while Figure 1 depicts the main flow and skip connections.
- Micro search: Micro search selects promising cells within each cell block for feature extraction.
- Macro search: Macro search selects backbone connections among cell blocks to combine features at selected levels.
- Model flow: The super-resolution flow comprises feature extraction, nonlinear mapping, and restoration, with the search space describing the mapping while the other stages remain fixed.
- Model flow: Figure 1 depicts the neural architecture of super-resolution, with arrows denoting skip connections.
- Model composition: A complete model combines a predefined feature extractor, n searched cell blocks, macro-space connections, and subpixel-based upsampling and restoration.
A. Cell-Level Micro Search Space
The architecture search space combines configurable cell operators and connectivity patterns, yielding exponentially growing micro, macro, and total spaces as the number of cell blocks increases.
- Cell-level micro search space: Each cell block shares the same cell search space S for micro-level architecture selection.
- Cell-level micro search space: The micro space varies convolutions, channel counts, in-cell residual connections, and repeated-block counts.Available convolutions include 2D, grouped convolution with groups in {2, 4}, and an inverted bottleneck with expansion rate 2; channels are {16, 32, 48, 64}.
- Cell-level micro search space: For n cell blocks, the micro search space contains 192^n models.
- Macro search space: For the i-th cell block CBi, the macro space offers n + 1 − i connection choices to following cell blocks.The indexing convention starts with i = 1.
- Macro search space: The macro space contains 2^(n(n+1)/2) configurations, making the total space 192^n × 2^(n(n+1)/2).
IV. MODEL GENERATOR
The model generator combines evolutionary search with reinforcement learning and constructs diverse candidate architectures through sampled cells and connection patterns.
- Hybrid controller: The hybrid controller uses an evolutionary algorithm for iteration and reinforcement learning for exploitation.NSGA-II controls the iteration through population initialization, selection, crossover, and mutation.
- Model pipeline: The upsampling stage contains a 2D convolution with 32 3×3 filters followed by a 3×3 convolution with one filter and unit stride.
- Model encoding: A model is encoded as a chromosome containing forward-connected cells and their information connections.The chromosome is represented by the pair (Mmic, Mmac), using operator indices for cells and a nested list for connections.
- Population initialization: Initial populations are generated by randomly sampling a cell from S and repeating the process n times, with sampling without replacement when N exceeds the size of S.
- Connection initialization: Connections are sampled from a categorical distribution, with uniform selection within each category.The connection categories are random, dense, and no connections, determined by probability intervals.
C. Tournament Selection
The method uses crowding distance and tournament selection to maintain model diversity while controlling evolutionary pressure, and applies simultaneous crossover in micro and macro spaces.
- Crowding distance renders a uniform distribution of models during evolution.
- Tournament selection with k = 2 controls the evolution pressure.
- Single-point crossovers are performed simultaneously in both micro and macro spaces to encourage exploration.
- The crossover procedure contributes more to exploitation than to exploration.
E. Mutation
Mutation combines categorical strategy selection, random mutation, and roulette wheel selection to balance exploration and exploitation while addressing the multi-objective search.
- A categorical distribution balances exploration and exploitation during mutation.
- Mutation combines random mutation with roulette wheel selection for FLOPS and parameter objectives.
- Mutation strategies are selected by probability ranges using upper-bound pmr and lower-bound pmf thresholds.
- For roulette wheel selection, the macro structure remains unchanged while micro operators are sampled from the shared operator set.
- The roulette wheel selection approximation uses 192 basic operators instead of evaluating 192n models, with FLOPS and parameters scaled logarithmically.
2) Exploitation:
The exploitation stage uses a reinforcement-driven neural controller to mutate architectures, generating macro-level connections from micro-level embedding features and optimizing through discounted rewards.
- 2) Exploitation:: A reinforcement-driven mutation enhances exploitation.
- 2) Exploitation:: The neural controller concatenates Mmic embedding features and injects them into three fully connected layers to generate Mmac.
- 2) Exploitation:: Its last layer uses n(n + 1)/2 neurons to represent connections, with output denoted Omac.
- 2) Exploitation:: The controller network generates cells and connections.
- 2) Exploitation:: Network parameters are partitioned into θmic and θmac, representing probabilities for cell operators and connections.
- 2) Exploitation:: The gradient g(θ) is calculated from discounted accumulated rewards Ri and Rj with discount parameter γ = 1.0.
V. EVALUATOR
The evaluator uses incomplete training to quickly distinguish generated models, then selects candidates for full training and comparison under matched computational budgets. Experiments evaluate super-resolution models using PSNR and SSIM, with Pareto-front analysis across accuracy and complexity objectives.
- Evaluator: Incomplete training roughly differentiates generated models before final evaluation on test datasets.An RNN performance regressor was abandoned because its validation error was too high.
- Evaluator: About 10k models are generated, with populations of 64 per iteration, and one pipeline run takes less than 3 days on 8 Tesla-V100 GPUs.DIV2K is used as the training set.
- Evaluator: Four models with large Pareto-front crowding distances between mean squared error and mult-adds are selected for full training.The selected models are trained on DIV2K for 24000 epochs with batch size 16, taking less than 1.5 days.
- Comparisons: Comparisons use PSNR and SSIM on the ×2 task, considering only methods with comparable FLOPS and measuring mult-adds on 480 × 480 inputs.Large models such as RDN and RCAN are excluded from this comparison.
- Results: At comparable FLOPS, FALSR-A outperforms CARN and dominates DRCN and MoreMNAS-A over three objectives on four datasets.FALSR-A also achieves higher PSNR and SSIM with fewer FLOPS than VDSR, DRRN, and many other methods.
- Results: FALSR-B matches or exceeds CARN-M with fewer FLOPS and parameters, while FALSR-C also outperforms CARN-M using relatively sparse connections.FALSR-B’s connection complexity lies between residual and dense connections.
1) Cell Diversity:
Experiments indicate that diverse cells improve super-resolution results compared with repeated blocks. The paper also discusses why dense information flow may be difficult to train effectively and notes an assumption underlying incomplete training.
- Cell Diversity: Experiments show that cell diversity helps achieve better super-resolution results than repeated blocks.The authors report that models with repeated blocks underperform models with diverse cells.
- Cell Diversity: Different cell types may handle input features more effectively than monotonous cell structures.This explanation is presented as a speculation rather than a demonstrated causal mechanism.
- Optimal Information Flow: Dense connections may be suboptimal because models can struggle to ignore useless information from lower layers.Although dense connections can cover other non-dense configurations in principle, training may not learn to suppress irrelevant features.
- Evaluator Assumption: Incomplete training relies on the assumption that models performing well after full training usually also perform well during incomplete training.The paper states that models outside this pattern must be trained as fully as possible.
- Conclusion: The proposed NAS method searches both micro and macro levels and generates models ranging from fast and lightweight to larger and more accurate in one run.The authors present this as a way to support constrained-device architecture design and compression.