Learning to Optimize: A Primer and A Benchmark

1.1 Background and Motivation

• L2O trains optimizers offline to improve online speed and potentially solution quality on difficult problems, while narrow task distributions can let learned methods exploit patterns conventional methods do not target.The paper also describes faster completion and higher-quality solutions at similar computing budgets as potential strengths of L2O.Source: §1.1 · pp. 2–3
• On unseen same-distribution compressed-sensing problems, [1] reach equal accuracy with an order-of-magnitude fewer iterations than ISTA/FISTA.The introduction presents this as an example of L2O's potential on repeated tasks drawn from a narrow distribution.Source: §1.1 · pp. 2–3

1.2 Preliminaries

• L2O parameterizes an update rule using available optimization information and trains its parameters on sample optimizees; architectures may be model-free or incorporate existing methods as a starting point.Model-free L2O can lack convergence guarantees and need many training samples, while model-based L2O narrows the search space to mitigate these shortcomings.Source: §1.2 · pp. 3–4

1.3 Broader Contexts

• L2O overlaps with meta-learning and AutoML but also applies beyond learning tasks and can discover optimizers rather than only select or tune existing ones; it is also a subset of learning-augmented algorithms.The boundary with hyperparameter optimization can be blurry when an L2O method only configures or predicts parameters for existing optimizers.Source: §1.3 · pp. 4–5

1.4 Paper Scope and Organization

• The article reviews model-free and model-based L2O, emphasizing recurrent networks, algorithm unrolling, plug-and-play, effective training, and the use of optimizee and classic-method structure.Its coverage is broader than prior reviews focused on algorithm unrolling or model-based deep learning.Source: §1.4 · pp. 5–6, §1.3 · p. 5
• The authors benchmark existing L2O approaches on representative optimization problems and release software and test cases in Open-L2O to support reproducible research and fair benchmarking.The paper identifies Open-L2O as its released package.Source: §1.4 · p. 6

2. Model-Free L2O Approaches

• Model-free L2O typically unrolls an LSTM to learn short trajectories, sharing parameters across steps; its inputs include optimizee states such as zero- and first-order information.The workflow has offline training on optimizees sampled from task distribution T, followed by online testing on new optimizees assumed to come from T.Source: §2 · p. 6

2.1 LSTM Optimizer for Continuous Minimization: Basic Idea and Variants

• LSTM-based L2O [5] showed faster convergence than SGD, RMSprop, and Adam on small tasks, but faced scalability and generalization limits, while longer unrolling brought memory bottlenecks and training instability.Generalization challenges include extrapolating to deeper or wider networks and to unseen activation or loss functions; most LSTM methods truncate unrolling, and extending a short-trained optimizer can yield unstable, poor-quality solutions.Source: §2.1 · pp. 7–8
• To address truncation bias, samples unroll lengths from a heavy-tailed distribution and uses a hierarchical three-RNN architecture, while [60] replaces the LSTM with an MLP and smooths the loss landscape.The hierarchical design uses smaller bottom-layer hidden states to reduce memory and compute overhead while improving generalization; [58] also uses random coordinate-wise scaling and a convex training term.Source: §2.1 · pp. 8–9
• Progressive unroll-length training and off-policy imitation learning [73] improved unseen-optimizee training losses across existing L2O models without changing their RNN architectures.Other training work reported faster wall-clock training of convolutional networks with reduced test losses, and robustness to unseen image corruptions [71].Source: §2.1 · pp. 8–10

2.2 Other Common Implementations for Mode-Free L2O

• RL-based alternatives learn update policies or optimizer hyperparameters [61; 74], while neural-symbolic search [62] selects mathematical operations to form update rules that transfer across tasks and architectures.Coordinate grouping [28] reduced computation cost and enabled extension to wider networks; the neural-symbolic optimizer improved training performance on ImageNet classification and Google’s neural machine translation system.Source: §2.2 · p. 10

2.3 More Optimization Tasks for Model-Free L2O

• For black-box optimization, [6] used input-output observations to balance exploration and exploitation, while L2O-Swarm [18] modeled uncertainty and outperformed off-the-shelf PSO and non-uncertainty-aware LSTM L2O on test functions and protein docking.The black-box method was competitive with Bayesian optimization packages; L2O-Swarm combines point- and population-based features and uses posterior entropy in its meta-loss.Source: §2.3 · pp. 10–11
• In minimax optimization, L2O improved inner maximization and resulting robust models [82], while Twin L2O extended learning end to end and compared favorably with state-of-the-art solvers on low-dimensional tests.Scaling L2O to practical minimax problems such as adversarial training or GANs remained an open challenge.Source: §2.3 · pp. 11–12
• Model-free L2O has also been applied to game theory [16] [17] and few-shot classification [85], including learned policies for agent equilibria and LSTM-based task-specific classifiers.[17] learns update-rule gradient and Hessian coefficients for constrained non-zero-sum games; [85] simplifies few-shot optimization to one gradient step with learned initialization and step size.Source: §2.3 · p. 12

• Model-based L2O replaces general-purpose LSTMs with learnable iterative update rules inspired by analytic optimization algorithms [50].Source: §3 · p. 12
• These methods blend model-based structures and priors with data-driven learning, balancing compactness, data efficiency, interpretability, and performance when the optimization model is available or partly inferred.Source: §3 · p. 12
• Plug-and-play (PnP) inserts a previously trained neural network into an analytic update, such as a proximal step, and applies the modified algorithm without further training.Source: §3 · pp. 12–13
• PnP can replace a total-variation proximal denoising step with an existing neural-network denoiser [88] or BM3D.Source: §3 · p. 13
• Algorithm unrolling truncates an optimization algorithm to K iterations, makes its updates neural-network layers, and trains layer parameters end-to-end using the final iterate xK [50].Unlike PnP, unrolled schemes train parameters jointly through xK; PnP trains its inserted operator separately.Source: §3 · p. 13, Figure 4 · p. 14

3.1 Plug and Play

• The original 2013 PnP method replaced one ADMM proximal operator with a denoiser and empirically surpassed the original ADMM algorithm [86]; convergence guarantees came later under doubly stochastic derivative [109] and bounded-denoiser assumptions.Source: §3.1 · p. 14
• Learning proximal operators as deep networks improved empirical performance over prior PnP methods and could accelerate optimization subroutines or produce better solutions [122] [123] [124].These works introduced learning operators into PnP in place of manually designed denoisers.Source: §3.1 · p. 14
• Learned regularizers included Denoising Autoencoders modeling image distributions for likelihood maximization and a Wasserstein discriminator used in gradient descent for inverse problems [125].The Denoising Autoencoder is differentiable, enabling gradient-based optimization.Source: §3.1 · p. 14
• A reinforcement-learning policy tuned PnP parameters and achieved results comparable to oracle parameters tuned using inaccessible ground truth [128].Source: §3.1 · p. 14
• Convergence can be encouraged by constraining denoiser Lipschitz constants, while RED-PRO extensions report global convergence and a learned operator can approximate projection onto a data manifold [88] [130] [131].The manifold projection result assumes manifold properties and sufficient representation by sampled data, and approximation is proven in probability.Source: §3.1 · p. 15

3.2.3 Objective-Based v.s. Inverse Problems

• Objective-based unrolling trains for rapid convergence on a data distribution, whereas inverse-problem unrolling trains to recover the true signal rather than minimize the surrogate objective.Objective-based losses include expected objective value, gradient norm, solution distance, or fixed-point residual; safeguards can fall back to a classic update when a learned step fails a descent check, preserving theoretical convergence guarantees. In inverse problems, a common setup unrolls a parameterized algorithm for a fixed number of iterations and minimizes expected squared distance to the true signal.Source: §3.2 · p. 15, §3.2.3 · pp. 18–19

3.2.1 Different target problems

• Unrolled L2O targets probabilistic graphical models, sparse and low-rank regression, differential equations, and quadratic optimization; sparse regression is especially prominent.Inverse-problem methods may instead prioritize recovering original signals, while some unrolled outputs serve downstream tasks such as clustering or classification.Source: §3.2.1 · pp. 15–16

3.2.2 Different Algorithms Unrolled

• Unrolled algorithms are grouped into forward-backward splitting, primal-dual methods, and others, with first-order methods prominent for low complexity and trainability.ISTA and its variants are widely unrolled for sparsity problems, alongside FISTA, AMP, and IHT; choices of learned parameters and safeguards can make methods based on the same algorithm perform differently.Source: §3.2.2 · pp. 17–18

3.2.4 Learned Parameter Roles

• Unrolled methods parameterize updates by learning algorithm parameters, data-dependent priors, or black-box update agents; learned hyperparameters can also control classic iterative algorithms.Untying parameters across layers can enlarge model capacity but risks training instability, while ALISTA reduces trainable parameters to two scalars per layer and provides a theoretical convergence guarantee.Source: §3.2.4 · pp. 19–20

Image Restoration and Reconstruction

• Image-restoration L2O addresses denoising, deblurring, super-resolution, inpainting, compressive sensing, and other tasks, though many works only loosely follow the original algorithm architecture [165; 170; 124; 155; 210; 2; 189; 127; 211; 123; 135; 212; 104; 178; 183; 1; 134; 214; 9; 176; 8; 215; 122; 101; 216].Model-based methods include Plug-and-Play and unrolling approaches; some replace algorithm components with convolutions or other modern deep-learning building blocks.Source: §3.3 · p. 21

Medical and Biological Imaging

• Medical and biological imaging applies model-based L2O to MRI and CT reconstruction, where few measurements are desirable and complex-valued inputs challenge methods extended from natural images [165] [218] [171].The imaging goals include accurate reconstruction with minimal discomfort or side effects.Source: §3.3 · p. 21

Wireless Communication

• In wireless communication, unrolling addresses resource management, channel estimation, signal detection, and LDPC coding [221] [172]; model-based methods show robustness and stability under low-SNR and system mismatches.For MIMO detection, deep unfolding of iterative algorithms with trainable parameters, such as LISTA, benefits sparse recovery [14].Source: §3.3 · p. 21

Seismic Imaging

• Seismic imaging commonly uses Plug-and-Play methods that train CNN-based projectors on data and integrate them into classic iterative updates, retaining emphasis on physical modeling.Source: §3.3 · p. 21

Miscellaneous Applications

• Other applications include clustering and classification [227], phase retrieval, RNA structure prediction [19], speech tasks [150] [229], remote sensing, smart grids, graph recovery, and photometric stereo.Source: §3.3 · p. 21

3.4 Theoretical Efforts

• Theory remains scarce; model-based L2O is more amenable to classical analysis, while trainability guarantees are very limited because training objectives are highly nonconvex.The cited exceptions analyze local minima and gradient explosion or vanishing for quadratic minimization [37], and use sorting activations and Lipschitz networks to encourage gradient-norm preservation [131].Source: §3.4 · pp. 21–22

3.4.1 Capacity

• Capacity analyses show that neural networks can approximate WMMSE, and that LISTA can converge faster than ISTA/FISTA, with later models refining its guarantees [1] [178] [236].Other convergence work analyzes differentiable linearized ADMM and imposes practical operator conditions, including continuity [88], smoothness [237], convex regularizers [171], or manifold and data-representation assumptions [131].Source: §3.4.1 · pp. 22–23
• Safeguards can extend convergence to out-of-distribution optimizees by accepting learned updates only when an energy or residual test passes, otherwise falling back to a classical update.With a monotonically decreasing energy, the overall algorithm converges to an energy minimizer; energies may use the variational objective or its differential, or residuals relative to a classical update [24].Source: §3.4.1 · p. 23

3.4.2 Interpretability

• Interpretability studies connect learned behavior to analytic methods, explaining LISTA through dictionary factorization and speed–accuracy tradeoffs, and unrolled gradient descent through Chebyshev step sizes.Other work links unrolled IHT to improved dictionaries and broader RIP conditions, derives LISTA weights, and relates graph-recovery models to adaptive convex programs.Source: §3.4.2 · p. 23

3.4.3 Generalization

• Generalization research bounds gaps using Rademacher complexity [160] [240] [241] and relates model depth to capacity–generalization tradeoffs [160].Related stability analyses estimate Lipschitz constants [201], study behavior as regularization vanishes, or use Φ-regularization with null-space networks; generalization concerns unseen samples from the training distribution, not distribution shifts.Source: §3.4.3 · p. 24

4. The Open-L2O Benchmark

• Open-L2O addresses inconsistent and sometimes unfair comparisons by benchmarking representative sparse, LASSO, Rastrigin, and neural-network problems under problem-specific task distributions.The benchmark implements applicable model-free and model-based methods in TensorFlow with identical training and testing data, and releases datasets and software to support reproducibility and coordinated development.Source: §4 · pp. 24–25

4.1.1 Learning to perform sparse optimization and sparse-signal recovery

• ALISTA leads sparse recovery in noiseless and larger-scale settings, while GLISTA leads with 20dB noise and a coherent dictionary.LISTA-CPSS, LAMP, and GLISTA also outperform LISTA in the noiseless case; LFISTA does not. At larger scale, ALISTA’s few learnable parameters help it retain nearly its smaller-scale performance, while other methods degrade.Source: §4.1.1 · p. 26, Figure 5 · p. 27

4.1.2 Learning to minimize the Lasso model

• On both LASSO problem sizes, 16-layer LISTA and ALISTA reach high-precision solutions, demonstrating the advantage of incorporating problem-specific structure.At (5, 10), they match FISTA’s precision after hundreds of iterations and outperform ISTA at 1,000 iterations; FISTA extensions sustain their advantage. At (25, 50), model-free methods and sub-gradient baselines remain poor, although L2O-RNNprop is faster than ISTA and comparable to FISTA at lower precision.Source: Figure 6 · p. 28, §4.1.2 · pp. 28–29

4.2 Test 2: Minimization of non-convex function Rastrigin

• For Rastrigin minimization, L2O-Swarm finds higher-quality solutions for both dimensions, especially n = 10, but converges more slowly than most alternatives.For n = 10, its final loss is significantly better than other methods; for n = 2, almost all methods end in [5.5, 6.0]. Other model-free L2Os perform similarly to analytic optimizers, with larger error bars at n = 10; L2O-Swarm’s population of LSTM-based particles incurs higher search cost.Source: Figure 7 · p. 30, §4.2 · p. 30

4.3 Test 3: Neural Network Training

• On unseen neural-network tasks, L2O-Scale and L2O-RNNprop transfer to ReLU-MLP, but learned optimizers do not effectively optimize ConvNet.L2O-DM diverges on ConvNet; L2O-RNNprop works only for fewer than 4,000 iterations, while L2O-Scale transfers to larger iteration counts, attributed to random scaling during training. Current L2O methods also struggle to scale to networks with over 1×10^6 parameters because unrolling requires substantial memory.Source: §4.3 · p. 31, Figure 8 · p. 32
• L2O-enhanced achieves the best performance on both unseen networks after extended optimizee training iterations.It uses increasing unroll length and an off-line imitation strategy.Source: §4.3 · p. 32
• On ReLU-MLP and ConvNet, all L2O optimizers show larger loss variances as iterations grow, causing more severe error fluctuations than analytic optimizers.The passage reports this testing-stability issue for all L2O optimizers.Source: §4.3 · p. 32

4.4 Take-home messages

• When model or problem structures are available, model-based L2O methods outperform model-free methods and analytic optimizers, with solid, consistent, reliable performance on same-task optimizees.Source: §4.4 · p. 32
• Model-free L2O gains over analytic optimizers are inconsistent, weaken on larger optimizees or with more test iterations, and no approach consistently wins across most tests.The gains are also questioned on out-of-distribution optimizees.Source: §4.4 · p. 32

Concluding Remarks

• The article presents the first panoramic review of the emerging L2O field alongside its first-of-its-kind benchmark, and finds that the field remains in its infancy.It identifies open challenges and research opportunities spanning practice and theory.Source: §5 · p. 32
• L2O lacks theoretical foundations for model-free methods, convergence guarantees for training, and well-studied generalization or adaptation guarantees on out-of-distribution optimizees.The lack of convergence results for training leaves the general feasibility of obtaining L2O models in question.Source: §5 · pp. 32–33
• Model-free L2O scalability remains a major practical hurdle, while model-specific success is limited to special inverse-problem and sparse-optimization instances requiring case-by-case modeling.The authors call for broader applications and potentially more general frameworks.Source: §5 · p. 33
• Unrolled networks may bridge generic networks and analytic algorithms as more data-efficient models [50] and robust starting points for data-driven model search.The authors also suggest improving end-to-end learning with continuous optimization algorithms to gain their theoretical guarantees and state-of-the-art methods.Source: §5 · p. 33
• Most discussed approaches remain exploratory and are not yet ready as general-purpose or commercial solvers, but the authors expect L2O research to grow.They characterize machine learning as only beginning to influence classical optimization.Source: §5 · p. 33

Appendix A. List of Abbreviations

• HPO means Hyperparameter Optimization, the problem of choosing optimal hyperparameters for a machine-learning algorithm.Source: Appendix A · p. 34
• L2O means Learning to Optimize: learnable optimizers predict update rules for optimizees, fitted from data.Source: Appendix A · p. 34
• LSTM means Long-short Term Memory, a recurrent neural-network variant typically used for sequence data; MLP means Multi-layer Perceptron, a multilayer feedforward network.Source: Appendix A · p. 34