PaperlayerView exact revision on arXiv

Source-linked AI summary

Symbolic Learning to Optimize: Towards Interpretability and Scalability

Wenqing Zheng, Tianlong Chen, Ting-Kuei Hu, Zhangyang Wang

arXiv:2203.06578v4cs.LGcs.AI

TL;DR

L2O models learn numerical optimization rules, but neural-network predictors make those rules difficult to interpret and add memory costs that limit scale. This paper develops a symbolic representation and distillation framework for L2O; its lightweight optimizer was meta-trained on ResNet-50 within 30 GB GPU memory and achieved SOTA performance on larger or different deep models. The authors identify estimating interpretability metrics for more complicated symbolic forms as future work.

  • Problem

    Neural-network L2O predictors make learned optimization rules hard to explain and compare, while their memory overhead and unrolled training limit scalability.

    Source: §1 · pp. 1–2

  • Method

    The paper builds a symbolic L2O framework that distills a numerical predictor into symbolic rules using symbolic regression and can further tune those rules.

    Source: §1 · p. 2, §3.3 · pp. 4–5

  • Results

    Meta-training on ResNet-50 (23.5 million parameters) fit within 30 GB GPU memory, and the symbolic optimizer achieved SOTA on ResNet-152 (58.2 million parameters) and MobileNet v2.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors present symbolic L2O as a path toward practical performance with interpretable, lightweight optimization rules.

    Source: §5 · p. 9

    Estimating interpretability metrics for more complicated symbolic forms remains a challenge, and more generic designs require investigation and validation as future work.

    Source: §3.5 · p. 6

Abstract

from arXiv · show

Recent studies on Learning to Optimize (L2O) suggest a promising path to automating and accelerating the optimization procedure for complicated tasks. Existing L2O models parameterize optimization rules by neural networks, and learn those numerical rules via meta-training. However, they face two common pitfalls: (1) scalability: the numerical rules represented by neural networks create extra memory overhead for applying L2O models, and limit their applicability to optimizing larger tasks; (2) interpretability: it is unclear what an L2O model has learned in its black-box optimization rule, nor is it straightforward to compare different L2O models in an explainable way. To avoid both pitfalls, this paper proves the concept that we can "kill two birds by one stone", by introducing the powerful tool of symbolic regression to L2O. In this paper, we establish a holistic symbolic representation and analysis framework for L2O, which yields a series of insights for learnable optimizers. Leveraging our findings, we further propose a lightweight L2O model that can be meta-trained on large-scale problems and outperformed human-designed and tuned optimizers. Our work is set to supply a brand-new perspective to L2O research. Codes are available at: https://github.com/VITA-Group/Symbolic-Learning-To-Optimize.

1 INTRODUCTION

The introduction presents L2O as a data-driven alternative to manually designed optimizers, but identifies interpretability and scalability as barriers. It proposes symbolic representations and analysis to address both, with a lightweight optimizer that scales to large models and outperforms manually designed optimizers.

1 INTRODUCTION

  • L2O learns optimizers from performance on optimization tasks, which can fit narrow task distributions but may lack a solid theoretical basis.The introduction motivates data-driven optimization for repeated, similar tasks whose inputs vary across instances.Source: §1 · p. 1
  • Existing L2O rules are difficult to interpret because neural-network predictors leave their learned optimization rules opaque and hard to compare.The paper also identifies memory overhead from predictors and memory growth from backpropagating through unrolled updates as scalability barriers.Source: §1 · p. 2
  • The proposed framework converts numerical L2O predictors into lightweight, tunable symbolic rules and develops interpretability metrics for analyzing them.The framework aims to preserve problem-specific performance while reducing memory overhead and supporting further meta-training.Source: §1 · p. 2
  • The paper establishes a symbolic L2O framework applicable to existing approaches, including a symbolic optimizer space and a fitting method for converting numerical L2O models.Its symbolic-domain analysis examines learned shortcuts and differences across methods.Source: §1 · p. 2
  • Meta-training on ResNet-50 (23.5 million parameters) fit within 30 GB GPU memory; the symbolic optimizer achieved SOTA on ResNet-152 (58.2 million parameters) and MobileNet v2, outperforming tuned optimizers.The paper reports these outcomes across various datasets.Source: §1 · p. 2

2 RELATED WORKS

Related work describes L2O across a range of optimization domains and outlines its evolving model designs. It also reviews training difficulties and interpretability analyses, noting that prior explanatory approaches have been limited to specific objectives or algorithm types.

2 RELATED WORKS

  • L2O spans applications from signal processing and image processing to graph learning, medical imaging, and computational biology, and covers several optimization problem types.The passage also relates L2O to meta-learning through rapid within-task learning and transfer across similar tasks.Source: §2 · p. 2
  • Early L2O models used an LSTM to map optimizee gradients to updates or treated objective-value history as a reinforcement-learning state.The LSTM approach is attributed to Andrychowicz et al. (2016), and the reinforcement-learning approach to Chen et al. (2017).Source: §2 · p. 2
  • Generalization efforts included random scaling and convex-function regularizers, as well as hierarchical RNNs trained on ensembles of small representative problems.These approaches are attributed to Lv et al. (2017) and Wichrowska et al. (2017), respectively.Source: §2 · p. 3
  • Full-length LSTM unrolling is impractical for memory and trainability, so truncation is common but can introduce bias that hampers generalization to unseen tasks.The passage notes that practical optimizers may require thousands or more iterations.Source: §2 · p. 3
  • Prior interpretability work identified mechanisms such as momentum and gradient clipping, but explanatory analyses remained restricted to particular objectives or algorithm types.Other analyses connected learned algorithms to matrix factorization, projected gradient descent, iterative algorithms with data-driven hyperparameters, and Chebyshev step-sizes.Source: §2 · p. 3

3 TECHNICAL APPROACH

The approach meta-trains numerical L2O predictors, distills them into symbolic rules, and analyzes those rules for interpretability and scalability. It also permits lightweight re-parameterization of symbolic rules for adaptation.

3.1 NOTATION AND PRELIMINARIES

  • L2O learns update rules g(zt, φ) from optimization tasks, but numerical predictors create interpretability and memory challenges that symbolic rules are intended to address.Meta-training seeks parameters that produce desirable iterates; recurrent predictors also require memory for unrolled sequences, parameters, and hidden states.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 A SYMBOLIC DISTILLATION FRAMEWORK FOR L2O

  • Symbolic distillation first meta-trains a numerical predictor, then uses symbolic regression to fit an equation approximating its input-output behavior.This separates numerical rule search from symbolic representation, retaining gradient-based search in a continuously parameterized space.Source: §3.3 · pp. 4–5
  • The symbolic search fits teacher outputs to sequences of optimization inputs, evolving equation candidates with operators from a predefined search space.An offline database contains input sequences within a fixed unrolling length and corresponding outputs from the trained numerical predictor.Source: §3.3 · p. 5

3.4 CONCRETIZING THE SYMBOLIC DISTILLATION FRAMEWORK

  • The framework uses numerical L2O inputs by default, allows extra optimization features, and models rules as FIR filters with a default 20-step input horizon.The operator set combines basic optimizer building blocks with nonlinear activations and thresholding functions; inputs are rescaled to uniform variance.Source: §3.4 · p. 5
  • A proof-of-concept distilled an LSTM-based optimizer trained on randomly initialized LeNets for MNIST using genetic-programming symbolic regression.The study considered rules across complexity levels, scored by operator count and diversity and the number of input variables used.Source: §3.4 · p. 5

3.5 INTERPRETABILITY FROM SYMBOLIC L2O REPRESENTATIONS

  • Symbolic rules expose recognizable mechanisms, including momentum-like gradient averaging, time-decaying step sizes, and normalization-like or clipping operations.The framework proposes Temporal Perception Field (TPF) and Mapping Complexity (MC) to characterize effective history length and predictor complexity.Source: §3.5 · p. 6
  • The authors hypothesize that larger TPF favors faster, more stable convergence within a task distribution, while smaller MC may aid transfer after meta-tuning.Estimating TPF for heterogeneous symbolic forms remains a challenge, with more general designs left for future work.Source: §3.5 · p. 6

3.6 SCALABLE L2O TUNING WITH SYMBOLIC REPRESENTATIONS

  • A distilled symbolic rule can remain frozen for testing or have its coefficients re-parameterized through meta-tuning on new data.The workflow moves from numerical meta-training to symbolic regression and then meta-tuning of the symbolic equation.Source: §3.6 · pp. 6–7, Figure 1 · p. 7

4 INTERPRETABILITY AND SCALABILITY EVALUATIONS FOR SYMBOLIC L2O

The evaluations test symbolic regression against known optimizer rules, use TPF and MC to characterize learned optimizers, and assess symbolic L2O on large-scale tasks. The symbolic model outperforms most human-engineered optimizers on three tasks in both convergence speed and accuracy.

4.1 SANITY CHECKS FOR SYMBOLIC REGRESSION ON OPTIMIZATION TASKS

  • Symbolic regression faithfully reproduces known optimizer equations using recent gradients from ResNet-50 training on CIFAR-10.The regression inputs retain the most recent 20 steps.Source: §4.1 · p. 7

4.2 INTERPRETABILITY OF TRADITIONAL OPTIMIZERS AND LEARNED OPTIMIZERS

  • Across three problems, higher TPF is associated with expected faster convergence, while more diverse input features accompany better performance even at lower MC.RP(small)(extra) achieves the best performance among RP and DM variants and has the most diverse input features.Source: §4.2 · p. 8

4.3 SCALABILITY EVALUATIONS

  • The team meta-pre-trained RP(small) on P1, distilled it into a symbolic equation, and meta-fine-tuned the equation on large-scale optimizees.Meta-training over ResNet-50 used less than 30GB GPU memory; initialized from pre-trained RP(small), it took one pass of 200 epochs.Source: §4.3 · pp. 8–9
  • Across three tasks, symbolic L2O outperforms most human-engineered optimizers in convergence speed and accuracy.Figure 2 compares the meta-fine-tuned symbolic model with baseline optimizers.Source: Figure 2 · p. 9, §4.3 · p. 9

5 CONCLUSION

The conclusion presents a framework combining symbolic and numerical optimizers to address interpretability and scalability limitations. It describes distilling a neural L2O model into a symbolic equation and meta-tuning that equation on testing problems.

5 CONCLUSION

  • The framework aims to address L2O’s interpretability and scalability limitations by combining symbolic and numerical optimizers.The conclusion identifies both as important limitations of current L2O methods.Source: §5 · p. 9
  • The procedure numerically learns a neural L2O model, distills it into a symbolic equation, then meta-tunes the equation on testing problems.Source: §5 · p. 9
  • The authors hope symbolic L2O will encourage wider adoption and eventually replace laborious manual optimizer design or case-by-case tuning.Source: §5 · p. 9

A.1 EXPERIMENTAL DETAILS

The comparison distinguishes RP from DM by their gradient inputs and the presence of an input projection layer.

A.1 EXPERIMENTAL DETAILS

  • RP uses two gradient features, whereas DM uses the pure gradient as input.Source: §A.1 · p. 14
  • RP has an input projection layer; DM does not.Source: §A.1 · p. 14
  • The comparison identifies these as the two differences between RP and DM.Source: §A.1 · p. 14

A.2 MORE DETAILED DISCUSSIONS

RP(small) is reported as the table-3 winner, with the smallest evaluation loss and the simplest model. The discussion examines how input features, pretraining choices, symbolic interpretation, and fine-tuning relate to optimizer performance and complexity.

A.2 MORE DETAILED DISCUSSIONS

  • RP(small) is the table-3 winner with the smallest evaluation loss and is also the simplest model, combining complex gradient features.The paper cautions that the optimal rule need not be the simplest rule; it can be a simple combination of complex features.Source: §A.2 · p. 14
  • Across different input sets, models with richer features outperform simpler-input models while having smaller TPF and MC; within one input type, larger TPF/MC accompany better test loss.The authors suggest that increasing input-feature diversity can improve performance while reducing both TPF and MC.Source: §A.2 · p. 14
  • Experiments pretrain on P1/P2/P4, distill an equation skeleton, train its coefficients during meta-fine-tuning, and compare pre/post test accuracy on ResNet-50 CIFAR-10.The paper says the equation skeleton is the main target of symbolic regression, while its coefficients remain available for fine-tuning.Source: §A.2 · pp. 14–15, Table 5 · p. 15
  • With a correctly distilled skeleton, meta-pretraining problem choice has no significant effect on symbolic L2O performance; diverse-feature RP aligns more consistently across settings than DM.The authors characterize the diverse-feature model as more tolerant or stable to the choice of meta-pretraining problem.Source: §A.2 · p. 15
  • Equation 6 combines thresholded gradient features that may help escape flat minima and smooth noisy gradients; Equation 7 resembles SGD in P2 and Adam with tanh thresholding in P4.For P1, the authors describe asinh as clipping the gradient around a log(x + 1) term that is near-linear at small magnitudes.Source: §A.2 · p. 15
  • The lower-MC symbolic rule from RP(small) transfers better than DM, while the higher-complexity DM model is less stable across meta-pretraining problems.The experiment evaluates before and after fine-tuning on ResNet-50 with CIFAR-10; the supplied table caption identifies test accuracy as the metric.Source: §A.2 · p. 15, Table 5 · p. 15
  • Symbolic distillation reveals whether a numerical optimizer learned a simple rule, allowing RP to be simplified into RP(small) for lighter weight, interpretability, and better performance.The authors stress that the numerical optimizer alone does not reveal the simplicity of its learned rule.Source: §A.2 · p. 15
  • The reported timing breakdown covers meta-pretraining, symbolic regression, meta-fine-tuning, and final evaluation; TPF and MC are available once symbolic regression finishes.Symbolic-regression and TPF/MC times were measured on a CPU, while other computation times were measured on an Nvidia A6000 GPU.Source: §A.2 · p. 16, Table 6 · p. 16

B.1 EXPERIMENTAL DETAILS

The appendix reports symbolic-regression settings, seed stability, and CNN training conditions. For RP(small), the distilled equation retained the same form across tested random seeds.

B.1 EXPERIMENTAL DETAILS

  • Experiments used complexity 100 for distilled equations, which the authors report gave the best fitting ability and performance.Sanity-check complexity limits were under 5 for SGD, under 50 for Momentum, and 100 for Adam.Source: §B.1 · p. 16
  • RP(small)'s distilled equation (6) remained the same across tested random seeds.The authors attribute this stability to the learned rule's low MC.Source: §B.1 · p. 16
  • CNN optimizees were trained for 200 epochs in both meta-fine-tuning and final evaluation.Batch size was 128.Source: §B.1 · p. 16

B.2 MORE DETAILED DISCUSSIONS

The appendix examines symbolic-rule fitting, complexity, equation behavior, and evaluation on GLUE. It reports that symbolicL2O achieved the best results across the GLUE tasks and describes how fit and equation form varied across analyses.

B.2 MORE DETAILED DISCUSSIONS

  • RP(small)'s distilled rule usually took equation (6)'s simple nonlinear-thresholding form and achieved R2-score 0.88.The rule was distilled from the best numerical model, RP(small).Source: §B.2 · p. 16
  • Figure 4 plots evaluation-loss trajectories for numerical and distilled rules on DM and RP(small) using ResNet50 on CIFAR-10.The plotted lines correspond to evaluation loss for each rule and model in table 3.Source: §B.2 · pp. 16–17, Figure 4 · p. 17
  • As equation complexity increases, fitting performance tends to rise and then fall, with simpler equations underfitting and more complex ones tending to overfit.Figure 5 plots R2-score against equation complexity and also reports estimated TPF over complexity levels.Source: §B.2 · p. 17, Figure 5 · p. 18
  • Hyperbolic functions are approximately linear for small inputs, so their output differs significantly from a linear function mainly at large magnitudes.The authors suggest these functions may be selected through mutation because thresholding can reduce uncertainty with noisy gradients.Source: §B.2 · pp. 17–18
  • In distilled LSTM equations, tanh may reflect the teacher LSTM's nonlinear activation and updating dynamics.Source: §B.2 · p. 18
  • Removing hyperbolic functions did not improve R2-score; equations remained similar in form with slightly different coefficients.The authors explain that symbolic regression compares and screens candidates both with and without these functions.Source: §B.2 · p. 18
  • SymbolicL2O achieved the best results across all evaluated GLUE tasks against AdamW.BERT base cased was fine-tuned for 100 epochs with both optimizers at learning rate 0.0005; symbolicL2O's coefficients were fixed.Source: Table 9 · p. 19, §B.2 · p. 19

C SYMBOLIC REGRESSION PRELIMINARIES

The preliminary section describes symbolic regression as iterative mutation of equation candidates and lists its principal hyperparameter settings. The authors used 300 iterations, a population of 200, and 5,000 samples.

C SYMBOLIC REGRESSION PRELIMINARIES

Related papers

Loading 2203.06578v4…