PaperlayerView exact revision on arXiv

Source-linked AI summary

OMLT: Optimization & Machine Learning Toolkit

Francesco Ceccon, Jordan Jalving, Joshua Haddad, Alexander Thebelt, Calvin Tsay, Carl D. Laird, Ruth Misener

arXiv:2202.02414v2stat.MLcs.AIcs.LGmath.OC

TL;DR

Trained neural networks and gradient-boosted trees can be difficult and error-prone to translate into solver-ready optimization formulations. OMLT uses Pyomo blocks to automate that translation, enabling these models to be included in larger optimization problems and allowing users to compare alternative formulations.

  • Problem

    Translating trained neural networks and gradient-boosted trees into solver-ready optimization formulations is tedious and error-prone.

    Source: §1 · p. 2

  • Method

    OMLT translates higher-level representations of trained neural networks and gradient-boosted trees into Pyomo formulations using blocks that connect surrogate models to larger optimization problems.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    OMLT supports incorporating neural networks and gradient-boosted trees into Pyomo optimization models and comparing formulations that are often treated as competitors.

    Source: §5 · p. 4, §4 · p. 4

  • Takeaways & Limitations

    Users can extend Pyomo optimization models with neural networks or gradient-boosted trees and select among formulations for a specific application.

    Source: §5 · p. 4, §4 · p. 4

Abstract

from arXiv · show

The optimization and machine learning toolkit (OMLT) is an open-source software package incorporating neural network and gradient-boosted tree surrogate models, which have been trained using machine learning, into larger optimization problems. We discuss the advances in optimization technology that made OMLT possible and show how OMLT seamlessly integrates with the algebraic modeling language Pyomo. We demonstrate how to use OMLT for solving decision-making problems in both computer science and engineering.

1. Introduction

OMLT 1.0 is an open-source toolkit for incorporating trained neural networks and gradient-boosted trees into larger optimization problems. It translates these models into Pyomo formulations, reducing the effort of coding and debugging them.

1. Introduction

  • OMLT 1.0 transforms trained neural networks and gradient-boosted trees into Pyomo formulations for larger decision-making problems.It supports GBTs through ONNX and neural networks through ONNX and Keras interfaces.Source: §1 · p. 1
  • OMLT automates the otherwise tedious, error-prone translation of trained models into optimization formulations suitable for solver software.A ReLU example switches among complementarity, big-M, and partition formulations in at most one line of code, with 248, 308, and 428 constraints, respectively.Source: §1 · p. 2

2. Use cases for optimizing trained machine learning surrogates

OMLT supports decision-making applications in computer science and engineering by embedding trained machine-learning surrogates in optimization problems. The examples span neural-network verification and engineering design or operations.

2. Use cases for optimizing trained machine learning surrogates

  • Neural-network verification can be formulated as optimization to test whether a differently labeled image exists within a fixed perturbation of a target image.The MNIST notebooks verify dense and convolutional networks; the passage also notes that some related applications, including minimally distorted adversaries and lossless compression, can be implemented in OMLT.Source: §2 · p. 2
  • Engineering examples use neural-network surrogates for complex processes to improve optimization convergence reliability or replace simulation-based models with equation-oriented formulations.The auto-thermal-reformer example develops its surrogate using data from an IDAES-PSE process model.Source: §2 · p. 2

3. Library design

OMLT uses Pyomo blocks and higher-level model representations to generate optimization formulations for neural networks and gradient-boosted trees. Its design supports alternative formulations and lets users connect surrogate inputs and outputs to larger models.

3. Library design

  • Pyomo provides OMLT with solver flexibility, automatic differentiation for nonlinear activations, and extensions for large-scale optimization approaches.Users can select a Pyomo-enabled solver without explicitly interfacing with each solver.Source: §3 · pp. 2–3
  • Pyomo blocks encapsulate surrogate components so users can link model inputs and outputs to a larger formulation while trying competing optimization formulations.The block abstraction also provides a flexible semantic structure for developers creating specialized algorithms.Source: §3 · pp. 2–3
  • OmltBlock delegates formulation generation to a PyomoFormulation and can use user-specified variable scaling and bounds.Scaling and bounds may be absent from ONNX or Keras representations but are required by some formulations.Source: §3 · p. 3
  • For neural networks, OMLT uses NetworkDefinition as an intermediate representation connecting higher-level models to alternative optimization formulations.For gradient-boosted trees, OMLT generates the formulation directly from the higher-level representation.Source: §3 · p. 3
  • OMLT supports dense and convolutional neural-network layers and provides smooth-activation formulations using full-space variables and constraints or reduced-space expressions.The smooth activations listed are linear, softplus, sigmoid, and tanh.Source: §3 · p. 3
  • For ReLU networks, OMLT offers big-M, complementarity, and partition formulations, with the partition formulation limited to dense layers in OMLT 1.0.The library also provides default activation constraints, which users can override to try other formulations.Source: §3 · p. 4

4. Comparison to related work

OMLT combines support for multiple model types, input representations, layers, activation functions, and optimization formulations. Rather than treating formulations as competitors, it presents them as alternatives users can compare for a particular application.

4. Comparison to related work

  • OMLT incorporates both neural networks and gradient-boosted trees, multiple input-model formats, dense and convolutional layers, and several activation functions.The related-work comparison describes OMLT as a more general tool than the listed specialized packages.Source: §4 · p. 4
  • OMLT lets users switch among optimization formulations to find the best fit for a specific application.The paper notes that partition-based and big-M formulations for ReLU networks are often presented as competitors, but OMLT offers them as alternatives.Source: §4 · p. 4

5. Outlook & conclusions

OMLT extends higher-level neural-network and gradient-boosted-tree representations into Pyomo optimization models by automating their conversion to solver-ready variables and constraints. It also enables comparison of alternative formulations.

  • OMLT automates transformation of pre-trained models into variables and constraints suitable for optimization solvers.Source: §5 · p. 4
  • OMLT allows Pyomo optimization models to include neural networks or gradient-boosted trees.Source: §5 · p. 4
  • The implementation makes it possible to compare different optimization formulations presented as competitors in the literature.Source: §5 · p. 4
  • OMLT 1.0 includes ReLU, linear, softplus, sigmoid, and tanh activations.Source: §5 · p. 4

Related papers

Loading 2202.02414v2…