PaperlayerView exact revision on arXiv

Source-linked AI summary

Optimal Control Via Neural Networks: A Convex Approach

Yize Chen, Yuanyuan Shi, Baosen Zhang

arXiv:1805.11835v5math.OC

TL;DR

The paper addresses the difficulty of using accurate but nonlinear and nonconvex neural-network models in tractable control. It constructs input-convex neural and recurrent networks for system identification and solves convex model predictive control problems, with experiments demonstrating potential across locomotion and building energy-management tasks.

  • Problem

    Neural networks can model complex system behavior but are difficult to optimize over for control because they are typically nonlinear and nonconvex.

  • Method

    The paper constructs input-convex neural and recurrent networks that learn system dynamics and support convex model predictive control.

  • Results

    Experiments on MuJoCo locomotion and building energy-management tasks demonstrate the methodology’s potential.

  • Takeaways & Limitations

    The framework combines data-driven system-dynamics representation with tractable convex optimization for control applications.

Abstract

from arXiv · show

Control of complex systems involves both system identification and controller design. Deep neural networks have proven to be successful in many identification tasks, however, from model-based control perspective, these networks are difficult to work with because they are typically nonlinear and nonconvex. Therefore many systems are still identified and controlled based on simple linear models despite their poor representation capability. In this paper we bridge the gap between model accuracy and control tractability faced by neural networks, by explicitly constructing networks that are convex with respect to their inputs. We show that these input convex networks can be trained to obtain accurate models of complex physical systems. In particular, we design input convex recurrent neural networks to capture temporal behavior of dynamical systems. Then optimal controllers can be achieved via solving a convex model predictive control problem. Experiment results demonstrate the good potential of the proposed input convex neural network based approach in a variety of control applications. In particular we show that in the MuJoCo locomotion tasks, we could achieve over 10% higher performance using 5* less time compared with state-of-the-art model-based reinforcement learning method; and in the building HVAC control example, our method achieved up to 20% energy reduction compared with classic linear models.

I. INTRODUCTION

The paper addresses the tradeoff between neural networks’ modeling accuracy and control tractability by constructing input-convex networks for system identification and optimal control. Its approach learns system dynamics with input-convex networks, including recurrent architectures, then solves convex model predictive control problems.

  • Motivation: Complex physical systems often have complicated or poorly understood dynamics, while detailed physics models may be unavailable or intractable to construct.Building physics models can take years, motivating data-driven alternatives.
  • Motivation: Deep neural networks capture complex relationships but are typically nonlinear and nonconvex, making optimization for control challenging.This difficulty has led many real-time control applications to favor tractable linear models despite poor fitting performance.
  • Approach: The proposed method uses input-convex neural networks to represent system dynamics and obtain tractable computational optimization problems.Convexity is imposed from network input to output to balance predictive accuracy with control tractability.
  • Approach: The method first learns system dynamics with an input-convex network and then computes control decisions by solving a convex model predictive control problem.The resulting MPC problem is described as tractable and having optimality guarantees.
  • Contributions: The paper claims that input-convex neural networks can represent all convex functions and system dynamics and are exponentially more efficient than widely used convex piecewise-linear approximations.This is presented as a theoretical contribution alongside the control framework.
  • Approach: The work extends input-convex networks to recurrent architectures to capture dynamical systems and temporal behavior.The authors also position the framework as distinct from model-free end-to-end controllers that directly map inputs to outputs.

II. CLOSED-LOOP CONTROL WITH INPUT CONVEX NEURAL NETWORKS

The paper constructs input convex neural networks for closed-loop decision making, extending them to recurrent architectures for dynamical systems. Convexity makes the resulting optimization tractable while retaining neural-network representation power.

  • The framework targets closed-loop systems where optimizing system performance goes beyond evaluating the network’s learning performance alone.
  • All-positive weights preserve convexity when multiple network steps are composed for dynamical systems.The construction uses the linear constraint v = −u, which can be included in convex optimization.
  • The proposed architecture adds direct passthrough layers and expanded inputs containing both u and −u.These changes support representation power while maintaining convexity under nonnegative-weight restrictions.
  • ICNNs enable neural networks to participate in decision-making optimization while guaranteeing unique and globally optimal solutions.
  • ICNN optimization over a convex feasible space is a convex problem that can be solved efficiently to global optimality.The approach trades some potential predictive or classification performance for tractability.

B. Closed-loop control and recurrent neural networks

Input convex recurrent neural networks model temporal system behavior while preserving convexity with respect to control inputs. Under convex cost and feasible-set conditions, they yield efficiently solvable optimal receding-horizon control problems.

  • B. Closed-loop control and recurrent neural networks: ICRNNs extend ICNNs to capture complex dynamics and make time-series control decisions.Recurrent networks carry an internal state and can represent temporal dependencies.
  • B. Closed-loop control and recurrent neural networks: The control formulation uses ICRNNs for system dynamics and outputs over a finite memory window and prediction horizon.The cost depends on system inputs and outputs, while state and action feasibility are imposed as constraints.
  • B. Closed-loop control and recurrent neural networks: The proposed ICRNN maps duplicated control inputs through recurrent dynamics so future states and outputs remain convex in the controls.The input vector includes both control actions and their negations, with consistency constraints linking them.
  • B. Closed-loop control and recurrent neural networks: The optimization is convex when the cost is convex in controls, convex and nondecreasing in states and outputs, and feasible sets are convex.Its equality constraints are affine, and gradients can be computed by back-propagation toward the inputs.
  • B. Closed-loop control and recurrent neural networks: Convexity guarantees that the control problem can be solved efficiently and optimally using gradient descent.
  • B. Closed-loop control and recurrent neural networks: The receding-horizon controller applies the first action of the optimized sequence, then repeats the optimization after updating the state prediction.

III. EFFICIENCY AND REPRESENTATION POWER OF ICNN

This section analyzes the representation ability and efficiency of input convex neural networks as system-identification models. The analysis addresses both predictive accuracy and computational tractability.

  • The theoretical analysis evaluates ICNN representation ability and efficiency alongside their computational tractability as system-identification models.

A. Representation power of input convex neural network

The paper proves that ICNNs and ICRNNs can approximate broad classes of convex functions and dynamics with nonnegative weights and ReLU activations. It also argues that neural-network parameterizations can be more efficient than directly learning maxima of affine functions.

  • A. Representation power of input convex neural network: For any Lipschitz convex function on a compact domain, a nonnegative-weight ReLU ICNN can approximate it within ε.
  • A. Representation power of input convex neural network: The construction achieves nonnegative weights by duplicating each input as x and −x and enforcing a linear consistency constraint.The proof first approximates a convex function by a maximum of affine functions.
  • A. Representation power of input convex neural network: The same representational result extends to ICRNNs for systems with convex rolled-out dynamics.The paper states that a nonnegative-weight ReLU recurrent network can approximate such dynamics within ε.
  • A. Representation power of input convex neural network: The framework can capture a broad range of systems, including linear-quadratic-Gaussian regulation and convex battery degradation control.Battery degradation is described as convex in charge and discharge actions, supporting a data-driven control formulation.
  • B. ICNN vs. convex piecewise linear fitting: The paper motivates learning ICNNs instead of directly fitting maxima of affine functions because ICNNs provide a more efficient parameterization.
  • B. ICNN vs. convex piecewise linear fitting: An ICNN with K ReLU activations may require Ω(2^K) affine functions for exact max-affine representation, whereas K affine functions require O(K) activations in an ICNN.

IV. EXPERIMENTS

Experiments evaluate ICNN-based control on MuJoCo locomotion tasks and building energy management. Across locomotion settings, ICNN control outperforms random shooting and is faster and more sample-efficient.

  • Experiments cover MuJoCo locomotion benchmarks and large-building energy management.ICNN represents MuJoCo dynamics, while ICRNN is used end-to-end for building control.
  • A. MuJoCo Locomotion Tasks: ICNN policies can initialize subsequent model-free reinforcement learning algorithms.This makes the learned policies usable beyond the direct model-based control procedure.
  • A. MuJoCo Locomotion Tasks: ICNN predicts robotic states accurately and supports finite-horizon convex predictive control.The controller optimizes actions over a look-ahead horizon using the learned dynamics model.
  • A. MuJoCo Locomotion Tasks: ICNN outperforms random shooting across all four locomotion tasks and varying horizons T.The comparison uses random shooting with K = 100 and validation rollouts during aggregated training iterations.
  • A. MuJoCo Locomotion Tasks: Larger action spaces expose random shooting’s search limitations, illustrated by stagnating ant control performance.With additional samples and more accurate MLP dynamics, random shooting still becomes stuck finding better policies.
  • A. MuJoCo Locomotion Tasks: ICNN control is much faster than random shooting because it avoids evaluating and selecting among shooting trajectories.In the Swimmer case, the method uses 1⁄5 of the time compared with the baseline and can use two orders of magnitude less training data than model-free RL methods for similar validation rewards.

B. Building Energy Management

Building HVAC control uses an input convex recurrent model to forecast dynamics and optimize future energy consumption under comfort constraints. ICRNN achieves competitive fitting accuracy, higher savings than the baselines, and more stable actions.

  • B. Building Energy Management: ICRNN models building dynamics from recurrent input-output histories and state transitions.The model uses a memory window and predicts building energy consumption and state evolution.
  • B. Building Energy Management: The building task optimizes HVAC actions over future steps while maintaining temperature and system-state constraints.The objective minimizes total energy consumption over a model predictive control horizon, with temperature comfort regions enforced.
  • B. Building Energy Management: Conventional RNN optimization has no guarantee of reaching a global or even local optimum.This limitation arises because conventional RNNs are generally nonconvex from inputs to outputs.
  • B. Building Energy Management: ICRNN test RMSE is 0.054 versus 0.051 for conventional RNN and 0.240 for the RC model.The ICRNN therefore provides competitive fitting performance while substantially outperforming the linear RC model on this metric.
  • B. Building Energy Management: 23.25% energy reduction follows ICRNN temperature-setpoint control under the same [19°C, 24°C] temperature constraints.Conventional RNN reduces electricity by 11.73%, while the RC model saves 4.07%.
  • B. Building Energy Management: ICRNN produces stable control actions, whereas conventional RNN set points show undesirable drastic variations.The comparison examines two randomly selected building zones.

V. SUMMARY AND DISCUSSION

The paper proposes neural networks engineered to be convex from input to output, using input convex networks for system dynamics and data-driven control. Experiments indicate potential across locomotion and building energy-management problems.

  • The framework constructs deep neural networks that are convex from inputs to outputs.This design targets the gap between neural-network identification accuracy and tractable control optimization.
  • Input convex and recurrent neural networks represent system dynamics for optimization and closed-loop control.The approach uses ICNNs and ICRNNs to connect learned dynamics with control decisions.
  • The proposed architecture casts data-driven control problems as convex optimization problems.The paper evaluates this methodology on MuJoCo locomotion and building energy-management benchmarks.

APPENDIX

The appendix illustrates ICNN decision boundaries and develops their construction and approximation properties. It shows how nonnegative-weight architectures preserve input convexity while representing convex piecewise-linear functions.

  • A. Toy Example: The toy optimization minimizes a neural-network output to find an input with the desired class probability.The paper compares conventional and input convex classifiers trained on the same circle-data setup.
  • A. Toy Example: In the circle-classification example, conventional-network decision boundaries have many zigzags, while ICNN level sets are convex by construction.Convex level sets make the associated optimization problem convex.
  • Appendix B. Proof of Theorem 1: Continuous Lipschitz convex functions on convex compact sets can be approximated using maxima of affine functions and ICNNs.The proof connects convex piecewise-linear representations with ICNN expressivity.
  • Appendix B. Proof of Theorem 1: An ICNN can exactly represent maxima of affine functions using ReLU layers and a linear second layer.The constructed two-layer network is shown to implement the same function as the maximum-of-affine representation.
  • Appendix B. Proof of Theorem 1: Nonnegative interlayer weights and passthrough links make the network input convex after expanding the input to include x and −x.The consistency relation for the expanded input is imposed as a linear equality constraint, preserving convexity.

Appendix C. Proof of Theorem 2

The proof constructs an ICNN whose piecewise-linear representation can require exponentially many affine pieces, while showing that K layers can represent a maximum of K affine functions.

  • A K-layer ICNN with a single ReLU per layer can represent the maximum of K affine functions.
  • A one-hidden-layer ICNN with K nodes contains 3K parameters: w0i, w1i, and bi.
  • Representing the same function as a maximum of affine functions requires evaluating every potential piece.
  • The constructed network has 2K potential combinations of piecewise-linear functions.
  • Consequently, representing the single linear ICNN as a maximum over affine pieces requires considering 2K pieces.

Appendix D. Experimental Details on MuJoCo Tasks

The MuJoCo experiments train ICNN and MLP dynamics models from rollout data and compare ICNN-based control with random shooting and TRPO. ICNN achieves comparable multistep prediction quality while finding effective policies efficiently and supporting model-free initialization.

  • Training and data collection: ICNN and MLP dynamics models are trained from random and policy-generated rollouts, with Gaussian noise added during aggregated iterations.
  • Training and data collection: Both models use two hidden layers of size 512−512, Adam optimization, a 0.001 learning rate, and mini-batches of 512.
  • Control results: ICNN control uses less or equal training time than random shooting with K = 100 while achieving better rewards than K = 1000 across different horizons.
  • Dynamics modeling: ICNN multistep prediction errors are comparable to those of the normal MLP across different rollout lengths, including the complex ant dynamics.
  • Control results: The ICNN approach is more efficient than random shooting for finding control policies and can provide initialization samples that accelerate model-free reinforcement-learning training.
  • Control results: With few rollouts, TRPO is hard to learn directly, whereas ICNN first learns dynamics and then finds actions through optimization.

Appendix E. Details on Building Energy Management

The building-energy experiments use ICRNNs to optimize electricity cost under time-varying prices and control constraints. The model captures long-term effects and shifts consumption away from peak-price periods.

  • Problem formulation: The building-control objective minimizes total energy cost under a varying time-of-use electricity price λ.
  • Problem formulation: ICRNNs parameterize the building-state model within the predictive-control optimization problem.
  • Problem formulation: The building example imposes constraints on control actions, system states, temperature set points, and real measurements.
  • Energy-management results: ICRNN captures long-term relationships between control variables and final costs, slightly increasing off-peak consumption while reducing peak-hour consumption.
  • Energy-management results: Under different set-point constraints, ICRNN finds similar off-peak solutions, corresponding to necessary consumption such as lighting and ventilation.
  • Energy-management results: Without system constraints, ICRNN cuts more than 80% of total energy during peak hours.
Loading 1805.11835v5…