Source-linked AI summary
Input Convex Neural Networks
Brandon Amos, Lei Xu, J. Zico Kolter
TL;DR
The paper addresses how to make neural-network inference over selected inputs globally tractable without restricting the network to a manually specified simple architecture. It introduces input convex neural networks and associated optimization and learning procedures, then evaluates them across structured prediction, imputation, and reinforcement learning. The authors report improvements over the state of the art in many settings.
Problem
Existing neural networks do not generally provide globally tractable optimization-based inference over selected inputs, limiting direct use of such inference in structured prediction and related tasks.
Method
The paper introduces scalar-valued ICNNs whose parameters enforce convexity in selected inputs, together with specialized inference, optimization, and learning procedures.
Results
The methods improve over the state of the art in many evaluated settings spanning multi-label prediction, image completion, and reinforcement learning.
Takeaways & Limitations
ICNNs provide a flexible framework in which inference can be performed by optimizing a learned scalar energy over convex inputs.
Takeaways & Limitations
Gradient-descent inference can suffer from step-size and accuracy-assessment issues and is harder to integrate with procedures requiring differentiation through the optimization chain.
Abstract
from arXiv · showhide
This paper presents the input convex neural network architecture. These are scalar-valued (potentially deep) neural networks with constraints on the network parameters such that the output of the network is a convex function of (some of) the inputs. The networks allow for efficient inference via optimization over some inputs to the network given others, and can be applied to settings including structured prediction, data imputation, reinforcement learning, and others. In this paper we lay the basic groundwork for these models, proposing methods for inference, optimization and learning, and analyze their representational power. We show that many existing neural network architectures can be made input-convex with a minor modification, and develop specialized optimization algorithms tailored to this setting. Finally, we highlight the performance of the methods on multi-label prediction, image completion, and reinforcement learning problems, where we show improvement over the existing state of the art in many cases.
1. Introduction
ICNNs are scalar-valued neural networks constructed to be convex in selected inputs, enabling global convex optimization for inference while retaining nonconvexity in other inputs and training. The paper develops these models for structured prediction, imputation, and continuous-action reinforcement learning, and evaluates them across several applications.
- Architecture: ICNNs constrain f(x, y; θ) to be convex in selected inputs y, while allowing nonconvexity in remaining inputs x.The convexity concerns inference over inputs, not the parameter-learning objective.
- Inference: Convexity lets the network replace feedforward prediction with globally solvable optimization over selected inputs given fixed others.The scalar network effectively serves as an energy function for inference.
- Applications: In structured prediction, minimizing the energy over a convex output space yields globally optimized predictions for each input.This distinguishes the approach from energy networks without convexity guarantees.
- Applications: Missing variables can be imputed by minimizing an ICNN over the missing subset while holding observed variables fixed.The paper gives image inpainting as an example.
- Applications: Continuous-action reinforcement learning selects actions by minimizing an ICNN representation of the negative Q function.The action-selection problem is therefore convex in the action when the model is convex there.
- Contributions: The paper proposes ICNN architectures, specialized inference and optimization procedures, training methods, and evaluations on multi-label prediction, image completion, and reinforcement learning.The authors report improvements over the state of the art in many settings.
2. Background and related work
The paper positions ICNNs between manually structured models with easy inference and unrestricted deep models with difficult inference. Its central novelty is to guarantee convexity of the decision space while retaining deep, nonlinear representations over optimized variables.
- Prior approaches: Earlier energy-based structured methods either hard-code architectures for easy inference or leave inference over a general output-space model unrestricted.The paper identifies these as the two prevailing design choices.
- Positioning: ICNNs occupy an intermediate position by constraining inference to be convex while specifying little architecture beyond the convexity-preserving parameter constraints.This preserves a flexible deep architecture over variables being optimized.
- Positioning: Unlike approaches using a structured linear final objective, ICNNs permit deep output-space architectures with multiple nonlinearities.The goal is to learn complex functions over outputs without manually specifying their form.
- Connections: The work relates to MAP-based structured prediction, approximate inference, and earlier energy-based learning approaches.These connections place ICNNs within a broader inference-and-learning literature.
- Convex function fitting: Compared with fitting sums of rectified half-planes, the deep ICNN architecture supports a richer representation class while maintaining convexity.The training scheme is also described as different.
3. Convex neural network architectures
The paper develops fully and partially input-convex architectures by imposing nonnegative-weight and activation constraints that preserve convexity. PICNNs relax full joint convexity, supporting richer models while containing both FICNNs and ordinary feedforward networks.
- Fully input-convex architectures: FICNNs model a neural network over y alone and are convex in the entire input under specific parameter and activation constraints.The architecture is presented as a fully convex k-layer network.
- Fully input-convex architectures: The network is convex in y when intermediate W^(z) weights are nonnegative and activations are convex and non-decreasing.The proof uses closure of convexity under nonnegative sums and convex non-decreasing composition.
- Fully input-convex architectures: Passthrough layers directly connect y to deeper hidden units because nonnegative hidden-to-hidden weights restrict identity-like information flow.These connections compensate for a restriction specific to input-convex architectures.
- Fully input-convex architectures: Convolutions can be included without changing the convexity properties, allowing ReLU-based architectures such as AlexNet, VGG, and GoogLeNet to be made input-convex.The paper evaluates both fully connected and convolutional ICNNs.
- Partially input-convex architectures: PICNNs are convex in y but not necessarily x, avoiding the unnecessary restriction of full joint convexity in applications such as structured prediction.They are defined over input pairs (x, y).
- Partially input-convex architectures: PICNNs allow arbitrary products between x-path and y-path hidden units while requiring nonnegativity only for the relevant hidden-to-hidden weights.This structure increases representational flexibility.
- Partially input-convex architectures: A k-layer PICNN can represent any k-layer FICNN and any k-layer purely feedforward network.This proposition establishes that PICNNs subsume both architecture classes.
4. Inference in ICNNs
ICNN inference is posed as convex optimization over selected inputs, enabling approximate methods that exploit convexity while avoiding impractical exact inference. The section develops gradient-based and bundle entropy approaches, with the latter offering stronger practical performance and an optimality-gap certificate.
- ICNN prediction requires solving a convex optimization problem over the network inputs being inferred.
- Exact inference can be formulated as a linear program for ICNNs with ReLU or linear units, but this approach is impractical.
- Gradient-based inference uses backpropagation to compute ∇_y f(x, y; θ), with projected gradient descent as the simplest approach.
- Gradient descent requires step-size choices, lacks an obvious accuracy criterion for piecewise-linear ReLU objectives, and often underperforms alternative methods.
- The bundle entropy method adapts bundle optimization to bounded Y, using negative entropy as a barrier that keeps solutions in the interior of the feasible hypercube.
- With typically 5 iterations, bundle entropy approximations substantially outperform gradient descent, while the maintained lower bound provides an optimality gap.
5. Learning ICNNs
The paper considers several ways to train ICNNs, including direct functional fitting, max-margin structured prediction, and argmin differentiation. Argmin differentiation trains parameters through predictions obtained by optimization, with gradients computed by implicit differentiation of KKT conditions.
- ICNN training can use direct functional fitting, max-margin structured prediction, or argmin differentiation, depending on the setting.
- Direct functional fitting: Direct functional fitting applies existing fitting procedures in domains such as Q-learning, with feasibility-preserving projection of constrained weights after updates.
- Max-margin structured prediction: Max-margin training posed difficulty in selecting margin scaling for continuous-valued outputs and in supporting losses beyond the hinge loss.
- Argmin differentiation: Argmin differentiation directly minimizes loss between true outputs and predictions produced by approximate optimization over the network inputs.
- Argmin differentiation: The argmin gradient is obtained by implicit differentiation of KKT conditions and has complexity linear in k, the number of active constraints.
6. Experiments
The experiments evaluate ICNNs on multi-label classification, image completion, and continuous-action reinforcement learning. Results are favorable in many settings, while performance varies across tasks and optimization methods.
- Experimental setup: ICNNs are evaluated on BibTeX multi-label classification, Olivetti face completion, and OpenAI Gym continuous-action reinforcement learning.The experiments use macro-F1 for BibTeX, MSE for image completion, and maximum test reward for reinforcement learning.
- Multi-label classification: 0.415 macro-F1 for PICNN exceeds the baseline feedforward network’s 0.396 on BibTeX.SPENs achieve 0.422, but the paper notes that the 0.007 difference may reflect setup, architecture, or random noise.
- Image completion: 833.0 MSE for ICNN with bundle entropy is lower than 872.0 with gradient descent, 850.9 for the nonconvex ICNN, and 942 for sum-product.The results suggest that five inner optimization iterations can be sufficient for good image-completion performance.
- Image completion: The image-completion results indicate that relaxing convexity slightly improves over the convex model, but not over the bundle-entropy method.This suggests that convexity does not substantially inhibit PICNN representational power in this experiment.
- Continuous-action reinforcement learning: ICNNs show clear advantages on HalfCheetah, Reacher, and HumanoidStandup while performing comparably on many other reinforcement-learning tasks.No method strictly dominates; ICNNs also have notable poor performances on Hopper and Walker2D.
7. Conclusion and future work
The paper establishes ICNNs as a framework that combines neural-network function approximation with optimization-based inference. It identifies broader applications as future directions beyond the limited subset studied here.
- Conclusion: Simple parameter constraints let existing network architectures represent general convex functions and use optimization as an inference procedure.The framework makes the inference procedure learned along with the network rather than requiring explicitly constructed structured-prediction architectures.
- Future work: The paper explores only a small subset of ICNN applications, leaving additional domains as promising future directions.
A. Additional architectures
The appendix extends ICNN constructions to convolutional architectures and examines exact inference as a linear program. Although the formulation is exact, its computational demands make it impractical for large neural networks.
- Convolutional architectures: ICNNs can incorporate convolutions because convolution is a linear operator.The construction depends on whether input and output spaces share spatial structure.
- Convolutional architectures: For similarly structured spatial inputs and outputs, convolutional PICNN layers apply separate scaled convolutions to the x and y inputs.The kernels are sized to match the previous feature map, with some Hadamard-product terms omitted for presentation.
- Convolutional architectures: When only the input is spatial, the output-space convolution can be replaced by a matrix-vector operation.This handles outputs with structures such as the simplex.
- Exact inference: For ReLU or linear units, ICNN inference can be formulated as a linear program that reproduces the network equations through positivity and inequality constraints.At the optimum, constraints become tight and recover the ReLU nonlinearity; the same procedure applies to PICNNs.
- Exact inference: The exact linear-program formulation is impractical because it scales with all network activations and requires expensive matrix inversions.These solvers also fail to exploit GPU-accelerated neural-network forward and backward passes.
C. The bundle method for approximate inference in ICNNs
The bundle method exploits convexity by optimizing a piecewise-linear lower bound formed from first-order approximations. The paper adapts this idea for ICNN inference and structured-prediction learning, while retaining nonconvex parameter training.
- Bundle method: The basic bundle method maintains a piecewise-linear global lower bound using first-order cutting planes.Each iteration adds a first-order approximation and optimizes the resulting lower bound.
- Learning: Max-margin structured prediction trains ICNN parameters by requiring lower energy for the target output and adding loss-scaled margins for alternatives.Suitable margin-scaling choices preserve convexity of the loss-augmented inference problem.
- Learning: Training remains nonconvex in the network parameters and can use subgradient updates with projection onto the non-negative orthant.The procedure can be adapted to mini-batches, AdaGrad, ADAM, or approximate inference.
G. Proof of Proposition 3
The proof derives Jacobians for the optimization-based inference procedure using chain-rule calculations and implicit differentiation of the active KKT conditions. It then obtains Jacobian products and gradients for the relevant variables.
- The proof begins by applying the chain rule to derive the proposition’s expressions.
- Implicit differentiation of the active KKT conditions computes the challenging derivative terms.
- Differentiating with respect to h yields a linear system whose solution provides the corresponding Jacobian.
- Differentiating with respect to G produces analogous matrix equations and corresponding Jacobian products or gradients.
- The derivation concludes by substituting the defined quantities into the formula stated in the proposition.
H. State and action space sizes in the OpenAI gym MuJoCo benchmarks.
The section presents the state and action space sizes for the OpenAI Gym MuJoCo benchmarks. These benchmark dimensions contextualize the reinforcement-learning experiments.
- Table 4 lists state and action space sizes for the OpenAI Gym MuJoCo benchmarks.
J. Multi-Label Classification Training Plots
The training plots compare model learning and test performance across the paper’s experiments. On BibTeX multi-label classification, the PICNN achieves a higher final test macro-F1 than the feedforward network.
- Figure 5 plots training and test macro-F1 for a feedforward network and PICNN on BibTeX multi-label classification.
- 0.415 final test macro-F1 for PICNN exceeds 0.396 for the feedforward network.
- Figure 6 reports train and test image-completion losses for PICNNs using bundle entropy, back optimization, and relaxed convexity constraints.