PaperlayerView exact revision on arXiv

Source-linked AI summary

Imposing Hard Constraints on Deep Networks: Promises and Limitations

Pablo Márquez-Neila, Mathieu Salzmann, Pascal Fua

arXiv:1706.02025v1cs.CV

TL;DR

Deep networks often use soft output constraints, but these require difficult penalty weighting and do not guarantee satisfaction; hard constraints promise a principled alternative for using limited labeled data. The paper develops a Lagrangian method with Krylov subspace solvers to make hard constraints computationally feasible. Experiments show that hard constraints are tractable and produce meaningful results, but do not outperform softer, cheaper alternatives because constraints are not perfectly maintained.

  • Problem

    Soft output constraints require difficult penalty weighting and do not guarantee constraint satisfaction, while generic hard constraints on deep networks were considered impractical.

  • Method

    The paper trains deep networks with a Lagrangian hard-constraint method that uses Krylov subspace approaches to solve very large linear systems.

  • Results

    Hard constraints are computationally feasible and yield meaningful results, but perform no better than soft constraints, which are less computationally demanding.

  • Takeaways & Limitations

    Hard constraints’ expected theoretical benefits do not materialize in this setting, motivating better ways to preserve their advantages.

  • Takeaways & Limitations

    Using only a changing subset of active constraints prevents perfect constraint satisfaction and discards effects from previously used constraints.

Abstract

from arXiv · show

Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.

1. Introduction

Output constraints can improve deep-network predictions and exploit unlabeled data when labeled data are scarce. Hard constraints avoid soft-constraint weighting and satisfaction issues, and Krylov methods make large constrained systems computationally feasible, although soft constraints perform better.

  • Motivation: Output constraints provide weak supervision or domain knowledge, helping deep networks when labeled training data are limited.Examples include data augmentation, synthetic data, and symmetry constraints for human pose estimation.
  • Soft and hard constraints: Soft constraints add penalty terms to the loss, requiring relative weights that are difficult to choose.They also provide no guarantee that constraints will be satisfied in practice.
  • Soft and hard constraints: Hard constraints can automatically select constraint weights and satisfy feasible constraints to machine precision.These benefits motivate replacing soft penalties with constrained optimization.
  • Contribution: Krylov subspace methods make the large linear systems arising in constrained optimization computationally solvable for deep networks.The resulting method is slow but returns meaningful results.
  • Main result: Soft constraints produce better results than hard constraints while requiring substantially less computation.The paper attributes this counterintuitive outcome to difficulty ensuring independence among linearized constraints.

2. Related Work

The paper situates hard output constraints within risk minimization and constrained optimization. It targets generic nonlinear constraints on network outputs, addressing limitations of penalty-based methods and prior specialized approaches.

  • Deep learning formulation: Deep learning trains network parameters to minimize a risk function so predictions approximate labeled outputs.The parameter vector can be very high-dimensional.
  • Constraint formulations: Soft constraints add penalties to the risk, creating a compromise between prediction loss and constraint satisfaction.Hard constraints instead impose equality conditions directly during optimization.
  • Constraint formulations: Penalty methods can have poor convergence and require difficult weighting when constraints differ in nature.Constrained optimization addresses these issues by optimizing constraint-term weights during optimization.
  • Related work: Prior work applied constrained optimization to kernelized latent-variable models, Gaussian Processes, and neural-network outputs.The cited image-segmentation approach used linear constraints to synthesize training labelings from network outputs.
  • Research gap: This paper addresses completely generic nonlinear constraints, a problem not covered by the specialized solution discussed in related work.The distinction motivates the paper’s broader constrained-learning formulation.

3. Formalization

The paper formulates data-dependent output constraints over labeled and unlabeled inputs, contrasting direct hard enforcement with weighted soft penalties. It studies optimization methods suited to very large deep-network parameter spaces.

  • Problem setup: Training minimizes the risk of a network while seeking parameters whose outputs approximate labeled targets.The parameter vector w is learned from the labeled dataset D.
  • Data-dependent constraints: Hard constraints are imposed on network outputs rather than parameters, especially for relative joint positions in 3D human pose estimation.These data-dependent constraints explicitly incorporate each input vector.
  • Data-dependent constraints: Because labeled examples implicitly satisfy the constraints, the formulation introduces unlabeled inputs on which constraints must also hold.The constraints are applied for every selected unlabeled point.
  • Hard versus soft formulations: Soft constraints weight each constraint with a positive scalar, trading off risk minimization against constraint satisfaction.Too-small weights neglect constraints, whereas too-large weights neglect other objective terms.
  • Optimization: The soft-constraint problem can use standard deep-learning optimizers such as momentum SGD or Adam.The paper studies constrained optimization in the setting of very large parameter vectors.

4. Dealing with Millions of Variables

The paper handles high-dimensional hard-constraint optimization by solving implicit KKT linear systems with Krylov subspace methods and Jacobian-vector operators. It reduces system size through active-constraint selection, while recognizing ill-conditioning from incompatible or repeated constraints.

  • Millions of parameters and large datasets make direct constrained optimization impractical for deep networks.
  • KKT-based iterations compute parameter increments and Lagrange multipliers by repeatedly solving linearized constrained systems.The method alternates between linearization and updates because the loss and constraints are generally nonlinear and nonconvex.
  • Krylov subspace methods avoid explicitly storing the immense system matrix by using matrix-vector products instead.These products are computed through R-op and L-op directional-derivative operations supported by neural-network backpropagation.
  • MINRES-QLP is selected because the KKT matrices are squared, symmetric, non-positive-definite, and often ill-conditioned.Ill-conditioning commonly results from incompatible or repeated constraints, especially when constraints are linearized over many samples.
  • Selecting active constraints reduces system size, including by sampling constraints or prioritizing those with the largest violations.The same mechanism can extend to inequalities by activating violated constraints and ignoring satisfied ones.
  • The constrained method can incorporate an Adam-like step using first- and second-order moment vectors as optimization quantities.The constrained Adam system uses m as the current gradient and √v as the parameter-wise learning rate.

5. Experiments

The experiments compare soft- and hard-constrained deep networks for 3D human-pose estimation using matched training and constraint datasets. They evaluate prediction error and constraint violation across several optimization variants.

  • Dataset and task: The experiments use Human3.6m Walking images to predict the 3D coordinates of 17 human-body joints.The dataset contains 162008 color images cropped to 128×128 pixels and is split into 80%-20% training and testing subsets.
  • Model and constraints: The network uses convolutional layers, max-pooling, ReLU nonlinearities, and fully connected layers, with a deliberately simple architecture for comparing constraint methods.The architecture is not state-of-the-art but is described as sufficient for comparative experiments.
  • Model and constraints: Six symmetry constraints enforce equal left-right limb lengths and equal shoulder- and hip-to-spine distances.Each constraint compares distances between specified pairs of predicted joints.
  • Evaluation: The evaluation measures prediction error as mean Euclidean joint-position distance and constraint violation as the median absolute value of all C_jk.The constrained models use either random constraint minibatches of 128 samples or hard constraint mining with 16 samples.
  • Results: Soft-Adam performs best, closely followed by Hard-SGD, while hard methods are generally similar to soft methods and do not enforce constraints perfectly.The imperfect enforcement occurs because only subsets of constraints are used at each iteration, allowing previously satisfied constraints to become violated.
  • Results: Hard-constraint methods require substantially more computation: Soft-SGD and Soft-Adam take 0.8–1.6 s/it, Hard-SGD 3–32 s/it, and Hard-Adam 103 s/it.The paper summarizes hard-constraint techniques as 10–100 times slower on average than soft-constraint counterparts.

6. Discussion

Hard-constraint learning is computationally tractable but does not outperform soft constraints because changing active subsets can make constraint satisfaction unstable. Synthetic experiments link this behavior to ill-conditioned KKT systems caused by dependent linearized, data-dependent constraints.

  • Discussion: Hard-constraint learning is computationally tractable but more expensive than soft-constraint learning and does not guarantee perfect constraint satisfaction or better performance.In human pose estimation, constraints may stop being satisfied when only a subset is enforced at each iteration.
  • Discussion: Data-dependent constraints can have linearly dependent linearizations, making KKT systems ill-conditioned and complicating MINRES-QLP solves.The paper identifies this as the root cause of difficulties with hard constraints.
  • Demonstration on a Synthetic Example: The synthetic problem minimizes distance to x0 while requiring w to lie at the intersection of 200 slightly incompatible hyperspheres.The hypersphere centers are normally distributed around the origin, with variance 0.01 and radius two orders of magnitude larger.
  • Demonstration on a Synthetic Example: With two nearly parallel linearized constraints, hard updates move toward a distant hyperplane intersection, creating a worst-case optimization step.The experiment compares soft and hard updates from the same starting point and extends the comparison across many iterations.
  • Demonstration on a Synthetic Example: In the large synthetic experiment, soft constraints initially improve steadily, whereas hard-constraint updates are unpredictable and chaotic before reaching a similar loss.The experiment uses d = 1e6, 20 randomly selected active constraints per iteration, and 500 iterations.

7. Conclusion

The paper shows that hard constraints can be trained efficiently enough for deep networks by solving very large linear systems with a Krylov-subspace-based Lagrangian method. However, experiments found good performance without improvement over soft constraints, partly because computational limits required replacing the active constraint subset at every iteration.

  • Conclusion: A Lagrangian method using a Krylov subspace approach makes training deep networks with hard output constraints practical.The approach targets the very large linear systems generated by constrained optimization.
  • Conclusion: The proposed hard-constraint approach performs well but not better than a soft-constraint approach.This outcome contradicts the expected advantage of hard constraints.
  • Conclusion: Computational limits required selecting a new active-constraint subset at every iteration, forgetting the effects of previously used constraints.The paper identifies this as the explanation for the negative result and notes that implementing persistent active constraints is difficult for deep networks.
Loading 1706.02025v1…