PaperlayerView exact revision on arXiv

Source-linked AI summary

Only Project Once: Projection-Adaptive Loss for Exact Constraint Satisfaction

Tim Aebersold, Soheyl Massoudi, Mark Fuge

arXiv:2610.04572v1cs.LGcs.AIcs.CE

TL;DR

Precise constraint satisfaction matters for deploying learned models, but current repair-aware training unrolls multiple repair steps and softly penalizes remaining violations. PAL trains with one detached projection step and adapts penalties using its residual. It matches or outperforms current methods across the reported benchmarks, including under extremely nonlinear constraints.

  • Problem

    Learned models need to satisfy hard constraints for deployment, while penalty methods typically do not ensure exact satisfaction and repair-based methods rely on costly multi-step training.

    Source: §1 · p. 1

  • Method

    PAL uses one detached projection step and its post-projection residual to adaptively weight constraint penalties on raw predictions.

    Source: §1 · pp. 1–2

  • Results

    PAL achieves best-in-class results simultaneously on synthetic stress tests, engineering applications, and extremely high-curvature constraints; it trains 2.5× faster than DC3 on its flagship benchmark.

    Source: §6 · p. 9

  • Takeaways & Limitations

    A single detached repair step suffices for good feasibility or optimality and enables tractable training when constraints are expensive to evaluate.

    Source: §1 · p. 1, §6 · p. 9

    PAL requires differentiable constraint evaluation, and problems with extremely many constraints remain compute-intensive because Jacobian assembly requires K backward passes.

    Source: §6 · p. 9

Abstract

from arXiv · show

Precise constraint satisfaction is a prerequisite to deploying learned models in many areas, motivating methods that repair raw neural predictions with a repair procedure. Current methods unroll multiple repair steps in training and softly penalize constraint violations that remain after the unroll. This is compute- and memory-intensive, lacks robustness when the repair fails to converge, and surrenders most of the constraint satisfaction work to the repair. Our central finding is that, contrary to common practice, a single detached projection step suffices in training. We accomplish this with a Projection-Adaptive Loss (PAL), which uses the constraint residual after this single step to adaptively weigh constraint penalties on the raw prediction. In experiments, PAL is the only method that retains virtually perfect feasibility on extremely nonlinear constraints, and matches or outperforms current methods on synthetic and engineering benchmarks. Because it only requires a single detached projection step, PAL trains 2.5x faster than the canonical repair-based method (DC3) on its own ACOPF benchmark. PAL can also be trained when constraints are expensive to evaluate (e.g., via neural surrogates), a setting where current unrolled methods are memory-intractable.

1 Introduction

Learned models need precise hard-constraint satisfaction for deployment, but existing approaches either use penalties that typically do not satisfy constraints exactly or rely on repair. PAL uses a single detached projection step and adapts penalties using its residual to shift feasibility work toward the neural network.

1 Introduction

  • Current methods either penalize constraints without typically satisfying them exactly or repair raw predictions to enforce constraints.The need arises in settings including conservation-law modeling, power systems, robotics, and mechanical or aerospace design.Source: §1 · p. 1
  • Unrolled repair is compute- and memory-intensive, soft penalties can be insufficient when projection fails to converge, and the repair often carries most feasibility work.These drawbacks limit expensive constraint evaluation and motivate training the neural network to learn more of the feasibility task.Source: §1 · p. 1
  • PAL uses a single detached projection step during training, matching or outperforming multistep unrolled-repair baselines.Detaching avoids backpropagation through the repair and supports settings where unrolled-repair training is infeasible.Source: §1 · p. 1
  • PAL adapts raw-prediction constraint penalties using the residual after one projection step, shifting feasibility work toward the neural network.The formulation treats projection as final clean-up and aims to avoid feasibility degradation when the projector sequence fails to converge.Source: §1 · p. 2, Figure 1 · p. 2

2 Preliminaries

The paper frames its setting as repeated constrained optimization over a problem family under varying conditions. It defines the model, evaluator, constraints, and projection iterates used in that setting.

2 Preliminaries

  • The focus is amortized optimization, where instances from the same problem family are solved repeatedly under varying conditions.The optimization objective and constraints depend on a condition or learned embedding z.Source: §2 · p. 2
  • The problem minimizes f(y, z) subject to equality constraints h(y, z)=0 and inequality constraints g(y, z)≤0.Here y is the design variable and z is an input condition or learned embedding.Source: §2 · p. 2
  • A neural backbone predicts candidate y values, while the differentiable evaluator returns objective and constraint values.The evaluator Φ is required to be differentiable in y.Source: §2 · p. 2
  • The projector iteratively moves the raw prediction toward the feasible set, with residuals recorded at each iteration.The repair displacement is the difference between consecutive projection iterates.Source: §2 · p. 2

3 Related Work

The related-work discussion covers penalty and repair-based approaches, then identifies limits of existing repair-aware training. PAL is positioned against costly unrolling and fixed penalties on post-repair residuals.

3 Related Work

  • Methods that enforce constraints by construction or use differentiable optimization layers often require known structure that the evaluator does not expose.The paper focuses on smoothly differentiable evaluators without exploitable algebraic form.Source: §3 · p. 3
  • Penalty methods apply fixed or adaptive weights to raw constraint violations, but often fail to satisfy constraints exactly.Adding repair only at test time can yield poor solution quality because the trained model does not anticipate the repair.Source: §3 · p. 3
  • Projection-based repair methods commonly use multiple steps, while FSNet uses gradient descent and avoids full Jacobian construction but requires more repair iterations.The discussion also notes method-specific constraints, including known linear or convex structure for some approaches and known equality-variable dependence for DC3.Source: §3 · p. 3
  • Existing repair-aware methods unroll and backpropagate through multiple repairs and apply fixed-weight soft penalties to post-repair residuals.When the unroll converges, the constraint loss can exert little pressure on the backbone; when it does not, the remaining residual competes with the objective and feasibility degrades.Source: §3 · p. 3

4 Method

PAL combines a projection step with an adaptive loss to train the neural backbone to satisfy constraints while keeping repair as final clean-up. Its training procedure uses one projection iterate and post-projection metrics to adapt penalties.

4 Method

  • PAL's training procedure is presented with a damped Levenberg–Marquardt repair step and can be used analogously with SQP or Interior-Point projection.Algorithm 1 lists the backbone, evaluator, training epochs, and PAL hyperparameters as inputs.Source: §4 · pp. 3–4

4.1 PAL Training

  • For each prediction, PAL evaluates the objective and constraints, assembles the constraint Jacobian, and performs one projection iterate.The paper uses up to 10 projection steps with early stopping at test time, as do its baselines.Source: §4.1 · p. 4, §4 · p. 4
  • The training loss uses the raw prediction's objective and constraint violations together with its displacement from the detached one-step projection.The objective is evaluated at the raw prediction because it progresses more reliably in early training than the repaired objective.Source: §4.1 · p. 4
  • PAL adapts constraint and displacement penalties using the residual after one projection step and the projection displacement.The residual serves as a proxy for how easily each constraint can be repaired; the displacement measures the repair step's distance in y.Source: §4.1 · p. 4
  • The adaptive penalty scheme aims to bring every post-step constraint residual below τc and displacement below τd.Both quantities are trained toward their respective thresholds.Source: §4.1 · p. 4
  • The toy comparison contrasts pre-projection and first-step violations across ALM, soft-loss training, and PAL.ALM does not distinguish constraints by repair difficulty, while PAL applies training pressure to drive first-step residuals below τc.Source: Figure 2 · p. 5, §4.1 · p. 6

5 Experiments

Across synthetic and engineering benchmarks, PAL combines strong feasibility with competitive optimality, including under extreme nonlinearity. Ablations and repair-step comparisons examine its weighting mechanism, projection choices, and inference precision.

5.1 Comparison on Synthetic Stress-Test Benchmarks

  • PAL uses the fewest evaluator passes among projection methods; at high constraint counts, FSNet requires fewer backward passes, while ALM+Bolt-On is cheapest overall but performs poorly.PAL needs two evaluator forward passes and K + 1 backward passes per training step, with K passes building the Jacobian.Source: §5.1 · p. 6
  • Across S1–S6, PAL matches or outperforms repair-aware baselines on optimality; all baselines reach high feasibility except DC3 on ill-conditioned problems.ALM+Bolt-On reaches near-perfect feasibility but very poor optimality, while FSNet performs well except on S5.Source: §5.1 · p. 6, Table 1 · p. 7

5.2 Case Studies in Science and Engineering

  • Across E1–E3, PAL is the only method combining high feasibility with a competitive objective; on E3 it leads in feasibility and optimality while training 2.5× faster than DC3.On E1 and E2, unrolled baselines are out of memory even at batch size 1, with the stated SnareNet exception on E1.Source: §5.2 · p. 7, Table 2 · p. 8

5.4 Ablation Studies

  • Slower adaptive-weight growth can improve optimality, excessive growth can collapse it unless Dmax is low, and tightening τd beyond a sufficiently small value has no effect.Replacing f(ŷ) with f(ỹ(1)) in the training loss degrades both optimality and feasibility.Source: §5.4 · p. 8

5.3 Highly Nonlinear Constraints

  • At curvature κ = 10^6, PAL retains the highest feasibility and reaches optimality on par with full-unroll methods.Its adaptive update raises pressure on pre-projection constraints when the first projection step leaves large residuals.Source: §5.3 · p. 7, Figure 3 · p. 8

5.4 Ablation Studies

  • Disabling either loss term preserves performance, but disabling both collapses optimality and degrades pre-projection feasibility, although the projector alone recovers about 99% feasibility.The results support a non-trivial constraint-pressure role for the backbone in achieving ideal outcomes.Source: §5.4 · pp. 8–9
  • SQP and Interior-Point repair roughly match LM across synthetic benchmarks when the projection step is robust on ill-conditioned Jacobians.Newton projection without damping is identified as a non-robust example.Source: §5.4 · p. 9
  • Single-step training does not limit the precision of constraint satisfaction PAL reaches when more repair steps are used at inference.The study varies the number of inference-time repair steps N.Source: §5.4 · p. 9

6 Conclusion

PAL uses single-step projection to adapt constraint weights, shifting feasibility work toward the neural network and leaving projection as clean-up. The conclusion reports strong results across synthetic, engineering, and high-curvature settings, alongside remaining limitations and future directions.

6 Conclusion

  • PAL is the only method reported to achieve best-in-class results across synthetic stress tests, engineering applications, and extremely high-curvature constraints.The conclusion also states that fully unrolled repair is unnecessary for good feasibility or optimality.Source: §6 · p. 9
  • PAL trains 2.5× faster than DC3 on its flagship benchmark while providing better feasibility and optimality.Single-step repair also reduces constraint-evaluator calls and enables training with expensive constraint evaluations that prevented full-unroll methods from training.Source: §6 · p. 9
  • PAL requires differentiable constraint evaluation, and problems with extremely large constraint counts remain compute-intensive because Jacobian assembly takes K backward passes.Using one projection step alleviates, but does not eliminate, the cost of assembling the Jacobian.Source: §6 · p. 9
  • Future work includes testing PAL beyond amortized optimization and improving efficiency or auto-calibrating its remaining hyperparameters.The authors also note that not backpropagating through repair permits future study of specialized, non-differentiable repair methods.Source: §6 · p. 9

A Notation Summary

  • Table 3 is a notation summary.The supplied passage identifies the table's purpose but does not provide its notation entries.Source: Table 3 · p. 12

B Ablation Study

The ablations examine damping, multiplier updates, tolerances, objective placement, and load-sharing in PAL. They show that multiplier growth and pre-projection objective choice matter, while backbone constraint pressure supports optimality even when projection alone can recover feasibility.

Projector Damping (A–D)

  • Residual-scaled damping is insensitive to δ across the tested sweep, but removing both damping and its floor exposes ill-conditioned systems to failed Cholesky solves and fallback damping.Only S1 runs fully undamped; there, two of ten seeds degrade in objective.Source: B Ablation Study · p. 12

Multiplier-Update Timescale (E–J)

  • Multiplier growth rate γ and per-step cap Dmax act jointly: pairing high γ with low Dmax recovers Base performance, while loosening Dmax alone performs several times worse.Lowering γ marginally improves optimality; raising it collapses optimality as the penalty overwhelms the objective. Tightening Dmax alone halves Base’s worst-bench objective.Source: B Ablation Study · pp. 12–13

Tolerances and Learning Rate (K–L)

  • The displacement threshold τd can be loosened by two orders of magnitude without effect, while the reported learning-rate sensitivity is generic to neural training.The threshold only needs to be small enough.Source: B Ablation Study · p. 13

Pre-Projection Objective (M–O)

  • Replacing f(ŷ) with f(ỹ(1)) degrades both optimality and feasibility across two τd values, while the matched control recovers near-baseline performance.The authors hypothesize that the degradation relates to gradient flow through projection.Source: B Ablation Study · p. 13

Load-Share (P–R)

  • Disabling both PAL load-share terms collapses optimality and raises pre-projection constraint residuals, although the projector still recovers about 99% feasibility.Disabling either term alone preserves objective and feasibility on this benchmark; the authors describe the terms as redundant safety nets here, though both may be needed for stiffer systems.Source: B Ablation Study · p. 13

C Constraint Curvature Ablation

The curvature ablation tests PAL and baselines on a single equality constraint whose curvature varies substantially. PAL’s adaptive constraint pressure increases with curvature and its results outperform full-unroll baselines under high curvature.

C Constraint Curvature Ablation

  • The benchmark varies curvature from 0 to 10^6 for an eight-dimensional problem with one equality constraint and a unique, non-multimodal optimum.Runs use ten seeds, 512 evaluations per seed, double precision, and a feasibility threshold of 10^-4.Source: C Constraint Curvature Ablation · pp. 13–14
  • At κ = 0, the purely linear constraint is repaired exactly in one projection step, leaving PAL with zero constraint-training pressure.All methods reach 100% feasibility with virtually no optimality gap at zero curvature.Source: C Constraint Curvature Ablation · p. 14
  • PAL’s constraint-training pressure rises with curvature and tighter residual targets, reaching 91% at κ = 10^4 and τc = 10^-4.The reported pressure is the ratio of constraint-loss gradient norm to total-loss gradient norm.Source: C Constraint Curvature Ablation · p. 14
  • PAL performs better than existing full-unroll baselines under high-curvature constraints, where single-step residuals trigger adaptive load sharing.The experiment reports that single-step repair does not limit PAL to near-linear constraints.Source: C Constraint Curvature Ablation · p. 14
  • The reported high-curvature runs use double precision; DC3’s undamped Newton completion diverges on ill-conditioned Jacobians, so a damped completion is used instead.Above κ = 10^4, single-precision rounding in y alone produces violations above the feasibility threshold.Source: C Constraint Curvature Ablation · p. 14

D Alternative Repair Steps

The appendix compares SQP and Interior-Point single-step repairs with LM under PAL, finding that SQP matches or improves on LM across six synthetic benchmarks while IP has one feasibility exception.

D Alternative Repair Steps

  • The SQP step uses elastic slacks with penalty ρ = 103, while the Interior-Point step uses barrier weight µ0 = 0.01; both retain PAL’s loss and weight update.At inference, each variant iterates the same repair step with a budget of 10 steps.Source: D Alternative Repair Steps · p. 14
  • 100% feasibility was reached by all repair-step methods on S1–S6, except IP on S3, which reached 86 ± 5%.Table 6 reports mean results across 10 seeds.Source: Table 6 · p. 16
  • SQP matches or slightly improves on LM on all six benchmarks, tying on S2, S3, and S5 and improving on S1, S4, and S6.SQP never underperforms LM.Source: D Alternative Repair Steps · p. 16
  • PAL accommodates LM, SQP, and Interior-Point without adaptation when the repair is a linearized projection robust to ill-conditioned Jacobians.The authors describe this as agnosticism to the specific repair step.Source: D Alternative Repair Steps · p. 16

E Additional Details on f(˜y) vs. f(ˆy) for Training Loss

This appendix explains why placing the objective after projection can stall early training, details the damped LM repair and its convergence rationale, and reports inference-precision and toy-problem demonstrations.

E Additional Details on f(˜y) vs. f(ˆy) for Training Loss

  • Post-projection objective gradients pass through the projector, which attenuates directions normal to the constraint manifold and can impair objective convergence when projection starts early.The authors therefore use the pre-projection objective; its gradients are not routed through the projector.Source: E Additional Details on f(˜y) vs.… · pp. 16–17
  • Four orders of magnitude smaller objective-gradient magnitude left the post-projection-loss variant near initialization, while switching after early pre-projection training retained objective and feasibility quality.The mid-training switch shows the failure mode is specific to early training.Source: E Additional Details on f(˜y) vs.… · p. 17, Figure 5 · p. 18

F Levenberg–Marquardt Projector

  • The LM repair step linearizes active constraints and uses damping to regularize the step; removing damping can degrade objectives and fail on ill-conditioned constraints.The active set is updated between iterates as inequalities become violated or satisfied.Source: F Levenberg–Marquardt Projector · pp. 19–20

F.1 Convergence

  • Under fixed-active-set regularity assumptions, the idealized LM iteration converges quadratically near the active feasible set, but this result does not automatically cover the deployed masked, damped projector.The stated recurrence is for the smooth fixed-active-set map.Source: §F.1 · p. 20
  • A single-step residual below τc is PAL’s practical heuristic for treating a constraint as easy to repair, not a sufficient guarantee of entering the local convergence basin.The proxy aligned with stable projector behavior in experiments, including extremely nonlinear constraints, for sufficiently small problem-dependent τc, typically around 10−4.Source: §F.1 · p. 20

G Inference Precision

  • Inference violations decrease monotonically with repair steps, and both float64 and float32 reach their numerical precision floor after about 2–5 steps except float64 on S5.The first-step target τc = 10−4 is distinct from the precision ultimately reached with additional iterations.Source: G Inference Precision · p. 21, Table 7 · p. 21

H Residual Intuition on a Toy Problem

  • The toy comparison holds the LM projector and its hyperparameters fixed while varying training projection frequency and loss across bolt-on ALM, ten-step Soft Loss, and single-step PAL.The toy problem combines a stiff linear equality with a mildly nonlinear equality.Source: H Residual Intuition on a Toy Problem · pp. 21–22

I Synthetic Benchmark Details

This section details six synthetic benchmarks and several application setups—including chip placement, aircraft evaluation, and urban planning—to examine how projection-based optimization interacts with coupled constraints and differentiable surrogates. It also explains why urban-planning clearance is evaluated pairwise and how the added constraints affect projection and backpropagation costs.

I Synthetic Benchmark Details

  • Five synthetic benchmarks couple objective-relevant head variables to objective-irrelevant tails, so projection can displace the head from its optimum; S5 isolates projection geometry with a finite feasible set.S5 is an overdetermined, consistent equality system with m > n; its feasible set is {±y⋆(z)}, and the implicit-function partition required by DC3 does not apply.Source: I Synthetic Benchmark Details · pp. 22–23

J Chip Placement Case Study

  • PAL reaches 100% feasibility; FSNet and SnareNet reach 0% at substantially higher wall-clock time, while ALM+Bolt-On averages 80%±42% feasibility.ALM+Bolt-On reaches 100% feasibility on 8 of 10 seeds; the remaining two have very high violations. Figure 6 reports the full results; methods use canonical defaults without per-method tuning.Source: J Chip Placement Case Study · pp. 23–24, Figure 6 · p. 23

K.1 Unmanned Aircraft

  • The E1 aircraft application maximizes blended-wing-body range under structural constraints using an evaluator composed of pretrained generative models and neural surrogates.The evaluator includes geometry, aerodynamic, surface-pressure, and structural components.Source: §K.1 · p. 24

K.1.1 Geometry / K.1.4 Iso-slicing

  • E1 uses a neural implicit SDF for differentiable training-time surface queries and a more expensive analytical Bézier SDF as a high-fidelity visualization reference.The neural SDF is queried by quadtree-based marching-squares iso-slicing to obtain surface points, normals, and integration weights; sampled span stations are detached from the gradient graph.Source: §K.1.1 · pp. 24–25, §K.1.4 · p. 26

K.1.2 Aerodynamics / K.1.3 Surface Pressure

  • A smooth MLP predicts global lift, drag, and pitching-moment coefficients, while a FiLM-modulated BlendedNet model predicts surface pressure and shear for structural loading.Global drag comes from the aerodynamic MLP rather than integration of the FiLM shear field.Source: §K.1.2 · p. 25, §K.1.3 · p. 26

K.1.5 Structural Model / K.1.6 Helper Functions

  • The structural evaluator uses a 16-dimensional latent design and an MLP to predict section properties; it assumes ribs add mass but not stiffness, and uses differentiable planform and atmosphere helpers.Structural training targets use SDF-based section geometry and beam integrals rather than FEA; thin-skin contributions to stiffness are retained.Source: §K.1.5 · pp. 26–27, §K.1.6 · p. 27

K.1.7 Beam Quadrature, Mass, and Constraints / K.1.8 Constraints. / K.1.9 Objective

  • The reported objective is −R/Rref with Rref = 10^7 m; maximum material strain uses per-station residuals and is aggregated, with zero aggregate when every station satisfies its strain bound.With K = 1, the formulation gives a smaller Jacobian than one constraint per station; the benchmark also constrains tip deflection using wtip = w(ytip).Source: §K.1.7 · p. 27, §K.1.9 · p. 28, §K.1.8 · p. 28

K.2 Urban Planning

  • The urban-planning setup places 10 buildings on a 750 × 750 m site to maximize built volume under five inequalities, including pedestrian-wind comfort evaluated by differentiable WinDiNet.WinDiNet is an LTX-Video 2B diffusion model that predicts velocity fields over an 1100 × 1100 m raster; the constraint scores the tail-mean field over the last 10 of 57 frames.Source: §K.2 · pp. 28–29
  • Pairwise clearance avoids gradient cancellation from mean-pooling overlaps, giving each pair its own multiplier and Jacobian row to restore directional information and per-pair pressure.A building wedged between violating neighbours can receive near-opposite push vectors, leaving a small center gradient despite gross infeasibility.Source: §K.2 · p. 30
  • With B buildings, clearance creates 1/2B(B −1) constraints and projection Jacobian construction scales with total K, but the expensive wind-model backpropagation occurs only once for Danger.Most constraints require backpropagation only through the clearance graph, while diffusion-model cost dominates.Source: §K.2 · p. 30

K.3 Power Grid Benchmark

The IEEE-57 ACOPF benchmark evaluates PAL on a 576-constraint power-grid task. PAL finishes roughly 2.5 times faster than DC3, although absolute wall-clock times depend on hardware.

K.3 Power Grid Benchmark

  • The IEEE-57 ACOPF task minimizes generation cost while meeting bus demand and respecting nonlinear grid physics, using 128 design variables.The variables include generator setpoints and bus voltages in polar form.Source: §K.3 · p. 30
  • The benchmark has 576 constraints: 114 nonlinear power-balance equalities and 462 inequalities covering generator, voltage, line-flow, and angle limits.PAL assembles Jacobians across all constraint rows; DC3 assembles them over the 114 equalities only.Source: §K.3 · p. 31
  • Compared with DC3’s released implementation, this variant adds 320 thermal-line and angle-difference inequalities, mostly inactive on sampled solutions but included in feasibility metrics.Angle limits are vacuous by construction, while thermal limits can bind on extreme off-manifold predictions.Source: §K.3 · p. 31
  • All methods use the same regenerated condition pool, retaining samples with converged offline ACOPF solves under relaxed thermal limits for internally fair comparisons.The loader and conventions differ slightly from DC3’s, with small adjustments for generator-base and slack-bus bounds.Source: §K.3 · p. 31
  • PAL finishes roughly 2.5 times faster than DC3; its per-step cost is dominated by Jacobian assembly, while DC3 takes several correction steps.Cross-method ratios are reliable, but absolute wall-clock numbers are hardware-specific.Source: §K.3 · p. 31

L Failure Modes of Baselines in Engineering Benchmarks

Appendix L offers hypotheses, rather than an in-depth study, of baseline failures on engineering benchmarks; the following appendices document tuning, implementation, hardware, and hyperparameters.

L Failure Modes of Baselines in Engineering Benchmarks

  • DC3 is structurally inapplicable on E1 and E2, while IPOPT reaches feasible points on E1 but no restart converges on E2.On E2, only 31% of IPOPT final iterates are feasible, and all restarts end with local-infeasibility status.Source: L Failure Modes of Baselines in… · p. 31
  • On E3, all ten SnareNet seeds diverge within 300 epochs, while ENFORCE either diverges ungated or rejects every gated projection step.With its gate enabled, ENFORCE later diverges at test time, producing numerical errors on 2 to 13 of 64 queries per seed.Source: L Failure Modes of Baselines in… · p. 32
  • On E3, the FSNet port uses generic L-BFGS, whereas the FSNet ACOPF paper uses an LM feasibility step that is absent from the public code.The authors speculate that aggregating many mixed-scale constraints into one score weakens FSNet on E3.Source: L Failure Modes of Baselines in… · p. 32
  • On E1 and E2, 39 of 40 ENFORCE seeds run out of memory while assembling the Jacobian; the sole completed E1 run runs out of memory at evaluation.The IFT variant did not reduce peak memory usage.Source: L Failure Modes of Baselines in… · p. 32

M Hyperparameter Tuning

  • Synthetic-suite tuning generally used 32 Bayesian-optimization trials per method and three seeds per trial, then selected among the top three trials using eight seeds.The ten reported result seeds were disjoint from tuning and selection; ALM+Bolt-On and FSNet were exceptions to the general trial budget.Source: M Hyperparameter Tuning · p. 32, Table 8 · p. 33

N Reproducibility

  • The reproducibility appendix states that E3 learned methods follow the published DC3 schedule, while PAL uses defaults and SnareNet uses a five-step repair budget.FSNet otherwise uses the authors’ defaults, including truncated unrolled differentiation of up to 50 repair steps and backpropagation through the first 30.Source: N Reproducibility · p. 33, §N.2 · p. 34

N.1 Hardware and Wall-Clock

  • Runs generally used GH200 nodes, with E1 and E2 using four GH200s per seed and synthetic benchmarks, E3, E4, and ablations run CPU-only.The appendix reports median end-to-end wall-clock times over 10 seeds, or n=5 where the corresponding results tables note it; Table 9 covers S1–S6 and Table 10 engineering benchmarks.Source: §N.1 · p. 33, Table 9 · p. 34, Table 10 · p. 35

N.2 Hyperparameters

  • For ALM+Bolt-On on E1 and E2, inference adds one Levenberg–Marquardt projector pass, and the benchmark sets PAL’s residual-gap target.The appendix describes bidirectional log-gap decay as the default and monotone-only growth as enabled only on E1; Table 11 lists PAL and ALM hyperparameters.Source: §N.2 · p. 33, Table 11 · p. 35

O Compute Scaling

The appendix compares PAL’s evaluator and projection costs with unrolled projection methods and FSNet, and discusses constraint-count and evaluator-cost regimes. It reports PAL’s application to E3 with 576 constraints and faster training than DC3.

Cost Model

  • PAL costs 2Cf + (K + 1)Cb evaluator passes per training step, using two forward passes, K Jacobian-building VJPs, and one loss-backpropagation pass.When K is much larger than dim(y), forward-mode JVPs can assemble the Jacobian in min(K, dim(y)) passes.Source: O Compute Scaling · p. 35

Comparison with Full-Unroll Projection Methods

  • PAL uses one Jacobian assembly, whereas projection-based methods such as DC3, ENFORCE, and SnareNet assemble it at each of N repair iterates.Their stated costs are N(Cf + KCb) for full-unroll methods versus PAL’s single assembly; the appendix identifies PAL as the cheapest member of this class.Source: O Compute Scaling · p. 35

Comparison with FSNet

  • FSNet avoids Jacobian assembly but unrolls many repair steps; the stated PAL–FSNet break-even point is roughly K ≈ 2N.FSNet still stores the unrolled tape, which can be memory-intractable.Source: O Compute Scaling · p. 36

Heterogeneous Constraint Complexity

  • When one or more constraints are exceptionally expensive to evaluate, PAL’s Jacobian-assembly count K virtually drops out of its cost, while FSNet still pays for the full unroll.The appendix gives E2’s diffusion-model wind-speed danger constraint as an example alongside trivial constraints.Source: O Compute Scaling · p. 36

Large-K Applications

  • PAL applies to E3 with K = 576 constraints without modification and trains faster than DC3 despite building the full Jacobian.Table 12 presents full synthetic stress-test results, including evaluator passes per training step, with means and standard deviations across 10 seeds.Source: O Compute Scaling · p. 36, Table 12 · p. 36

Related papers

Loading 2610.04572v1…