PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision-Driven Regularization: A Blended Model for Learning and Optimization

Gar Goei Loke, Qinshen Tang, Yangge Xiao, Xun Zhang

arXiv:2608.15124v1cs.LGmath.OC

TL;DR

The paper addresses limited control for prediction in integrated learning and optimization and proposes decision-driven regularization as a blended bi-objective framework. It reports superior numerical performance and identifies bias-variance trade-offs as relevant to estimation.

  • Problem

    Integrated learning and optimization literature lacks careful control for prediction, while separating learning from optimization may be suboptimal.

  • Method

    The paper proposes decision-driven regularization, a blended bi-objective framework for integrated learning and optimization.

  • Results

    The paper reports superiority of SLO under the non-misspecified regime and attributes DDR’s superior numerical performance over benchmark models to improved estimation.

  • Takeaways & Limitations

    Applying the bias-variance trade-off can improve estimation within the proposed framework.

  • Takeaways & Limitations

    Further work is needed to examine specialization of decision-driven regularization to other learners, such as neural networks.

Abstract

from arXiv · show

In contextual optimization, the decision-maker seeks optimal decisions to minimize a cost function, that varies based on observed features. This context is common in many business applications ranging from on-demand delivery and retail operations to portfolio optimization and inventory management. In this paper, we study the learning and optimization approach, which first learns how outcomes result from the features, and then selects optimal decisions based on these outcomes. We focus on the integrated learning and optimization literature, and identify that a lack of control for prediction accuracy can lead to overfitting and a loss of decision effectiveness against simple separate learning and optimization models. Instead, we propose a bi-objective formulation that balances prediction accuracy and cost minimization, termed decision-driven regularization. It also addresses ambiguity in the definition of the cost function via a surrogate that depends on a new hyperparameter. We additionally show that alternative perspectives for formulating the problem, namely robust optimization and regret minimization, lead to models that are closely related to our proposed model. As a consequence, our framework generalizes models such as SPO+. Our model is shown to be numerically superior to other benchmarks, such as OLS, Random Forest, XGBoost, SPO+, Perturbation Gradient, and Learning and Rank, in our synthetic studies.

1. Introduction

Contextual stochastic optimization uses features to estimate uncertain outcomes and select adaptive decisions, but separating prediction from optimization can be suboptimal. The paper proposes decision-driven regularization, which balances prediction accuracy with cost minimization and is numerically superior to examined benchmarks.

  • Contextual stochastic optimization: Contextual stochastic optimization uses observed features to estimate uncertain outcomes and adapt decisions to those features.The setting appears in applications including delivery, retail, portfolio optimization, and inventory management.
  • Learning and optimization: Learning and optimization proceeds by fitting a predictive model for z|x and then minimizing expected cost under the predicted distribution.Sequential learning and optimization separates these stages, including predict-then-optimize and estimate-then-optimize approaches.
  • Motivation: High prediction accuracy does not necessarily produce good downstream decisions, because estimation-loss and decision-cost objectives can be misaligned.The literature identifies potential suboptimality from separating learning and optimization, while parameter directions minimizing prediction loss may differ from those minimizing decision cost.
  • Integrated learning and optimization: Integrated learning and optimization embeds the optimization problem in training and selects parameters by empirical decision cost rather than conventional accuracy loss.This approach includes adaptations of both PTO and ETO methods, as well as robust models addressing uncertainty in distribution estimation.
  • Evidence motivating the proposal: ILO models often outperform OLS only under high misspecification, while OLS shows consistent superiority across multiple applications and cost functions against ILO models.These observations motivate controlling prediction accuracy rather than relying exclusively on decision-cost minimization.
  • Contributions and results: Decision-driven regularization is a blended bi-objective framework balancing prediction accuracy and cost minimization through a surrogate that combines empirical and estimated cost.The paper relates this formulation to robust optimization and regret minimization and reports that DDR outperforms examined benchmarks under low misspecification; SPO+ can outperform DDR under large samples and high misspecification, but not when small-order polynomial terms are added to both learners.

2. Decision-Driven Regularization

Decision-driven regularization combines prediction accuracy with cost minimization to address drawbacks of cost-driven integrated learning and optimization. The framework anchors cost-related learning in observed outcomes while retaining closeness to true outcomes.

  • Learning and optimization pipeline: The contextual optimization pipeline first estimates outcome-model weights from noisy training data, then solves an approximate decision problem using the resulting predictor.The framework uses a parametric map g(x) = f(x;w) and produces predictions f(x;ŵ) for new features before optimization.
  • Motivation: Unlike separate learning and optimization, integrated approaches can select weights using cost or regret without accounting for prediction accuracy.The paper identifies this cost-driven criterion as a potential source of misalignment between lower prediction loss and better decisions.
  • Motivation: Under low misspecification, SPO+ is systematically outperformed by an approach placing greater emphasis on predictive accuracy.The paper reports recovering this pattern in its numerical simulations.
  • Illustration: decision regions: In the route-choice illustration, OLS recovers the ground-truth decision boundary, whereas cost-driven ILO produces a degenerate region that selects route b sub-optimally.OLS chooses route b iff 1.5x̂_2 ≤ x̂_1; ILO chooses it iff x̂_2 ≤ ωx̂_1 for 1 ≤ ω < 2, adding regions where route a is optimal.
  • Illustration: decision regions: Decision-focused learning uses noisy observed outcomes to define its in-sample objective, which can introduce noise into the optimization criterion.In the illustration, noisy costs label the route-choice data and flip a decision relative to the oracle outcome.
  • Decision-driven regularization: DDR proposes a blended bi-objective framework that accounts for cost structure while retaining closeness to true outcomes.The cost function is anchored using observed outcomes, and the formulation includes a regularization parameter and a surrogate controlled by an additional parameter.

3. Relationship between regularized learning, robust optimization and regret minimization

The paper connects DDR to robust optimization and regret minimization, showing that these perspectives yield closely related formulations under stated assumptions. These connections establish DDR as a unifying framework while clarifying the assumptions and scope of its theoretical interpretations.

  • Robust optimization: Robust-DDR balances prediction accuracy and cost minimization through worst-case weights over an uncertainty set shaped by the loss function.The uncertainty set has statistical interpretations tied to the geometry of the loss.
  • Robust optimization: Under convex compact decision sets and concavity of f(x;w) in w, Robust-DDR and DDR attain coinciding optimal weights when µ ∈ [0,1).Theorem 1 provides the correspondence for any λ > 0 and an associated ρ > 0.
  • Robust optimization: The negative regularization sign in DDR is explained by the robust formulation, whereas a positive sign would impose poorer prediction accuracy in the uncertainty set.The paper interprets the negative sign as consistent with learning-oriented uncertainty constraints.
  • Scope of the robust interpretation: The robust interpretation requires concavity in w, which is not generally satisfied by common nonlinear predictive models, although DDR’s formulation and practical applicability do not rely on it.The assumption is used for the theoretical robust-development result rather than for DDR itself.
  • Related robust formulations: JERO and its DDR-related formulations connect robustness to prediction-loss uncertainty while imposing a cost target, with corresponding solutions under stated parameter mappings.For every λ > 0 in SP, a corresponding τ in JERO yields coinciding minimizers and decisions up to degeneracy; when µ = 0, DDR and JERO-Like solutions correspond through λ(τ).
  • Regret minimization: Regret-DDR considers worst-case oracle outcomes constrained by predictive-error and cost-alignment conditions, yielding an upper bound on true regret when those outcomes lie in the ambiguity set.The ambiguity-set size reflects the quality of the available training data.
  • Regret minimization: Under a triangular-inequality fidelity measure, DDR is the canonical approximation of cost-ambiguous regret minimization, and SPO+ is a special case with λ = 1.The paper states that the objective value of DDR is related to the regret formulation up to a dataset-dependent constant.

4. A Numerical Illustration on the Shortest Path Problem

The numerical illustration evaluates decision-driven regularization on synthetic shortest-path problems against separate and integrated learning-optimization benchmarks. Results examine parameter sensitivity, prediction and regret performance, misspecification, network size, and computational efficiency.

  • Benchmarks: Benchmarks include OLS, random forest, XGBoost, SPO+, Perturbation Gradient, and Learning to Rank.OLS, random forest, and XGBoost represent SLO models; SPO+, PG, and LTR represent ILO models.
  • Computational efficiency: The DDR model is computationally efficient, and its computational performance is insensitive to µ and λ despite solution time increasing with grid size.The reported solution time increases monotonically with grid size, while µ and λ have negligible computational impact.
  • Parameter calibration: Large λ causes performance to fall off earlier for smaller µ values.Smaller µ corresponds to a larger dependence on the true cost function.
  • SLO comparisons: DDR outperforms the tested SLO models; it achieves better regret than OLS 63% of the time, while RF and XGBoost show over-fitting and degraded decision quality.DDR remains close to OLS in prediction accuracy while improving performance.
  • ILO comparisons: Under no misspecification, DDR makes better decisions on every testing data point than the ILO benchmarks.With higher misspecification, SPO+ eventually catches up around β = 8.0, whereas PG and LTR do not.
  • Misspecification and scalability: DDR consistently outperforms OLS in regret across examined network configurations, although its advantage decreases as network size grows.DDR also continues to outperform SPO+ under larger network structures.

5. Conclusion

The paper proposes decision-driven regularization (DDR), a blended bi-objective approach that balances predictive accuracy with cost minimization in contextual optimization. It relates DDR to robust and regret-based formulations, generalizes SPO+, and attributes superior benchmark performance to its design elements.

  • DDR is a blended bi-objective framework that balances prediction accuracy and cost minimization.
  • Controlling predictive accuracy in integrated learning and optimization is important because insufficient control can lead to overfitting.
  • DDR incorporates a bias-variance trade-off to improve estimation of the cost function.
  • The paper attributes DDR’s superior numerical performance over benchmark models to its specific design elements.
  • DDR is related to robust and regret-based models and consequently generalizes SPO+.
  • DDR is compatible with general learners, although its specialization to other popular learners, including neural networks, requires further study.

E-Companion of “Decision-Driven Regularization: A Blended Model for

The E-Companion contains deferred proofs from the paper’s main text.

  • The E-Companion presents deferred proofs.
  • The proofs in this segment come from the paper’s main text.
  • This segment serves as a proof supplement to the main paper.

A.1. Proof of Claim 1 in Illustration 1

The proof characterizes when integrated and separate learning-optimization models choose route b, and connects the uncertainty-set construction to likelihood-ratio testing.

  • OLS estimates W through the least-squares solution W =(X⊤X)−1(X⊤Z).
  • SLO chooses route b if and only if 1.5x̂2 ≤ x̂1.
  • For ILO, route b is optimal when 1 ≤ w̄/w ≤2 and w ≥0.
  • ILO chooses route b if and only if x̂2 ≤ (w̄/w)x̂1 for some 1 ≤ w̄/w ≤2.
  • The uncertainty set based on likelihood differences contains weights whose likelihood ratio against w̃ is bounded by the specified threshold.
  • The Neyman-Pearson Lemma links this uncertainty set to weights that can be distinguished from w̃ under the corresponding significance level.

A.6. Proof of Theorem 1

The proof establishes concavity of the valuation function and strong duality under the stated convexity assumptions, while identifying a limiting-case issue as μ approaches 1.

  • The valuation function vμ(w) is concave in w.
  • For μ ∈[0,1], the objective is convex in y and concave in w, yielding concavity after taking the infimum over y.
  • Slater’s condition holds because w = w̃ is feasible and interior when ρ > 0, so strong duality applies.
  • The solution of the auxiliary problem coincides with the solution of the corresponding main problem.
  • At μ = 1, the regularization term no longer depends on w, so the argmin coincides with the minimizer of L(w).
  • As μ approaches 1 with λ bounded away from 0, simulations do not recover ŵ, and consistency beyond μ = 1 is not ensured.

A.7. Proof of Proposition 4 and Corollary 1

This section develops the robust reformulation of decision-driven regularization and establishes its equivalence to the original model under an appropriate parameter choice. It also relates the resulting out-of-sample guarantee to the learning model and its surrogate valuation function.

  • Robust DDR: The Robustness Decision-driven Regularization (RDDR) model is defined for μ ∈ [0,1) as a robust optimization problem.RDDR is introduced as the robustness-based formulation of decision-driven regularization.
  • Dual reformulation: Strong duality holds because ρ > 0 makes the reference solution an interior feasible point, satisfying Slater’s condition.The proof also restricts attention to α > 0.
  • Equivalence: Setting λ = β⋆ makes the RDDR solution coincide with the DDR solution.Here β⋆ is the optimal β obtained from the reformulated RDDR problem.
  • Out-of-sample interpretation: The out-of-sample guarantee bounds v_μ(ˇw) with probability tied to ˇw belonging to the uncertainty set U(ρ), while the valuation remains a surrogate for true cost.The bound’s strength is related to the learning model when L is a log-likelihood function.
  • Worst-case regret: The worst-case regret is bounded by relaxing the uncertainty conditions to two decision points: y⋆(z_n) and the optimization-derived y_n.The relaxed uncertainty set Z⋆(w) contains Z(w), so its supremum provides an upper bound.

A.9. Proof of Proposition 5

This section connects DDR to regret-based and SPO formulations while motivating decision-focused accuracy through a knapsack example. It argues that accuracy in the context region affecting decisions can matter more than globally lower prediction error.

  • Connection to SPO+: SPO+ occurs when μ = −1, so DDR reduces to an SPO+-type formulation for some λ ≥ 0.The paper presents this as a special case of DDR.
  • Connection to SPO: When ϕ = 0, η = 0, and t, ψ are sufficiently large, Regret-DDR becomes exactly SPO.Under these settings, the additional conditions in Z(w) disappear.
  • Interpretation: DDR approximates Regret-DDR in a way analogous to how SPO+ approximates SPO.The paper describes this parallel as a canonical construction within the two formulations.
  • Decision-relevant regions: A model with smaller prediction error in the decision-relevant region is expected to have greater decision power, even if its overall prediction error is larger.The illustrative broken-line confidence interval sacrifices accuracy in non-critical regions for the region affecting decisions.
  • Decision-relevant regions: In the knapsack setting, optimization emphasizes extreme z values associated with extreme x regions, where global prediction error can be largest.The cost is c(y;z) = y⊤z with feasible set Y = {y ∈ [0,1]^N, y⊤1 = 1}.

B.2. Relationship to other regularizers

This section compares decision-driven regularization with norm-based regularizers through local directions around the OLS solution. The simulations show that DDR’s decision-oriented direction differs substantially from Lasso and Ridge, with corresponding differences in cost performance.

  • Local comparison: Around the OLS solution, adding a regularizer moves weights in the gradient direction of that regularizer because the OLS loss gradient is zero.For DDR and norm-based regularization, these local directions are represented by optimal dual variables ξ.
  • Local comparison: DDR and norm-based regularizers have the same local objective form, but their directions differ according to the underlying feasibility sets.The relevant ξ solutions depend on how those feasibility sets differ.
  • Simulation evidence: Around 60° angles separate DDR’s local direction from those of Lasso and Ridge across 100 simulations.Table B.1 reports summary statistics for these local-direction angles.
  • Simulation evidence: Lasso and Ridge improve prediction accuracy over OLS but lead to significantly poorer cost-function performance.The simulations associate their differing local directions with worse eventual cost outcomes.
  • Implication: Blindly adopting regularizers for prediction accuracy can worsen performance by pulling the solution away from a direction that improves decisions.The section cautions that prediction-oriented regularization need not improve the eventual cost.
  • Robust interpretation: Under the stated assumptions, DDR and the worst-case weights from RDDR coincide for an uncertainty radius ρ⋆ and a suitable λ(τ).The relationship is established for all μ ∈ [0,1) under the conditions of the cited corollary.

D.1. A Closer Look at Over-fitting in the Tree-based SLO Benchmarks

This appendix examines overfitting in tree-based SLO benchmarks by varying tree depth while keeping OLS and DDR as references. It reports that changing hyperparameters may not eliminate the overfitting behavior.

  • Experimental setup: Table 2 compares in-sample and out-of-sample RMSE for OLS, Random Forest, XGBoost, and DDR, indicating potential overfitting in the tree-based learners.The appendix uses these models’ prediction accuracy as the basis for further analysis.
  • Experimental setup: The appendix refines Random Forest and XGBoost results by sweeping maximum tree depth across {2, 4, 6, 8, Default}.OLS and DDR remain unchanged as reference models.
  • Finding: Changing the tree-based learners’ hyperparameters might not resolve overfitting, which may instead be inherent to the function.This conclusion is drawn from the depth-sweep analysis.

D.2. Robustness of Our Model Across Various Settings

The experiments vary training size, feature dimensionality, noise, and model misspecification to assess DDR's robustness. DDR generally outperforms OLS and consistently outperforms SPO+, although its advantage over OLS narrows at N = 1000.

  • DDR versus OLS: DDR consistently attains superior regret performance to OLS across the evaluated settings.Figure D.2 summarizes the DDR–OLS comparison.
  • DDR versus OLS: DDR improves head-to-head performance across all sample sizes except N = 1000.The exception is attributed to improved OLS accuracy with larger sample sizes, which reduces the performance gap.
  • DDR versus OLS: Figure D.2 shows DDR outperforming OLS on both regret reduction and head-to-head metrics in most cases.This comparison indicates that DDR's advantage is broad but not universal against OLS.
  • DDR versus SPO+: DDR consistently outperforms SPO+ across all settings in regret reduction and head-to-head performance.The SPO+ comparison is presented in Figure D.3.
  • Extensions to tree learners: DDR can be applied with learners beyond the linear learner used in the numerical results, including a tree-based learner.The paper formulates DDR loss for decision trees and describes greedy recursive and integer-programming construction approaches.
Loading 2608.15124v1…