PaperlayerView exact revision on arXiv

Source-linked AI summary

Wasserstein Distributionally Robust Stochastic Control: A Data-Driven Approach

Insoon Yang

arXiv:1812.09808v4math.OCeess.SY

TL;DR

The paper studies stochastic control when uncertain-variable distributions are unavailable or inaccurately estimated. It uses Wasserstein ambiguity sets around empirical distributions within a dynamic-game and dynamic-programming framework, obtaining tractable algorithms, multi-stage out-of-sample guarantees, and explicit linear-quadratic policies.

  • Problem

    Stochastic control typically assumes known uncertainty distributions, but empirical distributions can be inaccurate because of limited, corrupted, or mismatched data.

  • Method

    The paper formulates distributionally robust control as a zero-sum dynamic game and uses Wasserstein ambiguity sets, Bellman operators, and Kantorovich duality.

  • Results

    The framework yields tractable value and policy iteration, extends single-stage out-of-sample guarantees to multiple stages without confidence degradation, and provides explicit linear-quadratic policies.

  • Takeaways & Limitations

    Dynamic programming and Kantorovich duality provide a data-driven route to computationally tractable and statistically robust stochastic control.

  • Takeaways & Limitations

    The a priori radius bound can be conservative, producing a smaller ambiguity radius than necessary; bootstrapping and cross-validation may reduce this conservativeness.

Abstract

from arXiv · show

Standard stochastic control methods assume that the probability distribution of uncertain variables is available. Unfortunately, in practice, obtaining accurate distribution information is a challenging task. To resolve this issue, we investigate the problem of designing a control policy that is robust against errors in the empirical distribution obtained from data. This problem can be formulated as a two-player zero-sum dynamic game problem, where the action space of the adversarial player is a Wasserstein ball centered at the empirical distribution. We propose computationally tractable value and policy iteration algorithms with explicit estimates of the number of iterations required for constructing an $ε$-optimal policy. We show that the contraction property of associated Bellman operators extends a single-stage out-of-sample performance guarantee, obtained using a measure concentration inequality, to the corresponding multi-stage guarantee without any degradation in the confidence level. In addition, we characterize an explicit form of the optimal distributionally robust control policy and the worst-case distribution policy for linear-quadratic problems with Wasserstein penalty. Our study indicates that dynamic programming and Kantorovich duality play a critical role in solving and analyzing the Wasserstein distributionally robust stochastic control problems.

1 Introduction

The paper addresses stochastic control when uncertain-variable distributions are inaccurately known by using Wasserstein ambiguity sets around empirical data. It develops dynamic-programming methods with tractability, out-of-sample guarantees, and explicit linear-quadratic solutions.

  • Motivation: The paper replaces fully known disturbance distributions with Wasserstein ambiguity sets centered on empirical distributions.This directly incorporates data samples while modeling distributional uncertainty.
  • Contributions: It develops computationally tractable value and policy iteration algorithms with explicit iteration estimates for constructing ε-optimal policies.Kantorovich duality reformulates the infinite-dimensional Bellman optimization into a tractable form.
  • Contributions: The framework provides an out-of-sample performance guarantee by extending a single-stage concentration result to multiple stages without confidence-level degradation.The extension relies on the contraction property of the associated Bellman operators.
  • Contributions: For linear-quadratic problems with Wasserstein penalty, the paper derives explicit optimal control and worst-case distribution policies.The paper also characterizes the solution through a Riccati-type equation.
  • Problem formulation: The study frames distributionally robust stochastic control as a dynamic game and uses dynamic programming and Kantorovich duality to solve and analyze it.The adversarial player selects distributions from the ambiguity set.

2 Distributionally Robust Control of Stochastic Systems

The paper formulates data-driven stochastic control as a two-player zero-sum game in which an adversary selects disturbance distributions from an ambiguity set. Wasserstein balls around empirical distributions support robust policies, dynamic-programming analysis, and tractable reformulations.

  • Ambiguity in Stochastic Systems: Empirical-distribution control can degrade under corrupted, limited, or distributionally mismatched data, motivating policies robust to errors in the empirical distribution.The robust policy minimizes worst-case cost over distributions in a prescribed ambiguity set.
  • Distributionally Robust Policy: The controller minimizes discounted total cost while an adversary selects disturbance distributions from the ambiguity set to maximize that cost.This creates a two-player zero-sum dynamic game with the ambiguity set as the adversary’s action space.
  • Existence and assumptions: Under the stated assumptions, an optimal policy exists and is deterministic and stationary, with an optimal value function in the lower-semicontinuous function space.The assumptions include compact admissible action sets and continuity conditions.
  • Wasserstein Ambiguity Set: The Wasserstein ambiguity set is a statistical ball centered at the empirical distribution, with distance defined through minimum-cost mass transport.Its radius controls the modeled distributional uncertainty.
  • Computational formulation: Kantorovich duality converts the Wasserstein reformulation into a finite-dimensional optimization problem, enabling computationally tractable value and policy iteration.This avoids solving the original optimization directly over probability measures.

3 Dynamic Programming Solution and Analysis

The paper develops a dynamic-programming framework for Wasserstein distributionally robust control, establishing contraction-based optimality and tractable iteration schemes. Kantorovich duality converts the Bellman problem into a finite-dimensional, generally semi-infinite program, while policy-iteration and worst-case-distribution results provide computational and structural guarantees.

  • 3.1 Bellman’s Principle of Optimality: The Bellman operator is a τ-contraction and monotone under Assumption 1, enabling a unique optimal value function and deterministic stationary policy.The contraction factor is τ = αβ ∈ (0, 1), and the optimal value function is the unique Bellman-equation solution.
  • 3.2 Value Iteration: Value iteration applies v_{k+1} := Tv_k and converges to v⋆ by contraction, with explicit iteration requirements for constructing an ε-optimal policy.The tractable reformulation is used at each state and iteration to avoid the original infinite-dimensional minimax problem.
  • 3.2 Value Iteration: Kantorovich duality reformulates the Wasserstein Bellman optimization without a duality gap, yielding finite-dimensional variables but semi-infinite support constraints.The reformulation uses u, λ, and ℓ variables, while constraints must hold for every disturbance in the support.
  • 3.3 Policy Iteration: Policy iteration remains convergent when policy evaluation is approximate, and its resulting policy is ε-optimal under the stated iteration conditions.The modified method evaluates π_k using an approximate value ˜v_k rather than computing the exact fixed point of T^{π_k}.
  • 3.4 The Worst-Case Distribution Policy: The worst-case distribution policy is deterministic and stationary, so Player II can use the same state-dependent worst-case distribution at every stage.Its explicit structure is obtained by applying a Wasserstein maximization result to the adversary’s problem.

4 Out-of-Sample Performance Guarantee

The paper establishes probabilistic out-of-sample guarantees for distributionally robust policies by combining Wasserstein measure concentration with Bellman-operator contraction. The same ambiguity-set radius preserves the confidence level when extending single-stage guarantees to multistage performance.

  • Motivation: The approach addresses the optimizer’s curse by seeking guarantees for policies evaluated on independent test samples rather than only the training dataset.The empirical-distribution policy may perform poorly out of sample even when training and testing data share the same distribution.
  • Radius selection: Measure concentration determines a Wasserstein-ball radius that contains the true distribution with probability at least 1−β.The concentration bound depends on the light-tail assumption and the sample size through the selected radius.
  • Multistage extension: Bellman-operator contraction extends the single-stage out-of-sample guarantee to multiple stages without additional requirements on the Wasserstein radius.The same radius can achieve the same confidence level 1−β for multistage performance as for the single-stage guarantee.
  • Out-of-sample guarantee: The optimal distributionally robust policy receives a probabilistic performance guarantee for expected costs evaluated under unseen samples from the true distribution.The guarantee is measured relative to the optimal value function under the true distribution and applies with probability at least 1−β when the chosen radius contains the true distribution.
  • Caveat: The concentration-based radius can be conservative, and bootstrapping or cross-validation may reduce that conservativeness.The paper notes that conservative constants can produce a smaller radius than necessary.

5 Wasserstein Penalty Problem

The Wasserstein penalty formulation replaces an explicit ambiguity set with a distribution-perturbation penalty and remains tractable through dynamic programming and duality. For linear-quadratic systems, it yields explicit policies and worst-case distributions whose robustness is controlled by the penalty parameter.

  • Penalty formulation: The Wasserstein penalty problem has a unique fixed-point value function and admits value iteration plus an optimal deterministic stationary policy.The Bellman operator is a contraction, enabling value iteration under the Banach fixed-point theorem.
  • Modeling scope: Unlike standard LQG, the Wasserstein-penalty formulation does not require Gaussian disturbance distributions.Its stated motivation is to obtain useful control policies when the true disturbance distribution deviates from a Gaussian distribution.
  • Linear-quadratic problem: For linear-quadratic problems, the value function is quadratic and dynamic programming produces an explicit optimal policy under a Riccati-type equation.The stated solution has the form v′(x)=x⊤Px+z when the required positive semidefinite matrix solution exists.
  • Policy and worst-case distribution: The optimal linear-quadratic policy is linear in the state, while each worst-case distribution support element is affine in the state.The control gain is independent of the disturbance covariance matrix, whereas the worst-case support shifts with the system state.
  • Robustness control: Increasing λ reduces permissible deviation from the empirical distribution, equivalently shrinking the Wasserstein ambiguity radius.As λ tends to infinity, the robust policy converges pointwise to the standard linear-quadratic optimal control policy.

6 Numerical Experiments

The numerical experiments evaluate Wasserstein distributionally robust control on investment-consumption and power-system frequency-control problems. Results show reliability depends on the ambiguity radius and sample size, while appropriately calibrated robust policies improve out-of-sample and frequency-control performance.

  • 6.1.1 Out-of-sample performance guarantee: Reliability increases with both the Wasserstein ball radius θ and the number N of samples, consistently across single-stage and multi-stage settings.The same radius θ achieves the same reliability level in both settings, as predicted by the theoretical guarantee.
  • 6.1.1 Out-of-sample performance guarantee: 8% lower out-of-sample cost than the SAA policy is achieved by the proposed DR policy when N = 10.The performance gap decreases as the number of samples increases, while the DR policy remains effective on independently generated test data.
  • 6.2 Power System Frequency Control Problem: The power-system experiment uses an IEEE 39-bus New England test case to evaluate Wasserstein-penalty LQ control under renewable-energy disturbances.The model combines swing dynamics with a linearized DC power-flow approximation and uses a discretized state-space representation.
  • 6.2 Power System Frequency Control Problem: Mean frequency deviation falls below 1% after 16.7 seconds under the proposed DR policy, compared with 41.8 seconds for standard LQG control.The DR method accounts for possible nonzero-mean disturbances, whereas standard LQG assumes zero-mean disturbances.
  • 6.2.1 Worst-case distribution policy: Reliability decreases as the penalty parameter λ increases and tends to increase with the number of samples used to design the policy.Because larger λ corresponds to a smaller Wasserstein radius, the relationship can guide penalty selection for a desired out-of-sample reliability.

7 Conclusions

The paper develops a data-driven Wasserstein distributionally robust stochastic-control framework with computational, statistical, and linear-quadratic guarantees. Its analysis uses Bellman-operator contraction and Kantorovich duality to connect tractable dynamic programming with out-of-sample performance.

  • 7 Conclusions: The framework provides computational tractability with error bounds, out-of-sample performance guarantees, and an explicit solution for linear-quadratic problems.The paper identifies Kantorovich duality as critical to its dynamic-programming solution and analysis.
  • 7 Conclusions: Bellman-operator contraction extends a single-stage concentration-based guarantee to the multi-stage setting without degrading the confidence level.This is the paper’s stated insight concerning the propagation of out-of-sample guarantees.

A Proof of Lemma 1

The proof establishes equivalence between two ambiguity-set representations using Kantorovich duality. It does so by proving both set inclusions.

  • A Proof of Lemma 1: Kantorovich duality is used to rewrite the Wasserstein distance through functions satisfying the constraint ϕ(w) + ψ(w′) ≤ d(w,w′)^p.This dual representation supports the subsequent ambiguity-set equivalence argument.
  • A Proof of Lemma 1: The proof shows ˆD ⊆ D by selecting an arbitrary µ in ˆD and applying the defining dual inequality.Membership in the first set implies membership in the second.

B Linear-Quadratic Problems

For Wasserstein-penalty linear-quadratic control, the Bellman equation admits a quadratic value function and a linear optimal policy under a sufficiently large penalty parameter. The same formulation also identifies worst-case distributions and handles nonzero disturbance means through normalization.

  • B Linear-Quadratic Problems: For λ ≥ ¯λ, strict concavity of the inner maximization and positive definiteness of the control Hessian yield a unique worst-case disturbance and a unique linear minimizer u⋆ = Kx.The condition λI − αΞ⊤PΞ ≻ 0 ensures strict concavity in the disturbance variable, while R ≻ 0 supports uniqueness in the control variable.
  • B Linear-Quadratic Problems: The quadratic function v′(x) = x⊤Px + z solves the Bellman equation, and the optimal policy is π′(x) = Kx.The matrix P and scalar z satisfy the corresponding Riccati and constant-term equations.
  • B Linear-Quadratic Problems: The worst-case distribution policy is obtained by substituting the empirical sample and optimal control into the inner maximization characterization.The resulting γ′(x) is identified as one of the worst-case distributions.
  • B Linear-Quadratic Problems: Nonzero-mean disturbances are converted to the zero-mean case by centering the samples, expanding the state, and modifying the quadratic cost matrix.The resulting optimal policy is π′(x) := K′((x − ¯x)⊤, 1)⊤.
Loading 1812.09808v4…