Source-linked AI summary
Stagewise Safe Bayesian Optimization with Gaussian Processes
Yanan Sui, Vincent Zhuang, Joel W. Burdick, Yisong Yue
TL;DR
Safe Bayesian optimization must optimize unknown utility while keeping every sequential decision within unknown safety constraints. STAGEOPT separates safe-region expansion from utility optimization, then provides theoretical guarantees and strong synthetic and clinical results, including better therapies than experienced physicians. Its scope is limited by the static-environment assumption and dependence on a well-specified Gaussian-process prior and kernel.
Problem
Safe Bayesian optimization seeks to optimize unknown utility under unknown safety constraints while ensuring every chosen action remains safe.
Method
STAGEOPT models utility and safety functions with Gaussian processes and separates safe-region expansion from subsequent utility optimization.
Results
STAGEOPT achieves theoretical safety and optimality guarantees, outperforms state-of-the-art safe optimization in experiments, and produces therapies outperforming experienced physicians.
Takeaways & Limitations
STAGEOPT provides an efficient tool for safety-critical online optimization, including preference-based clinical neural stimulation therapy.
Takeaways & Limitations
The framework assumes a static environment and requires a well-specified Gaussian-process prior and kernel for effectiveness.
Abstract
from arXiv · showhide
Enforcing safety is a key aspect of many problems pertaining to sequential decision making under uncertainty, which require the decisions made at every step to be both informative of the optimal decision and also safe. For example, we value both efficacy and comfort in medical therapy, and efficiency and safety in robotic control. We consider this problem of optimizing an unknown utility function with absolute feedback or preference feedback subject to unknown safety constraints. We develop an efficient safe Bayesian optimization algorithm, StageOpt, that separates safe region expansion and utility function maximization into two distinct stages. Compared to existing approaches which interleave between expansion and optimization, we show that StageOpt is more efficient and naturally applicable to a broader class of problems. We provide theoretical guarantees for both the satisfaction of safety constraints as well as convergence to the optimal utility value. We evaluate StageOpt on both a variety of synthetic experiments, as well as in clinical practice. We demonstrate that StageOpt is more effective than existing safe optimization approaches, and is able to safely and effectively optimize spinal cord stimulation therapy in our clinical experiments.
1. Introduction
Safe Bayesian optimization must choose actions that are both informative and known to satisfy safety constraints. STAGEOPT addresses this by first expanding the safe region and then optimizing utility, with theoretical guarantees and clinical evidence of improved safe exploration and stimulation strategies.
- Motivation: Safety-constrained applications require actions to remain safe while the algorithm explores an initially small known-safe region.The safe region is expanded iteratively from an initial seed.
- Motivation: Clinical therapy selection motivates sequential exploration because treatment efficacy and safety are initially unknown.The approach starts from well-studied therapies and explores similar therapies incrementally.
- STAGEOPT: STAGEOPT first expands the safe region and then optimizes utility, rather than interleaving the two activities.This stagewise design supports settings where safety and utility differ in scale or measurement.
- STAGEOPT: STAGEOPT models safety and utility with separate Gaussian processes and uses confidence bounds to assess unexplored decisions.The analysis assumes bounded RKHS norms and Lipschitz-continuous safety functions.
- Results: In clinical experiments, STAGEOPT explored a larger safe region and found a better stimulation strategy than expert physicians.The result is reported for spinal cord stimulation in patients with spinal cord injuries.
2. Related Work
Prior work established Bayesian optimization and confidence-bound methods for unknown utilities, but safety constraints remained insufficiently addressed. Existing constrained approaches either lacked theoretical guarantees or were not designed to preserve safety during optimization.
- Bayesian optimization: Bayesian optimization uses regularity assumptions and Gaussian processes to efficiently optimize unknown functions.Gaussian processes provide flexible nonparametric models that encode prior knowledge and uncertainty.
- Sequential decision problems: Multi-armed bandit and best-arm identification formulations formalize exploration–exploitation tradeoffs for unknown rewards.Best-arm identification seeks the highest-reward decision with minimal trials.
- Safe optimization: GP-UCB and active level-set sampling provide kernelized confidence-bound methods, but they do not target safety-constrained optimization.The cited discussion states that such methods often violate safety constraints in practice.
- Safe optimization: Constrained Expected Improvement and information-based search address unknown constraints without the theoretical guarantees emphasized for STAGEOPT.These approaches are presented as algorithms without theoretical guarantees.
- Safe exploration: Safe exploration methods from control and reinforcement learning typically restrict policies to recoverable, ergodic behavior in MDPs.These methods differ from the bandit and optimization setting considered here.
3. Problem Statement
The paper formulates safe Bayesian optimization as maximizing an unknown utility while every sampled point satisfies unknown safety thresholds. Because a zero-mean prior alone cannot identify safe points, optimization begins from a known safe seed and targets the safely reachable optimum.
- Optimization problem: The objective is to maximize f(x) subject to gi(x) ≥ hi for every safety function and sampled decision.The utility and safety functions are unknown and observed through noisy evaluations.
- Regularity assumptions: The utility and safety functions are modeled as Gaussian processes with bounded norms in their associated RKHSs.The GP formulation uses zero-mean priors without loss of generality.
- Feedback models: Under noisy feedback, observations take the form yt = f(xt) + nt with independent Gaussian noise.The posterior is characterized by updated mean, covariance, and variance functions.
- Feedback models: Preference feedback represents pairwise comparisons as Bernoulli outcomes governed by a link function of the two utility values.The paper gives the logit function as a common example.
- Preference feedback: The paper also considers restricted dueling feedback in which each round compares the current point with the previous point.This setting is used because no existing algorithms cover the full safe Bayesian dueling-bandit problem.
- Safe reachability: The optimization target is the safe region reachable from an initial seed within a finite horizon, rather than necessarily the globally optimal point.A disconnected safe region around the global optimum may be unreachable from the seed.
4. Algorithm
STAGEOPT separates safe optimization into a safe-region expansion stage and a utility-optimization stage. It uses GP confidence intervals to select uncertain expanders, then applies Bayesian optimization within the expanded safe region, supporting preference-based variants and outperforming SAFEOPT on optimization choices.
- Overview: STAGEOPT runs safe expansion for T0 steps and utility optimization for T1 = T − T0 steps.The two-stage schedule explicitly separates exploration of safety from optimization of utility.
- Confidence intervals: STAGEOPT models utility and safety functions with Gaussian processes and uses confidence intervals to explore while preserving safety.The intervals are maintained sequentially to support both safety and expansion progress.
- Stage One: Safe region expansion: In Stage One, the algorithm computes likely expander points within the current safe region and selects the expander with the highest predictive uncertainty.The expansion stage stops when confidence widths meet the tolerance criterion.
- Stage Two: Utility optimization: In Stage Two, STAGEOPT applies GP-UCB to optimize utility within the expanded safe region.The paper also describes a preference-based version that replaces GP-UCB with a kernelized dueling-bandit method.
- Stage Two: Utility optimization: The safe region can still expand during utility optimization, although this is typically unlikely.This is a property of the staged procedure rather than a strict prohibition on later expansion.
- Comparison with SAFEOPT: Unlike SAFEOPT, STAGEOPT uses a UCB criterion to select better optimization points after the initial safe expansion behavior coincides.Both methods construct confidence intervals and define the safe region similarly.
5. Theoretical Results
STAGEOPT’s two-stage analysis provides finite-time guarantees for safe-region expansion and utility optimization. With high probability, it preserves safety, expands the initial safe region to an ε-reachable set, and approaches the optimal utility value within that region.
- Stagewise analysis: STAGEOPT analyzes safe-region expansion and utility optimization as two distinct stages.The expansion stage searches for the total safe region, while the optimization stage operates within the expanded region.
- Overall guarantees: STAGEOPT’s theoretical results are finite-time bounds for convergence to the reachable safe region and toward the optimal utility value within it.Unlike interleaved approaches, the analysis does not require similarity or comparability between safety and utility functions.
- Safety guarantee: With probability at least 1 − δ, the expansion stage preserves every safety constraint throughout its evaluations.The guarantee applies under the stated GP, RKHS-norm, Lipschitz-continuity, seed-set, and sub-Gaussian-noise assumptions.
- Safe-region expansion: After at most t* iterations, the expansion stage reaches the ε-reachable safe region from the initial safe set S0.The required horizon depends on the reachable-region size, accuracy parameters, confidence level, function complexity, and GP information gain γ_t.
- Utility optimization: The optimization stage guarantees an ε-optimal decision within the expanded safe region after its prescribed finite horizon.The theorem states f(x̂*) ≥ f(x*) − ζ for the utility value attained within the expansion-stage region.
6. Experimental Results
STAGEOPT was evaluated on synthetic settings and a clinical spinal cord stimulation task, with experiments examining safe-region expansion and utility optimization under safety constraints. It expanded safe regions at least as effectively as SAFEOPT and safely surpassed the physician’s best clinical choice after about 400 iterations.
- Synthetic Data: STAGEOPT and SAFEOPT were compared across synthetic settings involving different safety constraints, feedback types, and utility formulations.Experiments used single and multiple safety functions, Gaussian and dueling feedback, and 30 sampled function combinations from 10 safe seeds.
- Synthetic Data: STAGEOPT grew the safe region at least as fast as SAFEOPT while reaching an optimal sample point more quickly.Both reward and safe-region growth were at least as good as SAFEOPT in the reported synthetic scenarios.
- Clinical Experiments: The clinical study optimized stimulation patterns for gripping rehabilitation using preference-based STAGEOPT in a tetraplegic patient.The experiment involved 564 therapeutic trials over 10 weeks with a 32-channel electrode.
- Clinical Experiments: STAGEOPT did not sample unsafe stimulating patterns in the spinal cord therapy experiment.Physicians judged that the total safe region had been reached between 300 and 400 iterations.
- Clinical Experiments: After around 400 iterations, STAGEOPT’s stimulation utility exceeded the physician’s best choice.The figure reports utilities within the safe region and includes a GP fit of STAGEOPT’s observed utilities.
7. Conclusion & Discussion
The paper presents STAGEOPT as a safe Bayesian optimization framework for optimizing utility under unknown safety constraints. It reports theoretical safety and optimality guarantees, strong synthetic performance, and clinical therapies outperforming experienced physicians, while identifying static environments and GP specification as limitations.
- Conclusion: STAGEOPT targets online optimization problems in which every sampled decision must satisfy unknown safety constraints.The framework is intended for applications including medical therapies, safe recommender systems, and safe robotic control.
- Conclusion: STAGEOPT separates safe-region expansion from utility maximization and provides high-probability safety guarantees with theoretical bounds for reaching an ϵ-safe region and ζ-optimal utility.The framework supports non-comparable safety constraints and Gaussian-process safety and utility models.
- Conclusion: Across synthetic experiments, STAGEOPT achieved its theoretical safety and optimality guarantees, ranked among the best for safe expansion, and outperformed state-of-the-art utility maximization.The evaluation covered single and multiple safety functions, real-valued utility, and dueling-feedback utility.
- Conclusion: In clinical experiments using dueling-feedback utility, therapies proposed by STAGEOPT outperformed those suggested by experienced physicians.The application involved gripping rehabilitation therapy for tetraplegic patients.
- Limitations: The study assumes a static environment, so the clinical setting treats patients’ condition and responses to stimulation as not improving over time.The authors identify incorporating dynamics as future work leading toward multi-criteria safe reinforcement learning.
- Limitations: Gaussian-process approaches require a well-specified prior and kernel to be effective, while uniform priors can converge slowly.The paper suggests learning kernels or modeling low-dimensional structure as alternatives.
A. Proofs
The proofs establish high-probability confidence coverage, safe-region expansion and containment, safety preservation, and utility optimization guarantees for STAGEOPT under stated GP and noise assumptions.
- Confidence guarantees: With probability at least 1 −δ, the confidence intervals contain f(x) for all x ∈D during the expansion stage.The result uses βt and applies across all expansion iterations.
- Safe-region expansion: If the recursively reachable region beyond the current safe set is nonempty, another expansion step remains possible.This property underpins the argument that expansion cannot stop while reachable safe points remain outside the current set.
- Safe-region guarantees: The safe sets remain contained in the recursively reachable safe region, and their expansion preserves the safety threshold under Lipschitz continuity.The proof combines safe-region containment with an induction argument showing f(x) ≥h throughout the expanding sets.
- Optimization guarantee: During optimization, after Y iterations the average regret satisfies ¯rY ≤ζ, yielding a sampled point with utility at least f(x∗)−ζ.The guarantee is stated for optimization within the safe region produced by the expansion stage.
B. STAGEOPT with dueling feedback
The dueling-feedback version of STAGEOPT uses preference feedback for utility while retaining real-valued safety feedback, and applies the same staged expansion and optimization structure.
- Feedback model: In the dueling-feedback setting, utility observations are Bernoulli outcomes generated by a link function comparing the current and previous sample points.Safety functions continue to receive real-valued observations to preserve the safety guarantees.
- Algorithm inputs: The algorithm initializes GP priors, confidence quantities, safety thresholds, a safe seed set, and separate expansion and optimization accuracies.The pseudocode specifies safety feedback for each gi and preference feedback for f.
- Two-stage procedure: STAGEOPT runs an expansion loop followed by an optimization loop, updating utility and safety observations and recomputing confidence quantities over the current safe set.The two loops are shown in the dueling-feedback pseudocode.
- Relation to prior methods: The formulation is similar to KERNELSELFPARRING, but selects one point per iteration; with multiple samples per iteration, KERNELSELFPARRING can replace GP-UCB.This identifies the relationship between the dueling-feedback implementation and the comparison-based baseline.