Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods

• The paper asks what common mechanism drives vanishing gradients in high-dimensional BO and how much depends on the surrogate model.Source: §1 · p. 2
• Pulling back the Fisher metric through the surrogate map measures how local input changes alter predictive distributions.Source: §1 · p. 2
• The analysis separates acquisition-side sensitivity from surrogate-induced geometry to explain when input-space movement changes acquisition values.Source: §1 · p. 2
• FITR replaces kernel-lengthscale scaling with local pullback-Fisher weights and applies to differentiable probabilistic surrogates without explicit kernel lengthscales.Source: §1 · p. 2
• FITR improves on TuRBO-REI on several benchmarks and remains competitive with strong baselines on the remainder.Source: §1 · p. 2

• BO seeks to maximize an expensive black-box objective over a feasible input set using sequential observations.Source: §2 · p. 2
• At each iteration, BO fits a probabilistic surrogate and maximizes an acquisition function to select the next candidate, then evaluates and adds it to the data.Source: §2 · p. 2
• A surrogate models the predictive distribution at each input, with its parameters determined by the model and observed data.Source: §2 · p. 2
• The surrogate can be viewed as a smooth map from inputs to a statistical manifold of predictive distributions, and the Fisher–Rao metric describes local distribution sensitivity.Source: §2 · pp. 2–3
• An acquisition function is expected utility under the predictive posterior and depends on the input through predictive parameters, with reparameterization covering Monte Carlo forms.Source: §2 · p. 3

• Other information-geometric applications include natural-gradient learning, Fisher-based sharpness-aware minimization, and learned manifolds for robotic motion.Source: §3 · p. 3
• Earlier geometry-aware BO uses geodesic distances for intrinsically non-Euclidean spaces, whereas this paper studies geometry induced by the surrogate posterior map.Source: §3 · p. 3
• GIT-BO identifies a global active subspace using predictive-mean gradients, unlike this paper’s local trust-region geometry based on predictive distributions.Source: §3 · p. 4
• Related pullback-Fisher work uses probabilistic-model geometries for latent-space interpolation, sampling, and decoder representations.Source: §3 · p. 4
• Prior high-dimensional BO work identifies vanishing gradients and proposes dimension-scaled lengthscales and RAASP as remedies.Source: §3 · p. 4
• Existing trust-region methods such as TuRBO use local axis-aligned regions, but their constructions rely on GP surrogates with explicit lengthscales.Source: §3 · p. 4

4 Manifold View of Bayesian Optimization

• The pullback Fisher tensor measures how sensitively the predictive distribution changes under local input perturbations.Source: §4 · p. 4
• When input dimension exceeds predictive-parameter dimension, the pullback tensor is degenerate, with null directions leaving predictive parameters unchanged.Source: §4 · p. 5

4.1 Acquisition Gradient Bound via Pullback Fisher Tensor

• The acquisition-gradient bound separates surrogate geometry, through the pullback trace, from acquisition-side sensitivity, which is shared across surrogates using the same Gaussian predictive family.Source: §4.1 · p. 5
• Where the posterior reverts to its prior and the pullback trace collapses, bounded acquisition sensitivity forces the acquisition gradient to become small.Source: §4.1 · pp. 5–6

4.2 Unified FIM-based Framework for HDBO analysis

• For isotropic kernels, slower-than-√Dx lengthscale growth makes the posterior nearly constant away from data, collapsing the trace; the same argument helps explain RAASP around observations.Source: §4.2 · p. 6
• FITR is motivated by pullback-metric ellipses that reflect local predictive change and can be used with surrogates whose predictive derivatives are available.Source: §4.2 · p. 6
• Ellipses formed from the pullback tensor illustrate induced geometries for SE-GP, IBNN-kernel GP, and BNN surrogates.Source: Figure 3 · p. 6

5 Fisher-Information Trust Region (FITR)

• TuRBO’s existing lengthscale-based trust regions rely on GP kernels with explicit lengthscales, such as SE or Matérn kernels.FITR instead uses local pullback-Fisher geometry to set its weights.Source: §5 · p. 6
• FITR uses the pullback Fisher tensor to define local trust-region geometry for differentiable probabilistic surrogates satisfying Assumption 4.1.The tensor measures how sensitively the predictive distribution changes under local input perturbations; FITR replaces ARD lengthscale weights with regularized pullback-Fisher sensitivities.Source: §5 · pp. 6–7
• The full local ellipsoid requires tensor inversion or whitening, and scalar predictive families have pullback rank at most two, leaving rank deficiency when Dx > 2.Without regularization, near-null directions can produce arbitrarily large radii; boundary-clipped whitened regions also require less efficient polytope sampling.Source: §5 · pp. 6–7
• The practical method estimates diagonal tensor entries by Monte Carlo, then regularizes them to prevent unbounded stretching in near-null directions.The cited estimator is unbiased, and diagonal regularization is applied after estimation.Source: §5 · p. 7
• Diagonal FITR uses a location-dependent axis-aligned box, avoiding clipped-polytope sampling while remaining compatible with standard BO toolchains.The diagonal approximation is also described as more practical in higher dimensions than the full whitening-based variant.Source: §5 · p. 7
• FITR changes only local weight construction; it retains TuRBO’s volume normalization and outer control logic, while FullFITR is treated as an ablation.TuRBO’s trust-region scaling rule remains unchanged.Source: §5 · p. 7

6 Experiments

• The evaluation covers nine continuous benchmarks with an SE-kernel GP, plus an IBNN-kernel transferability check.The SE-kernel comparisons include TuRBO, DSP, and BOUNCE; the IBNN comparison is limited to a non-trust-region baseline.Source: §6 · pp. 8–9
• FITR improves on TuRBO-REI on several SE-kernel benchmarks and remains competitive with DSP and BOUNCE on the remainder.Figure 4 compares BO performance, with lower values better; FITR-REI and TuRBO-REI share the GP pipeline and trust-region center selection but use different region shapes.Source: §6 · p. 8, Figure 4 · p. 8
• FITR’s anisotropy remains bounded across the benchmark suite, while its trust-region length κt stays qualitatively similar to TuRBO’s.The authors suggest the Figure 4 performance differences arise from trust-region shape rather than a different restart or length-adaptation schedule.Source: §6 · p. 9
• On the evaluated subset, FullFITR did not outperform diagonal FITR, so the authors use diagonal FITR as the practical default.Rapid local geometry variation, boundary clipping, and the full tensor’s higher numerical cost may contribute.Source: §6 · p. 9
• The same FITR pipeline with a depth-3 IBNN kernel improves on the non-trust-region baseline on a subset of tasks and is comparable on the remainder.This is a transferability check, not a full non-lengthscale benchmark comparison; gains for anisotropic kernels in trust-region methods are task-dependent.Source: Figure 5 · p. 8, §6 · p. 9

7 Limitations

• Proposition 4.2 bounds the acquisition gradient during inner optimization, not regret or global convergence of the outer BO loop.The bound therefore does not establish an outer-loop guarantee.Source: §7 · p. 9
• In experiments, a KL-divergence-constrained variant solved with an Augmented Lagrangian Method became highly exploitative and less exploratory, degrading performance.The authors retain TuRBO’s outer trust-region scaling rule to preserve exploration behavior.Source: §7 · p. 9
• For non-isotropic kernels such as IBNNs, FITR’s performance gains are task-dependent, and conditions for measurable search advantages remain future work.The vanishing-gradient account applies less sharply because the pullback trace need not collapse with Dx as it does for many lengthscale-dominated GPs.Source: §7 · p. 9

8 Conclusion

• The pullback-gradient factorization explains vanishing acquisition gradients when the surrogate-induced trace collapses, including observations behind dimension-scaled lengthscales and local initialization near data.The paper proposes FITR, which replaces lengthscale-based scaling with local pullback-Fisher weights.Source: §8 · p. 9

A Gaussian Process

• A Gaussian process places a prior on the unknown objective and updates beliefs from noisy observations.For scalar outputs, the posterior predictive distribution is Gaussian.Source: A Gaussian Process · p. 13

B Common Acquisition Functions: Analytic and MC Forms

• Table 1 gives closed-form acquisition functions for scalar Gaussian predictions, with EI and PI using the incumbent and UCB using an exploration weight.The appendix also describes reparameterized acquisition forms for sequential and batch settings.Source: Table 1 · p. 13, B Common Acquisition Functions:… · p. 13

B.1 Trust-Region Bayesian Optimization (TuRBO)

• TuRBO centers an axis-aligned trust region at the incumbent, adapts its size using learned lengthscales, and restarts after collapse, optionally selecting a new center with REI.Successes expand the region and consecutive failures contract it.Source: §B.1 · p. 14

C Implementation Details

• SE-kernel experiments use dimensionally scaled LogNormal priors, while IBNN experiments use the BoTorch default depth of 3.DSP optimizes the acquisition over the full input domain.Source: C Implementation Details · p. 15
• FITR replaces TuRBO-REI's ARD lengthscale weights with regularized diagonal pullback-Fisher weights, while retaining its trust-region scaling and REI center selection.FullFITR tests a full-tensor variant only on a subset of problems because of its additional computational cost.Source: C Implementation Details · p. 15
• Experiments use 30 randomly sampled observations, sequential evaluations, and fixed acquisition-optimization settings rather than benchmark-specific tuning.The SE-kernel and IBNN experiments use 11 and 5 independently seeded trials, respectively.Source: C Implementation Details · pp. 15–16

C.1 Trust-region diagnostics

• FITR and TuRBO-REI have qualitatively similar trust-region length traces, so their Figure 4 performance differences are not explained by different restart or length-adaptation schedules.Figure 6 also reports candidate-generation time, Tikhonov regularization, and FITR anisotropy over 11 seeded runs.Source: §C.1 · p. 16, Figure 6 · p. 17

D.1 Interchange Conditions for Gaussian Reparameterizations

• The interchange argument assumes the reparameterized integrand is locally Lipschitz and almost-everywhere differentiable, and the utility is globally Lipschitz in the sampled output.These conditions cover the scalar Gaussian cases used later for MC-EI and the smoothed MC-PI surrogate.Source: §D.1 · pp. 17–18
• The proof bounds the parameter-gradient norm using the utility's Lipschitz constant and an affine-in-noise derivative, whose squared expectation is finite for Gaussian noise.Positive-definite predictive covariance makes the Cholesky factor smooth in the stated regime.Source: §D.1 · p. 18
• If predictive variance degenerates, derivatives of its square root can become singular; the discussion is limited to the scalar case for simplicity.A strictly positive noise level rules out this singularity for GP regression.Source: §D.1 · p. 18

D.2 Proof of Proposition 4.2 (Acquisition Gradient Bound)

• The gradient bound combines the posterior-map Jacobian, the inverse Fisher metric, acquisition sensitivity, and Jensen’s inequality.The proof inserts the Fisher metric into a chain-rule decomposition and applies Cauchy–Schwarz before Jensen’s inequality.Source: §D.2 · pp. 18–19

D.3 Gaussian Predictive Distribution

• For a univariate Gaussian predictive distribution, the pullback tensor has mean- and variance-gradient contributions that encode local exploitation and exploration geometry, respectively.The mean contribution grows where the posterior mean changes rapidly relative to uncertainty; the variance contribution grows where posterior variance changes rapidly.Source: §D.3 · p. 19

D.4 Acquisition Sensitivity under a Scalar Gaussian Predictive

• MC-UCB’s mean dependence outside the reparameterized sample prevents its variance sensitivity from satisfying the same bound condition as its mean sensitivity.The appendix also extends the sensitivity analysis to multivariate Gaussian outputs and parallel BO.Source: §D.4 · p. 21
• For scalar Gaussian predictions, standard acquisition sensitivities are bounded when posterior variance is bounded, and analytic PI is bounded globally.The bounds depend on the predictive mean and variance, so they apply to any surrogate with a Gaussian predictive distribution; the appendix limits its discussion to the scalar case.Source: §D.4 · pp. 20–21, Table 3 · p. 21

D.5 Pullback FIM Trace Under the Critical Lengthscale

• λ = Θ(√D_x) is the unique scaling for which the pullback-Fisher trace remains at the critical Θ(D^-1) order; off-critical scalings still yield vanishing acquisition gradients.For λ = cD^p, the trace collapses exponentially when p < 1/2 and decays polynomially when p > 1/2.Source: §D.5 · p. 22
• For isotropic Matérn kernels, the non-degenerate scaling remains λ = Θ(√D_x), but below that scale their trace decays more slowly than for the SE kernel.The SE trace decays as exp(−c1D_x/λ^2), versus exp(−cν√D_x/λ) for Matérn kernels in the stated regime.Source: §D.5 · p. 22

D.6 Connection between TuRBO and FITR

• Near a local optimum of the posterior mean, the pullback tensor is dominated by variance sensitivity, ordering trust-region widths consistently with ARD lengthscales.TuRBO’s lengthscale-scaled box preserves this qualitative ordering, motivating a location-dependent region scaled by predictive sensitivities.Source: §D.6 · p. 22

D.7 Local KL Expansion for Pullback-Fisher Trust Regions

• The full pullback-Fisher tensor gives the second-order term of the KL divergence between nearby predictive posteriors, so its trust region limits local predictive change.FITR uses a diagonal, regularized approximation to form an axis-aligned box compatible with TuRBO-style updates.Source: §D.7 · p. 23

D.8 Pullback FIM Diagonal via Backpropagation

• The pullback-Fisher diagonal equals the expected squared input derivative of the log predictive likelihood and can be estimated by sampling predictive outputs and backpropagating.The estimator applies the outer-product-of-gradients form of the FIM to input coordinates rather than distribution parameters.Source: §D.8 · p. 23

• The authors state that limitations are discussed in Section 5 and a dedicated Limitations section.Source: §2 · p. 24
• The authors report that the research involves neither human subjects nor sensitive data.Source: §2 · p. 27
• The authors characterize the work as theoretical BO research and say they see no broader societal impact.Source: §2 · p. 28
• The authors report that the work releases no models or datasets considered high risk for misuse.Source: §2 · p. 28