Source-linked AI summary
On the Brittleness of Maximum Likelihood Estimation for Gaussian Process Hyperparameter Optimization
Tyler R. Johnson, Kian Ben-Jacob, Christopher P. Muller, Ramin Bostanabad
TL;DR
MLE-based Gaussian process training can be brittle when engineering data violate its assumptions. This paper evaluates practical remedies and finds that carefully implemented MLE can produce competitive GPs that outperform TabPFN across prediction accuracy, uncertainty quantification, and inference cost.
Problem
MLE-based GP training can generalize poorly and miscalibrate uncertainty when engineering data violate assumptions such as correct specification, sufficient data, and independent observations.
Method
The paper systematically evaluates MLE, alternative loss functions, and practical remedies for GP regression and classification across 3,170 simulations spanning noise, data size, and dimensionality.
Results
Carefully implemented MLE can substantially mitigate failure modes and enable GPs to outperform TabPFN in prediction accuracy, uncertainty quantification, and inference cost.
Takeaways & Limitations
Proper MLE implementation is preferable to switching loss functions entirely, while kernel-flow objectives require combination with likelihood-based losses for uncertainty-aware applications.
Takeaways & Limitations
The studies were mostly limited to analytic benchmarks with Gaussian observation noise, leaving non-Gaussian noise and real-world engineering datasets for future work.
Abstract
from arXiv · showhide
Machine learning (ML) has become an indispensable part of modern engineering design workflows. A crucial step in training an ML model is the selection of the loss function which can be systematically formulated via various techniques such as maximum likelihood estimation (MLE) and cross-validation . While MLE is one of the most popular, effective, and intuitive mechanisms for training ML models, it is brittle: if the assumptions underpinning it are not met, the trained ML model may generalize poorly. This brittleness affects even Gaussian processes (GPs) which are widely used in engineering design and are often (incorrectly) presumed to be very robust to overfitting. In this paper, we fundamentally evaluate the brittleness of MLE in the context of training GPs for probabilistic regression or classification tasks. We compare theoretically grounded metrics against MLE and propose practical solutions. Our extensive studies demonstrate the effectiveness of our solutions in downstream design tasks such as Bayesian optimization and provide a blueprint for practitioners to build accurate and robust GPs that can even outperform tabular foundation models in terms of prediction accuracy, uncertainty quantification, and inference cost. Our contributions are publicly available via GitHub at https://github.com/Bostanabad-Research-Group/GP-vs-TabPFN-vs-GPyTorch.
1 Introduction
The introduction frames MLE as a widely adopted but potentially brittle loss-function strategy for Gaussian-process engineering models when its assumptions fail. The paper investigates this brittleness, proposes practical remedies, and compares GP training with modern foundation models across regression and classification settings.
- Motivation: ML increasingly supports engineering design by serving as a surrogate, preference model, data-fusion tool, sensitivity-analysis method, and computer-model calibration approach.Its use is expected to grow as engineering systems become more complex.
- Motivation: MLE is widely adopted because of its statistical foundations and computational tractability, alongside alternatives including cross-validation, proper scoring rules, convex surrogate modeling, and Kullback-Leibler Divergence.The loss function determines how model parameters are learned from data.
- Problem: MLE guarantees typically assume correct model specification, independent observations, and sufficiently large data sets, conditions often violated in engineering design.Under assumption violations, likelihood-trained models may fit training data well yet generalize poorly.
- Motivation: The study asks how to improve MLE-based GP training for mean prediction and uncertainty quantification and how foundation models compare with MLE-trained GPs.Foundation models are designed to generalize across many tasks with minimal task-specific training and increasingly use transformer architectures.
- Contributions: The paper systematically evaluates MLE brittleness in GP regression and classification, proposes practical remedies and alternative loss functions, and studies 3, 170 independent simulations.The simulations vary noise levels, dataset sizes, and input dimensionality.
2 Technical Background
This section introduces Gaussian-process modeling, maximum-likelihood hyperparameter optimization, TabPFN’s prior-data fitted architecture, and their roles in Bayesian optimization and classification. It also contrasts GP retraining with TabPFN’s in-context inference and outlines the computational and inference challenges involved.
- Gaussian processes: Gaussian processes are characterized by mean and covariance functions whose hyperparameters are optimized from training data before probabilistic prediction via multivariate-normal conditioning.The mean uses β and the covariance uses θ; predictions are made at new x∗ after hyperparameter optimization.
- Gaussian processes: Maximum likelihood estimation minimizes the negative log-marginal likelihood, combining a log-determinant complexity term with a quadratic data-fit term.The objective uses Cε = c(x, x′; θ) + σϵ2I and observations centered by m = m(x; β).
- TabPFN: TabPFN approximates Bayesian posterior predictive distributions by repeatedly training on datasets sampled from a user-defined prior, with predictive quality and uncertainty tied to its synthetic training distribution.Its synthetic generators include causal-style mechanisms and Bayesian neural networks.
- TabPFN: TabPFN conditions each query on the entire dataset through attention, while its in-context inference cost remains quadratic in the number of rows.The relevant quantities are Nc labeled context rows and Nq query rows; effective token length also depends on feature representation.
- Bayesian optimization: Bayesian optimization uses an emulator and acquisition function to select evaluations, but GP surrogates require MLE retraining at every step, creating overhead and exposure to MLE brittleness.GPs remain predominant because they learn from small data and make many acquisition functions analytic.
- Classification: GP classification requires approximate inference because discrete labels induce non-Gaussian likelihoods that make exact posterior inference intractable.Standard approaches include Laplace’s method, variational inference, and expectation propagation.
3 Mitigating the Brittleness of MLE
The paper identifies three interconnected sources of MLE brittleness in Gaussian-process training—finite-data effects, non-convex optimization, and mismatch between likelihood fit and generalization—and evaluates alternative losses and careful implementation strategies. These strategies include cross-validation-derived objectives, kernel-flow losses, probabilistic scoring rules, priors, reparameterization, and bounded soft clamping.
- Sources of MLE brittleness: Finite samples and decreasing data density undermine MLE’s asymptotic guarantees, while non-convex optimization can converge to bad local optima, especially affecting GP nugget estimates.The likelihood profile is also connected to dataset size: Gaussian-kernel GP objectives become increasingly convex with more data, but this finding may not transfer to deep kernels.
- Sources of MLE brittleness: MLE optimizes data explanation rather than model-class selection or unseen-data generalization, providing no direct feedback on whether a GP kernel is formulated optimally.The paper therefore treats MLE as potentially brittle even though GPs are often incorrectly presumed robust to overfitting.
- Alternative loss functions: Leave-one-out log-predictive probability directly approximates generalization error, but GPs trained with MLE tend to achieve lower training times and generalization error than those trained with this objective.The leave-one-out objective can be calculated analytically for GPs in one shot rather than by training N independent models.
- Alternative loss functions: Kernel-flow losses support batch-based training by comparing reduced- and full-dataset interpolation errors, but they lack UQ capabilities and are not recommended for UQ tasks without additional losses.A well-generalizing kernel should yield a small kernel-flow objective value.
4 Results and Discussions
Across regression, classification, emulation, uncertainty quantification, and Bayesian optimization, model performance depends strongly on kernels, loss functions, preprocessing, data regime, noise, and dimensionality. GP+ variants can outperform TabPFN, but some losses and settings are brittle and computational trade-offs remain.
- Overall comparisons: GP+, GP+ (PE), and PFN 2.5 perform similarly overall, while GP+ (LOO), PFN 2.0, and GPyTorch excel in some cases but fail in others.These results demonstrate the importance of selecting suitable kernels and loss functions.
- Model selection: The model with the best train-likelihood provides the best or nearly best test performance across the compared settings.This supports selecting among optimized loss-function settings using training loss.
- Buckling: With more Buckling data, GP+ using Gaussian and PE kernels captures categorical inputs better than other models, and its UQ meets or exceeds TabPFN.In the low-data regime, no model performs well and TabPFN’s UQ is significantly ahead of GPs.
- Target preprocessing: Log-normal target preprocessing improves GP+/GPyTorch RRMSE and NIS for Buckling relative to default standardization, while PFN 2.5 and PFN 2.0 remain essentially unchanged.The improvement is motivated by Buckling’s log-scale response distribution.
- Uncertainty and loss sensitivity: GP+ (LOO) is often the weakest UQ model in low-data settings because its narrow intervals produce overconfident point predictions, while LPL(·) is best in selected high-data Ackley cases but sensitive in small-data cases.GP+ (LOO) is also the only GP performing poorly in the cited small-noise example, whereas other GPs outperform TabPFN at small noise and perform similarly at large noise.
- Downstream performance and cost: GPs outperform TabPFN on all Bayesian-optimization problems except Zakharov, while PFN methods fit faster and fitted GPs predict faster; GP+ (PE) training can exceed alternatives by over ten times.Both TabPFN versions outperform GPs on Zakharov emulation because GP standardization mishandles its very large observation scale.
5 Conclusion
The paper concludes that MLE brittleness is most pronounced for Gaussian processes in very small-data, high-dimensional settings, but practical implementation remedies can substantially mitigate failures. It favors robust MLE practice over wholesale loss replacement, while identifying meta-learning, broader evaluation, and regularized MLE as promising directions.
- 5 Conclusion: MLE effectiveness depends on assumptions often violated in engineering applications, including correct model specification, sufficient data, and well-behaved optimization landscapes.The study investigated GP training for engineering regression and classification tasks through 3,170 independent simulations.
- 5 Conclusion: In very small-data, high-dimensional regimes, MLE can produce poorly calibrated GPs because non-convex NLL landscapes expose nugget hyperparameters to bad local optima.Log-scale reparameterization, multiple optimization initializations, and bounded hyperparameters can mitigate these failures without abandoning MLE.
- 5 Conclusion: LOO log-pseudo-likelihood is theoretically more robust to kernel misspecification but costs more and does not consistently outperform default MLE settings in GP+.The authors therefore view proper MLE implementation as a better use of computational budget than switching loss functions entirely; kernel flows lack native UQ capabilities.
- 5 Conclusion: GPs could gain robustness by learning hyperpriors or kernel families from broad synthetic engineering-task corpora, adopting TabPFN-inspired amortized model-selection principles.Such learned priors could warm-start or constrain GP hyperparameter optimization.
- 5 Conclusion: Future work should evaluate non-Gaussian noise, real-world datasets, and downstream tasks beyond BO, while exploring regularized MLE for very small datasets.The current studies mainly used analytic benchmarks with Gaussian observation noise, and UQ errors may compound across fidelity levels in multi-fidelity modeling.
A Additional Benchmark Results
The appendix provides additional benchmark comparisons of model performance, training-NLL rankings, and computational time across multiple problems. It evaluates GP variants, PFN models, and GPyTorch using predictive, uncertainty, and efficiency metrics.
- Performance and uncertainty: Table A1 reports RRMSE and NIS statistics for GP+, its PE and LOO variants, PFN 2.5, PFN 2.0, and GPyTorch across benchmark problems.Entries are median ± std over runs, with the lowest median per metric highlighted for each problem.
- Training NLL: Figure A1 ranks GP+, GP+ (PE), and GPyTorch by training NLL across nine benchmark problems.The problems include Buckling, Borehole, Wing Weight, and several Ackley, Griewank, Zakharov, Dixon-Price, and Rosenbrock instances with specified dimensions.
- Computational cost: Table A2 compares training and inference time for GP+, GP+ (PE), GP+ (LOO), PFN 2.5, PFN 2.0, and GPyTorch across benchmark problems.Times are reported as median ± std.