Source-linked AI summary
On the Importance of Strong Baselines in Bayesian Deep Learning
Jishnu Mukhoti, Pontus Stenetorp, Yarin Gal
TL;DR
UCI regression comparisons in Bayesian Deep Learning have often pitted converged models against baselines trained for a fixed number of iterations. The paper reevaluates those comparisons using MC dropout under matched settings and finds it substantially improves over timed-setting results, outperforming or competing closely with the compared methods.
Problem
Some UCI regression comparisons for approximate-inference methods used baselines trained under a different experimental setting, making matched evaluation important.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
The authors conclude that benchmark comparisons should use experimental setups identical to those used to evaluate peers.
Source: §3 · p. 3
Abstract
from arXiv · showhide
Like all sub-fields of machine learning Bayesian Deep Learning is driven by empirical validation of its theoretical proposals. Given the many aspects of an experiment it is always possible that minor or even major experimental flaws can slip by both authors and reviewers. One of the most popular experiments used to evaluate approximate inference techniques is the regression experiment on UCI datasets. However, in this experiment, models which have been trained to convergence have often been compared with baselines trained only for a fixed number of iterations. We find that a well-established baseline, Monte Carlo dropout, when evaluated under the same experimental settings shows significant improvements. In fact, the baseline outperforms or performs competitively with methods that claimed to be superior to the very same baseline method when they were introduced. Hence, by exposing this flaw in experimental procedure, we highlight the importance of using identical experimental setups to evaluate, compare, and benchmark methods in Bayesian Deep Learning.
1 Introduction
Bayesian Deep Learning applies Bayesian inference to deep networks, and UCI regression is a popular setup for evaluating approximate inference methods. Re-evaluating comparisons under matched settings, the paper finds MC dropout outperforms several methods that had claimed superior performance over it.
1 Introduction
- Empirical comparisons can be undermined by experimental discrepancies, as prior work has noted challenges in judging progress in reinforcement learning and language modeling.Source: §1 · p. 1
- Bayesian Deep Learning applies Bayesian inference to deep networks, with approximate-inference approaches including dropout, deep Gaussian processes, and SGHMC variants.Source: §1 · pp. 1–2
- UCI regression is a popular benchmark for Bayesian neural networks because it evaluates predictive log-likelihood alongside RMSE, allowing uncertainty estimates to be assessed.Source: §1 · p. 2
- Under the same experimental conditions, MC dropout outperforms VMG, HS-BNN, and SGHMC, and is a close second to PBP-MV.The authors say the earlier publications consequently claimed state-of-the-art performance erroneously.Source: §1 · p. 2
2 Regression Experiments
The experiments compare Monte Carlo dropout with prior UCI regression results obtained under different training regimes. When trained to convergence or with tuned hyperparameters, dropout substantially improves on timed-setting results and is competitive with the compared methods.
2 Regression Experiments
- The study uses the curated UCI nonlinear-regression datasets except YearPredictionMSD, with one hidden layer of 50 units, or 100 for Protein Structure.YearPredictionMSD was excluded because its 515,345 instances and 90 dimensions made hyperparameter tuning require an inordinate amount of time.Source: §2 · p. 2
- The experiment has a timed variant of 40 epochs and a convergence variant with more iterations; both evaluate test RMSE and log-likelihood.Source: §2 · p. 2
- The authors test whether timed and converged models are comparable by evaluating MC dropout, a de facto standard approximate-inference baseline.Source: §2 · p. 2
- MC dropout trained to convergence or with tuned hyperparameters substantially improves over timed-setting results in RMSE and log-likelihood.The reported RMSE wins are on Concrete Strength, Naval Propulsion Plants, Wine Quality Red, and Yacht Hydrodynamics; log-likelihood wins are on Boston Housing, Concrete Strength, and Wine Quality Red.Source: Table 1 · p. 3, Table 2 · p. 3, §2 · p. 3
- In the hyperparameter-tuning variant, 4,000-epoch training is paired with grid search over precision and dropout rate, selecting the best pair on a validation set.The validation set is a random 20% of the training data; the convergence variant also uses 4,000 epochs and Bayesian Optimization.Source: §2 · p. 3
- On the remaining datasets, MC dropout is competitive and second only to PBP-MV; the cited VMG, HS-BNN, PBP-MV, and SGHMC values come from their respective papers.Source: §2 · p. 3
3 Conclusion
The conclusion is that MC dropout performs substantially better when trained under convergence or hyperparameter-tuning settings than in the timed setting. The authors judge previous comparisons with timed-setting baselines unrepresentative and recommend identical experimental setups for benchmarking.
3 Conclusion
- Convergence and hyperparameter-tuning experiments give MC dropout substantially better RMSE and log-likelihood results than the timed setting.Source: §3 · p. 3
- The authors conclude that comparisons against timed-setting baselines were unrepresentative because those models had not reached convergence.Source: §3 · p. 3
- The paper recommends evaluating benchmarked methods with the same experimental setup used for their peers.Source: §3 · p. 3