Probabilistic Matrix Factorization for Automated Machine Learning

1. Introduction

• Bayesian optimization models loss as a function of hyperparameters and uses a regression model and acquisition function to choose evaluations iteratively.Unlike grid or random search, the approaches described do not exploit continuity in parameter space.Source: §1 · p. 1
• Bayesian optimization can struggle in high-dimensional spaces, and tuning whole pipelines makes the search increasingly impractical.The passage attributes the difficulty to needing enough sampled configurations to estimate a predictive posterior.Source: §1 · p. 2
• Pipeline search combines preprocessing, model selection, and hyperparameter tuning across continuous, discrete, categorical, and conditional dimensions.The paper fixes pipeline instantiations ahead of training because modeling continuity across this mixed space is challenging.Source: §1 · p. 2
• Pipeline performance prediction across datasets is cast as collaborative filtering and solved with probabilistic matrix factorization.Gaussian process latent variable models embed pipelines according to their performance across datasets.Source: §1 · p. 2
• The method significantly outperforms the current state of the art in the vast majority of evaluated datasets.Its probabilistic model supplies a predictive posterior that can be paired with acquisition functions to guide exploration.Source: §1 · p. 2

2. Related work

• Multi-task Bayesian optimization treats datasets as related tasks, while cold-start methods select initial settings using results from related datasets.Examples include modeling datasets as separate tasks or learning dataset-specific embeddings (Swersky et al., 2013; Schilling et al., 2015; Springenberg et al., 2016; Perrone et al., 2017).Source: §2 · p. 2
• Feurer et al. (2015) warm-start automated machine learning using dataset meta-features and optimization runs from the nearest datasets.Other cold-start approaches also use meta-features, while Wistuba et al. (2015) additionally consider configuration performance on the new dataset.Source: §2 · pp. 2–3
• Latent-variable methods have also selected algorithms for satisfiability and other combinatorial problems by decomposing solver-performance data.Some project hand-crafted instance meta-features into the latent space for one-shot selection.Source: §2 · p. 3
• Compared with Feurer et al. (2015), this work discretizes pipeline space and uses a multi-task model rather than continuity modeling with single-task smart initialization.Compared with Stern et al. (2010), it performs sequential model-based optimization with a nonlinear latent-to-observed mapping rather than supervised linear one-shot selection.Source: §2 · p. 3

3. AutoML as probabilistic matrix factorization

• Pipeline performance across datasets forms a matrix that can be factorized to predict pipeline performance on a new dataset, under a balanced-accuracy objective that can be replaced by another loss.The method draws on similarities in observed results across datasets and pipelines.Source: §3 · p. 3

3.1. Non-linear matrix factorization with Gaussian process priors

• A Gaussian process prior makes the mapping from pipeline latent variables to performance nonlinear, with the squared-exponential kernel imposing a smooth-function assumption.The paper uses the squared-exponential kernel and leaves testing Matérn kernels to future work.Source: §3.1 · pp. 3–4
• Adding pipeline or dataset metadata through extra kernels did not improve performance; the latent-variable model captured the needed information unsupervised.Figure 2 shows pipeline embeddings colored by model, and its caption says the latent space captures model structure.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2. Inference with missing data

• For sparse performance matrices, missing observations are handled by dropping them from the mean and covariance; inference updates global parameters and evaluated pipelines’ latent variables by stochastic gradient descent.Multiple pipeline runs across datasets are described as embarrassingly parallel.Source: §3.2 · p. 4

3.3. Predictions

• Predictions for a new dataset use Gaussian-process regression, with computational cost driven largely by the number of pipelines already evaluated there.The prediction requires inversion of an N_j × N_j matrix; the paper expects evaluations to number in the hundreds.Source: §3.3 · p. 4

3.4. Acquisition functions

• Expected improvement selects the next pipeline by accounting for predictive uncertainty, while its free parameter ξ encourages exploration.The paper notes that different problems may call for different acquisition functions.Source: §3.4 · p. 5

4. Experiments

• 553 filtered OpenML classification datasets were used; 89 formed the held-out test set and 464 the training set.The test set was selected with probabilities proportional to random-search regret, then reduced after removing datasets used to train auto-sklearn.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1. Generation of training data

• The training data came from 42,000 pipelines per dataset, with 80/10/10 training, validation, and test splits; the resulting performance matrix had about 21% missing entries.Individual pipeline models were limited to 30 seconds of training time and 16GB of memory.Source: §4.1 · p. 6

4.2. Parameter settings

• The model used 20 latent dimensions, a learning rate of 1e−7, batch size 50, PCA initialization, and 300 training epochs.Training took approximately three hours on a 16-core Azure machine; the acquisition function used ξ = 0.01.Source: §4.2 · p. 6

4.3. Results

• PMF achieves the best average rank from the first iteration, while auto-sklearn ranks second.Random 2x and Random 4x use larger computational budgets and are described as impractical baselines.Source: §4.3 · pp. 6–7, Figure 4 · p. 6
• PMF also outperforms the other methods in balanced-accuracy comparisons and remains best when trained on a subset of the data.With 90% of entries in Y missing, performance degrades only slightly while remaining best among competitors.Source: §4.3 · p. 7, Figure 5 · p. 7
• As pipeline evaluations accumulate, test-set prediction error decreases and posterior uncertainty falls, with the error appearing to asymptote after 200 iterations.The reported error is mean squared error averaged across test datasets.Source: §4.3 · p. 7, Figure 6 · p. 7
• Including pipeline metadata did not improve performance; the model captures pipeline composition and hyperparameter structure from Y when enough experimental data is available.The paper illustrates this with latent embeddings organized by model choice and by a PCA preprocessor's retained-variance setting.Source: §4.3 · p. 7

• The method automates preprocessing and model selection alongside tuning their hyperparameters for a given dataset.Source: §5 · p. 7
• Across 89 OpenML datasets, the approach outperformed the state of the art and strong baselines.Source: §5 · p. 8
• Although pipeline sampling could require exponentially many instances, conditioned dimensions and increased sample size mitigate the concern; traditional Bayesian optimization can further tune parameters if needed.The authors also report that dense sampling is needed only when performance is highly sensitive to small parameter changes, which they had not observed in practice.Source: §5 · p. 8
• Planned extensions include adding dataset metadata, accounting for pipeline runtime costs in acquisition functions, and testing variational-autoencoder-based matrix factorization.The experiments constrained model runtimes, which the authors note could be impractical in real applications with large datasets.Source: §5 · p. 8