PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta-Learning: A Survey

Joaquin Vanschoren

arXiv:1810.03548v1cs.LGstat.ML

TL;DR

Meta-learning asks how prior experience with related tasks can be systematically and data-drivenly used instead of starting new learning problems from scratch. This survey organizes approaches that exploit evaluations, task properties, learned defaults, model parameters, and task-specific learning mechanisms. It presents a broad set of ways to guide model and configuration selection, while noting that transfer is constrained when target tasks are dissimilar and that several methods do not yet generalize across tasks.

  • Problem

    Meta-learning seeks a systematic, data-driven way to collect and exploit prior task and model experience so new learning tasks need not start from scratch.

  • Method

    The chapter surveys approaches that learn from model evaluations, task characteristics, configuration defaults, trained parameters, and few-shot task experience.

  • Results

    The survey presents meta-learning as a broad family of data-driven approaches for recommending configurations, designing search spaces, transferring knowledge, and replacing hand-engineered base-learners.

  • Takeaways & Limitations

    Collecting learning exhaust across increasingly similar tasks can help AutoML systems improve their efficiency over time.

  • Takeaways & Limitations

    Transfer approaches work less well when the target task is not sufficiently similar, and many neural-network methods do not yet generalize across tasks.

Abstract

from arXiv · show

Meta-learning, or learning to learn, is the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks, and then learning from this experience, or meta-data, to learn new tasks much faster than otherwise possible. Not only does this dramatically speed up and improve the design of machine learning pipelines or neural architectures, it also allows us to replace hand-engineered algorithms with novel approaches learned in a data-driven way. In this chapter, we provide an overview of the state of the art in this fascinating and continuously evolving field.

1. Introduction

Meta-learning systematically learns from prior tasks and models to guide learning on new tasks. Its approaches range from using model evaluations and task characteristics to transferring parameters between inherently similar tasks.

  • Prior experience can reduce the examples and trial-and-error needed to learn new tasks, especially when those tasks are related.
  • Meta-learning collects prior task and model experience, then learns from it to guide the search for effective models on new tasks.The meta-data can include algorithm configurations, evaluations, learned parameters, and measurable task properties.
  • Task similarity determines how much prior experience and meta-data can be leveraged, while unrelated or noisy tasks provide little useful transfer.
  • The chapter categorizes methods by meta-data specificity, covering model evaluations, task characterization, and trained-parameter transfer.These categories support configuration recommendation, meta-modeling of task characteristics and performance, transfer learning, and few-shot learning.
  • Multi-task and ensemble learning can complement meta-learning systems but do not themselves learn from prior experience on other tasks.

2. Learning from Model Evaluations

Evaluation-based meta-learning uses prior configuration results to train a meta-learner for recommending configurations on a new task. New-task evaluations can be incorporated iteratively, sometimes after another method provides an initial warm start.

  • The meta-learner is trained on prior evaluations P and new-task evaluations Pnew to predict recommended configurations for tnew.P contains evaluations across known tasks, while Pnew contains evaluations observed on the new task.
  • The configuration space Θ may be discrete, continuous, or mixed and can include hyperparameters, pipeline components, or network-architecture components.
  • Prior evaluations are typically collected beforehand or extracted from meta-data repositories, whereas Pnew is learned iteratively by the meta-learning technique.

2.1 Task-Independent Recommendations

Task-independent recommendation methods use evaluations from prior tasks to rank or recommend configurations without initially using evaluations from the new task. The resulting recommendations can be evaluated directly or used to warm-start further optimization.

  • With no evaluations on tnew, a learned function can recommend configurations independently of tnew for later evaluation or optimization.The recommended configurations can be evaluated on tnew to select the best one or warm-start other optimization methods.
  • Candidate configurations are commonly organized into portfolios and ranked per task using success rates, AUC, or significant wins.
  • Ranking methods can trade off accuracy against training time so equally effective but faster algorithms rank higher.
  • An anytime strategy evaluates the top-K configurations sequentially on tnew and can stop after a predefined evaluation budget.
  • Differentiable per-task performance models can be optimized with gradient descent, with configurations from similar prior tasks helping warm-start Bayesian optimization.

2.2 Configuration Space Design

Meta-learning can improve configuration-space design by narrowing search toward relevant regions and by identifying which hyperparameters are worth tuning. These approaches learn defaults or importance from performance patterns across many tasks.

  • Learning a better configuration space Θ* can radically speed model search by restricting exploration to more relevant regions when computational resources are limited.
  • Functional ANOVA deems hyperparameters important when they explain most of the variance in algorithm performance on a task.One evaluation used 250,000 OpenML experiments covering 3 algorithms and 100 datasets.
  • Hyperparameter importance can instead be defined as the performance gain from tuning a hyperparameter rather than leaving it at an optimal learned default.
  • Defaults can be learned jointly across tasks by minimizing average risk, after which tunability is estimated from the improvement still available through tuning.The described evaluation used about 500,000 OpenML experiments on 6 algorithms and 38 datasets.
  • When default values depend on meta-features, simple functions can incorporate those features and statistical tests can assess whether leaving a hyperparameter at default is safe.

2.3 Configuration Transfer

Configuration transfer uses evaluations from prior tasks to identify tasks similar to a new task and guide which configurations to evaluate next. Methods transfer information through surrogate models, acquisition functions, learned task representations, or task-selection procedures.

  • Empirical task similarity is inferred by comparing prior evaluations with new-task evaluations, then incorporated into a meta-learner that repeatedly updates recommendations.Similarity can be based on comparable configuration performances, with new evaluations collected iteratively.
  • Relative landmarks compare pairwise performance differences between configurations and support tournament-style active testing from a globally best configuration.The landmark is defined as RLa,b,j = Pa,j − Pb,j for configurations θa and θb on task tj.
  • Per-task surrogate models define similarity by how accurately a prior task predicts new-task evaluations, commonly within Bayesian optimization.Each surrogate sj(θi) predicts Pi,j, and prediction error on Pi,new provides the similarity signal.
  • Gaussian-process methods combine prior-task predictions or predictive distributions using similarity- or generalization-based weights.Weights may use relative-landmark similarity, Nadaraya-Watson kernel averaging, or estimated generalization performance.
  • Meta-data can instead influence the acquisition function, with prior-task expected improvements weighted alongside new-task improvement and gradually reduced as new evidence accumulates.The weight of the new-task expected-improvement component increases with each iteration.
  • Other approaches learn joint task representations, select related source tasks with multi-armed bandits, or combine optimum distributions using KL-divergence.The KL-based approach was evaluated for tuning two SVM hyperparameters across five tasks.

2.4 Learning Curves

Learning curves record performance as training progresses and let meta-learning predict full-training outcomes from partial curves. This supports stopping poor configurations early and accelerating configuration search, although prior-task curves had not yet been used in neural architecture search.

  • Learning curves measure configuration performance after successive training steps, producing a trajectory across increasing amounts of training data.Performance is represented as P(θi, tj, st) = Pi,j,t after step st on task tj.
  • Partial learning curves on a new task can predict full-dataset performance and determine whether training should continue.Training may be halted after r iterations with r < t, using prior experience from other tasks.
  • Nearest-task methods compare partial-curve shapes across configurations and adapt complete curves from the k most similar tasks to predict new-task outcomes.Task distance is defined from partial curves, and complete curves from similar tasks provide the prediction basis.
  • Neural architecture search methods had predicted learning curves, but had not leveraged learning curves previously observed on other tasks.

3. Learning from Task Properties

Task properties are encoded as meta-feature vectors that support task similarity and transfer of performance information. Meta-features may be manually engineered, aggregated from data properties, specialized by task type, or learned jointly from performance meta-data.

  • Each task is represented by a K-dimensional meta-feature vector, and distances between vectors can identify prior tasks for information transfer.The representation is m(tj) = (mj,1, ..., mj,K), with features drawn from the set M.
  • Common meta-features span simple, statistical, information-theoretic, complexity, model-based, and landmarking groups.Table 1 also gives rationales and, where possible, formulas for computing these features.
  • Categorical features and classes can be summarized through categorical values, conditional and joint probabilities, and marginal probabilities.
  • Meta-feature construction requires application-dependent selection, aggregation with summary statistics or histograms, systematic extraction, and normalization before similarity computation.Studies on OpenML data found that the optimal feature set depends on the application.
  • Specialized meta-features include streaming landmarks, time-series autocorrelations and regression slopes, and cluster properties for unsupervised tasks.
  • Joint task representations can be learned from task meta-features and performance data, including binary configuration-pair landmarks or Siamese-network representations for shared input spaces.These approaches replace or augment manually defined meta-features with learned representations.

3.3 Warm-Starting Optimization from Similar Tasks

Warm-starting optimization transfers promising configurations from empirically similar tasks to initialize search on a new task. Approaches include genetic search, surrogate ranking or regression, Bayesian optimization, and collaborative filtering, with different meta-feature and scalability choices.

  • Meta-features estimate task similarity and initialize optimization with configurations that performed well on related tasks.This mirrors how experts begin model searches using experience from related tasks.
  • Genetic search can start near promising solutions by evaluating the best configurations from the k most similar tasks on the new task.Similarity uses L1 distance over 17 simple and statistical meta-features.
  • SCoT trains a surrogate ranking model using four meta-features to predict configuration ranks across tasks, avoiding direct comparison of differently scaled evaluation values.Its features comprise three simple features and one PCA-based feature.
  • A factorization-machine-based multilayer perceptron models latent task representations, while an ensemble of 100 networks supplies predictive means and simulated variances.The ensemble addresses the base model’s inability to represent uncertainties.
  • Bayesian surrogate models can restrict training to tasks similar to the new task, standardize evaluations, or warm-start with configurations from d similar tasks.The warm-starting method uses 46 simple, statistical, and landmarking meta-features and also searches preprocessing steps.
  • Collaborative filtering treats tasks as users and configurations as items, using performance ratings to recommend configurations, but it faces a cold-start requirement for new-task evaluations.Other methods use meta-features to address cold start, and some recommend configurations that are accurate and fast.

3.4 Meta-Models

Meta-models use prior-task evidence to recommend, rank, or predict promising configurations for a new task. Their outputs typically warm-start or guide a separate optimization method rather than tuning the new task directly.

  • Meta-models learn relationships between task meta-features and configuration utility to recommend promising configurations for a new task.
  • Meta-models can rank the top-K configurations by predicting task similarity or directly modeling which configurations perform best.
  • Performance meta-regressors predict configuration accuracy or training time from task meta-features, helping estimate which evaluations are worthwhile.
  • Most meta-models generate promising configurations but do not tune them on the new task themselves.
  • Predictions can warm-start or guide Bayesian optimization, evolutionary algorithms, and other optimization techniques.

3.5 Pipeline Synthesis

Meta-learning supports pipeline synthesis by using prior experience to control large configuration spaces and recommend promising components. These recommendations can guide beam search, planning, evolutionary methods, reinforcement learning, and Monte Carlo Tree Search.

  • Pipeline configuration spaces grow dramatically, so prior experience can warm-start Bayesian optimization with promising pipelines on similar tasks.
  • Meta-learners recommend pipeline steps or components that larger construction systems use in planning, beam search, or evolutionary techniques.
  • AlphaD3M uses self-play reinforcement learning and Monte Carlo Tree Search to generate pipelines and train an LSTM to predict their performance.
  • Meta-models can predict whether tuning an algorithm is worthwhile and estimate expected improvement relative to the additional time investment.

4. Learning from Prior Models

Learning from prior models transfers structures, parameters, representations, and inductive biases across related tasks. The chapter surveys transfer learning, learned optimizers, neural meta-learners, and few-shot methods, while noting that transfer weakens across dissimilar tasks and many architecture-search methods do not generalize across tasks.

  • 4.1 Transfer Learning: Transfer learning uses models from source tasks as starting points for creating models on similar target tasks.
  • 4.1 Transfer Learning: Neural networks are suited to transfer because source structures and parameters provide initialization that can be fine-tuned with target-task data.
  • 4.1 Transfer Learning: Transfer works less well when the target task is dissimilar, motivating inductive biases learned from many similar tasks.
  • 4.2 Meta-Learning in Neural Networks: Meta-learning in neural networks includes RNNs that modify their own weights, learned optimizers, and reinforcement-learning formulations of optimization algorithms.
  • 4.2 Meta-Learning in Neural Networks: Many neural architecture-search methods model performance for a specific task but do not yet generalize across tasks.
  • 4.3 Few-Shot Learning: Few-shot learning trains models from few examples by leveraging prior experience with very similar tasks and large training sets.
  • 4.3 Few-Shot Learning: Matching Networks learn a shared representation and match test instances to memorized examples using cosine similarity.
  • 4.3 Few-Shot Learning: Prototypical Networks classify examples by mapping them into a vector space and comparing them with class prototypes.

5. Conclusion

The conclusion presents meta-learning as a broad effort to systematically reuse experience from prior tasks. It argues that accumulating meta-data can make future learning more efficient, especially as the number and similarity of encountered tasks increase.

  • Meta-learning spans many techniques and treats each learning attempt as experience that can support learning new tasks.
  • More numerous and similar tasks provide more opportunity to reuse prior experience, potentially reducing how much learning is needed for new tasks.
  • The conclusion frames stored meta-data as a resource for building AutoML systems that improve their efficiency over time.
Loading 1810.03548v1…