PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Learning with Statistical Models

D. A. Cohn, Z. Ghahramani, M. I. Jordan

arXiv:cs/9603104v1cs.AI

TL;DR

Most machine-learning research treats the learner as a passive recipient of data, rather than addressing how it can select training examples. The paper applies active-learning principles to mixtures of Gaussians and locally weighted regression, deriving criteria for efficient, statistically correct data selection. Using the variance-minimizing criterion is significantly better than the learner’s selection.

  • Problem

    Most machine-learning research treats the learner as a passive recipient of data, rather than addressing how it can select training examples.

    Source: p. 1

  • Method

    The paper applies active-learning principles to mixtures of Gaussians and locally weighted regression, deriving criteria for efficient, statistically correct data selection.

    Source: p. 2, . Discussion · p. 14

  • Results

    Using the variance-minimizing criterion is significantly better than the learner’s selection.

    Source: p. 12

  • Takeaways & Limitations

    The tested criteria are inexpensive to compute and show good predictive power, with potential time and money savings when collecting examples is costly.

    Source: p. 1, . Discussion · p. 14

    Good coverage of the potential query space may require many candidate points.

    Source: p. 8

Abstract

from arXiv · show

For many types of machine learning algorithms, one can compute the statistically `optimal' way to select training data. In this paper, we review how optimal data selection techniques have been used with feedforward neural networks. We then show how the same principles may be used to select data for two alternative, statistically-based learning architectures: mixtures of Gaussians and locally weighted regression. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate. Empirically, we observe that the optimality criterion sharply decreases the number of training examples the learner needs in order to achieve good performance.

App endix A. Notation

This appendix defines notation for inputs, outputs, queries, predictions, and the quantities used by neural networks, Gaussian mixtures, and locally weighted regression.

Appendix A. Notation

  • The notation distinguishes an arbitrary input x, its true output y, a predicted output, and a query’s specified input and possibly unknown output.It also denotes the estimated variance of the prediction.Source: App endix A. Notation · p. 15
  • Neural-network notation includes the known input distribution, a weight vector, estimated weights from training data, and the network’s computed function.The notation also includes an average estimate of data noise.Source: App endix A. Notation · p. 15
  • Gaussian-mixture notation identifies the total number of Gaussians, component i, its point weight, and its estimated input and output means.It also names each component’s estimated input variance.Source: App endix A. Notation · p. 15
  • Additional mixture notation covers output variance and input-output covariance for component i, plus output variance conditional on an input.It also denotes the joint input-output distribution and the input distribution conditional on a component.Source: App endix A. Notation · p. 15
  • For locally weighted regression, the notation names the kernel smoothing parameter, the kernel’s example weights, and their sum.It also denotes kernel-weighted input and output means and the weight assigned to a new point.Source: App endix A. Notation · p. 15
  • The notation also specifies a training-set size, a query input and possible output, and the weight of a new point under a kernel.For mixtures, it defines the weight attributed to a new point for Gaussian i.Source: App endix A. Notation · p. 15

Ac kno wledgemen ts

The acknowledgements give the authors’ current institutional addresses, identify funding sources, and thank colleagues for editorial assistance.

Acknowledgements

  • David Cohn’s current address is Harlequin, Inc., in Cambridge, Massachusetts; Zoubin Ghahramani’s is the University of Toronto’s Department of Computer Science.The passage also gives the Toronto address’s postal details.Source: Ac kno wledgemen ts · p. 16
  • The work was funded by an NSF grant, the McDonnell-Pew Foundation, ATR Human Information Processing Laboratories, and Siemens Corporate Research.Source: Ac kno wledgemen ts · p. 16
  • The authors thank Michael Titterington and Jim Kay for careful attention and continued help correcting an earlier version.Source: Ac kno wledgemen ts · p. 16

Related papers

Loading cs/9603104v1…