Source-linked AI summary
What is the goal of unsupervised machine learning?
Aapo Hyvärinen
TL;DR
Unsupervised learning has no single agreed goal, unlike branches with more clearly defined objectives. This paper proposes a tentative taxonomy of four distinct goals and argues that they can be orthogonal. It concludes that the taxonomy may be extended and made more rigorous.
Problem
The paper addresses the lack of a single conceptual goal or objective function for unsupervised learning, a field it describes as heterogeneous.
Source: §1 · p. 1Method
The paper proposes four goals—distribution estimation, data generation, downstream feature extraction, and understanding—and examines how they differ and relate.
Source: §1 · pp. 1–2Results
The paper concludes that unsupervised learning has multiple distinct, potentially orthogonal goals rather than one single goal.
Source: §1 · p. 1, §4 · pp. 9–10Takeaways & Limitations
The proposed four-goal list can be extended, and developing a more rigorous taxonomy remains an open direction.
Source: §4 · p. 10Validating whether unsupervised learning produces human understanding is difficult, and domain experts are often the only proper evaluators.
Source: Validation · p. 8
Abstract
from arXiv · showhide
Unsupervised learning is one of the main branches of machine learning. Here I argue that unlike the other branches of machine learning (supervised and reinforcement learning), unsupervised learning is a rather heterogenous field that can serve several different goals. It seems futile to try to define one single goal for unsupervised learning. I identify four different goals for unsupervised learning: 1) Estimating the distribution, 2) Generating new data points, 3) Extracting features for downstream tasks, and 4) Understanding the data.
1 Introduction
Unlike supervised and reinforcement learning, which have relatively well-defined goals, unsupervised learning has no single agreed goal. The paper proposes four tentative goals and treats self-supervised learning as pursuing the same goals.
1 Introduction
- Supervised learning commonly maximizes prediction accuracy and reinforcement learning total reward, but unsupervised learning has no single conceptual goal or objective function.The paper describes relatively well-defined goals and broad consensus for the first two branches.Source: §1 · p. 1
- Unsupervised learning can serve distinct goals that may be orthogonal, so pursuing one need not advance another and goals may even conflict.Source: §1 · p. 1
- The paper treats self-supervised learning as an algorithmic technique using supervised algorithms to pursue the same goals as unsupervised learning.Source: §1 · p. 2
- The paper tentatively identifies four goals: estimating the distribution, generating data, extracting downstream features, and understanding the data.Source: §1 · p. 2
- The discussion justifies the goals and their relations, points out basic methods, emphasizes validation, and is not a comprehensive review; related analysis appeared in.Source: §1 · p. 2
Definitions
- The basic probabilistic setup assumes real-valued observations without labels and an i.i.d. sample, although the independence assumption could be relaxed.Source: Definitions · p. 2
Goal #1: Estimating the distribution
Distribution estimation learns a data density or related quantity, and can theoretically support supervised prediction. Its interpretability and validation remain limited: standard checks often permit comparisons without establishing intuitive fit.
Problem statement
- Learning the data density—or its logarithm or score—is presented as a fundamental unsupervised goal; known class densities can theoretically support classification and conditional-density regression.Classification can use the density ratio p1/p2; conditional densities can be obtained from a joint density over concatenated inputs and targets.Source: Problem statement · pp. 2–3
- Density estimation may not reveal data structure when its neural-network approximator is an uninterpretable black box, and its practical value for supervised tasks is unclear.Source: Problem statement · p. 3
Methods
- Kernel density estimators perform poorly in high dimensions [8]; deep energy- and score-based methods tend to be state-of-the-art or close, while normalizing flows enable maximum-likelihood estimation with possible quality costs.Flows impose neural-network restrictions to make maximum likelihood possible, which may reduce density-approximation quality.Source: Methods · p. 3
Validation
- Held-out objective values can compare trained density models, but their unclear scale prevents judging intuitive fit, limiting this validation to comparisons between architectures or similar cases.Source: Validation · p. 3
- Sampling from the learned distribution is another validation approach, but it recasts evaluation as data-generation validation, which is difficult.Source: Validation · p. 4
Goal #2: Generating new data points
Data generation samples from a distribution learned from observed data, though some methods generate directly without explicit density estimation. Its validation remains difficult because no simple measure establishes distributional fidelity, and common checks have important scope or design constraints.
Problem statement
- Modern generative AI learns to sample from the data distribution using observed data alone; text generation is outside this discussion.In practice, the distribution may be conditioned on a prompt or similar input.Source: Problem statement · p. 4
Methods
- Generation can estimate a density or score before sampling, while GANs learn to generate directly and latent-variable models can also generate samples.Langevin methods and diffusion models are sampling approaches; latent-variable models may be less competitive than methods optimized for generation.Source: Methods · p. 4
Validation
- No simple measure establishes whether generated points follow the original data distribution, and validation must also rule out copying training examples.Source: Validation · p. 4
- Inception scores or distances evaluate generated images against labeled benchmark classes, but are limited to domains with such benchmarks.Source: Validation · p. 5
- A neural-network discriminator is a more general proposed check, but it adds no new information for GANs and depends on the evaluator’s architecture and training.A general solution to the validation problem is hard to find.Source: Validation · p. 5
Goal #3: Extracting features for downstream tasks
Feature extraction learns representations from unlabeled data to make a downstream task easier, including supervised learning when labeled examples are scarce. Many methods can provide features, and downstream performance offers a straightforward way to compare them in this setting.
Problem statement
- Feature extraction learns a mapping from inputs to useful features for a downstream task, which may be supervised, reinforcement-learning, or further unsupervised learning.The discussion focuses on supervised downstream tasks, closely related to semi-supervised learning.Source: Problem statement · p. 5
- When unlabeled data are plentiful but labels are costly, features learned from unlabeled examples can make prediction possible with a small labeled set.Source: Problem statement · p. 5
Methods
- Autoencoders reconstruct their input through a bottleneck, while contrastive methods recast learning as classification; both are prominent self-supervised approaches.Variational autoencoders provide a probabilistic framework, but the text questions why they should be preferred to basic autoencoders.Source: Methods · pp. 5–6
- For reinforcement-learning-related tasks, successor representations and proto-value functions explore actions without using rewards, and are unsupervised in that sense.Source: Methods · p. 6
Validation
- In semi-supervised learning, conventional downstream measures such as classification accuracy can straightforwardly compare feature-extraction methods.This evaluation applies to one possible goal, not all unsupervised-learning methods.Source: Validation · p. 6
Goal #4: Understanding the data
Understanding data means seeking interpretable structure or mechanisms, not merely predictive performance. The paper distinguishes this goal from feature extraction, emphasizes identifiability and interpretable models, and notes that validation often requires domain experts.
Problem statement
- In scientific analysis, machine learning may aim to understand data and measured phenomena rather than predict, but neural-network decisions can be difficult to interpret.Source: Problem statement · p. 6
- Features useful for downstream prediction do not necessarily support human understanding, so understanding is a distinct learning goal.Complex computation or an overwhelming number of features can obstruct interpretation.Source: Problem statement · pp. 6–7
- Interpretation calls for identifiable features, meaning unique and well-defined features; features recoverable only up to a linear transformation can make interpretation futile.Identifiability may be less important for downstream feature extraction.Source: Problem statement · p. 7
Methods
- Latent-variable models can assign components to underlying phenomena, while simple linear models are often preferred for interpretation.Independent Component Analysis provides an identifiable alternative to linear factor analysis and PCA, which lack identifiability.Source: Methods · p. 7
- Nonlinear methods such as autoencoders and VAEs typically lack identifiability guarantees, making their features’ interpretation questionable.In a typical VAE, features are defined only up to an orthogonal transformation, with stronger indeterminacies also possible.Source: Methods · p. 7
- Causal discovery and clustering offer other routes to interpretable structure, including causal connections, subpopulations, disease subtypes, or spatial regions.Interpretable statistical parameters such as means and variances can also help describe data.Source: Methods · pp. 7–8
Validation
- Understanding is difficult to validate because the intended result is human understanding, so domain experts are often the only proper evaluators.Causal directions or known clusters may sometimes provide ground truth, though experiments to measure causal directions can be difficult.Source: Validation · p. 8
3 Discussion
The four goals can overlap, but success at one does not guarantee progress toward another and may sometimes conflict. The proposed list is explicitly provisional, and related work on evaluating generative models is distinguished from the paper’s broader framing of learning goals.
3.1 Connections between the goals
- Density estimation can support sampling, and learned features may serve both downstream tasks and understanding, though the optimal features can differ.Density estimation and sampling can be black-box goals that do not advance feature extraction or understanding; simple models for understanding may be less useful for other goals.Source: §3.1 · pp. 8–9
3.2 Criticism of this list and possible further goals
- The four-goal list is neither exhaustive nor definitive; causal learning, compression, and data restoration or cleaning may constitute additional goals.Density estimation may be an intermediate goal rather than a practical endpoint, though its theoretical importance and applications motivate its inclusion.Source: §3.2 · p. 9
3.3 Related work
- Prior work compares evaluation methods for statistical generative models, whereas this paper treats learning itself as serving distinct goals and adds understanding data as Goal #4.Source: §3.3 · p. 9
4 Conclusion
The conclusion reiterates that unsupervised learning has no single goal; its goals can differ and be orthogonal.
4 Conclusion
- Unsupervised learning has different goals that can be orthogonal to each other.The conclusion proposes four goals, while noting that the list can be extended.Source: §4 · pp. 9–10
- The proposed list of four goals can be extended in future work.Source: §4 · p. 10
- A more rigorous taxonomy of unsupervised-learning goals could be developed.Source: §4 · p. 10