Source-linked AI summary
Diffusion Models and Concept Formation
Zekun Wang, Karthik Singaravadivelan, Christopher J. MacLellan
TL;DR
Human categorization includes a basic level that a computational account should explain, and the paper asks whether diffusion models implicitly represent concepts hierarchically like Cobweb. It recovers diffusion hierarchies by mode-finding and compares their basic levels with Cobweb’s, reporting alignment on MNIST and Fashion-MNIST.
Problem
A computational account of human categorization should explain why a basic level supports recognition with least cognitive effort.
Source: §1 · p. 1Method
The paper recovers diffusion’s implicit hierarchy through score-based mode-finding and compares its basic level with Cobweb’s.
Source: §1 · p. 2, §3.4 · p. 9, §4.2 · p. 12Results
The recovered hierarchies and basic levels align between diffusion and Cobweb on MNIST and Fashion-MNIST.
Source: §1 · p. 2, §4.2 · p. 13Takeaways & Limitations
Diffusion models can be interpreted as hierarchical concept models, with their basic level at an intermediate noise level where reverse diffusion commits to class identity.
Source: §5 · p. 13The hierarchy is indexed by a sample-quality-tuned noise schedule, whose nonlinear mapping to signal-to-noise ratio makes abstraction levels unevenly resolved and the basic-level timestep cognitively uninterpretable.
Source: §5 · pp. 14–15
Abstract
from arXiv · showhide
Humans organize knowledge into a taxonomy of concepts with nested levels of abstraction and a \emph{basic level} at which people recognize and name objects with the least cognitive effort. Cobweb is a classic cognitive account of this ability, an incremental learner that builds a probabilistic concept hierarchy by maximizing category utility. We argue that diffusion models, although designed for image synthesis, implicitly perform the same computation. The noisy marginals of a diffusion model are Gaussian smoothings of the data distribution, and the modes of these marginals form a hierarchy that corresponds to a Cobweb tree of probabilistic prototypes in four respects. Both are hierarchical density models, both are hierarchical-Bayesian models with Gaussian prototypes, both treat categorization as score-following that reduces uncertainty, and in both a basic level emerges. We locate this basic level for a diffusion model at an intermediate noise level, where recent analyses show that the reverse process commits to the class identity of a sample. The two models differ mainly in how they represent and learn the taxonomy. Cobweb learns a discrete tree incrementally, whereas a diffusion model encodes a continuous, interpolable hierarchy in a single learned score field fit to the data distribution. We test the correspondence on MNIST and Fashion-MNIST by recovering the diffusion hierarchy through mode-finding and comparing the basic levels of the two models. This reframes diffusion as a cognitive model of concept formation and offers Cobweb a continuous, scalable instantiation.
1. Introduction
The introduction frames explaining the human basic level as a computational challenge and proposes that diffusion models’ smoothed densities implicitly form a concept hierarchy. It presents formal correspondences with Cobweb and an empirical test on MNIST and Fashion-MNIST.
1. Introduction
- The basic level balances informativeness about an instance’s features with distinctiveness from other categories, while supporting recognition with least cognitive effort.The introduction says a computational account of categorization should explain why this level emerges.Source: §1 · p. 1
- Cobweb incrementally builds a probabilistic concept hierarchy by choosing restructuring operations that improve category utility.The introduction notes that the resulting tree reproduces basic-level and typicality effects.Source: §1 · p. 1
- Diffusion models learn to reverse gradual noising and are usually studied as image samplers, but their progressively smoothed data densities can represent different abstraction levels.The paper argues that this family of densities has the structure of a concept hierarchy.Source: §1 · p. 2
- Diffusion hierarchies recovered by mode-finding align with Cobweb hierarchies and basic levels on MNIST and Fashion-MNIST.The paper proposes this as an empirical validation of their correspondence.Source: §1 · p. 2
2. Background
The background introduces Cobweb’s incremental hierarchy of probabilistic concepts, diffusion models’ noise-indexed smoothed densities, and related hierarchical clustering approaches. It motivates treating diffusion’s marginal modes as an implicit hierarchy comparable to Cobweb’s tree.
2.1 Concept Formation with Cobweb
- Cobweb incrementally routes instances through a probabilistic concept tree and chooses restructuring operations by category utility, which measures expected reduction in feature uncertainty.Its continuous-attribute version represents features with Gaussian means and variances; category utility favors internally predictable clusters that capture variation in the parent.Source: §2.1 · pp. 2–3
2.2 Diffusion Models
- Diffusion corrupts data with Gaussian noise and learns a reverse process; each noise level smooths the data distribution at a different bandwidth.The score locates marginal modes, and the noise level determines abstraction: small levels yield near-instance prototypes, while large levels yield broad class prototypes.Source: §2.2 · pp. 3–4
2.3 Hierarchical Clustering
- Hierarchical clustering includes tree-based latent-variable models and density-based methods that treat clusters as modes and assign points to attraction basins.The paper connects these traditions to diffusion: its smoothed marginal modes form centroids across noise levels, recovered by mode-seeking on the learned score.Source: §2.3 · pp. 4–5
3. Diffusion Models as Hierarchical Concept Formation
The paper maps diffusion’s noise-indexed modes onto Cobweb’s probabilistic hierarchy, then uses concept distinctiveness to locate an intermediate basic level. It closes by contrasting Cobweb’s explicit incremental tree with diffusion’s continuous score-field hierarchy.
3.1 Both Are Hierarchical Density Models
- As noise grows, diffusion modes merge from instance-level to broader class-level prototypes, forming a hierarchy that closely matches Cobweb’s tree on the same data.Source: §3.1 · p. 6
3.2 Both Are Hierarchical Bayes with Gaussian Prototypes
- Both models represent concepts with local Gaussian prototypes, while broader concepts pool or combine narrower ones into a hierarchy of densities.Source: §3.2 · pp. 6–7
3.3 Both Maximize Information About the Data
- Cobweb’s category utility and diffusion’s denoising objective both favor retaining information about the data, and each model searches from coarse structure toward finer detail.Source: §3.3 · p. 7
3.4 A Basic Level of Greatest Distinctiveness
- Distinctiveness D(c), the expected pointwise mutual information of a concept’s members, is the per-concept quantity used to locate the basic level.Source: §3.4 · p. 8
- For each datum, the basic-level concept is the most distinctive along its path; in diffusion, its level is where held-out log-likelihood peaks at an intermediate noise level.Source: §3.4 · pp. 8–9
3.5 Discrete versus Continuous Hierarchies
- Cobweb learns a discrete tree incrementally, whereas diffusion encodes a continuous hierarchy in one score field, allowing interpolable prototypes but requiring mode-finding to recover the hierarchy.Source: §3.5 · pp. 9–10
4. Validating the Connection
The experiments recover a diffusion hierarchy structurally similar to Cobweb’s and find corresponding basic levels on MNIST and Fashion-MNIST.
4. Validating the Connection
- Both models were fit on training images from MNIST and Fashion-MNIST, and their hierarchies and basic levels were evaluated using held-out test images.Source: §4 · p. 10
4.1 Does Mode-Finding Recover a Cobweb-Like Hierarchy?
- Score-based mode-finding builds a bottom-up taxonomy by grouping fine modes and merging groups into parents as noise rises, qualitatively resembling Cobweb trees.Source: §4.1 · p. 10, Figure 1 · p. 11
4.2 Does the Diffusion Basic Level Match Cobweb’s?
- Distinctiveness D peaks at Cobweb depth three and diffusion timestep near 150 on both datasets, yielding an interior basic level rather than the root or leaves.Source: §4.2 · pp. 10, 12, Figure 2 · p. 12
- At these peaks, both models yield object-level prototypes coarser than exemplars but finer than the dataset, with matched concepts aligning.Source: §4.2 · p. 12, Figure 3 · p. 13
- Matched-pair KL divergence was 0.45 nats on MNIST and 0.42 on Fashion-MNIST, versus 1.5–2.1 nats under shuffled pairing; size-weighted values were 0.36 and 0.28.Source: §4.2 · p. 13
5. Discussion and Conclusion
The paper reports that diffusion and Cobweb basic-level concepts align on MNIST and Fashion-MNIST, and argues that diffusion can connect cognitive theories to deep learning and support study of symbol acquisition. Its hierarchy is limited by timestep indexing, which does not map evenly onto levels of abstraction.
Discussion and Conclusion
- Matched basic-level concepts diverge three to seven times less than chance on MNIST and Fashion-MNIST, supporting alignment between the models.Figure 4 compares class-profile divergence for matched concepts with shuffled pairings.Source: §5 · p. 13, Figure 4 · p. 14
- Diffusion modes and noise levels give prototypes, graded typicality, and the basic level computational analogs that can support tests of categorization on naturalistic stimuli.The paper also frames those results as cognitive interpretations of what diffusion models have learned.Source: §5 · p. 14
- Mode-finding extracts discrete, stable prototypes from continuous densities, offering a proof of concept for studying symbol acquisition in neural models.The paper proposes investigating how words come to pick out density modes and at which hierarchy level words attach most readily.Source: §5 · p. 14
- Because diffusion hierarchies inherit timestep schedules tuned for sample quality, abstraction ranges are resolved unevenly and the basic level is reported in cognitively uninformative units.The authors suggest schedules based on human abstraction levels or equal increments of category utility.Source: §5 · pp. 14–15