PaperlayerView exact revision on arXiv

Source-linked AI summary

The Vendi Score: A Diversity Evaluation Metric for Machine Learning

Dan Friedman, Adji Bousso Dieng

arXiv:2210.02410v2cs.LGcond-mat.mtrl-scistat.ML

TL;DR

Existing ML diversity metrics are often domain-specific or require references, labels, or fixed features. The paper proposes the Vendi Score, an entropy-based metric using a user-defined similarity function, and demonstrates its applicability across models, datasets, and domains while identifying interpretation limitations.

  • Problem

    Many ML diversity metrics are domain-specific or require a reference dataset, predefined labels, or discrete features, despite diversity’s importance across ML applications.

  • Method

    The Vendi Score computes the exponential of the Shannon entropy of the eigenvalues of a similarity matrix induced by a user-defined similarity function.

  • Results

    The Vendi Score was shown to apply across molecular, image, text, GAN, and benchmark-dataset settings, including revealing diversity differences among GANs that capture all labeled modes.

  • Takeaways & Limitations

    Its interpretability supported diagnosing dataset diversity and distinguishing diversity differences that mode counts alone do not reveal.

  • Takeaways & Limitations

    The score measures internal diversity rather than relation to a reference distribution, can reward random noise, and depends on the chosen similarity function.

Abstract

from arXiv · show

Diversity is an important criterion for many areas of machine learning (ML), including generative modeling and dataset curation. However, existing metrics for measuring diversity are often domain-specific and limited in flexibility. In this paper, we address the diversity evaluation problem by proposing the Vendi Score, which connects and extends ideas from ecology and quantum statistical mechanics to ML. The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of a similarity matrix. This matrix is induced by a user-defined similarity function applied to the sample to be evaluated for diversity. In taking a similarity function as input, the Vendi Score enables its user to specify any desired form of diversity. Importantly, unlike many existing metrics in ML, the Vendi Score does not require a reference dataset or distribution over samples or labels, it is therefore general and applicable to any generative model, decoding algorithm, and dataset from any domain where similarity can be defined. We showcase the Vendi Score on molecular generative modeling where we found it addresses shortcomings of the current diversity metric of choice in that domain. We also applied the Vendi Score to generative models of images and decoding algorithms of text where we found it confirms known results about diversity in those domains. Furthermore, we used the Vendi Score to measure mode collapse, a known shortcoming of generative adversarial networks (GANs). In particular, the Vendi Score revealed that even GANs that capture all the modes of a labeled dataset can be less diverse than the original dataset. Finally, the interpretability of the Vendi Score allowed us to diagnose several benchmark ML datasets for diversity, opening the door for diversity-informed data augmentation.

1 Introduction

The paper motivates diversity measurement across machine-learning applications and proposes the Vendi Score as a general, reference-free metric based on a user-specified similarity function.

  • Motivation: Diversity is sought across dataset curation, generative modeling, reinforcement learning, active learning, and decoding algorithms.The paper links diversity measurement to usefulness in critical applications such as scientific discovery.
  • Motivation: Existing ML diversity metrics are often domain-specific or limited by reference datasets, classifiers, or predefined features.These requirements constrain how diversity can be defined and evaluated across domains.
  • Contribution: The Vendi Score defines diversity using a user-specified similarity function applied directly to the evaluated sample.This design makes the metric reference-free while allowing users to choose the form of diversity being measured.
  • Method: The metric is the exponential of the entropy of the eigenvalues of a similarity matrix.The similarity matrix contains pairwise similarity scores among the evaluated elements, connecting the construction to ecological and von Neumann entropy.
  • Contribution: The paper studies the Vendi Score’s properties and demonstrates its flexibility across different machine-learning domains.The stated contribution includes formal analysis of diversity desiderata and applications beyond a single model or data type.

2 Are We Measuring Diversity Correctly in ML?

The paper reviews ML diversity metrics and argues that many require references, labels, discretized features, or domain-specific constructions, while more general metrics may miss feature correlations.

  • Reference-based metrics: Reference-based metrics measure diversity through coverage of a reference distribution or dataset and often require domain-specific embeddings.Fréchet Inception distance and precision-recall variants are examples of this evaluation paradigm.
  • Classifier-based metrics: Classifier-based metrics such as Inception score and number of modes define diversity through predefined predicted labels.These metrics require labeled data and a separate classifier.
  • Feature-based metrics: N-gram diversity evaluates the distribution of discrete features and therefore requires an explicit feature representation.This construction is common in natural-language processing.
  • Similarity-based metrics: Similarity-based metrics such as average pairwise similarity and IntDiv use pairwise scores to quantify diversity.The cited text describes text variants using BLEU-style overlap and the biological metric IntDiv.
  • Similarity-based metrics: Similarity-profile entropy metrics may fail to account for correlations between features.This limitation distinguishes them from approaches intended to capture multiple dimensions of variation jointly.
  • Summary: Overall, existing metrics can require references, labels, or domain-specific assumptions, while otherwise general metrics may still inadequately capture diversity.The paper presents these limitations as the motivation for a more broadly applicable approach.

3 Measuring Diversity with the Vendi Score

The Vendi Score measures diversity from the entropy of normalized similarity-matrix eigenvalues, yielding an interpretable effective number with formal properties and efficient computation in embedding settings.

  • 3.1 Defining the Vendi Score: The Vendi Score is defined as the exponential of the Shannon entropy of the eigenvalues of K/n.The kernel matrix K is induced by a positive semidefinite similarity function with unit self-similarity.
  • 3.1 Defining the Vendi Score: The eigenvalues of K/n are nonnegative and sum to one, so the Shannon entropy and Vendi Score are well-defined.The paper also interprets the score as the effective rank of the kernel matrix.
  • 3.3 Calculating the Vendi Score: The formulation assumes independently sampled elements, although the score can be generalized to an explicit probability distribution over the sample space.The paper also relates the construction to kernel entropy, quantum statistical mechanics, and determinantal point processes.
  • 3.2 Understanding the Vendi Score: The Vendi Score can be interpreted as the effective number of dissimilar elements in a sample.A score of m corresponds to the diversity of m completely dissimilar elements.
  • 3.2 Understanding the Vendi Score: The score accounts for correlations between features and can increase when distinct similarity dimensions describe less-correlated variation.A combined similarity function may produce a score greater than the average of the individual scores.
  • 3.2 Understanding the Vendi Score: For completely dissimilar elements the Vendi Score equals n, while for identical elements it equals 1.These are the maximum and minimum values specified by the effective-number property.
  • 3.2 Understanding the Vendi Score: For disjoint sample groups, combined diversity depends on group diversities and their relative sizes.The formulation uses the Shannon entropy of the groups’ relative sizes and a weighted geometric-mean relationship.
  • 3.3 Calculating the Vendi Score: With embedding-based inner-product similarities, exact computation scales as O(d^2n) when d ≪ n.Without embeddings, the paper notes that column-sampling methods such as Nyström can approximate the score.

4 Experiments

The experiments apply the Vendi Score across synthetic data, molecular and image generation, text decoding, GAN mode collapse, and dataset analysis. Across these settings, it captures intuitive or finer-grained diversity patterns and identifies weaknesses that existing metrics can miss.

  • Synthetic data: VS behaves consistently on synthetic mixture datasets, tracking effective modes as component count, mixture proportions, or component variance changes.With an RBF similarity kernel, VS ranges from one to five in the first two settings and increases from five to seven as per-component variance changes.
  • Molecular generation: VS identifies molecular-model weaknesses that IntDiv misses, because HMM samples contain clusters of identical or nearly identical molecules despite low average pairwise similarity.The HMM and AAE samples have similar IntDiv scores, but HMM scores much lower on VS.
  • GAN mode collapse: VS provides finer-grained GAN diversity distinctions than number of modes: Presgan is more diverse than Self-conditioned GAN, and both are less diverse than the original dataset.Both GANs capture all 1000 modes according to number of modes, while VS distinguishes their diversity levels.
  • Image generation: On image generation, VS generally agrees with existing metrics on low-resolution datasets but can diverge from quality metrics on LSUN datasets.On LSUN, the diffusion model has the highest precision and recall but lower VS, while no model matches the diversity of its original training dataset.
  • Text decoding: Text-decoding experiments rank beam search lowest, human captions highest, and diverse beam search between them as its diversity penalty increases.The human diversity score is 4.88, while higher diversity penalties increase diversity scores but lower quality scores.
  • Dataset analysis: For CIFAR-100, pixel-based VS is highest for visually varied categories such as aquarium fish and lowest for visually similar categories such as cockroach.Inception-based diversity is less straightforward and may reflect semantic differences or classification difficulty.

5 Limitations

The Vendi Score has interpretation limits tied to its reference-free design, similarity function, and computational cost.

  • Reference-free scores measure internal diversity, so high scores can also arise from random noise and should be paired with a quality metric.This limitation also applies to IntDiv and n-gram diversity.
  • The Vendi Score depends on choosing a similarity function appropriate to the application.An overly sensitive, insensitive, or biased similarity function can respectively inflate, suppress, or skew diversity scores.
  • Computing the Vendi Score can be expensive when the similarity function is not associated with low-dimensional embeddings.

6 Discussion

The paper introduces the Vendi Score as an interpretable, unsupervised, and general diversity metric based on pairwise similarity scores.

  • The Vendi Score measures diversity as a function of pairwise similarity scores and can be interpreted as the effective number of unique sample elements.
  • The metric applies across domains wherever similarity can be defined and does not require labels, a reference probability distribution, or a reference dataset.
  • Users specify the form of diversity being measured through the similarity function.
  • The authors report efficient exact computation and applications across multiple machine-learning domains and datasets.

7 Proofs

The proofs establish how the Vendi Score behaves under dissimilarity, similarity, duplication, partitioning, symmetry, and finite-sample estimation.

  • The probability-weighted Vendi Score is defined from the eigenvalues of a probability-weighted kernel matrix induced by a positive semidefinite similarity function.
  • For completely dissimilar elements with weighting distribution p, the Vendi Score equals exp H(p).
  • N completely dissimilar uniformly weighted elements have Vendi Score N, whereas completely similar elements have Vendi Score one.
  • When dissimilar groups form a partition, the kernel matrix becomes block diagonal and the entropy yields the partitioning property.
  • The score is invariant to orthonormal transformations because these preserve matrix eigenvalues and entropy symmetry.
  • The empirical Vendi Score estimator has convergence rate proportional to 1/sqrt(n), with a constant depending on the true entropy.

8 Implementation Details

The implementation evaluates GANs, image models, and captioning systems using specified datasets, sampling procedures, metrics, and tokenization choices.

  • Stacked MNIST: Stacked MNIST GANs share a DCGAN architecture, train for 50 epochs, and are evaluated every five epochs using 10,000 samples.Both models use the same 60,000-image dataset and stated training setup.
  • Images: Low-resolution image models are evaluated with 50,000 generated samples using official code and specified sampling parameters.The models include VDVAE, DenseFlow, and IDDPM on CIFAR-10 and ImageNet-64.
  • Images: Image quality and diversity metrics are computed with evaluation code from Dhariwal and Nichol, using provided references except for CIFAR-10.The resulting table values may not be directly comparable with prior reported results.
  • Text: Caption samples use a pretrained image-captioning model with beam size 5, while diverse beam search uses beam and group sizes of 10 and returns five sequences.
  • Text: Text metrics use word n-grams tokenized by the captioning models’ pretrained wordpiece tokenizer.

9 Additional Results

The Vendi Score is evaluated across controlled mode-dropping experiments, molecular and image generators, caption data, molecular odors, text datasets, and CelebA. Across these applications, results show that its interpretation depends on the chosen similarity function and can reveal diversity patterns missed by alternative metrics.

  • 9.1 Assessing Mode Dropping in Datasets: VS generally increases as more classes or genres are represented, showing a stronger correlation with the number of modes than IntDiv.In MNIST, VS rises roughly linearly for digits 0–5 before fluctuating; in MultiNLI, it increases monotonically with genres.
  • 9.2 Evaluating Molecular Generative Models for Diversity: IntDiv ranks the HMM among the most diverse molecular generators, whereas VS ranks it as the least diverse.The evaluation uses the first 2,500 valid molecules from each MOSES sample and Morgan fingerprint similarity.
  • 9.4 Evaluating Text Generative Models for Diversity: VS is highly correlated with n-gram diversity on MS-COCO captions, although the metrics sometimes rank samples differently.The differences arise because n-gram diversity conflates within-sentence n-gram diversity with diversity across sentences.
  • 9.3 Evaluating Image Generative Models for Diversity: VS can measure image diversity with either Inception representations or pixels, but the resulting scores do not always agree.The two similarity choices encode semantic versus visual similarity, so their disagreement reflects different notions of diversity.
  • 9.5 Diagnosing Datasets for Diversity: Molecular diversity scores correlate with odor-descriptor diversity, linking structural variation with differences in reported odor descriptions.Descriptors such as “green” occur across structurally varied tag sets, while “waxy” tends to co-occur with repeated tags.
  • 9.5 Diagnosing Datasets for Diversity: In MultiNLI and Amazon reviews, n-gram overlap and SimCSE assign different diversity rankings across genres and star ratings.In MultiNLI, both rank Slate highest; in Amazon reviews, 5-star reviews have highest n-gram diversity but lowest SimCSE diversity, while SimCSE ranks 3-star reviews highest.
  • 9.5 Diagnosing Datasets for Diversity: Almost all tested similarity functions indicate that CelebA is more diverse for men than for women.The comparison uses Inception embeddings, pixel vectors, and binary attribute vectors.
Loading 2210.02410v2…