PaperlayerView exact revision on arXiv

Source-linked AI summary

Finite-Sample Sensitivity of Kernel Metrics for Generative Evaluation

Tomasz Radzikowski, Mateusz Modrzejewski

arXiv:2610.34477v1cs.LGcs.SD

TL;DR

Kernel distances use fixed kernels, so their reported values do not identify which changes they detect at a given sample budget. The paper tests default KID and KAD configurations on real music, audio, and image embeddings, finding that both detect shifts but rarely detect rotations at 780 samples per side. It finds that sensitivity-restoring configurations depend on the embedding and recommends reporting the sample budget and detected changes.

  • Problem

    A reported kernel-distance value does not show which distributional changes its fixed kernel detects at the evaluation’s sample budget, although sensitivity depends on kernel, embedding geometry, and sample count.

    Source: §1 · p. 1

  • Method

    The authors run default KID and KAD configurations as permutation two-sample tests on real embeddings with controlled changes matched in transport distance.

    Source: §3 · p. 3

  • Results

    At 780 samples per side, both defaults rarely detect rotations but detect matched mean shifts in every run; the median heuristic detects contractions while the cubic kernel misses most.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors recommend reporting a kernel distance’s sample budget and the changes its configuration detects at that budget.

    Source: §8 · p. 8

    The experiments measure sensitivity to constructed changes in selected embedding spaces, not specific generator failures or perceptual quality, and two spaces share an encoder.

    Source: §8 · p. 7

Abstract

from arXiv · show

Kernel distances such as the Kernel Inception Distance (KID) and the Kernel Audio Distance (KAD) compare generated and reference samples in an embedding space, and model comparisons rest on their values. Each ships with one fixed kernel, and a reported value does not say which changes that kernel detects at the sample size used. We run each default as a permutation two-sample test on real embeddings of symbolic music, audio and images, comparing a sample with an independent sample changed by a mean shift, an isotropic contraction, or a rotation that keeps the mean and the covariance trace and is matched to the shift in transport distance. At 780 samples per side, both defaults rarely detect the rotations in five evaluation spaces, while they detect the shifts in every run. The median-bandwidth Gaussian kernel of KAD detects the contraction and regains power against rotations on image embeddings with an order of magnitude more samples. The cubic kernel of KID contains second- and third-moment information but misses the contraction in most runs, and our experiments do not establish a high-power rotation regime for it over the tested sample sizes. Narrower Gaussian bandwidths restore sensitivity in music and audio, an $\ell_1$ Laplace kernel does so for images, and aggregated tests combine both at nominal level. We recommend that a reported kernel distance state its sample budget and the changes its configuration detects at that budget.

1 INTRODUCTION

The paper asks what changes the shipped KID and KAD kernels detect at an evaluation’s sample budget, since sensitivity depends on the kernel, embedding geometry, and sample count. It measures that sensitivity on real embeddings and finds distinct finite-sample behavior across defaults and kernel configurations.

Approach

  • The study tests controlled changes to independent samples from five real embedding spaces using permutation two-sample tests, matching change strengths in 2-Wasserstein distance.The tested changes include mean shifts, isotropic contractions, mean- and trace-preserving rotations, low-rank changes, and coverage losses; sample sizes range from 195 to 20,000 per side.Source: §1 · pp. 1–2

Main findings

  • At 780 samples per side, both defaults detect matched mean shifts in every run but rarely detect rotations; the median heuristic detects contractions while the cubic kernel misses most.On image embeddings, the median heuristic detects rotations in every run from 10,000 samples per side, while the cubic kernel does so in 2 of 5 runs at 20,000.Source: §1 · p. 2

Kernel behavior

  • The kernel’s retained information helps explain the results: wide-bandwidth Gaussian MMD emphasizes mean differences, while the cubic kernel includes mean, second-moment, and third-moment terms.The cubic kernel’s second-moment term is weighted by an additional factor of 1/d, and its observed weakness concerns finite-sample signal-to-noise rather than absence of covariance information.Source: §1 · p. 2

Practical implications

  • Sensitivity-restoring configurations vary by embedding: narrower Gaussian bandwidths work in music and audio, while an ℓ1 exponential kernel reaches high power on image features.The paper’s contributions also include separating sample-size, bandwidth, and kernel-family effects and proposing a reporting protocol.Source: §1 · p. 2

Scope

  • The results audit statistical sensitivity in chosen embedding spaces, not perceptual importance or model quality; larger transport distance and higher rejection power do not establish either.The authors frame the work as an audit of metric configurations and sample budgets rather than a comparison of perceptual quality measures.Source: §1 · p. 2

2 RELATED WORK

Prior work studies embedding distances, kernel-test power, and diversity diagnostics, but the paper focuses on which default KID and KAD configurations detect constructed distributional changes at particular sample sizes.

Evaluation metrics

  • FID, FAD, and FMD compare Gaussian fits, whereas KID, KAD, and CMMD use MMD with fixed kernels and embeddings.The paper situates its question within metric use for ranking models and assessing sample quality or diversity.Source: §2 · p. 2

Positioning

  • Prior metric studies vary data, dimensions, bandwidths, or desiderata; this work applies controlled alternatives to neural embeddings and shipped defaults across music, audio, and image domains.Concurrent CIFAR-10 work also finds KID weak against dispersion changes, while this study includes KAD and tests up to 20,000 samples per side.Source: §2 · p. 2

Kernel two-sample testing

  • Kernel-test theory establishes dependence of power on kernel, dimension, and sample regime, with ℓ1-distance tests capturing marginal-distribution differences in settings where ℓ2 tests target means and variances.Prior work also develops aggregated tests and kernel-fusion approaches; this paper evaluates MMDAgg on evaluation embeddings.Source: §2 · p. 3

Diversity diagnostics

  • The Vendi score measures diversity as exponentiated entropy of a similarity-kernel spectrum, while the paper’s spectral summaries apply that functional to normalized embedding-covariance spectra.The normalization is relative to the reference corpus.Source: §2 · p. 3

3 EXPERIMENTAL PROTOCOL

The protocol applies controlled transformations to independent samples from five real evaluation spaces and compares kernel configurations with permutation tests. It matches rotation and shift strength using transport distance while noting that equal distance does not imply equal statistical difficulty or perceptual severity.

Embedding spaces

  • Five evaluation spaces use CLaMP 2, VGGish, PANNs, or InceptionV3 embeddings; the two image spaces share an encoder.The protocol also includes an out-of-domain symbolic encoder only in a later analysis.Source: §3 · p. 3

Alternatives

  • Each alternative transforms one of two disjoint samples; the protocol includes mean shifts, mean- and trace-preserving rotations, isotropic contractions, low-rank changes, and coverage losses.Unless otherwise noted, each side has 780 samples, and weak and strong alternatives use s=0.1 and s=0.5.Source: §3 · p. 3

Matching by transport distance

  • Rotation strength is matched to the mean shift using a Gaussian-approximation 2-Wasserstein bound, so the rotation is at least as far in transport distance as the shift.At s=0.1, the rotation bound is 1.07 to 1.37 times the shift distance; equal transport distance does not imply equal statistical difficulty or perceptual severity.Source: §3 · p. 3

Configurations and tests

  • Single-kernel tests use unbiased MMD2 and 500 label permutations at level 0.05; the protocol also tests MMDAgg and Fréchet distances.Cells generally use 20 independent draws, with some extended to 50; intervals for extended cells are descriptive, and draws resample fixed corpora. Figure 1 plots rejection rates against bandwidth multiples at n=780 per side.Source: §3 · p. 3, Figure 1 · p. 4

Centering caveat

  • The real-pair arms center each sample once before permutation, making their rejection counts descriptive rather than calibrated power estimates for differences beyond an unknown location shift.For the linear MMD part, the observed centered split minimizes the statistic over equal-size splits because permuted splits generally have nonzero mean differences.Source: §3 · pp. 3–4

4 LOCATION, SCALE AND ORIENTATION AT 780 SAMPLES PER SIDE

At 780 samples per side, default kernels detect mean shifts but rarely detect matched rotations, and the two defaults differ on contraction. Results for low-rank changes and coverage loss depend on the direction or construction of the change.

4.1 THE DEFAULTS DETECT MEAN SHIFTS AND RARELY DETECT ROTATIONS

  • At the median heuristic, both defaults detect matched mean shifts in every run but detect the weak rotation in at most 4 of 20 runs in every space.The rotation moves KID and KAD by less than one standard deviation of the estimated noise floor, whereas the shift moves them by nine to twenty-nine; equal-transport matching leaves the result unchanged.Source: §4.1 · pp. 4–5

4.2 CONTRACTION SEPARATES THE TWO DEFAULTS

  • For a matched-distance isotropic contraction, the median heuristic detects every run in all five spaces, while the cubic kernel detects at most 6 of 20.The same difference appears for intra-mode collapse: every tested configuration except the cubic kernel detects it.Source: §4.2 · p. 5

4.3 LOW-RANK CHANGES

  • The cubic kernel has low rejection rates for four tested low-rank alternatives, while the median heuristic is more sensitive to rank-one rescalings than rotations.Window bandwidth and MMDAgg detect rotations in music and audio, but rank-two rotation is hard for every configuration on Inception; random-direction controls are detected in at most 2 of 20 runs.Source: §4.3 · p. 5

4.4 COVERAGE LOSS

  • Defaults detect removal of half the largest k-means mode through its mean displacement, but miss the remaining difference when that displacement is subtracted.Resampling replacements introduces duplicates that alone trigger window and MMDAgg in every run; with a disjoint reserve, detection ranges from 0 of 20 to 50 of 50 for MMDAgg across VGGish and PANNs, and least-displacing mode drops have low rejection under all tested configurations.Source: §4.4 · p. 5

5 SAMPLE SIZE AND KERNEL FAMILY

The median heuristic regains rotation power on image features at larger sample sizes, while the kernel configuration that restores sensitivity varies by embedding. Aggregation combines kernel families, with measurable power and computational trade-offs.

5.1 THE MEDIAN HEURISTIC REGAINS POWER WITH MORE SAMPLES

  • 3 of 20 runs reject the matched rotation at 780 samples per side on CIFAR-10 Inception features; the median heuristic rejects it in every run from 10,000.Figure 2a shows rejection rates against sample size per side; its points use 20 draws through 5,000 samples, 10 at 10,000, and 5 at 20,000.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 THE KERNEL THAT RESTORES POWER DEPENDS ON THE EMBEDDING

  • In music and audio, a fixed narrow Gaussian bandwidth is enough, and Gaussian-only aggregation detects the rotation in at least 84% of runs in all three spaces.The result varies by embedding, rather than following a single successful bandwidth configuration across spaces.Source: §5.2 · p. 6
  • 20 of 20 runs detect the image rotation with the median-scale ℓ1 exponential kernel, compared with 5 of 20 using the same kernel on ℓ2 distances.The ℓ1 advantage disappears on principal-component truncations of the same features; among tested variations, the ablations favor the norm as its explanation.Source: §5.2 · p. 6
  • MMDAgg detects the weak rotation in 18 of 20 to 50 of 50 runs in every space, at a pooled empirical level of 13/260.At package defaults it combines Gaussian and Laplace kernels, costs about 16 times as much as a single kernel, and does not detect every alternative.Source: §5.2 · p. 6

6 WHY THE DEFAULTS DIFFER

The two kernels weight distributional information differently: the cubic kernel includes moment terms through order three, while the Gaussian kernel at wide bandwidth emphasizes mean differences. These expansions describe weighting, not finite-sample power by themselves.

6 WHY THE DEFAULTS DIFFER

  • The cubic-kernel MMD is an exact sum of mean, second-moment, and third-moment terms, with the second-moment term further weighted by 1/d.A covariance difference with equal means therefore gives positive population KID, but the full statistic’s finite-sample signal-to-noise ratio determines observed power.Source: §6 · p. 6
  • At wide bandwidth, the Gaussian MMD’s leading term is ∥∆µ∥2/σ2, while covariance differences enter at order σ−4.Across fifteen symbolic-corpus pairs, the median-bandwidth ratio of the kernel statistic to ∥∆µ∥2/σ2 is 0.64 to 0.67; the expansion does not set the sample size where this weighting stops mattering.Source: §6 · p. 7
  • Effective dimension is associated with the power regime but does not determine it: the median heuristic detects the weak rotation in 20 of 20 Adversarial-MidiBERT folk-embedding runs, but only 4 of 20 runs on 16-component Inception features.Music and audio spaces need a narrow bandwidth, while in the highest-effective-dimension image spaces the best single Gaussian bandwidth reaches about half power or less.Source: §6 · p. 7

7 IMPLICATIONS FOR GENERATIVE EVALUATION

At the tested sample sizes, nearly every configuration distinguishes the real model pairs from their data, but centered comparisons are descriptive rather than calibrated power evidence. The paper proposes separating distributional tests from other evaluation questions and calibrating each separately.

7 IMPLICATIONS FOR GENERATIVE EVALUATION

  • Every configuration detects every real model–data pair except that the window bandwidth misses the Aria pair, which the defaults and MMDAgg detect.The battery covered ten model and data comparisons in twelve pair–encoder combinations; the authors note configuration matters for smaller samples, finer model-to-model comparisons, and interpreting differences.Source: §7 · p. 7
  • Once-centered comparisons are descriptive: preprocessing treats observed and permuted labels differently, so the rejection counts do not establish calibrated comparative power.For the GPT-2-class model against its training corpus, the cubic kernel rejects in 0 of 50 runs, the median heuristic in 25 of 50, and the window bandwidth and MMDAgg in 50 of 50.Source: §7 · p. 7
  • The Fréchet test detects the weak rotation in every run in CLaMP 2 and VGGish, but its 2048-dimensional results use n=390 and are not matched-budget kernel comparisons.Like the default kernels, it rarely detects the mean-matched mode drop.Source: §7 · p. 7
  • Table 2 separates distributional testing from descriptions of location, covariance, and spectral concentration, and from checks for training-data copying.A copying check requires its own reference set, statistic, and calibration.Source: §7 · p. 7

8 LIMITATIONS AND CONCLUSION

The conclusion recommends reporting a kernel distance’s sample budget and the changes its configuration detects at that budget. The authors limit their claims to permutation power on embedding-space probes and document AI use, datasets, and reproducibility practices.

8 LIMITATIONS AND CONCLUSION

  • Sensitivity is measured in embedding space using constructed probes, not models of specific generator failures; two evaluation spaces share an encoder and draws resample fixed corpora.Once-centered real-pair analyses are descriptive, perceptual validation is out of scope, and several alternatives—including learned kernels and CMMD—were not run.Source: §8 · p. 7
  • At 780 samples per side, both defaults rarely detect mean- and covariance-trace-preserving rotations, although they detect transport-matched mean shifts in every run.The median-bandwidth Gaussian detects scale changes and regains rotation sensitivity with an order of magnitude more samples; the cubic kernel misses most contractions and has no established high-power rotation regime in the tested range.Source: §8 · pp. 7–8
  • Table 2 treats each reporting question as separate, with its own quantity and calibration.The protocol distinguishes distributional testing from location, covariance, and spectral summaries and copying checks.Source: Table 2 · p. 8

AI USE STATEMENT

  • The authors used generative AI for implementation, derivation-writing assistance, draft feedback, and other disclosed tasks, while reviewing AI-assisted work and taking responsibility for the final content.They state that AI did not generate synthetic datasets or design the methodology or experiments.Source: AI USE STATEMENT · p. 8

ETHICS STATEMENT

  • The study uses public datasets and licensed generative models, involves no human subjects or personally identifiable data, and anticipates no harms beyond evaluation methodology generally.The copying analysis is intended to support training-data accountability.Source: ETHICS STATEMENT · p. 8

REPRODUCIBILITY STATEMENT

  • Reported counts map to per-cell files recording draw seeds, and stable cell checksums make results reproducible across processes.Files and scripts are available from the authors on request; analysis runs on CPU except embedding extraction.Source: REPRODUCIBILITY STATEMENT · pp. 8–9

A.1 Wide-bandwidth degeneracy of the Gaussian MMD.

At wide bandwidth, Gaussian MMD is led by mean separation, while covariance differences enter at a smaller order. Under equal means and Gaussian distributions, the next-order term depends on covariance differences.

A.1 Wide-bandwidth degeneracy of the Gaussian MMD.

  • For distributions with finite fourth moments, Gaussian MMD’s leading wide-bandwidth term is ||∆µ||²/σ², while covariance differences enter at order σ^-4.The expansion is for the Gaussian kernel as σ² tends to infinity.Source: §A.1 · p. 11
  • With equal means and Gaussian distributions, the order-σ^-4 term is [2(∆tr Σ)² + 4||∆Σ||²_F]/(8σ⁴).The derivation uses Gaussian fourth-moment identities.Source: §A.1 · p. 11
  • Across 15 symbolic corpus pairs, the statistic-to-first-order-prediction ratio at median bandwidth was 0.64–0.67 (s.d. 0.010).The expansion’s two orders were also checked against closed-form Gaussian expectations, with agreement to 6 × 10^-5 at σ²=10^6.Source: §A.1 · p. 11

A.11 Coverage losses with small mean displacement.

Mean-preserving coverage changes distinguish the tested configurations: the median heuristic and aggregated tests detect collapse, while the cubic kernel often misses it. Sensitivity to coverage loss does not extend uniformly to changes with small mean displacement.

A.11 Coverage losses with small mean displacement.

  • The least-displacing mode drop deletes half of the mode that moves the mean least, with ∥∆ˆµ∥/r ≤ 0.016 versus 0.036–0.055 for the largest-mode drop.The intra-mode collapse instead reduces within-mode spread while preserving the overall mean to numerical zero.Source: §A.11 · p. 17
  • The collapse is detected by the median heuristic in 19–20/20 runs and by the window and MMDAgg in 20/20, but by KID’s cubic kernel in 0–2/20 at strength 0.10.The least-displacing drop has low rejection rates: 0–7/50 for all four configurations in all five spaces, MMDAgg included.Source: §A.11 · p. 17
  • Narrow-bandwidth sensitivity to coverage loss holds for the subtraction-based construction, but does not generalize to coverage losses with small mean displacement.The cubic kernel’s low rejection rates persist across constructions for a coverage failure detected by the other configurations.Source: §A.11 · p. 17
  • In Inception, collapse lowers recall from 0.85 to 0.13 and raises density to 10.1, while both mode drops leave all four quantities within 0.05 of control.The collapse changes coverage while preserving the overall mean to numerical zero; it shrinks points toward their own cluster centroids.Source: §A.11 · pp. 17, 19

A.19 Injected copies.

Injected copies and a model temperature sweep probe different properties of evaluation statistics. Copy sensitivity is not a copying test, while covariance-spectrum summaries describe concentration rather than coverage generally.

A.19 Injected copies.

  • The narrow-bandwidth statistic responds monotonically and identically to exact and lightly perturbed injected copies, but MMDAgg rejects in only 5 of 360 runs.Copying lowers cross-set distances; the response is a sensitivity to duplication, not a copying test, and a copying test needs its own reference set, statistic, and calibration.Source: §A.19 · p. 20, Table 10 · p. 20

A.20 Spectral diversity across a temperature sweep.

  • Effective dimensionality, n90, and Vendi rise with temperature across five settings, but covariance-spectrum summaries do not respond to rotations and therefore describe spectral concentration, not coverage generally.The covariance trace is not monotonic; these summaries require a d×d eigendecomposition, compared with an n×n kernel spectrum for Vendi.Source: §A.20 · p. 20, Table 11 · p. 20

Related papers

Loading 2610.34477v1…