PaperlayerView exact revision on arXiv

Source-linked AI summary

PTED: A multi-dimensional two-sample test for scientific inference and generative machine learning

Connor Stone

arXiv:2609.38388v1astro-ph.IMphysics.data-anstat.ML

TL;DR

Multivariate two-sample testing has many possible approaches, while the commonly used KS test is defined only for one-dimensional data. PTED implements a permutation test using energy distance; it detects over- and under-fitting in the reported tests and is versatile across settings, though it may not always be the most sensitive test.

  • Problem

    The KS test is one-dimensional, and multivariate two-sample testing presents many possible approaches despite its relevance to inference and generative modelling.

    Source: §1 · pp. 1, 3

  • Method

    PTED applies a permutation test to energy distance, whose finite-sample estimate is built from pairwise distances.

    Source: §1 · p. 3, §2.1 · pp. 3–4

  • Results

    PTED detects over-fitting in the two-moons test and, with MMD, has the highest sensitivity in the coverage test, by a small margin.

    Source: §3.2 · p. 8, §3.4 · p. 13

  • Takeaways & Limitations

    PTED works across low or high dimensions, sample sizes, and sample balance, but may not always be the most sensitive test.

    Source: §4.4 · p. 16

    Sample-based tests may not reject a single far-off outlier when the samples could simply have missed the distribution’s small distant mode.

    Source: §4.3 · p. 16

Abstract

from arXiv · show

Two-sample tests are widely applicable in inference and generative modelling, yet users frequently fall back on heuristics and visual inspection due to lack of an accessible test that operates in multiple dimensions. I present Permutation Test using the Energy Distance (PTED), a Python implementation of a powerful two-sample test described in Sz\'ekely and Rizzo [2004]. The test statistic is the energy distance, a metric on probability distributions that admits a natural sample estimator built from pairwise distances. A permutation test is run on energy distances to produce an exact two-sample test. Because the statistic depends on the data only through pairwise distances, the test applies in high dimensions, on learned feature representations, at large or small or imbalanced sample sizes, and to any data type on which a distance can be defined. An approximation to the formula allows PTED to scale linearly with both number of dimensions and samples while retaining most discriminative power. I highlight a number of instructive sensitivity tests on PTED alongside other multi-dimensional two-sample tests for comparison. PTED is the only method that is sensitive to all tests, though conventional Maximum Mean Discrepancy is identical for all but an over-fitting test.

1 Introduction

Two-sample tests support scientific inference and generative modelling, but multivariate testing offers many choices and the widely used KS test is one-dimensional. PTED implements a permutation test using energy distance and is reported to be competitive in one and high dimensions.

1 Introduction

  • Two-sample tests assess whether two data collections come from the same distribution, a question common in scientific inference and generative modelling.They are often framed as null-hypothesis tests that compute a p-value.Source: §1 · p. 1
  • Seven steps in the inference flowchart suggest a two-sample test as a validation check, while generative modelling aims to produce data from its training distribution.Simulation-based inference primarily uses samples and typically places greater emphasis on validation.Source: §1 · p. 1, Figure 1 · p. 2
  • Although the KS test is widely used, it is defined only for one-dimensional data; projection and permutation yield many valid multivariate test options.Examples include Wasserstein distance, MMD, FID, and FLD as statistics for permutation tests.Source: §1 · p. 3
  • PTED implements a permutation test on energy distance, contributes a Python implementation and linear scaling technique, and is described as competitive in one and high dimensions.The method was previously described by Székely and Rizzo [2004].Source: §1 · p. 3

2 PTED method

PTED tests equality of two distributions by applying a permutation test to the energy distance, estimated from pairwise sample distances. The method also describes landmark scaling, single-element coverage testing, and an asymmetric containment test.

2.1 Core two-sample test

  • PTED estimates squared energy distance from pairwise distances, comparing expected cross-sample distances with within-sample distances.The finite-sample estimator excludes self-pairs and is unbiased, though it may be negative on a finite sample.Source: §2.1 · p. 3
  • Under the null, PTED’s permutation test has type-I error no greater than the nominal level for any distributions and sample size.Its attainable p-values lie on a discrete grid, with minimum 1/(N + 1).Source: §2.1 · p. 5

2.2 Large sample sizes

  • With m landmark points, PTED computes an n × m distance matrix in O(nmd), retaining permutation-test validity while potentially reducing power.Permutations exchange labels within the landmark set and its complement, not between them.Source: §2.2 · p. 5

2.3 The single-element limit (coverage testing)

  • When nx = 1, PTED reduces exactly to a rank test, but its minimum p-value is 1/(ny + 1) and consistency requires nx to grow as well.For ny = 128, a single trial cannot report p < 1/129; coverage testing combines multiple trials’ p-values with Fisher’s method.Source: §2.3 · p. 6

2.4 Containment testing

  • The containment test ranks each x element relative to y and combines the resulting p-values, but is asymmetric and conservative when x is genuinely contained in y.Its composite null is exact on the F = G boundary; the PIT plot can reveal low-p-value tails that Fisher combination may miss.Source: §2.4 · pp. 6–7

3 Comparison tests

The comparisons test PTED against several two-sample methods across one-dimensional, two-moons, MNIST, and posterior-coverage settings. PTED is competitive across these tests, but sensitivity varies by deviation and method.

3 Comparison tests

  • The comparison suite includes KS, PQM, RBF-kernel MMD, FID, and FLD; some multidimensional tests project samples onto the first principal component.FID and FLD produce uncalibrated scores, for which relative values are recommended.Source: §3 · pp. 7–8

3.1 1D Gaussian sensitivity tests

  • PTED’s median p-value is more sensitive than the KS test and PQM in both one-dimensional tests, while MMD is essentially identical.FID appears highly sensitive to out-of-distribution samples, whereas FLD responds only after considerable deviation from normality.Source: §3.1 · p. 8, Figure 3 · p. 9

3.2 Two-Moons test

  • PTED’s p-values drop sharply below the 0.08 scale, detecting over-fitting; the other tests also show blind spots on clear under-fitting.PQM’s p-value spikes near the 0.08 moon width, while FLD rises through that scale and falls as over-fitting becomes clear.Source: §3.2 · p. 8, Figure 4 · p. 10

3.3 MNIST sensitivity tests

  • PTED and MMD perform nearly identically across the MNIST tests; FLD is more sensitive to white noise but misses class drops and favors more severe digit blends.Many FLD runs returned NaN values, likely due to low variance in some MNIST dimensions.Source: §3.3 · p. 10, Figure 5 · p. 11

3.4 Coverage Test

  • PTED and MMD are slightly more sensitive than the other methods to over- and under-confidence, while MIRA’s null window extends nearly to c = 2.The KS test and HPD-region test appear similarly sensitive, with peaks on opposite sides of the ideal value.Source: §3.4 · p. 13, Figure 7 · p. 13

4 Discussion and Conclusions

Across the test suite, PTED was the only method with competitive sensitivity on every test, while MMD was essentially identical in most tests. The discussion also weighs runtime and high-dimensional limitations, and concludes that PTED is broadly applicable though not always the most sensitive.

4.1 Sensitivity across the test suite

  • PTED was the only method with competitive sensitivity on all tests, while MMD was essentially identical to it in all tests except the two-moons replication test.Figure 8 compares the severity at which each method rejects; the authors note that consistency may matter more than small sensitivity gains if the observed bimodal pattern generalizes.Source: §4.1 · pp. 13–14, Figure 8 · p. 14

4.2 Runtime and scaling

  • PTED was among the fastest algorithms in each test and had the lowest runtime on CIFAR-10, but rankings may change with dimensions or sample counts.Using one sixteenth of the samples as landmarks gave a near order of magnitude speedup with almost no loss in power; eight landmarks still rejected at severity 0.35.Source: §4.2 · pp. 14–15, Figure 9 · p. 15, Table 1 · p. 15

4.3 Practical considerations and high dimensions

  • A single extreme outlier may not lead any sample-based test to reject, because it can be consistent with the other sample simply missing a rare mode.More outlier draws, or a very small y sample containing the outlier, can instead make the null probability vanishingly small.Source: §4.3 · pp. 15–16
  • In high dimensions, a small off-manifold perturbation may barely change pairwise distances, while feature-space projection can obscure failures orthogonal to the projection.The authors note that aggregation across many coordinates can dilute discrepancies confined to a few coordinates.Source: §4.3 · p. 16, §4.4 · p. 16
  • When passing a PTED test is an unattainable bar, relative p-values or energy distances can rank-order models; the energy distance is also differentiable.The authors suggest training on energy distance as a possible avenue for further exploration.Source: §4.3 · p. 16

4.4 Conclusions

  • PTED works across low and high dimensions, small or large and imbalanced samples, and arbitrary underlying distributions as an exact test, though it may not always be most sensitive.Pre-built tests include the base two-sample, coverage, and containment tests.Source: §4.4 · p. 16

Software

The software section acknowledges the tools and methods cited alongside PTED.

Software

A Extra Gaussian sensitivity tests

The additional Gaussian tests examine skewness, a rare second mode, and bimodal mixtures as deviations from a Normal distribution.

A Extra Gaussian sensitivity tests

B 1D Gaussian sensitivity test visualizations

The visualization plots the sampling distributions behind the one-dimensional Gaussian sensitivity tests and shows how their differences progress in severity.

B 1D Gaussian sensitivity test visualizations

C MNIST sensitivity tests visualizations

Figure 12 visualizes the MNIST perturbations used in Section 3.3, helping interpret the sensitivity curves. For white noise, the visualized images show noise by S = 0.25, when PTED’s p-value drops significantly.

C MNIST sensitivity tests visualizations

D CIFAR-10 sensitivity tests

On CIFAR-10, the sensitivity-test results are largely similar to MNIST, with differences in class-drop detection and FLD’s responses to pair blending and white noise.

D CIFAR-10 sensitivity tests

  • CIFAR-10 results are largely similar to MNIST, but the class-drop perturbation is harder to detect.The text suggests this may be because CIFAR-10 classes are less separated in sample space.Source: D CIFAR-10 sensitivity tests · p. 22
  • For pair blending, FLD gives lower scores—supposedly indicating better matching—for more severely mismatched samples.Source: D CIFAR-10 sensitivity tests · p. 22
  • FLD’s sensitivity to white noise is much less intense on CIFAR-10 than on MNIST.The text suggests this may be because no single dimension is zero for all examples in a batch of 2048.Source: D CIFAR-10 sensitivity tests · p. 22

E Probability Integral Transform plots

PIT plots show the empirical CDF of p-values against a uniform reference, making over- and under-confidence visible. They can reveal structure missed by an aggregate, but no combination of the reported views can establish H0.

E Probability Integral Transform plots

  • A PIT plot compares the empirical CDF of multiple p-values with the U(0,1) reference F(t) = t.A curve above the diagonal indicates excess small p-values; one below indicates excess large p-values.Source: E Probability Integral Transform plots · pp. 22–23
  • Figure 14 shows overconfident, well-calibrated, and under-confident posterior examples; curves outside the shaded 95% containment band indicate null rejection at 95% significance.Overconfidence produces a rise above the diagonal, while under-confidence produces a fall below it.Source: Figure 14 · p. 23, E Probability Integral Transform plots · p. 23
  • PIT curves can reveal calibration structure that an aggregate measure may miss, but even Fisher aggregation, PIT, and direct sample visualization cannot establish H0.A curve remaining inside the simultaneous band is consistent with H0 at the stated level; this is not proof of H0.Source: E Probability Integral Transform plots · p. 23
Loading 2609.38388v1…