PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Classifier Two-Sample Tests

David Lopez-Paz, Maxime Oquab

arXiv:1610.06545v4stat.ML

TL;DR

Two-sample tests need to detect whether samples come from the same distribution, despite limitations of traditional fixed-representation and low-dimensional methods. The paper develops classifier two-sample tests, evaluates their theory and performance, and applies them to GAN evaluation and causal discovery. C2ST achieve strong reported performance while offering interpretable statistics, learned representations, and simple asymptotic distributions.

  • Problem

    Existing two-sample tests often apply only to one-dimensional data or require a fixed data representation, motivating a relatively unexplored classifier-based strategy.

  • Method

    C2ST pair samples from P and Q with binary labels, train a classifier, and use held-out classification accuracy and its null distribution as the test statistic.

  • Results

    C2ST-NN achieves maximum power on NIPS article comparisons while controlling type-I error at α, and is reported as the most efficient test in the described dependence experiments.

  • Takeaways & Limitations

    C2ST provide a flexible approach for evaluating and comparing generative models and support applications such as causal discovery.

  • Takeaways & Limitations

    The paper leaves optimal power for quadratic-time C2ST in high-dimensional problems, where power bounds depend on dimension, for future work.

Abstract

from arXiv · show

The goal of two-sample tests is to assess whether two samples, $S_P \sim P^n$ and $S_Q \sim Q^m$, are drawn from the same distribution. Perhaps intriguingly, one relatively unexplored method to build two-sample tests is the use of binary classifiers. In particular, construct a dataset by pairing the $n$ examples in $S_P$ with a positive label, and by pairing the $m$ examples in $S_Q$ with a negative label. If the null hypothesis "$P = Q$" is true, then the classification accuracy of a binary classifier on a held-out subset of this dataset should remain near chance-level. As we will show, such Classifier Two-Sample Tests (C2ST) learn a suitable representation of the data on the fly, return test statistics in interpretable units, have a simple null distribution, and their predictive uncertainty allow to interpret where $P$ and $Q$ differ. The goal of this paper is to establish the properties, performance, and uses of C2ST. First, we analyze their main theoretical properties. Second, we compare their performance against a variety of state-of-the-art alternatives. Third, we propose their use to evaluate the sample quality of generative models with intractable likelihoods, such as Generative Adversarial Networks (GANs). Fourth, we showcase the novel application of GANs together with C2ST for causal discovery.

1 INTRODUCTION

The paper addresses limitations of conventional two-sample tests by developing classifier-based tests that learn representations from data. It establishes C2ST theory, evaluates performance, and applies C2ST to generative-model evaluation and causal discovery.

  • Two-sample testing assesses whether samples from P and Q share the same distribution, but many existing tests are limited to one-dimensional data or fixed representations.These limitations motivate a classifier-based alternative.
  • C2ST trains a binary classifier to distinguish examples from P and Q, using held-out accuracy to test whether the distributions differ.Under P = Q, accuracy should remain near chance; under P ≠ Q, successful discrimination should raise accuracy above chance.
  • C2ST learn representations on the fly, produce interpretable test statistics, have simple asymptotic distributions, and support interpretation of distributional differences.The learned features and predictive uncertainty help indicate how P and Q differ.
  • The paper analyzes C2ST properties, evaluates them against state-of-the-art alternatives, and illustrates their interpretability on synthetic and real data.The evaluation covers theoretical properties, testing performance, and interpretation.
  • The paper proposes C2ST for evaluating generative models with intractable likelihoods, including GANs, and combines C2ST with GANs for causal discovery.These applications extend classifier-based testing beyond the core two-sample problem.

2 TWO-SAMPLE TESTING

Two-sample tests convert differences between independently and identically distributed samples into a statistic for deciding whether P = Q. They support hypothesis testing, dependence testing, and evaluation of generative-model samples, while classical tests are often restricted to one-dimensional settings.

  • Two-sample tests determine whether samples from P and Q were drawn from the same distribution by testing the null hypothesis P = Q.Rejecting the null favors the alternative P ≠ Q.
  • A two-sample test summarizes sample differences in a real-valued statistic, with small values associated with the null and large values with the alternative.This statistic provides the basis for accepting or rejecting the null hypothesis.
  • The testing procedure chooses significance level α, computes the statistic and p-value, then rejects H0 when p̂ < α.Otherwise, it accepts H0 in the paper’s terminology.
  • Type-I errors reject a true null, whereas type-II errors accept a false null; within a fixed α, higher power π = 1 − β is preferred.The significance level bounds type-I error probability, while power measures the complement of type-II error probability.
  • Two-sample tests can test independence by comparing P(X, Y) with P(X)P(Y), although specialized independence tests may have higher power.The permutation construction is consistent when all possible random permutations are considered.
  • They can also evaluate generative models with intractable likelihoods by comparing generated samples with real samples using a two-sample statistic.The statistic measures the fidelity of samples produced by the generative model.
  • Classical tests such as t-tests, Wilcoxon-Mann-Whitney tests, and Kolmogorov-Smirnov tests are efficient mainly for one-dimensional data, motivating kernel methods and classifiers.The section presents binary classifiers as a relatively unexplored strategy for overcoming these issues.

3 CLASSIFIER TWO-SAMPLE TESTS (C2ST)

C2ST constructs a labeled dataset from two samples, trains a binary classifier, and uses held-out accuracy to test whether their distributions differ. Its null behavior, power, interpretability, and classifier-flexibility trade-offs are analyzed.

  • 3 CLASSIFIER TWO-SAMPLE TESTS (C2ST): C2ST shuffles and splits the labeled examples into disjoint training and testing sets, trains a binary classifier, and returns held-out classification accuracy as the test statistic.The classifier estimates p(l_i = 1|z_i) from training data, while accuracy is computed on Dte.
  • 3.1 NULL AND ALTERNATIVE DISTRIBUTIONS: Under H0: P = Q, classification is impossible and nte t_hat follows a Binomial distribution with success probability 1/2, enabling a simple normal approximation for large test sets.The approximation uses N(1/2, 1/(4nte)).
  • 3.1 NULL AND ALTERNATIVE DISTRIBUTIONS: Under H1: P ≠ Q, the statistic follows a Poisson Binomial distribution because test-example correctness probabilities may differ across examples.The paper approximates this distribution with a Binomial(n, p̄) distribution and then applies a central-limit approximation.
  • 3.2 TESTING POWER: Theorem 1 analyzes C2ST power through classifier accuracy, test-set size, effect size, significance level, and the normal cumulative distribution function.The theorem's bound is stated under expected accuracies for the null and alternative hypotheses.
  • 3.2 TESTING POWER: C2ST power trades off classifier accuracy against test-set size: flexible classifiers can detect more nonlinear patterns but require more training data.The resulting balance depends on classifier capacity measures such as VC-dimension.
  • 3.3 INTERPRETABILITY: C2ST predictions identify which examples distinguish the samples, classifier features indicate important differences, and accuracy reports the percentage correctly distinguishable.These outputs complement p-values with predictive uncertainty, feature-level explanations, and interpretable test-statistic units.

4 EXPERIMENTS ON TWO-SAMPLE TESTING

The experiments assess C2ST variants against classical and state-of-the-art two-sample tests on synthetic, dependence, text, and facial-expression tasks. C2ST methods control type-I error and show competitive or task-specific performance across these settings.

  • Experimental setup: C2ST-NN and C2ST-KNN are evaluated alongside classical tests and modern alternatives across synthetic, dependence, text, and facial-expression problems.The study uses neural-network and k-nearest-neighbour classifier tests, with comparisons varying by data dimensionality and task.
  • Control of type-I errors: The evaluated tests keep type-I error at or below the prescribed significance level α = 0.05 across sample sizes from 25 to 10,000.This result is based on 100 random repetitions under a true null hypothesis.
  • Gaussian versus Student: In Gaussian-versus-Student testing, Kuiper performs best, while C2ST-NN and ME closely follow; Wilcoxon-Mann-Whitney performs worst.The distributions are shifted and scaled to zero mean and unit variance, with performance evaluated through type-II errors as n or ν varies.
  • Independence testing on sinusoids: For sinusoidal independence testing, C2ST-NN is most sample-efficient, C2ST-KNN most robust to high-frequency variation, and C2ST-NN and ME most robust to additive noise.The task compares samples from the joint distribution with samples formed using a random permutation.
  • Distinguishing between NIPS articles: The NIPS-article experiment reports type-I errors and powers for distinguishing Bayesian, neuroscience, deep-learning, and learning-theory categories.The evaluation covers 5,903 NIPS articles published from 1988 to 2015 at significance level α = 0.01.
  • Distinguishing facial expressions: C2ST-NN achieves near-optimal power on facial-expression discrimination, only marginally behind the perfect results of SCF-full and MMD-quad.The comparison distinguishes positive from negative expressions at significance level α = 0.01 over 500 trials.

5 EXPERIMENTS ON GENERATIVE ADVERSARIAL NETWORK EVALUATION

The paper evaluates GAN samples with classifier-based two-sample tests because GAN likelihoods are intractable, and uses classifier outputs and learned features to interpret sample differences. The experiments expose pixel-level artifacts, propose safeguards for comparing models and overfitting, and illustrate interpretable distributional differences.

  • GAN formulation: GAN training alternates discriminator and generator losses, with the discriminator functioning as a C2ST distinguishing real data from generator-produced samples.The formalization permits an arbitrary binary-classification loss that influences the divergence minimized by the adversarial game.
  • GAN evaluation: Two-sample tests provide a way to evaluate GAN sample quality when likelihood evaluation is intractable but sampling is available.The procedure withholds real data and compares it with an equal amount of synthesized data using a fresh two-sample test.
  • GAN experiments: Pixel-based tests achieve near-perfect accuracy across DCGANs regardless of visual quality because checkerboard-like artifacts distinguish fake from real images.The result motivates feature-based evaluation using a ResNet-34 pretrained on ImageNet to focus on natural image statistics.
  • GAN evaluation: C2ST comparisons should use similar, small statistics on both train and test samples, with the smallest generative model preferred when this condition is met.This recipe is proposed for evaluating model quality and possible overfitting.
  • Limitations and refinements: Different-quality GANs can share the same perfect C2ST statistic, so the paper recommends finite-norm margin classifiers or the full area under the C2ST training curve for finer comparisons.The recommended curve statistic may be computed on train-set or test-set samples.
  • Interpretability: C2ST predictive uncertainty identifies where distributions agree or differ, including captured or dropped modes and peaks distinguishing Gaussian from Student’s t samples.Figure 2 also presents real and fake examples selected by uncertainty or certainty, while learned features expose discriminative facial regions.

6 EXPERIMENTS ON CONDITIONAL GANS FOR CAUSAL DISCOVERY

The paper applies conditional GANs and C2STs to cause-effect discovery, comparing synthetic data generated in both directions to infer the simpler causal direction. This approach addresses limitations of additive-noise models and is evaluated on the Tübingen dataset.

  • Problem: Cause-effect discovery asks whether X causes Y or Y causes X from samples of their joint distribution.
  • Related work: Additive Noise Models analyze regression residuals in both directions but assume independent additive noise, which can be too simplistic for heteroskedastic or multiplicative noise.
  • Method: Conditional GANs model the cause as conditioning input, noise as an independent input, and the effect as the generated variable, allowing arbitrary interactions g(X, N).
  • Method: The method trains CGANs in both directions, generates synthetic datasets, and compares each with the real data using a two-sample test statistic.
  • Decision rule: The causal direction is selected when its synthetic dataset has the smaller two-sample test statistic, implementing an Occam’s-razor preference for simpler conditional generative modeling.
  • Experiment: The procedure is evaluated on 99 Tübingen cause-effect pairs, with an ensemble variant training 100 CGANs and comparing their top generators using C2ST-KNN.
  • Experiment: Ensembling reflects the unstable behavior of generative adversarial training while highlighting its promise for causal discovery.

7 CONCLUSION

The paper concludes that modern binary classifiers can serve as powerful two-sample tests with useful theoretical, practical, and interpretive properties. It also identifies applications in generative-model evaluation and causal discovery.

  • Modern binary classifiers can be readily turned into powerful two-sample tests.
  • C2STs learn representations on the fly, have simple asymptotic distributions, and provide interpretable test statistics and differences between samples.
  • C2STs provide a flexible and scalable approach for evaluating and comparing generative models such as GANs.
  • Combining GANs with C2STs opens applications such as causal discovery.

A RESULTS ON EVALUATION OF GENERATIVE ADVERSARIAL NETWORKS

The appendix reports GAN evaluation results on the LSUN and LFW datasets across training settings and several two-sample test statistics. Lower statistics indicate better generated samples.

  • The LSUN results vary training epochs, discriminator filters, generator filters, and the MMD, C2ST-KNN, and C2ST-NN test statistics.
  • For both datasets, a lower test statistic estimates that the GAN produces better samples.
  • The LFW results use the same training-setting variables and compare MMD, C2ST-KNN, and C2ST-NN test statistics.

B PROOF OF THEOREM 1

The proof derives the C2ST test’s type-II error and power from its null-rejection threshold. The displayed expressions connect the threshold choice to the resulting power calculation.

  • The proof identifies the threshold under which the null hypothesis is accepted and then expresses the probability of a type-II error.
  • The displayed equation encodes the test-power calculation through the threshold and significance-level terms used in the proof.
  • The test power is obtained as one minus the type-II error probability.
Loading 1610.06545v4…