PaperlayerView exact revision on arXiv

Source-linked AI summary

Kernel-based External Validity of Experimental Studies

Federico Matteucci, Gerhard Neumann

arXiv:2610.13625v1stat.MEcs.LGstat.ML

TL;DR

Study findings may not generalize beyond the original sample, and machine-learning results can vary with experimental design choices. The paper defines kernel external validity using MMD and kernels that encode relevant features of results, including rankings. In experiments, resampling estimates approach the target validity level with relatively small preliminary studies.

  • Problem

    Study findings may fail to hold beyond the original sample, while machine-learning results vary with design choices and existing external-validity methods are sought for nonnumeric results such as rankings.

    Source: §1 · p. 1

  • Method

    The paper defines kernel external validity as a statistic for experimental results and research questions, using MMD and kernels to encode relevant result features.

    Source: §1 · pp. 1–2, §3 · p. 4

  • Results

    With N=10, bootstrap and permutation estimates of actual validity are within 0.1 of α, and within 0.05 for N≥20.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    The framework is applied to rankings and case studies, and its resampling procedure can estimate a suitable study size from small preliminary studies.

    Source: §1 · p. 2, §4.1 · p. 6

    The framework assumes factorial designs with independently sampled random-factor levels; generalizing to unseen conditions requires a representative pool, and study-size estimation is validated on one synthetic distribution.

    Source: §5 · p. 8

Abstract

from arXiv · show

The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.

1 INTRODUCTION

External validity concerns whether study findings hold beyond the original sample, a concern sharpened in machine learning by sensitivity to design choices. The paper proposes a kernel-based framework for assessing it across result types, including rankings.

1 INTRODUCTION

  • Machine-learning results can vary with dataset counts, tuning, seeds, hardware, data perturbations, and competitors, motivating methods that also handle nonnumeric results such as rankings.The introduction situates this problem within broader concerns about whether findings generalize beyond their original samples.Source: §1 · p. 1
  • The paper applies maximum mean discrepancy (MMD) to external validity, with reproducing kernels allowing comparisons across varied result domains, including rankings.The introduction identifies rankings as prominent in experimental-result analysis and describes the work as extending MMD to external validity.Source: §1 · pp. 1–2
  • The framework formalizes experimental results and research questions, then uses a kernel-based statistic called kernel external validity (KEV) to assess external validity.The authors discuss selecting the kernel and threshold, and using KEV to help assess study size.Source: §1 · p. 2

2 BACKGROUND

The background establishes notation, models studies as experiments over conditions, and recalls MMD as a tool for comparing distributions. Its example uses rankings to express a question about consistency across conditions.

2.1 Notation

  • A ranking with ties is a total, transitive binary relation over alternatives, and the paper also introduces notation for probability measures and kernel mean embeddings.The mean embedding is defined relative to a kernel, subject to an integrability condition.Source: §2.1 · p. 2

2.2 Experimental Studies

  • An experimental study evaluates the same alternatives under varying conditions; the framework assumes a factorial design with independently selectable random-factor levels.In the checkmate example, piece positions are random conditions and the research question asks whether an LLM's ranking remains consistent across positions.Source: §2.2 · p. 3

2.3 Maximum Mean Discrepancy

  • For distributions P and Q, MMD is the RKHS norm of the difference between their kernel mean embeddings.The section also introduces MMD for empirical samples and defines MMD_n as a composition.Source: §2.3 · p. 3

3 KERNEL EXTERNAL VALIDITY

The section defines kernel external validity (KEV) as the probability that independent replications produce results within an MMD tolerance, under stated design and kernel assumptions. It then shows how kernel choice encodes the research question and how a power law can guide study-size estimation.

3 KERNEL EXTERNAL VALIDITY

  • The framework assumes bounded continuous kernels, factorial sampling, finitely many alternatives, and evaluation of every alternative under every experimental condition.The random-factor levels are sampled independently of observed results.Source: §3 · p. 4
  • KEV is the probability that independent samples of results under comparable conditions have MMD at most ε; a study is (α, ε, n)-valid when its KEV is at least α.Comparable conditions are modeled as independent draws of random-factor levels, and similar results as results whose MMD is at most ε.Source: §3 · p. 4

3.2 On the Size of a Valid Study

  • KEV is not pointwise monotonic in n for discrete result spaces, especially at small n, and stopping when the estimated n∗ first fits the current study can underestimate n∗.A safety margin is suggested as a possible mitigation but is not considered in this work.Source: §3 · p. 4, §3.2 · pp. 5–6

3 KERNEL EXTERNAL VALIDITY

  • Kernel choice determines which result features count as relevant; the framework does not require characteristic kernels because discarded features are intentional.When two outcomes have identical kernel feature maps, their MMD is zero, and the KEV is 1 for any P and ε.Source: §3 · p. 4

3.1 Kernels for Rankings

  • Borda targets consistency in one alternative’s rank, Jaccard targets the top-t alternatives, and Mallows targets consistency across rankings and is adapted to ties.The kernels are presented in an interpretable form k = g ◦ d under Assumption 7.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 On the Size of a Valid Study

  • Under Assumption 7, a stated lower bound on KEV increases with ε, n, and the expected kernel value; every study with n at least the bound’s right-hand side is valid.The paper notes that this bound is loose, motivating a tighter practical estimate of n∗.Source: §3.2 · p. 5
  • The power law relates MMD quantiles to study size, allowing log n∗ to be estimated by fitting βα and using log n∗ = 2βα − 2 log ε.The authors propose estimating quantiles from resampled data and fitting the relation to obtain an estimate of n∗.Source: §3.2 · p. 5

4 EXPERIMENTS

The experiments assess resampling-based study-size estimates, external validity in encoder and LLM case studies, and a simulated study execution. Results show that small preliminary studies can support reasonable size estimates, while validity patterns depend on the kernel and study setting.

4 EXPERIMENTS

  • The experiment section also compares bootstrap and permutation resampling, uses bootstrap estimation in both case studies, and reports a 7-minute encoder-case runtime on a consumer laptop.The stated implementation complexity is O(n^2 u nrep), where u is the number of unique results and nrep the number of resamples.Source: §4 · p. 6

4.1 Resampling Methods

  • Both bootstrap and permutation estimates yield actual validity within 0.1 of 0.95 at N = 10 and within 0.05 for N ≥20.The study fits a power law to MMD quantiles from preliminary experiments to estimate the required n*; the authors report reasonable estimates even from small preliminary studies.Source: §4.1 · p. 6

4.2 Case Study: Encoders

  • The encoder analysis treats dataset as random and model, tuning, and metric as fixed, asking whether encoder rankings or raw performances generalize across sampled datasets.It retains 32/32 alternatives and 1872/2036 conditions across 48 configurations, and examines four kernel-defined research questions.Source: §4.2 · pp. 6–7

4.2 Case Study: Encoders; 4.3 Case Study: LLMs

  • In the encoder and LLM case studies, required study size increases with α and decreases with δ, while KEV increases with sample size and δ; LLM rankings can also be tied across subtasks.For encoders, the median configuration under k_j reaches (0.95, ε(0.1), ≈15)-validity; widespread LLM task failures can produce coincident rankings and KEV near 1 for any n.Source: §4.2 · p. 7, §4.3 · pp. 7–8

4.3 Case Study: LLMs

  • The LLM analysis treats subtask as random and task and number of shots as fixed when evaluating validity across sampled subtasks.The filtered data contain 45/55 alternatives and 2269/5448 conditions across 24 configurations, with research questions covering rankings and raw performance.Source: §4.3 · p. 7

4.4 Proposed Execution of a Study

  • In the simulated study, the procedure stops at N = 20 with n* estimate 13, and later estimates stabilize at 12.The study applies the procedure in increments of 10 experiments and reports agreement with KEV estimated from all 40 available results.Source: §4.4 · p. 8

5 LIMITATIONS

The framework’s scope and estimates are bounded by its design assumptions and validation settings.

5 LIMITATIONS

  • The framework requires a factorial design with independently sampleable random-factor levels, so it does not cover active experimental designs.The estimates concern the study’s pool of experimental conditions; extending them to unseen conditions requires that pool to represent the population of interest.Source: §5 · p. 8
  • The effect of imputing missing evaluations with the worst rank on KEV is not quantified.Source: §5 · p. 8
  • The estimate of n* is validated on only one synthetic distribution.Source: §5 · p. 8
  • The implementation does not use more efficient computational methods for estimating the MMD distribution, such as Nyström approaches.Source: §5 · p. 8

AI USE STATEMENT

The authors describe their use of generative AI tools and the review of AI-assisted work.

AI USE STATEMENT

  • Generative AI tools were not used for synthetic-data generation, theoretical or conceptual development, hypotheses, methodology, experiments, or interpretation.Source: AI USE STATEMENT · p. 8
  • The authors used generative AI for literature discovery, coding, and drafting text, and reviewed all AI-assisted work.Source: AI USE STATEMENT · p. 9
  • The first author checked LLM-assisted proofs line by line, and the authors thoroughly checked and tested LLM-produced code.The code mainly concerns optimizations of preexisting code.Source: AI USE STATEMENT · p. 9

Kernel-based External Validity of Experimental Studies: Supplementary Materials

The supplementary materials provide appendices on factor classification, kernels, resampling, and proofs.

Supplementary Materials

A EXPERIMENTAL FACTORS (EXTENDED)

Experimental factors are classified by the role they play in a study, and that role depends on the study’s objectives.

A EXPERIMENTAL FACTORS (EXTENDED)

  • Factors are classified as noise, held-constant, design, or random according to their role in a study.Noise factors average out randomness; held-constant factors are fixed, design factors have a relatively small set of levels, and random factors have many levels whose variation the study aims to withstand.Source: A EXPERIMENTAL FACTORS (EXTENDED) · p. 13
  • A factor’s classification is semantic: the same factor can play different roles in studies with different objectives.Dataset may be random in a comparison intended to generalize across datasets, but held constant in a study focused on one dataset, where resamples may be random.Source: A EXPERIMENTAL FACTORS (EXTENDED) · p. 13

B.1 Kernels for Rankings

The appendix supplies positive-definiteness arguments for ranking kernels, describes a Euclidean-space RBF kernel, and gives additional resampling details for estimating study size.

B.1 Kernels for Rankings

  • The Borda kernel is positive definite by embedding rankings isometrically through their Borda counts.The proof uses the resulting distance to establish conditional positive definiteness and then positive definiteness of the exponential kernel.Source: §B.1 · pp. 13–14
  • The Jaccard kernel on rankings is positive definite because it applies the Jaccard coefficient to each ranking’s top-t alternatives.The passage identifies those alternatives as the set with rank at most t.Source: §B.1 · p. 14
  • The Mallows-style ranking kernel is positive definite because it is a Gaussian RBF kernel composed with a ranking-to-adjacency-matrix map.The proof uses the Gaussian kernel’s positive definiteness and its preservation under composition with a map.Source: §B.1 · p. 14

B.2 Kernels for Euclidean Spaces

  • The Euclidean-space RBF kernel is suited to asking whether performances are consistent.For results in [0, 1]^na, the passage gives its correspondence to Assumption 7 and notes that the experiments use γ = 1/na.Source: §B.2 · p. 14

C NOTES ON RESAMPLING

  • The resampling appendix evaluates estimators and sampling schemes to determine the minimum number of experiments needed for valid results.Source: C NOTES ON RESAMPLING · p. 14

C.1 Resampling schemes

  • Four resampling schemes are compared using disjointness and replacement choices, which affect MMD through within- and between-sample collisions.Within-sample collisions bias MMD upward, while between-sample collisions bias it downward; the table is described as comparing expected effects on estimated KEV.Source: §C.1 · p. 15, Table 2 · p. 15

C.2 Incremental vs. Fresh Sampling

Incremental and fresh sampling produce nearly coincident KEV curves in the two case studies, supporting use of the practical incremental workflow. The appendix also introduces auxiliary results beyond the main text.

C.2 Incremental vs. Fresh Sampling

  • Incremental sampling appends batches of N0 results, whereas fresh sampling independently redraws y at each step.The two schemes therefore differ in the joint law of the sequence of samples.Source: §C.2 · p. 15
  • The KEV curves nearly coincide across kernels for incremental and fresh sampling, with only a small-n Jaccard deviation in the LLM case study far below α and not affecting estimated n*.The comparison supports using incremental sampling in place of fresh sampling in practice.Source: §C.2 · p. 15, Figure 9 · p. 16

D PROOFS

  • The proofs section states that it presents auxiliary results not included in the main text.Source: D PROOFS · p. 16

D.1 Auxiliary Results

The appendix records auxiliary MMD results, including a constant-diagonal kernel lemma and a limiting-distribution result for the null case.

D.1 Auxiliary Results

  • For a kernel satisfying Assumption 7, the appendix states that its diagonal is identically 1.Source: §D.1 · p. 16
  • The appendix includes a limiting-distribution result for n MMD_n^2 under the null hypothesis that P and Q are equal.The theorem is stated for a bounded reproducing kernel and introduces the eigenvalues of an operator.Source: §D.1 · p. 17
  • Centering the kernel does not change MMD, and the appendix uses the centered kernel in its limiting-properties analysis.Source: §D.1 · p. 17

D.2 Proof of Lemma 5

These appendix proofs establish results through bounded-kernel arguments, expectation inequalities, Markov bounds, and convergence tools.

D.2 Proof of Lemma 5

  • With bounded k, MMD_m lies in [0,M], and the proof reduces Lemma 5 to comparing expected MMD powers at sample sizes n+1 and n.The stated comparison is E_P[MMD_{n+1}^{2n+2}] ≤ E_P[MMD_n^{2n}].Source: §D.2 · p. 18
  • The sample comparison uses paired feature differences Y_i; each appears in n of the U_j terms before a triangle-inequality expectation bound.Source: §D.2 · p. 18

D.3 Proof of Lemma 9

  • For Lemma 9, one implication derives E_{P⊗P}k(X,X′) ≥ g(δ) using monotonicity, convexity, Jensen’s inequality, and Assumption 7; the reverse uses Jensen, Lemmas D.1–D.2, the hypothesis, and n≥1.Source: §D.3 · p. 19

D.4 Proof of Lemma 10

  • For Lemma 10, the proof applies Markov’s inequality to MMD_n² and then Lemma D.2 to bound achieved external validity.Source: §D.4 · p. 19

D.5 Proof of Theorem 11

  • For Theorem 11, continuity of Q’s cdf supports convergence of quantile functions, and β_α = log q_α does not depend on n.Source: §D.5 · p. 19

D.6 Proof of Lemma 13

  • For Lemma 13, the proof rewrites MMD_n² as a one-sample second-order V-statistic, whose scaled form converges to a weighted sum of squared standard normals with operator (10)’s eigenvalues.Source: §D.6 · p. 19
  • The Lemma 13 argument checks integrability using bounded h and verifies first-order degeneracy through vanishing projection terms.The boundedness condition given is |h| ≤ 4k̄.Source: §D.6 · pp. 19–20

Related papers

Loading 2610.13625v1…