What Can a Gaussian Process Design Test

1 Introduction

• The paper distinguishes retrospective detection of incompatibility from prospective testability: which departures a design can reveal, and with what power.Predictive-uncertainty locations need not be the observations that most strongly challenge model assumptions.Source: §1 · p. 2
• Moving four observations to the centre makes curvature structurally detectable; with endpoint-only inputs, GP rejection reflects an improbable intercept, not a violated curvature relation.Figure 1 compares predicted power with simulated rejection rates for the two designs and shows the prospective power of adding a candidate observation.Source: §1 · p. 2, Figure 1 · p. 2
• Finite-feature GPs have exact consistency relations, while other kernels impose soft restrictions described through the kernel spectrum.The framework connects those design-implied restrictions to power calculations before measurement; its exact theory assumes a fixed mean, kernel, and noise variance.Source: §1 · p. 3

2 Related work

• The contribution is to use classical residual, group-diagnostic, and discrimination tools prospectively to ask what a GP design can contradict.The underlying diagnostics and power criteria are not new; the paper combines them to interpret the evidence a rejection represents.Source: §1 · p. 3, §2 · p. 4

3.1 Gale duality

• Gale duality represents feature configurations through vectors in a complementary space, encoding how observations depend on one another.The feature matrix’s null space contains the relations; the Gale representation records them observation by observation.Source: §3 · p. 4, §3.1 · p. 5

3.2 Consistency relations of a model

• A consistency relation is a linear constraint satisfied by every exactly representable response at the design.The complete non-redundant set vanishes for representable responses, and its failures arise from noise or model violation.Source: §3.2 · p. 6

3.3 Residual projector, leverage and noise

• Gale-vector lengths describe residual visibility: in the planar example, 80% of a centre perturbation but only 30% of a corner perturbation remains visible.A longer Gale vector corresponds to a smaller leverage and a larger residual footprint.Source: §3.3 · pp. 6–7
• Circuits are minimal supports of exact relations, so each identifies the smallest observation set that can witness a structural contradiction.Every structural contradiction is witnessed by a circuit; for degree-q polynomials in one dimension, any q + 2 distinct inputs form one.Source: §3.3 · pp. 7–8

3.4 Structural assumptions from feature maps

• The feature map determines the structural assumption: a straight-line design needs at least three distinct inputs to test curvature, and a 2 × 2 grid yields an additivity relation.With only two distinct inputs, curvature can be interpolated by a line; in multiple dimensions, input positions also matter.Source: §3.4 · pp. 8–9

4.1 Finite-rank kernels: exact relations

• For finite-rank kernels, null(K) gives the exact relations among function values; polynomial kernels recover degree tests, and repeated inputs yield checks of the noise model.The null-space dimension counts independent checks at the design.Source: §4.1 · p. 10

4.2 Full-rank kernels: spectral coordinates

• For full-rank kernels, spectral shrinkage separates flexible response directions from soft prior restrictions, which make patterns unlikely but do not exclude them.For smooth stationary kernels, soft restrictions can resemble approximate finite differences across neighbouring inputs.Source: §4.2 · p. 11

4.3 The residual matrix in the small-noise limit

• As noise vanishes, the residual matrix converges to the exact residual projector for finite-rank kernels, but to zero for full-rank kernels.Truncating a full-rank spectrum at a positive noise level is an approximation, not an exact Gale transform.Source: §4.3 · p. 12

4.4 Hard structural relations and soft prior restrictions

• The paper reserves structural relations and Gale vectors for exact finite-rank constraints, using soft restrictions and spectral coordinates for general kernels.A full-rank GP can fit any data pattern, but patterns that its prior regards as implausible incur residual energy.Source: §4.4 · pp. 12–13

5 Group consistency tests and their power

• A departure violating an exact relation has noncentrality that grows without bound as noise decreases, while prior-based detection of a representable departure stays bounded.At the end-point design, curvature acts as an intercept shift and is detectable only through the prior.Source: §5 · p. 13, Figure 4 · p. 14

5.1 The group test

• The group test asks whether observations in a candidate group can be predicted from the remaining responses within predictive uncertainty.Exact calibration assumes fixed model parameters; fitting hyperparameters or noise from the tested data can weaken it.Source: §5.1 · pp. 14–15

5.2 Power of the group test

• The group-test statistic has a noncentral chi-square distribution under a deterministic discrepancy, and its power can be computed from the design before observing responses.Its noncentrality reflects the part of the group’s discrepancy that the remaining observations cannot predict.Source: §5.2 · p. 15

5.3 Geometric meaning

• For finite-rank kernels, the affected observations’ Gale vectors determine power through their lengths and angles; short vectors make local departures less visible.The GP and residual tests can have different degrees of freedom and therefore different thresholds, even when their noncentralities agree.Source: §5.3 · p. 16

5.4 Structured departures

• For a departure family, structural testability in every direction holds exactly when G⊤D has full column rank.The mean noncentrality and worst-case criterion depend on the scaling of D, and mean noncentrality need not rank designs like average power.Source: §5.4 · p. 17

5.5 Relation to discrimination design and Fisher information

• For finite-rank models and specified alternatives, structural noncentrality corresponds to T-optimality, whereas the full-GP divergence also penalizes representable but prior-improbable components.At the end-point design, curvature has zero projected divergence because it is representable as an intercept shift.Source: §5.5 · p. 18

6 Fitted noise as aggregate test failure

• The fitted noise variance is soft-residual energy divided by the effective number of residual degrees of freedom.The numerator aggregates failures of hard relations and soft prior restrictions; the denominator is n minus the effective number of parameters.Source: §6 · p. 18
• Discrepancy energy can inflate the fitted noise variance, which then enters predictive variance and shrinks test statistics.In experiment S1, three shifted observations among 40 raised the fitted noise variance to almost four times its true value.Source: §6 · p. 18
• Tests using fitted noise therefore tend to reject less often when the model is wrong, making non-rejection weaker evidence.Known measurement noise, replicates, or a separate reference set are preferable for calibration.Source: §6 · p. 18

7.1 Structural contradiction or prior improbability (E1)

• At the end-point design, residual-test power stays at level for every curvature; GP power reaches 0.74 at c = 5 through prior improbability, while four centre inputs detect curvature structurally.With four centre inputs, both tests have power 0.69 and 0.68 at c = 0.125; equally spaced inputs give 0.75 and 0.73.Source: Figure 5 · p. 19, §7.1 · p. 19

7.2 Choosing for power, for discrimination or for predictive variance (E1, continued)

• 0.48 to 0.72 mean power: adding the observation selected by the power rule outperforms predictive variance at 0.51 and random selection at 0.49.Averaged over 20 designs and 19 bump locations; the KL rule also gives 0.72 and chooses the same input as the power rule in 77% of cases.Source: §7.2 · p. 20, Table 2 · p. 21

7.3 Discrepancies of unknown location (E2)

• 0.52 mean power after five additions makes the average-power rule best in all 20 designs, versus 0.46 for space-filling, a paired difference of 0.059 ± 0.005.Power against the least-detectable bump stays at most 0.16 for every rule, and the average-power rule's gain shrinks when the true bump is wider or lower than assumed.Source: §7.3 · p. 22, Figure 7 · p. 22

7.4 Testing additivity in two dimensions (E3)

• 0.999 power at c = 0.1 for the power-optimised design, versus 0.81 for the grid, 0.09 for Latin hypercube, and 0.08 for random designs.The grid contains nine mixed-difference relations; the power-optimised design places inputs on two lines and contains seven relations. Simulated rejection rates match predictions within 0.002 for both designs.Source: §7.4 · pp. 22–23, Figure 8 · p. 23

8.1 What the experiments show

• At the end-point design, curvature rejection comes from an improbable prior intercept, whereas moving four observations to the centre makes curvature violate an exact consistency relation.The end-point design contains no structural information about curvature.Source: §8.1 · p. 24
• The 4 × 4 grid contains mixed-difference circuits that test additivity directly, unlike a same-size Latin hypercube; full-GP KL can be large while projected KL and T-optimality are zero.Both designs may assign low prior probability to an interaction, but only the grid provides an exact algebraic check of additivity.Source: §8.1 · p. 24
• In E1, choosing the next observation by predicted power raised average power from 0.48 to 0.72, compared with 0.51 for predictive variance.The advantage mostly remained when the lengthscale or noise level used for choosing was wrong by a factor of two; Proposition 2 tracked simulated rejection probabilities closely.Source: §8.1 · p. 24

8.2 Limits of the present analysis

• Exact calibration assumes a fixed mean, kernel, and noise variance with Gaussian homoscedastic noise; estimating hyperparameters from tested observations removes exact nominal test-level calibration.Discrepancy energy can inflate fitted noise variance and weaken the test, although E1 suggests design choice is not overly fragile to moderate hyperparameter misspecification.Source: §8.2 · p. 24
• The power calculation requires a specified discrepancy or family, so optimised designs can be specialised; E2 shows a milder gain when discrepancy location is unknown.The known-failure-mode specification can be useful when that mode is scientifically meaningful.Source: §8.2 · p. 24
• The results cover fixed designs tested once, linear consistency relations, and synthetic low-dimensional experiments; rejection indicates incompatibility, not which observation or assumption is faulty.Repeated testing needs false-rejection control, while nonlinear hypotheses require a different treatment; model repair is a separate decision.Source: §8.2 · p. 24

8.3 Implications for sequential design

• In Bayesian optimisation, an evaluation spent challenging the model is not used directly to improve the objective.Testability and usefulness for the optimisation task are distinct.Source: §8.3 · p. 24
• Sequential use remains open: designs, GPs, and structural hypotheses evolve, requiring time-uniform error control, uncertainty propagation, and a rule trading testing against optimisation.The paper also leaves open how to handle uncertainty in the discrepancy family and GP hyperparameters.Source: §8.3 · p. 25

9 Conclusion

• A GP can reject curvature at an end-point design through an improbable intercept even though the design has no consistency relation involving curvature.The departure can be rejected without the design having observed the structure supposedly violated.Source: §9 · p. 25
• For finite-rank kernels, the null space separates structural contradictions from prior-based evidence, while general kernels require a spectral rather than algebraic analysis.Gale duality identifies observation roles in structural relations and their minimal supports; the calculation is available before responses are measured.Source: §9 · p. 25
• Sequential optimisation still requires error control, uncertainty about assumed structure, and a decision rule trading model tests against objective improvement.The design, GP, and structural hypothesis all evolve during optimisation.Source: §9 · p. 25

Funding

• Ivan De Boi is funded by the Research Foundation – Flanders (FWO) through postdoctoral fellowship 1217125N.Source: Funding · p. 25

A Block deletion identities

• For indices ordered as (R, C), the Schur complement of A_RR is denoted V_C.The passage defines V_C from the blocks of A.Source: A Block deletion identities · p. 25
• Reading off the C block of α = A⁻¹y gives α_C = V⁻¹.Source: A Block deletion identities · p. 25
• T_C is the drop in the quadratic term of the log marginal likelihood when C is deleted, while the determinant identity gives the corresponding log-determinant change.Source: A Block deletion identities · p. 26

• The conditional discrepancy mean on C is ΔC, and the corresponding precision-weighted term is (Qδ)C = QCCΔC.When δR = 0, ΔC = δC; the geometric form uses the squared norm of a projection divided by σ2.Source: B Proofs · p. 26
• For a quadratic form ηC(θ) = θ⊤MCθ, its minimum over unit θ is the smallest eigenvalue of MC.Its mean under N(0, τ2Ip) is τ2 tr MC.Source: B Proofs · p. 26
• As σ2 tends to zero, Q:CQ−1 0Q converges in probability to GG⊤, and G⊤Dθ = 0 exactly when Dθ lies in col(Φ).Source: B Proofs · p. 26

• At a maximum of the marginal likelihood, differentiating with respect to noise variance yields the fixed-point identity (27), though it is not a closed-form estimator.With a weak-prior MAP fit, as used in the experiments, the identity holds only approximately.Source: C The noise fixed point · p. 26
• Discrepancy energy along hard relations or soft prior restrictions passes almost unattenuated into residuals and can raise the fitted noise variance.At fixed noise, it raises the expected numerator of (27) while leaving the denominator essentially unchanged.Source: C The noise fixed point · p. 26
• The discrepancy effect on fitted noise is not generally monotone because it depends on discrepancy direction, its cross term with f, and changes in S and the denominator.Source: C The noise fixed point · p. 26

• The power study uses 401 candidate inputs, adds one observation, and scores candidates by test power with thresholds and degrees of freedom recomputed for each candidate.The detailed example uses x0 = 0.1, and the sweep covers 20 designs with x0 from −0.9 to 0.9.Source: D Experimental details · p. 27
• The mismatch checks select designs using alternative lengthscales or noise variances, then evaluate them using the true hyperparameters.The straight-line comparison uses kernel 1 + xx′, with curvatures up to 0.25 for three designs and up to 10 for the end-point design.Source: D Experimental details · p. 27
• At five bump locations, simulated rejection rates for the average-power rule’s final design agree with predicted power to within 0.01.Source: D Experimental details · p. 27
• For the interaction experiment, Latin-hypercube and random designs have soft prior restrictions, but the interaction has little energy along them.The greedy design adds candidates to maximize noncentrality for testing all observations against x1x2.Source: D Experimental details · p. 27
• Power evaluations solve a linear system of size n for every candidate, which is cheap for the tested designs of at most 45 observations but not large designs without updates.Source: D Experimental details · p. 27

• F2I retains observations to minimize predictive variance over a finite target set, but rewarding a smaller refitted noise estimate can remove high residuals from honest noise and cause overconfidence.A smaller predicted variance is not evidence that the model has become more correct.Source: E Background material · p. 27
• At fixed kernel and noise, deleting data never reduces uncertainty; any improvement after deletion must come from re-estimating model quantities such as noise variance.Removing observations carrying discrepancy energy tends to lower the fitted noise estimate by reducing the numerator of (27) more than its denominator.Source: E Background material · p. 28
• Holding out a closely spaced group jointly reveals a shared bias that leave-one-out residuals can mask through mutual prediction.The joint statistic uses the whole QCC block and avoids the cancellation that can make individual statistics small.Source: E Background material · p. 28
• Bonferroni or Holm corrections control false rejections for a fixed family of candidate groups, while adaptive remove-and-refit searches require further calibration care.Source: E Background material · p. 28
• The group test in Gale coordinates is identical to the classical multiple-case test unless the residual matrix is approximated, for example by hard spectral truncation.Source: E Background material · p. 28
• For smooth stationary kernels and closely spaced inputs, soft relations behave like approximate higher-order finite differences, and residual rows act as local stencils.These stencils nearly annihilate constant, linear, and quadratic trends.Source: E Background material · p. 29
• Gale duality characterizes whether points lie on a convex-hull face through the convex hull of the remaining points’ Gale vectors.It is especially effective when there are only a few more points than dimensions, so the dual space is small.Source: E Background material · p. 29
• The all-monomials feature map of degree at most q and the projective Veronese map yield the same consistency relations after setting x0 = 1.Source: E Background material · p. 29

Results

• In the full GP, fitted noise variance rose to 9.8 × 10−3 in S1 and 0.22 in S2, widening target-region intervals and raising coverage to 1.00 instead of 0.95.These variances were almost four times and about ninety times the true value, respectively.Source: F Supplementary experiments: detection… · p. 31
• F2I-style deletion produced the sharpest predictions but only 9 to 14% coverage, while test-based methods kept coverage between 0.93 and 0.97 and deleted at most 3.2% of honest observations.F2I-style NLPD exceeded 100, versus −1.5 to 0.2 for every other method.Source: F Supplementary experiments: detection… · p. 31, Figure 10 · p. 33
• In S2, even the best test found only about 80% of misfit observations, yet every test-based method's target-region accuracy remained within 15% of the oracle.The rough region includes stretches where the fast oscillation happens to look smooth.Source: F Supplementary experiments: detection… · p. 31

Deletion trajectories

• At the misfit-count stopping point, median noise variance in S1 and S3 was within 2% of truth and coverage was 0.94, but F2I-style deletion continued removing honest observations.Unlike the group test, F2I-style deletion stops only when another deletion no longer lowers its predictive-variance objective.Source: F Supplementary experiments: detection… · pp. 33–34, Figure 11 · p. 35

Calibration

• On cubic-null data, any deletion occurred in 6.7–7.6% of group-test and 8.8–9.4% of single-test sets versus 5% nominal; estimated GP-null rates were 6.3–8.5%.The pipeline is approximately rather than exactly calibrated; hyperparameter estimation explains most of the excess over nominal levels.Source: F Supplementary experiments: detection… · p. 34, Table 6 · p. 34, Table 7 · p. 34

Masking

• For S3's masked batch, the single-observation test found 68% of four shifted points and had 2.4 times the target error of group tests; all tests failed when the batch lay inside A.In the inside-A variant, the reference noise estimate included the batch and was about six times the value in other S3 runs.Source: F Supplementary experiments: detection… · pp. 34–35, Figure 12 · p. 36

Gale coordinates

• In S3, Gale-coordinate and classical group tests made identical decisions in 99 of 100 runs; in S1 and S2, hard spectral truncation never favored Gale coordinates.The Gale version found 80% versus 95% of S1 misfits and left nearly four times larger noise variance in S2, with equal accuracy.Source: F Supplementary experiments: detection… · pp. 35–36, Table 9 · p. 37