PaperlayerView exact revision on arXiv

Source-linked AI summary

The Price of Correlated Tests: How Strict Should a Model Release Gate Be?

Marco Pollanen

arXiv:2610.00993v1stat.MLcs.LGstat.ME

TL;DR

Release gates must balance trustworthy clearances against the loss of acceptable models, with test dependence and pool composition shaping the choice. The paper models this trade-off and provides a labelled-data certification procedure. In one configuration, a 99% target requires 8 independent tests but 5,182 tests at latent correlation 0.5, retaining fewer than one good model in ten.

  • Problem

    A release gate must limit unacceptable clearances without unnecessarily rejecting acceptable models, but correlated tests and pool imbalance complicate how many tests should be required.

    Source: §I · p. 1

  • Method

    A two-class latent-factor model analyzes r-of-k gates, and exact binomial bounds with a union bound certify precision when selecting from a fixed shortlist.

    Source: §I · p. 1, C. Direct certification without a… · p. 5

  • Results

    At a 99% target, the stated configuration requires 8 tests at latent correlation 0 and 5,182 at 0.5, with fewer than 10% of good models kept at 0.5.

    Source: §VIII · p. 5

  • Takeaways & Limitations

    Gate decisions should weigh clearance trustworthiness against good-model retention, and labelled validation data can certify a population precision target for a shortlisted gate.

    Source: §VIII · p. 5

    The quantitative results assume common per-test error rates and a single equicorrelated factor, while real suites may have multiple shared failure channels.

    Source: §VII · p. 5

Abstract

from arXiv · show

Before a machine learning model ships, it often has to pass a suite of automated tests. Requiring every test to pass looks safe, yet it can reject many models that would have served users well, and it does not say how trustworthy a passing model actually is. We treat the release gate as a design problem: choose how many tests a model must pass so that cleared models meet a stated reliability target, while keeping as many good models as possible. A two-class latent-factor model makes both costs explicit and reduces each calculation to a one-dimensional integral. We prove that when both classes share the same latent correlation, a stricter gate always raises reliability, so the gate that keeps the most good models is the most lenient one that still meets the target. Under pass-all gating, any reliability target short of perfection is attainable within the model, but the share of good models kept tends to zero as the suite grows. Correlation between tests sets the price. In one configuration, a 99 percent target needs 8 independent tests, but 74 tests at a latent correlation of 0.3 and 5,182 at 0.5, where the gate keeps fewer than one good model in ten. We also give a validation procedure, built on exact binomial bounds, that certifies a gate from labelled data even when the gate is chosen from a fixed shortlist.

I. INTRODUCTION

The introduction frames release gating as a trade-off between clearing unacceptable models and rejecting acceptable ones, with correlated tests and pool composition shaping the choice. It presents a gate-design rule, quantifies dependence costs, and outlines data-based certification.

I. INTRODUCTION

  • A gate can both clear unacceptable models and reject acceptable ones; deciding how many tests must pass depends on test errors, their dependence, and pool composition.With 1,000 flawed candidates for every 10 good ones, a 99% target requires false clearances at no more than one ten-thousandth the rate of good clearances.Source: §I · p. 1
  • The paper studies candidate-model retention rather than selective classification of inputs, and distinguishes this release problem from leaderboard reuse and adaptive overfitting.It situates the analysis alongside diagnostic testing, latent-class models, and learn-then-test methods.Source: §I · p. 1
  • With a common within-class correlation, precision rises with the required pass count, and the most lenient qualifying r-of-k gate keeps the most good models.Under pass-all gating, any precision target below one is attainable, while sensitivity tends to zero as the suite grows.Source: §I · p. 1
  • At a 99% precision target, raising latent correlation from 0 to 0.5 increases the pass-all suite from 8 to 5,182 tests and reduces good-model retention from 66% to under 10%.The figures hold per-test error rates fixed.Source: §I · p. 1
  • Exact binomial bounds with a union bound can provide a simultaneous precision lower bound and preserve a valid guarantee when choosing a gate from a fixed shortlist.The reported tables are model calculations, not measurements of deployed pipelines; precision concerns a candidate drawn before evaluation, not a winner selected afterward.Source: §I · pp. 1–2

II. MODEL: FROM TEST OUTCOMES TO RELIABLE CLEARANCE

The model labels candidates against a deployment criterion and evaluates each on a suite of noisy tests. A shared latent factor represents dependence between tests within each candidate class.

A. Candidates, tests and the gate

  • Candidates are labelled acceptable or unacceptable against a deployment criterion, and released when they pass at least r of k tests; mandatory requirements remain separate pass-all checks.The pool imbalance is Ω = n−/n+.Source: A. Candidates, tests and the gate · p. 2
  • The miss rate p is the probability an unacceptable candidate passes a test, while the false-alarm rate η is the probability an acceptable candidate fails one.These are error rates against the deployment criterion, not benchmark accuracies.Source: A. Candidates, tests and the gate · p. 2

B. Dependence between tests

  • A candidate-level shared latent factor captures tests' tendency to err together; ρ is within-class latent-score correlation, not mixed-population benchmark-score correlation.For unacceptable candidates, θ_i measures how well a flaw hides from the suite; for acceptable candidates, it measures how clearly the suite recognizes competence.Source: B. Dependence between tests · p. 2

C. Clearance probabilities and precision

The section computes clearance probabilities by conditioning on a candidate's latent factor and integrating over it. It then relates the false-clearance rate and sensitivity to deployment precision and pool-level outcomes.

C. Clearance probabilities and precision

  • Conditioned on the latent factor, tests pass independently and the pass count is binomial; integrating over the factor gives each class's gate-clearance probability.The resulting probabilities are q for false clearance and σ for the share of good candidates kept.Source: C. Clearance probabilities and precision · p. 2
  • Raising r lowers both false clearances q and good-candidate retention σ, so the gate trades false clearances against lost good candidates.Source: C. Clearance probabilities and precision · p. 2
  • Deployment precision is the probability that a pre-evaluation candidate is acceptable given clearance, and its target constrains the false-clearance-to-sensitivity ratio according to pool imbalance.For τ = 0.99 and Ω = 100, the stated ceiling is 1.01 × 10−4; the expected false-release count is n−q, while the good-candidate survival expression assumes independent candidates.Source: C. Clearance probabilities and precision · p. 2

III. CHOOSING THE GATE

With a common latent correlation, stricter r-of-k gates strictly raise precision, and pass-all precision rises with suite size toward one. The gate that retains the most good models while meeting the target is the most lenient qualifying gate.

III. CHOOSING THE GATE

  • For k ≥ 2, precision strictly increases with the required pass count; under pass-all gating it also increases with suite size and tends to 1.The theorem also shows that both acceptable and unacceptable clearance rates tend to zero as a pass-all suite grows.Source: §III · p. 2
  • Among deterministic r-of-k gates meeting the target, the most lenient gate keeps the most good candidates; common correlation also makes qualifying gates a contiguous range.If no gate qualifies, the rule reports infeasibility rather than defaulting to the strictest gate.Source: §III · p. 3
  • As a pass-all suite grows, sensitivity tends to zero even as precision approaches 1, so the chance that any of a fixed number of acceptable candidates clears tends to zero.For any target below 1, a sufficiently large suite meets it, provided more tests with the same error rates and dependence are available.Source: §III · pp. 2–3
  • At k = 20 and ρ = 0.3, even pass-all gating misses the 99% target.Under independence, precision rounds to 1.000 from r = 17, though those gates differ markedly in sensitivity.Source: Table II · p. 3
  • Each (r, k) evaluation reduces to a one-dimensional integral, and the search over r is discrete, making the full design cheap to compute.Under independence, choosing r/k to approach a pass fraction between the two classes' per-test pass probabilities can drive false clearance down exponentially while sensitivity tends to one.Source: §III · p. 3

IV. WHAT A STRICTER GATE BUYS, AND WHAT IT COSTS

With a fixed suite, increasing test correlation narrows the qualifying gate choices. When only pass-all qualifies, adding tests can meet the precision target but sharply reduces the share of good models retained.

IV. WHAT A STRICTER GATE BUYS, AND WHAT IT COSTS

  • At ρ = 0.3, no suite up to 40 tests qualifies; at 80 tests, only pass-all qualifies, discarding 72% of good candidates.As correlation grows, the freedom to tolerate partial failures shrinks and then disappears.Source: §IV · p. 3
  • At 20 tests, each extra required pass cuts false clearance by at least sixfold without correlation, but only 1.5- to 4-fold at ρ = 0.3.Correlated flaws that evade several tests tend to evade the rest.Source: §IV · p. 3
  • A single effective test count generally cannot reproduce both classes’ clearance probabilities because correlation changes false clearance and sensitivity differently.The results describe the one-factor family, not dependence in general; dependence can sometimes separate classes even when their per-test rates match.Source: §IV · p. 4
  • At ρ = 0.5, 5,182 pass-all tests reach 99% precision but retain fewer than one good candidate in ten.The required suite grows from 8 tests at ρ = 0 to 74 at ρ = 0.3 and 5,182 at ρ = 0.5.Source: §IV · p. 4, Table III · p. 4, Figure 1 · p. 4
  • Table I fixes suite size to find the most lenient qualifying gate, whereas Table III fixes pass-all and asks how large the suite must be.The latter prices the removal of infeasibility identified by the former.Source: §IV · p. 4

V. A RELEASE DECISION UNDER UNCERTAIN DEPENDENCE

In the 500-checkpoint example, the assumed 12-test setup cannot meet the 99% precision target. The decision is especially sensitive to the assumed test correlation.

V. A RELEASE DECISION UNDER UNCERTAIN DEPENDENCE

  • The worked pool has 10 acceptable and 490 unacceptable checkpoints, with estimated per-test miss and false-alarm rates of 0.3 and 0.05.The team assumes ρ = 0.2 and evaluates 12 benchmarks.Source: §V · p. 4
  • At 12 tests, the best available precision is 0.937 at pass-all, so the 99% target is out of reach under the assumed ρ = 0.2.Relaxing to 11 required passes raises sensitivity from 0.621 to 0.860 but lowers precision to 0.792.Source: §V · p. 4, Table IV · p. 4
  • If true ρ is 0.4 rather than 0.2, pass-all precision falls to 0.616 and expected false releases rise from 0.4 to 4.3.The precision shift of 0.32 exceeds the 0.15 shift from moving the gate from 12 to 11 at the assumed correlation.Source: §V · p. 4, Table IV · p. 4
  • For a 99% target, pass-all requires 22 tests at ρ = 0.2 but 216 at ρ = 0.4.The example uses 500 checkpoints, with 10 acceptable and 490 unacceptable candidates.Source: §V · p. 4, Table IV · p. 4

A. What data can identify

Estimating the model’s inputs requires labelled candidates and within-class dependence estimates; unlabelled score matrices alone do not identify them.

A. What data can identify

  • Unlabelled leaderboard score matrices identify none of the four model inputs without further assumptions.The inputs are p, η, ρ, and Ω.Source: A. What data can identify · p. 4
  • Estimating error rates requires labels for whether candidates meet the criterion, and ρ must be estimated within labelled classes.Across a mixed model population, variation in overall quality can inflate observed benchmark correlation.Source: A. What data can identify · p. 4
  • An independent held-out criterion can divide candidates for estimating p, η, and ρ, but transferring those estimates to true acceptability requires separate proxy evidence.Latent-trait models fitted by marginal maximum likelihood can be checked for structure that one factor misses.Source: A. What data can identify · p. 5

B. Robust design

The robust design evaluates gates over an uncertainty set that covers the true model parameters with high probability. It certifies gates meeting the target and selects the least strict feasible gate, while accounting for possible nonmonotonicity when class correlations differ.

B. Robust design

  • A robust guarantee requires a data-dependent uncertainty set U that covers the true parameter with probability at least 1−δ.The parameter vector includes prevalence, error rates, and separate correlations for the two classes.Source: B. Robust design · p. 5
  • With probability at least 1−δ, every gate whose worst-case precision over U meets τ meets the target within the model family.An empty feasible set means no gate is certified under U, not that the target is impossible.Source: B. Robust design · p. 5
  • Because sensitivity falls as the required pass count r rises, the smallest robust-feasible r keeps the most good candidates.When class correlations differ, precision need not be monotone in r, so all gates must be searched.Source: B. Robust design · p. 5

C. Direct certification without a Gaussian model

The procedure certifies gates from representative labelled candidates using exact binomial bounds and simultaneous coverage across a fixed shortlist. It preserves the guarantee when selecting among certified gates, but may require very large validation samples.

C. Direct certification without a Gaussian model

  • Exact one-sided Clopper–Pearson bounds and a union bound give simultaneous coverage for all 2J bounds across a fixed shortlist, even with a shared validation sample.The shortlist, suite, and benchmark thresholds must be fixed independently of the validation sample.Source: C. Direct certification without a… · p. 5
  • Validation candidates must be i.i.d. within each labelled class and representative of the candidates the gate will face.Checkpoints from one training run are correlated and do not automatically count as independent observations.Source: C. Direct certification without a… · p. 5
  • If prevalence Ω is estimated, an upper confidence bound can be substituted, but class-stratified samples do not estimate prevalence.The precision lower bound is set to zero if its denominator vanishes.Source: C. Direct certification without a… · p. 5
  • Choosing the smallest r whose precision lower bound meets τ preserves the guarantee and keeps the most good candidates among certified gates.This gate may still be stricter than the population-optimal choice.Source: C. Direct certification without a… · p. 5
  • With τ = 0.99, Ω = 100, δ = 0.05, and J = 20, certification needs approximately 221,000 unacceptable candidates at L+ = 0.30 or 682,000 at L+ = 0.097.These calculations assume the stated sensitivity lower bound was already established from the acceptable validation sample.Source: C. Direct certification without a… · p. 5
  • Pooling releases helps only if class-conditional clearance probabilities remain stable; repeatedly checking a fixed-sample bound is not anytime-valid, and population precision does not cover a later winner selection.The latter caveat concerns selection by another rule after the population-level gate evaluation.Source: C. Direct certification without a… · p. 5

VII. LIMITATIONS

The paper's conclusions depend on the assumed test-error and dependence structure, while its release-gate rule weighs clearance trustworthiness against good-model retention. Correlation can make a target costly, and labelled validation can certify a shortlist without resolving every reason a gate fails certification.

VII. LIMITATIONS

  • A single-factor model imposes equicorrelation, whereas suites with several shared failure channels require a multi-factor extension and multidimensional integration.The quantitative results also assume common error rates across tests; test-specific rates would require a Poisson–binomial tail.Source: §VII · p. 5
  • The survival probability 1−(1−σ)^n+ assumes independent candidates, although q, σ, Π, and expected false-release count do not.Quantitative statements in Sections IV and V are conditional on per-test error rates and dependence structure.Source: §VII · p. 5

VIII. CONCLUSION

  • In the common-correlation model, stricter gates raise precision, so the least strict gate meeting the target keeps the most good models.Pass-all can attain any sub-perfect target within the model, but sensitivity tends to zero as the suite grows.Source: §VIII · p. 5
  • Raising latent correlation from 0 to 0.5 changes the 99% pass-all target from 8 tests to 5,182 and leaves fewer than one good model in ten retained.In an illustrative sweep, misjudging correlation by a factor of two costs more than moving the gate by one test.Source: §VIII · p. 5
  • Labelled validation under the stated sampling assumptions can certify population precision while allowing a gate choice from a fixed shortlist.If none certifies, the evidence cannot distinguish too little data, a genuinely inadequate gate, or conservative bounds.Source: §VIII · p. 5

Related papers

Loading 2610.00993v1…