PaperlayerView exact revision on arXiv

Source-linked AI summary

BRIDGE: An Acceptance Gate to Block False Improvements in Model Updates, with a Retrospective Clinical Evaluation

Dwi Sunaryono

arXiv:2610.29515v1cs.LGstat.ML

TL;DR

Validation-based selection can make an update look better than it performs on new data, a concern for models whose updates may affect clinical decisions. BRIDGE uses cross-fitting and a bootstrap lower-bound margin to decide whether to accept each update. Across the tested collections, it reduces false improvements among accepted updates relative to the naive rule.

  • Problem

    A positive validation gain can be misleading when the same data select and judge an update, and false improvements can misguide clinical decisions.

    Source: §I · pp. 1–2

  • Method

    BRIDGE uses fixed settings and cross-fitting to separate update selection from judgment, accepting only when the gain’s lower bootstrap bound exceeds a margin.

    Source: §V · p. 19

  • Results

    False-improvement rates fall from 19.1% to 1.1% in (A), 20.7% to 1.7% in (B), 20.0% to 0% in (C), 12.5% to 0.5% in (D), and 25.6% to 1.0% in (E).

    Source: §V · p. 19

  • Takeaways & Limitations

    BRIDGE removes 97% to 100% of the false improvements accepted by the naive rule, and its mean reference gain per deployed update is 1.8 to 3.1 times the naive rule’s.

    Source: A. SUMMARY OF MAIN RESULTS · pp. 12–13

    The claims hold within the tested conditions listed in Table H8.

    Source: A. LIMITATIONS AND FUTURE WORK · p. 20

Abstract

from arXiv · show

Models are updated repeatedly, and an update that looks better on validation data but not on new data is a false improvement that can misguide clinical decisions. The naive rule accepts every positive validation gain and is optimistic when the same data choose and judge the update. BRIDGE is a model-agnostic acceptance gate that decides each update from predictions and the available labels, with fixed settings. Each fold of the data judges the update chosen on the other folds (cross-fitting) to reduce selection bias; acceptance requires the gain's lower bootstrap bound to exceed a margin. The four settings are (A) 11 new datasets, (B) synthetic tasks, (C) 12 retrospective clinical tasks, and (D) 39 supporting datasets. The false-improvement rate falls from 19.1% (naive) to 1.1% (BRIDGE) in (A), 20.7% to 1.7% (B), 20.0% to 0% (C), and 12.5% to 0.5% (D); every 95% confidence interval of the reduction over datasets or tasks excludes zero (not over the clinical sources). An equal-size random draw from the naive acceptances contains more false improvements in (A)-(E) (post hoc, p<0.001). On the OpenML collection (E) of 14 new datasets, the false-improvement rate falls from 25.6% to 1.0%. BRIDGE removes 97% to 100% of the false improvements that the naive rule accepts, and its mean gain per deployed update is 1.8 to 3.1 times the naive rule's. The 95% upper bound on the false-improvement rate is below 5.5%: formal in (B) for independent tasks (3.56%), and conditional on the real data tested (0.33% to 5.40%).

I. INTRODUCTION

Validation gains can be optimistic when the same data select and judge an update, and false improvements can misguide clinical decisions. BRIDGE addresses methodological and empirical gaps with a fixed acceptance gate and an internally pre-registered evaluation.

I. INTRODUCTION

  • 12.5% to 25.6% of updates accepted by the naive rule are false improvements across collections (A)–(E), concentrated among gains of +0.5 points or less.In that lowest-gain group, 38% to 64% are false improvements; above +2 points, 0% to 4% are.Source: §I · p. 2, Figure 1 · p. 2
  • The methodological gap is a fixed acceptance gate combining cross-fitted selection, a bootstrap decision score, and a margin; the empirical gap is large-scale pre-registered evaluation across change types, metrics, and domains.The evaluation includes a retrospective application to 12 clinical tasks from six sources using net benefit as the decision metric.Source: §I · pp. 2–3
  • BRIDGE cross-fits update selection and judgment, then accepts only when the gain’s lower bootstrap bound exceeds a minimum-gain margin.Its settings are fixed across tasks, and it uses available labels without requiring new data.Source: §I · pp. 2–3
  • The evaluation is internally pre-registered, with the design, pass criterion, dataset list, and seed rule fixed before each run; the open-source package reproduces stored results from 15 runs.Files are recorded with SHA-256 hashes in a manifest.Source: §I · p. 3

A. SELECTION BIAS AND DATA LEAKAGE IN VALIDATION-BASED MODEL CHOICE

Selecting the best of many candidates on validation data makes its measured gain optimistic, while leakage can add further optimism. BRIDGE addresses the selection issue with cross-fitting and returns an accept-or-reject decision for one update using labels already available.

A. SELECTION BIAS AND DATA LEAKAGE IN VALIDATION-BASED MODEL CHOICE

  • Choosing the best of many candidates and judging it on the same validation data makes its apparent gain exceed the gain expected on new data; the optimism can grow with candidate count and tuning.Group and temporal leakage can also make validation scores too optimistic; the paper audits both before splitting.Source: A. SELECTION BIAS AND DATA LEAKAGE IN… · pp. 3–4

B. STATISTICAL TESTS FOR COMPARING CLASSIFIERS

  • Standard classifier tests compare fixed trained models, fixed training procedures, or methods across datasets, but do not assess whether to accept a newly selected best configuration.Rank-based and Bayesian tests answer which method is better, not whether one update should be accepted.Source: B. STATISTICAL TESTS FOR COMPARING… · p. 4

C. GATES FOR APPROVING MODEL UPDATES

  • Prior update gates require additional data, while BRIDGE answers whether one update improves on the current model using labels already available.The closest calibration methods bound the risk of a selected policy; BRIDGE’s evaluation bounds the share of false improvements over many decisions.Source: C. GATES FOR APPROVING MODEL UPDATES · p. 4

III. THE BRIDGE PROCEDURE AND EXPERIMENTAL DESIGN

  • BRIDGE splits POOL into three folds, selecting a candidate on two folds and judging it on the third in each round; pooled judgments determine acceptance.An accepted update is re-selected on all of POOL and deployed; otherwise the baseline is retained. Cross-fitting addresses selection bias, the bootstrap lower bound addresses sampling noise, and the margin enforces a minimum gain.Source: §III · pp. 4–5

A. DATA PARTITION AND CANDIDATE SPACE

The procedure separates model fitting, candidate selection, gating, and evaluation through SEARCH, POOL, and a locked reference. Within POOL, cross-fitting assesses candidates on folds excluded from their selection, and a bootstrap lower quantile must exceed a fixed margin for deployment.

A. DATA PARTITION AND CANDIDATE SPACE

  • SEARCH fits the base learners, while POOL supplies the evidence for selecting and gating an update; the locked reference is reserved for scoring experiments.Base-learner parameters and preprocessing are fitted only on SEARCH; nothing is fitted on POOL.Source: A. DATA PARTITION AND CANDIDATE SPACE · p. 5
  • The binary candidate space combines threshold variants and weighted mixtures of HGB, DT, and SVC, with coarse grids chosen to limit selection bias.The default space contains 173 configurations; its grids can be refined in sensitivity tests.Source: A. DATA PARTITION AND CANDIDATE SPACE · p. 6

B. CROSS-FITTED SELECTION AND BOOTSTRAP GATE

  • Each of three POOL folds scores a candidate selected on the other folds, producing pooled out-of-sample-with-respect-to-selection counts.If no candidate strictly improves on the selection folds, the baseline is used.Source: B. CROSS-FITTED SELECTION AND… · pp. 6–7
  • BRIDGE accepts only when the 5th-percentile bootstrap decision statistic exceeds the fixed margin; otherwise it retains the baseline.The gate uses 20,000 multinomial resamples and a fixed margin ε = 0.005, or 0.5 weighted-F1 percentage points.Source: B. CROSS-FITTED SELECTION AND… · p. 7
  • After acceptance, the deployed configuration is re-selected on all of POOL and may differ from the per-fold candidates used to decide.The cross-fit estimate of the final configuration’s gain is slightly pessimistic on average, by 0.0034.Source: B. CROSS-FITTED SELECTION AND… · p. 7

C. GENERALIZATION TO METRICS AND MULTICLASS

The gate extends to metrics computed from confusion tables and to multiclass tasks without changing its decision rule. Its count-based evidence representation limits implementation to metrics based on hard predicted labels.

C. GENERALIZATION TO METRICS AND MULTICLASS

  • The same counts, metric difference, bootstrap, and decision rule extend to any higher-is-better metric computed from a confusion table and to multiclass tasks.A lower-is-better metric would require a sign transformation, which this implementation does not define.Source: C. GENERALIZATION TO METRICS AND… · p. 8
  • The count-based representation limits the implementation to metrics computed from hard predicted labels.Examples include weighted F1, balanced accuracy, macro-F1, F2, sensitivity, and specificity.Source: C. GENERALIZATION TO METRICS AND… · p. 8
  • For clinical binary tasks, the protocol uses net benefit at a decision threshold; for rare classes, it uses balanced accuracy and macro-F1.AUC is unchanged by threshold-only updates, so the gate uses threshold metrics instead.Source: C. GENERALIZATION TO METRICS AND… · p. 9

D. RISK AMONG ACCEPTED UPDATES: UPPER BOUND AND SCOPE

The paper defines risk as the share of accepted updates that are false improvements and gives a finite-sample upper bound under an i.i.d. task-population assumption. For real benchmark collections, where that assumption does not hold, it reports a conditional KL-ratio bound instead.

D. RISK AMONG ACCEPTED UPDATES: UPPER BOUND AND SCOPE

  • Under the assumption that decision pairs are i.i.d. across tasks, the bound covers the population risk with probability at least 95%, using accepted-update and false-improvement counts.The bound uses an exact binomial tail inequality and applies for any finite number of tasks.Source: D. RISK AMONG ACCEPTED UPDATES: UPPER… · p. 9
  • The false-improvement risk is the proportion of accepted updates whose locked-reference metric difference is nonpositive.The quantity is defined when the acceptance probability is positive.Source: D. RISK AMONG ACCEPTED UPDATES: UPPER… · p. 9
  • Counting accepted updates with uncertain reference outcomes as false improvements yields a valid but more conservative bound.The primary bound counts an update as false when the reference shows no improvement.Source: D. RISK AMONG ACCEPTED UPDATES: UPPER… · p. 9
  • The finite-sample proposition does not apply to real benchmark tasks, so their reported risk bound is conditional on the observed collection.It uses a ratio of one-sided KL bounds; the real tasks share a locked reference and form a fixed collection rather than an i.i.d. sample.Source: D. RISK AMONG ACCEPTED UPDATES: UPPER… · pp. 9–10

E. EXPERIMENTAL DESIGN: DATA, PARTITION, COMPARATORS, AND EVALUATION

The evaluation spans five collections of real and synthetic tasks, with fresh-seed trials and shared comparator conditions. It measures false improvements among accepted updates and applies a pre-registered criterion and hypothesis tests.

a: Data

  • The five collections cover new real datasets, synthetic tasks, retrospective clinical tasks, supporting datasets, and OpenML; together the real-data collections contain 76 units.(A)–(E) serve distinct roles, including new-data evaluation, formal-bound testing, clinical application, and supporting evidence.Source: E. EXPERIMENTAL DESIGN: DATA,… · p. 10

b: Partition and trials

  • Each trial runs one task under a fresh seed and ends with an accept-or-reject decision; the main settings contain 8,600 decisions, with 1,280 more in (E).Trials per dataset are 100 in (A), 200 per clinical task, and 60 per multiclass dataset; (B) has one trial per task.Source: E. EXPERIMENTAL DESIGN: DATA,… · p. 11

c: Comparators

  • All rules judge the same decisions with the same candidates, base model, POOL counts, reference, and learner configuration, isolating the acceptance rule.Comparators include the naive rule, holdouts, a fixed validation margin, and three classical-test rules.Source: E. EXPERIMENTAL DESIGN: DATA,… · p. 11

d: Evaluation metrics

  • The main quantity is the false-improvement rate among accepted updates, evaluated against a locked reference; weighted-F1 is the pre-specified primary metric.Clinical analyses also include net benefit, balanced accuracy, and other secondary metrics.Source: E. EXPERIMENTAL DESIGN: DATA,… · p. 11

e: Statistical analysis

  • A run passes only if its 95% upper bound is below 10% and all additional LOCO, yield, dataset-count, and LOCO0 conditions are met.The bound uses error level δ = 0.05; the bound component is equivalent to a one-sided test of r ≥ 0.10 against r < 0.10.Source: E. EXPERIMENTAL DESIGN: DATA,… · p. 11
  • T1 uses paired cluster bootstraps over datasets or tasks; T2 tests equivalence within a 2-point margin, and T3 tests whether BRIDGE acceptances resemble a same-size random draw from naive acceptances.The runs distinguish dataset-level resampling from trial-level comparisons because trials within a dataset are correlated.Source: E. EXPERIMENTAL DESIGN: DATA,… · pp. 11–12

IV. RESULTS

Across all five collections, BRIDGE lowers the false-improvement rate relative to the naive rule, and the reported reductions are statistically supported. Tests also examine comparator rules, the margin, and whether BRIDGE merely accepts fewer updates.

A. SUMMARY OF MAIN RESULTS

  • BRIDGE lowers false-improvement rates by 12.0 to 24.6 percentage points versus the naive rule, to at most 1.7% across the five collections.Every 95% interval for the reduction over datasets or tasks lies above zero; BRIDGE removes 97% to 100% of naive-rule false improvements and deploys updates with 1.8 to 3.1 times the mean reference gain.Source: A. SUMMARY OF MAIN RESULTS · pp. 12–13

B. HYPOTHESIS TESTS

  • Against the naive rule, equal false-improvement rates are rejected in all five collections; against test-based rules, they are rejected in (A), (B), and (D), and for the 5×2 cv test in (C).Rules that also separate choosing from judging have rates of 0.0% to 3.0%; T1 detects a difference in (D), where BRIDGE is 1.0 percentage point lower.Source: B. HYPOTHESIS TESTS · p. 13
  • The margin lowers false-improvement rates by 1.1 to 1.8 percentage points in (A), (B), and (D); changing it from 0.0025 to 0.01 changes rates by less than 2 percentage points.On binary datasets in (E), the margin lowers the rate from 8.9% to 1.2%.Source: B. HYPOTHESIS TESTS · p. 13
  • A same-size random draw from naive acceptances contains 49 to 240 false improvements, compared with 0 to 8 for BRIDGE, rejecting the random-draw null in every collection.Within-dataset evidence also appears in (D) and (E); in (A) and (C), selection is shown across datasets, and this analysis is post hoc.Source: B. HYPOTHESIS TESTS · p. 13, Figure 3 · p. 13

C. COMPARISON OF ACCEPTANCE RULES

The comparisons assess false-improvement rates, acceptance yield, and whether BRIDGE’s advantage persists against rules that also separate selection from evaluation. The retrospective clinical results are limited to existing records and a hypothetical action.

C. COMPARISON OF ACCEPTANCE RULES

  • On (A), test-based rules fail the criterion while BRIDGE passes; among 459 gains of at most 0.01, BRIDGE accepts none, while the naive rule accepts all and is wrong 164 times.The test-based failure is specific to (A); the tests assume the compared pair is fixed before seeing the data.Source: C. COMPARISON OF ACCEPTANCE RULES · pp. 13–14
  • The corrected t-test with per-split re-selection meets the criterion in (A)–(D); on (A), its false-improvement rate is 2.3% versus 1.1% for BRIDGE, a non-significant difference.At matched counts, BRIDGE has an equal or lower rate through about 450 accepted updates, then a slightly higher proportion beyond that point.Source: C. COMPARISON OF ACCEPTANCE RULES · p. 14
  • Against honest holdouts, BRIDGE accepts 1.1 to 1.9 times as many updates and deploys more above-margin updates, while false-improvement rates remain close.In (A), rates are 1.1% for BRIDGE versus 1.3% and 3.2% for the holdouts; in (C), neither BRIDGE nor the holdouts accepts a false improvement.Source: C. COMPARISON OF ACCEPTANCE RULES · p. 14
  • On synthetic population (B), BRIDGE accepts 475 updates with 8 false improvements, giving 1.7%; Proposition 1’s 95% upper bound is 3.56%.On a fresh batch, it accepts 462 with 7 false improvements and gives a 3.35% bound; cross-batch estimates fall below the opposite batch’s bound, a consistency check rather than validation of coverage.Source: C. COMPARISON OF ACCEPTANCE RULES · p. 15, Table 11 · p. 15

E. CLINICAL APPLICATION (RETROSPECTIVE)

  • For clinical net benefit at pt = 0.2, naive acceptances have a 20.0% false-improvement rate, versus none among BRIDGE’s 1,199 accepted updates.The reduction’s 95% interval is 4.5 to 40.3 over tasks, but 0.0 to 31.9 over the six sources; the source-level difference is not significant (p = 0.17).Source: E. CLINICAL APPLICATION (RETROSPECTIVE) · p. 16, Table 12 · p. 16
  • For clinical net benefit, BRIDGE adds 90.9 true positives and 197.0 false positives per 1,000 patients, while flagged workload rises from 98 to 386 per 1,000.The net-benefit run followed an earlier weighted-F1 run on the same tasks that failed the LOCO criterion, so it is a fresh-seed re-test, not independent confirmation; the application is retrospective, not evidence of practical clinical benefit.Source: E. CLINICAL APPLICATION (RETROSPECTIVE) · pp. 15–16

F. COMPONENT ANALYSIS: ABLATION AND CANDIDATE-SPACE SIZE

The component analysis compares the gate’s evidence, decision statistic, and margin, then tests performance as candidate spaces grow. Scope checks find the criterion met across many tested metrics, learners, and settings, with specific exceptions and boundaries.

F. COMPONENT ANALYSIS: ABLATION AND CANDIDATE-SPACE SIZE

  • The bootstrap lower bound alone gives the largest false-improvement-rate reduction in (A), (B), (D), and (E), at 7.6–14.1 percentage points below the naive rule; in (C), it ties the margin at 17.3 points.Both the bootstrap lower bound and margin reduce yield by rejecting some updates the naive rule accepts.Source: F. COMPONENT ANALYSIS: ABLATION AND… · p. 16
  • On binary datasets of (E), BRIDGE has the lowest false-improvement rate among eight cells (1.2%, bound 6.1%), but its LOCO bound is 10.3%, so no cell meets the 10% criterion; across all 14 datasets, BRIDGE reaches 1.0% (bound 5.4%).Source: F. COMPONENT ANALYSIS: ABLATION AND… · p. 16
  • On (E)’s binary datasets, increasing the space to 11,249 configurations raises the false-improvement rate to 3.4%; in the same-POOL cell, the rate rises from 3.4% to 9.2%.The default space of 173 configurations remains the tested condition of use; a denser grid needs separate confirmation.Source: F. COMPONENT ANALYSIS: ABLATION AND… · pp. 17–18
  • At 173, 1,825, and 11,249 configurations, BRIDGE meets the criterion on (A)–(D) with false-improvement rates at or below 1.7%, while the naive rule fails at every size.The multiclass datasets of (D) and (E) are excluded from the size analysis because no equally dense multiclass grid was pre-registered.Source: F. COMPONENT ANALYSIS: ABLATION AND… · p. 17, Figure 5 · p. 18

G. SCOPE AND CONDITIONS OF USE

  • BRIDGE meets the criterion for weighted-F1, balanced accuracy, and macro-F1; training sizes of 1,000, 2,430, and 6,000 rows; multiclass tasks; and the 39 supporting datasets of (D).Feature selection with SVC is the reported exception; the OpenML collection meets the criterion jointly with its extension.Source: G. SCOPE AND CONDITIONS OF USE · p. 18
  • The gate meets the criterion with every tested member set in (A)–(D), but both additional learner pairs miss it on OpenML (E).XGBoost and an MLP under early stopping meet the criterion in all five collections; an MLP with early stopping has lower final weighted-F1 than HGB in (D) and still meets the gate criterion.Source: G. SCOPE AND CONDITIONS OF USE · pp. 18–19
  • On imbalanced data, BRIDGE meets the criterion down to 5% minority prevalence under balanced accuracy and macro-F1; the stated condition of use is a minority-sensitive metric.POOL contains about 360, 180, and 90 minority cases at 20%, 10%, and 5% prevalence, respectively.Source: G. SCOPE AND CONDITIONS OF USE · p. 19
  • BRIDGE meets the criterion in all 24 real-data quantile–margin cells, and its gate cost is about 0.03 s; fitting bagged base models dominates total cost.The 16-bag setting cuts fitting cost by about half. Tuning can be proposed as an update, though it meets the formal criterion in only two of 12 tested cells; the default configuration is retained.Source: G. SCOPE AND CONDITIONS OF USE · p. 19

V. CONCLUSION

Across every tested collection, BRIDGE lowers the false-improvement rate among accepted updates; it removes 97% to 100% of the false improvements accepted by the naive rule. The evidence supports BRIDGE for tabular updates within tested conditions, with bounds and clinical findings limited by their stated populations and settings.

V. CONCLUSION

  • 19.1% to 1.1% in (A), 20.7% to 1.7% in (B), 20.0% to 0% in (C), 12.5% to 0.5% in (D), and 25.6% to 1.0% in (E) are the false-improvement rates for naive and BRIDGE, respectively.The collections tested all show lower rates with BRIDGE; the clinical comparison was significant at the task level, but not at the coarser source level.Source: §V · p. 19
  • BRIDGE removes 97% to 100% of naive-rule false improvements, and its mean reference gain per deployed update is 1.8 to 3.1 times the naive rule’s.Against an honest holdout on the same labeled data, BRIDGE deploys 1.1 to 1.9 times as many updates gaining more than the margin; the share gaining at most the margin does not differ significantly.Source: §V · p. 19
  • The results also meet the pre-registered criterion across tested variants of learners, candidate spaces, and metrics, including imbalanced data down to 5% minority prevalence.The evidence covers tabular learners and label-based metrics; deep and foundation models, raw images, and probability-level or ranking metrics remain future work.Source: §V · p. 19, What is bounded, and for which… · p. 20

What is bounded, and for which population?

  • The formal 95% upper bound applies to the specified i.i.d. synthetic task population, while the primary real-data bound is conditional on the observed collection and frozen outer partition.The locked reference is finite and imperfect, and the bootstrap quantile inside the gate is a decision rule, not a confidence interval.Source: What is bounded, and for which… · p. 20
  • The comparator evidence is limited: the best fixed validation margin was selected after results were seen, and an equal-label-budget empirical comparison with Learn then Test remains future work.At matched acceptance counts, BRIDGE retains its advantage between about 125 and 450 accepted updates.Source: What is bounded, and for which… · p. 20
  • In the clinical evaluation, net benefit describes the sample rather than the population, and prospective validation in further institutions is the next step.The retrospective evaluation covers six sources, including two unweighted surveys.Source: What is bounded, and for which… · p. 20

APPENDIX A PROOFS

The proofs derive a finite-sample upper bound for false improvements among accepted updates and specify the distinct real-data estimand. The appendix also documents the benchmark’s data scope, synthetic task population, and training and reference sizes.

APPENDIX A PROOFS

  • Conditioning on the number of accepted tasks makes the false-improvement count binomial under the stated i.i.d. assumptions, yielding a one-sided upper bound with coverage at least 1 − δ.The proof uses a Chernoff–Kullback–Leibler bound; on real data, the estimand is conditional on the observed collection and frozen outer partition.Source: Appendix A · p. 21
  • The real-data estimand is the ratio of expected false-improvement counts to accepted-update counts conditional on the observed collection, with inner splits and bag draws redrawn per trial.The finite-sample proof rests on Assumption 1 and a binomial tail bound.Source: Appendix A · p. 21

APPENDIX B DATA, AUDIT, AND GENERATOR

  • The benchmark required publicly available, tabular or image-derived-in-tabular-form data with recorded labels; it excluded synthetic or simulated data, time series, thresholded or unverifiable targets, exact 50% class ratios, over 25% duplicate rows, near-perfect single-feature prediction, and subject- or session-grouped rows.Source: Appendix B · p. 21
  • The synthetic analysis draws 1,400 independent tasks from a fixed population, with each task generated from either a tree or linear label mechanism and a 100,000-row reference.The population was pre-specified, and the naive false-improvement rate was designed to be realistic based on pilot work.Source: Appendix B · p. 23
  • Default tasks use 2,430 training rows split into 630 SEARCH and 1,800 POOL rows; synthetic SEARCH sizes of 360, 630, and 1,000 meet the criterion.Their respective bounds are 1.65%, 3.56%, and 3.11%, at yields of 32.1 to 33.9 per 100.Source: Appendix B · p. 23

APPENDIX C PRE-REGISTRATION CHRONOLOGY

The chronology distinguishes development runs that fixed the gate and candidate space from later runs testing the frozen procedure. It records run outcomes, reproducibility checks, and a clinical analysis that changed its endpoint after an initial criterion failure.

APPENDIX C PRE-REGISTRATION CHRONOLOGY

  • Nineteen development runs fixed the gate’s seven parameters and candidate space; every later run tests the frozen gate without changing its default.The pre-registration chronology records endpoints and outcomes for confirmatory runs, including predictions that were not confirmed.Source: Appendix C · p. 23
  • All confirmatory runs MINI-20 to MINI-38 completed without error, and the released package reproduces stored result rows across development and confirmatory runs.Trials reproduced bit-identically across tested Intel and AMD machines except for specified tuning and feature-selection trials, with decisions unchanged in the reported MLP–KNN case.Source: Appendix C · pp. 23, 25
  • The run table defines its endpoint codes and distinguishes development runs from confirmatory, sensitivity, descriptive, and OpenML analyses.Its manifest hashes verify file contents, not the recorded pre-registration times.Source: Appendix C · p. 25
  • The first clinical run failed its LOCO criterion with weighted-F1; a second pre-registered run used net benefit and balanced accuracy as co-primary endpoints on the same 12 tasks with fresh seeds.The first run’s KL bound was 8.15% and per-source LOCO was 12.8%.Source: Appendix C · p. 25

APPENDIX D DATA SEPARATION, FROZEN PARAMETERS, CANDIDATE SPACES, SENSITIVITY, FIXED MARGIN, AND COST

Appendix D documents BRIDGE’s data separation, candidate spaces, frozen settings, sensitivity checks, and computational cost. Its checks report robustness on synthetic and real data while identifying settings where fixed-margin performance or parameter variants are more limited.

Data separation and candidate spaces

  • Cross-fitting assesses each per-fold candidate on the fold that did not select it, then re-selects the final configuration on all of POOL.The default binary candidate space contains 173 configurations; the refined spaces contain 1,825 and 11,249, while the joint member space contains 711.Source: Appendix D · p. 26

Parameter sweeps

  • All ten parameter variants on synthetic population Q pass the pre-registered criterion, with Proposition 1 bounds of 2.0%–4.9%.With Bonferroni correction, every variant remains below 10%, with bounds from 2.75% to 5.73%.Source: Appendix D · p. 26
  • On Q, 16, 32, and 64 bags yield bounds of 3.98%, 3.56%, and 2.69%, respectively, and all meet the criterion.Their yields are 32.8, 33.9, and 33.2 per 100 decisions.Source: Appendix D · p. 26

Monte Carlo error

  • The bootstrap decision statistic is stable across seeds: a single alternative seed changes the decision in 0.3% of Q re-runs and 0.7% of real-data re-runs.At 2,000 bootstrap replicates, BRIDGE accepts 481 updates with 7 false improvements, versus 475 with 8 at 20,000.Source: Appendix D · p. 27

Fixed margin

  • The pre-registered fixed margin t*=0.05 fails under fresh seeds on 6,000-row tasks and new datasets under weighted-F1; post hoc margins pass all settings.These comparisons are exploratory and do not establish that BRIDGE dominates a fixed margin.Source: Appendix D · p. 28

Parameter sensitivity on the real data

  • Collection (D) passes every tested parameter variant, while two variants fail in (A): a 100% SEARCH bag fraction and a 1,000-row SEARCH.The latter finds no false improvement on six datasets but fails the power check (LOCO0 20.77%).Source: Appendix D · p. 28

Computational cost

  • The gate adds about 0.03 seconds per run, or 0.4%–3.8% of SEARCH fitting time, and its inference cost equals that of the deployed predictor.The 16-bag setting also meets the criterion, allowing fitting cost to be halved.Source: Appendix D · p. 28

APPENDIX E PER-DATASET RESULTS AND PAIRED COMPARISONS

Appendix E details per-dataset results and paired comparisons, including cases where BRIDGE and competing rules have similar false-improvement rates. It also reports exploratory fixed-margin comparisons and a descriptive audit of base learners.

Per-dataset results

  • BRIDGE rejects every update on KDD98, Santander, and seismic, where the naive rule accepts 54, 39, and 35 false improvements, respectively.The corresponding naive acceptances are 83, 90, and 94.Source: Appendix E · p. 28, Table E1 · p. 29

Paired comparisons

  • The corrected t-test with per-split re-selection and the best post hoc fixed margin do not differ significantly from BRIDGE in false-improvement rate.Two fixed-margin exceptions favor BRIDGE by 2.2 and 2.5 percentage points; fixed-margin rows are exploratory with nominal intervals.Source: Appendix E · p. 29, Table E2 · p. 29

Fixed-margin detail

  • Against the best fixed margin, BRIDGE has a lower false-improvement rate by 2.2 percentage points at the smallest training size and accepts 31 rather than 14 updates per 100 on large data.The large-data yield difference is +16.8, with paired-bootstrap 95% CI [+10.2; +23.7].Source: Appendix E · p. 29
  • In leave-one-setting-out analysis, BRIDGE passes all seven settings, whereas the best fixed margin passes six and fails on new datasets under weighted-F1.BRIDGE’s bound there is 4.23%, compared with the fixed margin’s 8.50%.Source: Appendix E · p. 30

Base-learner audit

  • Across base learners, AUC ranges from 0.566 to 0.579, while predicted probabilities and iteration control differ.The audit describes these differences as relevant to the gate, rather than differences in case ranking.Source: Appendix E · p. 30, Table E3 · p. 30

Dataset-level bootstrap

  • The dataset-level bootstrap upper bound is 3.1% for BRIDGE and 29.4% for the naive rule on the 11 new datasets.The 5×2 cv and corrected t tests have bounds of 13.0% and 12.5%, respectively.Source: Appendix E · p. 30

APPENDIX F PATIENT-IDENTITY AUDIT

Appendix F audits patient identity in the clinical sources and reports source-level patterns in naive-rule errors. It notes that one source cannot be verified at the patient level, although removing it leaves the primary result unchanged.

Patient-identity audit

  • The mammography source has no patient identifier, so its inclusion cannot be verified at the patient level; removing it leaves the primary result unchanged.The audit records identity keys, duplicate rows, and rows removed before any split.Source: Appendix F · p. 30, Table F1 · p. 30

Clinical results by source

  • Table F2 reports a patient-identity audit of clinical results organized by source.Source: Table F2 · p. 30

APPENDIX G OPENML COLLECTION (E): DATASETS, EXTENSION, AND SENSITIVITIES

Appendix G reports results for the pre-registered OpenML collection and its extension, alongside subgroup analyses and curation limitations. The pooled collections meet the reported criteria, but several subgroup results and dataset provenance qualify that conclusion.

OpenML collection (E)

  • In OpenML collection (E), BRIDGE accepts 192 of 1,280 decisions with 2 false improvements (1.0%), a 5.40% bound, and a yield of 15.The binary subgroup narrowly fails LOCO at 10.30%, while the three-dataset multiclass subgroup has a 32.59% bound.Source: Appendix G · p. 31, Table G1 · p. 32

Extension E2

  • Extension E2 passes jointly with 452 accepted updates and 7 false improvements (1.55%); the naive rule has 303 false improvements among 1,633 acceptances (18.6%).The BRIDGE-minus-naive difference is −17.0 percentage points (95% CI [−24.3, −9.6]).Source: Appendix G · p. 31

Pooled collections

  • Pooling E and E2, BRIDGE accepts 644 updates with 9 false improvements (1.40%) across 33 datasets and 2,980 decisions.The pooled result meets every criterion, with a bound of 3.38% and LOCO 4.04%; its difference from naive is −20.2 percentage points (95% CI [−26.7, −13.8]).Source: Appendix G · p. 31

Subgroup results

  • The pooled binary subgroup meets every criterion with 486 accepted updates and 7 false improvements (1.44%), while the E2 multiclass subgroup fails its 10% LOCO condition at 14.10%.The E2 binary subgroup’s pass depends on churn, a dataset flagged as artificial; without datasets flagged a–c, the combined binary subgroup of E and E2 meets LOCO.Source: Appendix G · p. 31

Curation and provenance

  • A provenance audit found that pendigits violates the frozen curation rule because its 44-writer grouping is unavailable to the row-splitting runner.The deviation is reported, the run is retained, and no third dataset is selected after results are seen.Source: Appendix G · p. 31

Sensitivity without pendigits

  • Without pendigits and SVHN, the multiclass subgroup has 2 false improvements among 43 accepted updates (4.65%) but fails the bound, LOCO, and power checks.The gap is reported as a limitation, not a confirmed violation.Source: Appendix G · p. 32

Volkert supplement

  • The exploratory Volkert supplement pooled with E and E2 meets every criterion across 34 datasets, with 648 accepted updates and 9 false improvements (1.39%).Because Volkert was frozen after confirmatory runs, this pooled result leaves the confirmatory claim unchanged.Source: Appendix G · p. 32

Collection outside the validated class range

  • The exploratory collection outside the validated 7-to-10-class range meets the criterion jointly, but its K=3 and K=6 groups fail.The result varies with the number of classes.Source: Appendix G · p. 32

APPENDIX H DETAILED RESULT TABLES

Appendix H provides detailed tables behind Sections III and IV, covering acceptance-rule results, hypothesis tests, component analyses, conditions of use, and clinical consequences. Its appendix tables define a one-sided 95% upper bound on the false-improvement rate and use the pre-registered criterion from Table 6.

Main results and hypothesis tests

  • In (A), among 459 naive-rule acceptances with validation gains 0 < ˆ∆≤0.01, 164 were false improvements, while BRIDGE accepted none.The passage identifies this comparison as limited to small validation gains.Source: Appendix H · p. 33, Table H1 · p. 33

Acceptance-rule comparisons

  • Among accepted updates, the cross-fit lower bound is reached in 80%–96% of cases across settings (A)–(E), with shortfalls and false improvements tabulated separately.The reported shortfall averages are 0.0001–0.0011 across all accepted updates; the candidate selected in each fold was not stored, so choice stability cannot be assessed.Source: Appendix H · p. 35
  • The post hoc random-selection control compares BRIDGE with a random draw of the same size from naive-rule acceptances, rather than estimating risk on future tasks.A within-dataset version preserves BRIDGE’s accepted count in each dataset or clinical task.Source: Table H4 · p. 34, Appendix H · p. 36

Component analysis

  • The appendix tables cover candidate-space sizes of 173, 1,825, and 11,249 configurations across collections (A)–(E), with the OpenML panel limited to binary datasets.The multiclass datasets in (D) and (E) are excluded because no equally dense multiclass grid was pre-registered.Source: Table H7 · p. 34, Appendix H · p. 36

Conditions of use

  • Across tested metrics, training sizes, and multiclass tasks, the criterion is met except with feature selection and SVC, where LOCO exceeds 10%.The imbalance study covers five large binary datasets; the base-learner study uses 1,400 fresh synthetic tasks.Source: Table H6 · p. 34, Appendix H · p. 36

Clinical consequences

  • Across clinical tasks, mean net benefit after the update exceeds both treat none and treat all; three named NHANES/ACTG tasks first reach that level after updating.The values are exploratory means over accepted updates for each task.Source: Table H14 · p. 36, Appendix H · p. 37

DATA AVAILABILITY STATEMENT

The statement identifies the public data sources, code, stored outputs, and archived release. GOSSIS is available only through its original Kaggle competition, so the repository provides its downloader and loader rather than raw data.

Data and code

  • Code, per-trial outputs, and implementation details are available in the bridge-gate repository, and the paper’s release is archived on Zenodo.A single script rebuilds the reported tables from stored per-trial outputs without training a model.Source: DATA AVAILABILITY STATEMENT · p. 37
  • The datasets are publicly available from OpenML, UCI, scikit-learn, CDC NHANES/BRFSS, Mexico’s open Secretaría de Salud releases, and the OpenML mammography source.GOSSIS is available only through its original Kaggle competition; the repository contains a downloader and loader, not raw data.Source: DATA AVAILABILITY STATEMENT · p. 37

Related papers

Loading 2610.29515v1…