PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepGOF-1: A Pretrained Convolutional Goodness-of-Fit Test for Logistic Regression with a Computable Consistency Certificate

Ebrahim Khaled Ebrahim

arXiv:2609.29575v1stat.MEstat.COstat.ML

TL;DR

Small-sample logistic-regression goodness-of-fit tests can miss nominal levels, and combining them can worsen the drift. DeepGOF-1 pairs a pretrained frozen network with bootstrap rank calibration and a computable alternative-specific consistency condition. It reports stable levels across its design grid and benchmark, while its overall power rank is seventh of thirteen.

  • Problem

    At sample sizes of fifty to a few hundred with modest event counts, standard logistic-regression goodness-of-fit tests can drift from their stated levels, and combining five tests can worsen the drift.

    Source: §1 · p. 2

  • Method

    DeepGOF-1 trains a convolutional score once on simulated departures and calibrates it with the analyst’s parametric-bootstrap rank p-value.

    Source: §1 · p. 3

  • Results

    58 of 60 pre-declared design-grid cells fall within the nominal level band; on the benchmark, DeepGOF-1 is most stable across thirteen methods, beats every partition test at every sample size, and ranks 7 of 13 overall.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The rank p-value makes level a property of calibration rather than learning, and one forward pass on the frozen weights checks the consistency condition for a named alternative.

    Source: §8 · p. 15

    The consistency theorem covers fixed alternatives, not local asymptotic power under contiguous alternatives, where the paper’s recession argument supplies no information.

    Source: §3.2 · p. 9

Abstract

from arXiv · show

Goodness-of-fit tests for logistic regression are least reliable where they are most needed: at small samples their levels drift from the nominal one, and combining them worsens the drift. We propose a test whose statistic is a convolutional network, trained once on simulated departures, that reads misfit as a picture: a grid of standardized residuals over covariate ranks. The analyst never trains. The network ships frozen, and the p-value is the rank of the observed score within the analyst's own bootstrap, so the level is a property of the calibration rather than of what the network learned. We prove exactness under pivotality, asymptotic exactness without it, and a consistency theorem whose key condition is computable from the frozen weights in one forward pass, giving a per-alternative certificate; we also measure the test's blind cone. On a pre-declared sixty-cell grid the deployed level stays in the nominal band in fifty-eight cells, while that grid's power criterion failed; on a published benchmark it is the most stable of thirteen levels across settings and sample sizes, and the test outpowers every partition test at every sample size while ranking seventh overall. A bank-failure application and a versioned release close the paper.

1 Introduction

DeepGOF-1 pairs a once-trained, frozen score with bootstrap calibration for logistic-regression goodness-of-fit. The paper develops a computable consistency certificate and reports stable levels alongside qualified power results.

Introduction

  • The paper targets a gap: existing approaches do not combine exact finite-sample control for an arbitrary frozen learned statistic with a composite regression null and estimated nuisance parameters.The parametric bootstrap is only asymptotically valid once parameters are estimated (Genest and Rémillard, 2008).Source: §1 · p. 2

First, the mechanism (Section 2)

  • The test uses a once-trained frozen statistic and the analyst’s parametric-bootstrap rank p-value, so level control is a property of calibration rather than learning.Training uses bootstrap replicates as its null class, matching the object used for deployed calibration.Source: §1 · pp. 2–3

Second, the theory (Section 3)

  • A computable consistency condition tests whether the frozen network’s slope is positive in an alternative’s limiting map direction, while the blind cone captures directions where it fails.The paper also notes a representation limit: misfit integrating to zero within every cell defeats any statistic computed from the map.Source: §1 · p. 3

Third, the evidence (Sections 4–6)

  • 58 of 60 pre-declared design-grid cells fall within the nominal level band; on the benchmark, the test is most stable and beats every partition test at every sample size, ranking 7 of 13 overall.Its mean benchmark level is 0.0506; the overall power rank is behind covariate-space, smoothing, and link tests and one oversized combination.Source: §1 · p. 3

Fourth, the release (Section 7)

  • DeepGOF-1 ships with its training corpus, generator, code, benchmark harness, replicate p-values, and pre-declared evaluation criteria for versioned comparison.The paper closes with a bank-failure application on public FDIC data.Source: §1 · p. 3

2 The test

The test turns logistic-regression residuals into a rank-based grid that a frozen convolutional network scores. The analyst calibrates that score with parametric-bootstrap refits rather than training a network.

2.1 The residual map

  • The 6 × 6 residual map bins observations by the empirical ranks of the two strongest covariates, with each cell holding a standardized residual sum.Cells are approximately standard normal under a correct model; rank-based axes make the map invariant to monotone covariate rescaling, while departures form spatial patterns.Source: §2.1 · p. 4, Figure 1 · p. 4
  • The pre-declared K = 6 grid balances standardization with spatial detail, while selecting the two largest |β̂j|σ̂j covariates leaves a third covariate off-grid.The paper says a 12 × 12 grid leaves too few observations per cell at target sample sizes, whereas K = 4 discards usable detail.Source: §2.1 · p. 4

2.2 The pretrained score and the deployed test

  • A 18,273-parameter convolutional network is trained once on simulated null and departure maps, then shipped frozen; its null training maps are bootstrap replicates.The network combines global max pooling for localized bumps with global mean pooling for diffuse trends.Source: §2.2 · p. 5, Figure 2 · p. 6
  • At deployment, the analyst fits the model, constructs and scores the map, then simulates and refits bootstrap datasets to obtain the score distribution and rank p-value.A failed refit is scored +∞ and counts against rejection; the analyst runs no training software.Source: §2.2 · p. 5
  • With B + 1 forward passes and B logistic refits, runtime is linear in B and dominated by refits; the released implementation is pure base R.For n = 150, a complete test takes 1.6 seconds on one desktop core; forward passes account for 0.4 seconds.Source: §2.2 · pp. 5–6

3 Theory

The theory establishes finite-sample exactness under pivotality and asymptotic validity without it, then gives a computable condition for consistency against fixed alternatives. It also characterizes blind spots and limits of the guarantee.

3 Theory

  • The consistency result requires regularity A1–A3 plus a nonzero misfit footprint and positive certificate; A6 is needed only for Proposition 1.The stated conditions include i.i.d. data, a unique Kullback–Leibler projection, and a unique top-two covariate set.Source: §3 · p. 7

3.1 Validity

  • Under pivotality, the rank p-value has size at most ⌊α(B + 1)⌋/(B + 1) ≤ α at every sample size, with equality when the score law is atomless.Pivotality means the score law is the same under every null parameter for a fixed covariate configuration.Source: §3.1 · p. 7
  • Without pivotality, under A1–A3 and A6 the test is asymptotically exact at its B + 1 attainable levels, and p ⇝ Unif(0, 1) if B → ∞.The result is first-order validity; the paper claims no Edgeworth refinement rates because responses are lattice and the score is piecewise affine.Source: §3.1 · p. 7

3.2 Certified consistency

  • The recession function exists for networks built from affine maps, rectifiers, and componentwise maxima; it equals the same network with offsets removed.This makes the alternative-specific consistency condition evaluable from the frozen weights in one forward pass, before data are seen.Source: §3.2 · pp. 7, 9
  • For a fixed alternative, consistency follows when the computable recession slope S∞(µ/s) is positive, provided A1–A5 hold, including a nonzero map footprint µ.The deployed test's power then tends to one at any attainable level.Source: §3.2 · p. 8
  • For DeepGOF-1, no 250,000 sampled directions fell in the blind cone, but minimization found a nonzero blind direction with slope −0.281.That direction is a checkerboard at the cell scale; the cone is characterized as sub-resolution oscillation shared as a failure mode by binned statistics.Source: §3.2 · p. 8
  • If misspecification integrates to zero within every population cell, no fixed locally bounded statistic of the map can be consistent through it.Under nondegeneracy conditions, the deployed test's power then converges to a limit strictly below one.Source: §3.2 · p. 8
  • The theorem covers fixed alternatives but gives no rate or local asymptotic power; the recession argument does not resolve contiguous alternatives.Nondifferentiable ReLU and max-pooling boundaries prevent the usual expansion without further assumptions on the null-map activation pattern.Source: §3.2 · p. 9

4 Simulation I: the level, pre-declared

The pre-declared grid found DeepGOF-1’s deployed level inside the nominal band in 58 of 60 cells, but its power criterion failed. Power was at parity with the strongest published test on two-covariate designs and lower on three-covariate designs.

4 Simulation I: the level, pre-declared

  • 58 of 60 cells fell within the pre-declared level band, with excursions of .070 and .029 that did not recur across training corpora.The grid crossed five sample sizes, four designs, and three corpora, with 2,000 null replicates per cell; the band was [.035, .065].Source: §4 · p. 9
  • At nominal .05, the five-test Cauchy combination reached size .066 on 15%-event designs, while individual tests drifted in both directions.Evaluation criteria were frozen before confirmatory data were generated, and per-replicate p-values were deposited with the release.Source: §4 · p. 9
  • The pre-declared power criterion failed; against the hindsight-best published test, mean power was −.03 overall, at parity on two-covariate designs and −.06 to −.08 lower on three-covariate designs.The per-cell comparator was the best of five published tests, favoring the classical side; the three-covariate deficit is identified as the representation limit of Corollary 2.Source: §4 · p. 9, Figure 3 · p. 10

5 Simulation II: a published benchmark, thirteen methods

On the extended Liu et al. (2024) benchmark, DeepGOF-1’s level varied least across blocks and its power beat every partition-based test at every sample size. It ranked seventh of thirteen overall, while omitted tests also solved Setting 2.

5 Simulation II: a published benchmark, thirteen methods

  • DeepGOF-1’s block-to-block size standard deviation was .004, versus .012 for le-Cessie and .011 for the projection test, despite near-nominal average sizes for all three.The table reports empirical size at nominal .05 and matched-level power by sample size, averaged over four settings and departure strengths.Source: Table 2 · p. 11, §5 · p. 12
  • DeepGOF-1 beat every partition-based test at every sample size, with margins over Hosmer–Lemeshow of +0.13, +0.15, and +0.06 at n = 50, 100, and 500.It trailed covariate-space and smoothing families, most narrowly at small n.Source: Table 2 · p. 11, §5 · p. 12
  • DeepGOF-1 ranked 7th of 13 overall; its rank declined from 5th at n ≤100 to 8th at n = 500 as tests retaining more covariate information overtook it.The ten-test Cauchy combination also ranked ahead, but its size of 0.220 made it an unfair power comparison.Source: §5 · p. 12
  • On Setting 2, le Cessie and van Houwelingen, Tsiatis, and Xie also solved the setting, with rejection-rate rises of +.95, +.83, and +.82; Stute and Zhu did not (+.30).The benchmark reproduced Liu et al. (2024)’s four data-generating processes at n ∈{50, 100, 500}, with all arms randomized to level .05 on held-out nulls.Source: §5 · pp. 11–12

6 Application: bank failures after 2008

The FDIC application tested level transfer on real bank covariate geometry and compared model-adequacy calls across state cells. DeepGOF-1’s deployed level was mostly within the pre-declared band, while the combination and partition family disagreed on Florida and Illinois.

6 Application: bank failures after 2008

  • The application used 8,395 FDIC-insured banks from 2008-12-31, including 397 failures during 2009–2011, and fitted five-covariate models within state cells with at least fifteen failures.Georgia was selected as the primary cell before real data were downloaded.Source: §6 · p. 12
  • The five-test Cauchy combination exceeded .065 in four of six cells, reaching .082 at 431 banks with 16 failures, while its size was .054 in pre-declared Georgia.Its individual component tests had sizes between .017 and .057 in Georgia.Source: §6 · pp. 12, 14
  • For Florida and Illinois, the combination rejected the fitted model at p-values .003 and .026, while every partition test retained both models and DeepGOF-1 rejected no cell.DeepGOF-1’s adequacy p-values ranged from .185 to .905; the authors describe its calls as consistent with the representation limit of Corollary 2.Source: §6 · p. 14
  • DeepGOF-1’s deployed size ranged from .0445 to .0675 across six state cells; five cells were within the pre-declared band, and the exception had sixteen failures.The experiment used 2,000 null outcomes per cell simulated on the real design matrix, with no failed refits.Source: §6 · p. 14, Table 3 · p. 14

7 DeepGOF-1 as a versioned object

DeepGOF-1 is released as a versioned, reproducible object with its implementation, weights, training materials, evaluation harness, and underlying p-values. The release preserves fixed criteria and documents directions for a successor.

7 DeepGOF-1 as a versioned object

  • The release includes the base R deployed implementation, frozen weights, training corpus and generator, training and export code, benchmark harness, and per-replicate p-values.The forward pass was verified against the training framework to 5 × 10−11, and the row-major/column-major porting trap is documented.Source: §7 · p. 14
  • A successor can be trained by editing one file, with the bootstrap-replicate null class and departure prior explicit and versioned.The pre-declared evaluation criteria and per-replicate p-values are included so DeepGOF-2 can be judged against the same frozen goalposts.Source: §7 · pp. 14–15
  • The release records five measured directions for a successor, including one attempted direction that failed and one probe ruled an invalid measurement.The paper presents DeepGOF-1 as a first model rather than a finished method.Source: §7 · p. 15

8 Discussion

The discussion frames calibration, rather than learning, as the source of a learned test’s level control, and presents a computable consistency certificate. It also defines the test’s scope and identifies a model-agnostic recipe for future tests.

8 Discussion

  • The bootstrap rank gives an arbitrary frozen statistic exact level under pivotality and asymptotic validity at attainable levels for a composite regression null with estimated coefficients.The discussion distinguishes this combination from existing exact-validity methods aimed at other problems.Source: §8 · p. 15
  • For a named alternative, one forward pass checks whether consistency applies by evaluating the network’s recession slope at that alternative’s limiting map direction.The blind cone is measured and characterized as sub-resolution oscillation.Source: §8 · p. 15
  • The test dominates its own partition-test family everywhere and is competitive with all tests at n ≤100, but larger-sample tests can exploit covariate information its 6 × 6 map discards.The discussion identifies that representation limit as proved and measured.Source: §8 · p. 15
  • Power on the pre-declared design grid missed its criterion, and changing the training departure-prior moved the same architecture’s margin against the same rival from +.09 to −.07.The discussion treats learned-test power as a property of the test–departure-prior pair and notes that the release is versioned.Source: §8 · p. 15
  • The proposed model-agnostic recipe renders misfit as a statistic-valued image, trains against the bootstrap null, calibrates by rank, and certifies consistency through the recession function.The discussion says each ingredient generalizes beyond logistic regression.Source: §8 · p. 15

Use of generative artificial intelligence

Generative AI tools supported implementation and exposition, not research design. The author states that AI contributed no research claims and takes responsibility for the paper.

Use of generative artificial intelligence

  • Generative AI was used to speed implementation and exposition for a large simulation study, not for research design.The disclosure says the author formulated the method, derived and checked every proof, and verified reported numbers against deposited results.Source: Use of generative artificial… · p. 16
  • Claude was used for simulation, benchmark, and figure code, prose drafting and editing, and internal manuscript reviews; Gemini provided one independent review pass.Source: Use of generative artificial… · p. 16
  • The disclosure states that no AI system contributed a method, proof, claim, or conclusion, and that the author takes full responsibility for the content.Source: Use of generative artificial… · p. 16

Data availability statement

The paper reports public data and a broad reproducibility archive, alongside an R package that distributes the deployable test and comparison tools.

Data availability statement

  • All data are public, and the FDIC data can be rebuilt from two keyless API calls using the deposited script.Source: Data availability statement · p. 16
  • The Zenodo archive includes per-replicate p-values, the training corpus and generator, frozen weights, code, and pre-declared evaluation criteria.The archive is also identified by DOI 10.5281/zenodo.22113220 and the DeepGOF-1 GitHub repository.Source: Data availability statement · p. 16
  • The deployable test is distributed as deepgof1() in the ebrahim.gof R package, whose run.all.gof() function runs the compared classical-test battery.All computation used R 4.4.1.Source: Data availability statement · p. 16
Loading 2609.29575v1…