PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability-Aware Checkpoint Selection for Domain Generalization

Jinshi Liu, Jiahao Li, Pan Liu, Yanfeng Li, Rui Qian, Zhao Tong, Yue Sun, Tao Tan

arXiv:2609.39934v1cs.LGcs.CV

TL;DR

Checkpoint selection for domain generalization must use held-out source data rather than target-domain validation, but accuracy alone does not measure probability quality. AC ranks reliability among near-best-accuracy checkpoints; on post-development runs it lowered mean target reliability errors relative to Source-Acc, while target-accuracy preservation is not established.

  • Problem

    Domain generalization requires choosing a checkpoint without target-domain validation, while source accuracy alone does not measure predictive probability quality.

    Source: §1 · p. 1

  • Method

    AC filters checkpoints within a tolerance of best source-validation accuracy and ranks eligible checkpoints by source reliability, without target data, additional training, or weight averaging.

    Source: §1 · p. 2

  • Results

    On 360 post-development runs, AC-NC/D∞ lowered mean target ECE, CwECE, and NLL; mean accuracy changed +0.2133 pp, with intervals spanning losses and gains.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    In the evaluated trajectories, similar source-validation accuracy can coexist with different probability quality, and reliability-aware reselection improves mean target probability quality relative to Source-Acc.

    Source: §5 · p. 9

    The evidence does not establish superiority or equivalence of joint NLL–CwECE ranking over accuracy-constrained single-objective ranking.

    Source: §4.4 · p. 7

Abstract

from arXiv · show

Checkpoint selection in domain generalization often relies on source-validation accuracy, yet the selected checkpoint need not provide reliable probabilities on unseen target domains. Source-target distribution shifts can alter accuracy rankings, while accuracy alone does not measure predictive probability quality. We identify an empirical selection opportunity within fixed training trajectories: reselecting among checkpoints with near-optimal source accuracy can improve mean target probability quality with small observed changes in mean target accuracy. We study accuracy-constrained reliability selection (AC), which retains checkpoints within a tolerance of the best source-validation accuracy and ranks them by source reliability. Our reference rule aggregates within-set normalized negative log-likelihood (NLL) and class-wise calibration error (CwECE) using $D_\infty$. AC uses no target data and requires neither additional training nor weight averaging. We evaluate five domain generalization training algorithms on three benchmarks, using PACS to develop the objectives and a 0.5-percentage-point tolerance. In exploratory aggregation comparisons on 360 OfficeHome and TerraIncognita runs, the reference rule reduces mean target soft-bin squared-gap ECE and CwECE by 0.240% and 0.182%, respectively, and NLL by 0.030 relative to Source-Acc. Mean target accuracy changes by +0.213 percentage points. These results identify opportunities for reliability-aware reselection, while the additional benefit of joint over single-objective ranking remains unresolved.

1 INTRODUCTION

Source-validation accuracy can miss differences in checkpoint probability quality, motivating reliability ranking among near-best-accuracy checkpoints. AC applies this selection within fixed training trajectories; post-development evaluations report improved mean target reliability with a small observed mean accuracy change.

1 INTRODUCTION

  • Domain generalization deploys models on unseen target domains, requiring checkpoint choice without target-domain validation; Source-Acc selects the highest mean source-validation accuracy.The passage attributes the Source-Acc protocol to Gulrajani & Lopez-Paz (2021).Source: §1 · p. 1
  • Accuracy alone does not measure probability quality, and reliability-only selection can choose low-accuracy checkpoints; near-best-accuracy checkpoints can still differ in reliability.The motivation cites calibration and uncertainty degradation under distribution shift (Guo et al., 2017; Ovadia et al., 2019), and accuracy-filtered calibration selection (Wald et al., 2021).Source: §1 · p. 1
  • AC retains checkpoints within δ percentage points of the best source-validation accuracy and ranks them by source reliability, without target data, additional training, or weight averaging.The reference rule combines normalized NLL and CwECE using D∞.Source: §1 · p. 2
  • On post-development runs, AC-NC reduced mean target ECE, CwECE, and NLL relative to Source-Acc, with a +0.21-percentage-point mean target-accuracy change.The D∞ aggregation was not fixed before these evaluations, so aggregation comparisons remain exploratory.Source: §1 · p. 2
  • Prediction-level case studies report lower error rates above a fixed confidence threshold and, on subsets of shared errors, lower incorrect-prediction confidence and higher true-label probabilities.The contribution statement also describes paired evaluations across three benchmarks and five training algorithms.Source: §1 · p. 2

2 RELATED WORK

Prior domain-generalization selection methods filter or score candidates using validation accuracy, risk, discrepancy, or shifted validation data. AC instead studies reliability-aware selection among saved checkpoints within a fixed trajectory, without fitting a calibrator or constructing new weights.

2 RELATED WORK

  • Lyu et al. (2023) combine classification risk with feature-space domain discrepancy after filtering candidates by validation loss; Mixup-guided selection uses shifted validation data (Lu et al., 2023).PAIR-s scores preference-aware ERM/OOD objectives and filters by validation accuracy across runs (Chen et al., 2023).Source: §2 · pp. 2–3
  • Wald et al. (2021) select the lowest mean source-domain ECE subject to an in-domain accuracy threshold and additionally recalibrate candidates; AC applies accuracy filtering to saved checkpoints without post-hoc calibration.AC measures its threshold relative to the trajectory's best source accuracy and normalizes NLL and CwECE within the feasible set.Source: §2 · p. 3
  • Unlike SWAD, which averages weights over an overfit-aware trajectory interval, AC returns one existing checkpoint without weight averaging or prediction ensembling.EoA ensembles moving-average models from independent runs (Arpit et al., 2022); SWAD is attributed to Cha et al. (2021).Source: §2 · p. 3
  • Temperature scaling adjusts probabilities without changing class predictions, while AC selects a saved classifier without fitting a calibrator and may change probabilities and class predictions.The passage cites Guo et al. (2017) for temperature scaling, Ovadia et al. (2019) for calibration deterioration under shift, and Gong et al. (2021) for multi-source temperature scaling.Source: §2 · p. 3

3 METHOD

The method selects checkpoints using source-validation information alone: source accuracy defines eligibility, and reliability ranks eligible checkpoints. Its source-accuracy control has explicit limits and does not guarantee target-domain performance.

3.1 PROBLEM SETUP AND SOURCE-ONLY LIMITATION

  • The trajectory is a finite, nonempty set of saved checkpoints, and a source-only selector may use its source-validation statistics but not target observations.Source-Acc chooses the earliest checkpoint maximizing mean source-validation accuracy.Source: §3.1 · p. 3
  • With unrestricted target distributions, no source-only selector can choose a target-accuracy-maximizing checkpoint with probability one for every target domain.This applies to Source-Acc and AC, but does not preclude useful source-side selection under structured shifts.Source: §3.1 · p. 4

3.2 ACCURACY-CONSTRAINED RELIABILITY SELECTION

  • AC separates source-accuracy eligibility from reliability ranking, retaining checkpoints whose empirical source accuracy is within δ of the trajectory maximum.It normalizes each reliability objective within the feasible set and selects by an aggregate score.Source: §3.2 · pp. 4–5
  • The reference objective set is NLL and CwECE; AC normalizes these source-validation errors within the feasible set and D∞ minimizes the largest normalized error.The rule breaks ties by higher source accuracy and then earlier training step; D1 and D2 are alternative aggregations.Source: §3.2 · pp. 4–5

3.3 FINITE-SAMPLE SOURCE-ACCURACY CONTROL

  • Under independent trajectory generation and i.i.d. source-validation samples, the finite-sample guarantee controls population source accuracy, not reliability or target performance.Reusing validation samples for reliability ranking is covered, but candidate generation changed by validation feedback is not.Source: §3.3 · pp. 5–6
  • Source reliability ordering need not transfer to an unseen target; the stated conditional ordering result requires a monotone source–target relation and bounded checkpoint-dependent residual variation.The target-dependent quantities required by that result are unobserved and are not inputs to AC.Source: §3.3 · p. 6

4 EXPERIMENTS

Across five domain-generalization algorithms and three benchmarks, the experiments test accuracy-constrained reliability selection within fixed training trajectories. On post-development runs, the reference rule improves mean target probability quality relative to Source-Acc, while results do not establish superiority of joint over single-objective ranking.

4.1 EXPERIMENTAL SETUP

  • The evaluation covered five algorithms, three benchmarks, and 540 fixed trajectories, each with 51 checkpoints; PACS developed the objectives and tolerance before 360 OfficeHome/TerraIncognita runs.The aggregation rule was not fixed before those post-development evaluations, so aggregation comparisons remain exploratory.Source: §4.1 · p. 6

4.2 MAIN RESULTS AND HETEROGENEITY

  • Gains were heterogeneous: mean CwECE and NLL decreased for all five algorithms on OfficeHome, three on TerraIncognita, and two on PACS.Across 540 pooled runs, 118 selections lost target accuracy, including 76 (14.1%) losing at least one pp; the fifth percentile was −3.0 pp.Source: §4.2 · p. 6

4.3 PREDICTION-LEVEL DIAGNOSTICS

  • Among 642 shared OfficeHome Art/ERM errors, AC-NC raised true-label probability in 55.9% and lowered incorrect-prediction confidence in 57.9%; >90%-confidence error fell from 7.09% to 5.51% with lower coverage.These post-hoc cases illustrate probability shifts, not general prediction-level improvement.Source: §4.3 · p. 6, Figure 3 · p. 7

4.2 MAIN RESULTS AND HETEROGENEITY

  • Across 360 post-development runs, AC-NC/D∞ lowered target ECE, CwECE, and NLL; mean accuracy changed +0.2133 pp, with an interval spanning losses and gains.The results support improved mean probability quality, not target-accuracy preservation.Source: §4.2 · p. 6, Table 1 · p. 7

4.4 WHAT DRIVES CHECKPOINT RESELECTION?

  • On OfficeHome, unconstrained ECE/CwECE selection chose step 0 in 94.4%/95.6% of runs, with target accuracy of 2.55%/1.91%; at δ = 0.5, AC-NC differed from Source-Acc in 264/391 eligible multi-checkpoint runs.The 264 runs are 67.5% of the 391 runs with multiple eligible checkpoints.Source: §4.4 · p. 7
  • Pooled comparisons establish neither superiority nor equivalence of joint ranking; aggregation comparisons establish neither a preferred distance nor a benefit from adding ECE to NC.AC-NC had the best PACS means among four reliability-ranking rules, but these development results do not establish post-development superiority; D1 and D∞ disagreed in 19/540 selections, and NEC’s larger pooled NLL reduction was sensitive to one OfficeHome/IRM run.Source: §4.4 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

4.5 TOLERANCE AND ADDITIONAL COMPARISONS

  • At δ = 1.0, pooled reliability errors were lower than at δ = 0.5, but paired accuracy intervals for both alternative tolerances and the NLL interval for δ = 1.0 included zero.Dataset-level patterns identified no uniformly preferable tolerance.Source: §4.5 · p. 9, Table 4 · p. 9
  • Separate comparisons found checkpoint-SWAD with BatchNorm recalibration outperformed AC-NC in mean accuracy and lowered all three reliability errors, while LODO reliability ranking reduced fixed-hyperparameter ECE without a resolved accuracy difference.In the separate temperature-scaling study, selectors had similar ECE after scaling, while AC-NC had slightly higher CwECE and NLL; gains were not consistently additive.Source: §4.5 · p. 9, Table 5 · p. 9

5 CONCLUSION

In evaluated domain-generalization trajectories, checkpoints with similar source-validation accuracy can differ in target probability quality. Reliability-aware reselection improves mean target probability quality over Source-Acc, but the evidence does not establish that joint NLL–CwECE ranking is superior to accuracy-constrained single-objective ranking.

5 CONCLUSION

  • Within evaluated domain-generalization trajectories, checkpoints with similar source-validation accuracy differed in target probability quality.This identifies a reliability-selection opportunity not explicitly exploited by Source-Acc.Source: §5 · p. 9
  • Reliability-aware reselection improved mean target probability quality relative to Source-Acc on the original post-development cohort.Source: §5 · p. 9
  • The evidence does not establish superiority of the joint NLL–CwECE reference over accuracy-constrained single-objective ranking.Source: §5 · p. 9

A QUALITATIVE CHECKPOINT CASE STUDIES

The case studies show how reliability-selected checkpoints can improve probability quality even when they retain the same incorrect top-1 prediction. The section also specifies the main-cohort reliability metrics and their averaging conventions.

A QUALITATIVE CHECKPOINT CASE STUDIES

  • For the Lampshade image, both checkpoints predict Flowers, but reliability selection lowers its probability from 0.9528 to 0.2762 and raises the true-class probability from 0.0091 to 0.2418.The top-1 error remains, while more probability mass is assigned to the correct class.Source: A QUALITATIVE CHECKPOINT CASE STUDIES · p. 12
  • For the Desk Lamp image, the correct-class probability rises from 0.9304 to 0.9591, while the confidence–accuracy gap decreases from 0.226 to 0.001.Example-level NLL decreases from 0.07 to 0.04, and no single 16 × 16 patch dominates the prediction.Source: A QUALITATIVE CHECKPOINT CASE STUDIES · p. 12
  • For the Computer image, both checkpoints incorrectly predict Laptop, but reliability selection lowers Laptop probability from 0.9944 to 0.8085 and reduces example-level NLL from 8.53 to 3.22.The true-class Computer probability rises from approximately 0.0002 to 0.0398, without changing the top-1 decision.Source: A QUALITATIVE CHECKPOINT CASE STUDIES · p. 12
  • The examples illustrate lower confidence in shared errors and greater retention of true-class probability, but do not imply that every misclassification is corrected.Aggregate calibration and NLL outcomes are evaluated separately in the main paper.Source: A QUALITATIVE CHECKPOINT CASE STUDIES · p. 12

B.1 MAIN-COHORT RELIABILITY METRICS

  • The main-cohort metrics include NLL and Gaussian soft-bin calibration errors, with class-wise terms averaged equally across classes and source-domain metrics averaged equally across domains.The class-wise terms use all validation examples, and empty-weight terms contribute zero.Source: §B.1 · pp. 13–14

B.2 IMPLEMENTATION CONVENTIONS

The supplement distinguishes the soft-bin calibration conventions used in the main cohorts from hard-bin metrics used for LODO. AC also normalizes objectives within each accuracy-feasible set, so changing the tolerance changes both eligibility and ranking scale.

B.2 IMPLEMENTATION CONVENTIONS

  • Gaussian soft-bin metrics use equally spaced centers and a run-fixed bandwidth; each run uses the same settings across checkpoints, and the selector consumes logged scores without rebinning.The exporter defaults to B = 15 and h = 0.1 unless run-specific settings override them.Source: §B.2 · p. 14

B.3 LODO HARD-BIN CALIBRATION METRICS

  • LODO uses hard-bin absolute calibration gaps, with class-frequency-weighted CwECE; these metrics are distinct from the main cohort’s squared-gap, uniformly averaged soft-bin metrics.The results are compared only within their respective panels.Source: §B.3 · p. 14

B.4 FEASIBLE-SET NORMALIZATION

  • AC computes reliability-objective extrema within the current accuracy-feasible set, so changing δ changes both the candidate set and the ranking scale.Min–max normalization can amplify small raw differences when the objective range is small; the specified nonzero offset can also break an exact two-checkpoint tie under D∞.Source: §B.4 · pp. 14–15

C PER-DATASET AND PER-ALGORITHM RESULTS

The section reports target-accuracy variability across pooled runs and contrasts checkpoint reselection with checkpoint-SWAD on OfficeHome. It also describes a dataset–algorithm breakdown, while cautioning that source-side tolerance does not bound target accuracy.

C PER-DATASET AND PER-ALGORITHM RESULTS

C.1 DISTRIBUTION OF TARGET-ACCURACY CHANGES

  • In the pooled 540-run analysis, AC-NC loses target accuracy in 118 runs; 76 runs (14.1%) lose at least one percentage point, and the empirical fifth percentile of the change is −3.0 percentage points.The pool includes PACS development and is descriptive; a positive pooled mean does not imply per-run accuracy preservation.Source: §C.1 · p. 15

D.1 CHECKPOINT-SWAD ON OFFICEHOME

  • Checkpoint-SWAD on OfficeHome uses 180 runs and averages LossValley-selected endpoint parameters, but sparse saved endpoints make it an approximation to dense online SWAD.The protocol evaluates 51 endpoints and uses no target-domain observations in valley construction, averaging, or BatchNorm processing.Source: §D.1 · p. 16
  • On OfficeHome, checkpoint-SWAD with BatchNorm recalibration has higher mean target accuracy and lower mean ECE, CwECE, and NLL than both Source-Acc and AC-NC.These comparisons concern the complete averaging-and-BatchNorm pipeline, not weight averaging alone.Source: §D.1 · p. 16, Table D.1 · p. 16
  • Without BatchNorm recalibration, checkpoint-SWAD has mean target accuracy of 48.36 versus 62.62 and mean NLL of 9567.3991 versus 2.0026.The severe cases are attributed to mismatches between averaged parameters and the first selected checkpoint’s buffers; this is a buffer-mismatch diagnostic.Source: §D.1 · p. 16

D.2 LODO VALIDATION DETAILS

The supplement compares LODO validation, alternative selectors, and ablations of AC’s distance and reliability objectives. Results support some reliability improvements in specific comparisons, but do not establish a general advantage for joint NLL+CwECE ranking.

D.2 LODO VALIDATION DETAILS

  • The LODO study uses pseudo-target validation from another excluded domain; the true outer target is never used for checkpoint or hyperparameter selection.It covers ERM and VREx on OfficeHome and TerraIncognita, using hard-bin calibration metrics that are not numerically comparable with the main cohort’s soft-bin metrics.Source: §D.2 · pp. 16–17
  • In fixed-hyperparameter LODO comparisons, AC and LODO-Acc have no resolved accuracy difference (+0.196 percentage points; 95% CI: [−0.755, +1.291]), while AC has higher NLL and ECE.AC-LODO reduces ECE relative to LODO-Acc, but the evidence does not show accuracy improvement or uniform improvement across reliability metrics; the comparison is limited to ERM and VREx on OfficeHome and TerraIncognita.Source: §D.2 · pp. 17–18, Table D.2 · p. 17, Table D.3 · p. 17

D.3 VREX-ONLY PAIR-S-STYLE COMPARISON

  • The PAIR-s-style diagnostic is restricted to VREx and is not a reproduction of PAIR under its original training setup.Table D.4 covers 36 runs per dataset.Source: §D.3 · p. 18, Table D.4 · p. 18

E ABLATIONS AND SELECTION DIAGNOSTICS

  • The ablation section compares AC’s distance, objective-set, and component choices; target-domain quantities serve only as evaluation diagnostics and never enter checkpoint selection..Source: E ABLATIONS AND SELECTION DIAGNOSTICS · p. 18

E.1 POST-DEVELOPMENT EVALUATION AND DIRECT DISTANCE COMPARISONS

  • On OfficeHome and TerraIncognita, all three tested distances reduce mean target calibration errors and NLL, while their accuracy intervals include zero; distance comparisons remain exploratory.The direct D∞-versus-D1 comparison uses the same objectives and tolerance.Source: §E.1 · p. 18, Table E.1 · p. 19, Table E.2 · p. 19

E.2 RELIABILITY-OBJECTIVE AND DISTANCE ABLATIONS

  • The NC results are close across D1, D2, and D∞, while objective sets with similar source-accuracy gaps have different target-reliability changes, particularly for NLL.The archived Pareto filtering changed no selected checkpoint in the reported objective-set, distance, or tolerance ablations.Source: §E.2 · p. 19, Table E.3 · p. 19

E.3 SINGLE-OBJECTIVE COMPONENT ABLATION

  • By dataset, AC-NC outperforms both single-objective rules across the reported target metrics on PACS, while results are close to AC-CwECE on OfficeHome and favor single-objective rules on TerraIncognita.All paired 95% target-metric intervals include zero, establishing neither superiority nor equivalence of the joint rule.Source: §E.3 · p. 20, Table E.6 · p. 20, Table E.7 · p. 21

E.3.1 SELECTOR-DISAGREEMENT COUNTS

  • Among 391 runs with multiple eligible checkpoints, AC-NC differs from AC-NLL in 110, AC-CwECE in 121, and AC-ECE in 207; these disagreements do not measure target improvement.Because all selectors share the same feasible set, this ablation does not isolate the value of accuracy filtering.Source: §E.3.1 · p. 21

E.4 ACCURACY-TOLERANCE ABLATION

The tolerance ablation shows that expanding the accuracy-feasible set changes both its size and reliability outcomes, without identifying a uniformly preferable tolerance. Later analyses characterize feasible-set trade-offs and the scope of local source–target ranking comparisons.

E.4 ACCURACY-TOLERANCE ABLATION

  • δ from 0.1 to 1.0 expands the mean candidate set from 1.30 to 9.71 checkpoints, while tested results identify no uniformly preferable tolerance.Relative to δ = 0.5, δ = 0.1 increases target ECE by 0.1314 on the ×100 scale and NLL by 0.017667; δ = 1.0 reduces them by 0.1385 on the ×100 scale and 0.332431. Target-accuracy differences are −0.0649 and −0.0190 percentage points, with both intervals including zero; 74 and 55 runs, respectively, lose at least one target-accuracy point.Source: §E.4 · pp. 21–22
  • PACS accuracy is highest at δ = 0.1, whereas TerraIncognita accuracy is highest at δ = 1.0 among the tested values.Source: §E.4 · p. 21

E.5 FEASIBLE-SET COMPOSITION

  • At δ = 0.5, 149 of 540 runs have a singleton feasible set; among the 391 multi-checkpoint sets, AC-NC differs from Source-Acc in 264 (67.5%).Of 540 runs, 347 have one nondominated NC candidate, leaving 193 with multiple candidates and a reliability trade-off. Among 85 two-checkpoint sets, 32 have opposing NLL and CwECE rankings; min–max scaling makes all three distances tie.Source: §E.5 · p. 22

E.6 PURE CALIBRATION-ONLY SELECTION

  • Pure calibration-only selection can select degenerate low-accuracy checkpoints.Source: Table E.11 · p. 23

E.7 ACCURACY-PLATEAU VARIATION

  • E.7 measures eligible-checkpoint counts and within-set metric ranges per run, then averages by dataset–algorithm group; CwECE ranges use [0, 1] and NLL its original scale.Source: §E.7 · p. 23, Table E.12 · p. 23

E.8 SOURCE–TARGET RANK CORRELATION

  • Within-Θ0.5 rank correlations include only runs with at least three eligible checkpoints and nonconstant utilities; they describe local rankings, not target-optimal selection.Source accuracy is paired with target accuracy, while source NC utility is paired with target NC utility; ties receive average ranks for Spearman ρ.Source: §E.8 · p. 23, Table E.13 · p. 24

F THEORETICAL GUARANTEES AND PROOFS

  • The theoretical section distinguishes unrestricted-target impossibility, conditional source–target ordering transfer, and a source-accuracy bound for a fixed, validation-independent checkpoint set.The bound is also extended to arbitrary fixed domain weights.Source: F THEORETICAL GUARANTEES AND PROOFS · p. 24

F.1 PROOF OF PROPOSITION 1

The theoretical and reproducibility sections delimit what source-only selection can guarantee and the conditions behind the reported comparisons. They cover unrestricted targets, conditional ordering transfer, validation-independent accuracy bounds, and analysis conventions.

F.1 PROOF OF PROPOSITION 1

  • No source-only selector, deterministic or randomized, can select a target-accuracy maximizer with probability one for every unrestricted target distribution.The two possible targets have disjoint sets of accuracy-maximizing checkpoints, despite identical source observations and saved trajectories; this does not establish an analogous impossibility for every reliability metric.Source: §F.1 · p. 24

F.2 CONDITIONAL SOURCE–TARGET ORDERING TRANSFER

  • The conditional ordering comparison is descriptive: it uses unobserved target-dependent quantities and does not certify target ordering from an empirical source margin.Its condition is sufficient but not necessary, and per-metric transfer does not ensure preservation of AC’s normalized multi-objective ranking.Source: §F.2 · pp. 24–25

F.3 PROOF OF THE FINITE-SAMPLE SOURCE-ACCURACY BOUND

  • Reliability ranking may reuse the source-validation samples after feasibility is formed when the fixed candidate trajectory was generated independently of those samples.Checkpoint independence is unnecessary, but validation feedback that changes candidate generation is outside the guarantee.Source: §F.3 · p. 26

F.4 WEIGHTED-DOMAIN EXTENSION OF THE SOURCE-ACCURACY BOUND

  • The weighted-domain extension bounds population source accuracy for feasible checkpoints, assuming checkpoints are validation-independent; it does not bound target risk.It allows arbitrary dependence among checkpoints and covers reliability selection using the same validation samples.Source: §F.4 · pp. 26–27

G REPRODUCIBILITY AND FIGURE DETAILS

G.1 UNCERTAINTY AND RESAMPLING CONVENTIONS

  • The main paired bootstrap intervals condition on observed trajectories and do not model shared seed dependence hierarchically; an accuracy-difference interval containing zero is not evidence of equivalence.Main intervals use 10,000 bootstrap resamples of paired run-level differences.Source: §G.1 · p. 27

G.2 AGGREGATION AND RESAMPLING DETAILS

  • The original paired analysis resamples 15 dataset–algorithm averages, whereas the NC tolerance analysis resamples 135 configuration minima for the same worst-held-out-domain point estimate.Paired resampling keeps all compared selectors together; tolerance intervals use a different seed from the reference NC, NE, and NEC intervals.Source: §G.2 · p. 27

G.3 FIGURE DEFINITIONS

  • Figure 1 plots mean target accuracy against selected-checkpoint reliability distance; Figure 4 reports pooled paired accuracy and NLL changes and source–target checkpoint-rank agreement.Figure 1’s selector distances are computed after selection; Figure 4’s panel (b) rankings differ in scope and utility definition from Table E.13.Source: §G.3 · pp. 27–28

G.4 TRAINING-CODE SETTINGS

The training-code table records defaults and logged trajectory steps for the main sweep.

G.4 TRAINING-CODE SETTINGS

  • Table G.1 records the main sweep’s training-code defaults and logged trajectory steps.Source: Table G.1 · p. 28
Loading 2609.39934v1…