PaperlayerView exact revision on arXiv

Source-linked AI summary

A Free Knob: Decoupling Calibration and Predictive Skill in Threshold-Based Evaluation

Md Tanveer Hossain Munim, Bijoy Ahmed Saiem, Al-Amin Sany, Tanzima Hashem

arXiv:2609.33457v1cs.LG

TL;DR

Fixed-operating-point pool-and-threshold CSI can mix spatial discrimination with amplitude calibration, complicating interpretation of rare-event comparisons. The paper applies a symmetric held-out monotone calibration audit to each system and finds that calibration can substantially alter CSI contrasts, including reversals. It recommends reporting pooled frequency bias alongside the audit.

  • Problem

    Pool-and-threshold CSI comparisons at fixed rare operating points do not equalize predicted event counts, so the score reflects frequency bias as well as discrimination.

    Source: §1 · p. 1

  • Method

    The FreeKnob Audit fits an order-preserving amplitude transform separately to each system on held-out data and re-evaluates the original comparison without changing pixel ordering.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Across 450 system contrasts, 51 reverse sign after calibration, and the bias-difference association with contrast movement is r = +0.796.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper recommends reporting pooled frequency bias and a symmetric held-out FreeKnob Audit alongside rare-event pool-and-threshold scores.

    Source: §1 · p. 3

    The measured relative-gain reductions cover two published comparisons among three released SEVIR systems, and the effect size is not established across independent forecasting families or native benchmarks.

    Source: §8 · p. 9

Abstract

from arXiv · show

Many dense-prediction benchmarks evaluate rare events by pooling prediction and target over spatial blocks, thresholding each, and scoring the contingency table. At a fixed rare operating point, the max-pooled Critical Success Index (CSI) confounds spatial discrimination with amplitude calibration: sharp observations promote many blocks above threshold, while attenuated predictions from squared-error regression leave the same blocks below it. We repurpose classical monotone calibration as a symmetric audit: a post-hoc transform fitted on held-out data and applied separately to each system. The transform cannot reverse pixel ordering, so any contrast it reproduces cannot establish improved spatial ranking. On SEVIR, two released checkpoints of one architecture differ by -29.5% in extreme-threshold CSI before the control and by +5.3% after it. Across 450 pairwise contrasts among 6 systems, the difference in pooled frequency-bias deviation is associated with how far the CSI contrast moves under the control (r = +0.796), and 51 contrasts reverse sign. At CasCast's published extreme-event operating point, the cascade-over-backbone CSI gap falls from 0.1601 to 0.0339, a 78.8% reduction; the remaining gap stays positive. The effect persists when the transform is fitted on a window before the test period, and calibration also reveals advantages hidden by a better-calibrated baseline. On geostationary infrared imagery the relative gain grows as events become rarer, crowd counting reproduces the bias-gain relationship under patch-sum pooling, and semantic segmentation, where frequency bias is already near one, shows little average change. The confound therefore requires both a fixed operating point and a training regime that leaves the output miscalibrated there. We recommend reporting pooled frequency bias and a symmetric held-out FreeKnob Audit alongside rare-event pool-and-threshold scores.

1 INTRODUCTION

Pool-and-threshold CSI for rare events depends on both spatial discrimination and frequency bias, so amplitude calibration can change system comparisons without changing event placement. The paper applies a symmetric held-out calibration control and finds substantial changes in released-model comparisons.

1 INTRODUCTION

  • At fixed rare operating points, pooled CSI depends on frequency bias as well as discrimination, because predicting too few events adds misses.A system may raise CSI by predicting more events without placing them better.Source: §1 · p. 1
  • Max pooling can make sharp observations yield more positive blocks than smoother predictions of the same event, lowering pooled frequency bias.At the highest commonly reported SEVIR threshold, only 0.081% of pixels are positive.Source: §1 · p. 2
  • The two released EarthFormer checkpoints shift from a -29.5% CSI difference to +5.3% after calibration, reversing their ranking at 5 of 6 thresholds.The CasCast cascade-over-backbone gap falls from 0.1601 to 0.0339, a 78.8% reduction, but remains positive.Source: §1 · p. 2
  • A held-out monotone transform applied separately to each system shifts amplitudes without reversing pixel ordering, serving as a symmetric control for calibration effects.It requires no retraining and tests how much of the original contrast survives.Source: §1 · p. 2
  • Across 450 contrasts among 6 systems, frequency-bias differences track contrast movement, with 51 sign reversals; crowd counting shows the relationship while segmentation changes little near bias one.The paper presents the FreeKnob Audit as a package reporting pooled frequency bias and repeating comparisons after symmetric held-out calibration.Source: §1 · p. 3

2 RELATED WORK

Prior verification work has examined frequency-bias sensitivity, spatial pooling, recalibration, and benchmark audits. This paper distinguishes its approach as a symmetric held-out marginal control that re-evaluates the original fixed-threshold CSI comparison.

2 RELATED WORK

  • Forecast verification has long studied CSI’s frequency-bias sensitivity, frequency matching, and bias equalization, including applications to deep-learning nowcasts.This paper examines how that sensitivity changes comparisons of released deep nowcasting systems at rare, max-pooled operating points.Source: §2 · p. 3
  • Earlier neighborhood-verification work linked fixed-threshold scores to intensity and spatial or size error, while percentile thresholds were proposed to reduce the effect.The paper instead audits released checkpoints with a symmetric held-out marginal control while retaining the original CSI operating point.Source: §2 · p. 3
  • Classical recalibration distinguishes calibration from resolution, while recent potential-CRPS methods ask a different stability question from this fixed-threshold CSI audit.Here, each arm receives an unpaired marginal map fitted on held-out data, then the original score is recomputed; the map is a negative control, not a replacement score.Source: §2 · p. 3
  • Prior work reports that CSI can respond differently to pooling size for sharp versus blurry forecasts, while reporting of frequency bias varies across nowcasting papers.The present study holds released predictions fixed and gives each the same held-out marginal control.Source: §2 · p. 4

3 METHOD: THE FREEKNOB AUDIT

The FreeKnob Audit applies an order-preserving amplitude map before pooling and thresholding, then compares raw and calibrated CSI while reporting pooled frequency bias. Its global map is fitted on held-out events and held fixed across thresholds and pooling conventions.

3 METHOD: THE FREEKNOB AUDIT

  • CSI is computed from hits, false alarms, and misses, while pooled frequency bias is the predicted-to-observed event-count ratio accumulated over the split.The reference cell uses max 16×16 pooling at threshold 219.Source: §3 · p. 4
  • The audit considers shifts, positive gains, and marginal quantile mapping as order-preserving amplitude adjustments before pooling and thresholding.Under max pooling, the quantile map corresponds to matched-percentile thresholding, subject to empirical tie conventions.Source: §3 · p. 4
  • An order-preserving map changes which amplitudes cross the threshold but cannot improve spatial ranking; the resulting CSI change is the knob gain.The observed marginal informs amplitude but supplies no paired location information; calibration is summarized by pooled frequency bias.Source: §3 · p. 4
  • The primary global quantile map is fitted once per arm on 33 calibration events and held fixed across thresholds and pooling conventions.A per-cell variant is also fitted on held-out data to show how conclusions depend on the allowed calibration budget; neither is selected on evaluation CSI.Source: §3 · p. 4
  • Figure 2 plots pooled frequency bias across thresholds and, for 30 arm-threshold points, CSI gain against bias under max 16×16 pooling.The figure describes gains as growing with bias deficit and generally small near Bias = 1 in the evaluated arms.Source: Figure 2 · p. 5

4 EXPERIMENTAL SETUP

The experiments use released SEVIR models and a common evaluation pipeline to examine published comparisons. The protocol follows the released CasCast guidance setting to reproduce the reported comparison.

4 EXPERIMENTAL SETUP

  • The SEVIR evaluation uses 1872 test events, 13 input frames, 12 output frames at 384×384, and six VIL thresholds.The five primary arms are persistence, pysteps Lucas–Kanade advection, the official EarthFormer checkpoint, and CasCast’s deterministic backbone and diffusion cascade; all learned arms are released artefacts.Source: §4 · p. 5
  • The common pipeline’s deterministic CasCast backbone scores differ from the authors’ reported cells by at most 2.8%.The released evaluation script uses classifier-free guidance weight 2, which is used throughout; unguided sampling is retained as a diagnostic arm.Source: §4 · p. 5
  • Using the unguided path produces a different extreme-threshold score, so the evaluation follows the released guided setting rather than that alternative.The unguided sample is kept only as a sixth diagnostic arm.Source: §4 · p. 5

5 THE MECHANISM AND ITS EFFECT SIZE

With max pooling, attenuation can leave sharp observed blocks above threshold while smooth forecasts remain below it, producing misses and frequency bias below one. The calibration effect varies with rarity and pooling, while a near-unbiased persistence control moves little.

The mechanism and its effect size

  • With sharp observations and smooth forecasts, max pooling raises the observed block maximum more, widening the threshold gap and producing misses that drive bias below one.The persistence control’s small change is expected when its pooled marginal is stable over the forecast window; storm growth or decay can violate that condition.Source: §5 · p. 5
  • Persistence’s pooled bias stays within [0.993, 1.020] across six thresholds, and global quantile mapping changes its CSI by at most 0.8%.Across 30 primary arm–threshold points, knob gain is ordered by distance below bias one (ρ = 0.675); the association is ρ = 0.599 across 150 cells and four pooling conventions.Source: §5 · p. 5
  • At τ = 219, the map that lifts CasCast’s own backbone by +79.8% moves the cascade, whose bias is already 1.037, by -2.9%.The cascade has the highest CSI scored in this comparison.Source: §5 · p. 6

Rarity

  • At the rarest infrared operating point, knob gain reaches +250.8%, versus -2.3% at the most common, as CSI itself collapses 31-fold.Across six operating points, gain is monotonically ordered against log10 event fraction (ρ = −1.000); absolute CSI change stays within [−0.0062, +0.0238].Source: §5 · p. 6

Pooling

  • For CasCast’s deterministic backbone at τ = 219, knob gain is -8.9% under average 16×16 pooling but +79.8% under max 16×16 pooling.Under max pooling, bias falls from 0.291 unpooled to 0.214 at 4×4 and 0.172 at 16×16, while gain rises from +39.3% to +61.1% and +79.8%.Source: §5 · p. 6

6 DECOMPOSING A PUBLISHED COMPARISON

A symmetric held-out calibration substantially shrinks the cascade’s extreme-threshold advantage over its backbone, while leaving a positive residual under the primary global map. Across pairwise SEVIR contrasts and a same-architecture comparison, calibration also moves or reverses reported rankings.

6 DECOMPOSING A PUBLISHED COMPARISON

  • At the reference cell, the cascade-over-backbone CSI gap falls from 0.1601 to 0.0339, a 78.8% reduction after symmetric global calibration.The relative advantage falls from 109.1% to 12.9%, an 88.2% reduction; the remaining contrast stays positive under the global map.Source: §6 · pp. 6–7
  • With a per-cell calibration control, the cascade’s residual relative advantage becomes -5.5%, compared with +12.9% under the primary global map.The global map uses one transform per arm fixed across thresholds and pooling conventions; the reported sensitivity range is [−5.5%, +12.9%].Source: §6 · p. 7

6.1 ALL PAIRWISE CONTRASTS

  • Across 450 pairwise contrasts among six SEVIR arms, calibration-gap differences track contrast movement (r = +0.796), and 51 contrasts reverse sign.These are repeated comparisons among the same six evaluated arms, not independent model replications; the primary-pooling correlation is +0.884.Source: §6.1 · pp. 7–8, Figure 4 · p. 7
  • For three pairs compared in prior work, calibration shrinks the cascade’s gains over its backbone and EarthFormer but reveals a larger EarthFormer advantage over persistence.Across all 450 contrasts, 23 of the 51 sign reversals exceed 0.01 CSI on both sides.Source: §6.1 · p. 8, Table 1 · p. 8

6.2 TWO RELEASED CHECKPOINTS OF THE SAME ARCHITECTURE

  • Two independently trained checkpoints of EarthFormer’s cuboid-attention architecture differ by -29.5% in CSI before calibration and +5.3% after it.The sign reverses at five of six thresholds; because the groups’ recipes differ in more than seed, the gap is not attributed to seed variance.Source: §6.2 · p. 8

7 GENERALITY BEYOND ONE BENCHMARK

The bias–gain relationship also appears in crowd counting under patch-sum pooling, while segmentation shows little average movement when many classes are already near bias one. The paper’s controls distinguish placement effects from amplitude adjustment and identify miscalibration at a fixed operating point as a condition for the confound.

Crowd counting

  • Across ShanghaiTech Part A CSRNet arms, pooled-bias deficit orders absolute knob gain (ρ = 0.568 over 320 cells) under 32×32 patch-sum pooling.Across five seeds using one recipe, pooled bias ranges from 0.002 to 1.146 and orders CSI at ρ = 0.90, while dense-patch count errors differ by a factor of 1.7.Source: §7 · p. 8

Separating placement from attenuation

  • In the target-corruption control, spatial displacement leaves pooled bias in [0.926, 0.979] and knob gain at most 0.0023, while local-mean compression yields bias 0.761 and gain +0.0809.The results distinguish placement sensitivity from the monotone control’s sensitivity to amplitude adjustment.Source: §7 · pp. 8–9

Semantic segmentation as the boundary case

  • Across 20 segmentation classes, event frequency and knob gain are essentially uncorrelated (ρ = −0.019), with mean movement of only -0.0020 IoU.Eleven classes already lie within 0.06 of bias one; among miscalibrated classes, gain tracks bias deficit (ρ = +0.663), rising to +0.836 under max 8×8 pooling.Source: §7 · p. 9
  • The confound requires both a fixed operating point and a training regime that leaves the model miscalibrated there; rarity alone is not the diagnostic.Per-class IoU uses the same contingency-table arithmetic as CSI (IoU = TP/(TP + FP + FN)).Source: §7 · p. 9

8 DISCUSSION AND RECOMMENDATIONS

The audit treats calibration as a control for how much of a fixed-threshold score contrast can arise from amplitude, not as a corrected leaderboard. The authors recommend reporting pooled frequency bias and repeating comparisons with the same held-out monotone calibration procedure when compared systems have different biases.

8 DISCUSSION AND RECOMMENDATIONS

  • Report pooled frequency bias beside each rare pool-and-threshold score, and calibrate both arms separately on held-out data when their biases differ.State the calibration split and transform family.Source: §8 · p. 9
  • The audit is an identification device, not a corrected leaderboard, and a residual contrast is not guaranteed to measure spatial skill.The global-map CasCast residual is +12.9%, with its paired bootstrap interval excluding zero at every threshold.Source: §8 · p. 9
  • A monotone marginal transform changes which amplitudes cross the operating threshold but cannot reverse pixel ordering, so amplitude at a physical threshold remains real predictive skill.The identification problem is reading the score as evidence that a system places extremes better.Source: §8 · p. 9
  • The measured reductions cover two published comparisons among three released SEVIR systems, not a literature-wide average or established effect across independent forecasting families.The analysis concerns contingency-table scores at fixed operating points after pooling, does not test CRPS, LPIPS, or SSIM, and the audit depends on the admissible transform family.Source: §8 · p. 9
  • Agreement indicates stability under the specified control, while shrinkage or growth measures its effect on the contrast; a sign reversal blocks reading raw ordering as spatial-discrimination ordering.Bias near one usually accompanies small movement, but exceptions warrant checking the fitted map and raw-versus-pooled marginal distinction.Source: §8 · pp. 9–10

REPRODUCIBILITY STATEMENT

The experiments use released checkpoints evaluated through a common pipeline, with stored artefacts generating reported results. The paper documents protocol and model decisions to support reproduction.

REPRODUCIBILITY STATEMENT

  • Every SEVIR result comes from a released checkpoint evaluated on the official preprocessed data through one common pipeline.The authors use deterministic calibration/evaluation splits where applicable.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Tables, figures, and quoted statistics are generated programmatically from stored experimental artefacts rather than entered manually.The repository provides the installable freeknob implementation of the audit.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A documents the evaluation protocol, statistical conventions, checkpoint settings, reproduction checks, and model-inclusion decisions.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendix discloses AI assistance and describes evaluation and reproduction details for the released-model analyses. It also records checkpoint exclusions where faithful evaluation could not be established.

AI USE STATEMENT

  • The authors used an AI assistant for protocol design, implementation, data assembly, interpretation, writing, and figures, and reviewed the AI-assisted work.They report checking code against a reference implementation reproducing all thirty SEVIR cells to 10−7.Source: AI USE STATEMENT · p. 10

A REPRODUCIBILITY DETAILS

  • The repository includes the audit package, while reproducibility details specify parity-based event splits, rank-correlation conventions, and CasCast sampling settings.The main SEVIR fit uses 33 calibration events; fitting on 128 events from an earlier window leaves conclusions unchanged.Source: A REPRODUCIBILITY DETAILS · pp. 13–14

A.1 REPRODUCTION OF THE CASCAST BACKBONE

  • The CasCast deterministic-backbone reproduction compares a re-run of the released checkpoint with Gong et al.’s four reported cells; DGMR and NowcastNet are excluded because evaluating their checkpoints on SEVIR would require retraining.Table 2 defines CSI-M as the mean over six thresholds.Source: §A.1 · p. 14, Table 2 · p. 14
  • These exclusions narrow the empirical comparison but do not change the audit definition, which can be applied to additional checkpoints.Source: §A.1 · p. 14

B PER-CLASS SEGMENTATION RESULTS

The segmentation audit reports per-class IoU before and after a monotone logit shift, with classes ordered by positive-pixel fraction. Near-unit bias corresponds to small average movement, but individual classes can still change appreciably.

B PER-CLASS SEGMENTATION RESULTS

  • At bias 0.963, tvmonitor loses -0.0248 IoU despite lying inside the near-unit-bias band.Its positive-pixel fraction is 0.414%; the calibration-half shift is described as noisy, and selecting the family by validation IoU recovers most of the loss, to -0.0101.Source: B PER-CLASS SEGMENTATION RESULTS · p. 14
  • Across 11 classes in the near-unit-bias band, 6 move by less than 0.001 IoU, 4 lose, 1 gains, and mean change is -0.0048.The average movement is small in this band, although individual classes can move appreciably.Source: B PER-CLASS SEGMENTATION RESULTS · p. 15
  • The reported mIoU is -5.3% relative to the published figure because the evaluation uses all val2017 images and retains iscrowd regions.The authors state these choices do not affect the within-class before-and-after comparison.Source: B PER-CLASS SEGMENTATION RESULTS · p. 15
  • The table reports each sufficiently supported COCO class as a binary mask at threshold 0.5, before and after a monotone logit shift fitted on disjoint calibration data.Classes are ordered by positive-pixel fraction, and classes without enough calibration support are omitted.Source: B PER-CLASS SEGMENTATION RESULTS · p. 14, Table 3 · p. 15

C FULL RESULTS OF THE RARITY SWEEP

The rarity sweep reports stronger calibration gains at rarer thresholds, while the infrared sensitivity check retains that pattern when all scenes are pooled. The infrared split also has temporal shift, and the pooling-convention results are presented as a comparison of calibration effects and model rankings.

C FULL RESULTS OF THE RARITY SWEEP

  • For the three rarest thresholds, graded and tiered variants separately show monotone rarity associations; departures occur at common thresholds and reflect fitting noise.Across all six levels, ρ = −0.943 for graded and ρ = −0.829 for tiered against log10 R; the tiered variant does not reach conventional significance, and the authors do not interpret the pooled correlation as strongest evidence.Source: C FULL RESULTS OF THE RARITY SWEEP · p. 15
  • The rarity-sweep table retains a coverage filter, and four runs with zero uncalibrated CSI are omitted from relative-gain calculations despite gaining 0.0258–0.0315 CSI after calibration.Their relative gain is unbounded, so omitting them can only understate the trend.Source: Table 4 · p. 16

D THE INFRARED DOWNSCALING TASK

  • The infrared experiment uses 1,500 chronologically selected scenes, with four ERA5 predictors and Himawari B08 brightness temperature as the target.Integrated vapour transport is stored but excluded from the scored models.Source: D THE INFRARED DOWNSCALING TASK · p. 16
  • Temporal shift separates the infrared validation window from train and test, and six-hourly autocorrelation makes the effective sample size smaller than the nominal scene count.At 220 K, full-grid positive fractions are 18.75%, 13.95%, and 19.62% for train, validation, and test, respectively; the validation-selected operating point is applied across that shift.Source: D THE INFRARED DOWNSCALING TASK · p. 16
  • +11.7% at 220 K to +92.2% at 197 K: the infrared knob gain still grows with rarity when every validation and test scene is retained.The rarity ordering is less exact in this sensitivity check (ρ = −0.829, p = 0.0583).Source: D THE INFRARED DOWNSCALING TASK · p. 16

E FULL RESULTS BY POOLING CONVENTION

  • Table 5 compares pooling conventions for the CasCast deterministic backbone and counts how often pysteps outranks EarthFormer across six thresholds under three calibration conditions.The caption identifies the conditions as uncalibrated, global quantile map, and per-cell knob; it does not provide the outcomes.Source: E FULL RESULTS BY POOLING CONVENTION · p. 16, Table 5 · p. 17

F FULL CROWD-DENSITY RESULTS UNDER THE PATCH-SUM PROTOCOL

This section examines the calibration audit across fitting periods, transform families, selection rules, guidance, and scoring choices, alongside alternative verification scores. It clarifies that shared held-out monotone calibration is a narrow diagnostic of how much of a pooled CSI gap survives—not a way to identify the better model—and that results can change with the control used.

F FULL CROWD-DENSITY RESULTS UNDER THE PATCH-SUM PROTOCOL

  • Across five crowd-density seeds, CSI varies 188-fold while dense-patch count error ranges from 4.7 to 8.1 persons; patch error orders the scores perfectly (ρ = −1.000).The worst seed’s 8.1-person error is below the all-zero-predictor bound of 10, but the comparison does not establish that every run converged adequately. Among trained arms, bias–gain rank correlation is ρ = 0.568 over 320 cells; including controls gives 0.629 over 1760 cells, where ties affect ranking.Source: F FULL CROWD-DENSITY RESULTS UNDER THE… · p. 17, Table 7 · p. 18

G TWO RELEASED CHECKPOINTS OF ONE ARCHITECTURE

  • The two independent trainings of one cuboid-transformer architecture are not a published head-to-head comparison, and their calibrated CSI contrast reverses sign at five of six thresholds.This measures how large an apparent gap can arise between trainings of one architecture; the table caption specifies max 16×16 pooling and a global quantile map fitted per arm.Source: G TWO RELEASED CHECKPOINTS OF ONE… · p. 18, Table 8 · p. 18

H FITTING THE TRANSFORM ON AN EARLIER VALIDATION WINDOW

  • With a map fitted on an earlier held-out window, the main cascade relative-gain reduction is 96%, versus 88% under the test-half fit.Across 450 contrasts, calibration-gap association is r = +0.766 under val-fit and r = +0.796 under test-half fit, with 49 and 51 sign reversals. The val-fit residual is a point estimate without an interval; the result supports persistence across fitting periods, not that earlier fitting is better.Source: H FITTING THE TRANSFORM ON AN EARLIER… · pp. 18–19

I THE GUIDANCE ARM

  • At max 16×16 pooling and τ = 219, guidance changes pooled bias from 0.172 to 1.037 and CSI from 0.1467 to 0.3069 across the backbone-to-guided-cascade ladder.The corresponding knob gains are +79.8%, +17.1%, and −2.9% for backbone, unguided cascade, and guided cascade, respectively; guidance is a sampling-time setting, not a training change.Source: I THE GUIDANCE ARM · p. 19

J SELECTION RULES FOR THE TRANSFORM

  • Selecting the transform by validation CSI largely removes the crowd-density MSE baseline’s worst −0.1286 CSI loss, while CSRNet still gains up to +0.2394.The conservative bias rule grants CSRNet up to +0.2162; on SEVIR, validation-CSI selection is not uniformly better out of sample, scoring 0.3977 versus 0.4030 at max 162, τ = 181.Source: J SELECTION RULES FOR THE TRANSFORM · pp. 19–20

K DECOMPOSITION UNDER THE EQUITABLE THREAT SCORE; O ALTERNATIVE VERIFICATION SCORES

  • Under ETS, calibration removes 41%–88% of the cascade’s gain at the five thresholds with gain to erase, with relative-gain reductions within 5 percentage points of CSI.ETS corrects for chance rather than bias; rare-event alternatives such as SEDI may be useful alongside CSI but do not remove the upstream effect of pooling on which cells enter the contingency table. PostCast reports a roughly sixfold pooled extreme-threshold CSI increase, but its diffusion transform reorders pixels and is not a free monotone control.Source: K DECOMPOSITION UNDER THE EQUITABLE… · p. 20, O ALTERNATIVE VERIFICATION SCORES · pp. 24–25

L BOOTSTRAP INTERVALS

  • +0.1171 CSI is the backbone’s knob gain at the reference cell, while +0.0339 [+0.0234, +0.0439] remains of the cascade advantage after both arms receive the same transform.The residual is positive with p < 0.001 at every threshold; intervals are paired-bootstrap intervals conditional on the fitted maps. The quoted range [−5.5%, +12.9%] compares the two controls specified in Section 6, not all possible transforms.Source: L BOOTSTRAP INTERVALS · pp. 20–21, Table 12 · p. 21

M FITTING THE MAP PER LEAD TIME; N JOINT SENSITIVITY TO TRANSFORM FAMILY AND SELECTION RULE

O ALTERNATIVE VERIFICATION SCORES

  • FSS’s large-neighbourhood asymptote is determined by frequency bias rather than placement.This carries the same vulnerability as pooled scores.Source: O ALTERNATIVE VERIFICATION SCORES · p. 25
  • MODE uses object matching, whereas SAL summarizes identified objects without one-to-one matching.These methods diagnose structure, amplitude, and location, but require object-identification choices; MODE also requires matching, and their amplitude sensitivity is not evaluated here.Source: O ALTERNATIVE VERIFICATION SCORES · p. 25
  • The control gives every arm a monotone map fitted without test targets and at no training cost, measuring how much of a reported gap survives rather than which model is better.Unlike alternatives that modify the score, it modifies the forecast; the map is fitted under the protocol of Section 3.Source: O ALTERNATIVE VERIFICATION SCORES · p. 25

Related papers

Loading 2609.33457v1…