PaperlayerView exact revision on arXiv

Source-linked AI summary

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Md Kawsher Mahbub, Milon Biswas

arXiv:2609.32672v1cs.LGcs.AI

TL;DR

Predicting OOD performance without target-domain data remains difficult, while existing approaches may require target data, model internals, or retraining. STAMP ranks models from semantic and random pairs of source images using output-space variation. It reports strong rank correlations across medical and natural-image shifts, alongside scope limitations involving pair availability and distributional assumptions.

  • Problem

    Pre-deployment OOD prediction is difficult when target-domain data are unavailable, and prior approaches require target data, multiple models, retraining, or model weights.

    Source: §1 · p. 1

  • Method

    STAMP estimates output-space correlation ratio η2 from semantically stable and random source-image pairs, using one model at a time without target data or model internals.

    Source: §1 · pp. 1–2

  • Results

    STAMP reports ρs ≥0.844 on three medical OOD datasets and ρs ≥0.905 on five natural-image benchmarks, with 44 and 27 models, respectively.

    Source: §10 · p. 9

  • Takeaways & Limitations

    At approximately 12 seconds per model, STAMP supports large-scale pre-deployment auditing and model selection.

    Source: §10 · p. 9

    STAMP requires source-domain semantic pairs; synthetic augmentation pairs yield lower ρs, and performance degrades under concept drift.

    Source: §9 · p. 9

Abstract

from arXiv · show

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio $\eta^2=S_B/S_T$ by contrasting semantically stable pairs with random pairs: higher $\eta^2$ indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of $0.844$--$0.855$ with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from $0.311$ to $0.663$. STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains $\rho{=}0.984$ on ObjectNet and $\rho\geq0.905$ on four additional distribution shifts, with partial correlations of $0.662$--$0.949$ after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.

1 INTRODUCTION

STAMP addresses source-only prediction of model performance under distribution shift, where existing estimators require target data, multiple models, retraining, or model internals. It ranks models by whether outputs remain consistent across semantically equivalent source images, with strong reported associations across medical and natural-image benchmarks.

1 INTRODUCTION

  • STAMP estimates the fraction of output variance explained by semantic groups, contrasting stable pairs with random pairs.Stable pairs are same-patient longitudinal radiographs or images sharing an ImageNet class.Source: §1 · p. 1
  • STAMP reports macro-AUROC across 44 chest X-ray architectures evaluated on four external hospitals; its temperature-scaled variant reaches ρs = 0.984 on ObjectNet across 27 ImageNet-pretrained models.Predictions are source-only and take about 12 seconds per model on one GPU.Source: §1 · p. 2
  • STAMP ranks models from paired source images without target samples, model internals, retraining, or target labels..Source: §1 · p. 2
  • The paper proves consistency of the class-conditional correlation-ratio estimator and derives an OOD error bound under bounded covariate shift.The longitudinal medical estimator requires empirical rather than purely i.i.d. assumptions.Source: §1 · p. 2
  • Across nine medical and natural-image OOD benchmarks and 71 models, the evaluation reports rank and partial correlations, confidence intervals, permutation tests, stability, and failure analyses..Source: §1 · p. 2
  • STAMP completely separates CE-supervised and frozen SSL/VLM probes, with Cliff’s δ=1.0 across n=43 models.The paper identifies training objective as a moderator of source-only OOD predictability.Source: §1 · p. 2

2 RELATED WORK

Prior OOD predictors trade off access to target data, model internals, or retraining. STAMP instead uses output probabilities from paired source inputs as an architecture-agnostic, source-only proxy.

2 RELATED WORK

  • Target-domain confidence methods and model-agreement methods require target data, while Accuracy-on-the-Line requires labeled target data for calibration.STAMP is source-only.Source: §2 · p. 2
  • STAMP uses output probability vectors from paired inputs without a learned scorer, model internals, or retraining, unlike several source-only and geometric diagnostics.It is architecture-agnostic and reports SRCC = 0.855 on VinDr, versus DDBout SRCC = 0.820 on Camelyon17.Source: §2 · p. 2
  • The correlation ratio η2 = SB/ST measures variance explained by group membership, and STAMP applies it to output probabilities for black-box evaluation.The paper states that its proxy has a closed-form generalization bound and is computable in O(n) forward passes without gradients.Source: §2 · p. 2

3 OUR APPROACH

STAMP seeks a source-computable scalar that predicts accuracy under a different target distribution. It estimates output-space semantic consistency from stable and random source-image pairs, with temperature scaling and domain-specific pair construction for its two settings.

3.1 PROBLEM SETUP

  • The problem is to compute a source-only scalar that monotonically predicts accuracy under a target distribution different from the source.The medical setting uses 14 NIH chest-pathology labels and same-patient longitudinal pairs; the natural-image setting uses 1,000 ImageNet classes and same-class pairs.Source: §3.1 · p. 2

3.2 THE STAMP SCORE

  • STAMP compares N semantic pairs with N independently drawn random pairs from the source distribution.Semantic pairs share a patient or class identity; random pairs differ in patients or classes.Source: §3.2 · p. 3
  • STAMP is zero when semantic and random pairs induce identical output divergence and approaches one when within-class output variance is negligible relative to total variance.The estimator consistently estimates η2 = SB/ST.Source: §3.2 · p. 3

3.3 TEMPERATURE SCALING FOR NATURAL IMAGE MODELS

  • ImageNet logits are temperature-scaled using NLL on 2,000 held-out validation images outside the pair pool, and models with calibrated T > 3.0 are excluded as having non-functional classifier heads.Retained and excluded models have a calibration gap: maximum retained T=1.37 versus minimum excluded T=5.76.Source: §3.3 · p. 3

3.4 PAIR CONSTRUCTION

  • Medical STAMP-T samples 2,000 consecutive same-patient radiograph pairs with follow-up gaps ≤3, while STAMP-L uses label-matched pairs only for ablation.Natural-image pairs comprise 2,000 same-class and 2,000 different-class pairs, sampled across the 1,000 classes.Source: §3.4 · p. 3

4 THEORETICAL ANALYSIS

The analysis connects STAMP to OOD accuracy under stated shift, smoothness, and identifiability conditions. It establishes estimator consistency and a bounded-shift error connection, while identifying conditions and failure modes that qualify rank predictions.

4 THEORETICAL ANALYSIS

  • The OOD error connection assumes label-preserving bounded covariate shift, an L-Lipschitz output map, and error behavior tied to projected within-class variance and inter-class margin.The bound also invokes centroid alignment and bounded within-class shift.Source: §4 · p. 4
  • STAMP consistently estimates η2 with Op(n^-1/2) error under i.i.d. stable and random pair sampling and nonzero output variance.η2 is the fraction of total output variance explained by between-class signal; STAMP jointly favors larger between-class and smaller within-class variation.Source: §4 · p. 4
  • The bounded-shift analysis connects the source-only output statistic to OOD error, and under A3 gives dH∆H ≤2LW2(Ps, Pt).The result uses Lipschitz control of target variance and centroid displacement without a Taylor approximation.Source: §4 · p. 4
  • The ordering induced by the variance term is algebraically exact, but full OOD ranking additionally requires inter-class margin to be non-decreasing with STAMP rank.That condition is measured from source outputs with Spearman ρ = +0.73, p < 0.001 across 40 models.Source: §4 · p. 4
  • Under A1–A5, higher STAMP predicts higher target accuracy, with empirical correlations ρ ≥0.844 across three medical datasets and ρ ≥0.905 across five natural-image benchmarks.The framework identifies class mismatch, concept drift, and degenerate outputs as failure modes.Source: §4 · p. 5
  • Across five independent pair samples, STAMP’s standard deviation is below 0.004, and rankings remain unchanged for N ∈{500, 1000, 2000, 5000}..Source: §4 · p. 5

5 EXPERIMENTAL SETUP

The experiments evaluate medical and natural-image models on external distribution-shift benchmarks and compare source-only prediction metrics with target-domain estimators.

5.1 MEDICAL IMAGING

  • The medical study uses 44 architectures and NIH ChestX-ray14 source data, evaluating VinDr-CXR, CheXpert, RSNA Pneumonia, and MIMIC-CXR.NIH ChestX-ray14 contains 112,120 frontal radiographs, 30,805 patients, and 14 binary disease labels.Source: §5.1 · p. 5

5.2 NATURAL IMAGE BENCHMARKS

  • The natural-image study evaluates 27 ImageNet models on ObjectNet, Dollar Street, ImageNet-A, ImageNet-R, and ImageNet-C.The protocol uses permutation tests, bootstrap confidence intervals, partial Spearman correlations, and leave-one-out stability checks.Source: §5.2 · p. 5

5.3 BASELINES

  • The comparison includes 21 source-only metrics, target-domain ATC and AoTL estimators, and labeled NIH source AUROC as a reference.The listed metrics include confidence and entropy measures, test-time augmentation, embedding instability, and STAMP variants.Source: §5.3 · p. 6

6 RESULTS

STAMP and its temperature-scaled variant predict performance across medical and natural-image distribution shifts. The results also compare STAMP with other model-ranking metrics and examine subgroup and corruption patterns.

6.1 MEDICAL IMAGING:

  • STAMP attains ρs≥0.844 on VinDr, CheXpert, and MIMIC across 44 models; RSNA is the benchmark with a known class-overlap mismatch.All three correlations have permutation p<10−10; the figure shows weaker, noisier RSNA trends.Source: Table 1 · p. 6, §6.1 · p. 6, Figure 1 · p. 6
  • STAMP has the best average source-only ranking (ρs=0.715), ahead of Aug-STAMP-rotation (0.709), and exceeds target-domain ATC (0.380) and AoTL (0.320).STAMP takes about 12 seconds per model on one GPU.Source: §6.1 · p. 7, Table 2 · p. 7

6.2 NATURAL IMAGES: STAMP-TS STUDY

  • STAMP-TS reaches ρs≥0.905 on all five natural-image benchmarks across 27 models, with the cross-family analysis reaching ρs=0.986.The cross-family analysis uses one representative model per architecture, and leave-one-out results show no single model drives the benchmark result.Source: Table 3 · p. 7, §6.2 · p. 7
  • On Dollar Street, STAMP-TS correlates more strongly with top-1 accuracy in the high-income stratum (ρs=0.873) than the low-income stratum (ρs=0.777); across corruptions, mean ρs=0.876.Across 15 corruption types, correlations range from 0.756 on contrast to 0.929 on brightness.Source: §6.2 · p. 8

7 ABLATION STUDIES

The ablations examine the STAMP ratio, its relationship to pretraining objective, and where semantic discriminability emerges within model layers.

7 ABLATION STUDIES

  • STAMP's joint ratio reaches ρs=0.89 (SV alone: −0.19; AV alone: 0.64), while temporal pairs improve correlation by +0.110 to +0.186 over labeled pairs.The passage attributes the temporal-pair improvement to NIH label noise contaminating the labeled semantic pool.Source: §7 · p. 8
  • CE-supervised models have higher STAMP than SSL probes (mean 0.786±0.011 vs. 0.717), with no rank overlap (Cliff’s δ=+1.0).The comparison covers 43 models with standard objective groupings; hybrid RAD-DINO is analyzed separately.Source: §7 · p. 8, Table 4 · p. 8
  • ViT and Swin STAMP rises +0.12 to +0.22 from early to penultimate layers; mid-to-penultimate gain and final-layer STAMP predict VinDr AUROC at ρs=+0.701 and +0.795.These layerwise findings are termed the Universal Generalization Motif: robust models build semantic discriminability late rather than carrying it from the input.Source: §7 · pp. 8–9

8 FAILURE ANALYSIS

The largest STAMP–VinDr rank disagreements reveal two distinct failure modes tied to model behavior and the source distribution.

8 FAILURE ANALYSIS

  • Rank disagreements of at least 15 include NIH-specific channel-attention suppression in ResNet50-SE and compressed STAMP range for highly confident DeiT-Base/16 and MViTv2-T outputs.The attention-suppression pattern does not transfer to VinDr; both failure modes are detectable without target-domain data.Source: §8 · p. 9

9 DISCUSSION

STAMP’s output-space semantic signal is linked to OOD error, while pretraining objective strongly moderates its predictability. It can be computed before deployment without target data, at substantially lower cost than the target-domain estimators discussed.

9 DISCUSSION

  • STAMP estimates between-class output variance relative to total variance, linking output-space class-discriminability to bounded OOD error.The paper argues that same-patient pairs expose inconsistent responses to nuisance variation among images with shared semantic content.Source: §9 · p. 9
  • CE-supervised models completely separate from SSL models (Cliff’s δ=1.0), the paper’s strongest empirical finding and one robust to architecture control.The authors relate this pattern to the view that SSL objectives prioritize representation quality over decision-boundary sharpness.Source: §9 · p. 9
  • ATC and AoTL have a small edge on MIMIC-CXR because they observe the target distribution, whereas STAMP is computable at training time.That timing difference matters when target data are unavailable before deployment.Source: §9 · p. 9
  • STAMP costs approximately 12 seconds per model using 4,000 cached source images, independent of target-dataset size.ATC requires 32,010 target images; AoTL requires all 44 models on the target set, about 1.1 million images and 56 minutes.Source: §9 · p. 9
  • STAMP requires semantic source-domain pairs; synthetic pairs yield lower correlation, class mismatch degrades macro-STAMP, and concept drift violates an assumption.The authors also caution that STAMP is a ranking signal, not a mechanistic decomposition of individual failure modes.Source: §9 · p. 9

10 CONCLUSION

The conclusion presents STAMP as a source-only predictor of OOD generalization and reports strong medical and natural-image results. The accompanying theoretical analysis establishes consistency under specified sampling assumptions, develops conditional ranking arguments, and documents empirical checks and limitations.

10 CONCLUSION

  • STAMP reaches ρs ≥0.844 on three medical OOD datasets and ρs ≥0.905 on five natural-image benchmarks, using 44 and 27 models, respectively, without target-domain data.The conclusion also identifies pretraining objective as the dominant moderator, with CE-supervised models outperforming SSL/VLM probes (Cliff’s δ=1.0).Source: §10 · p. 9

A.1 ASSUMPTIONS AND SETUP

  • Under A1–A6, the analysis assumes label-preserving bounded covariate shift, a Lipschitz output map, and error behavior related to within-class variance and inter-class margin.The empirical identifiability conditions are verified post hoc rather than imposed as premises.Source: §A.1 · p. 14

A.2 CONSISTENCY OF STAMP

  • For i.i.d. stable and random pair sampling with nonzero output variance, STAMP consistently estimates η² at rate Op(n^-1/2).This result applies directly to class-conditional or label-matched pairs; temporal pairs require separate assumptions about dependence.Source: §A.2 · p. 14

A.3 MARGIN PRESERVATION UNDER COVARIATE SHIFT

  • Under its stated shift and alignment assumptions, the margin analysis connects STAMP to an OOD error bound, while its class-averaged form is approximate.The approximation’s relative error is bounded by CV² and becomes exact as CV approaches zero.Source: §A.3 · p. 15, §A.4 · p. 16

A.4 APPROXIMATION ERROR

  • The class-averaged bound is not deterministic: across 44 models, CV=1.07 gives a worst-case relative bound of approximately 1.14.The analysis identifies this as a bounded-error approximation.Source: §A.4 · p. 16

A.6 APPROXIMATE RANK CONSISTENCY AND FAILURE MODES

  • Higher STAMP is monotonically associated with higher target accuracy under A1–A5 when the empirical margin ordering holds; class mismatch and concept drift are failure modes.Macro-STAMP gives ρ=0.311 on RSNA, while class-matched Pneumonia STAMP gives ρ=0.663 (p < 10^-4); the full model ordering remains empirical.Source: §A.6 · p. 16

A.7 PARTIAL CORRELATION AND SHIFT MAGNITUDE

  • Across the five natural-image benchmarks, partial correlations are 0.935 on ObjectNet, 0.804 on Dollar Street, 0.755 on ImageNet-A, 0.700 on ImageNet-R, and 0.662 on ImageNet-C.The analysis characterizes the expected increase in partial association with shift magnitude as a structural heuristic, not a formal monotonicity theorem.Source: §A.7 · p. 17

A.8 FINITE-SAMPLE CONCENTRATION

  • At n=m=500 and δ=0.05, the concentration bound gives ε≈0.067, while five-seed empirical standard deviation is <0.004.Temporal STAMP and labeled-pair STAMP estimate different functionals; temporal STAMP is not a consistent estimator of labeled-pair within-class scatter.Source: §A.8 · p. 17, §A.9 · p. 17

B.1 FULL COMPONENT ABLATION

Across the ablations and extensions, STAMP’s full variance ratio is more predictive than either component alone, and temporal pairs provide a stronger signal than label-matched pairs. The results also examine robustness, failure boundaries, deployment screening, and performance across medical and natural-image shifts.

B.1 FULL COMPONENT ABLATION

  • ρs=0.89 for the full 1−SV/AV ratio, versus −0.19 for SV alone and 0.64 for AV alone on VinDr-CXR.The ratio jointly rewards low within-class and high between-class variance; Table 7 covers 40 custom models with full ablation caches.Source: Table 7 · p. 18, §B.1 · p. 18

B.4 CONTROLLED ARCHITECTURE EXPERIMENT

  • With shared-backbone ViT-B/16 checkpoints, STAMP rank matches VinDr OOD rank exactly (ρs=1.000); NIH-ViT models lead NIH-CNN and NIH-Meta, while NIH-trained regimes separate from foundation probes.Across all 44 models, CE-supervised STAMP scores cluster at [0.754, 0.805], SSL probes score [0.666, 0.754], VL-Contrastive probes score [0.626, 0.701], and TXV models span [0.493, 0.731].Source: §B.4 · p. 19, §B.5 · p. 19, Figure 4 · p. 19

B.6 PARTIAL CORRELATION ANALYSIS

  • After controlling for NIH AUROC and parameter count, STAMP retains partial ρs=+0.536; on probit scale, it predicts residual OOD deviations and exceeds NIH-AUROC on class-matched RSNA.Residual correlations are +0.702 on VinDr, +0.750 on CheXpert, and +0.662 on MIMIC; on RSNA, STAMP and NIH-AUROC correlate at r=0.852 and r=0.596, respectively.Source: §B.6 · p. 20, §C.1 · pp. 22–23, Figure 10 · p. 23

B.7 FEATURE-LEVEL STAMP

  • Layer-wise STAMP rises by +0.12 to +0.22 from early to penultimate layers in ViT and Swin families, while CNN trajectories are flatter.The mid-to-penultimate STAMP gain and final-layer STAMP score are the dominant VinDr AUROC predictors (ρs=+0.701 and +0.795; n=35).Source: §B.7 · p. 20, Figure 6 · p. 20

B.7.1 NUISANCE SUPPRESSION

  • The nuisance ratio is negatively correlated with STAMP (ρs=−0.386) and VinDr AUROC (ρs=−0.393), but only 13/35 models satisfy the full predicted variance ordering.The ordering violations do not affect the main STAMP correlations, which use only the temporal-versus-random contrast.Source: §B.7.1 · p. 21, Figure 8 · p. 21, §B.7.2 · pp. 21–22

B.8 PRE-DEPLOYMENT MODEL SCREENING: FULL THRESHOLD ANALYSIS

  • At δ≤0.821, pre-deployment screening achieves Precision=1.0 and a +0.21 to +0.33 F1 advantage over always-alarm; the largest rank disagreements reveal dataset-specific variance suppression and output saturation.STAMP costs about 12 seconds per model on one GPU and needs no target-domain data; its advantage over always-alarm narrows to +0.03 at δ≥0.87.Source: §B.8 · p. 22, §B.9 · p. 22, Figure 9 · p. 22

C.2 DOLLAR STREET: GEOGRAPHIC AND INCOME-STRATIFIED SHIFT

  • On Dollar Street, STAMP-TS reaches ρs=0.940 versus 0.902 for the ImageNet-1K accuracy baseline; income-stratified correlations are 0.873 for high-income and 0.777 for low-income households.The CE-Supervised distribution separates completely from SSL and VL-Contrastive groups (Cliff’s δ=+1.0), while ImageNet-C correlations are positive for all 15 corruption types, ranging from 0.756 to 0.929.Source: §C.2 · p. 23, §C.3 · pp. 23–24, §D.4 · p. 26

B.2 TEMPORAL VS. LABELED PAIRS

  • Temporal STAMP exceeds labeled STAMP by +0.110 to +0.186 (mean +0.161) across all 31 custom models.The authors attribute the difference to label noise contaminating the labeled pair pool; follow-up gaps from 1 to 10 visits leave scores unchanged to four decimal places for three probes, and N=2,000 is stable across seeds.Source: §B.2 · p. 18, §B.3 · p. 18, Figure 3 · p. 19

D.5 STATISTICAL POWER ANALYSIS

The analysis reports that the study’s sample sizes provide high power for its observed correlations and that the results are stable to removing individual models.

D.5 STATISTICAL POWER ANALYSIS

  • At n=27 and n=44, all reported natural-image and medical correlations have power >0.998; removing any single model shifts reported ρs by at most 0.026 without changing sign or significance.The power estimates use exact permutation-based simulation with 10,000 Monte Carlo samples per cell.Source: §D.5 · p. 27, Table 12 · p. 27

Related papers

Loading 2609.32672v1…