CORE-STACK+: Meta-Learning for Deep Stacked Generalization

Motivation

• Near-collinear out-of-fold predictions make the Gram matrix ill-conditioned and the stacking optimum unstable.Small perturbations in out-of-fold estimates can produce widely different meta-weights.Source: §1 · p. 1
• Naive linear stacking can compound constituent miscalibration, yielding worse ECE than the worst single model.The passage notes that post-hoc temperature scaling may be needed to recover calibration [47].Source: §1 · p. 2

Why existing remedies are insufficient

• Ridge regularization, greedy selection, and weight-space fusion each leave a gap for heterogeneous prediction pools.The described limitations include unresolved redundancy, combinatorial cost, and architectural homogeneity requirements.Source: §1 · p. 2

Our approach

• CORE-STACK+ replaces four stacking heuristics with CKA pruning, a learned gate, spectrum-adaptive regularization, and Bayesian blending.The pipeline also introduces a PAC-Bayes bound coupling prediction-space redundancy and meta-learner capacity.Source: §1 · p. 2

Weight-space fusion

• Weight-space fusion methods exploit a shared loss landscape but presuppose a common architecture and training regime.The passage names SWA/SWAG [22] [29], snapshot ensembling, and model soups [50].Source: §2 · p. 2

Prediction-space aggregation

• Ridge and elastic-net shrink weights without removing redundant members, while greedy selection scales poorly and ignores nonlinear dependence.The related-work discussion contrasts these with the Pearson-based CORE-STACK prototype, extended here through CKA/HSIC.Source: §2 · pp. 2–3

Diversity-aware training

• Diversity-aware training encourages disagreement but provides no principled post-hoc pruning criterion for supplied heterogeneous pools.Examples include MC dropout [11], hyperparameter ensembles [47], and repulsive ensembles [5].Source: §2 · p. 3

Calibration and generalization

• Post-aggregation calibration remedies are decoupled from meta-learning, while prior ensemble bounds do not tie generalization penalties to measurable pool redundancy.The passage cites temperature scaling and Platt scaling, and contrasts prior PAC-Bayes work [33] [32] [31] with CORE-STACK+.Source: §2 · p. 3

3.1 Preliminaries and Pipeline Overview

• The pipeline transforms out-of-fold predictions through CKA projection, meta-feature gating, closed-form regularization, and Laplace-approximate blending.The base predictors may produce classification, segmentation, or detection outputs, flattened into prediction vectors or maps.Source: §3.1 · p. 3

3.2 Kernelized Redundancy Projection using CKA

• CKA detects nonlinear prediction dependence that Pearson correlation can miss, and removes a model when a lower-risk retained model exceeds the CKA threshold.The passage states that CKA-based redundancy projection properly generalizes Pearson-based projection; Nyström approximation and optional LSH address scaling.Source: §3.2 · pp. 3–4

3.3 Differentiable Meta-Feature Gate

• A sample-conditioned gate learns attention over ensemble statistics, using a two-layer MLP with a sigmoid head trained end-to-end.The statistics include prediction mean, standard deviation, median, range, quartiles, entropy, and divergence from the best single model.Source: §3.3 · p. 4

3.4 Spectrum-Adaptive Regularization

• Under the cluster and sub-Gaussian-noise assumptions, the optimal ridge penalty is λ⋆ = λmax(Chat)/SNR(Chat).The closed form is reported within 5% of the CV-optimal value and eliminates the outer cross-validation loop.Source: §3.4 · pp. 4–5

3.5 Laplace-Approximate Bayesian Blender

• The Laplace blender uses fitted meta-learners’ loss and Hessian curvature to weight their predictions, unlike inverse-RMSE weighting.The stated rationale is to down-weight overconfident meta-learners whose flat Hessians indicate fold-specific overfitting.Source: §3.5 · p. 5

3.6 Theoretical Guarantees

• The PAC-Bayes bound includes a redundancy term tied to the prediction Gram spectrum, and the paper states that projection reduces it versus the unprojected bound.Under the same assumptions, the Laplace blender’s expected ECE is bounded by the maximum ECE among the fitted meta-learners.Source: §3.6 · p. 5

Benchmarks

• Six benchmarks cover classification, corruption robustness, dense prediction, long-tail recognition, and domain shift.They are ImageNet-1K, ImageNet-C, ADE20K, COCO, iNaturalist-2021, and DomainNet-126.Source: §4 · p. 5

Backbone pool

• The ImageNet-1K pool contains 14 public backbones across convolutional, transformer, hierarchical-transformer, and modernized-convolutional families.ADE20K and COCO use task-specific UperNet and Mask R-CNN heads, respectively.Source: §4 · p. 5

Baselines

• Comparisons include twelve baselines, from single-model and averaging methods to deep ensembles, SWAG, AutoGluon, and CORE-STACK.The list also includes Ridge stacking, greedy hill climbing, model soups, snapshot ensembles, and MC dropout.Source: §4 · p. 6

Metrics and protocol

• Evaluation reports task metrics, ECE and NLL, robustness, and efficiency on held-out splits using out-of-fold predictions.The protocol uses 95% bootstrap confidence intervals from 5,000 resamples and Bonferroni-corrected paired t-tests.Source: §4 · p. 6

5 Results

• CORE-STACK+ achieves the best ImageNet-1K top-1 and calibration without post-hoc correction, while retaining fewer models than any multi-member baseline.Table 1 reports top-1, ECE, NLL, retained models, and test FLOPs relative to the full 14-model ensemble.Source: Table 1 · p. 6

5.1 Robustness: ImageNet-C

• CORE-STACK+ reduces ImageNet-C mCE by 4.2 points versus deep ensembles and 2.6 versus CORE-STACK, with the largest family gains on Noise (−7.4) and Blur (−4.4).Its ∆vs. clean gap narrows from 42.1 for deep ensembles to 35.7, and it improves all four corruption families.Source: §5.1 · p. 6, Table 2 · p. 7

5.2 Dense Prediction: ADE20K and COCO

• On ADE20K, CORE-STACK+ reaches 53.0 mIoU, +0.9 over CORE-STACK, retaining 5 of 8 models; on COCO, it gains +1.3 AP over CORE-STACK with 4 of 7 retained.The results use flattened spatial predictions; relative FLOPs are 0.63× on ADE20K and 0.58× on COCO.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Long-Tail Recognition: iNaturalist-2021

• CORE-STACK+ reaches 74.2% overall top-1 on iNaturalist-2021, +1.1% over CORE-STACK, with a +2.3% gain on Tail versus +0.6% on Head.CKA preserves rare-class specialists that Pearson-based pruning can remove as redundant.Source: §5.3 · p. 7, Table 4 · p. 8

5.4 Domain Shift: DomainNet-126

• On DomainNet-126, CORE-STACK+ improves average transfer accuracy by +2.1% over deep ensembles, with gains of +2.0% on Clipart, +2.2% on Painting, and +2.3% on Sketch.Base models are trained on Real and evaluated zero-shot on the three target domains, without adaptation or fine-tuning.Source: §5.4 · p. 8, Table 5 · p. 8

5.5 Component Ablation

• The cumulative ImageNet-1K ablation attributes +0.5% top-1 and a 28% lower condition number to CKA projection, then +0.2% and a 0.002 ECE reduction to the feature gate.The gate suppresses 2–3 high-variance, low-signal prediction dimensions. Spectrum-adaptive regularization is accuracy-neutral (−0.1%) but cuts training from 625 s to 194 s; Bayesian blending adds +0.3% accuracy and reduces ECE from 0.023 to 0.018, with one further model removed.Source: §5.5 · pp. 8–9, Table 6 · p. 9

5.6 Efficiency, OOD Detection, and Scaling

• CORE-STACK+ raises A100 throughput from 312 to 528 img/s, a 1.69× speedup, and its entropy scores beat deep ensembles by +3.1 AUROC while matching SWAG (83.9 vs. 84.5).It retains 40–50% of submitted models as pool size grows from 4 to 20; accuracy and ECE improve through K = 14 before plateauing.Source: §5.6 · p. 9

• CORE-STACK+ replaces Pearson filtering, hand-crafted features, cross-validated ridge, and inverse-RMSE weighting with CKA pruning, a differentiable gate, spectrum-adaptive regularization, and Laplace blending.Source: §6 · p. 9
• Across six vision benchmarks, CORE-STACK+ improves top-1, ECE, NLL, mCE, mIoU, AP, and AUROC while retaining fewer models than any competitive ensemble.Source: §6 · p. 9
• The PAC-Bayes bound couples the KL penalty to the Gram spectrum of the pruned prediction matrix; proposed extensions include weight-space integration and online adaptation under distribution shift.The paper also reports releasing code and OOF matrices for reproducibility.Source: §6 · p. 9

Impact Statement

• CORE-STACK+ replaces four deep-stacking heuristics and reports improved accuracy, calibration, and robustness with 35–57% fewer retained models and up to 41% lower inference FLOPs.Source: Impact Statement · p. 10
• The authors identify potential value for resource-constrained deployment, uncertainty-aware perception, and decision-making in autonomous driving, medical imaging, and robotics.Source: Impact Statement · p. 10
• The statement flags dataset bias, distribution shift, and misuse in surveillance or autonomous decision-making as risks, and calls for responsible deployment and continued scrutiny.Source: Impact Statement · p. 10

A Theoretical Justification and Theorems

• The appendix develops results on CKA projection, spectral preconditioning, spectrum-adaptive ridge, gate approximation, PAC-Bayes generalization, Bayesian blending, leakage-freeness, calibration, and complexity.Source: A Theoretical Justification and Theorems · p. 16

B Notation and Standing Assumptions

• The OOF prediction matrix is formed by training each predictor outside a fold and evaluating it on that held-out fold; dense-task predictions are vectorized.Source: B Notation and Standing Assumptions · p. 16
• The analysis assumes predictors form disjoint CKA clusters, with high within-cluster and bounded across-cluster dependence, bounded meta-features and targets, and isotropic sub-Gaussian residual noise.For jointly Gaussian predictions with a Gaussian kernel, population CKA is a strictly monotone transform of absolute Pearson correlation.Source: B Notation and Standing Assumptions · p. 16

C.1 CKA strictly dominates Pearson correlation

• CKA detects dependencies Pearson correlation misses: for universal characteristic kernels, zero CKA implies independence, and a dependent pair can have zero correlation but positive CKA.For example, Q = P^2 with P standard normal has zero correlation with P but is deterministically dependent on it.Source: §C.1 · pp. 16–17

C.2 Concentration of the empirical HSIC

• With probability at least 1 − δ, empirical HSIC concentrates around population HSIC at a rate proportional to 1/√N, supporting high-probability recovery of the cluster structure.The stated sample-size condition is N ≳ (Cκ/ε)^2.Source: §C.2 · p. 17

C.3 Block structure of C under CKA clustering; D Spectral Preconditioning

• CKA clustering yields a block-structured correlation matrix, and projecting to one representative per cluster improves the stated condition-number bound relative to the unprojected matrix.The bound concerns C and its post-projection counterpart Ceff.Source: §C.3 · p. 17, D Spectral Preconditioning · p. 18

E.1 Signal-noise decomposition; E.2 Closed-form optimal penalty

• The spectrum-adaptive Ridge penalty is λ⋆ = λmax(Chat)/SNR(Chat), and replacing the inner-CV search with this choice preserves the excess-risk bound up to 1 + o(1).The penalty is bounded between the spectral noise threshold and the largest signal eigenvalue.Source: §E.2 · p. 18

G Differentiable Meta-Feature Gate: Approximation Theory

• The gate can approximate continuous target aggregators on compact sets, strictly contains the prototype’s hand-crafted feature class, and adds at most 15,000 parameters.At ImageNet-1K scale, the stated contribution to Rademacher complexity is approximately 0.011.Source: G Differentiable Meta-Feature Gate:… · p. 19

H.2 The PAC-Bayes excess-risk bound; H.3 Comparison to prior PAC-Bayes ensemble bounds

• The supplementary PAC-Bayes bound combines a KL penalty with a redundancy-dependent term, and is strictly tighter than the prototype bound when CKA retains fewer clusters than Pearson pruning.On ImageNet-1K, the reported ratio GCKA/GPearson is approximately 6/9, corresponding to a 1.22× improvement.Source: §H.2 · p. 20, §H.3 · p. 20

I.2 Variance reduction: the oracle bound; I.3 Suboptimality of heuristic weights

• Under the oracle blending result, the minimum variance of a convex blend is no greater than that of any individual meta-learner, with equality only for rank-1 covariance.The Laplace-weight analysis states that flat Hessians indicate fold-specific overfitting and reports ΔECE = −0.005.Source: §I.2 · p. 21, §I.3 · p. 21

J Leakage-Freeness of the Nested OOF Construction

• Refitting the CKA threshold, gate parameters, and blending weights within each outer fold makes the full OOF pipeline leakage-free and supports unbiased OOF risk estimates.The unbiasedness statement is conditional on the complement training folds.Source: J Leakage-Freeness of the Nested OOF… · p. 21

K Calibration Guarantees

• The Laplace blender’s expected ECE is at most the maximum ECE among its constituent meta-learners, with equality only when the blend selects a single meta-learner.The bound follows from convexity of ECE and the blending weights summing to one.Source: K Calibration Guarantees · pp. 21–22
• ECE = 0.018 without temperature scaling, matching the SWAG result with temperature scaling.The paper attributes the result to Laplace blending down-weighting meta-learners with flat Hessians.Source: K Calibration Guarantees · p. 22

L Computational Complexity

• At ImageNet-1K scale, CORE-STACK+ is reported to be 3.2× faster than the CORE-STACK prototype because closed-form λ⋆ eliminates the L|Λ| factor in Phase 3.The stated setting uses N = 128K, L = 5, |Λ| = 50, M = 3, and deff ≈ 30.Source: L Computational Complexity · p. 22

M Summary of Theoretical Results; N Extended Analysis and Additional Experiments

• Table 8 summarizes theoretical guarantees and marks which are new versus sharpened or refactored from the CORE-STACK prototype analysis.The appendix also outlines extended analysis of baseline selection, evaluation protocol, threshold sensitivity, inference costs, and the condition-number/accuracy relationship.Source: Table 8 · p. 22, N Extended Analysis and Additional… · p. 23

M Summary of Theoretical Results

• The combined theoretical analysis states a PAC-Bayes excess-risk bound tighter than the unprojected baseline by a factor of at least √(K/G).The summary attributes the result to CKA-based projection and the resulting reduction in effective dimension.Source: M Summary of Theoretical Results · p. 23
• The cited comparison reports that GPearson/GCKA is tighter than the CORE-STACK prototype; the passage provides no ImageNet-1K ratio or numerical improvement.Source: M Summary of Theoretical Results · p. 23

N.1 Positioning Relative to Prior Stacking Work

• CORE-STACK+ extends Pearson pruning with CKA for nonlinear redundancy, derives a closed-form spectrum-adaptive Ridge penalty, and bounds risk jointly by prediction redundancy and meta-learner capacity.The Ridge penalty removes the nested cross-validation loop and yields a 3.2× wall-clock speedup at no accuracy cost.Source: §N.1 · p. 23

N.2 Baseline Selection and Scope

• At Nmeta ≈25K after an 80/20 split, a three-layer dropout MLP meta-learner achieved 83.2% top-1, 1.0% below Ridge stacking, supporting its exclusion for severe overfitting.CORE-STACK+ operates strictly post-hoc on pre-trained public checkpoints, so training-modifying diversity methods are described as complementary rather than directly comparable.Source: §N.2 · p. 24

N.3 Benchmark Coverage and Evaluation Protocol

• The six benchmarks probe distinct generalization axes, with evaluations on held-out splits; ImageNet-O, ADE20K, and COCO are external to meta-training.For ADE20K and COCO, base models are frozen and the meta-layer is fit only on training-set out-of-fold predictions.Source: §N.3 · p. 24, Table 9 · p. 24

N.4 Formal Definitions

• Out-of-fold predictions are leakage-free by construction, while Laplace blending weights depend on each meta-learner’s OOF loss and loss-minimum Hessian.The formal condition states that the meta-features are conditionally independent of the target given the training data excluding that sample’s fold.Source: §N.4 · p. 25

N.5 CKA Threshold Sensitivity

• Top-1 accuracy stays within 0.1% of optimum for τCKA ∈[0.80, 0.90]; a largest-gap CKA-histogram elbow is offered as a parameter-free threshold choice.Below 0.75, pruning removes diverse models; above 0.90, near-duplicates remain.Source: §N.5 · p. 25, Table 10 · p. 25

N.6 Deployment Efficiency: Latency and Memory

• 528 img/s matches the best single model despite six parallel forwards; peak memory is 17.9 GB, 37% below the 14-model ensemble, with +1.5% top-1 accuracy.CORE-STACK+ is reported as the only multi-backbone method fitting within a single 20 GB GPU in half precision.Source: §N.6 · pp. 25–26, Table 11 · p. 26

N.7 Condition Number as a Performance Predictor

• ρs = −0.91 (p < 0.001) links lower κ to higher top-1 across methods; as components are added, κ falls 392→218 while top-1 rises 84.5%→85.4%.Figure 1 places CORE-STACK+ at κ = 218 and 85.4% top-1; the ablation points trace the same direction.Source: Figure 1 · p. 26, §N.7 · p. 26