PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Multi-Layer Subspace Modeling for Cross-Domain OOD Detection

Gerhard Krumpl, Henning Avenhaus, Horst Possegger

arXiv:2609.06785v1cs.CV

TL;DR

Neural networks can be overconfident on inputs outside their training distribution, and OOD detectors often rely on representations whose usefulness varies across domains and OOD types. PRISM fuses shallow-to-deep features and models them in an ID-learned subspace; across natural, medical, and industrial benchmarks, it achieves the best overall mean AUROC and average rank with a shared configuration.

  • Problem

    OOD detection is needed because models may make highly confident, incorrect predictions on out-of-distribution inputs, while the most informative representation layer varies by dataset and OOD type.

    Source: §1 · pp. 1–2

  • Method

    PRISM fuses shallow-to-deep representations into one feature space, learns a compact subspace from ID data, and combines projected distance with residual-based signals.

    Source: §1 · p. 3

  • Results

    Across natural, medical, and industrial benchmarks, PRISM achieves the best overall mean AUROC and average rank with a shared configuration and no OOD-based calibration.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The findings support fused shallow-to-deep ID geometry as an approach for more consistent cross-domain OOD behavior without per-benchmark layer selection or calibration.

    Source: §5 · p. 15

    PRISM uses fixed, architecture-aware layer sampling and global pooling; more task-specific weights or adaptive pooling could improve individual benchmarks but add priors and hyperparameters that may reduce transferability.

    Source: §5 · p. 15

Abstract

from arXiv · show

Out-of-Distribution (OOD) detection remains a fundamental challenge for neural networks, whose predictions can be overconfident on inputs that deviate from the training distribution. Most post-hoc OOD detection methods derive scores from a single representation level (eg., logits or penultimate features) or combine multiple layers via depth selection or OOD-calibrated weighting. However, because OOD shifts are diverse, the most informative representation level can vary strongly across OOD types and domains, making fixed-layer choices and OOD-calibrated aggregation brittle. In this paper, we propose PRISM (Projected Representation with Intermediate-layer Subspace Modeling), a model-agnostic post-hoc OOD detection method that models a unified multi-layer feature representation rather than aggregating independently scored layers. PRISM fuses intermediate and deep features into a single hierarchical embedding, estimates an in-distribution (ID) principal subspace, and then combines two complementary signals: (i) a class-conditional Mahalanobis distance in the projected subspace and (ii) the residual energy orthogonal to the learned manifold. This simple design avoids OOD-tuned layer weighting while capturing both in-subspace semantic deviations and off-subspace anomalies. Across diverse benchmarks spanning natural images, medical imaging, and industrial visual inspection, PRISM achieves consistent state-of-the-art cross-domain OOD detection performance with a single default configuration across all evaluated domains and architectures. We further show that PRISM incurs minimal inference overhead, making it practical for real-world deployment.

1 Introduction

OOD detectors can behave inconsistently across domains because useful representation depth varies with the dataset and OOD type. PRISM fuses shallow-to-deep features before statistical modeling, and achieves the best overall mean AUROC and average rank across evaluated benchmarks with a shared configuration.

1 Introduction

  • OOD detection matters because models can make highly confident but incorrect predictions on inputs outside the training distribution, risking silent failures in high-consequence applications.Examples include diagnostic errors from scanner variability and missed damage in industrial inspection.Source: §1 · pp. 1–2
  • The most informative layer varies with the ID dataset and OOD type, while intermediate and shallow features can provide cues that final representations miss.Existing approaches select one layer, aggregate layer-wise scores, or calibrate choices using synthetic or held-out OOD data.Source: §1 · pp. 2–3
  • PRISM fuses shallow-to-deep representations into one feature space, then models that representation using ID data and combines distance- and residual-based signals.This avoids layer selection, layer-wise aggregation heuristics, and OOD-dependent calibration.Source: §1 · p. 3
  • Layer-wise aggregation and OOD-based calibration are sensitive to the chosen validation OOD set, producing unstable cross-domain behavior.The proposed unified multi-layer subspace modeling principle aims to avoid those dependencies.Source: §1 · p. 3
  • Across natural, medical, and industrial benchmarks, PRISM achieves the best overall mean AUROC and average rank under a shared configuration without OOD-based calibration.The evaluation uses one configuration across the benchmarks.Source: §1 · p. 3

2 Related Work

Prior post-hoc OOD methods use model outputs, deep features, or multi-layer information, but multi-layer scoring often adds aggregation and calibration choices. PRISM instead models a fused shallow-to-deep representation in a low-dimensional ID subspace.

2 Related Work

  • Output-based post-hoc methods score prediction probabilities or logits, while feature-based methods commonly model penultimate-layer distributions or distances.Examples include MSP, MLS, energy-based detection, Mahalanobis distance, nearest neighbors, and ViM.Source: §2 · pp. 3–4
  • Multi-layer approaches use intermediate activations or layer-specific statistics, then aggregate scores, often with extra design choices or OOD-dependent calibration.Examples include ATS, GRAM, CORES, NAP, and multi-layer Mahalanobis methods.Source: §2 · p. 4
  • Many detectors validated on CIFAR or ImageNet-based benchmarks show reduced reliability in medical and industrial settings, motivating domain-agnostic feature geometry.The cited work notes that feature-space statistical and distance-based approaches have been observed to be more stable across settings.Source: §2 · p. 4
  • PRISM jointly uses shallow-to-deep representations and models dominant ID structure in a low-dimensional subspace, rather than selecting layers or aggregating per-layer scores.Its unified score avoids aggregation heuristics, OOD-based calibration, and per-benchmark tuning.Source: §2 · pp. 4–5

3 Method

Section 3 frames OOD detection as thresholding a score and reviews how sensitivity to feature depth and OOD calibration motivates PRISM. PRISM fuses representations across layers, models ID structure in a PCA subspace, and combines projected distance with residual energy.

3.1 Problem Statement

  • OOD detection is a binary decision based on a score, with the threshold typically set to retain a high fraction of ID samples.The passage gives 95% true positive rate as an example.Source: §3.1 · p. 5

3.2 Mahalanobis Distance for OOD Detection

  • Mahalanobis detection is sensitive to feature depth, while OOD-calibrated layer weights vary with the validation set, motivating fusion without OOD-dependent calibration.The cited figure passages compare shallow, deep, and full-hierarchy features and describe evaluation across benchmark tasks.Source: §3.2 · p. 6, Figure 2 · p. 6, Figure 3 · p. 7

3.3 PRISM: Projected Representation with Intermediate-layer Subspace Modeling

  • PRISM pools, normalizes, and concatenates features from selected layers into one vector that captures low-level structure and high-level semantics.The pooling is architecture-agnostic and applies to CNN feature maps and ViT token sequences.Source: §3.3 · pp. 7–8
  • PCA on fused ID representations estimates a principal subspace that retains dominant ID variability while discarding weakly supported low-variance directions.The subspace is estimated from ID training data.Source: §3.3 · p. 8
  • At inference, PRISM separates projected coordinates aligned with dominant ID patterns from residual components orthogonal to the learned ID manifold.The projected component represents in-subspace patterns, while the residual captures components outside the manifold.Source: §3.3 · p. 8
  • PRISM combines projected Mahalanobis distance and residual-energy anomaly magnitudes multiplicatively, without balancing coefficients or OOD-dependent calibration.The product emphasizes samples deviating from ID structure both within and outside the principal subspace.Source: §3.3 · pp. 8–9

4 Experiments

Experiments evaluate PRISM across natural, medical, and industrial OOD benchmarks, comparing it with 22 post-hoc methods and examining setup, ablations, architectural generalization, and inference cost. PRISM achieves the best overall mean AUROC across the three domains, while the ablations assess its components and robustness to design choices.

4.1 Experimental Setup

  • Evaluation covered three benchmarks, 21 OOD datasets, and natural, medical, and industrial shifts, using benchmark backbones and a fixed PCA dimension of d = 512.The PCA dimension was held constant across datasets, domains, and backbones.Source: §4.1 · pp. 9–10

4.2 Benchmark Results

  • PRISM maintained high AUROC from near- to extreme-OOD, while competing approaches showed pronounced domain-dependent variability.Reported errors often involved ID–OOD overlap or subtle near-OOD confusions.Source: §4.2 · pp. 10–12, Figure 4 · p. 14
  • 93.92% mean AUROC gave PRISM the best overall result across three domains, 4.79 points above MDS++ (89.13%).Table 1 compares methods across the natural, medical, and industrial benchmarks.Source: §4.2 · p. 10, Table 1 · p. 11
  • PRISM scored 90.36% on OpenOOD, 94.86% on OpenMIBOOD, and 96.56% on ICONIC-444, while competitors' strengths varied by domain.It was the strongest ID-only alternative to MDSEns on OpenMIBOOD and closely matched GRAM on ICONIC-444.Source: §4.2 · p. 11

4.3 Ablation Study

  • Either score alone improved over penultimate-layer baselines when computed on the unified multi-layer representation.The reported gains were attributed to modeling fused low-, mid-, and high-level cues, not merely combining scoring terms.Source: §4.3 · pp. 12–13
  • PRISM ranked among the top-performing methods across OpenOOD and ICONIC-444, independent of the evaluated architecture.The comparisons included convolutional and transformer-based backbones.Source: Table 3 · p. 13, §4.3 · pp. 13–14
  • 93.47% overall average AUROC and 23.65% FPR95 were achieved by combining the two scores, although component dominance varied by domain.The projected term performed strongest on OpenOOD and ICONIC-444; the residual term dominated on OpenMIBOOD.Source: §4.3 · pp. 12–13
  • Performance varied only slightly across a broad range of PCA dimensions; d = 512 gave the best overall performance.The reported dependence on projection dimension was generally weak.Source: §4.3 · p. 13

5 Conclusion

PRISM responds to depth-dependent OOD performance by modeling fused representations rather than relying on a single layer or aggregated layer-wise scores. It is presented as a consistent cross-domain default, with fixed layer sampling and global pooling chosen to preserve transferability.

5 Conclusion

  • Neither a single layer nor layer-wise score aggregation is consistently reliable across domains, and OOD-calibrated strategies do not robustly transfer.The conclusion motivates preserving information across representation depths rather than relying on benchmark-specific depth preferences.Source: §5 · p. 15
  • PRISM fuses shallow-to-deep representations before statistical modeling, combining projected class-conditional Mahalanobis distance with orthogonal residual energy.This avoids per-benchmark layer selection, aggregation heuristics, and OOD-based calibration.Source: §5 · p. 15
  • Fixed, architecture-aware layer sampling and global pooling support an ID-only evaluation protocol, while tuned weights or adaptive pooling could improve individual benchmarks but add priors and hyperparameters that may reduce transferability.Developing adaptive depth or pooling strategies that retain transferability is identified as future work.Source: §5 · p. 15

A.1 Implementation Details

The implementation uses OpenOOD, incorporating OpenMIBOOD and ICONIC-444 into a shared evaluation pipeline. It adds selected detection methods and modern model architectures, with runtime benchmarks measured on an RTX A4000.

A.1 Implementation Details

  • OpenOOD provides the standardized evaluation framework, with OpenMIBOOD and ICONIC-444 integrated to unify data handling, preprocessing, and protocols across domains.The integrated benchmarks cover natural, medical, and industrial settings.Source: §A.1 · p. 21
  • The implementation adds FeatNorm, ATS, and MDS++ and supports timm models, including ConvNeXt and Vision Transformers.OpenMIBOOD is incorporated as a fork of OpenOOD.Source: §A.1 · p. 21
  • Runtime and throughput benchmarks were measured on an RTX A4000; experiments used servers with Intel i9-9900X CPUs and NVIDIA GPUs.The systems ran Ubuntu 22.04, Python 3.11.3, PyTorch 2.7.1, CUDA 12.6, and cuDNN 9.5.1.Source: §A.1 · p. 21

A.2 Benchmark Setup

The benchmark setup combines natural, medical, and industrial evaluations, with near-, far-, and extreme-OOD categories used where applicable. Protocols specify model sources, preprocessing, calibration requirements, and PRISM’s ID-based feature and subspace estimation.

A.2 Benchmark Setup

  • The evaluation maps each benchmark and ID task to near-, far-, and extreme-OOD sets, with ICONIC-444 defining extreme shifts as large, semantically unrelated cross-domain changes.For benchmarks without official extreme-OOD categories, additional datasets standardize robustness analysis without changing official categories.Source: §A.2 · p. 21, Table 4 · p. 22
  • OASIS-3 follows official OpenMIBOOD preprocessing, including 1mm3 isotropic resampling and HD-BET skull-stripping; 2D OOD images are converted to grayscale and replicated across channels.Inputs are resized for backbone compatibility.Source: §A.2 · p. 22
  • Models use official OpenOOD and OpenMIBOOD checkpoints, supervised torchvision and timm models for ImageNet-1k, and the official training protocol for ICONIC-444.The architecture and checkpoint overview is provided in Table 5.Source: §A.2 · p. 22, Table 5 · p. 23
  • The evaluation uses a unified pipeline across OpenOOD, OpenMIBOOD, and ICONIC-444, with benchmark-specific hyperparameter searches following official procedures and merged search ranges.The search spaces and calibration requirements are summarized in Table 6.Source: §A.2 · pp. 22–23, Table 6 · p. 24
  • Primary comparisons exclude methods requiring real OOD calibration, except MDSEns as a strong OpenMIBOOD baseline; other methods use ID training data for calibration.FeatNorm uses synthetic OOD samples from ID data, while ImageNet-1k uses 20% of its official training split for ID calibration.Source: §A.2 · p. 23
  • PRISM fits PCA on class-balanced ID calibration samples, retains 512 components by default, and estimates class-conditional means with a shared covariance and ridge regularization λ = 10−3.Layer taps vary by architecture: four stage outputs for ResNet-like and ConvNeXt models; spaced blocks for ViT and Swin-T; and all seven blocks for the compact transformer.Source: §A.2 · p. 23

B Detailed Results

Across heterogeneous OOD datasets, PRISM has the best mean AUROC rank, while detailed results show substantial ranking changes across datasets, domains, and OOD categories. Its false accepts reflect different kinds of ambiguity across benchmarks, and ablations favor the fused, normalized subspace design.

B Detailed Results

  • No detector is uniformly best: method rankings change across datasets and near-, far-, and extreme-OOD categories, while PRISM remains consistently among the top performers.PRISM is characterized as a robust ID-only default across natural, medical, and industrial settings.Source: B Detailed Results · pp. 24, 29
  • 94.50% AUROC on CIFAR-10 and 86.62% AUROC on CIFAR-100 are PRISM’s strongest aggregated results, although individual datasets still produce baseline wins.The cited tables report ResNet-18 results for CIFAR-10 and CIFAR-100.Source: B Detailed Results · p. 24, Table 7 · p. 26, Table 8 · p. 26
  • On ImageNet-1k with ResNet-50, PRISM reaches 89.96% AUROC and 35.36% FPR95 overall, but deep-feature baselines outperform it on SSB-Hard.RMDS records 75.47% AUROC and 70.78% FPR95 on SSB-Hard; the section also notes PRISM’s strength on several far- and extreme-OOD sets.Source: B Detailed Results · p. 25, Table 9 · p. 31
  • Across MIDOG, PhaKIR, and OASIS-3, PRISM performs strongly under ID-only calibration, while medical-task results show substantial cross-task re-ranking.GRAM records 83.20% AUROC on MIDOG but 36.46% AUROC on PhaKIR; MDSEns requires real OOD validation data for aggregation calibration.Source: B Detailed Results · p. 25, Table 11 · p. 34
  • On ICONIC-444, PRISM has the best architecture-average AUROC (95.67%) across three backbones, while GRAM’s performance varies substantially by architecture.GRAM records 97.55 AUROC and 8.41% FPR95 on ResNet-18, but 88.75% AUROC and 34.64% FPR95 on CCT-7/7x2.Source: B Detailed Results · p. 25, Table 10 · p. 32
  • PRISM achieves the best mean AUROC rank (3.04, 95% CI [2.48, 3.61]); at α = 0.05, its rank gaps over GRAM (3.25) and MDS++ (3.29) exceed CD 2.08.A Friedman test rejects equal performance across methods (χ2 = 1185.59, p ≪0.001).Source: B Detailed Results · pp. 25–26, Figure 6 · p. 27
  • ImageNet false accepts often involve ID–OOD overlap or fine-grained ambiguity, whereas medical and industrial errors more often reflect subtle shifts, artifacts, and similarities among categories.PhaKIR and almond examples also reflect how the ID task’s class structure shapes borderline assignments; across settings, false accepts remain largely class-consistent and interpretable.Source: B Detailed Results · pp. 26–28, Figure 7 · p. 29, Figure 8 · p. 30
  • 93.47% AUROC / 23.65% FPR95 is PRISM’s average across large-scale benchmarks, ahead of MDS++ and ViM; the ablation favors fusion, subspace modeling, and pre-PCA normalization.Removing normalization yields 87.87% AUROC / 36.82% FPR95; one fused-space set of class-conditional statistics also avoids the repeated layer-wise modeling described for Mahalanobis methods.Source: B Detailed Results · pp. 28–29
Loading 2609.06785v1…