SVD-Based Typicality Maps for Out-of-Distribution Detection in Vision Transformers

Aldo Sean Sartor; Leandro de Souza Rosa; Andriy Enttsel; Mauro Mangia; Riccardo Rovatti

Source-linked AI summary

SVD-Based Typicality Maps for Out-of-Distribution Detection in Vision Transformers

Aldo Sean Sartor, Leandro de Souza Rosa, Andriy Enttsel, Mauro Mangia, Riccardo Rovatti

arXiv:2608.23499v1Aug 24, 2026cs.CV

TL;DR

The paper addresses how to characterize evolving ViT representations for OOD detection without relying only on output scores. It uses SVD-based projections and class-conditional density modeling to build typicality maps, then derives PAS and MLSV; on CIFAR-100, MLSV is the strongest unsupervised detector among those evaluated.

Problem
The geometry and depth-wise evolution of ViT representations are difficult to characterize, yet layer-wise class structure matters for interpretability and OOD detection.
Method
The method projects activations onto leading right singular vectors, fits class-conditional densities, forms typicality maps, and derives PAS and prototype-free MLSV scores.
Results
MLSV achieves mean AUROC 97.0% and mean FPR@95 11.9%, outperforming the strongest baseline by 1.7 pp in AUROC and 10.4 pp in FPR@95.
Takeaways & Limitations
Typicality maps provide a depth-resolved view of class-specific evidence, while MLSV detects OOD inputs without stored prototypes.

Abstract

from arXiv · show

We present a method for analyzing the internal representations of Vision Transformers (ViTs) exploiting the geometry of their learned parameters. Each affine layer's weight matrix is factored via Singular Value Decomposition (SVD), and activations are projected onto the leading right singular vectors to obtain compact, layer-intrinsic representations. A class-conditional density model is then fitted at each layer, producing per-class \emph{typicality scores} that are stacked across depth into \emph{typicality maps}: two-dimensional summaries of how class-specific evidence evolves through the network. From these maps, we derive two post-hoc scores for Out-Of-Distribution (OOD) detection: a \emph{Prototype Alignment Score} (PAS), measuring agreement with class reference prototype patterns, and a \emph{Multi-Layer Soft Voting} (MLSV) score, capturing cross-layer consensus without stored prototypes. On ViT-B/16 fine-tuned on CIFAR-100, the proposed scores achieve competitive detection performance without retraining or OOD exposure.

I. INTRODUCTION

The paper introduces a probabilistic, SVD-based framework for interpreting class-relevant structure across ViT depth and detecting OOD inputs. It replaces clustering-based processing with typicality maps and derives two post-hoc scores evaluated on CIFAR-100.

ViT representations are progressively transformed through self-attention and MLP blocks, motivating layer-wise analysis for interpretability and OOD detection.
The method projects activations onto directions that maximally amplify inputs and fits class-conditional densities in the resulting compact space.This produces per-layer, per-class typicality scores with a direct probabilistic interpretation.
Typicality maps summarize how class-specific evidence evolves through depth, using a simpler pipeline with fewer hyperparameters than MACS classification maps.
On ViT-B/16 fine-tuned on CIFAR-100, PAS and MLSV achieve competitive OOD detection against established baselines while retaining depth-resolved internal representations.
PAS measures agreement with class reference maps, while MLSV captures cross-layer consensus without stored prototypes.

II. MATHEMATICAL MODEL

The mathematical model treats selected affine layers as linear operators and uses their SVD geometry to construct compact representations of internal activations. Leading right singular vectors define the layer-intrinsic projection coordinates.

The pipeline selects L affine candidate layers and represents each layer’s transformation using an augmented input.
Each augmented layer operator is factorized with SVD, whose orthonormal matrices and ordered singular values characterize the operator’s geometry.
The projected representation uses the first k right singular vectors, aligning activations with the principal directions of the affine operator.
Choosing k ≪ dℓ+1 yields a compact, geometry-aware representation at each layer.

B. Class-Conditional Density Modeling

The method models projected activations separately for each layer and class using class-conditional Gaussian mixture densities. This replaces MACS’s clustering and feature-label association with likelihood-interpretable density modeling.

Training activations are projected at each selected layer, then grouped by class label for density estimation.
Each layer–class density is modeled as a Gaussian Mixture Model with M components.
The mixture weights, component means, and covariances parameterize each class-conditional density.
The replacement of clustering and feature-label association yields quantities with a direct likelihood-based interpretation.

C. Typicality Scores

Typicality scores provide probabilistic, layer- and class-specific measures of how likely an input representation is under class-conditional models. ECDF normalization makes these scores comparable across layers and classes.

At each layer, the method evaluates a class-conditional negative log-likelihood for the projected representation of an input.
ECDF normalization converts likelihood-derived scores into a scale-independent measure comparable across layers and classes.
The ECDF is estimated separately for each layer–class pair using training data, with N_c denoting the number of training samples in class c.
High τ_ℓ,c(x) values indicate that an input’s representation is typical for class c at layer ℓ, whereas low values indicate atypical or unlikely behavior.
Unlike clustering and feature-label association, the scores arise from probabilistic modeling and likelihood evaluation, giving them a direct probabilistic interpretation.

D. Typicality Maps and Class Prototypes

Typicality scores are assembled into maps that summarize class-specific evidence across network depth. Averaging correctly classified training maps produces class prototypes representing expected depth-wise patterns.

The typicality map T(x) collects all layer- and class-specific typicality scores for an input.
Each row shows typicality for one class across depth, while each column shows the class-wise typicality profile at one layer.
The map provides a depth-resolved, class-wise summary of the network’s internal decision process, analogous to MACS’s classification map.
A prototype map for class c is obtained by averaging the typicality maps of correctly classified training samples.
Class prototype maps represent the expected depth-wise typicality pattern for their respective classes.

E. Scalar Scores

The typicality map supports two scalar scores: PAS compares an input map with its predicted-class prototype, while MLSV measures consensus across layers without relying on stored prototypes.

The map structure yields two compact scalar scores for post-hoc analysis.
Prototype Alignment Score (PAS): PAS measures alignment between an input’s map and the prototype of its predicted class using a normalized Frobenius scalar product.
Prototype Alignment Score (PAS): High PAS values indicate strong agreement with the typical class-specific pattern, paralleling MACS’s cosine-similarity comparison.
Multi-Layer Soft Voting (MLSV): MLSV first applies a softmax across classes within each map column, then aggregates the normalized scores across layers and applies a second softmax.
Multi-Layer Soft Voting (MLSV): High MLSV values indicate cross-layer consensus toward one class, whereas low values indicate disagreement across layers.
The overall framework is organized as a five-stage pipeline with distinct offline and online phases.

III. EXPERIMENTAL SETUP

The experiments use a ViT-B/16 fine-tuned on CIFAR-100, with SVD-based projections at 24 intermediate hook points and a classification-head hook. Class-conditional GMMs use a fixed, empirically selected configuration throughout the evaluation.

The ViT-B/16 model is pretrained on ImageNet-1k, fine-tuned on CIFAR-100, and achieves 86.6% top-1 test accuracy.
The framework is applied to MLP sub-blocks at 24 intermediate hook points, formed from two locations in each of 12 ViT encoder layers, plus the classification-head input.
Activations are projected onto the top-k right singular vectors, using k = 200 for intermediate layers and k = 100 for the classification head.
Each class-conditional density uses a GMM with M = 4 diagonal-covariance components and covariance regularization ϵ = 10^-4.
The hyperparameters {k, M, ϵ} were selected empirically, then held fixed without task-specific tuning.

B. Out-of-Distribution Datasets

The evaluation uses SVHN, Places365, and DTD as semantically disjoint OOD datasets spanning different visual domains. Typicality maps show concentrated class evidence for correct in-distribution inputs but diffuse patterns for OOD inputs.

OOD datasets: Three semantically disjoint datasets—SVHN, Places365, and DTD—cover house-number, scene, and natural-texture image domains.They are evaluated as OOD shifts relative to CIFAR-100.
Typicality maps: Correctly classified inputs increasingly concentrate typicality on the ground-truth class as depth increases, whereas early layers spread evidence across classes.Class prototypes average correctly classified training maps into expected depth-wise profiles, while semantically related classes retain moderate typicality.
Typicality maps: Misclassified inputs retain competing or fluctuating class typicality even in late layers, unlike the concentration pattern of correctly classified inputs.The comparison uses correctly classified and misclassified lion examples alongside a class prototype.

B. Out-of-Distribution Detection

The study evaluates PAS and MLSV against established post-hoc OOD detectors on CIFAR-100 with SVHN, Places365, and DTD shifts. MLSV performs best among unsupervised methods overall, while SVHN remains an exception favoring several simpler scores.

Overall performance: MLSV achieves 97.0% mean AUROC and 11.9% mean FPR@95, outperforming the strongest baseline by 1.7 and 10.4 percentage points, respectively.Table I reports AUROC↑ and FPR@95↓ for the ViT-B/16 backbone with CIFAR-100 as the in-distribution dataset.
Dataset-specific performance: On Places365 and DTD, MLSV reaches 99.4% AUROC with FPR@95 below 2.5%, closely approaching the supervised DMD-A oracle.These are the shifts where the advantage over logit-based scores is most pronounced.
Dataset-specific performance: PAS attains 94.0% mean AUROC, comparable to MLS at 94.6% and k-NN at 94.1%, while exceeding MACS at 90.6% by 3.4 percentage points.The accompanying comparison reports a 17.7-point improvement over MACS in mean FPR@95.
Dataset-specific performance: SVHN is the exception: Energy and ReAct each reach 95.3% AUROC and k-NN reaches 95.9%, exceeding MLSV at 92.3% and PAS at 94.9%.The text attributes this to the large distributional gap between digit images and CIFAR-100 categories.
Score distributions: Normalized score distributions show moderate ID–OOD overlap for logit-based methods, while MLSV shifts OOD scores toward the low-score tail across all three benchmarks.DMD-B and k-NN improve separation on specific shifts but remain inconsistent across datasets.

V. CONCLUSIONS

The paper introduces SVD-based projections and class-conditional density modeling to produce depth-resolved typicality maps and post-hoc OOD scores. MLSV uses cross-layer consensus without stored prototypes and performs best among the evaluated unsupervised detectors.

Framework: SVD-based projections and class-conditional density modeling yield typicality maps that track how class-specific evidence consolidates across network depth.The maps also provide the basis for two post-hoc OOD detection scores.
Proposed scores: PAS measures alignment with stored class prototype maps, outperforms MACS, and requires significantly fewer parameters.Its formulation uses the corresponding class prototype as the reference pattern.
Proposed scores: MLSV discards stored prototypes and aggregates lightweight per-layer typicality votes to leverage cross-layer consensus.It achieves the best results among the considered unsupervised detectors across all three OOD benchmarks and approaches the supervised oracle on two.
Implication: Independent per-layer modeling suggests that representational consistency across depth is a reliability signal underused by single-layer and logit-based approaches.This conclusion connects the modeling choice to the interpretation of cross-layer evidence.