PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting prior-data fitted networks for tabular anomaly detection

Maximilian Bershtman, Niv Cohen

arXiv:2610.06693v1cs.LG

TL;DR

Tabular anomaly detection must work without labeled anomalies, and its reference set may itself be contaminated. The paper adapts TabPFN representations with frozen-feature scoring and fine-tuning; FOCUS achieves the highest mean AUROC in both evaluated regimes, and ZEN also transfers across other tabular foundation models.

  • Problem

    Adapting prior-data fitted networks for anomaly detection remains mostly unexplored, although the task has no labeled anomalies and its reference set may be contaminated.

    Source: §1 · pp. 1–2

  • Method

    The paper uses frozen TabPFN embeddings for ZEN’s trust-weighted nearest-neighbor scoring, then fine-tunes TabPFN in FOCUS to better separate normal samples from anomalies.

    Source: §1 · p. 2

  • Results

    FOCUS reaches the highest mean AUROC in both clean and contaminated regimes, while ZEN exceeds every baseline in the contaminated regime and transfers to four other tabular models.

    Source: §4 · p. 7, §7 · p. 9

  • Takeaways & Limitations

    Pretrained PFNs, once adapted to the task, make a strong anomaly detector in the evaluated settings.

    Source: §7 · p. 9

    The largest reference set evaluated contains 10,000 samples, so performance and cost with larger reference sets remain untested.

    Source: §6 · p. 9

Abstract

from arXiv · show

While deep features have transformed anomaly detection in images and video, their impact on tabular data has been less substantial, partly due to the limited availability of strong deep representations. Recently, prior-data fitted networks (PFNs) have emerged as a promising source of such representations for tabular data. In this work, we investigate how PFN representations can be adapted and leveraged for anomaly detection. The question is harder than it looks. No anomalies are available before deploy- ment, so model parameters cannot be tuned with supervision, and the reference set that defines normal behavior may itself contain the very anomalies it is supposed to reveal. We begin our study using frozen TabPFN features. Scoring each sam- ple by its distance to its nearest neighbors in feature space already gives strong results. We identify which layers to use and a feature-extraction procedure suited to the task. Next, to further improve performance, we use the reference set to fine- tune the model, so that the resulting features better separate normal samples from anomalies. On the ADBench benchmark, our fine-tuning free approach (ZEN) reaches a higher mean AUROC than every baseline, and our fine-tuned method (FOCUS) improves on it further. Our approach also generalizes across PFN models.

1 INTRODUCTION

Tabular anomaly detection must identify unknown anomalies without labels, even when its reference set may contain anomalies. This paper adapts TabPFN representations into ZEN and FOCUS, reporting strong results under contamination and transfer to other PFN models.

1 INTRODUCTION

  • The task is unsupervised: detectors must score new samples against a reference set that may itself contain anomalies.The anomaly structure is unknown in advance, and contaminated reference samples can make similar test anomalies look normal.Source: §1 · p. 1
  • PFN-based anomaly detection has used dedicated synthetic-task pretraining or frozen TabPFN scoring, while adapting a pretrained PFN to the task remains mostly unexplored.Adaptation is also of interest because a detector built by adapting TabPFN would inherit future PFN improvements.Source: §1 · p. 2
  • Under contamination, ZEN achieves higher mean AUROC than every baseline in the comparison, and FOCUS achieves the highest mean AUROC overall.FOCUS builds on ZEN by fine-tuning TabPFN so its features better separate normal samples from anomalies.Source: §1 · p. 2
  • ZEN scores samples by nearest-neighbor distances in frozen TabPFN embeddings and weights reference samples by trust to remain reliable under contamination.The paper also develops a feature-extraction procedure suited to this task.Source: §1 · p. 2
  • The approach is applied without changes to four other tabular foundation models, significantly improving each of them.The paper presents this as evidence that the approach is not tied to a single PFN model.Source: §1 · p. 2

2 METHOD

The methods use TabPFN representations for nearest-neighbor anomaly scoring, first with frozen weights in ZEN and then with compactness fine-tuning in FOCUS. ZEN handles contaminated reference sets through trust weighting, while FOCUS adapts embeddings using a fixed center and averaged epoch checkpoints.

2.1 PRELIMINARIES

  • Reference samples are embedded as leave-one-fold-out queries because TabPFN representations depend on the full reference context and differ between context and query roles.Source: §2.1 · p. 3

2.2 ZEN: ZERO-TRAINING EMBEDDING NEIGHBORS

  • ZEN scores test samples by mean distance to their k nearest reference samples in frozen TabPFN embeddings from the last four of its 24 blocks.Block scores are standardized before averaging; ZEN also averages scores across random feature subsets and the full feature set.Source: §2.2 · pp. 3–4
  • ZEN runs TabPFN on random half-feature subsets and the full feature set, adding uniform synthetic samples to each context to regulate feature scale.Source: §2.2 · p. 4
  • ZEN estimates reference-sample isolation across feature subsets and blocks, then inflates distances to isolated samples through trust weights rather than removing them.This reduces the chance that a test anomaly near an undetected reference anomaly receives a small distance and appears normal.Source: §2.2 · p. 4

2.3 FOCUS: FINE-TUNED ONE-CLASS UNSUPERVISED SCORING

  • FOCUS fine-tunes the last model blocks so reference embeddings contract toward a fixed center computed from trust-weighted frozen embeddings.It trains for two epochs and averages the saved weights; the adapted model then uses ZEN's feature subsets, contexts, and soft-cleaned readout.Source: §2.3 · pp. 4–5

3 RESULTS

Across 47 ADBench datasets, ZEN exceeds every baseline in the contaminated regime, and FOCUS improves further; in the clean regime, FOCUS and ZEN attain the highest mean AUROCs while raw-feature kNN retains the best baseline mean rank. The evaluation uses fixed configurations and five-seed means across clean and contaminated splits.

3.1 EXPERIMENTAL SETUP

  • The evaluation covers all 47 ADBench datasets, uses fixed method configurations, and averages AUROC over five seeds per dataset.The clean split uses 70% of normal samples for reference; the contaminated split uses a stratified 70/30 split preserving anomaly rates.Source: §3.1 · p. 5

3.2 MAIN RESULTS

  • 3.6 AUROC points above Isolation Forest (p = 0.01), FOCUS reaches the highest contaminated-regime mean AUROC, improving 0.3 points over ZEN and 6.1 points over raw-feature kNN.ZEN also exceeds every baseline, by 3.3 points over Isolation Forest (p = 0.02); FOCUS is significantly ahead of every ranked baseline in pairwise tests.Source: Figure 2 · p. 6, §3.2 · p. 6
  • 31 of 47 clean-regime datasets favor FOCUS over ZEN (p = 0.02), while raw-feature kNN retains the best baseline mean rank.FOCUS and ZEN have the two highest mean AUROCs, and FOCUS leads every deep and TabPFN-based detector by more than four AUROC points.Source: §3.2 · p. 6
  • When the reference-set regime is unknown, FOCUS matches the best classical detector if the set is clean and gains 3.6 AUROC points over the best baseline if it is contaminated.Source: §3.2 · p. 6

4 DISCUSSION

ZEN’s feature-extraction steps improve the frozen model’s anomaly-detection results, while FOCUS is less affected by reference-set contamination than raw-feature kNN. ZEN also transfers across tabular backbones, and the reported results are relatively insensitive to many configuration choices.

Ablations and analysis

  • 79.7 AUROC takes the frozen model from 74.1 to above every baseline after ZEN’s steps are added.The plain readout starts at 74.1; soft cleaning contributes the largest individual step, adding 2.8 points.Source: §4 · p. 6

Additional results

  • ZEN’s AUROC margin over each baseline is positive and significant in the contaminated regime across the 47-dataset comparison.The figure reports Wilcoxon p < 0.05 for every margin; uTabPFN covers 44 of 47 datasets.Source: Figure 3 · p. 7

Ablations and analysis

  • 5.9 AUROC points are lost by FOCUS between clean and contaminated regimes, versus 11 points for raw-feature kNN.The comparison is reported as average loss across datasets.Source: §4 · p. 7
  • FOCUS’s pooled isolation estimate assigns reference anomalies an average trust weight of 0.70 and normal samples 1.16.The isolation estimate reaches 76.7 AUROC as a reference-sample anomaly score; multi-subset averaging is described as essential to soft cleaning.Source: §4 · p. 7

Transfer across backbones

  • ZEN improves all four other tested tabular foundation models by roughly five to eight AUROC points, with every gain statistically significant.The method is applied with the same feature subsets, blocks, constants, and protocol; the improvements are shown under contamination.Source: §4 · p. 7

Sensitivity analysis

  • No tested constant has a sharp optimum, and the soft-cleaning gain rises from 0.1 AUROC points at 2% contamination to 2.0 at 20%.Reading the last one to four TabPFN blocks gives similar results under contamination, while reading more or earlier blocks is worse.Source: §4 · p. 7

5 RELATED WORK

Related work spans classical detectors, per-dataset deep methods, pretrained anomaly detectors, and transfer or adaptation approaches. This paper’s distinction within PFN-based work is using pretrained classification-model embeddings and adapting them for anomaly detection.

Classical anomaly detectors

  • Classical detectors include neighborhood and local-density methods such as kNN and LOF [1], partitioning and projection approaches such as Isolation Forest [2] and LODA [3], and distributional methods such as HBOS [5] and COPOD [6].ZEN’s feature subsets are closest to feature bagging [4], though each subset changes the in-context representation itself.Source: §5 · p. 8

Deep anomaly detectors

  • Deep tabular detectors use varied objectives, including transformations [1, 2, 3], diffusion or flow scoring [4, 5], masked modeling [6, 7], and noise evaluation [8].These methods incur per-dataset training; in this paper’s pipeline, training occurs only in FOCUS’s two-epoch fine-tune.Source: §5 · p. 8

Pretrained anomaly detectors

  • Pretrained anomaly detectors apply models across datasets, but synthetic-task methods and frozen TabPFN prediction scorers differ from ZEN’s embedding-based scoring and FOCUS’s adaptation.Examples include TACTIC [1], FoMo-0D [2], OutFormer [3], and OFA-TAD [4].Source: §5 · p. 8

Transfer and adaptation in anomaly detection

  • Prior work transfers or adapts pretrained features in image anomaly detection [1, 2, 3], while contamination-aware approaches refine training data or infer latent labels [4, 5, 6].FOCUS soft-cleans the reference set using the initial representation.Source: §5 · p. 9

6 LIMITATIONS

The evaluation and runtime impose scale boundaries: reference sets above 10,000 samples are untested, and scoring requires repeated TabPFN forward passes. Wide datasets are also constrained by a fixed feature subset.

Hardware and runtime

  • 3,500 samples per minute on one A100 is the reported throughput, and the method requires a GPU for its TabPFN forward passes.Most runtime comes from ten leave-one-fold-out passes per feature subset; fewer subsets trade marginal accuracy for significant speedup.Source: §6 · p. 9

Scale

  • 10,000 samples is the largest reference set tested because ADBench subsamples larger datasets; scoring cost grows with reference-set size, and larger sets remain untested.Three datasets exceed 200 features and use a fixed subset of 200; a more sophisticated treatment of wide tables may improve the method.Source: §6 · p. 9

7 CONCLUSION

The paper concludes that adapted pretrained PFNs are strong anomaly detectors, with FOCUS achieving the highest mean AUROC in both regimes. It also reports cross-model improvements, describes its reproducibility materials, and states its ethical scope and deployment caution.

7 CONCLUSION

  • FOCUS reaches the highest mean AUROC in both clean and contaminated regimes and is significantly ahead of every baseline in the contaminated regime.The paper’s training-free approach also exceeds every classical and deep baseline in mean AUROC under contamination.Source: §7 · p. 9
  • ZEN’s feature-extraction method significantly improves four other tabular foundation models without tuning or changes to the method.The authors state that none of the examined models was pretrained for anomaly detection, yet all showed anomaly-detection capabilities when adapted appropriately.Source: §7 · p. 9

AI USE STATEMENT

  • The authors report that AI tools assisted with implementation, result interpretation, translation, code, figures, text editing, literature review, and reference formatting.They state that authors directed and reviewed the work and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • Every reported number aggregates stored results from five fixed seeds, and the authors release code to reproduce the results.The paper points to configuration details and per-dataset tables in its appendices.Source: REPRODUCIBILITY STATEMENT · p. 10

ETHICS STATEMENT

  • The study uses public, anonymized benchmark data and no human subjects or personally identifiable information.The authors caution that deployment in fraud screening or medical triage should still audit error rates across affected populations.Source: ETHICS STATEMENT · p. 10

A EXPERIMENTAL PROTOCOL IN FULL

The protocol uses regime-specific splits, avoids duplication from ADBench’s official resampling on small datasets, and documents dataset- and detector-specific constraints.

A.1 SPLITS AND SUBSAMPLING

  • Clean-regime training uses 70% of normal samples, with the remainder of normals and all anomalies in testing; contaminated-regime splits are stratified 70/30.If too few normals are available, the training fraction steps down by 0.1; eight datasets with anomaly rates above 30% use 0.6.Source: §A.1 · p. 14

A.2 THE RESAMPLING LEAK IN THE OFFICIAL ADBENCH PIPELINE

  • ADBench’s official resampling duplicates samples on 12 small datasets; the study instead evaluates them at natural size, making its results non-comparable cell-for-cell with official-pipeline numbers.Under the official protocol, an average 88.5% of contaminated-regime test samples have a training twin, which can inflate distance- or density-based detector scores.Source: §A.2 · p. 14, Table 3 · p. 15

A.3 NOTES ON SPECIFIC DATASETS AND DETECTORS

  • On three datasets with more than 200 features, ZEN and FOCUS use a fixed 200-feature subset, while uTabPFN results cover only 44 datasets.uTabPFN takes about five to seven GPU-hours per seed on those datasets, so all five seeds were not run; four PyOD detectors also compute some statistics over the scored batch.Source: §A.3 · p. 15

B.1 DATASET STATISTICS

  • The benchmark comprises 47 datasets, and Table 3 reports each dataset’s natural sample size, dimensionality, and anomaly rate, with markers identifying protocol conditions for resampling, subsampling, and fixed-feature subsets.Markers flag sub-1,000-sample datasets affected by official resampling, datasets subsampled to 10,000 samples, and three datasets where ZEN and FOCUS use a fixed 200-feature subset.Source: §B.1 · p. 15, Table 3 · p. 15

B.2 PER-DATASET AUROC

Across ADBench’s 47 datasets, FOCUS leads the baselines under contamination, while clean-regime rankings favor raw-feature kNN; the section also reports AUPRC, seed variability, and performance gaps to per-dataset best methods.

B.2 PER-DATASET AUROC

  • Tables 4–7 report five-seed per-dataset AUROC for the compared methods across the 47 datasets in both regimes.Tables 4–5 cover contaminated data and Tables 6–7 clean data; each regime’s methods are split across two tables.Source: §B.2 · p. 16, Table 4 · p. 17, Table 5 · p. 18

B.3 AUPRC

  • FOCUS has the best mean AUPRC in both regimes and best mean rank under contamination, but its contaminated lead over Isolation Forest is not significant (p = 0.08).On clean data, raw-feature kNN retains the best AUPRC rank; FOCUS’s lead over it is not significant (p = 0.52).Source: §B.3 · p. 16, Table 8 · p. 21

B.4 SEED VARIABILITY

  • The 47-dataset mean AUROC varies by less than one point across seeds for 14 of 18 reported methods under contamination and 17 of 19 on clean data.Every method except DeepSVDD has seed-to-seed spread below 1.3 points in the 47-dataset mean.Source: §B.4 · p. 16, Table 9 · p. 21

C.1 CONSTRUCTION OF THE MAIN COMPARISON (FIGURE 2)

  • FOCUS exceeds all 16 full-coverage baselines significantly under contamination; it is 13.1 AUROC points ahead of uTabPFN on 44 datasets and beats Isolation Forest on 28 of 47.ZEN is not included as a baseline in this comparison; the comparison with FOCUS is reported separately.Source: §C.1 · p. 16

C.3 REGRET PROFILES AND THE REMAINING GAP

  • FOCUS and ZEN have the smallest average gaps to the per-dataset best: 4.3 and 4.6 AUROC points, respectively.Isolation Forest leads on four datasets with at most 21 features; on Wilt, the cleaning misfires and costs 6 points, while on Speech no compared method reaches 55 AUROC in either regime.Source: §C.3 · p. 22, Figure 7 · p. 23

C.4 ABLATIONS: EVERY STEP OF ZEN AND FOCUS

The section evaluates how each step in the ZEN readout contributes to isolation estimation and how much FOCUS fine-tuning adds as those steps are introduced.

C.4 ABLATIONS: EVERY STEP OF ZEN AND FOCUS

  • The ladder recomputes each readout step on saved embeddings using the official constants, with adjacent steps compared across 47 dataset means.The final step reproduces the main-run ZEN and FOCUS means to within 0.05 AUROC.Source: §C.4 · p. 23
  • Pooling all 24 representations yields a 76.7 AUROC isolation estimate, beating single representations on 43 of 47 datasets (p < 10^-10).Among the 5% of reference samples it distrusts most, 31% are anomalies, versus 12% in the reference set; labels are used only for this diagnostic.Source: §C.4 · p. 24, Figure 9 · p. 24
  • Under contamination, FOCUS’s gain over the frozen model falls from about 1.9 AUROC points on the plain readout to 0.3 after soft cleaning.The authors offer, but do not directly test, the possibility that fine-tuning and soft cleaning repair the same defect; on clean data, fine-tuning retains a significant 0.6–1.2-point gain at every step.Source: §C.4 · p. 24, Figure 8 · p. 24

C.5 ABLATION: THE READOUT WINDOW

The appendix tests readout choices, fine-tuning settings, context construction, controlled contamination, and transfer of ZEN to other PFN backbones.

C.5 ABLATION: THE READOUT WINDOW

  • Under contamination, the last one to four blocks perform equivalently within 0.3 AUROC, while reading all 24 costs ZEN 2.7 points and FOCUS 2.9; on clean data, FOCUS stays within 0.05 AUROC across one to twelve blocks.The paper’s last-four-block window is not a tuned optimum: it ties the best windows under contamination, and the clean-data best is 0.3 points away.Source: §C.5 · pp. 24–25, Figure 10 · p. 25, Table 11 · p. 26

C.6 SENSITIVITY OF THE READOUT

  • Under contamination, k = 50 is on a plateau, cleaning strengths λ = 0.25–2 improve on no cleaning, and the equal feature-mixing weight is best; on clean data, every λ > 0 costs 0.4–2.5 points.Three random subsets already place ZEN above every baseline, while adding subsets helps significantly through the fourth under contamination.Source: §C.6 · pp. 25–26, Table 12 · p. 26

C.7 ABLATIONS OF THE FINE-TUNE

  • Under contamination, fine-tuning 3 or 6 blocks is equivalent, while 12 and 24 blocks cost 2.6 and 3.2 AUROC points; on clean data, 12 blocks is best and 24 costs 3.0.A ten-times-larger learning rate destabilizes clean-regime fine-tuning (−19.9 points, p < 0.001); lower rates than the paper’s are safe in that setting.Source: §C.7 · pp. 26–27, Table 13 · p. 27

C.8 ABLATION: THE CONTEXT OF EACH RUN

  • Using two or five folds instead of ten changes contaminated-regime mean AUROC by +0.6 or +0.5, neither significantly, so two embedding passes have no measurable cost.Augmented context is a small, consistent component: up to 2,000 synthetic samples gain 0.8 points, not significantly.Source: §C.8 · p. 27, Table 14 · p. 27

C.9 INJECTED CONTAMINATION

  • With injected contamination rising from 2% to 20%, ZEN’s margin over Isolation Forest grows from 2.7 to 4.6 AUROC points, significant from 5% onward.Injected anomalies receive mean trust weights of 0.62–0.79 versus 1.12–1.20 for normal reference samples; Figure 11 reports the controlled-rate comparison and trust weights.Source: §C.9 · p. 28, Figure 11 · p. 28

C.10 ZEN ON OTHER BACKBONES

  • ZEN’s steps help every tested backbone under contamination; on clean data, the gain is significant for TabPFN-3, TabPFN v2, TabICL, and Mitra, but not TabDPT.TabPFN-3 significantly leads TabPFN v2 by 4.2 and 5.3 AUROC points in contamination and clean regimes, respectively; FOCUS on TabPFN v2 reaches 76.1 AUROC on 44 datasets.Source: §C.10 · pp. 28–29, Figure 12 · p. 29, Table 15 · p. 29

D.1 METHOD CONFIGURATION

The configuration section specifies fixed, non-dataset-specific settings for feature extraction and fine-tuning, then documents the baseline choices and runtime measurement protocol.

D.1 METHOD CONFIGURATION

  • Training constants are fixed once per regime, not per dataset; FOCUS trains for two epochs and averages the epoch-1 and epoch-2 parameters.FOCUS updates the last 12 blocks in the clean regime and the last 6 in the contaminated regime; ZEN and FOCUS use k=50.Source: §D.1 · pp. 29, 31
  • ZEN extracts embeddings with fixed arbitrary labels and leave-one-fold-out queries, then adds random feature subsets and synthetic context samples.The synthetic samples are not scored or used as neighbors; they only affect the model context.Source: §D.1 · p. 30
  • FOCUS uses frozen-embedding trust weights to define a fixed weighted center and minimize trust-weighted compactness over reference-sample batches.The weights and center are computed once and kept fixed during adaptation; FOCUS trains on the reference set without anomaly labels.Source: §D.1 · p. 31

D.2 BASELINE CONFIGURATIONS

  • Baselines use default or author-recommended settings without dataset-specific tuning, and the comparison omits pretrained anomaly detectors because methods start without anomaly-specific pretraining.Raw-feature kNN uses mean distance to its 20 nearest neighbors; PCA is capped at 95% cumulative explained variance to avoid infinite scores on singular covariance cases.Source: §D.2 · pp. 31–32

D.3 RUNTIME

  • Runtime measurements use one NVIDIA A100; baseline times are single seed-0 probes, while proposed-method stage times are medians across 235 runs.FOCUS adds a one-time fine-tuning cost before the same inference stages used by ZEN.Source: §D.3 · p. 32, Table 17 · p. 32

E NEGATIVE RESULTS

The tested alternatives generally did not improve on the paper’s plain kNN-based scoring and pooling choices; several performed worse.

E NEGATIVE RESULTS

  • LOF loses to plain kNN distance on raw features and embeddings, with exact duplicates disrupting reachability normalization.Removing duplicates did not close the gap.Source: E NEGATIVE RESULTS · p. 32
  • Hubness correction is worse than plain distance, and whitened or Mahalanobis-style kNN is not significantly better in either regime.Source: E NEGATIVE RESULTS · p. 32
  • Scoring by distance to the center is far worse than the kNN readout in both regimes, while reference-set bagging brings no gain.Source: E NEGATIVE RESULTS · p. 32
  • Agreement-weighted, pessimistic, calibrated, and two-neighborhood pooling do not improve on the plain mean of the isolation estimate.Source: E NEGATIVE RESULTS · p. 32
  • The classifier head is at chance as an anomaly score under arbitrary context labels, and mixing it into the embedding score hurts.Source: E NEGATIVE RESULTS · p. 33
  • Fine-tuning feature subsets separately, through augmented context, with pooled-isolation trust weights, or toward local centers brings no gain.Source: E NEGATIVE RESULTS · p. 33
Loading 2610.06693v1…