PaperlayerView exact revision on arXiv

Source-linked AI summary

The Selection Rule Decides the Winner: A Pre-Registered Audit of Open-Set Graph Anomaly Detection

Farhan Shahriyar Hossain, Taufikur Rahman Fuad, Md Abrar Jahin, Md Rizwan Parvez

arXiv:2609.33370v1cs.LG

TL;DR

Open-set graph anomaly detection must find anomaly classes absent from labeled training data, while common evaluation conventions complicate comparisons. The authors rerun three detectors under shared splits and seeds, scoring each with oracle and deployable validation rules. The rule changes which method leads, and the oracle bonus varies with benchmark provenance.

  • Problem

    Open-set GAD must detect anomaly classes absent from labeled training data, but reported results often use best-test-epoch scores, copied baseline numbers, and relabeled minority classes.

    Source: §1 · p. 1

  • Method

    The study reruns DEMO and NSReg with OUTPOST on shared splits and seeds, scoring each run under oracle and validation checkpoint-selection rules.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    Under validation, NSReg leads five graphs versus three each for OUTPOST and NSReg under the oracle rule; OUTPOST’s oracle bonus is 0.045–0.080 AUC-ROC on semi-synthetic graphs and 0.002–0.014 on observed-fraud graphs.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The authors conclude that rankings depend on checkpoint selection and component contributions depend on whether anomalies are observed or constructed.

    Source: §6 · p. 9

    The provenance comparison rests on only three binary observed-fraud graphs, too few to separate provenance from their other shared properties.

    Source: §6 · p. 9

Abstract

from arXiv · show

Open-set graph anomaly detection trains on a few labeled anomalies from one class and must also find anomaly classes that were never labeled. Published results share three conventions: the test score is read at the best epoch on the test set, baseline numbers are copied from earlier papers, and most anomalies are minority classes relabeled as anomalous. We ask how much of the reported ranking these conventions decide. We re-run two recent methods, DEMO and NSReg, together with OUTPOST, a small first-order detector built for this study. All three use one protocol with identical seeds and splits on eight graphs (seven for the baselines, which cannot run on ogbn-mag), ten seeds each, and every run is scored under both the best-epoch rule and a deployable validation rule. Before the runs that test them, we registered 40 predictions. Three findings hold. First, the rule changes the leader: under the best-epoch rule, OUTPOST and NSReg each lead three of seven graphs, while under the validation rule, NSReg leads five. Second, the best-epoch bonus depends on how the benchmark was built: 0.045--0.080 AUC-ROC on the three small relabeled-class graphs and 0.002--0.014 on the three real fraud graphs. Third, pseudo-labeling in OUTPOST is worth 0.038--0.065 AUC-ROC on the same three graphs but gives no benefit on any real fraud graph. We also show that a 0.002 tie band for hyperparameter selection lies below the paired standard error on all six graphs tested, even at ten seeds. Twelve of our 40 predictions were falsified, and we report them. We close with a short reporting checklist.

1 INTRODUCTION

Open-set graph anomaly detection must identify anomalies from classes absent from labeled training data, but common evaluation conventions leave the reported rankings uncertain. The audit compares methods under a shared protocol and finds that checkpoint selection and anomaly provenance shape the results.

1 INTRODUCTION

  • Published open-set GAD evaluations commonly select the best test epoch, copy baseline numbers from incompatible runs, and relabel minority classes as anomalies.The paper asks how much these conventions determine reported rankings and which conclusions survive deployable evaluation.Source: §1 · p. 1
  • The audit reruns DEMO and NSReg alongside OUTPOST on eight graphs with identical splits and seeds, scoring every run under oracle and validation rules.The validation rule selects the epoch with highest validation AUC-ROC; three graphs have observed-fraud labels.Source: §1 · pp. 1–2
  • OUTPOST’s oracle bonus declines across semi-synthetic, OGB, and real graphs, while the real–OGB mean gap is 0.003 and lies within seed spread.Figure 1 shows the per-graph bonuses with standard-deviation bars over ten seeds.Source: Figure 1 · p. 2
  • The selection rule changes the leader on two graphs: OUTPOST and NSReg each lead three graphs under the oracle rule, while NSReg leads five under validation.The oracle bonus is 0.045–0.080 AUC-ROC on semi-synthetic graphs and 0.002–0.014 on graphs with observed fraud.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses graph anomaly detection, limited supervision, unseen anomaly types, and benchmark comparison through distinct methods and evaluation settings. This paper situates open-set GAD at their intersection.

Graph anomaly detection

  • Graph anomaly detectors use reconstruction, contrastive learning, or one-class modeling, while limited-supervision methods address label scarcity but were not designed to detect unseen anomaly classes.Examples include ConsisGAD, GGAD, and SpaceGNN.Source: §2 · pp. 2–3

Open-set GAD and detector design

  • Open-set methods extend detection to unseen anomaly types: NSReg uses normal-node-oriented relations, DEMO combines mixup and adaptive pseudo-labeling, and OUTPOST uses conformal calibration and similarity-based neighbor sampling.The cited foundations include open-set recognition, GraphSAGE, conformal calibration, and consistency-based pseudo-labeling.Source: §2 · p. 3

GAD benchmarks and anomaly provenance

  • BOND and GADBench compare detectors on synthetic or organic anomalies, but neither evaluates open-set GAD; this study contrasts relabeled minority classes with observed fraud.GADBench uses standardized tuning and validation AUC-PR for model selection.Source: §2 · p. 3

Model selection and reproducible comparison

  • Prior re-evaluations show that graph-model conclusions can depend on splits, training, and benchmarks, but open-set GAD had not received a comparable re-evaluation.The paper measures checkpoint-selection effects within the same runs using both oracle and validation rules.Source: §2 · p. 3

3 PROBLEM SETTING AND EVALUATION PROTOCOL

The protocol defines open-set graph anomaly detection as ranking both seen- and unseen-class anomalies above normal nodes, then compares methods across shared splits, seeds, and selection rules. It distinguishes relabeled-class benchmarks from observed-fraud graphs.

Task

  • A detector must rank anomalies from both the seen training class and unseen classes above normal nodes, in a transductive setting where the whole graph is visible during training.The unlabeled self-training pool is the test node set, whose labels methods do not read.Source: §3 · p. 3

Graphs

  • The eight graphs comprise five multi-class graphs with relabeled anomalies and three binary graphs with observed-fraud labels.The paper grounds open-set claims in the five multi-class graphs; the binary fraud graphs test the few-label regime with real anomalies.Source: §3 · p. 3

Splits and rotations

  • Each rotation uses 50 seen-class anomalies and 5% of normal nodes for training, plus 30 anomalies and 1% of normal nodes for validation; remaining nodes are test data.Each anomaly class takes a turn as the seen class, and graph scores average over rotations.Source: §3 · p. 3

Two selection rules

  • The oracle rule reports the best test metric across epochs, while the deployable validation rule selects the epoch with highest validation AUC-ROC and reads no test labels.The oracle bonus is their score difference; the study reports AUC-ROC and AUC-PR.Source: §3 · p. 4

Baselines under the same protocol

  • DEMO and NSReg receive the same graph tensors, splits, ten seeds, metrics, training budget, and selection rules as OUTPOST; earlier-method numbers remain separate because their runs are not paired.DEMO’s second-order energy term is omitted; NSReg runs from released code.Source: §3 · p. 4

Statistics

  • Main comparisons pair methods by seed and use two-sided Wilcoxon tests; ten seeds lower the smallest attainable p-value to 0.002.The study applies Holm correction within ablation families and treats comparisons with transcribed values as weak evidence.Source: §3 · p. 4

Pre-registration

  • The authors preregistered 40 predictions with falsifying outcomes; the ledger records 17 confirmed, 12 falsified, 3 split, and 1 null.The remaining seven predictions were superseded, corrected, or partial.Source: §3 · p. 4

4 OUTPOST: A FIRST-ORDER OPEN-SET DETECTOR

OUTPOST is a first-order open-set detector built to model normality, handle unseen anomaly classes, and allow component-level evaluation. Its test ranking uses a GraphSAGE-based score, while training adds an atlas, conformal pseudo-labeling, and graph-specific modules.

Encoder and score

  • OUTPOST scores nodes with a two-layer GraphSAGE encoder, projection, and scoring head; the sigmoid score alone ranks test nodes, including unseen anomaly classes.The atlas shapes training but does not score test nodes.Source: §4 · p. 4, Figure 2 · p. 5

Atlas of normality

  • The normality atlas represents labeled normals as a union of spherical caps and measures each node’s distance from that atlas.Spherical k-means initializes prototypes from normalized labeled-normal embeddings; prototype radii use a geodesic-distance quantile.Source: §4 · pp. 4–5

Conformal pseudo-labeling with an atlas gate

  • Each epoch, split conformal calibration sets the positive pseudo-label threshold from held-out normal scores, while an atlas gate prevents pseudo-labeling nodes outside the atlas as normal.The gate is intended to guard against anomalies pulled toward normal neighbors by score propagation.Source: §4 · p. 5

Synthetic anomalies

  • On relabeled-class graphs, OUTPOST adds synthetic positives through similarity-weighted anomaly mixup and halo extrapolation away from the mean normal embedding.The halo multiplier is sampled from U[1.2, 2.0]; synthesis is off on observed-fraud graphs.Source: §4 · pp. 5–6, Figure 2 · p. 5

SimSample

  • SimSample orders neighbors by raw-feature cosine similarity, then mixes the most similar neighbors with uniform draws according to a graph-specific setting.It has no parameters or labels and incurs no sampling-time cost after the ordering is built.Source: §4 · p. 6

Full objective

  • OUTPOST uses first-order components and graph-kind-specific configurations; on real-fraud graphs, synthesis and the atlas loss are off, leaving only the atlas gate.Its first-order objective lets it run on ogbn-mag, where neither baseline can run.Source: Figure 2 · p. 5, §4 · p. 6

5 RESULTS

The results show that selection rules and benchmark provenance shape reported rankings and component gains. OUTPOST’s pseudo-labeling gains are concentrated on relabeled-class graphs, while SimSample’s effect varies across graphs.

5.1 THE SELECTION RULE CHANGES THE LEADER

  • Five of seven comparable graphs are led by NSReg under validation, versus three each for OUTPOST and NSReg under the oracle rule.The rule changes the leader on two graphs; on Computers, OUTPOST’s margin over NSReg reverses from +0.0140 to −0.0140.Source: §5.1 · p. 6
  • The three-seed paired standard errors exceed the 0.002 tie band on every graph, and ten seeds still leave standard errors at 0.0021–0.0045.Reaching the band would take 11–51 seeds; under a one-standard-error requirement, every selection would have kept its default.Source: §5.1 · pp. 7–8, Figure 3 · p. 7

5.2 BENCHMARK PROVENANCE SETS THE SIZE OF THE ORACLE BONUS

  • 0.045–0.080 AUC-ROC is the oracle bonus on semi-synthetic graphs, versus 0.002–0.014 on graphs with observed fraud.Across the eight graph means, bonuses are ordered by benchmark provenance with no overlap; graph size does not explain the ordering.Source: §5.2 · p. 8
  • The oracle bonus can exceed method differences: 13 of 14 paired oracle-rule differences are smaller than the smallest semi-synthetic bonus, 0.045.On Photo, restricting scores to unseen classes widens the bonus to 0.178.Source: §5.2 · p. 8
  • On Photo, removing pseudo-labeling and the atlas gate costs 0.038 under the oracle rule but only 0.015 under validation, not significantly.The reported oracle-rule benefit includes variance collected by the best-epoch selection.Source: §5.2 · p. 8

5.3 COMPONENT CONTRIBUTIONS DEPEND ON ANOMALY PROVENANCE

  • 0.038–0.065 AUC-ROC is lost when pseudo-labeling is removed on Photo, Computers, and CS, while removal is neutral or improves scores on observed-fraud graphs.On Amazon removal is neutral; it improves Yelp and T-Finance. Conformal calibration matters on Computers, while the atlas gate moves AUC-ROC by at most 0.0016 across seven graphs.Source: §5.3 · p. 9
  • On Yelp, disabling SimSample costs 0.0504 AUC-PR and 0.0263 AUC-ROC; a shuffled-order placebo falls below the disabled arm.Both metrics rise with the similarity fraction, and the placebo indicates the gain comes from which neighbors are chosen rather than reduced sampling randomness.Source: Figure 3 · p. 7, §5.3 · p. 9
  • SimSample helps Yelp and T-Finance, hurts Amazon and Photo, and is neutral on CS and ogbn-arxiv; the registered account does not explain this pattern.On T-Finance, validation selection chose the weaker arm, understating OUTPOST by 0.0128 AUC-ROC.Source: §5.3 · p. 9

6 CONCLUSION

The audit finds that open-set GAD rankings depend on the epoch-selection rule and anomaly provenance, so reported gains should identify both. Its conclusions are bounded by the evaluated encoders, graph labels, methods, and tuning settings.

6 CONCLUSION

  • The selection rule changes which detector leads, while the effect of components depends on whether anomalies were observed or constructed.The conclusion frames these as measurement findings: 12 of 14 oracle-rule differences between OUTPOST and the baselines are at or below 0.020 AUC-ROC.Source: §6 · p. 9
  • The provenance comparison uses only three graphs with observed labels, all binary, so it does not test real anomalies with unseen classes.The conclusions are also conditioned on GraphSAGE-type low-pass encoders, one split generator, one label budget, and three evaluated methods.Source: §6 · p. 9

OVERVIEW OF THE APPENDIX

  • The appendix provides supporting materials on graphs, settings, baselines, rankings, oracle bonuses, tuning, ablations, model size, and the pre-registration ledger.It also includes a reporting checklist and analyses of sampling, propagation, and signal-to-noise.Source: OVERVIEW OF THE APPENDIX · p. 14

A GRAPHS

The graph set combines relabeled anomaly classes with observed fraud, and its structural patterns do not support the prediction that real fraud is always more scattered. The appendix also reports implementation settings, reproducibility checks, and sensitivity tests.

A GRAPHS

  • Real fraud is not uniformly more scattered than relabeled anomalies: T-Finance’s same-class fraction is 0.63, while Yelp’s is 0.16 and Amazon’s is 0.08.Together with ogbn-mag classes lacking same-class neighbors, these patterns contradict prediction P4.Source: A GRAPHS · p. 15

B SETTINGS, HARDWARE, AND SENSITIVITY

  • OUTPOST uses two GraphSAGE layers and graph-specific prototype and augmentation settings, with ρ selected on validation for Amazon and T-Finance.The complete settings distinguish semi-synthetic/OGB graphs from real graphs; Amazon and T-Finance use the Yelp block except for input width.Source: B SETTINGS, HARDWARE, AND SENSITIVITY · p. 15
  • Deterministic reruns matched metrics to six decimals, but shared-machine timing varied enough that the authors make no relative-speed claim.The same Photo pair produced time ratios of 0.75 and 1.40 on separate occasions.Source: B SETTINGS, HARDWARE, AND SENSITIVITY · p. 15
  • Five of six hyperparameter sweeps fall well inside default-seed noise.On Yelp, hidden width 16/32/64 spans 0.0078 AUC-ROC (0.5× seed sd) and 0.0350 AUC-PR (1.03×); on Photo, the λun sweep spans 0.0116 and 0.0107 AUC metrics (0.39× and 0.48×).Source: B SETTINGS, HARDWARE, AND SENSITIVITY · p. 15

C BASELINE CONFIGURATIONS

The baseline and published-field comparisons show why consistent reruns matter, while paired rankings show that selection rules can change method order. Scope differs across methods: neither baseline runs on ogbn-mag, and transcribed values are not directly comparable with reruns.

C BASELINE CONFIGURATIONS

  • Neither DEMO nor NSReg runs on ogbn-mag: DEMO requires a dense 2.17 TB PPR matrix, while NSReg’s all-pairs regularizer entails 1.34 billion pairs.Subsampling the pairs would no longer be NSReg.Source: C BASELINE CONFIGURATIONS · p. 16
  • At 400 epochs, five of seven baseline reruns exceed published AUC-ROC values: NSReg on all four available graphs and DEMO on CS.The table compares published scores with reruns at each method’s released budget and at the study’s 400-epoch budget.Source: Table 5 · p. 16

D PUBLISHED FIELD AND AUC-PR

  • On Yelp, OUTPOST reaches 0.3824 AUC-PR, ahead of re-run NSReg at 0.3610 and DEMO at 0.3504.Figure 4 also notes that most of OUTPOST’s margin over the best published value, 0.3029, comes from the protocol change, which lifts rerun baselines too.Source: Figure 4 · p. 17
  • The published-field tables separate transcribed results from study reruns because their splits, budgets, and seed counts differ.Tables 6 and 7 mark best and runner-up separately within each block; Table 8 gives AUC-PR under the study protocol over ten seeds.Source: Table 6 · p. 17, Table 7 · p. 18, Table 8 · p. 18

E PAIRED TESTS, UNSEEN CLASSES, AND RANKINGS

  • The paired comparisons report OUTPOST-versus-baseline results across shared seeds, including win counts and a separate unseen-class analysis.The unseen-class table omits the three binary real graphs, which have no unseen class; wins are out of ten seeds.Source: Table 9 · p. 19, Table 10 · p. 19, Table 11 · p. 19
  • The selection rule changes the method order on Photo and Computers, while OUTPOST and NSReg each lead on only CS and Yelp under both rules.For unseen classes, the rule changes the order on two of four graphs, the same two as for all anomalies.Source: Table 10 · p. 19, Table 12 · p. 20, Table 13 · p. 20

F ORACLE BONUS, BUDGET, AND TUNING

The oracle bonus and training budget can change apparent gains and rankings, while validation-based tuning is tested against seed-level noise. The reported tuning outcomes include reversals and a case where moving OUTPOST to the selected Amazon arm does not change the leading method.

F ORACLE BONUS, BUDGET, AND TUNING

  • The oracle bonus is 3 to 35 times larger on semi-synthetic than real graphs, and in AUC-ROC it is larger still on unseen classes.The bonus compares test performance at the best test epoch with performance at the best validation epoch, averaged over ten seeds.Source: Table 14 · p. 20
  • On semi-synthetic graphs, 43 to 62 percent of full OUTPOST’s apparent benefit over the variant without pseudo-labeling and the gate comes only from the oracle rule.Runs use each graph’s development budget: 200 epochs on Photo, Computers, and CS; 400 elsewhere.Source: Table 15 · p. 21
  • On Photo, DEMO ties OUTPOST at 200 epochs but leads at 400, and 77% of DEMO’s gap to its published Photo value closes with budget alone.The study uses 400 epochs throughout because the baseline is properly trained there, although that costs OUTPOST the Photo tie.Source: Table 16 · p. 21, F ORACLE BONUS, BUDGET, AND TUNING · p. 21
  • The tuning appendix reports validation-only selections for OUTPOST and NSReg and measures noise in the statistic used by the rule.The tuning tables use three seeds for selection; the noise table reports paired validation AUC-ROC spreads and the seed count needed to fall within the 0.002 tie band, with T-Finance only an order-of-magnitude estimate.Source: F ORACLE BONUS, BUDGET, AND TUNING · p. 21, Table 17 · p. 21, Table 18 · p. 22
  • Two of three tuning choices that departed from defaults reverse or vanish when retested at ten seeds.Table 19 compares selected and default arms using validation AUC-ROC margins, paired test differences, and wins out of ten seeds.Source: Table 19 · p. 22
  • Moving OUTPOST to the validation-selected Amazon arm yields 0.9494 ± 0.0076 oracle AUC-ROC instead of 0.9534, but does not change which method leads.Its deficit to DEMO grows from −0.0021 to −0.0061 (p = 0.037); the selected-arm preference does not persist at ten seeds.Source: F ORACLE BONUS, BUDGET, AND TUNING · p. 22

G ABLATIONS

The ablations show that component effects vary by graph, and some registered expectations about interactions and synthesis were falsified.

G ABLATIONS

  • Table 21 reports paired, variant-minus-full ablations at each graph’s development budget; five paired seeds give a minimum attainable p-value of 0.062.HopMix and the spectral gate were tested and rejected; the table records wins by seed, with two ties in Yelp conformal cells.Source: Table 21 · p. 23
  • Under the validation rule, removing the atlas gate costs 0.0014 on CS (p = 0.037), while no other cell is significant.At the best test epoch, no gate-removal cell approaches significance and the largest movement is 0.0016.Source: Table 22 · p. 23
  • Removing pseudo-labeling and removing the gate plus pseudo-labeling produce equal results to four decimals on Computers and CS, falsifying the predicted interaction.Removing synthesis raises Photo AUC-ROC by 0.018 on five of five seeds, while adding it lowers Yelp AUC-PR by 0.032.Source: G ABLATIONS · p. 24

H SIMSAMPLE

The registered degree-budget account only partly matches SimSample’s measured effects, and the Yelp intervention shows its gain persists above the median degree.

H SIMSAMPLE

  • The registered degree-budget account fits Photo, Yelp, and T-Finance, but not CS, ogbn-arxiv, or Amazon.Figure 5 shows that two of three graphs above the budget improve while Amazon does not; among four below it, only Photo is clearly harmed.Source: Table 23 · p. 24, Figure 5 · p. 25
  • +0.0217 AUC-PR remains for SimSample on Yelp at budget 200, despite the prediction that its gain would vanish above the median degree.At budget 200, 37% of Yelp nodes still exceed the budget; the two-term model predicted −0.083 on ogbn-arxiv, but the measured value was +0.0008 and the model was discarded.Source: H SIMSAMPLE · p. 24
  • On Yelp, the “on” arm falls by 0.020 across budgets 25 to 200 while the “off” arm rises by 0.010 from 25 to 50 and is then flat.The sweep therefore mainly measures neighbor selection rather than receptive-field size.Source: H SIMSAMPLE · p. 24

I TRAINING-FREE DIAGNOSTIC

The training-free diagnostic relates class clustering to propagation gains in the observed graphs, but its relation does not predict new classes; a signal-to-noise account also fails as a closed-form rule.

I TRAINING-FREE DIAGNOSTIC

  • Across 37 classes in eight graphs, best prototype AUC correlates with same-class neighbor fraction (Spearman ρ = 0.778); among 21 classes helped by propagation, the correlation is 0.896.Clustered classes gain up to 0.48 AUC from propagation, while scattered classes lose; the diagnostic uses no trained model and computes class-level AUC-ROC over propagation depths zero to three.Source: I TRAINING-FREE DIAGNOSTIC · pp. 25–26, Table 24 · p. 27
  • The fitted clustering relation predicts unseen classes worse than a constant (mean absolute error 0.239 vs. 0.166), and the diagnostic has no rank correlation with trained-model AUC on ogbn-mag (ρ = −0.02, p = 0.94).The out-of-sample comparison holds out one graph at a time; Figure 6 also shows that the class rated easiest by the diagnostic ranks sixth of fifteen after training.Source: Figure 7 · p. 26, Figure 6 · p. 26, I TRAINING-FREE DIAGNOSTIC · p. 27

J A SIGNAL-TO-NOISE ACCOUNT OF PROPAGATION

  • The signal-to-noise rule agrees with measured propagation-gain signs on only 8 of 16 classes, and the paper makes no theoretical claim from the closed form.The authors identify correlated neighbor noise and the AUC ceiling as failed assumptions; the continuous margin still ranks with gain (Spearman 0.756).Source: J A SIGNAL-TO-NOISE ACCOUNT OF… · pp. 27–28

K MODEL SIZE AND MEMORY

  • OUTPOST is the smallest model on every graph; its parameter margin over DEMO is large throughout, while its margin over NSReg is large only on the three observed-fraud graphs.The Yelp plot compares AUC-PR with trainable parameter count; every OUTPOST arm clears the rerun DEMO and best published result with at most about one fifth of DEMO’s parameters.Source: Table 26 · p. 28, Figure 8 · p. 28, K MODEL SIZE AND MEMORY · p. 28

L REPORTING CHECKLIST

  • The reporting checklist calls for paired reruns, both validation- and oracle-rule scores, tie bands calibrated to seed noise, and preregistered predictions with falsifying outcomes.It also recommends testing component benefits on at least one benchmark with observed anomalies and distinguishing infeasible baselines from merely expensive ones.Source: L REPORTING CHECKLIST · p. 29

M PRE-REGISTRATION LEDGER

The ledger records 40 preregistered predictions and their outcomes, including falsifications about the paper’s own diagnostic and explanations.

M PRE-REGISTRATION LEDGER

  • The ledger distinguishes blind predictions from those registered before the relevant seeds were read, and marks the first diagnostic predictions as superseded or corrected.P1–P3 were superseded by the eight-graph analysis, and P4 was corrected.Source: M PRE-REGISTRATION LEDGER · p. 29
  • On ogbn-mag, OUTPOST stayed below 0.60 AUC-ROC yet exceeded the best published value by 0.096 on ten of ten seeds, making its registered prediction split.On ogbn-arxiv, the result fell inside the published band as registered.Source: M PRE-REGISTRATION LEDGER · p. 29
  • Seventeen of 40 predictions were confirmed, twelve falsified, three split, one null, and seven superseded, corrected, or only partly held.Several falsified predictions concerned the authors’ own method or explanations and changed the paper’s text.Source: Table 27 · p. 29
Loading 2609.33370v1…