PaperlayerView exact revision on arXiv

Source-linked AI summary

When Identical Rows Disagree: From Benchmark Identifiability to Replication-Robust Anomaly Detection

Jie Deng

arXiv:2609.29580v1cs.LGstat.ML

TL;DR

Repeated rows make it unclear whether multiplicity reflects meaningful frequency or an artifact, while identical features can also limit what evaluation labels are distinguishable. The paper uses exact-row equivalence classes to analyze those limits and develops SCOUT to separate support from exposure-aware count evidence. Across four detectors, row-to-support weighting changes AUROC by at least 0.05 on 50–61 datasets.

  • Problem

    When identical released features carry different labels, deterministic feature-based detectors must tie them, and the table may not reveal whether multiplicity represents business frequency or replication.

    Source: §1 · p. 1

  • Method

    SCOUT uses one representative per normal equivalence class for structural modeling and a separate exposure-normalized conditional count model, with factorwise conformal calibration and support-only scoring as the safe default.

    Source: §1 · p. 2

  • Results

    Changing evaluation from rows to observed support changes AUROC by at least 0.05 on 50–61 datasets per detector across Isolation Forest, ECOD, HBOS, and COPOD.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The same equivalence classes expose feature-identifiability limits, metric weighting, size-biased training, and the nuisance-invariant structural sample, linking measurement analysis to detector design.

    Source: §8 · p. 8

    Exact equality misses near-duplicates and repeated entities with changing fields, while conformal validity requires exchangeable normal support classes and may fail under temporal drift, entity dependence, or contaminated calibration.

    Source: §8 · p. 8

Abstract

from arXiv · show

A released table is often treated as an i.i.d. sample, although its repeated rows may encode business frequency, repeated entities, joins, resampling, or extraction errors. We show that this ambiguity creates a hidden measurement layer with three consequences: feature-identical rows impose an attained evaluation ceiling, row-weighted AUROC is sensitive to replication, and row-trained detectors learn a multiplicity-size-biased law. An exact-row audit of all 690 OddBench datasets finds train-test overlap in 355, feature-identical label conflict in 147, and a test anomaly identical to a training normal in 137. Switching from row to support weighting changes AUROC by at least 0.05 on 50-61 datasets across four classical detector geometries. We introduce SCOUT (Support-Count Orthogonalized Unsupervised Testing), a factorized anomaly detector that separates replication-invariant support evidence from exposure-aware count evidence. Factorwise split-conformal calibration yields marginal false-positive-rate control, while the support channel is exactly invariant to arbitrary positive row replication. On 686 OddBench datasets and five seeds, support-only SCOUT is non-inferior to row-wise Isolation Forest in raw AUROC and improves replication-invariant AUROC. External normal-support evaluations track nominal false-positive levels, and four backbones remain exactly unchanged under controlled replication. Semi-synthetic interventions show that conditional count modeling helps materially only under strong rate heterogeneity. These results specify when multiplicity should be treated as signal, nuisance, or uninterpretable without additional information.

1 Introduction

Repeated rows in a released table may represent different underlying processes, but the table alone exposes only their features and multiplicities. The paper uses exact feature-equivalence classes to characterize evaluation limits and replication effects, then builds SCOUT to separate support evidence from exposure-aware count evidence.

1 Introduction

  • Identical feature rows can encode independent transactions, replicated records, or observations with missing entity and time identifiers, so their multiplicity has no unambiguous meaning in the released table.Ordinary geometry-based detectors let multiplicity reshape feature geometry, while global deduplication discards frequency evidence.Source: §1 · p. 1
  • The observational quotient partitions rows into support patterns and multiplicities, yielding an exact AUROC decomposition, an attained feature-identifiability ceiling, and a replication-invariant sensitivity metric.The framework also shows that row sampling is multiplicity-size-biased and business versus artifact frequency is not identifiable without auxiliary information.Source: §1 · pp. 1–2
  • Across the four detectors, changing evaluation from rows to observed support changes AUROC by at least 0.05 on 50–61 datasets.The audit also finds exact train–test overlap in 355 of 690 datasets, conflicting labels on identical features in 147, and a test anomaly identical to a training normal in 137.Source: §1 · p. 1
  • SCOUT models one representative per normal equivalence class structurally and separately models exposure-normalized log count conditional on that representative.Support-only scoring is the safe default; the frequency channel is used when exposure is comparable, and the observed business and artifact intensities cannot be separated from their product alone.Source: §1 · p. 2

2 Related Work and Novelty Boundary

Prior work advances tabular representations, data-quality checks, conformal detection, event-frequency analysis, and pretrained or graph-based features. This paper distinguishes its contribution as making repeated-row semantics and replication orthogonality explicit for static released tables.

Tabular anomaly detection and benchmarks

  • Tabular anomaly-detection benchmarks and methods improve evaluation breadth, representations, or scoring geometries, but do not specify whether repeated rows are independent units or expansion artifacts.The cited benchmarks include ADBench [1], an independent JMLR study, and MacrOData [3].Source: §2 · p. 2

Data quality and evaluation

  • Data-quality work ranks duplicates and label errors, whereas this paper derives what identical released features make unidentifiable without assuming duplicates are errors or automatically deleting them.Its AUROC transformation changes weights within each label through equivalence-class replication while keeping feature support fixed [11].Source: §2 · p. 2

Robustness and conformal novelty detection

  • Unlike environment-based invariance or conformal procedures for novelty and contamination, SCOUT's stated contribution is the support–count sampling unit and replication orthogonality.The paper describes the exact-feature-class nuisance as algebraic rather than distributional and does not claim conformal ranks, Bonferroni correction, or contamination robustness as new [12] [14] [15].Source: §2 · p. 2

Semantic and frequency anomalies

  • Semantic and frequency anomaly methods use event time or system structure, while the paper studies static released tables where that context may be absent.The quotient supplies an impossibility boundary and a support-only fallback rather than inventing event semantics [16, 17].Source: §2 · p. 2

Foundation and graph features

  • Pretrained and graph-derived features can add information or serve as SCOUT backbones, but deterministic encoders of the same released row cannot identify why it was replicated.The paper therefore did not retain a pretrained-feature-plus-XGBoost pipeline as its central contribution [19] [20] [21].Source: §2 · p. 2

3 The Observational-Equivalence Certificate

The observational-equivalence certificate separates feature-identifiability limits from multiplicity-dependent AUROC weighting. It also defines a support-weighted sensitivity metric that is invariant to replication.

3.1 Exact decomposition and an attained ceiling

  • The certificate decomposes AUROC over feature-equivalence classes, separating unavoidable within-class ties from multiplicity-dependent cross-class weights.Replicating observed row–label pairs can change AUROC without changing scores or feature support.Source: §3.1 · p. 3
  • Ranking equivalence classes by their anomaly fractions attains the sharp finite-sample AUROC ceiling for deterministic scores measurable from released features.The ceiling is a post-hoc certificate based on test labels, not an unsupervised detector or a model-selection quantity.Source: §3.1 · p. 3

3.2 A replication-invariant sensitivity estimand

  • RI-AUC evaluates each observed class–label pair once, remains unchanged under arbitrary positive replication, and is canonical among separable pair weights.Ordinary AUROC remains appropriate when rows are the intended unit; the signed RI-AUC difference measures multiplicity leverage, not automatically bias.Source: §3.2 · pp. 3–4

4 Multiplicity as a Hidden Measurement Layer

Expanded-row sampling overweights support patterns with higher expected multiplicity, so row-trained density models reflect both support prevalence and conditional counts. Business intensity and artifact replication are not separately identifiable from observed counts and exposure alone.

4 Multiplicity as a Hidden Measurement Layer

  • Uniformly selecting an expanded row yields a multiplicity-size-biased version of the support law, rather than the support law alone.This follows when expected multiplicity is positive and finite.Source: §4 · p. 4
  • Ordinary row-density models therefore learn a mixture of support prevalence and conditional count, with multiplicity affecting both evaluation and model fitting.The same multiplicity that changes cross-class AUROC weights also weights representation learning and model fitting.Source: §4 · p. 4
  • Business intensity and artifact replication are not separately identifiable from observations of support, multiplicity, and exposure without restrictions or auxiliary metadata.Repeated windows, provenance, entity identifiers, or interventions are needed to assign cause.Source: §4 · p. 4

5 SCOUT: From Certificate to Detector

SCOUT uses support classes to separate replication-invariant geometry from exposure-aware count evidence. Its support scores are exactly invariant to replication, while conformal guarantees depend on exchangeability and provide marginal class-level control.

5.1 Orthogonal support and count channels

  • SCOUT's support channel uses one representative per normal equivalence class, while a separate model estimates exposure-normalized log count conditional on that representative.The count channel is used when exposure is comparable; the safe default is support-only scoring.Source: §5 · p. 4, §5.1 · p. 4
  • Replicating any training or scoring row a positive integer number of times leaves support nonconformities, p-values, and anomaly scores exactly unchanged.The proper-training and calibration split is performed after quotienting, so replication cannot alter the partition.Source: §5.1 · p. 4, §5 · p. 4
  • Under exchangeability of null test and support calibration classes, the support p-value has marginal false-positive control; comparable exchangeable count residuals give the same guarantee for the count channel and their combination.The combined guarantee does not assume channel independence.Source: §5.1 · p. 4
  • The conformal guarantee is marginal class-level FPR control, not row-wise batch FDR or protection under temporal drift.It depends on the sampling unit and exchangeability.Source: §5.1 · p. 4
  • The count regressor is a flexible working model rather than a claimed count likelihood, and can be replaced without changing SCOUT's factorization.The API returns support-only output unless frequency is explicitly enabled with exposure.Source: §5.1 · p. 5

6 Study Design

The study tests five questions about benchmark multiplicity, channel mechanisms, calibration, utility, and replication invariance. Its design combines a full benchmark audit, controlled mechanism tests, calibration evaluations, and replication interventions.

6 Study Design

  • Five linked questions address whether the hidden measurement layer matters in a public benchmark, whether the two channels recover their mechanisms, and how calibration, utility, and replication invariance perform.The questions are tested across benchmark audit, mechanism, calibration, utility, and intervention studies.Source: §6 · p. 5

6.1 Complete audit and evaluation sensitivity

  • The audit uses bitwise exact numeric rows after canonicalizing signed zero, while four detectors are compared using raw AUROC and RI-AUC with bootstrap intervals and Holm-adjusted paired tests.The audit covers released arrays; detector rankings are also compared.Source: §6.1 · p. 5

6.2 Mechanism identification and frequency boundary

  • Controlled Gaussian supports test geometry shifts and abnormal conditional counts across 15 dimension-and-seed settings, with row-wise, deduplicated, count-augmented, and unconditional-frequency baselines.A separately reported stress test raises dimension to 50 and 100 and scales the log-rate signal by 1.5.Source: §6.2 · p. 5
  • A semi-synthetic count test uses normal support from 100 duplicate-rate-stratified OddBench datasets, with weak or strong covariate rate heterogeneity and paired comparisons against unconditional frequency.Counts and burst labels are semi-synthetic because OddBench has no window exposure; test support receives rate boosts of 1.5, 2, or 4.Source: §6.2 · p. 5

6.3 Calibration, utility, and intervention

  • Calibration is evaluated on 72,000 synthetic null tests per channel and 500 real dataset–seed runs using held-out normal support at each alpha.The external evaluation includes 691,385 support evaluations per alpha, with run-level rates also bootstrapped.Source: §6.3 · p. 5
  • Utility evaluation covers 686 datasets with five matched seeds and a prespecified −0.01 raw-AUROC non-inferiority margin; separate replication interventions refit four detectors after adding replicas to selected support classes.The intervention records score changes, rank correlation, and AUROC changes across 100 datasets stratified by training duplicate rate.Source: §6.3 · p. 5

7 Results

Across OddBench, exact-row disagreement and replication-sensitive evaluation are common. SCOUT preserves full-benchmark utility, while the value of count modeling depends on rate heterogeneity and its support channel remains unchanged under replication.

7.1 RQ1: disagreement is common and changes conclusions

  • 51.4% of datasets had exact train–test overlap and 21.3% had feature-identical label conflicts; 17 had ceilings below .95 and seven below .90.These counts do not label duplicates as errors; source semantics determine the appropriate observational unit.Source: §7.1 · p. 6
  • Across detectors, 50–61 datasets had |∆rep| ≥0.05, and 30 changed their top-ranked detector under row versus support evaluation.Raw and replication-invariant rankings had Kendall τ = 0.929 across 2,760 detector–dataset cells.Source: §7.1 · p. 6

7.2 RQ2: factorization recovers mechanisms, not every advantage

  • On mixed anomalies, combined SCOUT reached .9847 versus .9133 for count-augmented Isolation Forest, but was .0146 below support-only on pure support anomalies.The mixed-anomaly paired difference was .0714 with interval [.0599, .0828]; the support-anomaly gap argues against always combining channels.Source: §7.2 · p. 6
  • High-dimensional regimes won all ten mixed/frequency settings, while conditional-count gains on real geometry were .0031–.0061 under weak versus .0567–.0831 under strong heterogeneity.The semi-synthetic gains were larger under strong heterogeneity, with 92–99 dataset wins out of 100.Source: §7.2 · p. 6

7.3 RQ3: support calibration transfers to real geometry

  • Across 691,385 external tests per alpha, micro FPR was .00958, .04936, and .09949 at nominal .01, .05, and .10, respectively.This experiment validates the support channel; OddBench has no exposure data with which to validate real count calibration.Source: §7.3 · p. 6, Figure 2 · p. 7

7.4 RQ4: robustness preserves full-benchmark utility

  • On 686 datasets, mean raw AUROC fell by .00258 yet cleared the −.01 non-inferiority margin; RI-AUC improved by .00487.The comparison isolates changing the observational unit while holding the backbone family fixed; it is not a state-of-the-art claim.Source: §7.4 · p. 6, Table 6 · p. 8
  • In the highest duplication quartile, raw AUROC was .0104 lower while RI-AUC was .0198 higher; median fit time was .387 seconds versus .333 seconds.The timing comparison is descriptive, not a hardware-independent complexity claim.Source: §7.4 · pp. 7–8

7.5 RQ5: intervention verifies a backbone-independent guarantee

  • Across Isolation Forest, ECOD, HBOS, and COPOD, support-score change was exactly zero and minimum rank correlation was 1.0 after replication.The reported mean raw/RI-AUC pairs vary by backbone; exact stability follows from quotient construction, not equal detector strength.Source: §7.5 · p. 8

8 Discussion

The discussion frames SCOUT as a joint response to ambiguity in repeated-row meaning, separating stable support evidence from exposure-aware frequency evidence. It recommends enabling frequency only when exposure semantics make counts meaningful and identifies limits on the evidence and scope.

What the fusion changes

  • The framework links identical-row equivalence classes to feature identifiability, evaluation leverage, size-biased training, and a nuisance-invariant structural sample.Its contribution is the closed measurement-to-method loop, rather than any single off-the-shelf component.Source: §8 · p. 8

When to enable frequency

  • Use frequency evidence when observation windows are comparable, exposure is measured, and abnormal rates are operationally meaningful; disable it when counts may reflect undocumented data-generation artifacts.When normal rate barely depends on covariates, conditional count modeling adds little and an unconditional count test may be preferable.Source: §8 · p. 8

Why neither global deduplication nor richer features resolves the problem

  • Global deduplication discards legitimate rate evidence, while appending raw count can distort detector geometry and underperform the factorized channel in mechanism experiments.Richer backbones cannot infer missing factorization from the same feature vector alone.Source: §8 · p. 8

Limitations and external validity

  • Exact equality misses near-duplicates and repeated entities with changing fields, and approximate equivalence lacks the guarantees established for exact matches.Conformal validity requires exchangeable normal support classes; the count channel cannot identify the cause of an unexpected rate, and its power evidence is synthetic or semi-synthetic because OddBench lacks exposure metadata.Source: §8 · p. 8

Negative directions

  • Mean raw AUROC for support-wise versus row-wise Deep Isolation Forest changed by −.0036 with interval [−.0166, .0059], failing the −.01 non-inferiority criterion; RI-AUC changed by −.0017 with interval [−.0145, .0074].A random-projection hypersphere detector scored .679 on a 20-dataset real pilot, below quotient Isolation Forest at .705.Source: §8 · pp. 8–9

9 Conclusion

The conclusion presents the observational quotient as a way to make repeated-row measurement explicit, connecting evaluation limits and size bias to a detector that separates support from count evidence. It recommends declaring the observational unit before optimizing an anomaly detector.

9 Conclusion

  • The observational quotient makes the hidden measurement layer explicit, yielding an evaluation ceiling and replication-invariant sensitivity estimand while separating stable support evidence from exposure-aware count evidence.Audits, matched fits, external calibration, mechanism tests, and replication interventions support the framework while identifying boundaries on its use.Source: §9 · p. 9

Reproducibility

  • The repository provides source code, fixed seeds, successful and failed experiment rows, unit tests, audit tables, bootstrap outputs, and figure builders.Source: §9 · p. 9

Data scope

  • Four utility datasets were excluded because each had fewer than 40 distinct normal support points.The public raw data are downloaded from official repositories and are not redistributed.Source: §9 · p. 9

A Proofs

The proofs partition AUROC comparisons by feature-equivalence classes and establish an attained score ordering for the ceiling. Within-class comparisons contribute a fixed tie term because identical features force identical scores.

A.1 Equivalence-class AUROC

  • The AUROC pair sum can be partitioned by the equivalence classes of its positive and negative members, with each class pair contributing its multiplicity times the score-comparison value.Source: §A.1 · p. 9
  • Pairs drawn from the same feature-equivalence class contribute a fixed 1/2 comparison value because identical features force identical deterministic scores.Source: §A.1 · p. 9

A.2 Attained ceiling

  • Ordering equivalence classes by nonincreasing positive-to-negative count ratio qc attains the global AUROC ceiling.Adjacent swaps improve or preserve the objective, including zero-count boundaries.Source: §A.2 · p. 9

A.3 Canonical replication-invariant weights

The proofs characterize replication-invariant support weighting, show that expanded rows follow a multiplicity-size-biased law, and establish limits on attributing count rates to causes. They also prove exact support-channel invariance under positive replication and factorwise conformal validity under stated conditions.

A.3 Canonical replication-invariant weights

  • Replication invariance forces the positive- and negative-class weights to be constant across positive counts, giving the support-indicator weighting in Eq. (3).Source: §A.3 · pp. 9–10

A.4 Row-law size bias

  • The expanded-row law is the support law weighted by multiplicity and normalized by expected multiplicity.Source: §A.4 · p. 10

A.5 Frequency-cause non-identifiability

  • Different business and artifact intensity functions can yield the same observable multiplicity mean, so the causal decomposition is not identifiable from the observed count law alone.Source: §A.5 · p. 10

A.6 Exact replication orthogonality

  • Positive replication leaves the representative set, deterministic order, class split, support scaler, backbone fit, calibration scores, and support-channel scores unchanged; only the explicit count path can change.Source: §A.6 · p. 10

A.7 Factorwise and combined validity

  • Under proper training and exchangeability, each valid factor has P(p_j ≤ t) ≤ t, and Bonferroni combination controls the false-positive probability at α without channel independence.Without count exchangeability or comparable exposure, the validity argument applies only to p_S.Source: §A.7 · p. 10

Related papers

Loading 2609.29580v1…