PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Novelty Detection via Data-enhanced Localization

Xiaoyang Wu, Jiajun Liu, Haojie Ren, Changliang Zou

arXiv:2610.34999v1stat.MEstat.ML

TL;DR

Conditional novelty detection asks whether relationships between variables change, rather than whether observations differ only in their marginal pattern. The paper develops DEL, using calibration and test data in localized conformal p-values and conditional calibration; it reports finite-sample FDR control and improved power in ablation comparisons.

  • Problem

    Conditional novelty detection concerns changes in the distribution of outcomes given covariates, such as counts that are unusual for a particular time period despite their overall pattern.

    Source: §1 · p. 2

  • Method

    DEL enhances a kernel conditional-distribution estimator with calibration and test data, adjusts for distribution shift, and recalibrates the resulting localized conformal p-values.

    Source: §1.1 · p. 3, §2.2 · p. 6

  • Results

    The final rejection set ensures FDR ≤ α under PX = QX and a permutation-invariant density-ratio estimator, while ablations report that DEL consistently has the highest power across score functions and sample sizes.

    Source: §3.1 · p. 8, §4.1.2 · p. 13

  • Takeaways & Limitations

    Conditional calibration integrates the proposed p-values into a novelty-detection procedure with finite-sample FDR control, and adaptive bandwidth selection is designed to enhance detection power.

    Source: §1.1 · p. 3, §3.1 · p. 8

    The adaptive-bandwidth procedure can be computationally burdensome when the candidate bandwidth set is large, and high-dimensional conditional distribution estimation remains a challenge.

    Source: §5 · p. 16

Abstract

from arXiv · show

Novelty detection is an important area of research in both statistics and machine learning. In this paper, we focus on the conditional novelty detection problem, where novelties arise from the relationship between different variables. We first adopt the conformal inference framework and propose a data-enhanced localized conformal p-value, constructed by recalibrating an enhanced conditional distribution estimator. This estimator efficiently captures conditional information by incorporating both calibration and test data into its kernel estimation. We show that our p-values are valid in finite samples and can improve detection efficiency. Based on this p-value, we then develop a novel conditional novelty detection algorithm, along with a data-driven bandwidth selection method that ensures finite-sample false discovery rate control while enhancing detection power. Both simulated and real data experiments demonstrate the advantages of our proposed approach.

1 Introduction

Novelty detection asks whether observations depart from a reference pattern, and conditional novelty detection focuses on changes in relationships between variables. The paper motivates and introduces DEL, which uses calibration and test data to construct localized conformal p-values and supports conditional novelty detection with finite-sample FDR control.

1 Introduction

  • Novelty detection seeks observations that deviate from an expected pattern, with applications including fraud, disease outbreaks, machine errors, and unexpected scientific patterns.The paper frames detection as testing whether a test sample comes from the reference distribution.Source: §1 · p. 2
  • Conditional novelty detection tests whether the conditional distribution of W given X differs between reference and test data, including cases where an observation is unusual for its user group or time period.This differs from testing only whether the overall marginal pattern changes.Source: §1 · p. 2

1.1 Related works and motivation

  • Existing model- and machine-learning-based approaches may rely on distributional assumptions or heuristic scores without theoretical guarantees, while conformal methods offer finite-sample guarantees for marginal novelty detection.Localized conformal p-values extend this framework to conditional distributions but use calibration data alone and can lose efficiency, especially with limited calibration data; their performance is also bandwidth-sensitive.Source: §1.1 · pp. 2–3
  • DEL constructs localized conformal p-values by enhancing a kernel conditional-distribution estimator with both calibration and test data, followed by recalibration.The paper presents the method as addressing efficiency and bandwidth-selection challenges in conditional novelty detection.Source: §1.1 · p. 3, §1.2 · p. 3

1.2 Our contributions

  • The proposed data-enhanced p-value is finite-sample valid and more efficient than LCP, and the method is described as applicable when calibration data are limited.The p-value is based on conditional distribution estimation using both calibration and test data, followed by recalibration.Source: §1.2 · p. 3
  • The resulting detection algorithm uses conditional calibration for finite-sample FDR control and offers data-dependent bandwidth selection that preserves this control while targeting greater power.The paper notes that data enhancement can help identify more novelties, especially in heteroscedastic scenarios.Source: §1.2 · p. 4

2 Data-enhanced localized conformal p-value

The section recaps conformal p-value novelty detection, then develops a localized conditional p-value that uses calibration and test data to improve efficiency while retaining finite-sample validity. Its enhanced estimator has a faster convergence rate under stated regularity assumptions.

2.1 Recap: novelty detection with conformal p-value

  • The recap compares each test point's score with calibration scores to form conformal p-values, then applies the Benjamini-Hochberg procedure to identify novelties.The p-values are super-uniform under the null when the calibration and test data are exchangeable; the shared-calibration p-values are also described as PRDS.Source: §2.1 · pp. 4–5

2.2 Data-enhanced localized conformal p-values

  • The earlier localized p-value captures nearby covariate information but is not finite-sample valid, and localization can reduce efficiency, particularly with small calibration sets.The proposed approach motivates recalibrating the localized conditional-CDF estimate.Source: §2.2 · p. 5
  • Recalibrating an enhanced conditional-CDF estimator that uses calibration and test data yields a finite-sample-valid localized conformal p-value.The estimator weights test data using a conditional density-ratio estimate and a parameter ω.Source: §2.2 · pp. 6–7
  • Under regularity assumptions, the nonparametric convergence rate uses n + ωm in place of n, with a separate term for density-ratio estimation accuracy.The accuracy term is controlled by ω.Source: §2.2 · p. 7

3 Conditional novelty detection algorithm

The procedure uses conditional calibration to control FDR with data-enhanced p-values, then selects bandwidths to improve detection power while preserving validity.

3 Conditional novelty detection algorithm

  • The section presents a novelty-detection algorithm using data-enhanced p-values and a separate data-driven bandwidth-selection method intended to boost power.Algorithm 1 uses conditional calibration for finite-sample FDR control; Algorithm 2 adds adaptive bandwidths.Source: §3 · p. 8, §3.1 · p. 8, §3.2 · p. 10

3.1 Novelty detection

  • Conditional calibration replaces direct BH application because data-enhanced p-values need not satisfy PRDS, and auxiliary p-values enable term-wise FDR analysis.The procedure decouples dependence between each p-value and the rejection number.Source: §3.1 · p. 8

3.2 Data-driven bandwidth selection

  • For each test sample, the bandwidth is chosen from candidates by maximizing the BH rejection count while keeping it fixed given the unordered calibration set.The modified selection preserves the exchangeability needed for p-value validity.Source: §3.2 · pp. 9–10
  • Adaptive bandwidth selection has complexity O(|Λ|·m^2), with reusable auxiliary p-values and kernel calculations limiting added computation.The authors report runtimes up to about 30 seconds on one core when m is in the hundreds.Source: §3.2 · p. 10
  • The data-driven bandwidth procedure retains finite-sample FDR control when the density-ratio estimator is permutation invariant and the relevant calibration and null test samples are exchangeable.The theorem states FDR ≤ α under these conditions.Source: §3.1 · p. 8, §3.2 · p. 11

4 Experiments and evaluation

Across simulations and real-data experiments, the authors evaluate FDR and power for DEL against benchmarks and ablations; DEL generally combines FDR control with higher detection power.

4.1 Simulated experiments

  • The simulations use a heterogeneous linear regression model, and experimental results are based on 200 replications.The simulation setting is adapted from a labeled scenario.Source: §4 · p. 11, §4.1 · p. 11

4.1.1 Comparison with baselines

  • All three methods control FDR below the nominal level, while DEL consistently has higher power than CP and LCP across the reported test and calibration sample sizes.The comparisons are shown for varying test sample sizes and calibration sample sizes.Source: §4.1.1 · p. 11, Figure 1 · p. 12, Figure 2 · p. 12

4.1.2 Ablation studies

  • Power increases stepwise from UEL to DEL-fix to DEL; DEL has the highest power across both score functions and all sample sizes, while all versions control FDR below nominal.The ablation compares unenhanced localization, fixed bandwidths, and adaptive bandwidths.Source: §4.1.2 · pp. 12–13, Figure 3 · p. 13
  • As conditional variance σW decreases or test sample size m increases, the selected optimal bandwidth becomes smaller.Conversely, larger σW or smaller m leads to a larger selected bandwidth.Source: §4.1.2 · p. 13, Figure 4 · p. 14

4.2.1 U.S. house price

  • On the House Prices dataset, all three methods control FDR below nominal, and DEL detects the largest number of novelties.Source: §4.2.1 · p. 14, Table 1 · p. 15

4.2.2 Bike sharing

  • On Bike Sharing, LCP has slightly inflated FDR under temporal dependency, whereas DEL stays below nominal FDR and achieves the highest power for both score functions.Source: Table 2 · p. 15, §4.2.2 · p. 15

5 Concluding remark

The concluding remarks identify kernel-estimation and computational limits, while the related-work discussion positions DEL among conditional novelty detection and conformal testing methods.

5 Concluding remark

  • Kernel conditional-distribution estimators have boundary effects and are inefficient in high-dimensional settings.The authors suggest modern machine-learning-based conditional-distribution estimation as a possible way to address these challenges.Source: §5 · pp. 15–16
  • Large candidate bandwidth sets can make the adaptive-bandwidth procedure computationally burdensome.The authors call for more efficient algorithms that avoid conditional calibration while retaining finite-sample FDR control.Source: §5 · p. 16

Appendix A Related works

  • Earlier conformal novelty-detection studies with finite-sample FDR control primarily address the marginal case (Bates et al. 2023; Marandon et al. 2024; Liang et al. 2024).Source: Appendix A · p. 16
  • Localized conformal inference by Guan (2023) and Hore and Barber (2024) targets prediction intervals valid conditional on covariates, whereas this work constructs p-values for testing.Guan uses a kernel-weighted calibration distribution and an adjusted quantile level; Hore and Barber add random sampling.Source: Appendix A · p. 16
  • The paper extends localized conformal p-values to conditional testing and selects bandwidth to maximize novelty-detection power, in a spirit similar to score selection by Bai and Jin (2024).Wu et al. (2025) introduced localized conformal p-values for conditional testing.Source: Appendix A · p. 16

Appendix B Experiment details

The appendix records experiment-specific score, kernel, bandwidth, and data-splitting choices, followed by proof details for exchangeability and conditional-distribution estimation.

Appendix B Experiment details

  • In the general experiment, localization uses the true conditional dimension d_con = 1, a Gaussian kernel, and five bandwidth candidates indexed by λ ∈ {−1, −0.5, 0, 0.5, 1}.The two score choices are residual and quantile-regression scores; the density ratio estimator is fitted with a GLM.Source: Appendix B · pp. 16–17
  • The house-price experiment removes identified outliers, splits remaining observations 20%/40%/40% into null-test, calibration, and training sets, then adds selected outliers to the test set.Neighborhood is encoded as a high- versus low-price indicator, and the DEL bandwidth candidates are specified in the passage.Source: Appendix B · p. 17

C.1 Proof of Theorem 1

  • The Theorem 1 proof uses permutation invariance of the density-ratio estimator and exchangeability of the calibration data to establish exchangeability of the transformed calibration values.Source: §C.1 · p. 17

C.2 Proof of Theorem 3

  • The Theorem 3 proof compares estimates involving calibration and test observations with the conditional CDF of the score given covariates.The passage defines that conditional CDF as F_{V|X=x}(v); the supplied proof excerpts do not state a full conclusion.Source: §C.2 · pp. 18–19

C.3 Proof of Theorem 4

The proof of Theorem 4 derives finite-sample FDR control by conditioning on a statistic that preserves null p-value validity and fixes the rejection set. Theorem 5 extends this argument to adaptive bandwidth selection.

C.3 Proof of Theorem 4

  • Under the null, p_DE,j remains super-uniform conditional on Φ_j because conditioning on the unordered calibration set preserves exchangeability.With the rejection set fixed given Φ_j, summing the conditional argument yields FDR control.Source: §C.3 · pp. 21–22

C.4 Proof of Theorem 5

  • The adaptive bandwidth h_j is measurable with respect to Φ_j, so its data-enhanced p-value remains conditionally super-uniform under exchangeability of the test data.The proof then reuses the conditional-calibration argument for Theorem 4.Source: §C.4 · p. 22

Appendix D Additional experiments

Additional experiments vary the density-ratio estimator and ω, examining their effects on FDR and power.

Appendix D Additional experiments

  • The appendix reports additional simulations using the main-text settings with varying parameters or algorithms, unless otherwise stated.The SVM comparison fixes the test sample size at m = 1000.Source: Appendix D · p. 22, §D.1 · p. 22

D.1 Estimation of br

  • DEL is more robust than DEL-fix across both score functions when the density-ratio estimator changes from GLM to SVM.With SVM, DEL-fix is severely affected for the residual score, while adaptive bandwidth mitigates this issue; DEL’s quantile-score power is similar across algorithms.Source: §D.1 · p. 22, Table D1 · p. 23

D.2 Paramter ω

  • Changing ω among 0.25, 0.5, and 0.75 only slightly affects performance; DEL has the highest power in most cases.The exception is ω = 0.75 with the quantile score, where DEL and DEL-fix have quite close power.Source: Table D2 · p. 23, §D.2 · p. 23

D.3 Relationship between br1 and w

The comparison of estimated and oracle density ratios empirically supports choosing a larger bandwidth parameter w when the density-ratio estimator is more accurate.

D.3 Relationship between br1 and w

  • The experiment compares power curves across w for DEL using estimated r and an oracle approximation trained on a leave-out dataset ten times larger.The setting matches Section 4.1.2 with m = 500.Source: §D.3 · p. 23
  • DEL with oracle r achieves relatively higher power, with an optimal w around 0.6 versus 0.1 for the estimated-r version.This pattern supports the proposed rule for selecting w.Source: Figure D1 · p. 24, §D.3 · p. 24

D.4 Novelty ratio

Across the appendix comparisons, DEL has higher power than AdaDetect, while increasing the novelty ratio raises power across methods and preserves the relative pattern seen in the main simulations.

D.4 Novelty ratio

  • Across novelty ratios 0.1, 0.15, and 0.2 at m = 1000, power improves for all methods as the ratio increases, while relative comparisons match the main simulations.Tables D3-D4 report the novelty-ratio comparisons for DEL, LCP, CP, DEL-fix, and UEL.Source: §D.4 · pp. 24–25, Table D4 · p. 25

D.5 Comparison with AdaDetect

  • Both DEL and AdaDetect control FDR at the target level, while DEL has substantially higher power because AdaDetect omits conditional distributional information.The power comparison covers different test sample sizes.Source: Figure D2 · p. 25, §D.5 · p. 25

D.6 Training sample size |Dt|

DEL maintains the highest power as training sample size increases. Under tested covariate shifts, its FDR remains controlled within a range, though reduced variance tends to inflate it.

D.6 Training sample size |Dt|

  • DEL consistently achieves the highest power across training sample sizes, and power rises for all methods as training data increase.The comparison fixes m = 1000 and reports that all methods control FDR at the target level.Source: §D.6 · pp. 25–26, Figure D3 · p. 26

D.7 Influence of covariate shift

  • Under mean shifts and enlarged variance, DEL's FDR does not increase; reduced variance tends to inflate FDR, but it remains around nominal even at half variance.The clean labeled data retain Xd ∼ N(0, 1), while test covariates vary from that distribution.Source: Figure D4 · p. 26, §D.7 · p. 26, Figure D5 · p. 27

D.8 Categorical responses

With categorical responses, all methods have reduced power, which the paper attributes to the limited information in a conditional distribution with only two point masses.

D.8 Categorical responses

  • The example uses a binary response with opposite logistic conditional probabilities for inliers and outliers, creating conditional distribution drift.X is uniform on [0, 1]; the experiment varies m with |T| = |C| = 500 and uses random-forest probability scores.Source: §D.8 · p. 27
  • All methods show reduced power in the categorical-response setting, with power below 0.5 despite substantial inlier-outlier differences.The paper attributes the degradation primarily to W being discrete, leaving only two point masses in its conditional distribution given X.Source: §D.8 · pp. 27–28
  • The paper notes that a well-calibrated predictive model with accurate probability estimates can be particularly beneficial in this setting.Source: §D.8 · p. 28

Related papers

Loading 2610.34999v1…