Source-linked AI summary
What Changed? Drift Detection with Real, Virtual, and Incomparable Diagnosis
Kentaro Oda
TL;DR
Drift alarms do not by themselves identify whether the input–output mechanism changed or only the input distribution moved. The paper estimates covariate and functional change together, and reports that typed monitoring is Pareto-optimal across real- and virtual-drift regimes while allocating labels differently by diagnosis.
Problem
Standard error-stream detectors watch mistakes that can rise under real drift or covariate-induced extrapolation, while input-space detectors observe P(X) only, leaving drift type unresolved by construction.
Source: §1 · p. 1Method
Results
Typed monitoring is Pareto-optimal across real- and virtual-drift regimes, gains up to 25.8 accuracy points over never-retraining, and spends zero labels on virtual drift where always-retrain pays 2000 labels for a +0.002 change.
Source: §5 · p. 6Takeaways & Limitations
The diagnosis directs label spending: typed monitoring matches always-retrain on real drift while avoiding its label cost on virtual drift.
Source: §5 · p. 6Window-based estimation limits detection resolution to about one window at the stated settings, and the functional axis requires labels to arrive within the window.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Sharing a deep encoder does not, by itself, fix the central confound of task-comparison scores. We show that cross-evaluated heads on a frozen shared representation inherit the extrapolation confound of shallow exchange scores: pure input rotations with fixed labels inflate a deep exchange score from about 0 to 0.80, while representation-novelty scores are blind in the complementary direction (flat under label permutations that change the task completely). Transplanting a conditional two-discriminator discrepancy into the embedding space resolves both blind spots: the functional axis stays within +-0.001 under rotations and tracks label-permutation drift mass monotonically. Built into a mixture-of-heads lifecycle, the two-axis gate attains better decision quality with fewer heads than exchange or novelty triggers at a matched training budget. On generalized category discovery, the same chunk-level functional axis separates semantic novelty from photometric shift with AUROC 0.98-0.99 where per-input OOD scores (MSP, Energy, Mahalanobis, KNN) sit near chance for that distinction. All findings replicate across frozen ImageNet-21k ViT-B/16 and self-supervised DINOv2 backbones on CIFAR-100, and extend to residual adapter pools with recurrence, where a null-calibrated novelty trigger never fires on mechanism changes while the two-axis gate handles them with full recurrence reuse. We state explicitly the common-factoring condition under which embedding-space conclusions transfer to the original mechanism.
1 Introduction
Drift alarms alone do not identify the response: real mechanism change calls for retraining, while covariate shift calls for other actions. The paper frames monitoring as estimating covariate and functional change separately.
1 Introduction
- Error-stream detectors respond to mistakes that can rise under either real drift or covariate-induced extrapolation, while input-space detectors observe P(X) only.The type question is therefore not answerable by construction for either standard detector family.Source: §1 · p. 1
- Real drift calls for retraining, whereas covariate-only movement calls for reweighting, recalibration, or targeted data collection rather than retraining.An alarm without a diagnosis does not tell the system what action to take.Source: §1 · p. 1
- DDM alarms on 80% of stationary runs at default settings and labels every covariate stream “real”; a D3-style classifier detects covariate shift in 1000 samples but misses every SEA and STAGGER concept change.Baseline detectors do not claim a type explicitly; the study scores their implied action, and a covariate shift may leave error-stream detectors silent.Source: §1 · p. 2
- The monitor decomposes I(Z; X, Y) into covariate Ix = I(Z; X) and functional DCJS = I(Z; Y | X), which is zero when P(Y | X) is unchanged, regardless of how far P(X) moves.Both axes use one pooled discriminator pair and provide block-level estimates with per-point confidence intervals.Source: §1 · p. 2
2 Method
The method estimates covariate and functional discrepancy between a reference window and a sliding recent window, then assigns one of three responses. It treats severe loss of overlap as incomparable rather than evidence that the mechanism is intact.
2 Method
- At each block boundary, the monitor compares reference window R with recent window St using a two-discriminator cross-entropy difference and can report real, virtual, or incomparable.Real alarms require the standardized functional statistic and DCJS to cross their gates; virtual alarms require the covariate statistic to cross its threshold without exceeding the comparability ceiling.Source: §2 · p. 2
- When covariate discrepancy exceeds the comparability ceiling, the functional axis is vacuous, so low DCJS cannot establish mechanism equality and the monitor reports incomparable.The ceiling is a design threshold; the recommended response is to collect data or reset the reference.Source: §2 · p. 2
- The ceiling choices 0.8, 0.9, and 0.95 ln 2 leave benchmark decisions unchanged, with zero false alarms in 40 runs and zero misses in 30.The benchmark shifts remain below the near-disjoint regime where the ceiling binds.Source: §2 · p. 2
- For continuous monitoring, two betting e-processes use per-point loss-difference increments from discriminators frozen before the block is used for training.The level is split across axes; bounded-memory recency uses the restarted e-detector construction with level spent across restart instances.Source: §2 · p. 2
- Time-monotonic index features must be excluded from the covariate discriminator, while the functional discriminator inherits the underlying discrepancy’s robustness and can be downward-biased under weak discriminators.The method avoids model exchange and its extrapolation false signal.Source: §2 · p. 2
3 Benchmark
Across synthetic streams, the two-axis monitor achieves exact detection and typing without per-stream tuning, at a delay cost. On INSECTS it decomposes real drifts and recovers recurrence, while sequentially valid monitoring avoids confident near-random decisions on borderline cases.
3 Benchmark
- The benchmark covers SEA-style and STAGGER-style real changes, pure covariate shift, and stationary null streams, with 10 seeds and a change at 10k in streams of length 20k.Detectors are assessed at defaults and at matched false-alarm operating points calibrated on null streams with disjoint evaluation seeds.Source: §3 · pp. 2–3
- The two-axis monitor needs no per-stream tuning and trades about 1–3 window-halves of delay for exactness in all three benchmark columns.At matched zero-observed-false-alarm calibration, DDM and EDDM have no viable setting, Page–Hinkley misses all SEA drifts at its calibrated threshold, ADWIN retains a 0.6 SEA miss rate, and calibrated error-stream detectors still imply “retrain” on pure covariate shift.Source: §3 · p. 3
- On INSECTS-abrupt, every function-based measure detects the five documented change points, while the two-axis monitor reports mixed covariate and functional drift with Ix ∈[0.26, 0.31] and DCJS ∈[0.13, 0.24].Recurring segment pairs (0, 2), (0, 5), and (2, 5) have DCJS ≤0.03, recovering recurrence in the controlled temperature profile without segment supervision.Source: §3 · p. 3
- Under continuous monitoring, repeated testing at every block is invalid; on boundary cases, it converts uncertainty into confident, near-random alarms (Fig. 4).Source: §3 · p. 3
- The unwindowed betting e-process is anytime-valid and maintains a 0.00 false-alarm rate with a 20% delay premium.The restarted e-detector supplies post-drift recency with the same guarantee; validity is for the frozen-discriminator surrogate and applies per monitoring segment, with the reference and process reset after an alarm.Source: §3 · p. 4
4 Related work
The paper distinguishes its model-free, two-axis monitor from prior systems that combine separately designed monitors, and compares it directly with WATCH under the paper’s protocol. The comparison highlights a speed–false-alarm and coverage trade-off.
4 Related work
- WATCH monitors deployed-model nonconformity scores with weighted conformal martingales, whereas this monitor compares data distributions and derives both axes from one pooled discriminator pair.The paper describes its functional statistic as having an exact covariate-null and producing a per-drift two-axis anatomy.Source: §4 · p. 4
- The INSECTS figure depicts covariate and functional components for documented real drifts and recurring segments clustered with nulls, exposing the temperature cycle.The figure presents this as a per-drift decomposition and recurrence pattern.Source: Figure 3 · p. 5
- In the re-implementation, WATCH is much faster on strong shifts, with STAGGER delays of 34–51 samples and covariate-shift typing within 55–103 samples, while the two-axis monitor trades 500–1650 samples for zero observed false alarms, zero misses, and exact typing on the same runs.These measurements are for the paper’s re-implementation and protocol, not the authors’ pipeline; WATCH also targets adaptation under mild shift.Source: §4 · p. 5
- Conditional two-sample tests provide p-values for P(Y | X) equality but not a bounded, decomposable statistic, streaming validity, or a covariate companion from the same learned object.The monitor derives both axes from one pooled discriminator pair with a shared statistical treatment.Source: §4 · p. 5
- For borderline DCJS ≈ τ cases, repeated confidence-interval testing commits in 63% of runs, split nearly evenly between the two actions, whereas valid monitoring defers and incurs a 1.8–2.6× delay ratio on clear cases.The comparison illustrates the cost of deferral against confident but near-random decisions.Source: Figure 4 · p. 6
5 Does the diagnosis matter? An action-value experiment
Typed monitoring is Pareto-optimal across the evaluated regimes and ties drift diagnoses to intervention costs, matching retraining on real drift while avoiding label expenditure on virtual drift.
5 Does the diagnosis matter? An action-value experiment
- The intervention policy retrains on the recent window after real drift, keeps the deployed model after virtual drift, and resets the reference and collects data for incomparable drift.Retraining on real drift costs 2000 labels.Source: §5 · p. 6
- Typed monitoring is Pareto-optimal in every regime and the only non-oracle policy Pareto-optimal across both real- and virtual-drift regimes.It gains up to 25.8 accuracy points over never-retraining.Source: §5 · p. 6
- On real drift, typed monitoring matches always-retrain, within 0.6 points of the known-change-point baseline on SEA and above that baseline on STAGGER.On STAGGER, re-alarms trigger the follow-up retrains required by the rule change.Source: §5 · p. 6
- On virtual drift, typed monitoring spends zero labels, while always-retrain spends 2000 labels for a +0.002 change.The deployed model remains pointwise correct under virtual drift; reweighting matters for continued training, not the predictor.Source: §5 · p. 6
6 Limitations
Detection resolution is bounded by the monitoring window, and the functional axis depends on timely labels; delayed labels shift real-drift detection but leave virtual-drift detection and typing unaffected.
6 Limitations
- Window-based estimation bounds detection resolution to approximately one window of delay at the reported settings.The window therefore constrains how quickly the monitor can resolve a drift.Source: §6 · p. 6
- Labels must arrive within the window for the functional axis; delays of 1000 or 2000 samples shift real-drift detection by the same delay.Virtual-drift detection and typing remain unaffected because the covariate axis does not need labels.Source: §6 · p. 6
- Under simultaneous strong covariate and mechanism change, the monitor reports real drift when the functional statistic clears its gate.This is the monitor’s stated behavior by design.Source: §6 · p. 6