PaperlayerView exact revision on arXiv

Source-linked AI summary

The Anatomy and Boundary of Adaptation under Temporal Tabular Shift

Tianyu Wang, Xi Vincent Wang, Lihui Wang, Mian Li, Zhihao Liu

arXiv:2609.12136v1stat.MEcs.AI

TL;DR

Adaptation under temporal drift has mixed outcomes, and model architecture alone does not reveal which mechanisms produced a gain. The paper audits gains under a leakage-corrected streaming protocol and finds that neither drift recovery nor unlabeled adaptation is a general rule.

  • Problem

    Adaptation under temporal drift improves some deployments and harms others, while method rankings alone do not identify why.

    Source: §1 · p. 1

  • Method

    The paper audits four recurring gain mechanisms under a prequential streaming protocol and uses leave-one-channel-out replay to measure channel contributions.

    Source: §1 · p. 2, §3 · p. 6, §5.4 · p. 15

  • Results

    Across eight temporal streams, observed improvements and one harm case were attributable to a small set of mechanisms, supporting neither adaptation claim as a general rule.

    Source: §8 · p. 38

  • Takeaways & Limitations

    Channel-resolved attribution matters because architecture alone can miscredit sophisticated components for gains reproduced by a running mean.

    Source: §5.4 · p. 15

    Whether a window’s drift belongs to a covariate- or label-shift class cannot be tested from unlabeled data, so invoking those classes is a modeling commitment.

    Source: §4.1 · p. 12

Abstract

from arXiv · show

Prequential adaptation of frozen tabular foundation models under temporal drift, with each label revealed only after prediction, helps some deployments and harms others, yet current practice does not predict which. We study the sources and limits of these gains. A diagnostic anatomy attributes gains to four recurring mechanisms under a streaming protocol that removes three optimistic biases and quantifies a fourth. Within an agnostic total-variation drift class, the target conditional is only partially identified: its identified-set diameter, the \emph{wall}, is irreducible from unlabeled data uniformly in sample size. A second, orthogonal $L^2$ projection wall quantifies what the frozen representation cannot express. Two canonical mechanism priors collapse the first wall. Under stated nuisance-rate conditions, the wall can be estimated from labeled historical windows at a $\sqrt N$ rate above the margin threshold $\gamma^\star=d_0/(2\alpha_s)$. At $\gamma=0$, the conditional lower-bound program depends on an open affinity estimate; the positive-margin lower branch also remains open. Semi-synthetic data illustrate the finite-sample mechanism with calibrated exponents. Stream-level proxies on eight industrial streams fall on the difficult side under a stated roughness bound, while the equality case $\gamma=\gamma^\star$ remains unresolved.

1 Introduction

The introduction asks how much degradation under temporal tabular shift is correctable, from which information, and at what price. It previews a mechanism audit, an identification boundary, and deployment diagnostics, while emphasizing that adaptation can help or harm.

1 Introduction

  • Across eight industrial streams, streaming residual correction improves seven and costs 2.28 AUC points on the eighth, motivating channel-level diagnosis.The paper attributes gains to four recurring mechanisms and treats them as an audit of this protocol, not a unique additive decomposition of every algorithm.Source: §1 · pp. 1–2
  • Within the agnostic drift class, the target conditional remains set-identified from unlabeled data, and the wall cannot shrink uniformly with sample size.A separate representation wall measures information the frozen embedding discarded; the two obstructions call for labels versus a new embedding.Source: §1 · p. 2
  • The wall has a proved sqrt(N)-rate learning branch above γ⋆=d0/(2αs), subject to nuisance-rate conditions, while the γ=0 lower-bound program and positive-margin lower branch remain open.The equality case γ=γ⋆ is also unresolved; examined stream proxies fall below the threshold under αs≤1.Source: §1 · p. 2
  • The deployment analysis adds a local-correction screening rule, a Wall-A/Wall-B/correctable-error attribution procedure, and a rate linking anchors to wall reduction.The screening rule uses no labels beyond those already consumed by the gated corrector and retrospectively detects the reported harm case.Source: §1 · p. 2
  • The streaming protocol removes future-neighbor leakage, temporal-twin noise floors, and duplicate-deflated intrinsic dimension, while measuring the cost of safety gates scored in a different loss.The loss mismatch is described as structural rather than tunable.Source: §1 · p. 3

2 Related Work

The related work spans test-time adaptation, tabular models, shift theory, representation limits, semiparametric methods, and drift monitoring. The paper positions its contribution as measuring what available information identifies and separating that limit from what a frozen representation can express.

Test-Time Adaptation Practice

  • Unlabeled test-time adaptation methods and direct tabular adaptation methods provide context and baselines, but do not ask which parts of conditional drift are determined by time-respecting information.The paper's prequential system consumes each label after prediction, unlike the label-free constraint that delimits the cited TTA wall theory.Source: §2 · p. 3

Distribution-Shift Theory: Bounds versus Identification

  • Classical domain-adaptation bounds and covariate-shift reweighting do not determine a drifting conditional; the paper measures the remaining identified-set ambiguity as a wall.The wall is the diameter of conditionals consistent with label-free observables under the agnostic drift class.Source: §2 · p. 4

Representation-Level Limits

  • The paper separates information discarded by a frozen representation from information no representation can identify, assigning these distinct limits to Wall B and Wall A.The attribution procedure uses this distinction to differentiate when labels are needed from when the embedding must change.Source: §2 · p. 5

Semiparametric Efficiency and Non-Smooth Functionals

  • The regular wall-estimation branch draws on semiparametric efficiency and debiased-learning methods, while its threshold uses the margin condition in a new role.The cited margin condition indexes whether the wall functional is smooth or non-smooth at the relevant scale; the conditional lower-bound construction retains an open affinity-estimate dependency.Source: §2 · p. 5

Drift Monitoring and Evaluation Practice

  • Drift monitors often rely on promptly available labels or loss-denominated scores, whereas this paper's screening rule uses labels already consumed by its corrector and accounts for conditional drift through a budget.Its future-neighbor leak is compared with look-ahead bias managed by purged cross-validation.Source: §2 · p. 5

3 Setting, Protocol, and Measurement Conventions

The paper studies a frozen tabular model on temporally ordered streams: predictions precede labels, and correction may use only strictly past labels. Its theory assumes an agnostic total-variation drift budget and its multi-window results require additional sampling, smoothness, and nuisance conditions.

3 Setting, Protocol, and Measurement Conventions

  • The setting fixes a frozen representation and readout, with predictions measurable from that representation and no deployment-time retraining.The empirical analysis uses eight industrial TabReD streams and extends the proxy study to fifteen shift benchmarks.Source: §3 · p. 6
  • Each label is revealed after prediction, and a corrector may use only the strictly-past labeled prefix; the unlabeled window supplies covariate-law information but not its conditional labels.The drifted conditional is constrained only to a pointwise total-variation ball around the source conditional, without cross-covariate coupling or an assumed drift mechanism.Source: §3 · p. 6

3.1 Standing Assumptions

  • The multi-window wall-learning results assume sequential windows with MCAR anchors, a stationary response curve, novelty support, smooth drift in representation coordinates, and controlled ratio-estimation errors.Additional conditions include shape and exogeneity assumptions for the response and ratio error.Source: §3.1 · p. 7
  • The boundary theory covers conditional-mean decisions scored with proper-loss currency; ranking measures such as AUC remain measured counterexamples outside that theory's scope.The ranking-currency analogue of the theory and screening rule is left open.Source: §3.1 · pp. 7–8
  • The evaluation uses time-respecting retrieval oracles, temporally decoupled noise-floor pairs, and intrinsic-dimension estimates after exact-duplicate removal.A second independently written implementation recomputed the reported numbers.Source: §3.1 · p. 8
  • The roughness bound αs≤1 is an assumption, while the margin condition describes drift mass thinning near the crossing; neither is presented as a derived property.The text identifies MCAR selection as especially easy to violate in practice.Source: §3.1 · pp. 7–8

4 The Anatomy of Adaptation Gains

The section audits the evaluated corrector’s gains through four recurring mechanisms and two canonical drift classes that collapse the identified set. The audit is mechanism-oriented, not a unique decomposition of every adaptation algorithm.

4.1 Four Gain Mechanisms and Two Collapse Classes

  • On delivery-eta, a running-mean-only corrector reproduces 93% of the probe corrector’s +0.66% RMSE improvement.C1 tracks the slowly moving mean of past residuals; on cooking-time, region-resolved offsets add at most +0.07 percentage points over one global offset.Source: §4.1 · p. 9

4.2 The Eight-Stream Decomposition and the Harm Case

  • Every positive stream’s gain is accounted for by C1–C4, led by C1 on delivery-eta, C2 on sberbank-housing, C3 on weather, and C4 on classification streams.Table 1 summarizes the per-stream decomposition; its gain column reports the single-prior reconstruction, while the dominant-channel column attributes the historical deployed system.Source: Table 1 · p. 10, §4.2 · p. 12

4.1 Four Gain Mechanisms and Two Collapse Classes

  • On sberbank-housing, the freshest, random past-only, and oldest-label arms deliver +4.9%, +4.1%, and −1.1% RMSE, respectively, isolating freshness at matched buffer size.The oldest labels are actively harmful in this comparison.Source: §4.1 · p. 10
  • On weather, embedding neighbors share residual errors, and local re-estimation delivers +3.9% RMSE improvement.The mean absolute residual difference between embedding neighbors is 78% of that between random pairs; the time-respecting ceiling is 0.067 of residual variance with a causally growing buffer, versus 0.0995 for the batch ceiling.Source: §4.1 · pp. 10–11
  • Region-resolved offsets improve AUC by +1.3–1.9 points on homesite-insurance and +1.3–1.8 points on ecom-offers, while deployed systems gain +0.7 and +0.9 points.These classification gains coexist with squared-error ceilings that are zero, negative, or marginal.Source: §4.1 · p. 11
  • Pure covariate shift makes the identified set a singleton at n=0; under label-marginal shift, known independent class-conditionals identify the target prior from unlabeled covariates.For label-marginal shift, the window marginal must also be consistent with the class, and the prior is estimated at the parametric rate.Source: §4.1 · pp. 11–12
  • Membership in these collapse classes is untestable from unlabeled data, so invoking C5 is a modeling commitment rather than a measurement.Agnostic, covariate-shift, and label-shift worlds can induce the same unlabeled law at every sample size.Source: §4.1 · p. 12

4.2 The Eight-Stream Decomposition and the Harm Case

  • On homecredit-default, naive residual correction costs 2.46 AUC points on the deployed system, as rare-positive local smoothing becomes noisy base-rate estimation.The stream has a 2.2% default rate; likelihood-optimal regional offsets that improve log-loss by 2.1% reduce AUC by 1.47 points in-sample.Source: §4.2 · p. 12

5 Measurement Methodology for Streaming Adaptation

This section measures and corrects optimistic biases in streaming evaluation, while treating channel attribution as a reporting correction rather than a numerical one.

5.1 Future-Neighbor Leakage Inflates Signal Ceilings

  • Time-respecting ceilings deflate the batch estimate by up to 4.5× and reverse sberbank-housing’s apparent signal from +0.080 to −0.048.On homesite-insurance, R2 falls from 0.0735 to 0.0164; a temporal-gap stress test lowers it further to 0.006.Source: §5.1 · p. 13, Figure 1 · p. 14

5.2 Temporal Twins Deflate Noise Floors

  • On weather, the naive nugget is 0.0009, while the temporally decoupled noise-floor estimate is 0.13–0.62 of residual variance.Because close pairs are temporal twins sharing noise realizations, the corrected estimate requires pairs at least 1,000 stream steps apart; on weather, this leaves no close-pair support, so the result is an extrapolated range.Source: §5.2 · p. 14

5.3 Duplicates Deflate Intrinsic Dimension

  • After four-decimal deduplication, unsw nb15’s intrinsic dimension rises from 0.87 to 3.58, and acs employment’s rises from 1.62 to 3.11.The correction is inert on nearly duplicate-free TabReD streams, where estimates move by at most 0.08.Source: §5.3 · p. 15

5.4 Architectural Attribution Miscredits Gains

  • On delivery-eta, a running mean reproduces 93% of the probe corrector’s gain, illustrating why architecture alone cannot identify the gain mechanism.The proposed attribution replays the same prequential protocol with one channel disabled at a time.Source: §5.4 · p. 15

5.5 Currency-Mismatched Gates Miss Deployment Harm

  • On homecredit-default, the log-loss gate locks at 15.1% of the stream, after 84% of the AUC harm has accrued.Log-loss improves from 2% to 5% of the stream while ranking harm continues; locking at half-stream more than doubles never-lock harm, from −2.36 to −5.26 AUC points.Source: Figure 1 · p. 14, §5.5 · p. 16

5.6 Ceilings versus Realized Gains

  • Time-respecting regression ceilings are 0–2.5% of residual variance, while hindsight capacity gains flatten, saturate, or reverse out of time.Paired bootstrap resolution matters: the frozen-model interval half-width exceeds reconstructed gain on four streams; sberbank-housing’s historical gain is attributed to freshness, not the local channel reconstructed here.Source: §5.6 · pp. 16–17, Figure 2 · p. 17

6 The Boundary: What Unlabeled Data Can and Cannot Determine

This section distinguishes what unlabeled covariates cannot identify from what a frozen representation cannot express, and examines when historical labels can support estimation of the former. It tests the margin threshold using stream-level proxies and a controlled semi-synthetic experiment, while the conditional lower-bound program and empirical phase classification remain incomplete.

6 The Boundary: What Unlabeled Data Can and Cannot Determine

  • The boundary analysis treats label-free observation as each prediction-time state in the prequential protocol, uses proper-loss conditional-mean theory, and excludes ranking from its scope.The section frames the wall as the limitation a corrector faces before labels arrive.Source: §6 · p. 18

6.1 The Identified Set and Its Diameter

  • Under pointwise drift budgets, the identified-set diameter has no additive covariate-geometry floor; in binary mean-gap prediction it is 2E[β] minus a nonnegative clipping term.The mean-gap diameter lies between E[β] and 2E[β], while the decision-flip diameter is the probability that the budget reaches the source prediction's 0.5 threshold.Source: §6.1 · pp. 19–20

6.2 Irreducibility: The Wall Is Assumed, Not Measured

  • Across every sample size, unlabeled-only estimation of the agnostic conditional wall has constant positive minimax risk whenever drift is positive on the region.Unlabeled tests have power equal to size, so the wall is maximal only among budget-respecting priors. Covariate shift collapses it at n=0, while label shift collapses it using unlabeled moments at an n^-1/2 rate.Source: §6.2 · pp. 20–22

6.3 The Second Wall: What the Representation Discarded

  • Wall B is the L2(pW) residual variance of ηW after projection onto σ(ϕ), equivalently the discarded conditional information under log loss.It is geometrically orthogonal, not statistically independent, from Wall A and satisfies WallB ≤ E[β^2]. For injective ϕ, exact Wall B vanishes; refining ϕ can reduce Wall B yet increase deployable Wall A on sub-fiber regions.Source: §6.3 · p. 23
  • Residual error separates into a Wall-A share addressed by acquiring labels, a Wall-B share addressed by changing or fine-tuning the representation, and a correctable share addressed by the Part I channels.The two walls call for different interventions, so treating them as a single adaptation failure can prescribe the wrong remedy.Source: §6.3 · p. 24
  • For fiber-union regions, deployable and raw walls coincide; on sub-fiber regions, the signed gap is positive when the region over-selects high-drift points with pinned fiber-mates and negative when it under-selects.The deployable wall averages drift widths over the full fiber population, and its label-free plug-in estimand is given the representation, baseline conditional, drift bound, and window distribution.Source: §6.3 · p. 25
  • In the worked two-point fiber example, the drifting cell’s raw wall is 0.60, while the frozen readout’s deployable wall is 0.30 because its pinned mate constrains the fiber-level prediction.The deployable wall is half the raw wall in this example.Source: §6.3 · p. 25, Figure 4 · p. 26

6.4 Learning the Wall from History

  • When γ > γ⋆ = d0/(2αs), the wall is estimable at the parametric √N rate under the theorem’s stated nuisance-rate conditions, and no regular estimator improves on that rate.The theorem also requires overlap, cross-fitting, proportional target size, and a sup-norm density-ratio rate; equality at the threshold is not established.Source: §6.4 · pp. 26–28
  • The equality case γ = γ⋆ remains unresolved, while the positive-margin lower branch remains open and the γ = 0 lower bounds are conditional on an unproved affinity estimate.If that estimate fails, the conditional lower-bound floors lapse.Source: §6.4 · pp. 27, 29
  • At γ = 0, the conditional lower-bound results imply a non-regular fixed-K floor; for growing K, the conditional worst-case rate does not improve because coherent drift preserves the single-window bias floor.These conclusions depend on the open Appendix-K affinity estimate (2c).Source: §6.4 · p. 27
  • Regime recurrence helps average stochastic error only under an incoherence prior; coherent worst-case drift gets no averaging and incurs the per-window price.The gain is on the stochastic-error component, not the bias component.Source: §6.4 · p. 30
  • At γ=0, conditional lower bounds depend on an open affinity estimate; positive-margin sub-threshold adaptation remains open, with a growing-K interval lacking a proven attaining estimator.For growing K, the interval (γ*, γ*(λ)) has a proven N^-1/2 lower bound but no proven estimator attaining it.Source: §6.4 · pp. 29–30
  • With margin γ>γ*=d0/(2α_s), sign recovery makes wall estimation from historical labels a smooth problem and supports √N-rate learning under stated nuisance conditions.The rate theorem assumes the sup-norm density-ratio condition H6; the program establishing that premise from primitives remains partial.Source: §6.4 · p. 30

6.5 Proxy Estimates and Sensitivity Analysis on Twenty-Three Data Sets

  • For all eight TabReD streams, 2α_s bγ<d0 under α_s≤1 and representative margins; sberbank-housing compares 2.12 (1.63 at measured bαs=0.77) with d0=3.01.This is conditional evidence, not an uncertainty-quantified phase assignment; the smoothness proxy overstates smoothness, favoring the below-threshold reading.Source: §6.5 · p. 30, Figure 5 · p. 31
  • The fifteen other benchmarks fall below threshold only under assumed α_s=γ=1; deduplication materially shifted low-uniqueness cases, so these data do not establish a population phase map.The six high-d0 rows were 99.97–100% unique, while two of the re-measured low-uniqueness data sets left the low-dimensional reading entirely.Source: §6.5 · p. 31, Table 4 · p. 32

6.6 Semi-Synthetic Illustration of the Threshold Mechanism

  • The semi-synthetic design isolates sign recovery with a known representer, and changing either d0 or γ alone moves scaling across the threshold.Its finite-grid tests report effective slopes rather than certify asymptotic exponents.Source: §6.6 · pp. 32–33

7 Beyond the Boundary: Diagnostics and Decision Rules

The section turns the boundary results into screening, attribution, and labeling procedures. Retrospective evaluations show the screening rule avoids false adaptations but forgoes gains, while attribution points to different remedies across streams.

7.1 The Local-Correction Screening Rule

  • The rule adapts only when local signal clears the calibrated noise floor and the decision-flip fraction is material; it freezes when the identified set is wide or the buffer is sparse or stale.The rule is a screening rule, not a certificate: it has no finite-sample false-adapt guarantee.Source: §7.1 · p. 34
  • The screening rule made no false adaptations retrospectively, but froze four streams whose gains came wholly or partly through channels it cannot see.The harmed stream was frozen; loss-denominated gates miss temporal and ranking gains as well as some global-debias gains.Source: §7.1 · pp. 34–35

7.2 The Degradation-Attribution Procedure

  • WallBρ was indistinguishable from zero on every stream; local correction recovered error only on weather and homesite-insurance, so no uniform remedy fits all streams.Sberbank-housing had the largest room (0.34) with none locally correctable, while homecredit-default's entire 0.15 was remainder and naive correction harmed.Source: §7.2 · p. 36, Table 6 · p. 37

7.3 The Price of Labels

  • Halving region-averaged uncertainty costs 4× the labels, while halving pointwise uncertainty costs 32× at d0 = 3 and about 3 × 10^4× at d0 = 13.The section therefore favors regional wall audits and decision-flip monitoring over pointwise drift maps at measured intrinsic dimensions around 3–18.Source: §7.3 · pp. 37–38

7.4 Deployment Recommendations

  • Deployment recommendations combine out-of-time validation, local-channel screening, three-way attribution, and label allocation when the attribution places degradation on the identifiability wall.A sound ranking-currency gate remains open, so ranking deployments must monitor their deployment currency.Source: §7.4 · p. 38

8 Conclusion

The conclusion finds no general rule that adaptation under drift either recovers drift or cannot help. The paper separates non-identifiability from representation error, while stating unresolved theoretical branches and empirical scope limits.

8 Conclusion

  • Across eight temporal streams, the evidence supports neither universal claim about adaptation, and the examined proxy settings do not establish a universal empirical phase law.The reported improvements and one harm case are attributed to recurring mechanisms under a leakage-corrected protocol.Source: §8 · p. 38
  • The equality case γ = γ⋆ remains uncovered; the lower-bound program is conditional at γ = 0 and open for 0 < γ < γ⋆.The regular-branch rate also depends on stated nuisance-rate hypotheses, including H6.Source: §8 · p. 38
  • All measurements use one backbone family, and schema or semantic drift that changes the covariate space lies outside the formalism.The stream-level proxy estimates are noisy, although the verdict under the rough-field bound is insensitive to that noise.Source: §8 · p. 38
  • The screening rule's retrospective one-sidedness on eight streams is not a bound on unseen-stream soundness, and its loss-denominated gates forgo ranking gains.A sound ranking-currency gate and a ranking-currency analogue of the wall theory remain open.Source: §8 · pp. 38–39

Reader’s guide to the appendices

The guide explains where the appendix proofs and status records appear, then states the standing setup and the label-free observation property used in the irreducibility argument.

Reader’s guide to the appendices

  • The reader can verify claim status at contract level, inspect six pillar proofs at pillar level, or follow the full developments across the appendices and Online Appendix 1.Appendices B, H, and J are included here; many others and the program contract appear in the supplementary document.Source: Reader’s guide to the appendices · p. 44

B.0 Standing assumptions

  • The standing setup uses a Polish covariate space, binary labels as the primary case, a frozen source conditional, and an assumed drift budget.Unlabeled covariates are i.i.d. from the window distribution; labels are never observed in this setup.Source: §B.0 · pp. 44–45

B.1 Construction and ancillarity

  • With the covariate distribution fixed, unlabeled covariate data have the same observation law across candidate conditionals because marginalizing the unobserved labels erases the conditional.The appendix calls the full covariate sample ancillary for the conditional parameter.Source: §B.1 · p. 45

B.2 Le Cam two-point and the irreducibility theorem

The section uses two-point comparisons to establish that unlabeled covariates cannot identify the conditional within the agnostic drift class. The resulting minimax and confidence-set bounds persist at every sample size.

B.2 Le Cam two-point and the irreducibility theorem

  • The binary mean diameter is Eµ[wβ], attained by the extremal pair, and the minimax upper bound is attained by a zero-data midpoint estimator.The wall is strictly positive when the region has positive probability mass where β > 0.Source: §B.2 · p. 46
  • Any test distinguishing two candidate conditionals has power equal to size, since their unlabeled observation laws coincide..Source: §B.2 · p. 46
  • A uniformly valid 1 − α confidence procedure has expected parameter diameter at least (1 − 2α)Dmean(β; R) for every n.The bound is stated in inner/outer-expectation form because the data-dependent diameter need not be measurable.Source: §B.2 · p. 46
  • For every n, including n = ∞, the minimax risk remains bounded below by half the largest pairwise parameter separation for any real functional on these worlds.The bound is unchanged even if an oracle reveals the covariate distribution.Source: §B.2 · p. 47

B.3 Scope delimitation: the scope theorems

The scope theorems show that the wall depends on the assumed mechanism class: two canonical classes collapse it, while budget-respecting classes remain bounded by it. The magnitude-wall branch adds a margin-dependent estimation result whose lower-bound status remains qualified.

B.3 Scope delimitation: the scope theorems

  • Under covariate shift, the identified set is the unchanged source conditional and the wall is zero for every discrepancy, region, and budget, even at n = 0.This is a prior-driven collapse; no unlabeled sample is needed.Source: §B.3 · p. 47
  • With known, linearly independent class-conditionals, unlabeled covariates identify the target prior and collapse the label-shift identified set to one conditional.The prior is recovered by a bounded-moment system and estimated at rate n^-1/2.Source: §B.3 · pp. 47–48
  • The unrestricted budget-free class can exceed that envelope, so the maximal-wall claim is scoped to budget-respecting mechanism priors.The budget-free class has wall 1 when the budget wall is less than 1.Source: §B.3 · p. 48
  • The agnostic mechanism class remains unfalsifiable from unlabeled data: competing drift classes induce the same observation law at every sample size.The wall is maximal only among mechanism classes whose compatible conditionals fit within the drift budget.Source: §B.3 · p. 48

H The magnitude-wall margin boundary (Conjecture E.6 of Online Appendix 1, regime (II))

  • For the magnitude-wall branch, the margin threshold separates a proved regular branch from lower-bound arguments that remain conditional or open.The section records open affinity and rate issues; rate transfer also requires a mixture-density lower bound that is conjectured and may fail near boundary priors or barely overlapping supports.Source: §B.3 · p. 48, H The magnitude-wall margin boundary… · p. 49

N above the margin threshold

Above the margin threshold, sign recovery turns magnitude estimation into a signed, linear target and supports a root-N result under stated nuisance-rate conditions. The strict threshold and additional regime qualifications delimit that guarantee.

N above the margin threshold

  • Recovering the sign makes the magnitude target linear on correctly signed regions, while raw labels leave sign error as the outcome-bias source.The margin makes that sign-error contribution second-order; the estimator avoids the smoothing bias of a plug-in absolute value.Source: N above the margin threshold · pp. 49–50
  • Above γ⋆ = d/(2αs), the sign-corrected DML2 one-step is asymptotically normal at rate N^-1/2 under the stated nuisance-rate and fixed-K conditions.The result inherits qualifications from the regular regime, including the ratio-channel condition.Source: N above the margin threshold · pp. 50–51
  • The threshold is strict: equality is unresolved, and at a generic crossing the regular branch requires d < 2αs.For larger intrinsic dimension relative to smoothness, the generic crossing is at or below threshold.Source: N above the margin threshold · p. 51
  • With growing K, the threshold changes with the growth rate, and sufficiently rapid growth requires stronger Stage-1 tails and variance stabilization.For log K/log N tending to λ, the stated threshold is γ⋆(αs,d,λ) = 1/[2a(1−λ)] − 1.Source: N above the margin threshold · p. 51

H.2 Route B: the Cai–Low floor below the margin threshold

Below the margin threshold, the absolute-value target is non-smooth, and the no-margin lower-bound program uses moment-matched alternatives that are hard to distinguish from labels. Its strongest floor remains conditional on an open affinity estimate.

H.2 Route B: the Cai–Low floor below the margin threshold

  • Moment-matched drift-height priors can be nearly indistinguishable from Bernoulli labels while their absolute-value functionals differ by order 1/D.The resulting non-regularity lies in the labels-to-magnitude map.Source: §H.2 · p. 52
  • For K = 1 at γ = 0, the stated lower bound is k^-a up to logarithmic factors, conditional on the open affinity gap and transfer steps.The construction uses a Hölder interior class with zero crossing.Source: §H.2 · pp. 51–53
  • The fixed-K no-margin lower-bound conclusion is non-regularity, but it depends on the open permutation-mixture affinity estimate.The stated conditional rate is k^-a up to logarithmic factors.Source: §H.2 · p. 52
  • For positive margins below threshold, both the proposed adaptive upper bound and the matching lower-bound adaptation remain open.The available polynomial-debiasing achiever is rate-sharp only at K = 1; the general-K lower bound also carries an affinity gap.Source: §H.2 · pp. 51, 53

H.3 The regular branch, boundary estimator, and lower-bound status

The magnitude wall has a regular branch above the margin threshold and a conditional no-margin lower bound below it. The boundary case and positive-margin lower-bound adaptation remain unresolved, while the proposed boundary estimator is only partially established.

H.3 The regular branch, boundary estimator, and lower-bound status

  • For fixed K and γ > γ⋆, the covered-part magnitude wall is regular at rate N^-1/2, subject to the Section-G qualifications.The result uses the sign-recovery reduction to the regular branch.Source: §H.3 · pp. 53–54
  • At γ = 0, the fixed-K lower bound implies non-regularity only conditional on the affinity gap and transfer steps.For 0 < γ < γ⋆, the proposed upper-bound adaptation and matching lower-bound adaptation remain open.Source: §H.3 · pp. 53–54
  • Although the polynomial exponents meet at γ = γ⋆, no boundary-case claim follows because the regular-branch condition is strict.The equality case remains unresolved.Source: §H.3 · pp. 53–54
  • The polynomial-debiasing boundary estimator is rate-sharp up to logarithmic factors only for γ = 0 and K = 1, conditional on the affinity gap.Its general-K aggregation and positive-margin adaptation are not established.Source: §H.3 · p. 54

H.4 Summary of status (Section H)

Section H summarizes supporting estimation and inference steps alongside stream-level measurement proxies and evaluation protocols, which motivate but do not establish population-level phase claims. It also records unresolved boundary cases: the lower branch is conditional or open, equality remains unresolved, and the candidate upper-rate exponent applies below a stress-test threshold within a specified scope.

H.4 Summary of status (Section H)

  • Above γ⋆ = d/(2αs), the magnitude wall has an N-regular established branch; the γ = 0 lower bound remains conditional, positive-margin adaptation below threshold is open, and equality is unresolved.The general or growing-K conditional program is taken up in Appendix I.Source: §H.4 · p. 55
  • The sup-norm variable-radius k-NN estimator attains rate (log n/n)^(αs/(2αs+d)) with failure probability at most n^-2 under the stated minimal-mass and smoothness conditions.The bound extends to probability at least 1−n^-c for any fixed c>0, supporting the tail used in Corollary H.4(b).Source: §H.4 · pp. 55–56
  • The inid empirical-to-population step is oP(N^-1/2) when the fold-trained nuisance error is oP(1), but the result requires fold-balanced designs and does not cover window-level folding.Stationarity (A1) identifies the stated mixture variance; without it, the variance is smaller by the average squared per-window means, and the confidence-interval constant is conservative.Source: §H.4 · pp. 56–57
  • The inid extension of the ratio-type bracketing argument for Theorem C.12, proof step (3), remains an open point not settled in this section.Source: §H.4 · p. 57

J Measurement protocol and the proxy/sensitivity study

  • The measurement appendix uses eight TabReD streams as noisy single-stream proxies; fifteen related benchmarks lack labeled temporal drift fields and enter only sensitivity analysis at (αs, γ) = (1, 1).These analyses motivate, but do not establish, a population-level claim about dataset phases.Source: J Measurement protocol and the… · p. 57

J.1 Measurement protocol

  • For TabReD, αs and γ are estimated from a single-stream residual-field proxy, with αs bounded by 1; γ is unmeasurable on label-free benchmarks, and duplicate-row correction is applied before intrinsic-dimension estimation.The αs proxy is biased high by k-NN smoothing and unstable across the two-range fit; label-free readout smoothness is not substituted for drift smoothness.Source: §J.1 · p. 58
  • The evaluation specifies out-of-time probe and reconstruction protocols, a retrieval-scale-matched wall resolution, and label-free screening gates, with reported verdicts robust across the stated threshold ranges and bootstrap block multipliers.The wall resolution is tied to the deployed k = 20 retrieval scale; reconstruction reports the local corrector alone, not the tuned seven-prior ensemble.Source: §J.1 · pp. 59–60
  • The strict reverse inequality is a lower branch conditional on gap (2c) at γ = 0 and remains open for 0 < γ < γ⋆; equality is unresolved.The current logarithmic-rate proof does not resolve the equality case.Source: §J.1 · p. 61
  • With (αs, γ) = (1, 1), the threshold check reduces to d0 > 2, useful for stress testing but not data-set-specific phase measurement.Source: §J.1 · p. 61
  • Below threshold, the candidate upper-rate exponent is a(1 + γ), with a = αs/(2αs + d0); its matching scope is specified in Appendices H and I of Online Appendix 1.Source: §J.1 · p. 61

J.2 The proxy and sensitivity display for the twenty-three data sets

The proxy and sensitivity display places both stream-level proxies and favorable sensitivity scenarios below the threshold for their stated inputs. The threshold verdict depends on the rough-field bound, and the reported rate exponents are implications of proxy inputs, not measured minimax rates.

J.2 The proxy and sensitivity display for the twenty-three data sets

  • Both stream-level proxies and favorable sensitivity scenarios lie below the threshold for their stated inputs, but neither set has uncertainty intervals.The stream-level proxies use observed drift labels; favorable scenarios do not.Source: §J.2 · p. 61
  • sberbank-housing would need αs > 1.42 versus measured ∼0.77, while ecom-offers needs > 1.90 versus ∼0.14 and homecredit-default > 2.03 versus ∼0.50.The remaining five streams need αs > 5.44–9.22, against measured values of 0.05–0.95.Source: §J.2 · p. 61
  • The plug-in upper-rate exponent a(1+γ) ranges from 0.007 to 0.35 and quantifies proxy-input implications, not minimax rates measured from the streams.Most values are below 0.15; the extremes are 0.007 for homesite-insurance and 0.347 for sberbank-housing.Source: §J.2 · p. 61
  • The below-threshold verdicts depend on αs ≤ 1; the variogram estimator cannot detect αs > 1, so it cannot rule out smoother drift fields crossing the threshold.The stated crossing points are αs > 1.42 for sberbank-housing, > 1.90 for ecom-offers, > 2.03 for homecredit-default, and > 5.44–9.22 for the remaining streams.Source: §J.2 · p. 61

J.3 The deduplication correction

The correction removes duplicate-driven downward bias in intrinsic-dimension estimates, materially changing readings for low-uniqueness data while leaving essentially duplicate-free streams and benchmarks largely unchanged. Under the stated sensitivity inputs, the corrected examples do not enter the regular branch.

J.3 The deduplication correction

  • Duplicate rows bias intrinsic-dimension estimators downward, so the protocol measures FM-embedding dimension after deduplication.The correction is driven by many points at first-neighbor distance r1 ≈ 0.Source: §J.3 · p. 61
  • Table 7 compares unique fractions and raw versus deduplicated MLE d0 across nine re-measured related benchmarks and six high-d0 benchmarks, with d0/2 shown for αs = 1.For the six high-d0 benchmarks, the deduplication shift is a noise bound rather than a correction.Source: Table 7 · p. 62, §J.3 · p. 62
  • On unsw nb15, d0 rises from 0.87 to 3.58 after deduplication; the raw estimate is estimator-unstable, with TwoNN giving 2.93 on the same data.The stream has 41% duplicate flows.Source: §J.3 · p. 62
  • On acs employment, d0 rises from 1.62 to 3.11 after deduplication, making d0/2 = 1.56 under the stated sensitivity inputs.The true drift smoothness is not measured, so the raw reading’s apparent explanation does not hold after correction.Source: §J.3 · p. 62
  • Across the eight TabReD streams, all are 99.8–100% unique and deduplication shifts d0 by at most 0.08.The minimum deduplicated TabReD MLE d0 is 2.94.Source: §J.3 · p. 63, Table 8 · p. 63
  • After correction, none of the re-measured rows enters the regular branch under (αs, γ) = (1, 1), but this does not establish the true drift fields’ threshold status.Neither αs nor γ is measured for the related benchmarks.Source: §J.3 · p. 63

J.4 Reading the display

Deduplication changes the apparent threshold reading for three low-uniqueness datasets, while the display supports comparative stress testing rather than a ranking of empirical wall heights. The ACS family shows an ordered dimension pattern, but the inputs and phase calls remain uncertain.

J.4 Reading the display

  • Deduplication moves three low-uniqueness datasets below the regular branch under the sensitivity inputs; only the deduplicated column enters Table 4 and Figure 5.The raw readings placed two datasets in or near the regular branch and a third at its border.Source: §J.4 · p. 63
  • Under αs = 1, d0/2 ranges from 1.43 for cicids to 7.03 for mimic iv mortality, a dimension-driven spread that is not a ranking of wall heights.Interpreting it as wall height would also require measuring drift smoothness.Source: §J.4 · p. 63
  • Deduplicated ACS d0 increases with target complexity: employment 3.11, unemployment 3.49, public coverage 4.78, and income 6.49.The ordering persists after correction, but it is not an independent test of the threshold theorem.Source: §J.4 · p. 64
  • TabReD drift proxies are noisy, and both drift quantities are unobserved for related benchmarks; phase validation would require repeated labeled windows and uncertainty intervals.The αs and γ proxies show two-range slope disagreement on five of eight streams and upward bias from k-NN smoothing.Source: §J.4 · p. 64

Related papers

Loading 2609.12136v1…