Rethinking Tabular Foundation Models On Data Streams

1 Introduction

• Earlier streaming-TFM evaluations were narrow in their stream and cost coverage, leaving backbone, memory, label delay, and drift effects without a controlled joint account.On streams, TFMs adapt by updating context memory rather than model parameters.Source: §1 · p. 1
• Backbone quality and memory size dominate predictive performance, while memory management policy matters mainly under delayed labels.The study examines memory composition, memory size, label delay, context refresh rate, and query batching.Source: §1 · p. 1
• TFMs are the most accurate but also the most costly learners in comparisons with streaming learners and streaming AutoML systems.The comparison uses real and synthetic streams with different types of drift under a shared protocol.Source: §1 · p. 1, Figure 1 · p. 2
• Repeated context encoding is the dominant serving cost; compact causal models with cached context keep cost nearly constant across memory sizes while remaining close to re-encoding in accuracy.The context is re-encoded at every prediction despite changing by only one row between steps.Source: §1 · p. 2

Data streams

• A data stream is an unbounded sequence of examples arriving one at a time from a distribution that may change, requiring prediction and adaptation under bounded memory and time.Evaluation is prequential: each example tests the current model before becoming available for learning, and its label may be delayed.Source: §2 · p. 2

Tabular foundation models

• TFMs predict from labelled context rows and a query in one forward pass without updating parameters, while stream adaptation updates the context memory.TabPFN, TabICL, and TabFM use different context-encoding designs, but encoding costs time quadratic in the number of rows.Source: §2 · p. 2
• Prior fixed-table context construction retrieves or compresses rows from a fixed pool drawn from one distribution, unlike streaming adaptation to changing data.Source: §2 · p. 2

Streaming TFMs

• DualFIFO uses short-term FIFO and class-balanced long-term stores, while CURE evicts redundant rows; neither prior evaluation jointly covers drift types, serving cost, and delayed labels.DualFIFO was evaluated on real streams and CURE on one synthetic abrupt-drift stream; both assume immediate labels.Source: §2 · p. 2

Streams

• The evaluation uses real and synthetic classification streams, including repository data with arrival-order rows and synthetic generators for four drift types.Because each drift type is paired with a different generator, the drift types are descriptive conditions rather than directly comparable groups.Source: §3 · p. 3

Methods

• The comparison includes classical streaming learners, streaming AutoML, and multiple TFM backbones and variants, with No-Change and Majority-Class as reference baselines.Streaming learners and AutoML run on CPU, while TFMs run on GPU; the TFM set spans different designs, pretraining strategies, and scales.Source: §3 · p. 3

Evaluation protocol and metrics

• All methods follow a prequential protocol with delayed labels; memory-based methods retain at most B labelled examples and use a shared budget unless varied.Reported metrics include accuracy, balanced accuracy, Cohen’s κ, temporal κt, latency per instance, and memory growth per instance.Source: §3 · p. 3

Experimental setup

• The study releases its experimental code, run configurations, and raw result files to support reproducibility.Source: §3 · p. 3

• TabFM+FIFO ranks first in accuracy on every stream but one; Drift-Resilient TabPFN narrowly leads on abrupt drift.Source: §4 · p. 3, Table 2 · p. 4
• In-context methods sit at the most accurate and expensive end, with predictive gains requiring more than an order of magnitude higher latency than the cheapest ensemble.Source: §4 · p. 4, Figure 3 · p. 5
• TFMs lead early and recover higher after abrupt drift, but advantages vary: gradual and incremental streams separate little, while recurring-drift swings can exceed paradigm gaps.Source: §4 · p. 4, Figure 2 · p. 5, Figure 4 · p. 6
• Memory size affects accuracy more than policy; FIFO changes little while reducing cost, and the abrupt-drift stream peaks below the largest budget.Source: §4 · p. 5, Table 3 · p. 6
• TabFM+FIFO retains the highest mean performance at every label delay, but CURE's benefit is confined to smart-meter, where it surpasses TabFM+FIFO at the longest delay.Source: §4 · pp. 5–7, Figure 6 · p. 7
• Refresh-rate reductions save little before harming accuracy, whereas shared-query batching cuts per-instance time with similar recurring-drift accuracy but larger smart-meter declines.Source: Table 4 · p. 7, §4 · p. 7, Table 5 · p. 8
• Fixed call overhead dominates single-query serving and is largely amortisable; budget scaling indicates most default-budget TabICL and TabPFN cost is context encoding, though its exact share is unmeasured.Source: §4 · p. 7, Table 6 · p. 8
• Cached compact models keep time and GPU memory flat as memory grows, while re-encoding scales superlinearly in time and near-quadratically in memory, taking up to ten times longer; this is a proof of concept, not a competitive model.Source: §4 · pp. 7–8, Figure 8 · p. 8

Conclusions

• TFMs lead the evaluation in predictive performance, drift recovery, and accuracy under label delay, but serve more than an order of magnitude slower than the cheapest streaming ensemble.Backbone choice and memory size matter more than memory policy, which helped only under delayed labels and on one stream.Source: §5 · p. 9

Limitations

• The study covers stream classification only, and its context limits and serving costs leave substantially larger memories unexplored.Context-based adaptation also depends on labelled examples in memory, which constrains adaptation under delayed feedback and changing concepts.Source: §5 · p. 9
• Caching experiments use compact models and fixed attention windows, so they do not establish that the efficiency gains transfer to larger pretrained TFMs.Future work also faces the challenge of scaling efficient architectures while retaining current TFMs’ predictive strength.Source: §5 · p. 9

Future work

• Future models could encode arriving examples once and reuse their representations as the stream evolves.The paper identifies local attention, recurrent state reuse, and key-value caching as useful foundations, alongside pretraining on drifting sequences.Source: §5 · p. 9

AI Use Statement

• The authors used Claude and ChatGPT to refine manuscript text and code for analysis, figures, and tables, then reviewed the output and verified reported numbers.They state that experimental design, scientific claims, and interpretation remain their own.Source: AI Use Statement · p. 9

A Streams

• The streams include repository datasets, smart-meter recordings from Dryad, and person-activity recordings from UCI.The appendix says included repository streams are available through the USP data stream repository.Source: A Streams · p. 13

A.1 Real streams

• Real streams include electricity, weather, household energy use, building images, and activity recordings, selected only when row order reflects observation arrival.The electricity stream has strongly autocorrelated labels; posture no8 retains ten activities to fit TabPFN v1’s class limit.Source: §A.1 · p. 13

A.2 Synthetic streams

• The four synthetic-stream generators are implemented through CapyMOA and are absent from the pretraining prior of every model trained for the study.CapyMOA wraps the MOA implementations.Source: §A.2 · p. 13
• Synthetic streams cover abrupt, gradual, incremental, and recurring drift using AGRAWAL, SEA, RandomRBF, and RandomTree generators.Their schedules include three abrupt switches in 30,000 AGRAWAL instances, a 10,000-instance SEA transition, continuous centroid movement in RandomRBF, and cycling among three trees in RandomTree.Source: §A.2 · p. 13

Evaluation protocol

• All experiments use single-pass prequential test-then-train evaluation, with predictions made before labels are revealed.Streaming learners and AutoML methods run on CPU hosts, while TFMs and backbone-efficiency measurements run on GPU hosts.Source: B Reproducibility · p. 13

Hardware and cost comparisons

• Cross-method wall-clock comparisons use each paradigm’s respective platform, while within-backbone efficiency comparisons use fixed hardware.Because the platform asymmetry favours TFMs, normalisation would increase their measured cost relative to CPU methods.Source: B Reproducibility · pp. 13–14

Implementation details

• The appendix reports backbone parameter counts of 25.8M for TabPFN v1, 27.6M for TabICL v2, and 1.64B for TabFM.Compact-model comparisons vary datapoint-axis attention masks within each architecture family while keeping the remaining training configuration fixed.Source: B Reproducibility · p. 14

Reproducibility materials

• Released result files include metrics, per-window histories, computational costs, memory measurements, host information, and reproduction configurations.Default evaluation settings and hardware/software details are summarized in Tables 7 and 8.Source: B Reproducibility · p. 14, Table 7 · p. 14, Table 8 · p. 14

C.1 Accuracy

• TabFM+FIFO is best on every stream except abrupt drift, where Drift-Resilient TabPFN leads; in-context methods separate most from classical learners on meter and rialto.Table 9 gives per-stream accuracy behind Table 2.Source: §C.1 · p. 15, Table 9 · p. 15

C.2 Label delay

• TabFM+FIFO has the best mean accuracy at every tested label delay, though CURE overtakes it at the longest delay on meter.Most of the decline with delay comes from meter.Source: §C.2 · p. 15, Table 10 · p. 16

C.3 Variation across streams

• Temporal κt distributions show that similar medians can conceal different spreads and that strong overall rankings do not imply uniform performance across streams.Figure 9 also encodes per-instance serving cost with fill colour on a logarithmic scale.Source: §C.3 · p. 15, Figure 9 · p. 16

C.4 Statistical comparison

• TabFM+FIFO ranks first on all but one stream, but the paired Wilcoxon–Holm comparison does not separate the leading in-context methods with so few streams.The exception is Drift-Resilient TabPFN on the abrupt-drift stream.Source: §C.4 · p. 15, Figure 10 · p. 17

C.5 Decision boundaries on a synthetic stream

• On one synthetic stream, shared-memory tests show irrelevant features hurt kNN, while larger FIFO memories sharpen steady-state boundaries but prolong drift recovery.Recovery time grows nearly in proportion to memory size; hindsight size selection is worth several points, and selectors over all sizes capture some of that headroom.Source: §C.5 · p. 17, Table 11 · p. 17, Figure 11 · p. 18

C.6 Computational Cost

• Serving time varies across streams, and the reported resident-memory growth is process-level behaviour rather than isolated model state.Table 13 reports per-stream serving time in milliseconds per instance and mean latency rank.Source: §C.6 · p. 20, Table 13 · p. 20

C.7 Cost at cloud prices

• At the stated cloud prices, the least expensive TFM costs about ten times the most expensive classical ensemble and over 250 times the Hoeffding tree.Charging CPU methods for a full 8-vCPU instance is conservative for the paper’s cost comparison.Source: §C.7 · p. 20

D.1 Fine-tuning

• Prefix-based fine-tuning produces negligible changes and no clear improvement over pretrained TabICL in the evaluated cases.The result is specific to the tested prefixes, training budget, and checkpoint-selection procedure.Source: §D.1 · p. 20

D.2 Similarity instead of attention

• The similarity-based predictor stores labelled rows and uses cosine-similarity-weighted voting with recency decay and a drift-dependent gate.Source: §D.2 · p. 20
• Raw features outperform frozen TabICL representations on most streams, but both remain below TabFM+FIFO; frozen representations also require substantially greater serving time.They improve performance only on the smart-meter stream.Source: §D.2 · p. 21, Table 17 · p. 22
• Recency weighting improves overall performance, especially on several real streams, while the drift gate helps more on synthetic drifting streams and has mixed effects on real data.The ablation reports Δ as off minus on, so negative values favor the component.Source: §D.2 · p. 22, Table 18 · p. 22

D.3 Pre-training small models

• Compact nanoTabPFN and nanoTabICL models are pretrained from scratch on synthetic drifting sequences that exclude the evaluation streams.Source: §D.3 · p. 22
• The experiments vary bidirectional, causal, and sliding attention masks within each architecture family while keeping the remaining training configuration fixed.Source: §D.3 · p. 22
• Sliding attention restricts each row to a window of W observations, giving O(W) attention cost per step at fixed model size.Source: §D.3 · p. 22

D.4 Datapoint-axis masks

• Causal and sliding attention improve accuracy under abrupt, gradual, and incremental drift in both architecture families, with larger gains for nanoTabPFN.Source: §D.4 · p. 22, Table 19 · p. 23, Figure 13 · p. 23
• Recurring drift behaves differently across the two architectures, and nanoTabICL’s shared column-stage dependencies limit full cache compatibility.Source: §D.4 · p. 23, Figure 14 · p. 23
• Wider sliding-attention windows mainly benefit recurring and stationary sequences, while other drift types change less beyond moderate window sizes.Source: §D.4 · p. 23
• The mask comparison and window sweep are separate training experiments, so their values should be compared only within each experiment.Source: §D.4 · p. 23, Table 19 · p. 23

D.5 Transfer to evaluation streams

• Both compact models outperform No-Change on several evaluation streams but perform poorly on electricity, where short-lag label autocorrelation is strong, and remain below the released TabICL backbone.A learned recency bias improves NOAA but reduces performance elsewhere; its slopes remain at their initial values, leaving no evidence that useful recency weighting was learned.Source: §D.5 · pp. 23–24, Table 21 · p. 24

D.6 Cached serving

• At a fixed attention window, cached serving stays nearly constant in time and peak GPU memory across memory budgets, while re-encoding becomes more expensive in both.Re-encoding memory grows faster than time because its attention footprint scales with the square of retained memory.Source: §D.6 · p. 24, Table 22 · p. 25

E Prior work

• Prior streaming-TFM evaluations leave gaps in drift-type coverage and single-instance serving cost, while existing TFM approaches also differ in context scaling and cache compatibility.The cited prior work includes methods evaluated only on real-world streams or with runtime compared only against another TFM method.Source: E Prior work · p. 26

F Accuracy and cost of streaming TFMs

• The appendix frames streaming-TFM accuracy and cost in terms of pretraining-prior fit, memory age, and context re-encoding.Source: F Accuracy and cost of streaming TFMs · p. 27, §F.1 · p. 27, §F.2 · p. 27

F.1 Why TFMs are accurate on streams

• Pretraining toward the prior predictive distribution can explain good predictions from small memories when the stream is represented by the prior, but does not guarantee accuracy on a given stream.Source: §F.1 · p. 27
• With FIFO memory and label delay d, the memory is fully replaced B + d instances after an abrupt drift; this is a replacement time, not a recovery bound.Accuracy may recover earlier or remain incomplete after replacement, and larger memory or delay increases context age.Source: §F.1 · p. 27
• Hoeffding-tree ensembles need enough instances to resolve split tests, and after each reset must pay that sample requirement again, helping explain slower cold starts and drift recovery.Resolving a candidate gap of η takes on the order of R^2 ln(1/δ)/(2η^2) instances per leaf.Source: §F.1 · p. 27

F.2 Why TFMs are costly and how to reduce the cost

• Re-encoding the memory at every prediction incurs repeated cost even though only one row changes per step; batching reduces fixed overhead but does not remove per-call context cost.With causal attention over a fixed window, rows can be encoded once and read from a cache, whose storage no longer depends on B; cached and re-encoded predictions may differ because cached rows can retain evicted observations.Source: §F.2 · p. 27