PaperlayerView exact revision on arXiv

Source-linked AI summary

Adapting Linear-Time Architectures for Tabular In-Context Learning

David Schnurr, Felix Sarnthein, Thomas Hofmann, Imanol Schlag

arXiv:2609.36337v1cs.LGstat.ML

TL;DR

Linear-time architectures could ease the scaling limits of softmax attention in tabular ICL, but their generalisation beyond the pretraining context is uncertain. This paper compares training setups and recurrent mixers, diagnoses length-related state instability, and adapts DeltaNet’s write rates and readout. The resulting linear-time model closes most of the gap to softmax attention in the reported evaluation.

  • Problem

    Softmax attention scales quadratically with context size, while it remains unclear whether linear-time architectures can construct a state that generalises for tabular ICL.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    The paper controls comparisons of training design and recurrent update rules, diagnoses length generalisation with a hidden-state oracle, and stabilises DeltaNet with time-dependent write-rate decay and final-state readout.

    Source: §1 · p. 1, §4.1 · p. 6, §6 · p. 9

  • Results

    The adapted model is the best-performing linear-time model in the evaluation, closing most of the gap to softmax attention, especially with Interleaved Multi-Target or final-state readout.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Across the considered settings, stable state construction is the key challenge to learning a generalising tabular ICL algorithm.

    Source: §6 · p. 10

    Recurrent variants degrade beyond roughly 2–4× the pretraining context length, and tested mitigations do not solve the issue.

    Source: §1 · p. 2

Abstract

from arXiv · show

Tabular foundation models achieve strong performance by conditioning on labelled examples in context, but softmax attention limits their use on large datasets. Existing linear-time alternatives, however, are mostly causal, and their potential for tabular in-context learning (ICL) remains underexplored. To address this, we (1) revisit causal training setups, (2) compare linear sequence mixers, and (3) investigate their ICL generalisation beyond the pretraining context length. First, we show that the best training setup for causal models resembles next-token prediction. Then, perhaps surprisingly, the most promising linear sequence mixer is causal: DeltaNet outperforms even non-causal linear attention. However, it degrades beyond $2$-$4\times$ the pretraining context length, and existing mitigation strategies such as bidirectionality defer the problem at best. A hidden-state oracle shows that this is not a capacity problem. Instead, our analysis points to an instability in the recurrent state, which drifts in deeper layers of causal models. Since DeltaNet's learned write rates overfit to the pretraining regime, we modulate them with a time-dependent decay schedule intervention to stabilise length generalisation. Finally, re-introducing non-causality by reading out from the final state allows us to closely match a controlled softmax attention baseline on OpenML-CC18 and TabArena.

1 INTRODUCTION

Tabular in-context learning offers an alternative to tree-based methods, but softmax attention scales poorly on large datasets and recurrent mixers’ suitability for exchangeable rows remains uncertain. The paper studies training design, sequence mixers, and length generalisation, finding that delta-rule updates and stabilised final-state readout offer promising linear-time alternatives.

Training design space

  • Interleaved Multi-Target is strongest for causal models, while Combined Single-Target is strongest for non-causal models.Source: §1 · p. 2

Controlled recurrence comparison

  • Delta-rule updates outperform other causal recurrences and non-causal linear attention at moderate sequence lengths.The paper also finds that DeltaNet degrades beyond roughly 2–4× the pretraining context length, while order-invariant models remain stable.Source: §1 · p. 2

Length-generalisation failure

  • Every recurrent variant degrades beyond roughly 2–4× the pretraining context length, and tested mitigations including bidirectionality do not solve the problem.Order-invariant models remain stable in this comparison.Source: §1 · p. 2

Failure mechanisms and stabilisation

  • A time-dependent decay to DeltaNet’s write rate βt mitigates its sequence-length overfitting, while deeper causal layers show directional state drift.A hidden-state oracle indicates that fixed-size state capacity is not the bottleneck.Source: §1 · p. 2

Final-State DeltaNet

  • Final-State DeltaNet lets every query read directly from the stabilised final state to remove the readout-level causal bottleneck.Source: §1 · p. 2

2 BACKGROUND

Tabular foundation models predict query labels from labelled examples in context, while linear-time alternatives seek to replace costly pairwise attention with a compressed state. The background introduces recurrent update rules, their test-time regression interpretation, and the unresolved challenge of constructing a state that generalises.

2 BACKGROUND

  • PFNs cast supervised learning as prediction from labelled context examples, training on held-out targets without fitting task-specific parameters at test time.TabPFN applies this approach to tabular classification.Source: §2 · p. 2

2.1 SCALING BOTTLENECK OF TRANSFORMERS

  • Softmax attention requires quadratic computation across context samples, motivating linear-time models that compress the context into a parametric state.Whether a generalising state exists and how to construct it remain open questions.Source: §2.1 · p. 2

2.2 LINEAR ATTENTION AND RECURRENT ARCHITECTURES

  • Linear attention uses a feature map to construct a fixed-size state, while recurrent mixers differ in how they update that state from past key-value pairs.The listed mechanisms include input-dependent forgetting, delta-rule overwriting, and state-space updates.Source: §2.2 · p. 3

2.3 CAUSAL LINEAR RECURRENCES AS LAYER-LOCAL TEST-TIME REGRESSORS

  • The test-time regression view treats recurrent state as an online approximation to a layer-local associative-memory problem.Prior analysis motivates testing whether that state converges to a stationary solution as more context samples arrive.Source: §2.3 · p. 3

2.4 TABULAR FOUNDATION MODELS WITH LINEAR BACKBONES

  • Earlier tabular linear-backbone studies vary sequence mixers, label embeddings, and target formulations together, making individual contributions difficult to attribute.This paper separates those design axes and examines length generalisation.Source: §2.4 · pp. 3–4

3 DISENTANGLING TRAINING STRATEGIES AND SEQUENCE MIXERS

The study controls training design and sequence-mixer comparisons to assess linear-time architectures for tabular ICL. DeltaNet performs strongly among linear mixers at moderate lengths, but longer real-world contexts expose a generalisation gap.

3.1 PRETRAINING AND EVALUATION PROTOCOL

  • Models are matched at 12.5M parameters and pretrained on synthetic datasets capped at 1,000 rows and 20 features.Evaluation uses OpenML-CC18* and TabArena*, with the latter measuring generalisation on larger datasets.Source: §3.1 · p. 4

3.2 TRAINING STRATEGIES

  • Interleaved Multi-Target best suits causal models, with teacher forcing as query feature-target pairs enter later queries’ contexts; interleaving doubles sequence length.Combined Single-Target remains competitive and is the only setting permitting fair causal versus non-causal comparison.Source: §3.2 · p. 5

3.3 SEQUENCE MIXERS

  • The study compares linear attention and recurrent update rules under Combined Single-Target, alongside a parameter-matched softmax baseline.The controlled Table 2 comparison reports results for the preprocessed OpenML-CC18* and TabArena* benchmarks.Source: §3.3 · p. 5, Table 2 · p. 6
  • Non-causal softmax attention is the strongest controlled architecture, while causal DeltaNet and Gated DeltaNet perform best among linear mixers.Delta-rule updates outperform vanilla linear attention; on TabArena, DeltaNet deteriorates as dataset sizes exceed the 1,000-row pretraining cap.Source: §3.3 · pp. 5–6, Table 2 · p. 6

4 UNDERSTANDING GENERALISATION FAILURE

Causal recurrent models lose performance beyond their pretraining context length, but the evidence points to unstable state construction rather than insufficient state capacity. An explicit write-rate decay stabilises DeltaNet at longer lengths, while tested mitigations alone do not eliminate degradation.

4.1 LENGTH GENERALISATION GAP AND HIDDEN-STATE CAPACITY

  • Beyond roughly 2-4× the 1,000-row pretraining range, causal models degrade while non-causal attention continues to generalise; DeltaNet can outperform non-causal linear attention when predictions are ensembled across 3K-sample blocks.A hidden-state oracle achieves strong 128k-context performance, showing that fixed-size state capacity is not the bottleneck, although practical deployment requires careful length-dependent tuning.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 MITIGATION STRATEGIES

  • Bidirectionality, final-state readout, and state weaving help near the pretraining range but do not solve length generalisation; long-sequence pretraining shifts degradation later at a short-context cost.State passing reduces degradation for GLA but lowers overall DeltaNet performance, and non-causality increases DeltaNet degradation.Source: §4.2 · p. 7

4.3 HIDDEN-STATE DRIFT IN CAUSAL LINEAR ATTENTION

  • Causal linear attention’s hidden states drift directionally from their 1,000-row states as context grows, with stronger drift in deeper layers and degradation occurring alongside it.Normalising each layer’s state matrix does not remove degradation, so magnitude drift alone is insufficient to explain the failure.Source: §4.3 · p. 8

4.4 DELTANET LENGTH OVERFITTING AND WRITE-RATE DECAY

  • Explicitly decaying DeltaNet’s effective write rate lets it improve beyond the pretraining length instead of collapsing, with only minor degradation at long sequence lengths.The intervention is motivated by learned write rates that remain excessively large at long lengths; the decay uses η_t = c/(t + c), with c = 256.Source: §4.4 · pp. 8–9

5 FINAL-STATE READOUT DELTANET WITH WRITE-RATE DECAY

The final-state readout gives DeltaNet non-causal information flow from a shared final context state in a single forward pass. Combined with write-rate decay, it is the strongest evaluated linear-time variant on the real-world benchmarks and substantially narrows the significance gap to non-causal softmax attention on TabArena.

5 FINAL-STATE READOUT DELTANET WITH WRITE-RATE DECAY

  • Final-state readout uses the same final context state for every query, enabling non-causal information flow in a single forward pass.Write-rate decay stabilises that state and supports length generalisation on synthetic tasks compared with causal DeltaNet.Source: §5 · p. 9
  • DeltaNet with final-state readout and write-rate decay is the strongest evaluated linear-time variant and improves over standard DeltaNet on both benchmarks.It slightly outperforms causal Delta + Decay with Interleaved Multi-Target on three of four metrics while processing half as many tokens per dataset.Source: §5 · p. 9
  • On TabArena, the final-state-readout model substantially closes the significance gap to non-causal softmax attention.The authors report that the improved length generalisation transfers to real-world datasets.Source: §5 · p. 9

6 DISCUSSION

The study finds that stable state construction is central to generalising tabular ICL: causal DeltaNet performs strongly at moderate lengths, but length generalisation remains challenging. A time-dependent decay to DeltaNet’s write rates improves stability, while the evaluation prioritises controlled comparison over absolute performance.

6 DISCUSSION

  • Interleaved Multi-Target best suits causal models, but requires 2× sequence length and is invalid for non-causal models; Combined Single-Target shows update rules matter more than non-causal information flow at moderate lengths.In Combined Single-Target, causal DeltaNet variants outperform the non-causal linear-attention baseline and remain competitive with the externally trained TabFlex checkpoint.Source: §6 · p. 9
  • The time-dependent write-rate decay produces the best-performing linear-time model in the evaluation, closing most of the gap to softmax attention, especially with Int-MT or final-state readout.The authors also report that the hidden-state oracle rules out fixed-size state capacity as the bottleneck; recurrent update dynamics instead drift beyond the pretraining range.Source: §6 · p. 9
  • The authors prioritise controlled comparison over absolute performance and use row-wise tabular ICL; larger pretraining budgets and further tuning may improve performance.Extension to two-axis attention or token preprocessing is expected, not established by this study.Source: §6 · p. 10
  • The study identifies stable state construction as the key challenge to learning a generalising ICL algorithm across the settings it considers.It adapts ordered fast-weight updates in causal and non-causal models to aggregate information across exchangeable tabular rows.Source: §6 · p. 10

Appendices

Experiments used two GPU cluster nodes, with longer-context pretraining assigned to the higher-memory node. Pretraining and length-generalisation evaluation together required approximately 100–125 GPU-days, excluding additional preliminary and development compute.

A COMPUTATIONAL RESOURCES

  • Longer-context pretraining used a node with 8 NVIDIA RTX A5000 GPUs, each with 24 GB of VRAM, because GPU memory constrained this work.The node also had an AMD EPYC 7742 64-Core Processor.Source: A COMPUTATIONAL RESOURCES · p. 16
  • Approximately 100–125 GPU-days covered pretraining and evaluation across the reported model families, training setups, mitigation strategies, and stabilised DeltaNet variants.Each model’s pretraining took approximately one to two days on a single GPU, while length-generalisation evaluations took several additional hours per model.Source: A COMPUTATIONAL RESOURCES · p. 16
  • Preliminary experiments, implementation development, and selection of fair parameter configurations required additional compute beyond the reported total.Source: A COMPUTATIONAL RESOURCES · p. 16

B.1 LINEAR-TIME ARCHITECTURE DETAILS

The appendices describe the shared recurrent-memory view and matched model setup, then detail pretraining, external baselines, a hidden-state oracle, and a layer-local ridge alternative. Together, these details specify how the comparisons were configured and what the oracle and ridge model modify.

B.1 LINEAR-TIME ARCHITECTURE DETAILS

  • Table 4 presents the evaluated linear-time mixers in a unified recurrent-memory view, with state S_t and unnormalised readout o_t.The table omits normalisation, output projections, and implementation-specific details.Source: Table 4 · p. 16, §B.1 · p. 16

B.2 PRETRAINING AND HYPERPARAMETER DETAILS

  • Controlled model comparisons use approximately 12.5M parameters, with embedding dimension 320, four heads where applicable, and feed-forward/intermediate dimension 640.Layer counts vary by architecture; Table 5 gives exact parameter counts, and the Delta-FS and decay variants match Delta’s count.Source: §B.1 · p. 16, Table 5 · p. 17, Table 6 · p. 17
  • Models are pretrained on fresh synthetic datasets sampled from the TabPFN v1 prior, with validation losses used for model and hyperparameter selection.Short convolutions are disabled except for DeltaNet Int-MT, which uses width 4; exploratory evaluations found these benefited that setting but reduced or did not materially improve others.Source: §B.2 · pp. 16–17

B.3 EXTERNAL BASELINES

  • External baselines include TabPFN v2.5, TabICLv2, TabFlex, and CatBoost, but released foundation-model checkpoints are not controlled for model size or training budget.The authors report 6.4B rows of pretraining exposure for their models versus an estimated 40B–100B+ for the foundation-model baselines, which also have comparable or larger parameter counts and more advanced priors.Source: §B.3 · p. 17

B.4 HIDDEN-STATE ORACLE DETAILS

  • The hidden-state oracle freezes model parameters and jointly optimises per-layer fast-weight states on the context-set regression objective.A 10% context holdout supports tuning and early stopping because optimised states can overfit even at 128k sequence length.Source: §B.4 · p. 18

B.5 LAYER-LOCAL RIDGE STATE DETAILS

  • The layer-local ridge model computes each layer’s state independently using a length-scaled ridge objective, rather than jointly optimising states end to end.Its state is permutation-invariant, accounts for key-representation covariance, and scales regularisation with context length.Source: §B.5 · p. 18

C.1 METRICS USED IN EVALUATION

The appendices define the evaluation metrics and paired significance-testing procedures, then describe the real-world TabArena length-generalisation comparison. On TabArena, DeltaNet variants are competitive with softmax attention at moderate dataset sizes, while non-causal models outperform causal models as size increases.

C.1 METRICS USED IN EVALUATION

  • The main evaluation averages raw ROC-AUC and accuracy over splits and datasets; length-generalisation experiments report normalised metrics over 500 synthetic datasets with 128k rows and a fixed 100-query set.Normalisation makes confidence intervals more comparable across datasets of differing difficulty.Source: §C.1 · p. 18

C.2 SIGNIFICANCE TESTING

  • Significance tests use paired two-sided Wilcoxon signed-rank comparisons with Holm correction at α = 5%.The procedure follows a pairwise comparison protocol.Source: §C.2 · p. 19

C.2.1 SIGNIFICANCE TESTING FOR TRAINING STRATEGIES

  • Training-strategy tests average over cross-validation splits and model families before pairwise Wilcoxon comparisons across datasets, applying Holm correction separately by benchmark and metric.The analysis includes GLA, DeltaNet, Gated DeltaNet, and Mamba-2, excluding standard Linear Attention because comparable interleaved-label performance requires custom positional encodings.Source: §C.2.1 · p. 19

C.2.2 SIGNIFICANCE TESTING FOR SEQUENCE MIXERS

  • Sequence-mixer tests compare dataset-level paired samples after averaging over cross-validation splits, with Holm correction applied separately within each table’s model-pair set.Shared bars indicate non-significant differences, not equivalence.Source: §C.2.2 · p. 19

D.1 REAL-WORLD LENGTH GENERALISATION ON TABARENA

  • At moderate TabArena dataset sizes, DeltaNet variants are competitive with softmax attention, but non-causal softmax and linear attention outperform causal models as size increases.Figure 4 shows model ranks smoothed in log10-space with a Gaussian kernel of σlog10 = 0.1.Source: Figure 4 · p. 20, §D.1 · p. 20

D.2 HIDDEN-STATE ORACLE ON REAL DATA

The real-data oracle supports the claim that fixed-size state capacity is not the bottleneck, while comparisons identify where recurrent models succeed or fail. Longer pretraining and GLA-specific mitigations shift long-context degradation, but do not eliminate it.

D.2 HIDDEN-STATE ORACLE ON REAL DATA

  • The hidden-state oracle outperforms causal DeltaNet and non-causal linear and softmax attention on all five TabArena datasets and on mean ROC-AUC.This supports the conclusion that a fixed-size state can achieve strong performance at these context lengths.Source: §D.2 · p. 20, Table 7 · p. 20

D.3 LAYER-LOCAL RIDGE STATE COMPARISON

  • The layer-local ridge state is competitive, but stabilised DeltaNet outperforms it, with significant differences on TabArena.On synthetic length-generalisation comparisons, the ridge state is length-stable, while final-state DeltaNet with write-rate decay closely matches or sometimes exceeds it.Source: §D.3 · pp. 20–21, Figure 5 · p. 21

D.4 SCALING PRETRAINING SEQUENCE LENGTH

  • Longer-context pretraining improves performance at longer evaluation contexts but shifts degradation to longer sequences, in runs that are not compute-matched.The models see the same number of datasets but substantially more total rows during pretraining.Source: §D.4 · p. 21, Figure 6 · p. 22

D.5 GENERALISATION AND MITIGATION STRATEGIES FOR GLA

  • For GLA, mitigation strategies reduce but do not eliminate degradation; final-state readout leads to around 16× pretraining length, after which bidirectionality is strongest.State passing and mimetic initialisation help; state weaving shows no significant improvement in addressing degradation. The authors hypothesise that some improvements move GLA toward more stable causal or non-causal linear attention.Source: §D.5 · pp. 21–22, Figure 7 · p. 22

D.6 GENERALISATION FAILURE ANALYSIS FOR LINEAR ATTENTION

Causal linear attention’s recurrent state drifts with context length, especially in deeper layers. Constraining state magnitude has little effect on performance and does not remove degradation.

D.6 GENERALISATION FAILURE ANALYSIS FOR LINEAR ATTENTION

  • The causal recurrent key-value state’s Frobenius norm drifts relative to other layers as context grows; this effect is much weaker in the non-causal setup.The pattern appears both with and without K-sum denominator normalisation.Source: §D.6 · p. 23, Figure 8 · p. 24
  • Frobenius-normalising the recurrent state has only a very limited effect on causal and non-causal linear-attention performance.The normalisation preserves state direction while controlling magnitude.Source: §D.6 · p. 23, Figure 8 · p. 24
  • As context length grows, causal linear attention’s state orientation drifts most in deeper layers, while the non-causal state stays close to its 1000-row configuration.The comparison uses cosine similarity of flattened recurrent states against the corresponding layer and head at 1000 rows.Source: §D.6 · p. 23

D.7 DELTANET UPDATE DYNAMICS ANALYSIS

The analysis links DeltaNet’s length behaviour to its ordered recurrent updates and learned write rates, finding that explicit decay stabilises long-context performance. Extended comparisons also examine model variants and two-axis processing, which does not resolve length-generalisation failure.

D.7 DELTANET UPDATE DYNAMICS ANALYSIS

  • Unlike additive linear attention’s order-independent set aggregate, DeltaNet’s state is an ordered online optimisation trajectory, so final-state readout does not make it fully order-independent.The DeltaNet write rate controls how much the current sample overwrites prior memory along the key direction.Source: §D.7 · p. 25
  • The write-rate decay variant remains nearly stable across the evaluation range, while standard DeltaNet collapses beyond its pretraining range.The intervention decays effective write rates toward zero across sequence length in every layer; the standard model reduces them less aggressively, especially in deeper layers.Source: §D.7 · p. 25, Figure 9 · p. 26
  • Long-context-pretrained DeltaNet retains more contribution from earlier samples, while 1000-row-pretrained variants focus mostly on recent samples, supporting write-rate overfit to pretraining length.The contribution analysis is a lower-bound proxy under a simplified identical-unit-norm-key assumption.Source: §D.7 · p. 25, Figure 11 · p. 27

D.8 EXTENDED REAL-WORLD MODEL COMPARISON

  • The extended tables compare ridge-state, subsampled DeltaNet, and Int-MT variants on OpenML-CC18 and TabArena; 3K variants average logits from disjoint blocks of up to 3000 rows.The OpenML-CC18* table notes that its 1K sample cap makes each Subs. 3K variant identical to its full-context counterpart.Source: §D.8 · p. 27, Table 8 · p. 28, Table 9 · p. 28

D.9 TWO-AXIS ROW-FEATURE PROCESSING

  • Two-axis processing does not resolve length-generalisation failure: it increases degradation for DeltaNet, while causal linear attention remains comparatively stable.The two-axis models have 14.9M parameters and incur substantially higher memory and runtime costs; they are not parameter-matched to row-wise models.Source: §D.9 · p. 27, Table 10 · p. 28

E MEMORY AND COMPUTE SCALING ANALYSIS

Recurrent models reduce context-processing and query costs relative to softmax attention by storing fixed-size state, though ensemble cache costs and end-to-end timing impose qualifications. A stateless prediction patch enables parallel independent queries without copying the context state for each query.

E MEMORY AND COMPUTE SCALING ANALYSIS

  • Linear attention and DeltaNet process context in linear time and queries in constant time from fixed-size state; softmax attention has quadratic context processing and linear query processing with a cache.Softmax KV-cache size grows with context length, whereas recurrent models store a fixed-size hidden state.Source: E MEMORY AND COMPUTE SCALING ANALYSIS · p. 29, Figure 12 · p. 29, Figure 13 · p. 29
  • The reported figures omit separated end-to-end fit-and-predict timing because implementing ensemble caching is beyond the project’s scope.They report pure context time, query time, and cache size instead.Source: E MEMORY AND COMPUTE SCALING ANALYSIS · pp. 29–30
  • At 128k samples, a 10-member ensemble requires 23.4 GB for softmax attention’s KV cache versus 11.7 MB for DeltaNet’s recurrent state.The comparison illustrates the memory advantage of parametric models in the stated setting.Source: E MEMORY AND COMPUTE SCALING ANALYSIS · p. 30

F ADDING STATELESS PREDICTION TO FLASH-LINEAR-ATTENTION MODELS

  • The stateless patch predicts each query independently from the shared cached context state, without materialising query-updated states or nq copies of the state.The patch replaces the recurrent kernel update during query prediction while retaining standard FLA kernels for context processing.Source: F ADDING STATELESS PREDICTION TO… · pp. 30–31
Loading 2609.36337v1…