Source-linked AI summary
Tydra: An Efficient Hybrid Model for Tabular Data
Mieszko Komisarczyk, Saurabh Mathur, Maurice Kraus, Sriraam Natarajan, Kristian Kersting
TL;DR
Tabular foundation models face a trade-off between TabPFN’s predictive performance and quadratic inference cost, while hybrid attention–SSM architectures remain unexplored for tabular data. Tydra interleaves attention and SSM layers, achieving up to 1.43× faster inference than TabPFN while retaining comparable predictive performance across OpenML benchmarks.
Problem
Hybrid attention–SSM architectures remain unexplored for tabular foundation models, which face a trade-off between predictive performance and inference cost.
Method
Tydra interleaves TabPFN-style attention layers with Hydra’s efficient SSM sequence mixing for tabular in-context learning.
Results
1.43× faster inference than TabPFN while retaining comparable predictive performance across OpenML benchmarks.
Takeaways & Limitations
Combining subquadratic sequence mixers with PFN-style learners is a promising direction for scaling tabular foundation models.
Takeaways & Limitations
Extending Tydra to large-scale, long-context tabular data remains an important direction for future work.
Abstract
from arXiv · showhide
Transformer-based tabular foundation models such as TabPFN achieve strong predictive performance but incur quadratic computational cost with context length. On the other hand, subquadratic SSM-based alternatives such as Hydra trade away accuracy for efficiency. To balance both, we introduce Tydra, a hybrid Transformer-State Space Model (SSM) architecture for tabular in-context learning that interleaves attention and SSM layers. Across 30 OpenML datasets, Tydra reduces inference time by 30% relative to TabPFN while retaining much of its predictive performance. Tydra also outperforms an approximately ten-times-larger Hydra model while providing faster inference. The results indicate that hybrid architectures are a promising direction for tabular foundation models.
Introduction
Tydra is introduced as a hybrid Transformer–SSM model for tabular in-context learning to address TabPFN’s quadratic inference cost. Across 30 OpenML datasets, it matches TabPFN’s accuracy at up to 30% lower inference time while substantially outperforming Hydra.
- Motivation: TabPFN delivers strong tabular prediction without task-specific training, but its Transformer backbone scales quadratically with context length.This makes inference prohibitively expensive for large-scale or long-context tasks and restrictive for institutions unable to use serverhosted inference.
- Method: Tydra interleaves TabPFN attention layers with Hydra SSM layers as the first hybrid Transformer–SSM architecture for tabular in-context learning.The approach targets the accuracy–efficiency trade-off while accounting for tabular data’s permutation invariance over rows and columns.
- Results: 30% lower inference time on 30 OpenML datasets is achieved by Tydra while matching TabPFN’s accuracy and substantially outperforming Hydra.The reported efficiency improvement makes strong tabular in-context learning practical on small-to-medium-scale data when serverhosted inference is unavailable.
- Study scope: Tydra’s architecture family is studied extensively across different combinations and ratios of Hydra and TabPFN layers.The paper then presents related work, the Tydra architecture, and empirical results before concluding.
Related Work
Tydra builds on TabPFN’s attention-based tabular in-context learning, SSMs’ efficiency advantages, and hybrid architectures designed to balance efficiency and accuracy. It adopts an interleaved design combining TabPFN attention layers with Hydra state-space layers.
- TabPFN: TabPFN performs tabular classification through in-context learning and a single forward pass over training data and test queries.Its Transformer architecture adapts self-attention for permutation invariance across rows and columns and is meta-trained on millions of synthetic classification tasks.
- TabPFN: TabPFN’s later versions broaden task coverage and increase model capacity while retaining the same attention mechanism.The cited extensions include regression support, roughly doubled Transformer depth, and a higher supported class count.
- State Space Models: SSMs address limitations of RNNs and Transformers through near-linear sequence-length complexity, with Mamba providing selective state-space filtering and linear-time inference.The passage presents selectivity as enabling Mamba to match or exceed Transformer performance.
- State Space Models: SSMs provide an efficiency advantage over attention-based tabular foundation models but can reduce predictive performance, especially on standard small-to-medium-scale datasets.The passage contrasts TabPFN’s quadratic cost with Hydra’s subquadratic scaling and practicality.
- Hybrid Architectures: Tydra adopts an interleaved hybrid strategy that combines TabPFN’s attention layers with Hydra’s state-space layers.Hybrid architectures are presented as a solution to the efficiency–accuracy tradeoff by combining Transformer layers with subquadratic layers.
The Tydra Family of Architectures
Tydra is a hybrid tabular in-context learning family that interleaves Hydra state-space layers with Transformer attention, combining efficient bidirectional sequence mixing with content-dependent interactions. Its architecture preserves permutation-aware, full-row interactions with linear-time processing and is meta-trained on synthetic classification tasks using a prior-data fitted network pipeline.
- Architecture: Tydra interleaves four Hydra–Transformer pairs, yielding eight layers, with each layer preserving the same m-dimensional representation.The evaluated fully interleaved model is referred to as Tydra; Transformer layers use self-attention and feed-forward sublayers with residual connections and normalization.
- Architecture: Each table row becomes one token; training rows add separately encoded labels, while test rows use only feature embeddings in a shared token space.The model processes labeled training rows and unlabeled test rows together before decoding test-row representations into class probabilities.
- Hydra layers: Hydra combines forward and reversed causal SSM scans with diagonal terms, enabling every row to attend to every other row while retaining O(L) time and memory.This bidirectional quasiseparable construction addresses tabular permutation invariance and matches full self-attention’s expressivity without materializing the mixer.
- Training: Tydra is meta-trained offline on synthetic tabular classification tasks generated by the neural-network component of the TabPFN prior framework.Training retains the prior-data fitted network pipeline and classification objective while replacing the original Transformer backbone with Tydra’s alternating Hydra–Transformer backbone.
Empirical Evaluation
Empirical evaluation compares Tydra with Transformer and SSM baselines across real-world and synthetic tabular benchmarks, measuring predictive performance and inference efficiency. Across 30 OpenML datasets, Tydra matches TabPFN’s predictive performance while achieving inference speedups of up to 29.9%.
- Datasets: 30 OpenML-CC-18 classification datasets with at most 2,000 observations, 100 features, and 10 classes were used alongside synthetic datasets for long-context speed and memory-scaling tests.Each table was split equally into context and query samples; synthetic data contained no categorical features or missing values.
- Methods: Tydra was compared with a 25.8M-parameter TabPFN-style Transformer, 16M- and 160M-parameter Hydra models, and hybrid variants differing in attention–Hydra layer ratio and placement.All models used the same prior-fitting procedure and synthetic-data prior.
- Metrics: AUROC measured predictive performance across five deterministic 50/50 train-test splits per dataset, with mean values reported in the main results.The corresponding standard deviation across splits was reported in the appendix.
- Metrics: Inference speed was defined as total test predictions divided by synchronized model inference time, measured over 75 trials after two warm-up runs.The trials consisted of 15 repetitions for each of the five splits.
- Results: Up to 29.9% inference speedups (mean +24.8%) were achieved against TabPFN while predictive-performance differences stayed within mean |∆AUROC| = 0.006 on all but one OpenML dataset.The comparison covered 30 OpenML datasets and a synthetic benchmark isolating runtime scaling.
49 training
Tydra matches TabPFN’s predictive performance while reducing inference cost through a hybrid Transformer–SSM architecture. Among evaluated variants, the 4HT configuration provides the strongest accuracy–speed trade-off, and Tydra scales better than larger baselines.
- Tydra matches TabPFN’s predictive performance while substantially reducing inference cost.
- Tydra scales substantially better than TabPFN and Hydra 160M while remaining competitive with Hydra 16M up to 213 rows.At the largest synthetic benchmark size, Tydra falls behind Hydra 16M.
- The 4HT architecture, combining Transformer and Hydra layers in a 1:1 ratio, provides the best accuracy–speed trade-off.Hybridization generally produces models faster than TabPFN without sacrificing too much predictive power.
- 1.24 points worse than TabPFN is the average predictive-performance cost of the T{6H}T variant with six Hydra and two Transformer layers.
Conclusions and Future Work
Tydra combines Hydra and Transformer layers for tabular in-context learning, achieving up to 1.43× faster inference than TabPFN while retaining comparable predictive performance. Future work includes alternative sequence mixers, large-scale long-context data, and looped transformers.
- Contributions: 1.43× faster inference than TabPFN is achieved while retaining comparable predictive performance across OpenML benchmarks.Tydra is introduced as a hybrid Hydra-Transformer architecture for tabular in-context learning.
- Contributions: Combining subquadratic sequence mixers with PFN-style learners is identified as a promising direction for scaling tabular foundation models.
- Future Work: Alternative subquadratic mixers, including gated linear attention and Gated DeltaNet, could replace Hydra while interleaving with attention layers.These alternatives may offer different points on the accuracy–efficiency trade-off.
- Future Work: Extending Tydra to large-scale, long-context tabular data remains an important direction for future work.
- Future Work: Looped transformers are also identified as a future research direction for Tydra.