PaperlayerView exact revision on arXiv

Source-linked AI summary

LimiX: Unleashing Structured-Data Modeling Capability for Generalist Intelligence

Xingxuan Zhang, Gang Ren, Han Yu, Hao Yuan, Hui Wang, Jiansheng Li, Jiayun Wu, Lang Mo, Li Mao, Mingchao Hao, Ningbo Dai, Renzhe Xu, Shuyang Li, Tianyang Zhang, Yue He, Yuanrui Wang, Yunjia Zhang, Zijing Xu, Dongzhe Li, Fang Gao, Hao Zou, Jiandong Liu, Jiashuo Liu, Jiawei Xu, Kaijie Cheng, Kehan Li, Linjun Zhou, Qing Li, Shaohua Fan, Xiaoyu Lin, Xinyan Han, Xuanyue Li, Yan Lu, Yuan Xue, Yuanyuan Jiang, Zimu Wang, Zhenlei Wang, Peng Cui

arXiv:2509.03505v2cs.LGcs.AIcs.CL

TL;DR

Tabular foundation models remain limited in generality, often requiring separate components for classification, regression, imputation, and generation. LimiX models structured data as a joint distribution and use episodic masked pretraining for unified conditional prediction. Across 11 benchmarks, LimiX-16M consistently surpasses strong baselines, while scaling studies quantify benefits from larger datasets and models.

  • Problem

    Existing tabular foundation models remain limited in generality and commonly require separate task-specific models, adapters, or external pipelines.

  • Method

    LimiX treats structured data as a joint distribution over variables and missingness and uses masked joint-distribution pretraining with episodic context conditioning.

  • Results

    Across 11 large structured-data benchmarks and multiple tasks, LimiX-16M consistently surpasses competitive baselines, while LimiX-2M performs strongly under tight compute and memory budgets.

  • Takeaways & Limitations

    A single LimiX model can address diverse tabular tasks through conditional queries without task-specific architectures or bespoke per-dataset training.

  • Takeaways & Limitations

    Random-mask conditional modeling may fail to recover the full conditional distribution when only a subset of mask patterns is covered.

Abstract

from arXiv · show

We argue that progress toward general intelligence requires complementary foundation models grounded in language, the physical world, and structured data. This report presents LimiX-16M and LimiX-2M, two instantiations of our large structured-data models (LDMs). Both models treat structured data as a joint distribution over variables and missingness, thus capable of addressing a wide range of tabular tasks through query-based conditional prediction via a single model. They are pretrained using masked joint-distribution modeling with an episodic, context-conditional objective, supporting rapid, training-free adaptation at inference. We evaluate LimiX models across 11 large structured-data benchmarks with broad regimes of sample size, feature dimensionality, class number, categorical-to-numerical feature ratio, missingness, and sample-to-feature ratios. LimiX-16M consistently surpasses strong baselines, as shown in Figure 1 and Figure 2. The superiority holds across a wide range of tasks, such as classification, regression, missing value imputation, and data generation, often by substantial margins, while avoiding task-specific architectures or bespoke training per task. Notably, LimiX-2M delivers strong results under tight compute and memory budgets. We also present the first scaling law study for LDMs, revealing how data and model scaling jointly influence downstream performance and offering quantitative guidance for tabular foundation modeling. All LimiX models are publicly accessible under Apache 2.0.

1 Introduction

LimiX addresses the limited generality of task-specific tabular pipelines with unified large structured-data models. Its models support multiple conditional-prediction tasks through one recipe and achieve strong results across diverse benchmarks, while scaling laws guide model and data choices.

  • Structured data supports evidence-based decisionmaking and quantitative analysis across finance, healthcare, logistics, and public policy.
  • Existing tabular systems require separate training for each dataset and task, limiting deployment speed and cross-domain knowledge reuse.
  • LimiX treats structured data as a joint distribution over variables and missingness, expressing classification, regression, imputation, generation, and sample selection as queries to one model.
  • LimiX uses masked joint-distribution pretraining with episodic context conditioning, enabling per-dataset adaptation without fine-tuning at inference.
  • Across 11 benchmarks, LimiX-16M consistently surpasses competitive baselines across multiple tasks, while LimiX-2M remains strong under tight compute and memory budgets.
  • The work introduces the first scaling law for LDMs, quantifying how data and model scaling shape downstream performance.

2 Architecture

LimiX embeds tabular cells and processes dependencies across feature and sample axes with transformer blocks. Discriminative feature encoding restores explicit column identity while keeping representations compact.

  • Embedding of Tabular Data: Each tabular cell is projected into a latent embedding space, with separate embedding modules for features and outcomes.
  • Embedding of Tabular Data: Raw cell and outcome values are transformed using two-layer MLPs with LayerNorm and GELU activations.
  • Discriminative Feature Encoding: Feature-level attention based only on cell-value interactions cannot reliably infer a cell’s originating column.
  • Discriminative Feature Encoding: Discriminative feature encoding separates feature representations while constraining them to low effective rank for compact identity sharing.
  • Transformer Architecture: LimiX-16M uses 12 transformer blocks with two feature-level attention passes and one sample-level pass per block.

3 Pretraining

LimiX pretrains by recovering masked cells from visible context, then aligns inference with episodic context-conditional prediction. This supports diverse conditional queries and training-free dataset adaptation.

  • Masked Joint-Distribution Modeling: Masked-cell recovery exposes LimiX to varied conditional dependencies among variables and supports prediction, imputation, and iterative sample generation.
  • Context-Conditional Masked Modeling: Context-Conditional Masked Modeling splits each dataset into context and query rows, conditioning query predictions on the context subset.
  • Context-Conditional Masked Modeling: Context acts as non-parametric memory during inference, enabling per-dataset calibration and adaptation without gradient updates.
  • Joint Distribution Learning: CCMM improves dependency-structure modeling by enforcing consistency across conditional predictions of all features rather than a single prefixed conditional.
  • Masking Strategy: Cell-wise, column-wise, and block masks train recovery of isolated entries, entire features, and higher-order variable groups.

4 Pretraining Data Generation

LimiX generates diverse synthetic pretraining data with hierarchical structural causal models and adapts sampled variables into downstream tasks. Graph-aware and solvability-aware sampling help construct useful training sets.

  • Data Generation Pipeline: Synthetic data generation proceeds through DAG generation, data sampling, and task adaptation.
  • DAG Generation based on Hierarchical SCMs: Hierarchical DAG generation represents complex causal dependencies through connected local causal structures.
  • DAG Generation based on Hierarchical SCMs: Within each local causal structure, child values aggregate parent transformations and observational noise through a structural causal equation.
  • Edge Functions: Edge dependencies use MLPs, convolutional layers, or decision trees, with randomized configurations or independently sampled tree hyperparameters.
  • Aggregation and Noise: Child-node inputs are combined by simple average, weighted average, or MLP-based aggregation, with noise scaled to feature distributions.
  • Training-Set Sampling: Graph-aware and solvability-aware sampling select higher-quality pretraining data than purely random sampling.
  • Task Adaptation: Classification discretizes the initially continuous target, whereas regression retains continuous targets and may transform clustered values toward a more uniform scale.

5 Retrieval-based Ensemble

LimiX uses inference-time retrieval and ensembling to customize in-context samples and features without additional training. Its attention mechanisms support retrieval based on both sample similarity and feature relevance.

  • Retrieval strategy: Inference-time retrieval reweights and selects representative in-context samples and features using learned attention scores, requiring no additional training.The customized context is intended to further improve LimiX model performance.
  • Ensemble strategy: Each dataset is processed through multiple pipelines that permute columns and categorical labels, with schema-preserving feature transformations; classification uses 4 pipelines and regression uses 8.Transformations include quantile normalization, log-normal transformation, and high-energy SVD components.
  • Retrieval strategy: Two forward passes are used per pipeline: the first retrieves customized in-context samples, and the second predicts from the retrieved context.The first pass uses all in-context samples, while the second uses the customized subset.
  • Bi-level attention: Feature-level attention supplies feature weights, which reweight sample-level attention scores before selecting in-context samples for a second prediction pass.The reweighted score is a feature-weighted average of sample-level attention across features.
  • Toy examples: In a 2D ten-class toy example, sample-level attention assigns larger weights predominantly to in-context instances sharing the query sample’s category.The attention captures dependencies beyond Euclidean distance by operating in a latent embedding space and incorporating input–category dependencies.

6 Theoretical Analysis on Context-Conditional Masked Modeling

The theoretical analysis formalizes context-conditional masked modeling as learning the joint conditional distribution of test samples given in-context data. It establishes why random masks and larger mask counts support recovery, sample efficiency, and generalization, while showing that restricted masking can be insufficient.

  • Random masks: Randomly sampled masks are necessary because conditionals for only a subset of mask patterns may be insufficient to recover p(Xte|Xct) and generalize effectively.The practical mask family uses masks containing 0.1d to 0.4d features, while the theory analyzes fixed-size masks k.
  • Problem formulation: At test time, any feature may serve as the target, so the model must estimate the corresponding conditional distribution from the remaining test features and the in-context samples.The target variable is treated as one potentially missing feature of the test sample.
  • Joint conditional modeling: Learning p(Xte|Xct) is equivalent to achieving all single-feature conditional predictions, making the joint conditional distribution central to test-time predictive performance.The analysis states that recovering all masked-feature conditionals allows recovery of the joint conditional distribution.
  • SCM interpretation: Masked pretraining can model p(Xte|Xct) through posterior inference over structural causal models and likelihood under each SCM.The data-generating process yields conditional independence of context and test samples given the SCM.
  • Sample efficiency: Larger mask counts reduce estimation uncertainty and can require fewer samples to reach the same uncertainty, yielding greater sample efficiency for sufficiently large fixed n.The theorem compares covariance bounds Γ_k and states Γ_k1 ⪯ Γ_k2 when k1 > k2.
  • Generalization: The generalization-error upper bound decreases monotonically as the number of masked cells increases, under the theorem’s regularity conditions.The result concerns the estimated conditional joint distribution relative to the true distribution.

7 Evaluation

LimiX is evaluated across diverse structured-data benchmarks and tasks, with LimiX-16M generally leading strong baselines while LimiX-2M remains competitive under constrained resources. The evaluation also examines robustness, representation quality, fine-tuning, missing-value imputation, data generation, and out-of-distribution generalization.

  • Classification: LimiX-16M achieves the best performance on BCCO-CLS across most datasets and the highest ranking on all three evaluation metrics for BCCO-CLS, OpenML-CC18, and TALENT-CLS.Critical differences indicate a significant margin in most cases.
  • Classification: LimiX-16M outperforms all baselines on every classification benchmark in both mean and rank across all three metrics.LimiX-2M also outperforms other strong baselines across classification benchmarks despite highly limited computational resources.
  • Subgroup analysis: LimiX-16M achieves leading performance across subgroups and remains comparatively stable as the proportion of categorical features increases, while most baselines degrade rapidly.Under some larger-training-sample subgroups, LimiX-16M is the only ICL-based model outperforming AutoGluon.
  • Regression: LimiX-16M always achieves the best performance on the reported regression benchmarks, leading in both normalized RMSE and R2.LimiX-2M has predictive performance comparable to AutoGluon and TabPFN-v2 while requiring substantially fewer computational resources and less runtime.
  • Missing Value Imputation: LimiX performs missing-value imputation on unseen datasets through in-context learning without additional training, unlike existing deep learning approaches described here.The capability follows from the masked modeling used during pretraining and targets the joint distribution P(X, Y).
  • Robustness: Under perturbations, LimiX maintains nearly unchanged normalized AUC with up to 90% uninformative features and avoids the outlier-related degradation observed for competing models.The evaluation reports superior robustness relative to TabICL, CatBoost, and TabPFN-v2 in classification and regression settings.

8 Scaling law

The study characterizes how LDM loss and downstream performance scale with dataset size and model parameters using power-law fits. Scaling generally improves outcomes, but returns diminish and differ across metrics and model capacities.

  • Motivation: Scaling laws for LDMs remain largely unexplored compared with those for large language models.The study addresses quantitative relationships among model size, dataset size, and downstream metrics.
  • Method: The analysis assumes power-law relationships and estimates their parameters with ordinary least squares across multiple resource scales.The evaluated metrics include classification Log-Loss, AUC, F1, and accuracy, plus regression RMSE-Loss and R2.
  • Dataset-size scaling: R2 > 0.99 fits describe loss scaling with dataset size across model configurations, with larger models achieving lower and more stable loss.Loss falls rapidly at smaller dataset scales before improving more gradually.
  • Parameter scaling: R2 > 0.9 fits hold for classification Log-Loss, AUC, and ACC, whereas F1 reaches R2 = 0.68 in the model-parameter analysis.Figure 27 reports real and fitted trends for Loss, AUC, F1, ACC, and R2.
  • Parameter scaling: Loss decays sharply with model size, while AUC, ACC, and F1 improve gradually, indicating later gains in fine-grained discrimination.Regression RMSE decreases and R2 rises with model size, with strong power-law trends reported at R2 > 0.98.
  • Conclusions and insights: Increasing data and model capacity produces predictable improvements, but progressively larger resources are required for marginal performance gains.The findings frame dataset expansion and model scaling as a trade-off between predictive accuracy and computational efficiency.

9 Conclusion

The conclusion presents LimiX-16M and LimiX-2M as unified LDMs for multiple structured-data tasks through conditional queries to a shared joint-distribution model.

  • Unified modeling: LimiX models express classification, regression, imputation, generation, and sample selection as conditional queries to one model.They model structured-data inputs as a joint distribution over variables and missingness.
  • Method: LimiX combines masked joint-distribution pretraining with episodic, context-conditional adaptation for versatile downstream use.Its architecture captures dependencies across features and samples and models causal relations among variables.
  • Evaluation: The work evaluates LimiX across 11 large structured-data benchmarks spanning diverse dataset and feature regimes.The conclusion describes broad variation in sample size, feature dimensions, class number, feature types, missingness, and sample-to-feature ratios.

10 Contribution

The supplied contribution passage contains an author list for the paper.

  • Authorship: The passage identifies Gang Ren as a paper author.It lists the authors in sequence.
  • Authorship: The passage identifies Han Yu as a paper author.It lists Han Yu among the paper’s authors.
  • Authorship: The passage lists additional coauthors alongside Gang Ren and Han Yu.The complete author list contains multiple contributors.

A.1 Details of Datasets with Distribution Shifts

This appendix documents distribution-shift datasets and supporting theoretical details for recovering structured-data conditionals. It also records benchmark metadata and baseline-search settings.

  • Datasets: The TableShift benchmark contains 10 public datasets spanning voting, unemployment, sepsis, insurance, income, hospital readmission, food stamps, diabetes, college, and education-assistance domains.The datasets are adopted from TableShift and include Voting4, Unemployment, Sepsis, Public Health Insurance, Income, Hospital Readmission, Food Stamps, Diabetes, College Scorecard, and ASSISTments.
  • Dataset statistics: Table 29 records feature counts, training samples, in-distribution and out-of-distribution test samples, shift types, and domain counts.These fields describe the benchmark’s dataset structure and distribution-shift splits.
  • Baselines: Table 30 gives the Optuna hyperparameter search space for tree-based baselines, with classification and regression differing only in optimization target.All other hyperparameter settings remain identical across the two task types.
  • Theoretical details: Under a positive-density assumption, the conditional distribution p(Xte|Xct) corresponds one-to-one with an appropriate family of masked conditionals.The proof derives recoverability through successive conditional relationships and induction.
  • Theoretical limitations: A single conditional p(Xte_j|Xte_-j, Xct) is insufficient to recover the full conditional distribution p(Xte|Xct).The appendix gives a two-feature Gaussian example showing that one conditional does not uniquely determine the full parameter set.
  • Model notation: The learned family qθ(Xte|Xct) induces a joint form by multiplying the conditional with p(Xct).The notation is extended as qθ(Xct, Xte) := qθ(Xte|Xct)p(Xct).

B.2.1 Sample Efficiency

The analysis shows that, under regularity and realizability conditions, empirical minimizers are locally unique and asymptotically normal. Increasing the number of masked variables improves sample efficiency through a non-increasing asymptotic covariance bound.

  • Sample Efficiency: √n(θ̂k,n − θ∗) converges in distribution to N(0, Γk), with Γk independent of n.The asymptotic normality result holds locally around θ∗ under the positive-definite Hessian condition.
  • Sample Efficiency: Under the stated assumptions, a sufficiently large sample yields a unique empirical-loss minimizer near the population minimizer.The result requires realizability, differentiability, gradient bounds, and a uniform law of large numbers.
  • Sample Efficiency: The proof establishes θ∗ as the minimizer of Lk(θ) for every k ∈[d] and extends the local asymptotic argument across masking levels.The derivation combines the minimizer characterization with the theorem establishing local uniqueness and asymptotic behavior.

B.2.2 Generalization for Joint Distribution Learning

The generalization analysis bounds conditional-distribution error when masked joint-distribution learning is trained under stated regularity assumptions. It further shows that increasing the number of masked cells reduces the relevant upper bound through monotonicity of the tensorization constant.

  • Generalization for Joint Distribution Learning: Approximate tensorization of entropy measures the complexity of the joint distribution through the mixing behavior of Gibbs-sampling dynamics.It evaluates how readily samples can be generated using local conditional distributions.
  • Generalization for Joint Distribution Learning: The analysis assumes support compatibility, a positive lower bound β for model conditionals, and controlled parameter-space covering and likelihood continuity.These conditions are used to obtain uniform high-probability bounds over θ and masking levels.
  • Generalization for Joint Distribution Learning: Under the theorem’s conditions, Ck+1(qθ) ≤ Ck(qθ) for masking sets Πk.The result extends approximate tensorization from Π1 to every 1 ≤k ≤d.
  • Generalization for Joint Distribution Learning: The generalization-error upper bound is reduced when the number of masked cells increases.The bound is scaled by Ck(qθ), whose monotonicity supports the reduction.
  • Generalization for Joint Distribution Learning: The resulting bounds hold simultaneously over 1 ≤k ≤d with probability at least 1 −dδ.The simultaneous statement follows by applying the per-k result and a union bound.
Loading 2509.03505v2…