PaperlayerView exact revision on arXiv

Source-linked AI summary

LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence

Xingxuan Zhang, Gang Ren, Hao Yuan, Hao Zou, Hongze Tan, Hui Wang, Jianhao Song, Jiansheng Li, Jiayao Zhang, Jinghan Zhang, Kaifang Li, Lang Mo, Li Mao, Mingchao Hao, Nuo Xu, Rui Ding, Ruiji Zhang, Shuyang Li, Siyu Mei, Tianyang Zhang, Weiyang Mu, Yancheng Dong, Yongxian Wei, Yuan Xue, Yuanrui Wang, Yue He, Zijia Yang, Ziyun Li, Dongzhe Li, Fuqiang Wang, Jiandong Liu, Jiawei Chen, Jiaxin Du, Kaijie Cheng, Kehan Li, Lei Sun, Linjun Zhou, Ningbo Dai, Qi Wang, Renzhe Xu, Shaoxing Du, Shumeng Yang, Wang Lu, Wenjing Chu, Xiannan Huang, Xiaoyu Lin, Xing Ai, Xinyan Han, Xuanyue Li, Xuanyue Su, Xukun Zhang, Yan Lu, Yaxin Zhang, Yi Qin, Yifei Huang, Yihan Xu, Yongle Lv, Yuanyuan Jiang, Yushan Han, Peng Cui

arXiv:2609.17488v1cs.AI

TL;DR

Structured-data models often require separate training and model selection for each dataset, limiting knowledge reuse across tasks. LimiX-2 addresses this with Contextual Mechanism Networks and Context-Conditional Masked Modeling over SCM-generated synthetic data. It outperforms current tabular foundation and dataset-specific models across TabArena, TALENT, and BCCO, while also supporting causal skeleton recovery through causally informative feature attention.

  • Problem

    Task-specific tabular methods require separate training and model selection for each dataset, motivating models that transfer across structured-data tasks.

  • Method

    LimiX-2 uses CMNs and CCMM to learn joint variable dependencies from SCM-generated synthetic datasets through target prediction and feature reconstruction.

  • Results

    LimiX-2 outperforms current tabular foundation and dataset-specific models across TabArena, TALENT, and BCCO, and its feature attention supports accurate causal skeleton recovery.

  • Takeaways & Limitations

    CMNs provide a unified structured-data framework spanning supervised prediction, missing-value imputation, conditional reasoning, and causal skeleton recovery.

  • Takeaways & Limitations

    Extrapolated 2B-model results are forecasts rather than measured performance, and benchmark Elo depends on the comparison pool.

Abstract

from arXiv · show

We introduce LimiX-2, a new model in the LimiX family, developed through model and data scaling guided by our previously established scaling laws. LimiX-2 adopts the Contextual Mechanism Networks (CMNs) paradigm and is pretrained with Context-Conditional Masked Modeling (CCMM). CMNs shifts the organizing principle of in-context learning from target-centric prediction to mechanism-oriented joint modeling. Rather than centering the network on the $p(y \mid x, D_{\mathrm{context}})$ objective of conventional tabular PFNs, it is designed around learning $p(x, y \mid D_{\mathrm{context}})$, a context-dependent representation of the joint structure underlying data generation. Pretraining uses synthetic datasets generated by structural causal models (SCMs) spanning diverse graph structures, functional mechanisms, and observation processes. Evaluations on TabArena, TALENT, and BCCO show that LimiX-2 outperforms current dataset-specific models and tabular foundation models. Beyond predictive performance, the CMN paradigm also promotes causal awareness in LimiX-2: its feature attention encodes direct causal relationships, enabling accurate causal skeleton recovery.

1 Introduction

LimiX-2 addresses the limited cross-task reuse of conventional tabular models by introducing CMNs, which learn joint dependency structure rather than only target prediction. It combines CCMM pretraining with SCM-generated synthetic data and outperforms current models across broad tabular benchmarks and causal skeleton recovery.

  • LDMs are defined as pretrained structured-data models that cover diverse tasks, use a unified task-independent paradigm, and perform new tasks without model updates.Classification and regression are canonical instances, but the framework is broader than those tasks.
  • CMNs shift structured-data modeling from target-centric p(y | x, D_context) prediction to joint dependency modeling of p(x, y | D_context).The paradigm uses multiple conditional prediction tasks over the same dataset to capture dependencies shared across variables.
  • CCMM integrates target prediction and feature reconstruction under varied observation patterns, making inter-variable inference an explicit pretraining objective.This provides a unified basis for supervised prediction, missing-value imputation, and broader conditional reasoning.
  • LimiX-2 is pretrained exclusively on synthetic SCM-generated datasets spanning broader graph structures, functional mechanisms, and observation processes than its predecessor.The model refines a Transformer architecture while retaining cell-level representations.
  • LimiX-2 outperforms current tabular foundation and dataset-specific models across TabArena, TALENT, and BCCO, despite being four times smaller than TabFM.The evaluations span varied sample sizes, feature dimensionalities, class counts, feature types, missingness, and sample-to-feature ratios.
  • LimiX-2 outperforms other tabular foundation, tree-based importance, and dedicated causal discovery methods on causal skeleton recovery.The result indicates that its feature attention encodes causal structural information.

2 Architecture

LimiX-2 preserves cell-level representations while scaling a dual-axis Transformer with separate feature and task pathways. Its backbone combines sample-axis attention, independent gated FFNs, and asymmetric feature-axis attention for conditional prediction.

  • 2.1 Embedding of Tabular Data: Cell-level representations preserve within-row feature relations and enable dataset-level statistics to be derived from context cells.Unlike row-level compression, each cell retains its own representation for fine-grained reasoning across columns and samples.
  • 2.1 Embedding of Tabular Data: LimiX-2 expands cell embeddings to d = 256, uses a shared missing-cell vector, and encodes targets into K = 4 task-embedding slots.Numerical and categorical targets use separate encoders, while query targets receive a learnable MASK embedding.
  • 2.2 Discriminative Feature Encoding: Low-rank discriminative feature encoding adds compact column identity codes, distinguishing columns without encoding their sequential positions.The codes are mapped into the embedding space and added to feature representations; joint column permutation should not create order dependence.
  • 2.3 Model Backbone Architecture: The dual-axis backbone propagates context across samples, models within-row variable relations, and restricts target representations to attend only to features.Query rows attend only to context rows, preventing predictions from depending on other query samples in the batch.
  • 2.3 Model Backbone Architecture: Each of the M = 24 blocks applies independent sample-axis attention, separate feature and target SwiGLUs, asymmetric feature-axis attention, and residual connections.The architecture is illustrated in Figure 2, whose final task embeddings feed classification and regression heads while shallow feature embeddings support masked-feature reconstruction.

3 Pretraining

LimiX-2 uses CCMM to learn joint variable dependencies through target prediction and masked-feature reconstruction under varied observation patterns. Separate feature and task pathways share a conditional interface across prediction and imputation tasks.

  • 3.1 Context-Conditional Masked Modeling: CCMM extends PFN-style target prediction across variables and conditioning sets by combining target supervision with masked-feature reconstruction.Its objective targets the joint dependency structure rather than only p(y | x, Dcontext).
  • 3.1 Context-Conditional Masked Modeling: Each episode splits rows into disjoint context and query sets, with query targets predicted from observed query features, masked-column locations, and context evidence.Masked features are reconstructed for each query row while the same conditioning information feeds the target heads.
  • 3.1 Context-Conditional Masked Modeling: Query rows attend only to context rows, making predictions invariant to query-batch composition for fixed inputs and context.Context representations are computed without access to queries, preventing direct and context-mediated information exchange among query rows.
  • 3.1 Context-Conditional Masked Modeling: The feature pathway models inter-variable dependencies through cell-level representations, while K = 4 task embeddings aggregate information for prediction without task-specific updates.Asymmetric attention directs feature information into the task readout as the architecture is scaled for wider tables and longer contexts.
  • 3.2 Masking Schemes: Three masking schemes—individual entries, selected query columns, and entry blocks—broaden the granularity of conditional tasks seen during pretraining.These schemes range from isolated-value recovery to target-column prediction and grouped missing-entry reconstruction.

4 Pretraining Data Generation

LimiX-2 pretraining data are synthesized with structural causal models whose graph, mechanism, observation, and task properties are broadly varied. A five-stage pipeline converts sampled causal systems into diverse prediction datasets.

  • 4.1 Overview: The five-stage generation pipeline samples hyperparameters, generates a DAG, propagates an SCM, samples data, and adapts tasks.LimiX-2 expands the pipeline inherited from LimiX to increase structural and statistical diversity.
  • 4.1 Hyperparameter Sampling: Hyperparameter sampling varies sample size, continuous and categorical feature dimensions, task type, and the context-query split.An evaluation position randomly divides each generated dataset into context and query rows.
  • 4.2 DAG Generation: DAGs are assembled from recursively expanded causal motifs representing directed structures such as chains, confounding, and colliders.Topology-constrained transformations further enrich local graph structure through operations including edge redirection and path replacement.
  • 4.3 Functional Mechanisms of SCM: SCM propagation samples root-node values and computes remaining variables in topological order using edge functions, aggregation functions, and stochastic noise.The engine supports MLP, CNN, tree, linear, kernel, piecewise, periodic, and multiplicative mechanisms, including compositions.
  • 4.4 Data Sampling and Task Adaptation: Task adaptation selects observable variables as features and targets, filters candidates multi-objectively, and applies stochastic observation transformations.Selection broadens structural and statistical task coverage, while transformations include scaling, nonlinear, logarithmic, exponential, and composed operators.

5 Evaluation

LimiX-2 achieves leading aggregate predictive performance across TabArena, TALENT, and BCCO, spanning classification and regression tasks. Its pairwise wins and subgroup analyses indicate broad relative strength across competing methods and dataset regimes.

  • TabArena: LimiX-2 ranks first across all four TabArena metrics, reaching Elo 1935 and exceeding TabFM+ by 117.4 points before rounding.It also records lower improvability, better average rank, and approximately 3.6 times as many aggregated wins as TabFM+.
  • TabArena: LimiX-2 ranks first on TabArena classification and regression subsets, with Elo scores of 1917 and 2206, respectively.Its average pairwise win rates are 94.5% for classification and 96.9% for regression.
  • TabArena: LimiX-2 maintains majority win rates against every displayed TabArena competitor, including 65.2% against TabFM+ and at least 95% against tuned-and-ensembled baselines.All off-diagonal win rates exceed 60%, while several foundation-model comparisons exceed 90%.
  • TALENT: On TALENT, LimiX-2 achieves the highest Elo across all five categories, reaching 1506 and reducing improvability by 26.4% relative to TabFM.It records 84.3 aggregated wins versus 50.1 for TabFM, approximately 1.7 times as many.
  • TALENT and BCCO: Across TALENT and BCCO, LimiX-2 retains strong classification and regression performance and remains competitive across dataset size and feature-dimensionality subgroups.It achieves the lowest reported average ranks in several classification and regression comparisons and maintains low fitted ranks across broad dataset regimes.
  • BCCO: On BCCO, LimiX-2 achieves the highest overall Elo of 1432, a 43.1% relative improvability reduction versus TabFM, and approximately three times as many wins.It exceeds AutoGluon, TabFM, and LimiX-16M by 56, 63, and 202 Elo points, respectively.

6 Scaling Law

LimiX-2 performance scales predictably with model size across TabArena, TALENT, and BCCO, with positive but task-dependent returns. The reported 2B-parameter results are extrapolations, while measured results show no saturation up to 406.2M parameters.

  • Scaling Results: Across all five evaluation series, performance follows a clear log-linear trend with model size, with R2 values of 0.9617–0.9808.Residual RMSE ranges from 3.84 to 9.03 Elo, and positive slopes vary substantially by task.
  • Scaling Results: 169 Elo separates LimiX-2 at 1935 from 1766 on TabArena as parameters increase from 12.5M to 406.2M.This corresponds to a roughly 32.5× model-size increase and a fitted slope of 34.68 Elo per doubling.
  • Scaling Results: 109 and 89 Elo gains occur on TALENT classification and regression, while BCCO classification and regression gain 56 and 147 Elo, respectively.These gains are measured across the same 12.5M-to-406.2M parameter range.
  • Scaling Results: Within the measured range, LimiX-2 compares favorably with structured-data foundation models at comparable scales, indicating strong empirical parameter efficiency under the reported protocols.Architecture, pretraining-distribution, and training-compute differences prevent attributing the performance gaps solely to model design.
  • Scaling Results: The extrapolated 2B-parameter results are forecasts rather than measurements because larger models may enter different optimization or data regimes and Elo depends on the comparison pool.The conclusion that capacity is a robust scaling axis is supported only within the studied regime.
  • Scaling Results: No clear performance saturation appears up to 406.2M parameters, motivating extension of LimiX-2 into the billion-parameter regime.The paper frames this as empirical motivation rather than measured evidence at billion-parameter scale.

7 Conclusions

The conclusion presents CMNs and CCMM as a broader alternative to PFN label prediction, instantiated in LimiX-2 for multiple structured-data tasks. It reports strong benchmark performance without task-specific training and supports causal inference in one forward pass.

  • 7 Conclusions: LimiX-2 performs classification, regression, missing-value imputation, and causal inference in a single forward pass without task-specific training.It is a transformer-based tabular foundation model with cell-level representations, pretrained on synthetic SCM-generated data.
  • 7 Conclusions: CMNs generalize PFN label prediction to arbitrary masked-column imputation, while CCMM supplies denser supervision through context-conditional masked modeling.The paradigm is intended to strengthen structured-data reasoning beyond a target-label objective.
  • 7 Conclusions: Evaluations on TabArena, TALENT, and BCCO show LimiX-2 outperforming a wide range of tabular foundation models and dataset-specific models.The conclusion attributes these results to the CMN-based design.

8 Contribution

This section lists the contributors to the paper.

  • 8 Contribution: The paper lists its contributors as Gang Ren, Hao Yuan, Hao Zou, Hongze Tan, Hui Wang, Jianhao Song, Jiansheng Li, Jiayao Zhang, Jinghan Zhang, Kaifang Li, Lang Mo, Li Mao, Mingchao Hao, Nuo Xu, Rui Ding, Ruiji Zhang, Shuyang Li, Siyu Mei, Tianyang Zhang, Weiyang Mu, Yancheng Dong, Yongxian Wei, Yuan Xue, Yuanrui Wang, Yue He, Zijia Yang, and Ziyun Li.
Loading 2609.17488v1…