PaperlayerView exact revision on arXiv

Source-linked AI summary

TabNSM: Neural Sparse Mixer for Tabular Regression

Ali Eslamian, Qiang Cheng

arXiv:2608.18026v1cs.LGcs.CE

TL;DR

High-dimensional tabular regression must identify instance-specific signal amid noisy and redundant features. TabNSM combines adaptive sparse interaction modeling with structured supervision and difficulty-aware sampling, achieving the best performance on seven of nine benchmarks and second on one.

  • Problem

    High-dimensional tabular regression contains instance-specific informative features alongside large amounts of noisy, redundant, or weakly informative features.

  • Method

    TabNSM combines adaptive sparse foreground discovery and local interaction encoding with Feature-Token Mixing, GridLoss, and RISE reweighting.

  • Results

    TabNSM achieves the best performance on seven of nine regression benchmarks and ranks second on one, outperforming GBDT baselines on most tasks.

  • Takeaways & Limitations

    The results support selective interaction modeling, structured regression supervision, and difficulty-aware sampling as complementary components for scalable high-dimensional tabular regression.

  • Takeaways & Limitations

    ASIM’s signal-to-noise benefit is conditional on its sparse support covering informative features while excluding most background dimensions.

Abstract

from arXiv · show

Large-scale, high-dimensional tabular regression remains challenging: tree-based models are robust but lack end-to-end representation learning, while deep models enable flexible feature learning but often incur costly interaction modeling and sensitivity to noisy or redundant features. We propose TabNSM, a scalable regression framework that extends our earlier sparse-attention and mixer architectures. At its core, the Adaptive Sparse Interaction Module (ASIM) integrates foreground feature discovery, sparse local interaction encoding, and Feature-Token Mixing, providing near-linear complexity under fixed sparse configurations. For regression, TabNSM introduces three complementary components: a Multi-Stage Regression Head for progressive prediction refinement; GridLoss, an ordinal-aware soft-binning objective that incorporates target structure into representation learning; and RISE (Reweighted Instance Sampling by Error), a difficulty-aware sampling strategy based on loss-quantile bins. Across nine real-world regression benchmarks, TabNSM delivers strong predictive performance and practical scalability, with particularly consistent gains on high-dimensional and heterogeneous datasets. These results demonstrate that selective interaction modeling, structured regression supervision, and difficulty-aware sampling provide an effective and scalable approach to deep tabular regression.

1 Introduction

TabNSM addresses high-dimensional tabular regression by combining instance-adaptive sparse interaction modeling with regression-specific prediction and training components. Across nine real-world benchmarks, it demonstrates strong predictive performance and practical scalability.

  • Motivation: High-dimensional tabular regression combines heterogeneous features with sparse, sample-dependent predictive signals that can shift across instances.The foreground-background view distinguishes decision-relevant features from a larger noisy background, with no globally fixed feature subset.
  • Motivation: Tree models provide sparse feature selection, whereas deep models trade expressive interaction modeling against foreground isolation and computational scalability.Dense self-attention incurs quadratic feature-dimension cost, while sequential state-space scanning does not directly provide sparse, instance-adaptive selection.
  • Method: TabNSM’s Adaptive Sparse Interaction Module combines instance-adaptive foreground discovery, sparse local interaction encoding, and Feature-Token Mixing.ASIM adapts components from the earlier TabNSA sparse-attention architecture and TabMixer mixing block within a high-dimensional regression framework.
  • Method: The Multi-Stage Regression Head progressively refines predictions through residual coupling and feature fusion, while GridLoss provides ordinal-aware soft-binning supervision.GridLoss aligns predictions and continuous targets in an ordinal grid space alongside standard pointwise regression losses.
  • Training and Evaluation: RISE periodically reweights instances using loss-quantile bins to emphasize difficult or underfit regions under non-uniform error distributions.The framework is evaluated on nine real-world regression benchmarks, with ablations validating the complementary roles of its components.

2 Related Work

Prior tabular regression methods trade off predictive flexibility, feature selectivity, scalability, and structured supervision. TabNSM is positioned to combine differentiable representation learning with sparse, instance-adaptive interactions and regression-specific difficulty-aware objectives.

  • Deep Learning versus Tree Ensembles for Regression: GBDTs remain strong tabular regression baselines, but they are not naturally end-to-end differentiable for joint representation learning with high-dimensional learned features.The passage identifies XGBoost, LightGBM, and CatBoost as representative GBDTs.
  • Feature Interaction and Scalability: Transformer-based tabular models capture global feature interactions through dense self-attention, but their attention mechanisms can be sensitive to feature noise and scale poorly with feature dimension.TabPFN and TabPFNv2 also achieve strong generalization through in-context learning, while transformer designs can incur substantial computational and memory costs.
  • Feature Interaction and Scalability: SSMs improve scalability and parameter efficiency through linear-time feature mixing, but typically impose feature ordering and do not explicitly model sparse pairwise feature interactions.The passage specifically describes selective scan operations along the feature axis.
  • Handling Heterogeneous Data and Text Features: Heterogeneous tabular datasets increasingly combine textual embeddings with numerical and categorical fields, motivating semantic representations for text-rich regression benchmarks.Pretrained language models and Sentence-BERT embeddings are identified as approaches for semantic text representation.
  • Regression Objectives and Difficulty-Aware Learning: Regression research extends beyond MSE and MAE toward robust, distribution-aware objectives, while curriculum learning and hard-example mining remain less developed for regression.The passage mentions Huber and quantile losses, target discretization, and label distributions as related approaches.
  • Positioning of TabNSM: TabNSM is positioned to address these gaps by combining GBDT-like feature selectivity, differentiable representation learning, scalable mixing, and explicit sparse, instance-adaptive feature interactions.The positioning passage contrasts GBDTs, dense-attention models, and SSM-based models as having complementary limitations.

3 Methodology

TabNSM unifies heterogeneous tabular inputs through leakage-controlled numerical, categorical, and semantic text representations, then models interactions with ASIM and regression-specific components. Its methodology combines sparse scalable computation, structured GridLoss supervision, and difficulty-aware RISE sampling.

  • Input Representation: Numerical, categorical, and textual columns are unified into dense representations, with preprocessing parameters fitted only on the training split to prevent leakage.Numerical features are standardized; low-cardinality categoricals use integer encoding, while high-cardinality or free-text fields use frozen all-MiniLM-L6-v2 embeddings.
  • Adaptive Sparse Interaction Module: Each scalar feature is projected into a d_tok-dimensional token, enabling feature-wise interaction modeling across the D input dimensions.The tokenized representation is X_tok = Tok(x) ∈ R^(D×d_tok), with one token per feature.
  • Adaptive Sparse Interaction Module: ASIM combines instance-adaptive sparse attention with Feature-Token Mixing through learnable residual blending, enriching representations with selective and global feature dependencies.Sparse attention uses local and compressed supports, while FTM mixes embedding and feature dimensions; the output is aggregated back into feature space.
  • Scalability: Under fixed sparse settings, one ASIM layer has O(Dd_tokL) sparse-attention complexity and O(D(d_tok + L)) activation memory, both linear in D.Here L is the maximum number of attended keys per query feature, while d_tok, heads, sparse hyperparameters, and feature mini-batch size remain fixed.
  • Sparse Feature Selection: When sparse support covers informative features while excluding background dimensions, sparse aggregation improves signal-to-noise ratio, with noise contribution vanishing when m_b = s.This is a simplified conditional analysis based on informative-feature coverage and exclusion of most background noise.
  • Regression Supervision and Sampling: GridLoss provides a monotone soft index, bounded gradients by K/(2τ), and Lipschitz continuity, while RISE reweights empirical loss-quantile bins toward harder regression examples.RISE periodically refreshes per-sample errors and prioritizes underfit or difficult target regions using bounded bin-based weights.

4 Experiments and Results

Across nine real-world regression benchmarks, TabNSM achieves the lowest RMSE on seven and statistically significant aggregate gains over baselines. Ablations and scalability analyses attribute these results to adaptive sparse selection, global feature mixing, and near-linear parameter growth, while component checks support FTM, GridLoss, and RISE.

  • Main results: TabNSM achieves the lowest RMSE on seven of nine benchmarks, with statistically significant aggregate gains over baselines after correction (p ≤0.02).The evaluation compares TabNSM with 38 baselines across classical, tree-based, neural, transformer, retrieval-based, deep tabular, and foundation-model families.
  • Main results: 22% and 21% are the largest reported margins, respectively, on TO (0.021 vs. 0.027) and EVP (1.199 vs. 1.507).The TO comparison is against all competing models, whereas the EVP comparison is against FT-Transformer.
  • Result interpretation: TabNSM is most effective on high-dimensional, interaction-rich datasets because ASIM selects informative feature subsets while FTM propagates global context without dense quadratic cost.The interpretation specifically highlights EVP with 392 features, ChD with 405 features, and AQ’s improvement over DCN-v2’s explicit cross-network mixing.
  • Ablation analysis: ASIM achieves the lowest RMSE in ablation comparisons, while full attention is unstable in high-dimensional settings and Mamba, despite improved efficiency, underperforms ASIM.The downstream head and training protocol are held fixed across the ablated variants.
  • Scalability analysis: Parameter count grows approximately linearly with input feature dimension when the retained feature fraction varies under fixed sparse-attention configurations.This behavior supports ASIM’s intended near-linear scaling across datasets.
  • Additional component checks: Removing FTM degrades performance, GridLoss matches or improves standard pointwise losses under heterogeneous errors, and RISE reduces validation variance by prioritizing difficult samples.FTM’s effects are reported on CP, TO, SA, and EVP, especially EVP.

5 Conclusion and Discussion … B.1 ASIM Complexity

TabNSM combines sparse interaction modeling, structured GridLoss supervision, and difficulty-aware reweighting for scalable tabular regression. Its ASIM module retains adaptive sparsity with linear-in-feature computation and activation memory under fixed configurations.

  • 5 Conclusion and Discussion: TabNSM extends TabNSA and TabMixer with an adapted sparse interaction module, Multi-Stage Regression Head, GridLoss, and RISE across nine regression benchmarks.The framework is explicitly regression-focused and combines progressive prediction refinement, ordinal grid supervision, and difficulty-aware reweighting.
  • 5 Conclusion and Discussion: ASIM replaces dense quadratic interaction with sample-specific foreground selection, achieving near-linear complexity while preserving adaptive sparsity.This summarizes the stated motivation and design of ASIM in the regression framework.
  • A Theoretical Intuition of GridLoss: GridLoss provides a differentiable coordinate on an ordered target grid by smoothly concentrating weights around nearby anchors.For finite temperature, its derivative is controlled by temperature and grid resolution, yielding bounded gradients.
  • A Theoretical Intuition of GridLoss: As target values increase, GridLoss shifts mass toward higher-index anchors, making its grid representation nondecreasing and ordinal-aware.The mapping is described as a smooth approximation to hard binning on an ordered grid.
  • A Theoretical Intuition of GridLoss: GridLoss addresses limitations of pointwise regression losses by explicitly encoding ordinal placement on the target scale.The passage contrasts this with residual-focused behavior in MSE, MAE, Huber, log-cosh, MAPE, and quantile loss.
  • B Proofs of Theoretical Results: Proposition B.1 analyzes ASIM time and memory complexity using feature count D, token dimension dtok, and sparse attention width L.The proposition assumes each query feature attends to at most L feature positions under a fixed sparse configuration.
  • B.1 ASIM Complexity: O(D(dtok + L)) is the sparse interaction complexity, while overall ASIM computation remains linear in D under fixed dimensions, batch size, and sparse hyperparameters.The stated fixed quantities include dtok, H, dh, Bf, and sparse hyperparameters (s, b, r, m).
  • B.1 ASIM Complexity: O(D(dtok + L)) is the sparse attention activation memory when sparse weights or indices are materialized.This combines O(DL) storage for the sparse pattern with O(Ddtok) storage for token representations.

B.2 Conditional SNR Improvement under Sparse Feature Selection … C.1 Full Benchmark Results

The theoretical analyses explain when sparse feature selection improves signal-to-noise ratio, establish GridLoss’s monotonicity, stability, and MAE consistency, and interpret RISE as reweighted empirical-risk optimization. Full benchmark results then position TabNSM against broad classical, tree-based, neural, transformer, retrieval-based, deep tabular, and foundation-model baselines.

  • B.2 Conditional SNR Improvement under Sparse Feature Selection: Sparse aggregation improves SNR when its support covers informative features while excluding most background dimensions.The result assumes S ⊆ T and shows the benefit relative to dense aggregation when mb > s.
  • B.2 Conditional SNR Improvement under Sparse Feature Selection: When mb = s, sparse aggregation contains no background noise, so its noise contribution vanishes under the simplified model.With fixed informative-feature count and tight sparse support, sparse SNR is dimension-independent while dense aggregation suffers signal dilution as D grows.
  • B.2 Conditional SNR Improvement under Sparse Feature Selection: Structured sparse selection motivates ASIM’s robustness by restricting interactions to supports that can retain informative features while excluding irrelevant dimensions.The analysis contrasts this with dense attention and with Mamba’s linear-time propagation without explicit sparse feature selection.
  • B.3 Properties of GridLoss: GridLoss’s soft index is nondecreasing in the scalar input, and every subgradient of its per-sample loss satisfies |g| ≤ K/(2τ).These properties provide ordinal monotonicity and bounded gradients under the stated grid and temperature definitions.
  • B.4 Asymptotic Consistency of GridLoss: Normalized GridLoss converges to MAE in the high-resolution, low-temperature limit.Finer grids improve target-scale resolution, while finite temperature smooths hard binning and yields bounded gradients.
  • B.5 Objective Interpretation: RISE follows stochastic gradients of reweighted empirical risks that emphasize higher-loss quantile bins when its bin-to-weight map is increasing.The sampling weights are recomputed at refresh steps and treated as fixed between refreshes, without inverse-probability correction.
  • C.1 Full Benchmark Results: Table 2 reports per-model RMSE across all nine benchmarks and compares TabNSM with a broad range of classical, ensemble, neural, transformer, retrieval-based, deep tabular, and foundation-model baselines.The compared families include classical baselines, tree-based ensembles, MLP-style architectures, transformer-based models, retrieval-based methods, other deep tabular architectures, and tabular foundation models.

C.2 Statistical Significance of Results

TabNSM’s improvements are statistically significant overall and against every baseline across nine benchmarks, although conservative rank-based procedures and missing-value imputation limit some pairwise conclusions.

  • Friedman Test: The Friedman test rejects equal model performance across five benchmarks for 37 complete models, with χ2(36) = 79.3 and p = 4.25 × …Ranks use 1 = lowest RMSE = best; SAINT and DNNR were excluded after divergence or memory failures on three benchmarks.
  • Nemenyi Post-Hoc Test: The Nemenyi post-hoc test is conservative with 37 models and five benchmarks, producing CD = 26.8 rank units and leaving most pairwise comparisons inconclusive.This limitation arises because the number of models greatly exceeds the number of benchmarks.
  • Pairwise Wilcoxon Signed-Rank Test: After Benjamini–Hochberg correction, TabNSM has a significantly lower rank than every baseline across all nine benchmarks, with all corrected p ≤0.02.The analysis uses 38 one-sided Wilcoxon signed-rank tests at α = 0.05 and imputes missing values as worst rank.
  • Pairwise Wilcoxon Signed-Rank Test: Worst-rank imputation is conservative for TabNSM but liberal for baselines with many incomplete runs, so pairwise results should be interpreted as aggregate trends.TabNSM has no missing runs, whereas some baselines have many incomplete entries.

D Brief Summary of Baseline Model Setup

Baseline models used a standardized preprocessing, data-splitting, tuning, and evaluation pipeline. Models lacking regression code or adequate tuning were excluded from Table 2.

  • Data preparation: ARFF data were loaded into pandas, categorical strings decoded and numerically converted where possible, missing values filled with zeros, and targets removed before training.Data were split into 80% training-plus-validation and 20% test, then the former into 90% training and 10% validation.
  • Evaluation and tuning: Test performance was evaluated using RMSE as the primary metric alongside MSE, MAE, and R2, with Optuna conducting 20-trial 3-fold cross-validation while maximizing negative RMSE.The best configuration trained a final model on the training set, using validation data when supported, before held-out test evaluation.
  • Reporting limitations: Models without provided regression-task code or without tuning on most large-dataset benchmarks were excluded from Table 2.Untuned models often showed non-convergence or outlier performance.

E Instance-adaptive feature selectivity

TabNSM exhibits instance-adaptive feature selectivity in test-set attribution visualizations. Its attributions reveal both globally influential predictors and input-dependent sparse feature emphasis, consistent with ASIM’s design.

  • Attribution visualization: Test-set feature attribution heatmaps visualize per-instance feature selectivity using gradient–embedding attribution normalized to [0, 1] within each instance.Rows represent test instances and columns represent input features; brighter values indicate stronger influence on the regression output.
  • Attribution patterns: Some features form consistently bright columns across many instances, suggesting globally influential predictors.This pattern appears alongside more instance-specific attribution structures in the heatmap.
  • Attribution patterns: Many instances show distinct sparse attribution patterns, indicating dynamically selected feature subsets based on input context.The behavior is consistent with ASIM encouraging instance-adaptive sparse feature interaction.

F Dataset Descriptions · F.1 Dataset Links · G Seed Variation

The paper evaluates TabNSM on diverse regression datasets spanning molecular, robotic, environmental, public-health, geographic, real-estate, and neuroimaging tasks. Dataset access information and 20-seed experiments further document reproducibility and performance stability.

  • F Dataset Descriptions: The benchmark suite covers diverse prediction tasks and dataset domains used to evaluate TabNSM.The datasets include molecular descriptors, robotic-arm measurements, vehicle registrations, air quality, chronic disease, crime, real-estate, and Alzheimer’s neuroimaging data.
  • F Dataset Descriptions: The Topo dataset uses 1,143 Adriana-computed molecular descriptors to predict oz267, while Sarcos predicts joint torque V22 from robotic-arm motion measurements.Sarcos contains joint positions, velocities, and accelerations from a 7-degree-of-freedom robotic arm and is used for inverse-dynamics regression.
  • F Dataset Descriptions: The Electric Vehicle, Air Quality, and U.S. Chronic Disease datasets predict electric driving range, quantitative air-quality measurements, and numeric chronic-disease metrics, respectively.Their inputs include vehicle and registration attributes, pollutant and environmental-health indicators, and standardized public-health indicators.
  • F Dataset Descriptions: The Crime and Real Estate Sales datasets predict incident geographic areas and sales ratios from temporal, categorical, spatial, property, and transaction features.The Crime records come from Los Angeles since 2020, whereas the real-estate transactions were reported by Connecticut municipalities.
  • F Dataset Descriptions: The SCAN benchmark combines neuroimaging, fluid biomarkers, and demographic variables to estimate Mini-Mental State Examination (MMSE) scores.It uses controlled-access multimodal data obtained through the NACC data-request process under the NACC Data Use Agreement.
  • F.1 Dataset Links: Eight benchmark datasets are publicly available, while NACC/SCAN data require access through NACC’s data-request process under the NACC Data Use Agreement.Table 4 provides the public source or access page for each dataset.
  • G Seed Variation: 20 random-seed repetitions show stable TabNSM test RMSE distributions, with relatively small interquartile ranges and limited variation across runs.Figure 6 reports the distributions using boxplots to assess sensitivity to initialization and training stochasticity.

H Efficiency and Memory Footprint … K.1 Ablation of the FTM Pathway

TabNSM combines sparse, adaptive feature interaction and feature-token mixing for sub-quadratic computation, with favorable runtime and memory behavior. Ablation results show that removing FTM worsens RMSE on several datasets, especially EVP.

  • H Efficiency and Memory Footprint: TabNSM maintains low runtime across evaluated settings, whereas CatBoost runtime increases with dataset size and feature dimensionality.The comparison records training time until each method reaches a predefined validation-loss threshold.
  • H Efficiency and Memory Footprint: TabNSM shows moderate GPU memory usage and lower CPU memory increase than CatBoost on evaluated large datasets.GPU memory is reported only for TabNSM because CatBoost runs on CPU.
  • I Sparse Feature-wise Attention: Full Formulation: ASIM restricts each feature token to a compact, instance-adaptive subset covering local, global, and instance-specific interaction patterns.The sparse operator avoids computing all D^2 pairwise feature interactions.
  • I Sparse Feature-wise Attention: Full Formulation: Compressed global summaries reduce global context to O(D/r) keys, while top-m block retrieval recovers fine-grained interactions within a fixed budget of mb keys per query.The method combines block compression with selective full-resolution retrieval.
  • I Sparse Feature-wise Attention: Full Formulation: The total attended context is N = 2s + ⌊(D −b)/r⌋+ mb ≪D, yielding complexity O(D dtok N) for fixed sparse hyperparameters.This is sub-quadratic in D under fixed sparse configurations.
  • J Feature-Token Mixing (FTM): Full Formulation: FTM uses parallel embedding-wise and feature-wise mixing branches, with feature tokens processed in disjoint mini-batches of size Bf to avoid quadratic cost in D.The embedding-wise branch captures intra-token structure, while the feature-wise branch mixes along the token axis.
  • J Feature-Token Mixing (FTM): Full Formulation: FTM combines its branches through element-wise multiplication, Layer Normalization, SiLU activation, and a residual connection.The residual connection supports stable gradient flow while enriching representations with globally mixed context.
  • K.1 Ablation of the FTM Pathway: Ablating FTM degrades RMSE on CP, TO, SA, and EVP, including 570.09 vs. 1.199 on EVP.The ablation keeps the sparse attention backbone and training protocol fixed, indicating that FTM contributes to performance while sparse selection remains primary.

K.2 Ablation on Loss Function … K.7 Effect of the Blending Coefficient α

The ablations show that GridLoss and Huber are complementary, RISE improves validation stability, and TabNSM remains robust across feature dimensionality and sample size. GridLoss sensitivity depends on target-distribution structure, while intermediate blending coefficients perform best and fixed defaults are generally stable.

  • K.2 Ablation on Loss Function: GridLoss achieves competitive or superior RMSE across datasets, with especially pronounced gains under heterogeneous error distributions.It is less sensitive to scale variation and large residuals than MAE and LogCosh while remaining competitive with Huber.
  • K.2 Ablation on Loss Function: Blending Huber with GridLoss yields stable optimization and strong predictive performance, particularly in high-dimensional regression settings.The combination joins Huber’s local robustness with GridLoss’s structure-aware ordinal supervision, and performance shows limited sensitivity to the fixed α choice.
  • K.3 Ablation on effect of RISE: Disabling RISE increases mean validation error and run-to-run variance, whereas enabling it produces more consistent validation performance.RISE adds sampling overhead, but the reported cost is modest relative to the observed validation gains.
  • K.4 Sensitivity to Feature Dimensionality and Sample Size: TabNSM maintains stable MSE as feature dimensionality increases and remains competitive when training data are reduced, unlike baselines with higher and more non-monotonic error.The analysis varies retained feature fractions and training-sample fractions while keeping other settings aligned with the main protocol.
  • K.5 Sensitivity to GridLoss Hyperparameters: GridLoss performance varies smoothly across multiple orders of magnitude of τ, indicating limited sensitivity within a broad range.The combined objective is L = αLHuber + (1 −α)LGrid, and the analysis evaluates τ across CP, TO, and SCAN-2.
  • K.5 Sensitivity to GridLoss Hyperparameters: The main experiments use fixed default values for τ and α, while the number of grid bins is tuned from dataset-specific target statistics.The effects of τ and α are analyzed separately below.
  • K.6 Sensitivity of τ to Target Distribution Properties: The optimal τ is primarily governed by bulk-target concentration, quantified by IQR/Std, rather than tail heaviness alone.CP favors increasing τ, TO favors small τ, and SCAN-2 has a degenerate bulk with highly non-monotonic response.
  • K.7 Effect of the Blending Coefficient α: Intermediate α values consistently outperform using either loss alone, with optimal performance observed for α ∈[0.4, 0.6].The result indicates complementarity between Huber robustness and grid-based structural supervision.

K.8 Component-wise Ablation of Sparse Attention Parameters

The study varies one sparse-attention hyperparameter at a time and evaluates average validation RMSE across four datasets. Among the reported settings, a compress block size of 10 performs best, with degradation at both smaller and larger values.

  • Compress block size: A compress block size of 10 achieves the best average validation RMSE.Performance degrades when the compress block size is either smaller or larger than 10.
  • Ablation protocol: The ablation varies number_of_selected_blocks, selection_block_size, compress_block_size, and sliding_window_size individually while holding other parameters fixed.Performance is measured using average validation RMSE across four validation sets, where lower RMSE is better.
Loading 2608.18026v1…