PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Tabular Deep Regression: Objectives and the Limits of Global Early Stopping

Danil Rudenko, Georgii Bulgakov, Nikolay Kartashev, Ivan Rubachev, Daniil Krasylnikov, Artem Babenko

arXiv:2610.13909v1cs.LGcs.AI

TL;DR

Objective choice for tabular DNN regression remains difficult to assess because prior comparisons vary in datasets, backbones, and tuning protocols. The paper systematically compares objectives and examines uncertainty-dependent stopping, introducing AdaQR’s input-dependent smoothing. Quantile loss is consistently strong against tuned MSE, and AdaQR improves over quantile loss in several settings.

  • Problem

    Prior objective comparisons use different datasets, backbones, and tuning protocols, making reliable practical comparisons difficult.

    Source: §1 · p. 1

  • Method

    The study compares regression objectives across tabular datasets and backbones, examines uncertainty-dependent stopping, and introduces AdaQR’s jointly learned input-dependent smoothing.

    Source: §1 · pp. 1–2

  • Results

    Quantile loss is a consistently strong alternative to carefully tuned MSE, and AdaQR improves predictions in several settings.

    Source: §1 · p. 2, §6 · p. 11

  • Takeaways & Limitations

    Different noise regions can peak at different epochs, so global early stopping may compromise between uncertainty regions.

    Source: §1 · p. 2, §6 · p. 11

    The underlying training dynamics remain incompletely understood, and adaptive smoothing does not fully resolve differences in preferred stopping times across uncertainty regions.

    Source: §7 · p. 11

Abstract

from arXiv · show

While deep learning models for tabular data have recently received significant research attention, the choice of objective functions for regression tasks remains an understudied degree of freedom. Several lines of research offer alternatives to the standard Mean Squared Error (MSE), but differences in datasets, backbones, and evaluation protocols make their practical benefits difficult to compare. In this work, we address this gap by performing a systematic re-evaluation of regression objectives across a broad collection of established tabular datasets and multiple neural backbones. Our comparison reveals that many proposed objectives provide limited or inconsistent gains over a properly tuned MSE baseline, while quantile loss yields consistent aggregate improvements over MSE across the tested backbones. We then identify a limitation that remains even with this strong objective: regions with different levels of target noise can reach their best predictive performance at different training epochs. Consequently, a single global early-stopping checkpoint can impose a compromise between uncertainty regions, even when the objective can represent heteroskedastic target distributions. Motivated by this limitation, we introduce a heteroskedasticity-aware extension of quantile loss that uses input-dependent label smoothing as a potential remedy and improves over quantile loss on several backbones. Overall, our results provide a modern empirical picture of regression objectives for tabular deep learning and highlight how differences in optimal stopping times across uncertainty regions can limit the effectiveness of global early stopping. The code is available at https://github.com/yandex-research/tabular-regression.

1. Introduction

Objective choice for tabular DNN regression is difficult to assess from prior comparisons, which vary in datasets, backbones, and tuning protocols. The comparison finds quantile loss consistently strong against tuned MSE and introduces AdaQR to address uncertainty-dependent stopping conflicts.

1. Introduction

  • The study benchmarks regression objectives on 41 established tabular datasets with extensive hyperparameter tuning, focusing on conventional DNNs.Pretrained PFNs are outside its scope.Source: §1 · p. 1
  • Gaussian-smoothed binning objectives and quantile loss stand out on MLPs, with quantile loss showing more consistent gains across backbones.Binning represents targets as distributions over discrete bins, while quantile loss predicts conditional quantiles.Source: §1 · pp. 1–2
  • Regions with different uncertainty levels can peak at different epochs, making a single early-stopping checkpoint a compromise between them.The paper examines this conflict with controlled synthetic experiments and uncertainty-based checkpoint selection on real datasets.Source: §1 · p. 2
  • AdaQR adds jointly learned, input-dependent smoothing to quantile training as a potential remedy for stopping conflicts.The smoothing reduces quantile updates’ sensitivity to individual labels in high-uncertainty regions; AdaQR improves on standard quantile loss on several backbones.Source: §1 · p. 2
  • Across 41 datasets, most objectives do not consistently improve on carefully tuned MSE, while quantile loss gives more consistent gains across tested backbones.The benchmark covers 18 objectives on MLPs and evaluates selected objectives on three advanced tabular DNN backbones.Source: §1 · pp. 1–2

2. Related Work

Related work spans alternative regression objectives, early-stopping analyses, and methods for modeling input-dependent uncertainty. The paper situates its contribution as an uncertainty-based analysis of training trajectories and a jointly learned adaptive-smoothing objective.

2. Related Work

  • Regression-objective alternatives include robust losses, target discretization, conditional distribution modeling, and representation regularization.The paper compares representative methods under a common tabular evaluation protocol.Source: §2 · p. 2
  • Prior work studies differing optimal training stages across feature directions, network layers, token groups, and calibration or refinement errors [17] [18] [19].The paper extends uncertainty-based analysis of tabular models [11] to training trajectories and checkpoint selection.Source: §2 · pp. 2–3
  • Prior heteroskedastic objectives and smoothing methods use joint uncertainty estimation or preliminary local-scale fits [22] [23] [24] [25] [28].AdaQR combines an energy-distance objective with an input-dependent smoothing head learned jointly with the neural quantile predictor.Source: §2 · p. 3

3. Revisiting regression objectives

The study compares regression objectives under a common tabular evaluation, finding that gains over tuned MSE are uneven while quantile loss improves more consistently across tested backbones. Its results also compare objective families and show optimizer-sensitive outcomes on MLP.

3.1 Losses

  • The compared objectives span traditional pointwise losses, target-binning methods, quantile and likelihood-based objectives, and additional objectives with representation regularization.Binning methods discretize targets and predict expectations over bin centers; quantile regression predicts conditional quantiles with pinball loss.Source: §3.1 · p. 3

3.2.1 Implementation details

  • The evaluation uses 41 BeyondArena datasets, validation-tunes configurations, retrains across 15 seeds, and compares MLP with TabPack, TabM-mini, and RealMLP.Three computationally demanding datasets are excluded; the first predefined split is used for each retained dataset.Source: §3.2 · p. 4, §3.2.1 · p. 4, §3.2.3 · p. 5

3.2.2 Results

  • On MLP, Soft Binning, HL-Gaussian, and quantile loss improve on MSE in aggregate by ranks and relative RMSE, while most objectives are inconsistent.Figure 1 compares average ranks and relative RMSE improvement over MSE across 41 datasets.Source: Figure 1 · p. 4, §3.2.2 · p. 5
  • Under AdamW, HL-Gaussian and quantile loss improve over MSE in aggregate; Muon configurations show larger median gains against the common MSE+AdamW reference.The table reports median relative RMSE improvement over MSE+AdamW and win/tie/loss counts.Source: §3.2.2 · p. 5, Table 1 · p. 5

3.2.3 Advanced backbones

  • Across tested backbones, quantile loss achieves positive median improvements and more wins than losses on every backbone, unlike HL-Gaussian’s weaker transfer.HL-Gaussian has more losses than wins on TabPack despite a small positive median improvement, and its median gain is limited on RealMLP.Source: §3.2.3 · p. 5

4. Quantile regression under heteroskedasticity

Quantile regression is more robust across backbones than other strong alternatives to MSE, but it does not remove the conflict between optimal stopping times in regions with different noise levels. Uncertainty-based cohort checkpoints improve test RMSE over global stopping on the benchmark datasets, though the gains are small and cohort selection has practical costs.

4. Quantile regression under heteroskedasticity

  • Quantile loss achieves positive median improvements and more wins than losses against MSE on every tested backbone, making it more consistent than HL-Gaussian.On TabPack, HL-Gaussian has more losses than wins despite a small positive median improvement; quantile loss is therefore the focus of the subsequent analysis.Source: §4 · p. 5, Table 2 · p. 6

4.1 Optimal stopping epochs vary by uncertainty region

  • With heteroskedastic noise, earlier epochs perform relatively better on high-uncertainty points and later epochs on low-uncertainty points, so global stopping is a compromise.The pattern disappears in the homoskedastic control, supporting a connection to varying noise levels rather than the shared feature space or target function.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 Measuring the impact of epoch misalignment

  • Uncertainty-based checkpoint selection improves median RMSE over global selection by 0.18% for five cohorts and 0.114% for ten cohorts, with three wins, four ties, and no losses in both settings.These comparisons cover the seven largest benchmark datasets; increasing the cohort count to ten is associated with lower overall performance, consistent with more checkpoint-selection noise.Source: §4.2 · p. 8, Table 3 · p. 8
  • Uncertainty-based cohort splits improve performance on the heteroskedastic synthetic data but worsen it on the homoskedastic variant, indicating that gains reflect differences in noise-driven training dynamics.Per-cohort stopping requires multiple checkpoints and routing test points to cohorts, so the paper presents it as a measurement strategy rather than a practical solution.Source: §4.2 · p. 8

5. Adaptive quantile regression

AdaQR adds jointly learned, input-dependent smoothing to quantile training as a potential remedy for stopping-time conflicts across uncertainty regions. It improves median RMSE over MSE on all four tested backbones, with gains over quantile loss varying by backbone, and its learned uncertainty estimates correlate strongly with CatBoost estimates.

5. Adaptive quantile regression

  • AdaQR uses jointly learned, input-dependent smoothing in quantile training to reduce sensitivity to individual labels in high-uncertainty regions and potentially delay overfitting there.The loss replaces the point mass at the observed target with a Gaussian centered on that target, with an input-dependent learned variance and a regularization penalty.Source: §5 · pp. 8–9

5.1 AdaQR results

  • AdaQR achieves positive median RMSE improvements over MSE on all four backbones, with larger median gains than quantile loss on TabM and RealMLP and a small difference on TabPack.The main comparison covers 28 datasets with at least 1,000 training samples; benefits vary across backbones.Source: §5.1 · pp. 9–10, Table 4 · p. 9

5.2 Importance of Adaptive Smoothing

  • Replacing AdaQR's per-sample smoothing with a fixed global smoothing value substantially degrades performance on RealMLP, highlighting the importance of adaptivity.The ablation excludes the Tiny subset and otherwise follows the standard protocol.Source: §5.2 · p. 10, Table 5 · p. 10

5.3 Uncertainty Estimation

  • AdaQR's data-uncertainty estimates show strong correlations with CatBoost estimates on three real-world datasets.The comparison uses test splits and Spearman rank correlation across test inputs; ground-truth uncertainty is unavailable for these datasets.Source: Figure 3 · p. 10, §5.3 · p. 11

5.4 Effect on the Global Stopping Compromise

  • Under AdaQR, cohort-based checkpoint selection yields a lower benefit than under quantile loss for both cohort counts, with the benefit decreasing on five of seven datasets in both settings.This real-world MLP experiment is consistent with the synthetic finding that AdaQR alleviates, but does not establish the disappearance of, stopping-time mismatch.Source: §5.4 · p. 11, Table 6 · p. 11

6. Conclusion

The paper systematically compares tabular regression objectives and finds quantile loss to be a consistently strong alternative to tuned MSE. It also identifies global early stopping as a compromise across noise regions and introduces AdaQR as one possible remedy, with benefits that vary across settings.

6. Conclusion

  • The study compares 18 objectives on 41 datasets and evaluates leading approaches across multiple backbones.Quantile loss emerges as a consistently strong alternative to carefully tuned MSE.Source: §6 · p. 11
  • Regions with different noise levels can peak at different training epochs, making a shared checkpoint a compromise.Source: §6 · p. 11
  • AdaQR uses jointly learned, input-dependent smoothing as one possible remedy and improves predictions in several settings, with benefits varying across backbones and datasets.Source: §6 · p. 11

7. Limitations

The study’s understanding of uncertainty-dependent stopping conflicts remains incomplete, and AdaQR does not fully resolve them. Its gains vary across backbones, and its main evaluation excludes Tiny datasets; the authors also describe AI use and reproducibility resources.

7. Limitations

  • The underlying training dynamics and the conditions that produce uncertainty-dependent stopping conflicts remain incompletely understood.The authors call for further study of when conflicts arise and of alternative objectives, training procedures, and checkpoint-selection strategies.Source: §7 · p. 11
  • Cohort-based checkpoint selection still adds gains for AdaQR, but smaller ones than for standard quantile regression, indicating that smoothing does not fully resolve stopping-time differences.The residual benefit is reported on real datasets.Source: §7 · p. 11
  • AdaQR’s predictive gains vary across backbones, and its main evaluation excludes Tiny datasets, though Appendix I reports results including them.Source: §7 · p. 12

AI use statement

  • The authors used generative AI for code implementation, literature identification, methodology and design discussions, and manuscript editing; they reviewed AI-assisted material and verified generated code.The authors state that they made final methodological and scientific decisions and take responsibility for the manuscript, code, and results.Source: AI use statement · p. 12

Reproducibility statement

  • The paper’s code and experiment configurations are available in the repository, whose README explains dataset downloading and experiment reproduction.Source: Reproducibility statement · p. 12

A. Datasets

The benchmark uses the lite split of 41 BeyondArena datasets, with three large or high-dimensional datasets excluded because exhaustive tuning exceeded the per-dataset compute budget. Failed MLP objective runs are assigned a conservative replacement score for aggregate comparisons.

A. Datasets

  • Three BeyondArena datasets were excluded because their dimensionality, categorical cardinality, or sample size made exhaustive tuning infeasible within the 24-hour per-dataset budget.The excluded datasets are maps_router_eta_1m, delivery_eta_1m, and cooking_time_1m; the benchmark uses the first split of each remaining dataset.Source: A. Datasets · p. 16
  • For GAR[MSE], GAR[MAE], and Ordinal Entropy on MLP, failed runs on two datasets were replaced with the worst result among methods that converged on the same dataset.Source: A. Datasets · p. 16

B.1 Experimental protocol

The appendices specify a shared preprocessing, training, tuning, and evaluation protocol, alongside implementation details for the evaluated backbones and regression objectives. They define rank and improvement summaries and describe how each objective represents or predicts targets.

B.1 Experimental protocol

  • Models share feature and target preprocessing, train with Muon for weight matrices and AdamW for biases and output weights, and use 100 tuning trials before evaluation across 15 random seeds.The tuning objective is validation RMSE; test RMSE is reported after retraining the selected configuration from scratch.Source: §B.1 · pp. 16–17

B.2 Ranks and metrics

  • The study reports test RMSE and aggregates per-dataset relative improvements by their median; ranks use a conservative mean-and-standard-deviation comparison to avoid counting negligible gains as wins.Ranks are assigned independently per dataset using a top-down procedure; Figure 1 summarizes mean and standard deviation of ranks across the benchmark.Source: §B.2 · p. 17

B.3 Support construction for distributional losses

  • Distributional-loss bin supports span the training-label 0.005–0.995 quantiles, expanded on both sides and uniformly partitioned into bins.The cited protocol uses quantiles rather than extrema to mitigate outlier effects while capturing effective target support.Source: §B.3 · p. 17

B.4 Advanced models from Section 3.2.3

  • The advanced-model experiments use TabM-mini with PLE-Q embeddings, TabPack with 384 bins or quantiles, and RealMLP with added HL-Gaussian, quantile regression, and AdaQR.TabPack’s AdaQR regularizer is fixed at 0.2; RealMLP uses the tabarena-new search space.Source: §B.4 · pp. 17–18

B.5 AdaQR implementation

  • AdaQR adds a learned input-dependent smoothing-scale logit alongside quantile predictions, centers its initial scale to mean 0.5, and averages predicted quantiles at inference.The smoothing-scale output is not used at inference; the implementation uses equally spaced quantile levels from 0.001 to 0.999.Source: §B.5 · p. 18

B.6 Hyperparameter Tuning Spaces

  • The appendices provide optimizer, architecture, and objective-specific tuning spaces, with RealMLP-specific parameters taken from the tabarena-new search space.Table 7 distinguishes the default MLP space from the decoder-specific space; the TabPack and AdaQR settings are described separately.Source: §B.6 · p. 18, Table 7 · p. 18, Table 9 · p. 19

C.1.1 Traditional losses; C.1.2 Binning-based losses

  • Canonical losses include MSE, MAE, and Huber, while binning objectives train categorical heads against target distributions and predict the expectation over bin centers.Binning variants include hard assignments, soft mappings with a global spread parameter, truncated-Gaussian probabilities, and learned target distributions with reconstruction.Source: §C.1.1 · p. 19, §C.1.2 · p. 21

C.1.3 Quantile loss; C.1.4 Modeling full conditional distributions in the arbitrary form; C.1.5 Other losses

  • Quantile loss predicts conditional quantiles with pinball loss; alternatives include autoregressive tokenized-target decoding, gradient-aligned regression, representation regularizers, β-NLL, and Gaussian-mixture density networks.The decoder is jointly trained with an MLP using next-token cross-entropy, while β-NLL reweights NLL by a detached predicted-variance power.Source: §C.1.3 · p. 21, §C.1.4 · p. 22, §C.1.5 · p. 22

D. Detailed description of the dataset used in Section 4

The synthetic experiments use controlled homoskedastic and heteroskedastic noise with clean validation and test targets. The appendix relates quantile loss to CRPS and energy distance, then describes AdaQR’s smoothed target distribution in quantile terms.

D. Detailed description of the dataset used in Section 4

  • The synthetic dataset has 50,000 examples, with noisy training targets but noise-free validation and test targets for evaluating recovery of the conditional expectation.Its heteroskedastic noise scale depends nonlinearly on informative features, while the homoskedastic setting has constant variance.Source: D. Detailed description of the dataset… · p. 23
  • The synthetic features combine 100 informative dimensions generated from a 40-dimensional Gaussian latent with 16 irrelevant Gaussian-noise dimensions, and a two-layer MLP generates clean targets.Source: D. Detailed description of the dataset… · p. 23

E. Quantile loss and Energy Distance

  • With a finite uniform quantile grid, quantile loss is equivalent to CRPS up to discretization and a constant factor; its energy-distance interpretation compares the predicted distribution with a point mass at the observed target.AdaQR replaces that point mass with an input-dependent Gaussian smoothing distribution.Source: E. Quantile loss and Energy Distance · p. 24

F.1 Full objective in terms of quantiles

  • The AdaQR objective can be expressed using predicted quantiles at levels from 0 to 1, with the standard normal CDF used in its formulation.Source: §F.1 · p. 24

F.2 Proof of Proposition 1

The appendix characterizes AdaQR’s uncertainty estimates under Gaussian assumptions and describes a scale regularizer for RealMLP. The regularizer improves the agreement and ensemble consistency of RealMLP’s estimated uncertainty, while AdaQR estimates also correlate positively with CatBoost uncertainty on additional datasets.

F.2 Proof of Proposition 1

  • Under Proposition 1’s assumptions, AdaQR’s population optimum captures relative heteroskedasticity, with the proof relying on a unique minimum.The stated assumptions include a sufficiently large quantile set, Gaussian predicted and target distributions, and convergence to Bayes-optimal predictions.Source: §F.2 · pp. 25–26

G. Scale regularization for RealMLP

  • AdaQR† augments AdaQR with an auxiliary scale regularizer, without architectural changes, while retaining the average predicted quantiles as the inference-time point prediction.The regularization strength is controlled by γ.Source: G. Scale regularization for RealMLP · p. 26, §G.2 · p. 27

G.1 Uncertainty estimation in RealMLP

  • Proposition 1’s population result does not guarantee that training reaches the optimum; a model may rank uncertainty correctly yet misestimate absolute scales.This distinction motivates scale regularization for RealMLP.Source: §G.1 · p. 26, G. Scale regularization for RealMLP · p. 26
  • RealMLP shows larger scale deviations and greater variation across ensemble members than TabM, motivating an auxiliary regularizer for inferred scales.TabM already recovers synthetic data uncertainty reasonably well and consistently across members.Source: §G.1 · p. 26

G.2 Auxiliary scale regularization

  • The auxiliary CRPS terms share AdaQR’s idealized scale optimum under Proposition 1’s assumptions and a correctly specified conditional mean.The regularizer directly trains quantile spread and smoothing scale, though shared-backbone effects can still alter point predictions.Source: §G.2 · p. 27

G.3 Effect on RealMLP

  • AdaQR† brings RealMLP’s estimated noise scales closer to true uncertainty and makes estimates more consistent across ensemble members.The improvement primarily concerns absolute scales; vanilla AdaQR already ranks inputs by uncertainty reasonably well.Source: §G.3 · p. 27

H. Uncertainty Estimation

  • AdaQR-derived uncertainty estimates have positive correlations with CatBoost estimates across nine additional real-world datasets.TabM branch scales are converted to data variances and averaged; both methods’ estimates use standardized-target variance units.Source: H. Uncertainty Estimation · p. 27

I. Extended Results

The extended-results appendix collects additional objective and checkpoint-selection results, alongside uncertainty-estimation comparisons. Its captions specify the reported metrics, datasets, and evaluation setups, but do not provide the underlying result values.

I. Extended Results

  • The uncertainty figures compare AdaQR estimates with true synthetic uncertainty and CatBoost estimates on additional real-world datasets.The synthetic figure reports relative-variation capture for all three models and improved low-uncertainty deviations and ensemble consistency with scale-regularized RealMLP.Source: Figure 4 · p. 28, Figure 5 · p. 29
  • Table 11 defines improvement as median relative RMSE improvement over MSE and summarizes win/tie/loss comparisons using the paper’s stated rule.The caption does not report the table’s underlying values.Source: Table 11 · p. 28
  • The extended results include cohort-based versus global checkpoint comparisons for MLPs trained with quantile loss or AdaQR on seven BeyondArena datasets.The table reports test RMSE over 15 seeds, with global or five- and ten-cohort validation-selected checkpoints.Source: §I · p. 27, Table 12 · p. 30
  • The appendix also presents dataset-wise MLP results for AdamW and extended Beyond results for Muon, including single-model and ensemble settings.The captions specify 41 Beyond datasets for the AdamW results; the Muon tables are described as extended Beyond results.Source: §I · p. 27, Table 13 · p. 31, Table 14 · p. 32
Loading 2610.13909v1…