Tabular Deep Learning for Algorithmic Trading: Cross-Regime Bayesian Optimisation for Equity Signal Generation

Joshua Le Grice

Source-linked AI summary

Tabular Deep Learning for Algorithmic Trading: Cross-Regime Bayesian Optimisation for Equity Signal Generation

Joshua Le Grice

arXiv:2608.27076v1Aug 27, 2026cs.LGq-fin.CPq-fin.TR

TL;DR

The paper asks whether equity prediction models can be tuned for robustness across changing market regimes and evaluated by trading outcomes rather than prediction metrics alone. It applies cross-regime Bayesian optimisation to tabular models and a cross-sectional US equity strategy, finding that an XGBoost–TabNet Hybrid achieves significant out-of-sample alpha while individual deep architectures do not surpass gradient-boosted trees.

Problem
Existing evaluations do not explicitly target regime robustness during hyperparameter selection, despite tabular deep learning models rarely surpassing gradient-boosted trees in static benchmarks.
Method
Five model classes are trained on daily observations from approximately 300 large-cap US equities, with hyperparameters selected through expanding-window validation across three market regimes.
Results
The XGBoost–TabNet Hybrid produces statistically significant out-of-sample alpha, while signal precision remains above the random baseline across all four quarterly test regimes.
Takeaways & Limitations
Cross-regime ensemble tuning supports generalisation under regime and input shifts, while alternative data contributes secondarily and more strongly to short-side signals.
Takeaways & Limitations
Survivorship bias and licensed Bloomberg data limit the absolute-return interpretation and independent replication of the results.

Abstract

from arXiv · show

Algorithmic trading now represents a market exceeding $20 billion, where even marginal gains in signal robustness can translate into economically significant returns. Existing evaluations of equity prediction models do not explicitly target regime robustness during hyperparameter selection. Five model classes are trained on daily observations from approximately 300 large-cap US equities over eleven years, with Bayesian optimisation configured to target trading performance across three statistically different market regimes. Regime-robust hyperparameter selection is associated with out-of-sample generalisation, as signal precision remains above the random baseline across all four quarters of the test period, and portfolio performance slowly degrades under simulated input noise before collapsing beyond a defined threshold. No individual tabular deep learning architecture outperforms gradient-boosted trees, but combining XGBoost and TabNet using rank aggregation produces a Hybrid ensemble with an annualised return of 51.26%, a Sharpe ratio of 2.44, and a statistically significant CAPM alpha of 0.423 (p = 0.011). A near-zero beta indicates this outperformance is driven by stock selection, not market exposure. Alternative data plays a secondary role once technical and fundamental features are accounted for, as well as contributing more strongly on the short side than the long, and varies by model class. An interactive application makes these results explorable in real time, with live data integration the remaining step toward practical deployment.

1 Introduction

The paper addresses regime instability, structural challenges in cross-sectional equity prediction, and the mismatch between predictive accuracy and trading performance. It evaluates tabular deep learning models using cross-regime Bayesian optimisation and trading-focused held-out evaluation.

Motivation: $20.23 billion in 2026, projected to reach $29.54 billion by 2031, highlights the economic importance of marginal improvements in trading robustness.Strategies that perform well in one market environment often fail as conditions shift.
Research gap: Cross-sectional equity prediction is difficult because daily returns are stochastic, while market regimes can change the relationship between inputs and returns.The paper therefore uses cross-sectional ranking rather than continuous-return prediction.
Research questions: Alternative data may add information beyond price and accounting variables, but its contribution to forecasting and trading under realistic conditions remains poorly understood.The study also examines how source contributions vary across models and signal directions.
Research gap: Predictive accuracy and trading performance are distinct objectives because classification metrics can overlook return magnitude, direction, and execution costs.Evaluating one as a proxy for the other can produce misleading conclusions.
Approach: The experiments train tabular deep learning models in a cross-sectional US equity strategy, select hyperparameters across market regimes, and evaluate held-out 2025 trading performance.Robustness analysis and SHAP attributions assess sensitivity to shifts and source-category contributions.
Contributions: The Hybrid ensemble maintains performance across distinct regimes and realistic input perturbations, while individual tabular deep learning architectures do not consistently improve over gradient-boosted trees.The ensemble combines XGBoost and TabNet and achieves significant out-of-sample alpha.

2 Literature Review

Prior work establishes the usefulness of machine learning and alternative data for financial prediction but highlights weaknesses of deep learning on tabular inputs and instability under regime shifts. The paper positions cross-regime optimisation and direction-specific source attribution as responses to these gaps.

Financial prediction methods: Classical statistical models provide financial forecasting benchmarks, while machine learning captures multivariate interactions that linear time-series methods cannot.Tree-based ensembles including Random Forests and XGBoost have improved forecasting over statistical baselines in financial applications.
Tabular deep learning: Neural networks can learn feature representations, but standard architectures lack mechanisms for temporal dependence or irregular tabular feature interactions.Tabular data often favors tree-based splits because target functions are irregular and piecewise.
Tabular deep learning: TabNet uses sequential attention for instance-wise feature selection, whereas FT-Transformer represents each feature as a distinct token.These architectures address tabular learning through mechanisms distinct from standard MLPs.
Research gap: Benchmark evidence finds tabular deep learning frequently outperforms standard MLPs but rarely surpasses gradient-boosted trees without careful tuning.The paper notes that regime changes make robust generalisation more difficult than in static benchmarks.
Alternative data: Existing alternative-data research reports predictive information in news sentiment and search volume, but much of the evidence is market-level rather than stock-specific.This limits direct relevance to daily cross-sectional stock differentiation.
Open questions: Prior literature identifies survivorship and look-ahead bias risks, while the relative contribution of source types across long and short signals remains unquantified.The paper addresses source attribution separately by signal direction.
Evaluation gap: Classification accuracy need not predict trading profitability because errors on high-magnitude moves and execution assumptions can dominate aggregate performance.This motivates evaluating models with realistic trading objectives rather than classification metrics alone.
Regime robustness: Regime-aware optimisation selects hyperparameters that perform across multiple environments without requiring regime detection during inference.This treats regime heterogeneity as an optimisation constraint rather than an input state.

3 Methodology

The methodology builds a large-cap US equity panel from market, fundamental, macroeconomic, sentiment, and search data, transforms returns into cross-sectional classes, and evaluates models with expanding-window cross-regime validation. Bayesian optimisation targets trading performance, while ensemble construction, transaction costs, and data-access constraints shape the evaluation.

Dataset and preprocessing: 300 large-cap S&P 500 constituents are observed from January 2015 to December 2025, with FRED, firm-level news sentiment, and Google Trends added to price and company data.The large-cap universe supports realistic liquidity and transaction-cost assumptions.
Dataset and preprocessing: Temporal lags shift macroeconomic, fundamental, jobless-claims, news, and search data so only information available at prediction time enters the features.The stated shifts are 21, 63, 5, and 1 trading days for the respective data frequencies.
Dataset and preprocessing: Four feature categories are used: technical indicators, company fundamentals, macroeconomic factors, and alternative data.Features are transformed into relative terms where possible to improve cross-sectional predictability.
Target construction: Next-day returns are ranked into daily deciles, assigning the top and bottom 10% to long and short classes in a three-class classification task.The task avoids direct continuous-return prediction because returns are noisy, heteroskedastic, and nonstationary.
Experimental setup: Training and validation cover 2015–2024, while 2025 is held out as an uninfluenced out-of-sample test period.Three expanding-window folds evaluate 2022, 2023, and 2024, with a one-day purge at each boundary.
Portfolio construction: Models generate separate long and short probability rankings, and the top n stocks in each ranking receive the corresponding positions.Separate rankings evaluate each signal direction on its own cross-sectional distribution.
Trading evaluation: 2.2 basis points per trade are charged per leg on entry and exit to represent transaction costs and slippage.The assumption exceeds the cited spread component to leave room for commissions and fees.
Ensemble construction: XGBoost and TabNet form the Hybrid ensemble through averaged rank aggregation, selected after eliminating combinations with negative returns in any validation regime.Rank aggregation reduces sensitivity to differences in probability calibration across models.

4 Results

Across market regimes and the 2025 out-of-sample period, the Hybrid ensemble delivered the strongest trading performance and statistically significant CAPM alpha. Its signal precision remained above the 10% random baseline across quarters, while performance degraded gradually under moderate input noise before collapsing at higher noise.

Trading performance: XGBoost, TabNet, FT-Transformer and the Hybrid returned positively across the bear, recovery and bull validation regimes, unlike LR and partly MLP.The S&P 500 returns were −18.99%, 26.00% and 25.28% across those regimes.
Trading performance: 51.26% total return and 2.44 Sharpe made the Hybrid the strongest 2025 out-of-sample performer, with a maximum drawdown of −7.76%.XGBoost was the strongest individual model, returning 33.61% with a Sharpe ratio of 1.37.
Statistical robustness: 0.423 annualised CAPM alpha (p = 0.011) and β = 0.048 made the Hybrid the only model with significant alpha and indicated limited market exposure.Its PSR was 0.960, corresponding to a 96% probability that its Sharpe ratio exceeded the S&P 500’s.
Statistical robustness: Table 3’s Wilcoxon and KS tests largely failed to reject distributional equality with the S&P 500, while the Friedman test found no significant differences in strategy rank distributions.The FT-Transformer was the exception in the KS test, with p = 0.042; the Friedman test returned p = 0.352.
Robustness and temporal stability: 34.36% total return and 1.53 Sharpe remained at σ = 0.10, but performance decreased sharply at σ = 0.20; the Random Null returned −8.21% with a Sharpe of −1.44.JS Divergence increased with noise, from 0.0126 Long and 0.0102 Short at σ = 0.05 to 0.0350 and 0.0261 at σ = 0.50.
Robustness and temporal stability: 13–17 percentage-point precision improvements over the 10% random baseline persisted in every 2025 quarter, with no significant quarterly collapse.Long precision ranged from 23.06% to 25.78%, while Short precision ranged from 24.34% to 26.67%.
Feature attribution: Technical features led both signal directions, while macroeconomic and alternative features together contributed under 6% of attribution.For XGBoost, Market Capitalisation led Long attribution at 0.0216, whereas Trading Volume led Short attribution after Market Capitalisation fell to fourth.
Portfolio sensitivity: 26.1% return at N = 10 reflected signal dilution, while maximum drawdown improved from −25.7% at N = 1 to −4.8% at N = 10.The Bayesian-optimised configuration N = 6 traded some peak return for broader-position stability, with Sharpe 2.44 and drawdown −7.76%.

5 Discussion

The discussion finds that ensemble diversification and regime-aware tuning improve trading robustness, while no individual tabular deep learning model surpasses XGBoost. Alternative data contributes unevenly across architectures and more strongly to short signals, but deployment remains constrained by concentration, data quality, universe, sample, and evaluation limitations.

Signal and Model Evaluation: 80% of observations belong to Hold, making accuracy and F1 poor indicators of trading-signal quality.A hold-only classifier would achieve 80% accuracy, while observed accuracy ranges only from 0.786 to 0.799.
Signal and Model Evaluation: XGBoost is the strongest individual model, achieving a 33.61% OOS return and 1.37 Sharpe ratio.Its edge may reflect the smaller effective number of independent market environments and lower tuning demands relative to deep learning models.
Signal and Model Evaluation: The Hybrid is the only model with statistically significant OOS alpha, benefiting from combining models whose errors are uncorrelated.TabNet was selected as XGBoost’s ensemble partner through the composite scoring function.
Signal and Model Evaluation: The Hybrid’s near-zero beta and 96% PSR probability support stock-selection-driven outperformance rather than market exposure.Its 0.254 short precision persisted across the OOS period, while KS and Friedman tests found no significant distribution-shape or daily-rank differences.
Signal and Model Evaluation: TDL architectures do not individually improve on gradient-boosted trees, with TabNet’s value emerging through signal orthogonality with XGBoost.This supports the interpretation that cross-regime Bayesian optimisation selects configurations that generalise beyond their estimation conditions.
Feature Contributions: Technical indicators account for 40–63% of SHAP attribution, while alternative-data contributions vary substantially by architecture.XGBoost assigns 43.73% attribution to fundamentals and 1.63% to alternative data; the strongest model therefore falls below the 10% alternative-data threshold, unlike LR and MLP.
Feature Contributions: Alternative data generally contributes more to short-side than long-side signals.The discussion links this pattern to stronger market reactions to negative information and uncertainty-related search activity.
Deployment Considerations: The highest-conviction portfolio peak at N = 3 is impractical for conventional institutional mandates because concentration and market impact would be excessive.Even N = 6 exceeds concentration thresholds and is better suited to flexible proprietary or hedge-fund mandates.

6 Conclusion

Cross-regime Bayesian optimisation and ensemble design produce the paper’s strongest result: a Hybrid combining XGBoost and TabNet achieves significant out-of-sample alpha, while individual tabular deep learning architectures do not consistently surpass gradient-boosted trees. Alternative data contributes secondarily, with architecture-dependent and stronger short-side effects.

Core conclusion: Cross-regime Bayesian optimisation of a TDL ensemble provides statistically significant OOS alpha that no constituent model achieves.The paper presents this as evidence that ensemble value is combinatorial and that regime-aware tuning may transfer beyond equities.
Model comparison: TabNet and FT-Transformer do not produce significant improvements over gradient-boosted trees, while XGBoost remains the strongest individual model.TabNet is selected as the Hybrid partner because its signal is more orthogonal to XGBoost’s than MLP’s signal.
Robustness: The Hybrid’s out-of-sample performance remains above the S&P 500 under realistic input perturbations, with signal precision above the random baseline across all four quarterly regimes.These results support generalisation across both regime and input shifts.
Alternative data: Alternative data contributes supplementary information, varies considerably across architectures, and contributes more strongly to short-side than long-side signals.The strongest individual model relies on alternative features only marginally, while attribution is architecture-dependent.
Practical implication: Portfolio-level backtesting should guide model selection, while ensemble construction should prioritise signal orthogonality over individual model performance.Classification metrics can remain consistent across models even when their production profitability differs.

A UMAP Feature Space Projections

The UMAP projections visualise the training feature space by raw next-day returns and cross-sectional return deciles. Their lack of systematic colour organisation motivates cross-sectional ranking, while visible structural clusters may reflect market regimes.

Projection setup: The UMAP projections use approximately 6,000 stratified observations from 2015–2024, coloured by raw next-day return magnitude and cross-sectional decile label.The two figures provide 2D and 3D views of the same feature-space sample.
Interpretation: The projections show no systematic colour organisation, suggesting absolute return magnitude and daily cross-sectional decile labels are not learnable from individual feature vectors alone.This motivates cross-sectional ranking during portfolio construction.
Interpretation: Three visible structural clusters may correspond to bear, recovery, and bull market regimes.The projection offers an unsupervised geometric visualisation of possible market-regime structure.

B Kolmogorov-Smirnov Tests and Distributional Analysis

Distributional tests support treating the validation folds as distinct market regimes. The accompanying figures and equity-curve comparisons organise regime distributions and model performance across the validation periods.

KS tests: Pairwise two-sample KS tests reject equal daily cross-sectional return and 5-day realised volatility distributions across all validation-fold pairs at the 1% level.This supports multi-regime validation.
KS tests: Table 7 summarises pairwise Kolmogorov-Smirnov test results across market regimes.The table is the reported tabular summary of the regime-separation analysis.
Distributional analysis: Figure 9 presents kernel density estimates of return and volatility distributions across the 2022–2025 market regimes.Figure 10 compares training-period distributions from 2015–2024 against the 2025 out-of-sample period.
Validation performance: Figures 11–13 present equity curves for all models against the S&P 500 across the 2022, 2023, and 2024 validation periods.All portfolios are initialised at $10 million, and the periods correspond to the three Bayesian optimisation validation folds.

D Confusion Matrices

The OOS confusion matrices show majority assignment to Hold under the 80% class imbalance, while Long and Short diagonal counts indicate above-random minority-class signal across models.

Class distribution: The confusion matrices show majority assignment to the Hold class, consistent with the 80% class imbalance and top-rank selection.This pattern appears across all evaluated models in the 2025 out-of-sample period.
Minority-class signal: Long and Short diagonal counts confirm that all models retain above-random signal in the minority classes.The matrices therefore provide evidence of minority-class predictive signal despite dominant Hold assignments.

E Statistical Test Results

The analysis tests return normality and compares model performance through pairwise CAPM regressions. Most models require non-parametric testing, while the Hybrid shows significant relative alpha against several alternatives.

Normality Tests: Only LR (W = 0.992, p = 0.177) and MLP (W = 0.989, p = 0.062) fail to reject normality at the 5% significance level.Non-parametric tests are therefore used for all models.
Normality Tests: The Shapiro–Wilk analysis covers daily returns for every model and the S&P 500 benchmark over the OOS period.
CAPM Regressions: The Hybrid produces statistically significant relative alphas against MLP (p = 0.015), TabNet (p = 0.009), and FT-Transformer (p = 0.005).Relative alpha captures return generation beyond what the comparison model explains.
CAPM Regressions: Pairwise CAPM regressions compare all model pairs, with annualised relative alpha reported as Ann. α.

F Robustness and Temporal Stability

Robustness analysis examines how the Hybrid responds to Gaussian input noise and whether quarterly OOS signal precision remains above a random baseline. The figures jointly track degradation, distribution shift, and temporal precision.

Noise Robustness: Figure 15 evaluates Hybrid strategy degradation and output distribution shift as Gaussian noise increases.It also reports the probability distribution shift at σ = 0.20.
Temporal Stability: Figure 16 compares Hybrid Long and Short signal precision across the four 2025 OOS quarters against the 10% random baseline.

G Feature Importances

Feature-importance analysis uses SHAP attribution to compare feature categories and top individual features across five base models and both signal directions. The reported rankings distinguish buy and sell signal contributions by architecture.

Category Attribution: SHAP analysis compares Technical, Fundamental, Macroeconomic, and Alternative category contributions across five base models for long and short signals.Category totals represent each category’s share of total mean absolute SHAP attribution across the full 50-feature set.
Architecture Comparisons: The appendix provides architecture-specific top-10 feature tables for Logistic Regression, XGBoost, MLP, TabNet, and FT-Transformer.
Feature Rankings: Daily Dollar Volume ranks first at 0.0181, followed by Shares Outstanding and Market Capitalisation at 0.0109.
Feature Rankings: Top-10 features are ranked by mean absolute SHAP value separately for buy and sell directions, and may differ between directions.

H Tuning, Training, and Inference Time

The paper reports tuning, training, and inference costs by model and operationalises the outputs through an interactive React application served by a REST API. The application exposes data exploration, alternative-data, backtesting, and research-assistant views.

Computational Cost: Table 16 reports tuning, final training, and inference time for all models to inform retraining cadence.Tuning covers the full 30-trial Optuna search; final training uses 2015–2024, and inference uses the 2025 OOS set.
Computational Cost: The Hybrid ensemble is excluded from tuning and training figures because rank aggregation combines pre-trained constituent outputs without separate training.
Interactive Application: A React frontend served by a REST API operationalises the study’s outputs through four primary views.
Interactive Application: The application includes stock-level data, alternative-data exploration, interactive backtesting, and an AI research assistant grounded in backtest and macroeconomic data.