PaperlayerView exact revision on arXiv

Source-linked AI summary

KellyBoost: Growth-Optimal Portfolio Construction with Gradient-Boosted Trees

Jiayu Li

arXiv:2608.23393v1q-fin.PMcs.LG

TL;DR

Portfolio learning commonly separates return prediction from portfolio decisions, while boosted trees have largely required surrogate objectives for allocation. KellyBoost trains a multi-output XGBoost model directly on negative log growth with exact derivatives and evaluates it under a separated deployment protocol. The growth objective wins in all four learner–loss cells, although the study’s evidence is bounded by its single universe, limited monthly-horizon history, and disclosed evaluation-segment peek.

  • Problem

    Most machine-learning portfolio methods optimize return prediction and portfolio allocation in separate stages, while boosted trees have been excluded from direct decision-focused learning.

  • Method

    KellyBoost uses one multi-output XGBoost model whose softmax output is a long-only, fully invested portfolio trained on the exact negative log-growth objective with analytically derived curvature.

  • Results

    Four of four learner–loss cells favor the growth objective, with a pooled paired effect of +0.18 per 20-day decision (×100) rising to +0.25 under 20 bps of transaction costs.

  • Takeaways & Limitations

    In this regime, replacing classification surrogates with the exact growth objective improves deployed performance while trading less in every cell.

  • Takeaways & Limitations

    The evidence covers one universe and a 13.6-year evaluation window, with leverage, shorting, taxes, market impact, and capacity out of scope; the evaluation segment was also scored twice.

Abstract

from arXiv · show

KellyBoost is a single multi-output XGBoost model whose softmax output is the portfolio: with y the vector of per-asset holding-period returns, the training loss is - log(1 + w y), the negative log growth rate, so the fitted model is the growth-optimal (Kelly) allocation conditioned on the features. The objective is exact rather than a surrogate: we derive the gradient, the analytic diagonal Hessian and the full Hessian in closed form, verify them by finite differences, and ship a dependency-free reference engine.

1 Introduction

KellyBoost addresses the mismatch between return prediction and portfolio decisions by training boosted trees directly on negative log growth. The study derives an exact objective, evaluates deployment under a separated protocol, and finds growth-loss training superior to classification surrogates across all tested learner and feature-pipeline cells.

  • Motivation: Most machine-learning portfolio methods forecast returns under a statistical loss before handing forecasts and a covariance estimate to an optimizer.This separates prediction from the portfolio decision and optimizes different objectives across the two stages.
  • Method: KellyBoost closes the end-to-end gap for boosted trees with a multi-output XGBoost model whose softmax output is a long-only, fully invested portfolio.Its loss is −log(1 + w⊤y), whose sample mean is the negative log growth rate, and its gradient and diagonal Hessian are derived analytically.
  • Method: A leave-one-out ensemble addresses material allocation instability caused by dropping a single training row, while retaining unchanged hyperparameters.The paper treats a single fit as one draw from a distribution and uses K fits trained with one row omitted in each member.
  • Evaluation: The evaluation separates development-segment hyperparameter selection from untouched 2013–2026 performance measurement using frozen parameters and expanding-window deployment.The development segment ends in 2012, and the evaluation uses an end-anchored decision grid.
  • Evaluation: Eight asset legs and a 7,871-column candidate feature set are evaluated over 2003–2026 using public market data with same-day, backward-looking inputs.The raw snapshot is committed to the repository, and the data require no API keys.
  • Results: Four of four learner–loss cells favor the growth objective, with a pooled paired effect of +0.18 per 20-day decision (×100) rising to +0.25 under 20 bps of transaction costs.The growth objective also trades less in every cell, while the two-stage pipeline beats every end-to-end learner in this regime.

2 Related work

The related work spans growth-optimal investing, end-to-end neural portfolio learning, custom objectives for boosted trees, and disciplined backtest evaluation. KellyBoost combines these strands by conditioning growth-optimal allocation on features and bringing the objective to boosted trees.

  • Growth-optimal investment: Kelly’s growth-optimal criterion maximizes expected logarithmic wealth, while KellyBoost estimates the log-optimal portfolio conditionally as a function of an arbitrary feature vector.Earlier literature conditions on no information or price history alone.
  • End-to-end portfolio learning: Prior end-to-end portfolio-learning work uses neural networks to optimize Sharpe ratio, utility, or risk-budgeting objectives directly.KellyBoost’s contribution is orthogonal to architecture novelty: it brings the end-to-end objective to boosted trees.
  • Boosting with custom objectives: Gradient boosting supports twice-differentiable custom losses, and XGBoost’s multi-output mode grows one tree per round with vector-valued leaves.KellyBoost uses this machinery unchanged while introducing the growth objective and exact curvature demonstration.
  • Evaluation discipline: Purging and embargoing overlapping labels, caution about selected backtest maxima, and an end-anchored decision grid motivate the study’s evaluation discipline.These practices address leakage, selection bias, and sensitivity to rebalance-grid phase.

3 Method

KellyBoost directly estimates the conditional growth-optimal portfolio with a multi-output boosted-tree model, using an exact log-growth objective and closed-form curvature. The method also addresses non-convex second-order optimization, vector-leaf boosting, feature selection, and tree instability.

  • 3.1 The objective and its exact derivatives: A multi-output boosted-tree model maps features to logits, applies softmax to produce a long-only, fully invested portfolio, and minimizes negative sample log growth.The portfolio return is S_t = w_t^T y_t, and minimizing the loss maximizes geometric wealth growth without a forecasting or covariance-optimization stage.
  • 3.1 The objective and its exact derivatives: The population minimizer is the growth-optimal Kelly portfolio on the simplex conditioned on the feature vector.The pointwise decomposition of the expected objective establishes conditional growth optimality.
  • 3.1 The objective and its exact derivatives: Closed-form gradients and Hessians are finite-difference verified, with the gradient increasing an asset’s logit when its return exceeds the portfolio return.The gradient includes the compounding factor 1/(1 + S), distinguishing log growth from mean-return optimization.
  • 3.2 The curvature, examined: Because the loss is non-convex, negative diagonal curvature is rectified with |h_k|; rectification preserves descent direction while changing step size.Negative curvature is expected for roughly the cells whose asset is outperforming the portfolio, and λ bounds the resulting leaf step.
  • 3.2 The curvature, examined: 50% of row-asset cells are rectified initially, declining to 27% by the final training round, while off-diagonal Hessian mass reaches 0.62 per row.Despite substantial off-diagonal mass, diagonal and full rectified steps have mean cosine at least 0.99, and the two leaf solvers score within noise.
  • 3.3 Second-order boosting with vector leaves: Vector-leaf trees decouple outputs after a shared split, so asset coupling enters through split gains summed across outputs; feature search jointly selects hyperparameters and features.The candidate library contains 7,871 backward-looking columns, ranked into a 300-feature pool and searched by stochastic local moves.

4 Data

The testbed uses eight investable asset legs, public daily market data, and backward-looking features constructed deterministically from a committed raw snapshot. Labels are overlapping forward 20-trading-day returns, motivating purged evaluation.

  • Asset and feature data: The raw dataset contains 38 daily series from 1985–2026, all freely downloadable from Yahoo Finance without an API key and deterministically processed downstream.The raw snapshot is committed to the repository, making the data pipeline reproducible.
  • Asset and feature data: The eight asset legs are growth equity, value equity, long-term Treasuries, international equity, energy equity, gold, silver, and cash.Mutual funds are used where their longer histories improve coverage, and adjusted closes include distributions.
  • Asset and feature data: Feature-only inputs include broad market, volatility, yield, commodity, sector, and cross-asset context series, none of which is an investable leg.Mutual-fund share classes extend histories when equivalent ETFs are too young.
  • Features: The hand-built feature set has 174 columns, while the candidate set has 7,871 columns and contains the hand-built set as a superset.Features include momentum, volatility, moving-average ratios, drawdowns, z-scores, yield changes, curve slopes, and VIX statistics over multiple windows.
  • Labels: Labels are per-leg forward 20-trading-day simple returns, with adjacent rows overlapping 19 label days and therefore requiring a purge.The purge is specifically designed to remove dependence created by this overlap.

5 Experiments

Experiments compare learned methods under identical candidate sets, labels, searches, ensembles, and walk-forward conventions. Evaluation uses realized log growth as the primary metric alongside annualized return, volatility, Sharpe ratio, and drawdown.

  • Methods compared: Every learned method uses the same candidate set and label frame, identical joint hyperparameter-feature search, and the same leave-one-out ensemble and walk-forward deployment.This design isolates differences among the learning objectives and pipeline structures.
  • Methods compared: The comparison includes KellyBoost, a two-stage LightGBM predict-then-optimize portfolio, a LightGBM multiclass surrogate, two MLP variants, and unconditional Kelly.The MLP variants use either the growth objective or cross-entropy, completing the 2 × 2 learner-by-loss design.
  • Methods compared: Unconditional Kelly is the constant simplex portfolio maximizing in-sample log growth and serves as the baseline for measuring the value of conditioning on features.It is refit at each decision date.
  • Evaluation: Table 1 evaluates 163 monthly decisions from 2013-01 to 2026-07, reporting gross-of-costs mean 20-day log growth and annualized daily excess-return Sharpe.The last column gives moving-block-bootstrap 95% intervals for annualized daily log-growth differences versus KellyBoost.
  • Evaluation: The primary metric is realized mean log growth per 20-day decision, supplemented by annualized geometric return, volatility, excess-return Sharpe ratio, and maximum drawdown.Pairwise daily-log-growth differences use 21-day moving-block bootstrap intervals with 5,000 draws.

6 Results

Across controlled loss comparisons, the growth objective consistently outperforms the argmax surrogate, while pipeline shape and feature search determine how much of that advantage survives deployment.

  • The loss effect: the 2 × 2: The growth objective wins all four controlled learner-loss comparisons, with positive paired per-decision differences ranging from +0.07 to +0.37.The comparisons hold learner class and feature pipeline fixed while swapping only the training loss.
  • The loss effect: the 2 × 2: +0.18 per decision is the pooled growth-objective advantage, with bootstrap Pr(∆> 0) = 0.89 across the four aligned cells.The direction remains positive across block lengths of 1 to 12 decisions, although the pooled interval includes zero.
  • The loss effect: the 2 × 2: Cross-entropy discards return-margin information, treating narrowly and decisively won months as the same label and producing a more brittle target.The growth objective instead moves weights according to estimated conditional growth, and its advantage sharpens after transaction costs because it trades less.
  • Across learner classes: no conclusion: Learner-class comparisons do not support a conclusion: tree–MLP gaps are confounded by capacity and optimization geometry and remain within bootstrap noise.The MLP leads under the growth objective, while the tree leads under the surrogate, so neither direction is interpreted.
  • The objective’s boundary: what shrinks, wins the window: The two-stage predict-then-optimize pipeline out-deploys every end-to-end learner in this regime, as near-zero forecasts induce mild, diversified portfolios.The paper interprets forecast shrinkage as implicit regularization, marking a boundary where objective faithfulness stops paying.
  • Feature search: Feature search roughly doubled development selection scores, but deployment gains depended on pipeline: the two-stage method doubled realized growth while searched MLP lists worsened.End-to-end search mainly reshuffled concentration, showing that selection-score improvements did not reliably transfer to deployment.

7 Ablations

The ablations separate optimization quality from deployment quality: exact curvature optimizes the training objective better, while shrinkage-oriented variants and the growth objective can deploy more robustly.

  • Hessian substitutions: The exact curvature scores +4.19 versus +1.43 for the constant-Hessian variant on the development protocol at identical hyperparameters.This confirms the exact Hessian is the stronger optimizer of the training objective.
  • Hessian substitutions: Deployed, the constant-Hessian variant’s smaller steps leave allocations closer to uniform and outperform the fully optimized full-Kelly allocation out of sample.The deployment ranking therefore reverses the development ranking.
  • Vector leaves: Vector leaves select worse but deploy better, matching the shrinkage signature of Hessian substitutions because independent trees move softmax weights less coherently.The alternative removes the shared partition and multiplies parameters by K.
  • Ensembling: Ensembling stabilizes weights without increasing realized growth, removing split-tie fragility while leaving performance flat in K within noise.One-row perturbations can move deployed legs by whole percentage points; the committee is retained for determinism.
  • Off-diagonal curvature: The diagonal and full Hessian score +0.0335 and +0.0339 mean log growth per block, respectively, within noise.Their ≥0.99 leaf-step cosines indicate that off-diagonal curvature adds little at tuned regularization.
  • Transaction costs: The growth objective trades less than its argmax counterpart in all four cells, and transaction costs widen the pooled paired difference from +0.18 gross to +0.25 at 20 bps.Annualized turnover is lower for the growth objective in every reported cell.

8 Limitations

The evidence is limited to one universe, two feature pipelines, and a 13.6-year evaluation window, with additional constraints from monthly data, portfolio structure, and omitted market frictions.

  • Scope: The study covers one universe, two feature pipelines, and one 13.6-year evaluation window, so monthly-horizon evidence remains intrinsically scarce.The leakage-controlled protocol cannot create more independent history.
  • Portfolio constraints: The formulation is long-only and fully invested; leverage and shorting are described as unevaluated affine extensions of the softmax.The reported claims therefore concern the long-only, fully invested allocation regime.
  • Statistical dependence: Overlapping labels reduce effective sample size despite purge handling and bootstrap blocks sized to the overlap.The limitation applies to metrics computed on daily curves.
  • Costs: Taxes, market impact, and capacity are out of scope because the main backtests are gross of costs, with sensitivity priced separately.Transaction-cost sensitivity does not cover these additional frictions.
  • Evaluation disclosure: The evaluation segment was scored twice, and the later feature-search design was informed by the first underperforming evaluation.The authors report both pipelines and state that the central negative finding holds in both.
  • Risk calibration: Full-Kelly is the aggressive end of the risk spectrum, and the paper does not answer how its fractional-Kelly or CRRA risk dial should be set.Choosing that setting is treated as a preference rather than a statistical question resolved by the study.
  • Scale: The paper claims the asset-allocation regime, not the stock-selection regime, because hundreds of assets would require architectural extensions around the same objective.The cited extensions include shared features, hierarchical softmax, and sparse top-m allocation.

9 Conclusion

KellyBoost brings boosted trees into end-to-end portfolio learning with an exact growth objective and closed-form derivatives. Experiments support a narrow positive claim for the objective, while exposing deployment boundaries for full-Kelly optimization.

  • Contribution: KellyBoost combines softmax portfolio weights, negative log growth, closed-form gradient and curvature, finite-difference tests, and a dependency-free reference engine.The model is a single multi-output XGBoost learner rather than a surrogate forecast pipeline.
  • Empirical claim: Replacing classification with the exact decision objective improves deployed growth in all four learner-by-feature-pipeline cells.The claim is explicitly limited to the learner classes and feature pipelines tested.
  • Boundary: End-to-end learners trail the two-stage pipeline because estimated conditional full-Kelly allocations concentrate beyond what monthly-horizon signals support.The two-stage squared-error forecasts shrink toward zero, producing milder portfolios.
  • Boundary: The exact Hessian is a better selector but loses deployed to de-tuned substitutes, showing that faithful optimization is not the deployment objective’s desired outcome here.The conclusion distinguishes optimization success on selection from out-of-sample deployment success.
  • Future direction: Future work should retain the exact objective while adding intentional shrinkage through robust, regularized, turnover-penalized, or risk-calibrated formulations.These directions are presented as constructive extensions rather than established results.

A Derivation of the gradient and Hessian

The derivation differentiates the softmax-linked portfolio growth loss to obtain gradient, diagonal curvature, and the full Hessian. The expressions are checked against finite differences, including symmetry and stated tolerances.

  • Diagonal Hessian: The diagonal second derivative is obtained by differentiating the gradient again, producing the stated g_k(1 − 2σ_k) term plus σ_k^2.The derivation identifies how numerator and denominator contributions combine.
  • Verification: Central finite differences verify both gradient and diagonal Hessian, with relative tolerances 10^-5 and 10^-3, respectively.The full-Hessian identity is also checked entry-by-entry, including symmetry.
  • Setup: For each row, the softmax portfolio weights feed the scalar portfolio return S, which enters the log loss through 1 + S.The general formulation writes the loss as ℓ = ϕ(S), with log loss ϕ(S) = −log(1 + S).
  • Gradient: The gradient uses a_k = σ_k(y_k − S), so asset k is favored when its return exceeds the portfolio return.The log-loss derivative additionally scales by the compounding factor 1/(1 + S).
  • Full Hessian: The full Hessian follows by differentiating a_k and applying the chain rule, with ∂a/∂z = diag(a) − σa^T − aσ^T.The resulting matrix is symmetric in the output indices.

B Search spaces and budgets

Methods are searched under identical, method-specific development protocols and fixed seeds, with wall-clock budgets and bounded feature-state spaces.

  • All methods use a fixed-seed search and identical development protocol, but selection scores are compared only across states within each method.The protocol uses single-row purged blocks every 11 rows, or every 63 rows for the MLP.
  • KellyBoost receives a 60-minute wall-clock budget, while each LightGBM pipeline and the MLP receives 45 minutes on 30 cores.
  • The search pool contains 300 candidates, starts from 20 root columns, and allows at most 40 columns per state.

C Reproduction

The committed snapshot supports full regeneration of the paper’s numbers, tables, and figures through documented commands, with fixed seeds and recorded budgeted-search winners.

  • Running uv sync && uv run pytest && bash run all.sh regenerates every number, table, and figure from the committed data snapshot.
  • The searches are wall-clock budgeted, so exact trees are hardware-dependent; committed winners under experiments/params/ are the records used for table regeneration.
  • Full regeneration takes roughly six hours on 30 CPU cores, while refreshing the snapshot requires one additional command and no API key.
Loading 2608.23393v1…