PaperlayerView exact revision on arXiv

Source-linked AI summary

Jacobian Rank Collapse in Decision-Focused Learning

Aojie Yuan, Haiyue Zhang, Zijian Su

arXiv:2609.39261v1cs.LGq-fin.PM

TL;DR

Decision-focused learning may not give a predictor an independent parameter-update direction, so the paper characterizes how the predictor Jacobian restricts available learning directions. It combines geometric analysis with financial and synthetic experiments; across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%.

  • Problem

    A decision-relevant error direction helps only if the predictor can express an update along it, motivating study of this restriction in decision-focused learning.

    Source: §1 · p. 2

  • Method

    The paper analyzes predictor-Jacobian geometry, sparse index-tracking support, and batch-subspace behavior to distinguish output signals from trainable parameter directions.

    Source: §1 · p. 2

  • Results

    Across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Pointwise rank restrictions do not establish common minimizers or collinear batch updates, and rank collapse does not rule out different signs or final decisions.

    Source: §1 · p. 2

    Chronological folds may be dependent, and unadjusted signed-rank p-values do not establish generalization across markets or tasks.

    Source: §6.2 · p. 8

Abstract

from arXiv · show

Decision-focused learning (DFL) trains predictors through downstream objectives, but a different loss need not provide an independent parameter-update direction. We characterize this restriction through the predictor Jacobian, using sparse index tracking to distinguish the covariance entries read by the optimizer from the parameter directions available to learning. Rank-one Jacobians make nonzero per-example gradients collinear; a conditional spectral bound describes near-collinearity. A batch-subspace characterization and counterexamples show why these local statements imply neither common minimizers nor collinear batch updates. Experiments examine when geometry translates into decision quality. Across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%; a 385-parameter conditional predictor also has pointwise rank one. In validation-tuned shortest-path and knapsack experiments, full-capacity SPO+ reduces mean regret by 11.6% and 10.6%, respectively; only knapsack survives correction across eight comparisons. The capacity contrast persists on fresh datasets across batch orders and training budgets. Holding expressivity fixed, invertible coordinate scaling lowers spectral effective rank and ordinary SGD gains; compensating for the scaling restores the original trajectories. Financial forward-target controls separate forecast accuracy from decision quality; a matched neural comparison finds no aggregate DFL advantage in the tested architecture. These findings distinguish local rank restrictions, coordinate-dependent optimization and predictive accuracy. Predictor geometry helps explain available learning directions, while held-out decision quality remains the test of practical benefit.

1 Introduction

The paper studies whether a predictor can express useful parameter updates from a decision-focused objective, separating local Jacobian geometry from empirical decision quality. It tests this distinction in financial and synthetic settings, with coordinate effects and validation choices delimiting the claims.

1 Introduction

  • At a fixed input, rank-one predictor Jacobians place nonzero MSE and task-gradient images on one line, while a conditional spectral bound describes near-collinearity.Rank collapse excludes a new independent direction, but not different signs, stationary points, or final decisions.Source: §1 · p. 2
  • Across 38 one-parameter equity configurations, DFL gains over MSE remain below 1.8%, while a 385-parameter conditional predictor also has pointwise rank one.The financial baseline reconstructs trailing covariance rather than forecasting future covariance.Source: §1 · p. 2
  • The experiments use matched initializers, separate validation, exact decision oracles, and a fresh-data follow-up varying minibatch order and training budget.Either loss can retain the unchanged baseline in the follow-up.Source: §1 · p. 2
  • A support-energy inequality, spectral bound, and batch-subspace characterization distinguish decision-observed outputs from parameter directions; counterexamples rule out inferring common minimizers or collinear batch updates from pointwise rank one.These are separate geometric restrictions, not a claim that every decision-relevant direction is lost.Source: §1 · p. 2
  • An invertible reparameterization holds expressivity fixed while spectral rank and ordinary SGD gains fall together; compensating for the scaling restores the original updates.This distinguishes a coordinate-dependent optimization effect from lost expressivity.Source: §1 · p. 2
  • The matched neural forward-target comparison reports a small aggregate difference, and validation often selects the unchanged initializer; regularization coefficients and benefit thresholds remain validation choices.Dynamic-selection experiments and negative results delimit the claims.Source: §1 · p. 3

2 Related Work

Related work spans sparse portfolio construction, decision-focused learning methods, and portfolio optimization or covariance estimation. This paper focuses on how the predictor Jacobian maps output-space signals into trainable parameter directions.

2 Related Work

  • Sparse portfolio construction includes evolutionary heuristics, mixed-integer programming, ℓ1-penalized formulations, and cardinality-constrained optimization (Xu et al., 2016).The paper focuses instead on training covariance predictors through resulting decisions.Source: §2 · p. 3
  • DFL approaches include differentiable QPs (Donti et al., 2017; Amos and Kolter, 2017; Agrawal et al., 2019), SPO+, combinatorial surrogates (Wilder et al., 2019; Vlastelica et al., 2020), and implicit differentiation (Blondel et al., 2022; Paulus et al., 2024).Other cited work studies guarantees, regret gradients, regret decomposition, online DFL, and prediction inflation.Source: §2 · p. 3
  • Prior work covers end-to-end portfolio construction and high-dimensional covariance estimation, while concurrent work by Jeon et al. (2026) applies DFL to sparse tangent portfolios and reports gains in larger universes.That concurrent study is described as complementary to this paper’s focus on local gradient geometry and low-capacity predictors.Source: §2 · p. 3

3 Problem Formulation

The formulation decomposes sparse index tracking into subset selection and a weight-optimization QP. Because selection is detached, the QP reads only a restricted set of covariance entries.

3 Problem Formulation

  • Tracking-error minimization uses the quadratic objective min_w (w−widx)⊤Σ(w−widx), and the joint selection-and-weighting problem is NP-hard, motivating a two-stage decomposition.Source: §3 · p. 3
  • The first stage selects K assets by market-cap ranking by default or by a tracking score that depends on estimated covariance.Under tracking-score selection, off-block estimation errors can propagate to stock selection.Source: §3 · p. 3
  • With selection detached, the QP reads at most 2NK − K^2 of N^2 covariance entries; Figure 2 contrasts direct MSE supervision with task-loss differentiation through the QP.Both objectives reach parameters through J⊤.Source: §3 · p. 3, Figure 2 · p. 4

4 Methodology

The methodology combines a two-stage tracking baseline with differentiable QP training and a support-aware regularizer. It evaluates multiple covariance-predictor capacities and gives a uniform bound on selected QP inputs, subject to solution-stability assumptions for decision guarantees.

4.1 Two-Stage Baseline and DFL

  • The financial baseline uses the preceding 63 trading days’ sample covariance as a trailing reconstruction target, not a future covariance label; the QP is differentiated through KKT conditions with selection detached.For shrinkage families, the same sample covariance is also the input, so the baseline is not an optimally specified supervised forecasting model.Source: §4.1 · p. 4

4.2 Block-Diagonal DFL (BD-DFL)

  • Detached selection restricts the task gradient to the current support, leaving errors outside it unpenalized even though they may matter after selection changes.The figure illustrates the support restriction and how rank-one Jacobians map nonzero gradients onto one line.Source: §4.2 · p. 4, Figure 3 · p. 5
  • BD-DFL adds relative MSE regularization, with β = 0 recovering pure DFL and larger β emphasizing covariance fit; β is chosen empirically.The penalty is motivated by entries outside today’s support potentially entering tomorrow’s QP; the cited DRO principles do not establish the model-specific swap-dependent coefficient.Source: §4.2 · p. 4
  • For every error matrix and subset, the Frobenius penalty bounds both selected-block and selected-row QP input perturbations; turning this into a decision or regret bound requires solution-stability assumptions.Source: §4.2 · p. 5

4.3 Covariance Models

  • The covariance models range from one-parameter shrinkage and factor models through neural and structured predictors to a 385-parameter conditional predictor.Training uses Adam, validation-TE early stopping, and cvxpylayers for QP differentiation.Source: §4.3 · p. 5

5 Local Geometry and Batch Limits

The section separates support restrictions in covariance space from the predictor Jacobian’s restrictions on parameter updates. It then shows why per-example rank statements do not determine batch-gradient geometry or shared minimizers.

5.1 Gradient Alignment Theory

  • The support bound depends on the task-relevant covariance entries and an assumption about MSE gradient energy there; sharpness for unrestricted vectors does not establish attainability by the tracking QP.The QP reads only selected covariance blocks, while MSE can place gradient energy elsewhere.Source: §5.1 · pp. 5–6

5.2 Predictor Jacobian and Gradient Alignment

  • At rank one, nonzero parameter-gradient images are collinear, while near-rank-one alignment depends on the spectral gap and nonzero leading gradient components.The bound does not rule out different signs, stationary points, or final decisions.Source: §5.2 · p. 6
  • Spectral effective rank, angular alignment, and the heterogeneity-based proxy are distinct; concentrated singular values alone do not ensure a task gradient has a nonzero leading component.The stated corollary requires the projected gradients to be non-collinear, not merely a Jacobian rank greater than one.Source: §5.2 · p. 7

5.3 From a Single Example to a Batch

  • Across examples, all batch gradients lie in the shared subspace spanned by the stacked Jacobians, and all possible nonzero batch gradients are collinear exactly when that subspace has rank one.Pointwise rank one is insufficient unless the examples’ nonzero row spaces share a common line.Source: §5.3 · p. 7
  • Counterexamples show that pointwise rank one implies neither common minimizers nor collinear batch updates.A scalar predictor can have different loss minimizers, while two rank-one example Jacobians can yield orthogonal batch gradients.Source: §5.3 · p. 7
  • The 385-parameter conditional predictor is rank one at each measured input, but this does not establish a one-dimensional training problem or guarantee test loss.Its scalar shrinkage intensity can have gradients that vary across examples; the analysis concerns raw Euclidean gradients at a fixed parameterization.Source: §5.3 · p. 7

6 Experimental Setup

The experiments use equity, spatial, and combinatorial settings with specified data protocols and comparison methods. Results are organized around matched folds or seeds, with dependence and evidence provenance explicitly qualified.

6.1 Data

  • The equity setup uses daily returns and rolling chronological train, validation, and test windows, with additional market, universe-size, and cross-domain evaluations.The primary neural evaluation uses nine folds covering 2016–2025; the main universe contains 97 stocks after a coverage filter.Source: §6.1 · p. 7

6.2 Comparison Units and Evidence Provenance

  • Comparisons use matched folds or seeds within experiments, not individual daily returns, and shared training histories limit generalization claims across markets or tasks.Unadjusted signed-rank p-values do not establish generalization; only explicit TOST margins support equivalence interpretations.Source: §6.2 · p. 8

6.3 Baselines

  • The baselines include portfolio heuristics, covariance estimators, validation-tuned decisions, DFL methods, and MSE-trained factor or neural predictors.The listed primary metric is annualized tracking error.Source: §6.3 · p. 8

7 Results

In the reported low-capacity equity settings, DFL brings little measured benefit despite substantially higher training cost. The one-parameter comparisons also meet the stated equivalence criterion.

7.1 Small Gains in One-Parameter Equity Models

  • ∼3,300× training-cost overhead accompanies little measured benefit from DFL in the tested low-capacity settings.At N=100, fitted one-parameter models achieve similar test performance across training objectives.Source: §7.1 · p. 8
  • All 12 pairwise comparisons among ValTuned, SPO+, and DFL meet the TOST criterion with a 20-bps margin.The table covers three pairs across four sparsity levels; POET and GLasso are different model families shown for reference.Source: Table 1 · p. 9
  • The reported 2K/N bound is vacuous, and the section claims no equivalence with the referenced baselines.Source: §7.1 · p. 9

K LW POET GLasso ValTuned SPO+ DFL

The experiments find modest financial DFL gains and a capacity-dependent contrast in synthetic tasks, while showing that predictor geometry, optimization coordinates, and forecast accuracy do not alone determine decision quality.

K LW POET GLasso ValTuned SPO+ DFL

  • Table 1 finds TOST equivalence within a 20-bps margin for all 12 pairwise comparisons among ValTuned, SPO+ and DFL, while each improves on Ledoit–Wolf by 7–19%.LODL also matches DFL, indicating that explicit KKT differentiation is not necessary to obtain this performance.Source: K LW POET GLasso ValTuned SPO+ DFL · p. 9

7.2 Architecture and Loss Choice at Larger Scale

  • At N=478, switching from MSE to DFL improves TE by 0.68%, 1.07% and 1.34% for the 1-, 12- and 385-parameter models, respectively, smaller than architecture-related differences.The 385-parameter conditional predictor remains pointwise rank one; five folds make p = 0.031 the smallest attainable one-sided Wilcoxon value.Source: §7.2 · p. 9, Table 2 · p. 9

7.3 Cross-Market Evidence for One-Parameter Models

  • Across 38 equity configurations, one-parameter DFL gains range from −0.57% to +1.76%, with no detected monotone association with sparsity.Structured-model reductions are 2.95% on ASX, 2.77% on Hang Seng, and 2.41% on S&P 1500; gains on Euro Stoxx, Nikkei and FTSE are below 1%.Source: §7.3 · pp. 9–10, Table 3 · p. 10

7.4 BD-DFL under Dynamic Selection

  • Under dynamic tracking-score selection, neural DFL increases mean TE by 8.6% at N=100 and K=10, while BD-DFL with β = 0.001 reduces it by 8.2%.Results vary with model, selection rule, and configuration, motivating validation of regularization rather than a universal benefit or coefficient.Source: §7.4 · p. 10

7.5 Limits of Transfer

  • Earlier transfer evidence has limits: the heterogeneity-based capacity proxy depends on the partition and is not a Jacobian rank, while NOAA and EPA have only one or two evaluation windows per configuration.Older shortest-path sweeps measure the input Jacobian rather than the parameter Jacobian; the sampled PyEPO benefit threshold does not transfer from equities.Source: §7.5 · p. 10

7.6 Validation-Tuned Transfer Beyond Finance

  • Full-capacity SPO+ reduces mean test regret by 11.59% on shortest path and 10.59% on knapsack; only full-capacity knapsack passes Holm correction across eight comparisons.On fresh datasets, full-capacity gains persist across batch orders and 20/80-epoch budgets, while scalar gains stay below 0.6%.Source: §7.6 · pp. 10–11, Figure 5 · p. 11, Figure 6 · p. 11

7.7 Holding Expressivity Fixed

  • Reducing ε from 1 to 0.01 lowers mean SPO+ regret reduction from 11.79% to 1.19% for path and from 10.17% to 0.52% for knapsack; compensation restores identity-coordinate trajectories.The invertible scaling preserves expressivity and exact rank while lowering spectral effective rank; this identifies a removable coordinate effect, not a rank-only causal effect.Source: §7.7 · p. 12, Figure 7 · p. 12

7.8 Forecast Accuracy Is a Separate Test

  • Future-target MSE reduces relative covariance error by 37.25% but increases mean tracking error by 4.13%; a matched neural comparison finds DFL 0.07% worse in relative mean annualized TE.Task-validation tuning reduces mean tracking error by 1.46%, and Ledoit–Wolf by 0.69%; the neural comparison is neither an equivalence test nor evidence about all neural predictors.Source: §7.8 · p. 12

8 Conclusion

The conclusion separates local rank restrictions from training outcomes and highlights coordinate-dependent optimization and forecast-to-decision gaps. It calls for broader tests before treating these findings as robust.

8 Conclusion

  • Rank one restricts local directions without fixing training outcomes; decision support, the predictor Jacobian and batch aggregation impose distinct restrictions.Capacity-dependent gains persist across tested batch orders and budgets, with stronger statistical evidence for knapsack.Source: §8 · pp. 12–13

Ethics statement

  • The financial tracking metrics do not establish profitability, deployment safety, or robustness to future market conditions.Backtests can be misinterpreted as prospective investment evidence.Source: Ethics statement · p. 13

AI use statement

  • AI tools assisted with analysis, auditing, literature checks, drafting, revision and figure design; the authors remain responsible for the scientific content.AI-assisted auditing identified evaluation-harness defects and unsupported theoretical steps, prompting corrections and narrower claims.Source: AI use statement · p. 13

Reproducibility statement

  • The release includes saved results, editable figures, numerical checks and rerunnable studies, but historical real-data training has not been independently repeated.The equity comparisons use a repaired harness; archived diagnostic scores are not prospective tests of its corrected results.Source: Reproducibility statement · p. 13

Appendix Guide

  • The appendices separate proofs and counterexamples, provenance and costs, equity comparisons, transfer diagnostics, combinatorial experiments, and controls for expressivity, coordinates and predictive targets.The neural future-target comparison shows no aggregate DFL gain in its tested architecture, but is descriptive rather than an equivalence test.Source: Appendix Guide · p. 16

A.1 Proof of Proposition 4 (Jacobian Rank Collapse)

The appendix separates local gradient alignment from broader optimization outcomes: Jacobian rank restricts available directions, while support bounds and counterexamples specify what that restriction does—and does not—imply. Its tracking-QP results are conditional on explicit energy or stability assumptions, and the selection-robustness argument does not establish a universal swap-dependent coefficient.

A.1 Proof of Proposition 4 (Jacobian Rank Collapse)

  • For d=1, any two nonzero parameter gradients are collinear with cosine ±1, but rank collapse does not determine whether their signs agree.Under the stated unimodality, smoothness, and nearby-minimizer conditions, the losses are concordant and the loss gap is bounded by Lδ²/2.Source: §A.1 · pp. 16–17
  • Across 20 folds, task loss was unimodal in 20/20, and the MSE–task minimizer gap averaged 0.154 with median 0.040.The tested conditions used four markets, K=20, and α∈[0.01, 0.99].Source: §A.1 · p. 17
  • Across 38 one-parameter configurations, DFL gains over MSE stayed below 1.8% with median 0.25%, and did not grow with K/N (ρ=−0.16, p=0.35).This empirical pattern is compatible with, but not implied by, the corollary; signs, stationary points, and test losses can still vary with sparsity.Source: §A.1 · p. 17

A.2 Counterexamples and Batch Geometry

  • Rank-one Jacobians constrain local directions, but do not imply common minimizers or collinear batch updates.The appendix gives distinct scalar-loss minimizers and orthogonal batch gradients from rank-one per-example Jacobians.Source: §A.2 · p. 19

A.3 Proof of Proposition 2

  • Under normalized-energy isotropy, the expected support bound is approximately 2K/N when K≪N; without that assumption, the numerical bound need not hold.Equal unnormalized second moments alone do not suffice, especially when selection concentrates error energy.Source: §A.3 · p. 20

A.4 Sharpness of the Support-Only Inequality

  • The support-only cosine inequality is sharp for ambient vectors, but the appendix does not claim a matching lower bound realized by the tracking QP.The construction attains the linear-algebra bound, while the earlier equicorrelation construction does not establish the required tracking-gradient structure.Source: §A.4 · p. 20

A.5 Proof of Proposition 3

  • Changing estimated covariance entries outside the selected set leaves the QP decision and its evaluated regret unchanged, given a fixed selection set and true covariance.The result assumes a common tie-breaking rule when needed.Source: §A.6 · p. 21

A.6 Task Regret and Local Perturbation

  • The local tracking-variance bound requires a positive-definite selected block, small perturbations, and a shared active set; it is not a global bound across active-set changes.Its perturbations involve the selected covariance block and index cross-covariance, not estimated off-support entries.Source: §A.6 · p. 21

B.1 Evidence Provenance and Reproduction Scope

The evidence record distinguishes corrected evaluations from archived diagnostics and limits what the experiments establish. Across the tested financial settings, DFL’s gains vary with model and protocol; compute, selection, and turnover trade-offs also matter.

B.1 Evidence Provenance and Reproduction Scope

  • The corrected release separates equity evaluations, archived diagnostics, and spatial or combinatorial experiments; the 33 archived cases are not independent validation of the corrected harness.Chronological folds may share training history, and limited windows constrain cross-domain replication.Source: §B.1 · p. 22

B.2 Training Details; B.3 Computational Cost Comparison

  • The default equity trainer uses Adam with learning rate 10^-3, weight decay 10^-4, gradient clipping at 1.0, and accumulation over four rebalancing dates.In one N=100, K=20 setting, measured per-fold cost ratios were roughly 3,329× for one-parameter DFL and 804× for structured DFL versus POET; timings depend on hardware and implementation.Source: §B.2 · p. 23, §B.3 · p. 24, Table 4 · p. 24

C.1 DFL Gains across Sparsity Levels

  • 7.0–16.5%: neural DFL reduces TE at tested K ≤30; at K = 50, the estimated reduction is 0.2%.The largest reduction is at K = 20, so these finite comparisons do not establish a monotone law in K/N.Source: §C.1 · p. 24, Table 5 · p. 24, Figure 9 · p. 25

C.3 Sparsity vs. Model Capacity Interaction

  • At K = 20, DFL reduces TE by 13.9–15.9% across factor counts, versus 0.4–2.5% at K = 50.Across fixed-K columns, the spread across factor counts is at most 2.1 percentage points; the grid does not isolate a causal mechanism.Source: §C.3 · p. 26, Figure 11 · p. 27

C.4 DFL with Richer Model Classes

  • The 12-parameter structured model gains 1.2% at K=10 and 1.6% at K=20, while the 385-parameter conditional model gains 0.7% and 1.0%.Parameter count alone does not predict relative gain, and the conditional model trains less stably across folds.Source: §C.4 · p. 27, Table 8 · p. 27

C.5 Cross-Market Dim-1 Validation; C.6 Cross-Benchmark Validation (Russell 2000 / Tech Sector)

  • Across 38 one-parameter equity configurations, gains over MSE remain below 1.8%; additional tested small-cap and sector benchmarks each show gains under 0.3%.The cross-market gains are unordered in K/N, an empirical observation rather than a consequence of the rank result.Source: §C.5 · p. 28, Table 9 · p. 28, Table 11 · p. 29

C.7 BD-DFL Detailed Results

  • At N = 451, corrected results show unregularized DFL improves at K = 10, with regularization adding a smaller further reduction; at K = 20, regularization’s direction is favorable but not significant at 5%.The corrected results differ from the earlier harness; β = 0.001 is empirical.Source: §C.7 · p. 30

C.9 Transaction Cost-Aware Optimization

  • At c = 100 bps, transaction costs add 4 bps at K=10 and 3 bps at K=20, against TE reductions of 49 and 57 bps, respectively.The tested DFL policies have higher turnover, but lower net cost at both cost levels and both sparsity settings.Source: §C.9 · p. 31

C.10 Design Choices and Robustness

Robustness checks report DFL advantages across several tested covariance windows, rebalance schedules, and position caps, while also identifying settings where added training complexity does not help. The comparisons remain specific to the evaluated protocols and do not establish prospective investment performance.

C.10 Design Choices and Robustness

  • DFL gains are +12.8% for 21-, 42-, and 63-day covariance windows and +13.9% for 126 days; gains are also similar across weekly, biweekly, and monthly rebalancing.The rebalancing comparisons report +14.9%, +14.4%, and +14.7%, respectively; those runs use their stated fold protocols.Source: §C.10 · pp. 32–33
  • With position caps of 10% and 20%, DFL gains are +12.7% and +14.7%, respectively, compared with +14.7% unconstrained.The tighter cap raises both MSE and DFL tracking errors while preserving the relative advantage in these tests.Source: §C.10 · p. 33

C.11 Negative Results: Multi-Step Lookahead and Curriculum Learning

  • Multi-step lookahead keeps TE within 0.1% of single-step DFL, while the tested MSE-to-task curriculum yields identical final TE to direct task-loss training.The finite ablations do not identify why either extension failed to improve results.Source: §C.11 · p. 33

C.12 Value of the QP Structure

  • At K=20, direct weight prediction has TE = 0.0589, compared with 0.0276 for the tested QP pipeline.At K=10, the corresponding values are 0.0806 and 0.0462; the comparison also changes the parameterization and optimization problem.Source: §C.12 · p. 33

C.13 Risk Metrics and Domain-Specific Analysis

  • DFL’s reported tracking-risk measures describe evaluated folds, not prospective investment performance; the released results also do not support a factor-neutrality claim.Regime variation is descriptive and does not identify covariance stability as its cause.Source: §C.13 · p. 34, Table 18 · p. 34

D.1 Practitioner Diagnostic

The practitioner diagnostic treats heterogeneity as a heuristic, not a universal performance threshold. Direct bound tests validate the rank-one endpoint but find the bound vacuous for the sampled structured model; synthetic and spatial experiments remain exploratory.

D.1 Practitioner Diagnostic

  • dproxy = dh scales parameter count by partition-based index-weight heterogeneity, but is distinct from measured Jacobian rank and does not establish a universal performance threshold.Equity experiments used h = 1 by convention; measured universe values ranged from 0.31 to 0.71.Source: §D.1 · pp. 34–36, Table 20 · p. 36
  • The archived diagnostic records 27 correct predictions among 31 committed cases and two abstentions; post-hoc rescoring with measured heterogeneity gives 25/28, not a paired accuracy comparison.The archived score predates the corrected equity harness, and the revised rule abstains more often.Source: §D.1 · pp. 35–36
  • Across 38 one-parameter equity configurations, gains remain below 1.8%, and measured heterogeneity, sparsity, and universe size do not explain the larger cases.The proposed attribution to heterogeneity does not survive the additional ASX result.Source: §D.1 · pp. 36–37

D.2 Testing the Spectral-Gap Bound Directly

  • Across 162 points, the spectral-gap bound has no recorded violations and is informative at 108 rank-one or near-rank-one points, but vacuous at all 54 structured-model points.For the one-parameter model, the bound and realised alignment are both 1 at all 54 points; for the structured model, task gradients are nearly orthogonal to the leading singular direction.Source: §D.2 · p. 37
  • The 385-parameter conditional predictor is pointwise rank one, but changing input-specific shrinkage gradients mean this does not determine stacked rank or test-gain ordering.The bound's intermediate regime is carried by the empirical diagnostic, not its quantitative guarantee.Source: §D.2 · p. 37

D.3 Synthetic dproxy Sweep

  • In the synthetic sweep, five of six settings above dproxy = 9 have mean gains of 2.5–4.7%, while the final setting returns to approximately 1.0%.Baseline tracking error also changes, so the sweep does not isolate effective dimension or establish a sharp universal transition.Source: §D.3 · pp. 37–38

D.4 Generalization beyond Portfolio Optimization

  • In sensor placement, all ten configurations report lower tracking error under DFL, while two measured alignments exceed the isotropic reference.The deterministic support-energy inequality remains the appropriate statement without the additional isotropy assumption.Source: §D.4 · p. 38
  • The exploratory spatial contrast reports negative changes for NOAA at dproxy = 1.4 and positive changes for EPA at dproxy = 6.1, but few evaluation windows preclude strong inference.These domains also differ in observations, targets, and index-weight heterogeneity.Source: §D.4 · p. 39, Table 23 · p. 40

E.1 Shortest Path Experiment Details

This appendix section examines how rank measurements, parameter-update constraints, capacity, validation selection, coordinates, and target selection relate to decision regret and forecast metrics. Its controlled results find gains concentrated in full-capacity settings, with adjusted support for knapsack but not shortest path; shifts in scalar-family forecast metrics do not establish that DFL dominates forecasting.

E.1 Shortest Path Experiment Details

  • On the 12×12 grid, SPO+ advantage reached +16.4%, a 4× amplification over the 8×8 grid; tests were unadjusted and the larger grid does not isolate the mechanism.Wider bottlenecks sometimes lowered prediction error while increasing regret. The measured input-Jacobian rank does not establish parameter-Jacobian rank; bottleneck width bounds input rank only.Source: §E.1 · pp. 41–42, Table 25 · p. 42

E.2 External Validation on PyEPO

  • Across a 45× measured-capacity range, PyEPO SPO+ gains rose from −0.25% at reff=1.00 to +12.99% at full rank (ρ=+0.93, p=0.0025).The reff ≥4 threshold did not transfer: gain was +0.72% at reff=7.02. The full-capacity result was +12.99% (p=0.001, 10/10 seeds).Source: §E.2 · p. 43, Table 26 · p. 43

E.3 Validation-Tuned Controlled Extension

  • Full-capacity SPO+ gains had unadjusted bootstrap intervals of [5.49, 16.81]% for shortest path and [7.30, 13.81]% for knapsack; only knapsack’s adjusted tests passed.All ten knapsack dataset pairs favored full-capacity SPO+, compared with eight of ten shortest-path pairs; shortest path had adjusted p = 0.0684.Source: §E.3 · p. 46, Table 28 · p. 46

E.4 Training Budget and Batch-Order Sensitivity

  • 12.12% to 12.54% was the full-capacity shortest-path gain across the tested budgets, while knapsack gains were 13.67% to 13.76%; knapsack passed adjusted tests at both budgets.Scalar gains stayed below 0.6%; limited sensitivity across these budgets is not proof of convergence, and nonsignificance is not equivalence.Source: §E.4 · p. 47, Table 29 · p. 47

F.1 Invertible Coordinates at Fixed Expressivity

  • With the function class fixed, compensating scaled-coordinate gradients by ε exactly recovers identity-coordinate SGD predictor trajectories under plain SGD.Exact rank is unchanged; the control does not separate spectral rank from conditioning and does not claim equivalence for adaptive optimizers.Source: §F.1 · pp. 47–48, Table 30 · p. 48

F.2 Chronological Forward-Target Financial Baselines

  • On a survivorship-biased 100-stock, 2005–2025 series, the forward-target control fits scalar covariance shrinkage to future covariance and selects variants by validation tracking error.It uses ten annual test folds from 2016–2025, which share histories and are unsuitable for independence-based significance claims.Source: §F.2 · pp. 48–49
  • The selected future-target estimator improves mean relative covariance error by 37.25% yet worsens mean TE by 4.13%; task validation improves mean TE by 1.46%, and Ledoit–Wolf by 0.69%.The estimator beats reconstruction in six years, but win counts alone miss the magnitude of its failures.Source: §F.2 · p. 49
  • These results show target sensitivity within one scalar family, not that reconstruction is the strongest predictive baseline or that DFL dominates forecasting.Point-in-time stock membership, transaction costs, and broader forecast architectures remain needed.Source: §F.2 · p. 49

F.3 Matched Neural Forward-Target Comparison

The matched forward-target comparison finds no aggregate DFL advantage in its tested neural family, while the unresolved findings limit how broadly the paper’s empirical patterns can be interpreted.

F.3 Matched Neural Forward-Target Comparison

  • The matched experiment uses a 197-parameter residual covariance predictor and compares future-covariance MSE with a tracking-return DFL objective under shared data, selection, and training budgets.The DFL objective minimizes mean squared tracking returns, whereas MSE minimizes squared covariance-entry errors; this restricted neural family does not establish behavior for unrestricted covariance forecasters.Source: §F.3 · p. 50
  • Mean TE was 5.711% for future-target MSE and 5.715% for DFL, a 0.07% relative increase for DFL; this descriptive result is not evidence of equivalence.Validation selected the unchanged initializer for 17/30 MSE models and 12/30 DFL models; the comparison cannot explain differences from historical financial results because architecture, support, and evaluation protocol differ.Source: §F.3 · pp. 50–51, Table 32 · p. 51

G.1 Unresolved Empirical Findings

  • Only three of 38 one-parameter equity configurations exceeded a 1% DFL gain, and the equity benefit threshold did not transfer to shortest path.The shortest-path SPO+ gain at sampled stacked rank 7.02 was 0.72%; a strong rank–gain ordering did not validate a shared cut point.Source: §G.1 · p. 52
  • Architecture rankings differed across equity universes, and pointwise rank one did not resolve the difference because input-dependent directions can span a larger batch subspace.The largest displayed gain was for the structured model at N = 100, while the conditional model had the largest relative reduction at N = 478.Source: §G.1 · p. 52
  • With full-capacity hyperparameters frozen, perturbedOpt lost to MSE at every intermediate sampled rank, then recovered to a 4.98% gain at full rank.At the lowest rank, the estimated change was −2.32% (p = 0.19), which does not establish equivalence; capacity-specific tuning is needed to separate optimization from geometry.Source: §G.1 · p. 52
  • Negative-identity training gave a 2.24% full-capacity gain but exceeded 1.5× baseline regret at reduced capacity, leaving the source of instability unresolved.The passage retains this as an experimental failure rather than excluding it by the geometric argument.Source: §G.1 · p. 52
  • The archived knapsack sweep had an imprecise 1.85% full-capacity gain and no detectable monotone capacity trend; spatial replication was also limited.NOAA and EPA mostly provide one evaluation window per configuration, so Table 23 supports only a descriptive contrast.Source: §G.1 · pp. 52–53
  • The matched neural control cannot identify why it differs from historical financial gains, and its shared annual histories and absent equivalence test limit inference.The paper calls for broader architectures, more training, and point-in-time membership before attributing the result to objective choice or predictor geometry.Source: §G.1 · p. 53

Related papers

Loading 2609.39261v1…