Source-linked AI summary
End-to-End Neural Shrinkage of Indefinite Pairwise Correlation Matrices for Small-Cap-Inclusive Portfolios
Christian Bongiorno, Lorenzo Villassero
TL;DR
Incomplete equity return panels make pairwise covariance estimates useful but potentially indefinite, limiting direct optimization and standard shrinkage. The paper adapts a rotation-invariant neural estimator to process signed spectra with overlap-aware information and trains it for realized GMV risk. Across 2000–2025, the estimator delivers the strongest reported risk-adjusted and realized-risk performance after modeled execution costs.
Problem
Pairwise-complete estimation preserves unequal return histories but can produce indefinite correlation matrices, while standard RMT shrinkage assumes a common rectangular panel.
Method
The estimator conditions a rotation-invariant spectral map on factor-aligned overlap information, maps signed eigenvalues to a positive spectrum, and trains end-to-end on realized five-session GMV variance.
Results
20.7% lower annualized five-session realized volatility than Factor, with RIEnet the only method retained by the 0.1% Model Confidence Set after modeled execution costs.
Takeaways & Limitations
RIEnet preserves its lowest-realized-risk and highest-risk-adjusted-performance advantage within the full portfolio-construction and execution pipeline.
Takeaways & Limitations
Component-level attribution on controlled synthetic benchmarks is left for future work.
Abstract
from arXiv · showhide
Small-cap-inclusive equity universes contain recently listed and intermittently traded securities, so enforcing a common look-back discards a substantial fraction of the available information. Pairwise-complete estimation preserves the longest overlap for each asset pair, but the resulting correlation matrix can be indefinite because its entries are computed on different samples. This prevents direct use in Markowitz optimization and falls outside the assumptions of standard random-matrix shrinkage. We adapt a rotation-invariant neural covariance estimator to this setting. The model computes mask-aware marginal moments and a pairwise correlation matrix proxy, processes its signed spectrum, and uses a bidirectional gated recurrent unit conditioned on factor-aligned effective sample lengths derived from the overlap matrix and eigenvector loadings. It maps all eigenvalues, including negative ones, to a positive inverse spectrum. The reconstructed covariance is positive definite and is trained end-to-end to minimize five-day realized global-minimum-variance risk. We evaluate 26 expanding-window models from 2000 to 2025 on up to 1,500 U.S. equities in a closing-auction simulator with point-in-time selection, commissions, financing, corporate actions, and market impact. Across the 26-year out-of-sample period, the neural estimator reduces annualized five-day volatility by approximately 20\% and increases the Sharpe ratio by approximately 40\% relative to the next-best covariance estimator. These improvements are consistent across realized risk, risk-adjusted performance, and drawdown control, remain after the modeled execution frictions, and are supported by a 99.9\% Model Confidence Set that retains only the neural estimator.
1 Introduction
Incomplete return histories make common-panel covariance estimation impractical for small-cap-inclusive universes. Pairwise estimation preserves information but can produce indefinite matrices outside standard RMT assumptions, motivating a neural extension that uses signed spectra and overlap information.
- Short or fragmented histories arise for recent IPOs, lower-capitalization securities, suspended stocks, and assets with incomplete coverage.
- Pairwise-complete estimation uses each asset pair’s maximum shared history, preserving more information than complete-case deletion.
- Different overlap samples can make the assembled pairwise matrix indefinite, with negative eigenvalues implying negative variance for some portfolios.
- Standard RMT shrinkage does not provide a principled rule because pairwise entries have heterogeneous sample sizes and the matrix may be indefinite.
- The proposed extension supplies signed eigenvalues and factor-specific sample-information measures to a rotation-invariant recurrent estimator.
- Evaluation uses up to 1,500 U.S. stocks from 2000 through 2025 in a broker simulation incorporating trading costs and market impact.
2 Indefinite Pairwise Risk Estimation
The method builds marginally standardized pairwise cross-moments from unequal overlaps, accepting that the resulting matrix may be indefinite. Neural regularization is then required to map its signed spectrum to a positive, usable covariance estimate.
- The return panel and validity mask identify observed entries, enabling overlap-aware estimation with incomplete histories.
- Marginal moments are estimated once for each asset and reused across every pair rather than recomputed separately on each overlap.
- The resulting pairwise cross-moment matrix is formed by combining marginal scales with the pairwise correlation proxy.
- Because entries use different subsets and denominators, the symmetric unit-diagonal correlation proxy may have negative eigenvalues and fail to be positive semidefinite.
- Marginal scaling alone cannot repair indefiniteness because the covariance and correlation forms have the same inertia.
- The neural covariance estimate is used in a common long-only GMV optimization after regularization.
- The required correction maps the signed spectrum to a positive regularized spectrum while retaining heterogeneous-overlap information.
3 RIEnet for Incomplete Return Panels
RIEnet transforms masked returns into marginal volatility and correlation representations, then processes overlap-conditioned spectral factors to reconstruct a positive-definite covariance matrix. Training differentiates through this pipeline using realized GMV risk while separating estimator learning from deployment constraints.
- 3.1 Lag Transformation: A shared lag transformation preserves the validity mask and supplies the common input to the marginal-volatility and correlation branches.The transformation learns lag weighting and soft clipping while ensuring invalid entries do not become observations.
- 3.2 Marginal Volatility: The marginal-volatility branch applies asset-specific history-length scaling to account for heterogeneous sampling uncertainty.
- 3.3 Correlation Estimation: The correlation branch computes signed eigenvalues and eigenvectors, then transfers pairwise overlap information to each spectral factor.
- 3.3 Correlation Estimation: Each factor’s effective sample length is a squared-loading-weighted average of pairwise overlap counts and equals the input-panel length for complete data.
- 3.3 Correlation Estimation: A bidirectional GRU processes ordered spectral tokens and produces positive inverse eigenvalues from the signed spectral representation.
- 3.4 End-to-End Training: The reconstructed correlation is positive definite, and combining it with positive marginal scales yields a valid covariance estimate.
- 3.4 End-to-End Training: The network is trained end-to-end by differentiating through lag transformation and covariance reconstruction while minimizing realized five-session GMV variance.
- 3.4 End-to-End Training: Training uses variable dimensions of 50–500 assets and 600–1,200 requested days, while tests extend to 1,500 assets across 26 expanding-window models.
4 Experimental Setup
The experiment compares incomplete-panel and complete-row covariance estimators in a point-in-time, small-cap-inclusive U.S. equity portfolio simulation with realistic execution costs. All methods use the same long-only GMV optimizer and five-session rebalancing protocol.
- Data and portfolio formation: The dataset covers point-in-time U.S. equities and ADRs from 1990–2025, with up to 1,500 stocks selected using only information available at each investment date.Securities must satisfy price, shares-outstanding, listing, history, and volatility filters.
- Data and portfolio formation: Signals are produced every five trading days from a 1,200-day return look-back and validity mask, excluding execution-day returns to avoid look-ahead bias.Target weights are rounded to 0.1% increments and renormalized before simulation.
- Compared estimators: Compared estimators include Anderson, Bootstrap, Factor, and Ridge for incomplete panels, plus MLE and QIS applied to complete-row subpanels.QIS relies on a common rectangular panel with a defined observation count, unlike pairwise and imputation-based inputs.
- Compared estimators: All covariance estimators feed the same long-only GMV optimizer, while Equal Weight provides an allocation-free benchmark; capitalization weighting is omitted to preserve small-cap exposure.The omission avoids mechanically concentrating the portfolio in the largest firms.
- Execution protocol: The broker simulator begins with USD 1 million and processes holdings, corporate actions, closing rebalances, settlement, marking, and financing chronologically.Target holdings are integer shares sized using a strict pre-execution NLV estimate based on opening prices.
- Execution protocol: Execution costs include tiered commissions, regulatory fees, financing, dividends, splits, reorganizations, and fractional-share cash-in-lieu.Market impact uses closing-auction order size, prior ten-day average daily volume, price, size bucket, and exchange-specific coefficients.
- Evaluation: Table 1 reports net out-of-sample performance for January 2000–December 2025, with annualized non-overlapping five-session volatility as the primary endpoint.Methods are ordered by increasing five-session volatility, and all figures are net of execution costs.
5 Results
After broker-simulated execution and modeled costs, RIEnet delivers the strongest realized-risk and risk-adjusted results across the 26-year out-of-sample period. Its advantage persists alongside lower drawdown and broad market exposure, despite non-minimal implementation drag.
- Realized risk: 11.17% annualized five-session volatility makes RIEnet the lowest-risk estimator, versus 14.08% for Factor, a 20.7% reduction.RIEnet is the only method retained by the 0.1% MCS across all tested block-length sensitivities.
- Risk-adjusted performance: 9.30% CAGR and 0.814 Sharpe ratio are RIEnet’s highest values, with Sharpe approximately 40% above the next-highest ratio.The comparisons are 8.07% and 0.560 for QIS and 7.33% and 0.580 for Factor.
- Risk-adjusted performance: −41.3% is RIEnet’s least severe maximum drawdown among covariance estimators, versus alternatives ranging from −49.4% to −70.0%.Figure 1 indicates a stable advantage across the full 26-year experiment and principal market drawdowns.
- Market exposure: 0.500 is RIEnet’s five-session Russell 1000 beta, below Factor at 0.592, QIS at 0.799, and Equal Weight at 1.125.The performance improvement is therefore not obtained by mechanically reproducing broad capitalization-weighted equity-index exposure.
- Implementation costs: 57.6 positions is RIEnet’s median effective portfolio size, between the concentrated incomplete-panel estimators and QIS at 754.1 positions.Its 31.53-times annual turnover accompanies 2.57 basis points of impact and 0.92 basis points of explicit fees per dollar traded.
- Implementation costs: 1.49 percentage points is RIEnet’s estimated total CAGR drag, above QIS’s 1.17 but below MLE, Bootstrap, Anderson, Factor, and Ridge at 3.06–4.03 points.The estimator preserves its realized-risk and risk-adjusted-performance advantage after broker-simulated execution.
6 Discussion and Conclusions
The proposed estimator jointly addresses heterogeneous sampling and indefiniteness in incomplete return panels, while RIEnet delivers the strongest empirical performance after modeled execution costs. Its evaluation uses a fixed specification, with broader component-level attribution left for future controlled synthetic benchmarks.
- Contribution: The estimator conditions a rotation-invariant spectral map on factor-aligned overlap information and transforms signed spectra into valid, regularized covariance estimates.This jointly addresses unequal return histories and indefinite pairwise-complete correlation matrices.
- Empirical findings: RIEnet achieves the lowest realized risk and highest risk-adjusted performance among evaluated covariance estimators, remains the only method retained by the Model Confidence Set, and preserves its advantage after execution costs.The empirical results identify the treatment as economically relevant within the evaluated pipeline.
- Empirical findings: The resulting portfolios remain substantially more selective than QIS and have the lowest Russell 1000 beta among the considered covariance estimators.The gains therefore are not obtained by collapsing toward a broadly diversified allocation.
- Limitations: The empirical analysis evaluates a fixed RIEnet specification within the full portfolio-construction and execution pipeline, leaving broader component-level attribution to future controlled synthetic benchmarks.Synthetic settings would make the population covariance and missingness mechanism known and allow architectural effects to be isolated without repeatedly reusing the historical backtest.