Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction

Kunhan Guo

Source-linked AI summary

Nyström Attention Matches Full Attention for Cross-Sectional Stock Prediction

Kunhan Guo

arXiv:2609.08106v1Sep 8, 2026cs.LGq-fin.ST

TL;DR

The paper asks what MASTER’s inter-stock attention learns and whether it can be made cheaper. It decomposes the module using diagnostics, ablations, spectral analysis, and efficient-attention experiments, finding that low-rank Nyström attention matches full attention while sparsification fails. The benefits do not trivially transfer to larger, adapted universes.

Problem
The study examines what MASTER’s inter-stock attention actually learns and whether the module can be made cheaper.
Method
The paper combines statistical diagnostics, controlled ablations, spectral analysis, and efficient-attention experiments, using controlled Step ➂ variants and multi-seed TOST validation for equivalence claims.
Results
Nyström attention with m = 32 landmarks matches full attention at O(mN) cost, while graph-based sparsification degrades performance and no cross-stock module significantly outperforms a per-stock LSTM at N ≈3,500.
Takeaways & Limitations
MASTER’s inter-stock value resides in a compressible, dynamic, near-global redistribution that rewards low-rank approximation but resists sparsification.
Takeaways & Limitations
The findings are established on MASTER in a single Chinese A-share market, and the large-scale experiments use an adapted pipeline that limits direct comparability with CSI300.

Abstract

from arXiv · show

MASTER's inter-stock multi-head attention -- the module responsible for modeling cross-sectional stock relationships -- accounts for 42.5% of model parameters and 25% of predictive value. We systematically decompose this module and uncover a surprising structure: the learned attention is near-uniform (perplexity 278/300), yet forcing exact uniformity eliminates all cross-sectional discrimination. Spectral analysis resolves this paradox: the deviation from uniformity is low-rank (effective rank ~65, top-10 modes capture 96.5% of energy), explaining why sparse approximations consistently fail while Nystrom low-rank attention (m=32 landmarks) matches full O(N^2) attention at O(mN) cost -- certified equivalent via TOST at both N=300 (5 seeds, Rank IC p=0.003) and N=800 (10 seeds, Rank IC p=0.034). Additional findings include: (i) attention anti-correlates with return similarity (Spearman rho = -0.614; on the industry-labeled subset, -0.645 unconditionally and -0.627 after controlling for industry, beta, and volatility), suggesting complementarity-seeking rather than correlation mining; (ii) all graph-based alternatives degrade performance, with hard masking worse than complete module removal; and (iii) at N ~ 3,500 with adapted architectures, no cross-stock module (GCN, Nystrom, or MASTER-style pipeline) significantly outperforms a per-stock LSTM baseline (n=4 seeds), indicating that the benefits observed at smaller scales do not trivially transfer. These results establish that the inter-stock attention's value resides in a compressible, dynamic, near-global redistribution that rewards low-rank approximation but resists sparsification.

1 Introduction

The paper decomposes MASTER’s inter-stock attention to determine what it learns and whether it can be made cheaper. It finds that predictive value comes from a dynamic, low-rank deviation from near-uniform attention: Nyström approximation succeeds, whereas sparsification fails, with limited transfer to larger universes.

The study combines statistical diagnostics, controlled ablations, spectral analysis, and efficient-attention experiments to examine MASTER’s inter-stock module.
Per-stock entropy is within 1.2% of maximum, yet forcing exact uniformity eliminates all cross-sectional discrimination.
Effective rank is ∼65 out of 300 stocks, while the top-10 singular values capture 96.5% of the Frobenius norm.
Nyström attention with m = 32 landmarks matches full attention at O(mN), with equivalent Rank IC across 5 seeds.The equivalence test reports TOST p = 0.003 within ±0.005.
Graph-masked, top-K, and deviation-thresholded variants all degrade performance, and hard correlation-neighbor masking is worse than removing the module.
At N ≈3,500, no cross-stock module significantly outperforms a per-stock LSTM baseline across multiple seeds.The experiments use adapted architectures and n = 4 seeds.

2 Background: MASTER Architecture

MASTER processes stock histories through temporal and cross-stock attention stages, with Step ➂ providing the main pathway for cross-stock information flow. Ablation identifies Step ➂ as the dominant value contributor.

MASTER processes N stocks over T lookback timesteps through five stages, including intra-stock temporal attention and Step ➂ inter-stock attention.
Step ➂ contains 329,216 parameters, or 42.5% of the model, including QKV projections, a feedforward network, and LayerNorm.
Step ➂ contributes the largest IC drop at −25.3% and is the only module enabling cross-stock information flow.
Removing Step ➃ increases Rank IC by 5.4%.Step ➃ performs temporal aggregation.

3 Experimental Setup

The experiments use MASTER’s CSI300 data and a larger full-market dataset, report correlation-based prediction metrics, and preserve the original pipeline across controlled Step ➂ variants. Mechanistic decompositions use one seed, while equivalence claims use multi-seed TOST testing.

CSI300 experiments use approximately 300 A-share stocks with 222-dimensional features and 619 test days.
Scale validation uses approximately 3,486 stocks per day from the full A-share market, with 17 features and 383 test days.
The study reports IC, Rank IC, ICIR, and long-short portfolio Sharpe ratios for scale experiments.
Step ➂ variants preserve the original treatment of the other pipeline stages, while equivalence claims are validated across 5–10 seeds with TOST.

4 Attention Diagnostic

The learned attention is near-uniform in distribution but dynamically structured in its deviations. It preferentially attends to stocks with dissimilar return dynamics, and its daily pattern reorganizes over weeks rather than remaining static.

The extracted attention weights have perplexity 278 out of 300, and the time-averaged matrix shows no visible industry block structure.
Pairwise attention and return correlation have Spearman ρ = −0.614, with stocks that co-move receiving less attention.
On the industry-labeled subset, the association is ρ = −0.645 unconditionally and partial ρ = −0.627 after controls.
Centered cosine similarity falls from 0.967 at lag 1 to 0.048 at lag 20, indicating attention-pattern reorganization over 2–4 weeks.
The 1.2% deviation from uniformity carries all of the module’s cross-sectional value despite entropy suggesting effective uniformity.

5 What Carries the Value?

The cross-sectional module’s value comes from a small, stock-specific deviation layered on a near-uniform broadcast, while graph-guided restrictions degrade performance. Its useful pattern is dynamic rather than static, so freezing attention recovers only part of the predictive value.

Graph-guided variants: Every graph-based variant degrades performance, and correlation-based hard masking falls below complete module removal (IC 0.045 ± 0.004 vs. 0.050 ± 0.001).The ordering Original > No Step ➂> GraphMask holds across seeds, with paired t p = 0.040 for Original versus GraphMask.
Cross-sectional structure: The time-averaged attention matrix shows no industry block-diagonal structure across 433 unique stocks, with approximately 300 active on any day.Stocks are sorted by Shenwan Level-1 industry, and the matrix is centered at 1/N.
Uniform and deviation components: The uniform component contributes 98.7% of output energy, but the 1.3% deviation component is the sole source of cross-sectional variance.The uniform component is identical across stocks as a rank-1 broadcast; value-vector heterogeneity does not alter this conclusion.
Dynamic redistribution: Pairwise attention is negatively associated with return correlation (ρ = −0.614; n = 73,910 pairs), while centered attention similarity fully decorrelates in approximately 20 trading days.The figure reports the pairwise association on the left and lagged centered cosine similarity on the right.
Dynamic redistribution: StaticAttn recovers approximately 47% of Rank IC value and approximately 9% of IC value relative to the floor, indicating that daily recomputation carries the value.The static oracle matrix is computed on the test set to give the fixed pattern its best chance.

6 Spectral Analysis and Nystr¨om Approximation

Spectral analysis shows that attention deviations from uniformity are robustly low-rank, motivating Nyström approximation, which preserves predictive quality when routing remains dynamic.

6.1 The Deviation Matrix Is Low-Rank: The deviation matrix has effective rank ∼65, while its top-10 singular values capture 96.5% of Frobenius energy.Across 208 day×head matrices, median effective rank is 65 [IQR: 45–81] and median top-10 energy share is 96.8%.
6.1 The Deviation Matrix Is Low-Rank: Low-rank structure is present before training, but training supplies the predictive pattern of deviations.Randomly initialized models have effective rank 59 ± 7 and top-10 energy 95.6 ± 0.9%, compared with trained rank ∼65 and top-10 energy 96.5%.
6.2 Nystr¨om Matches Full Attention: Nyström with m = 32 landmarks matches full attention at O(mN), with Rank IC equivalent within ±0.005 and IC equivalent within ±0.008.The equivalence tests report p = 0.003 for Rank IC and p = 0.028 for IC across 5 seeds.
6.3 Why Low-Rank Works but Sparse Does Not: Low-rank methods outperform sparse alternatives: Nyström matches the original on Rank IC, whereas TopK (K=16) retains at most 50% of either metric.Landmark-based Nyström also substantially outperforms Performer at the same m.
6.3 Why Low-Rank Works but Sparse Does Not: Small-K sparsification fails because near-equal top-K weights cannot represent the range around 1/N needed for meaningful differentiation.The 2×2 factorial confirms that support size and weight sharpening impose mutually exclusive requirements under sparsity.
6.4 Landmark Count: A Non-Monotonic Profile: Performance peaks at m = 32, with IC 0.059 ± 0.002 and Rank IC 0.066 ± 0.003, then declines non-monotonically as landmarks increase.Over-sizing worsens pseudo-inverse conditioning, increasing approximation variance faster than it reduces bias.
6.5 Dynamic Sampling Is Essential, Not Just Low-Rank Structure: Static rank-32 attention collapses to the no-attention floor, showing that low-rank approximation preserves performance only with dynamic, data-dependent routing.The static baseline reports IC = 0.0478 ≈0.0482.

7 Cross-Stock Aggregation at Scale

Nyström remains equivalent to full attention at N ≈800, but adapted large-scale experiments find no significant advantage for cross-stock modules over a per-stock LSTM at N ≈3,500.

7.1 CSI800: Same Codebase, Larger Universe: At N = 800, Nyström with m = 32 is equivalent to full attention within ±0.005 on both IC and Rank IC.The equivalence tests report p = 0.038 for IC and p = 0.034 for Rank IC across 10 seeds.
7.1 CSI800: Same Codebase, Larger Universe: Increasing landmarks to m = 80 does not improve performance, and the optimal m ≈32 remains stable across scale.Doubling the analysis from 5 to 10 seeds showed the initially suspected gap was a statistical-power artifact.
7.2 Large-N Transferability: At N ≈3,500, no cross-stock module significantly outperforms a per-stock LSTM baseline across 4 seeds.The test uses adapted architectures on the full A-share market.

8 Efficiency

Nyström incurs fixed overhead at small N but scales more favorably than full attention, delivering substantial latency and memory savings beyond the crossover point.

8 Efficiency: At N = 300, Nyström is slower than full attention, 5.90 versus 1.65 ms, because pseudo-inverse computation adds fixed overhead.The latency crossover occurs at N∗≈1,300.
8 Efficiency: Beyond N∗≈1,300, Nyström reaches 5.5× lower latency at N = 3,500 and 14.7× at N = 9,000.Its O(mN) scaling wins decisively at larger universes.
8 Efficiency: From N = 300 to 9,000, full-attention memory grows ×219 while Nyström grows ×20, consistent with quadratic versus linear scaling.Full attention exhausts 16 GB at N = 12,000, whereas Nyström uses 974 MB.
8 Efficiency: Nyström training is ∼2× slower per epoch, while recurring inference offers a 5–15× latency reduction with proportional memory savings for N > 3,000.At CSI300, the paper does not claim wall-clock savings because full attention is faster below N∗≈1,300.

9 SVD Factor Interpretation

The dominant attention-deviation modes align with established financial risk factors, indicating that inter-stock attention functions as an implicit multi-factor risk model.

9 SVD Factor Interpretation: PC1 explains 79.5% of the deviation structure and tracks cross-sectional volatility dispersion.High- and low-volatility stocks load on opposite ends of PC1.
9 SVD Factor Interpretation: PC4 provides the clearest industry signal, with η2 = 0.48, contrasting defensive banks and construction with cyclical autos and power equipment.Other components correspond to momentum, sector rotation, liquidity, and reversal.
9 SVD Factor Interpretation: MASTER’s inter-stock attention acts as an implicit 6-factor risk model despite never using factor labels during training.Its routing aligns with established financial risk dimensions.
9 SVD Factor Interpretation: Six interpretable components account for 91% of energy, while PCs 6 and 8–10 lack clear financial interpretations.The mapping is summarized in the paper’s factor-interpretation table.

10 Discussion

The discussion interprets inter-stock attention as complementarity-seeking, highlights metric- and preprocessing-dependent effects, and bounds the findings by scale, architecture, market, and validation scope.

Attention interpretation: Perplexity 278/300 cannot distinguish exact uniformity from near-uniform weights with structured low-rank deviations.The discussion recommends pairing entropy summaries with functional ablations such as uniform-forcing and oracle-static tests.
Attention interpretation: ρ = −0.614 unconditionally and −0.627 after controlling for industry, beta, and volatility, indicating attention favors stocks with different price dynamics.The association was negative on all 619 test days and contrasts with a correlation-mining interpretation.
Metric implications: ∼77% of IC covariance at CSI300 comes from the top return-magnitude tercile, while IC and Rank IC respond to different mechanisms.IC scales with aggregation breadth, whereas Rank IC depends on weight sharpness; this distinction is most relevant within MASTER’s native scale.
Related work: Static MLP-based cross-stock mixing in StockMixer achieves comparable performance, consistent with near-global redistribution as the effective mechanism.This related-work comparison echoes findings that fixed or random attention patterns can match learned attention in NLP.
Scope and limitations: All reported Sharpe ratios are frictionless and omit transaction costs, slippage, and market impact.This constrains direct interpretation of economic significance in the A-share setting.
Metric implications: IC differences of 0.005–0.01 are modest but practically relevant for institutional portfolios, with IC serving return-focused strategies and Rank IC serving pure ranking strategies.Long-short strategies with continuous position sizing benefit from IC, while top-K selection depends on Rank IC.
Scope and limitations: The TOST Rank IC equivalence weakens from p = 0.003 to p = 0.120 under improved feature normalization.Full attention gains +0.005 IC with 5.8× lower seed variance, while Nyström IC is unchanged, so equivalence narrows under optimized preprocessing.
Scope and limitations: At N ≈ 3,500, no cross-stock module significantly outperforms a per-stock LSTM baseline across multiple seeds.The large-scale experiments use an adapted MASTER pipeline, and the evidence comes from one model and the Chinese A-share market.

11 Conclusion

The paper finds that MASTER’s inter-stock attention derives its value from dynamic, near-global redistribution rather than interpretable stock relationships. Its low-rank structure supports Nyström compression but does not guarantee benefits at larger scales or across architectures.

Conclusion: MASTER’s attention implements dynamic, near-global redistribution whose deviation from uniformity is low-rank and compressible.This mechanism differs from the relationship-modeling interpretation implied by the module’s design.
Conclusion: Nyström approximation matches full attention on ranking metrics at O(mN) cost, while sparsification consistently fails.The equivalence is TOST-certified at N = 300 and N = 800.
Conclusion: At N ≈ 3,500 with adapted architectures, cross-stock modules do not significantly outperform per-stock baselines.The observed benefits therefore do not trivially transfer across scales and architectures.