PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecast Collapse in Time-Series Foundation Models

Shu Wan, Miles Ma, Hank Zhu, Guangqi Liu, Stephen Wang, Qingsong Wen, Huan Liu

arXiv:2608.14106v1cs.LGcs.AIcs.CEstat.APstat.ML

TL;DR

Forecast collapse produces nearly flat equity-return predictions with poor cross-sectional ranking, raising questions about predictability and cross-series structure. The paper analyzes these mechanisms across models and benchmarks, then introduces CalibRank to balance calibration with ranking and reports improved correlation while retaining a calibration anchor.

  • Problem

    Forecast collapse combines low forecast amplitude with poor cross-sectional structure, while the phenomenon differs between equity returns and traded volume under the same protocol.

  • Method

    The paper studies predictability, calibration, and cross-series structure across forecasting models and benchmark configurations, and introduces CalibRank to jointly balance calibration and cross-sectional ranking.

  • Results

    CalibRank improves cross-sectional correlation across all tested models while controlling forecast scale, whereas MSE produces nearly flat forecasts and IC-only optimization can produce severe amplitude inflation.

  • Takeaways & Limitations

    Per-series evaluation can miss cross-series structure required by downstream decisions, so forecasting should assess both calibration and the relevant multivariate relationships.

  • Takeaways & Limitations

    The cross-sectional diagnosis and remedy are demonstrated on one real panel and do not establish the same ordering failure in another domain.

Abstract

from arXiv · show

When forecasting hourly returns for 1,000 US equities, we observe an unexpected phenomenon: predictions become nearly flat and show poor stock ranking, as measured by cross-sectional correlation. We call this forecast collapse. Surprisingly, the phenomenon largely disappears when forecasting trading volume under the same setting. We investigate forecast collapse across time-series foundation models (TSFMs), twelve deep-learning forecasting models, and 97 public benchmark configurations, and find that it is closely tied to target predictability. We identify two distinct reasons behind it: low predictability limits the amplitude of calibrated point forecasts, while per-series objectives leave cross-series structure unidentified. These findings reveal a calibration-ranking tradeoff: optimizing squared error leads to flat predictions, whereas directly optimizing cross-sectional correlation improves ranking but can inflate forecast amplitude by more than an order of magnitude. To address this tradeoff, we introduce CalibRank, a simple objective that balances calibration and ranking. On Finance1K, CalibRank nearly triples cross-sectional correlation while keeping amplitude close to the target, and improves correlation on all tested models. Our results reveal a blind spot in conventional time-series evaluation: per-series metrics can hide failures in cross-series structure needed by downstream decisions.

1 Introduction

Forecast collapse is the combination of nearly flat forecasts and poor cross-sectional stock ranking, appearing for hourly equity returns but largely disappearing for traded volume under the same protocol. The paper explains this through predictability limits and unidentified cross-series structure, then introduces CalibRank to balance calibration with ranking.

  • Phenomenon: Forecast collapse denotes low forecast amplitude together with poor cross-sectional structure, observed when TSFMs forecast hourly US equity returns.The same phenomenon largely disappears when the target changes to traded volume without changing the stock panel or protocol.
  • Mechanisms: Low target predictability limits the amplitude of calibrated point forecasts, while per-series objectives leave cross-series structure unidentified.The paper formalizes these mechanisms through an exact amplitude relation and an invariance result for per-series risk.
  • Calibration-ranking tradeoff: Optimizing MSE can produce low-amplitude forecasts, whereas optimizing IC improves ranking but generated forecasts over 20 times the target amplitude.CalibRank combines MSE with cross-sectional correlation, with λ controlling their relative importance.
  • Implication: Forecasting evaluation can miss decision-relevant cross-series structure when a multivariate panel is treated as independent series.The paper argues that benchmarks should preserve and evaluate this structure when it matters downstream.
  • Proposed remedy: CalibRank improves cross-sectional correlation for every tested model while maintaining control of forecast scale.The tradeoff is validated across twelve forecasting models and 97 public benchmark configurations are used to examine amplitude and predictability.
  • Empirical setting: Finance1K compares 1,000 US equities using aligned hourly return and volume targets under a matched forecasting protocol.This design contrasts low-predictability returns with more predictable volume while holding the forecasting setting fixed.

2 Related Work

Related work spans variance and amplitude limits, structural evaluation beyond pointwise error, regression-ranking objectives, and joint forecasting. The paper positions CalibRank as a narrower composite for the cross-sectional functional used by downstream decisions.

  • Forecast amplitude and predictability: Prior work relates mean-optimal under-dispersion and low-amplitude transformer forecasts to variance-share or measurement-error limits.This paper connects those limits to best-scaled forecast amplitude while separately measuring cross-sectional ordering.
  • Beyond pointwise error: Image-restoration research motivates evaluating structural criteria beyond pixelwise L2, although its perception index differs from IC.The cited perception measure concerns divergence between marginal laws, whereas IC measures dependence between forecasts and targets.
  • Regression and ranking: Composite regression-ranking losses, rank-correlation objectives, correlation losses, and differentiable ranking operators are established in stock prediction and information retrieval.Listwise objectives have also been reported to reduce conflicts between regression and ranking.
  • Time-series foundation models: Surveyed TSFM objectives include squared-error-like, quantile, likelihood, cross-entropy, and flow-matching losses, but none scores a whole cross-section at once.The issue therefore extends beyond MSE: per-series scoring leaves the multivariate axis absent.
  • Positioning: CalibRank addresses the narrower cross-sectional functional used by a downstream decision, while related work suggests stronger composites may improve on its simple scalarization.The paper distinguishes this target from richer dependence modeling in probabilistic forecasters.

3 Preliminaries

The preliminaries define timestamp-level cross-sectional evaluation and distinguish target predictability, model error, forecast scale, and ordering. They also classify forecasting objectives by the quantities they identify and whether they score whole cross-sections.

  • Forecasting panel: At each forecast timestamp, the N series form one cross-section, and temporal reporting aggregates timestamp-level statistics.Forecasts and realized targets are collected into vectors before computing the relevant statistics.
  • Evaluation unit: The cross-section is the evaluation unit and timestamps are the replication unit; correlation must be computed within timestamps to preserve decision-relevant ordering.Pooling all series-time pairs would mix temporal and cross-sectional variation.
  • Analytical layers: The framework separates target-side predictability, model error relative to the predictable component, and forecast properties such as IC and amplitude.A forecast can therefore have weak scale, weak ordering, or both.
  • Objective survey: The objective survey covers nineteen models and eighteen sourced objectives, with fifteen disclosed forecasting losses and four squared-error-like losses.TimeGPT names no loss function and is omitted from the table.
  • What objectives identify: Squared error and robust surrogates identify conditional means, while pinball grids identify included quantiles and likelihood or cross-entropy scores identify predictive distributions.Flow matching regresses a velocity field, with distributional quantities obtained from its draws.
  • Scope: The sourced training objectives score per-series quantities rather than a whole cross-section, so output scale and within-timestamp ordering remain separate quantities to measure.The survey is about objective scope, not model performance.

4 Why Forecasts Collapse

Forecast collapse has two distinct sources: low predictability attenuates calibrated amplitude, while pointwise objectives leave cross-series dependence unidentified. Consequently, raw scale, calibration, and downstream ranking must be evaluated separately.

  • Per-series risk and coupling: Pointwise objectives score per-series, per-timestamp pairs and therefore do not directly identify cross-series dependence.This applies to squared error, absolute error, quantile, likelihood, and cross-entropy losses.
  • Predictability and amplitude: Low predictable variance limits the amplitude of a calibrated point forecast, even when the forecast is otherwise appropriate for the information set.The result concerns point forecasts; predictive distributions can retain conditional spread even when their mean or median is small.
  • Predictability and amplitude: After optimal scalar recalibration, forecast amplitude equals the absolute forecast-target correlation, so weak correlation cannot coexist with correct scale.The identity is a property of a centered forecast-target pair and does not imply that raw model amplitude equals the optimally rescaled amplitude.
  • Two mechanisms: The two collapse conditions are distinct: low-signal univariate forecasts can be small yet calibrated, whereas predictable multivariate forecasts can be well-scaled but incorrectly ordered.Forecast collapse combines amplitude attenuation from weak predictable variance with underdetermined cross-series structure.
  • Per-series risk and coupling: Per-series risk can remain unchanged while cross-sectional correlation changes, because recoupling preserves each coordinate’s law but alters dependence across series.A two-series Gaussian construction keeps squared-error risk at 1.40 while changing expected IC from 0.371 to 0.032.
  • Implication: CalibRank addresses the coupling gap by rewarding the downstream cross-sectional functional, while retaining a calibration anchor.It targets per-timestamp Pearson correlation rather than a full joint distribution or copula.

5 Formal Foundations and Proofs

The formal analysis decomposes forecast error into predictable-model and irreducible-noise components, then shows how weak signal attenuates calibrated amplitude and pointwise risk fails to determine cross-sectional ranking. Additional results connect these mechanisms to mean and median forecasts and to direction accuracy.

  • 5.2 Additivity of squared error: Under the declared information set, MSE decomposes exactly into forecast error Q and irreducible noise σ2.The conditional-mean representation makes the residual orthogonal to admissible forecasts, eliminating the cross term.
  • 5.2 Additivity of squared error: Irreducible noise can dominate squared-error comparisons across series, so low MSE does not necessarily reflect comparable model error.Heterogeneous noise can make cross-sectional squared-error rankings track noise levels rather than forecast quality.
  • 5.3 Attenuation and the amplitude identity: A forecast can covary with the target only through its predictable component, making the attainable correlation ceiling depend on the information set and target predictability.For weak signal, the ceiling can still exceed SNR; for example, SNR = 0.01 permits ICmax ≈0.10.
  • 5.3 Attenuation and the amplitude identity: Optimal rescaling links retained forecast amplitude directly to absolute correlation, so weakly correlated forecasts remain weak in amplitude after best scaling.The optimal scale is the least-squares slope of the target on the forecast.
  • 5.4 Mean and median point forecasts: For symmetric conditional targets with common scale, mean and median point forecasts both equal the conditional mean and therefore share the same attenuation result.Changing from squared-error to absolute-error point forecasting does not escape the conclusion under these assumptions.
  • 5.5 Insensitivity to cross-series structure: Pointwise risk depends only on per-coordinate forecast-target laws, whereas cross-sectional IC depends on their coupling, so equal risk can produce different rankings.Theorem 5.4 keeps risk at 1.40 while expected IC ranges from 0.371 to 0.032, an order-of-magnitude difference.
  • 5.6 The direction law: Under the direction law, sign agreement is linked to IC; IC = 0.05 corresponds to a 51.6% hit rate.This illustrates that a small correlation can coexist with a hit rate only modestly above one half.

6 Calibrated Cross-Sectional Ranking

CalibRank combines calibration and cross-sectional ranking in a two-term objective. Its weighting parameter traces a frontier between scale fidelity and ranking performance, with alternative calibration terms fitting the same design.

  • Objective design: CalibRank combines squared error with differentiable per-timestamp Pearson correlation to balance calibration and cross-sectional ranking.The squared-error term anchors scale, while correlation rewards ordering at each timestamp.
  • Calibration-ranking frontier: Sweeping λ traces the calibration-ranking frontier, with single-term MSE and IC objectives at the two endpoints.Huber calibration is also considered as a robust alternative that can reduce the influence of large target innovations.
  • Objective design: The objective design space separates calibration and ranking terms, leaving the specific functional choices open.The general form is L = C(ŷ, y; w) − λS(ŷ, y; w).

7 Experimental Design

The experiments compare calibrated forecasting, ranking, and CalibRank objectives on matched return and volume targets, using controlled, synthetic, foundation-model, and downstream decision evaluations. Synthetic panels vary signal-to-noise ratio and forecaster skill to test the theoretical mechanisms directly.

  • Real-data setup: Finance1K contains hourly data for 1,000 US equities with aligned log-return and log-volume-change targets, using a chronological train/test split.The task predicts each series’ next target from its previous 96 hourly observations.
  • Model and objectives: The controlled encoder maps a length-96 univariate window to a next-step scalar forecast through six patches, a three-layer Transformer, pooling, and a linear head.The reference encoder has approximately 0.38M parameters.
  • Model and objectives: The controlled comparison changes only the training objective across twelve forecasting families, with shared parameters across series and three fixed random seeds.Supervised models receive the same univariate history for each series and are trained with a common optimizer schedule.
  • Evaluation: Cross-sectional IC is computed within each trading hour and then averaged, preventing pooled observations from mixing temporal and cross-sectional variation.Squared error is reported on the full evaluation grid, while IC targets timestamp-level ranking.
  • Synthetic verification: Synthetic experiments vary signal-to-noise ratio and forecaster skill on disjoint fitting and evaluation draws, using known signal and noise to test the theoretical relations.The synthetic predictor adds controlled independent model error to the true signal, making alignment directly testable.
  • Synthetic verification: The synthetic close regime has signal variance equal to 1.2% of noise variance, whereas volume has 20× larger SNR and more homogeneous noise.This matched generator isolates the predicted transition from MSE–IC decoupling to partial coupling.
  • Additional evaluations: The additional protocols include a held-out decision return, a post-hoc λ sweep, and TSFM amplitude comparisons across 97 GIFT-Eval configurations.Figure 3 also contrasts return and volume targets under matched evaluation settings.

8 Empirical Results

The controlled results expose a calibration–ranking tradeoff: MSE yields nearly flat forecasts, IC improves ranking but leaves scale unidentified, and CalibRank combines ranking gains with controlled amplitude. Synthetic and pretrained-model analyses support the predictability mechanism, while decision returns and volume provide external checks.

  • Controlled objective comparison: MSE produces raw amplitude 0.027 of target amplitude and IC 0.046, whereas IC training reaches IC 0.124 but 25× target amplitude.Uniform CalibRank at λ = 1 reaches IC 0.126 while keeping raw amplitude within a factor of two of the target.
  • Calibration–ranking tradeoff: CalibRank places ranking and calibration on an operating frontier, while pure ranking improves ordering without constraining forecast scale.Rescaling alone cannot recover missing ranking information, and IC-only training produces forecasts over 20 times the target amplitude.
  • Calibration–ranking tradeoff: IC reaches 0.1246 by λ = 0.1 and changes by less than 0.002 through λ = 1, while raw amplitude continues to rise.At λ = 0.1, forecasts reach 87.5% of target amplitude with about half the squared error of λ = 1.
  • Synthetic verification: Across the synthetic grid, the amplitude identity predicts held-out best-scaled amplitude within 0.0015 and correlation within 0.0021.The sweep spans two decades of SNR and three forecaster-skill levels on disjoint fitting and evaluation halves.
  • Foundation-model evidence: Across 97 GIFT-Eval configurations, amplitude correlates with achieved R2 at 0.88 for Chronos and 0.87 for TimesFM.Both models retain about three quarters of target amplitude when achieved R2 is at least 0.30, but under six percent when it is below 0.05.
  • Downstream decision check: Held-out decision return rises from 0.052 ± 0.010 under MSE to 0.143 ± 0.029 under IC and 0.170 ± 0.030 under weighted CalibRank.The decision functional is absent from the training loss, providing an out-of-objective check.
  • Target contrast: On volume changes, MSE and uniform CalibRank both reach raw amplitude 0.391, with IC 0.530 and 0.538 respectively.The higher-predictability target therefore lacks the near-zero MSE amplitude observed for close returns.

9 Comprehensive Cross-Architecture Evidence

Across twelve forecasting backbones, CalibRank consistently improves cross-sectional ranking while keeping the calibration cost far below IC-only training. Conditioning effects depend on target predictability, and ranking gains largely saturate at modest λ values.

  • Cross-architecture objective comparison: The controlled cross-architecture comparison repeats matched MSE and CalibRank training across recurrent, convolutional, mixing, linear, and attention-based architectures.The same context, optimizer, schedule, and three-seed protocol are used across backbones.
  • Conditioning ablation: The conditioning contrast is consistent with low lagged predictability for hourly close volatility and stronger persistence in volume.The ablation separates this conditioning limitation from the core CalibRank coupling, which remains unchanged.
  • Conditioning ablation: Conditioning raises close IC in 4/12 backbones, but raises volume IC in 10/12 and lowers volume MSE in 9/12.Large standard deviations for Informer and Autoformer caution against treating isolated signs as universal gains.
  • Calibration–ranking sweep: Across backbones, the first nonzero ranking weight produces almost the entire correlation gain, after which correlation plateaus while squared error and amplitude rise.Differences in sweep curvature reflect backbone stability, with Informer showing larger seed dispersion.
  • Calibration–ranking sweep: Moving from λ = 0 to λ = 1 raises close IC for every backbone, with median gain 0.0489 at a median 1.280× MSE multiplier.Most models obtain nearly all of this ranking gain by λ = 0.25–0.5.

10 Discussion

The discussion argues that evaluation and model selection should preserve cross-sectional structure alongside per-series accuracy, especially when downstream decisions depend on relationships among channels. It also proposes diagnostics for anticipating forecast collapse and validation-based selection of the calibration-ranking tradeoff.

  • Evaluation: Benchmarks should report amplitude, cross-sectional dependence, and ordinary per-series error because scalar averages cannot reconstruct amplitude or joint structure.GIFT-Eval scores series independently, flattening multivariate configurations and omitting metrics for raw amplitude and cross-sectional structure.
  • Evaluation: Evaluation should preserve channel grouping and timestamps, compute ordering within each cross-section, and average afterward to avoid mixing temporal and cross-sectional effects.Amplitude should also be identified as raw output or post-calibration, so rescaling does not hide the model’s original output.
  • Evaluation: The relevant cross-series functional depends on the decision, ranging from correlation or concordance to tail dependence or a learned graph.Finance uses ordering, while environmental, clinical, energy, and traffic applications may depend on different relationships among channels.
  • Diagnostics: Forecast collapse is most likely when predictable variance is low and the downstream task requires a multivariate functional absent from the objective.Low achieved R2 warns of amplitude attenuation but cannot establish that the true predictability ceiling is low.
  • Diagnostics: Cross-sectional losses should improve the targeted functional under matched protocols, while tests across architectures and seeds can distinguish statistical structure from implementation defects.If high predictable variance coexists with flat MSE forecasts, optimization, regularization, or preprocessing becomes the leading suspect.
  • Model selection: λ should be selected on date-ordered validation data to satisfy a declared amplitude tolerance, with the untouched test result reported afterward.The paper treats λ = 1 as prespecified and the seven-point sweep as a measured frontier rather than test-set model selection.
  • Model selection: Reporting both calibration and structure turns model selection into a constrained choice rather than a single leaderboard ranking.Practitioners can impose an admissible calibration range and maximize cross-sectional structure within it.

11 Conclusion and Limitations

The conclusion presents forecast collapse as the combination of amplitude and ordering failures and recommends evaluating multivariate structure alongside per-series error. It qualifies the empirical reach of the predictability analysis, the single-panel demonstration, the dependence functional, and aggregation choices.

  • Conclusion: Forecast collapse combines low-amplitude forecasts with poor ordering, while Finance1K shows that CalibRank improves cross-sectional correlation while retaining a calibration anchor.MSE forecasts are nearly flat, IC-only forecasts are badly miscalibrated, and TimesFM and Chronos show a parallel amplitude-predictability association across 97 configurations.
  • Conclusion: A small forecast can be correctly calibrated for a low-predictability target, so audits should measure raw amplitude and decision-relevant structure alongside per-series error.Low per-series error alone does not guarantee preservation of relationships needed by the decision.
  • Limitations: Achieved R2 from fitted baselines is only a lower bound on available predictability, so GIFT-Eval shows where amplitude collapse appears but does not estimate the true ceiling.The amplitude identity is exact for best-scaled forecasts and checked in simulation, while the benchmark demonstrates an empirical association.
  • Limitations: The cross-sectional diagnosis and remedy are demonstrated on one real panel, so Finance1K does not support a market-wide performance claim or establish the same failure in another domain.The twelve-backbone experiment reduces dependence on one architecture but does not establish cross-domain generality.
  • Limitations: CalibRank targets per-timestamp Pearson correlation and may require a different cross-sectional term for rank, tail, or calibrated-scenario applications.Its scalarized frontier is not claimed to be fundamental, and the remedy requires a meaningful cross-section at each timestamp.
  • Limitations: Per-timestamp IC is averaged rather than pooled because drift can create Simpson effects, while ceilings based on average predictability are Jensen-loose.The paper avoids comparing empirical IC with a pooled ceiling percentage.

12 Generative AI Usage

The authors disclose that generative AI assisted with code and writing, while excluding research ideation, theory, proofs, experiment design, and result interpretation. They reviewed and edited all AI-assisted material and accept responsibility for its accuracy and integrity.

  • Use: Generative AI assisted with code, experiment infrastructure, figure and table rendering, validation, and writing clarity.The assistance covered code writing and refactoring plus drafting, editing, and clarity improvements.
  • Boundaries: Generative AI was not used for research ideas, theory or proofs, experiment design, or interpretation of results.
  • Accountability: The authors reviewed and edited all AI-assisted text and code and accept responsibility for its accuracy and integrity.
Loading 2608.14106v1…