Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision

Yong-Hoon Choi; Youngjin Cho

Source-linked AI summary

Which Histories Matter for Time Series Forecasting? Learning Predictive Relevance with Future Supervision

Yong-Hoon Choi, Youngjin Cho

arXiv:2608.23221v1Aug 24, 2026cs.IRcs.LG

TL;DR

Historical similarity is an imperfect proxy for which past examples will help forecast a current query. The paper uses future trajectories available only during training to supervise a past-only reranker, finding that relevance structures vary by domain and no retrieval principle dominates universally.

Problem
Past similarity does not fully specify predictive relevance, especially when similar histories diverge under nonstationarity or regime shifts.
Method
A normalized-pattern retriever forms a coarse candidate set, and a lightweight residual reranker learns future-supervised relevance while using only past/context features at inference.
Results
Future-supervised reranking improves Pattern retrieval across six benchmarks, including all 12 confirmatory tasks, while its advantage varies across domains and similarity rules.
Takeaways & Limitations
Historical relevance is structured and domain dependent: query-specific domains favor correct future supervision, while candidate-global or geometric structure can remain stronger elsewhere.
Takeaways & Limitations
The method reranks only Pattern-selected candidates and evaluates utility using future-trajectory MSE, so other utilities or candidate generators may yield different relevance structures.

Abstract

from arXiv · show

Historical retrieval for time-series prediction commonly treats past similarity as a proxy for usefulness. We ask a different question: which historical examples should be expected to matter for a query? We define predictive relevance as expected future utility conditioned on inference-time information, using realized futures only during training as privileged supervision. A normalized-pattern retriever first forms a coarse candidate set, and a lightweight residual multilayer perceptron (MLP) learns a listwise future-compatibility target while keeping inference-time scoring strictly past-only. Our method retains similarity-based candidate generation but reranks its candidates by a more predictive relevance criterion. Optimal relevance decomposes into candidate-level utility and query-specific compatibility, motivating Candidate-Prior and Shuffled-Future controls. Across six benchmarks, the reranker improves Pattern retrieval while revealing candidate-global, query-specific, and mixed relevance regimes. On all 12 confirmatory tasks, it improves Pattern and outperforms a matched-protocol Stationarity-Aware Retrieval-Augmented Time Series Forecasting (SARAF) retrieval rule. Architecture-matched ablations show that correct future supervision, rather than the MLP or added context alone, drives gains in query-specific regimes. Alternative-similarity experiments show that a strong last-value-anchored L2 rule remains superior in some domains, whereas future-supervised relevance is particularly strong where our diagnostics indicate query-specific relevance, especially on Solar. Candidate-pool diagnostics show that this contrast is not explained solely by coarse Pattern retrieval. Overall, historical relevance is structured and domain dependent rather than governed by a universally superior retrieval rule.

1 INTRODUCTION

The paper asks whether historical examples should be ranked by predictive relevance rather than past similarity alone. It uses training-time future outcomes to supervise a past-only reranker while retaining similarity for candidate generation.

Past-pattern similarity is a practical retrieval default but only a proxy for predictive relevance under nonstationarity and regime shifts.
Future outcomes provide privileged supervision for identifying useful historical candidates, while inference-time relevance prediction uses only past information.
Pattern similarity constructs a Top-M candidate pool, and a lightweight residual MLP reranks only those candidates using observable context and future-compatibility targets.Candidates excluded during initial retrieval cannot be recovered by the reranker.
Predictive relevance decomposes into candidate-level utility and query-specific compatibility, diagnosed with Candidate Prior and Shuffled-Future controls.
Across six benchmarks, future-supervised reranking improves Pattern retrieval and reveals domain-dependent relevance structures rather than a universally dominant rule.The experiments compare learned relevance with alternative similarities and matched retrieval protocols.

2 RELATED WORK

Related work applies retrieval augmentation and varied relevance principles to time-series forecasting and other nonstationary tasks. This paper instead isolates candidate-level predictive relevance and supervises past-only ranking with training-time future trajectories.

Retrieval-augmented forecasting methods retrieve historical segments and use their observed futures or learned augmentation modules to support prediction.
Recent approaches vary retrieval through stationarity awareness, semantic matching, spectral or channel-wise similarity, and selective aggregation.
These methods target forecasting, correction, probabilistic prediction, or retrieval design, whereas this paper isolates candidate-level predictive relevance.
The paper’s supervision differs from prior learned retrieval because historical future trajectories provide retrospective evidence of candidate utility.

3 PREDICTIVE RELEVANCE FROM PRIVILEGED FUTURES

The method defines relevance through future utility while preserving past-only inference. Pattern similarity generates candidates, and a small residual reranker learns to reorder them from observable context using future-supervised listwise targets.

Candidate memories are temporally valid because each candidate’s complete future is observed before the query time.
Future trajectories are normalized and used to define future distance and utility, which provide the training-time relevance signal.
The inference-time target is utility recoverable from observable information because the query future is unavailable when retrieval is performed.
Channel-wise normalized cosine similarity constructs the candidate set, while the learned score only reorders that set and cannot recover excluded candidates.
The reranker is intentionally a minimal residual MLP rather than a claim about the optimal retrieval architecture.
Training future distances within each candidate set become soft ranking targets for a listwise objective over reranker scores.
Chronological validation and refitting prevent test-query futures from entering training, normalization, candidate construction, or scoring.

4 WHAT MAKES HISTORY PREDICTIVELY RELEVANT?

Optimal predictive relevance is the conditional expected utility given past-only information. Its variation separates broadly useful candidates from query-specific compatibility, motivating controls that diagnose these structures.

The Bayes-optimal relevance function is the conditional expectation of candidate utility given observable query and candidate information.
Candidate-level utility is defined as expected utility conditioned on candidate information, while query-specific compatibility is the residual relevance after subtracting that utility.
The residual compatibility has zero conditional mean given candidate information, and the two components are orthogonal in variance.
Domains can share imperfect Pattern retrieval while differing in whether relevance is broadly candidate useful or query dependent.
Candidate Prior estimates query-independent candidate-global structure, whereas Shuffled Future preserves future marginals while destroying query–future correspondence.Correct-versus-shuffled performance therefore diagnoses query-specific signal rather than numerically estimating the residual component.

5 EXPERIMENTS

Across six benchmarks, future-supervised Learned reranking improves Pattern retrieval and SARAF-Matched under the confirmatory protocol, while diagnostics reveal distinct candidate-global and query-specific relevance regimes. Alternative similarities and controls show that gains depend on domain structure rather than a universally dominant retrieval rule.

Core retrieval results: Learned improves Pattern on all 18 benchmark–horizon settings, with 9 of 12 confirmatory gains statistically significant.It also beats SARAF-Matched on all 12 confirmatory tasks, with every bootstrap interval strictly above zero.
Predictive relevance regimes: ETTh1, Traffic, and Solar exhibit query-specific compatibility, Weather and Exchange exhibit candidate-global utility, and Electricity contains both.The two axes are Correct-versus-Shuffled gain and Candidate Prior-versus-Pattern gain.
Predictive relevance regimes: On Exchange, Candidate Prior improves Pattern by 55.8% while Correct is 3.3% worse than Shuffled; on Solar, Correct improves Shuffled by 70.5% and Pattern by 31.8%.Candidate Prior is 48.4% worse than Pattern on Solar, contrasting with its strong Exchange result.
Ablations: Pattern+Context improves Pattern in all 12 confirmatory tasks, but Learned beats it in 11/12, isolating value beyond context and architecture.Correct and Shuffled use the same MLP and features, so their gap tests the value of correct future correspondence rather than model capacity.
Alternative retrieval principles: Learned beats raw cosine, Pattern/Pearson, spectral cosine, and SARAF-Matched on all 12 confirmatory tasks, but beats last-value-anchored L2 on only 4/12.Learned remains 16.7% better than L2 on Solar within the identical pool, whereas restricted L2 beats Learned by 14.2% on Electricity and 11.4% on Exchange.
Forecasting implications: AnalogFutureMSE and RetrievalForecastMSE are distinct: on Traffic at H = 96, the former improves from 0.8751 to 0.8634 while the latter changes from 0.6115 to 0.6185.The financial case study similarly improves retrieval-based forecasting over Pattern but remains 2.2–2.5% behind the strongest direct forecasting baseline.

6 DISCUSSION AND LIMITATIONS

The experiments indicate that predictive historical relevance is structured and domain dependent, with different domains favoring query-specific supervision, candidate priors, or mixtures. The method is a useful demonstration rather than a universally optimal retriever, with important scope and pipeline boundaries.

Discussion: Query-specific domains benefit strongly from correct future supervision, candidate-global domains may favor simple priors or geometric similarity, and mixed domains contain both mechanisms.The lightweight MLP demonstrates that training-only privileged futures can teach an inference-time relevance function.
Limitations: Future-trajectory MSE defines the utility, so other downstream tasks or utility definitions may induce different relevance structures.The method also cannot recover useful examples excluded by the initial Pattern Top-M candidate pool, and relevance does not specify optimal aggregation or forecasting architecture.

7 CONCLUSION

The paper argues that historical relevance is structured and domain dependent: similarity remains useful for candidate generation, but future-supervised reranking helps most when relevance is query specific.

Future-supervised relevance preserves similarity for coarse candidate generation while learning a more predictive ranking criterion.
Predictive relevance decomposes into candidate-level utility and query-specific compatibility, with controls designed to diagnose both components.
No single retrieval principle dominates across domains: learned future compatibility is strongest for query-specific relevance, while geometric or candidate-level structure can dominate elsewhere.
Historical retrieval should be evaluated by why retrieved examples are expected to matter, not only by how they are searched.

REPRODUCIBILITY STATEMENT

The reproducibility materials document the main protocol and supporting implementation details across the paper and appendices.

The main text specifies the relevance target, reranker, controls, hyperparameters, SARAF comparison, similarity analysis, candidate-pool diagnostic, and statistical test.
Appendices B–F document preprocessing, temporal splits, normalization, context features, optimization, seed-level evaluation, bootstrap intervals, ablations, mechanism analyses, and the financial case study.

AI USE STATEMENT

The supplied materials combine an AI-use disclosure with descriptions of the study’s formal assumptions, datasets, features, controls, results, and target-scaling caveat.

ChatGPT supported experimental discussions, code development, and manuscript editing, while the authors independently reviewed and verified the scientific content.
The relevance framework defines conditional expected utility as the optimal observable ranking function and separates candidate-level utility from query-specific interaction.
The Shuffled-Future control preserves within-channel future marginals while destroying original query–future correspondence, providing an architecture-matched diagnostic for query-specific signal.
The study evaluates six benchmarks under fixed candidate, feature, optimization, and horizon protocols, with AnalogFutureMSE as the primary relevance metric.
The lightweight reranker improves Pattern across benchmark settings, beats SARAF-Matched on all 12 confirmatory tasks, and exhibits stable confirmatory-suite variability.
Weather exposed instability in a local-scale target because low-variance windows produced extreme normalized values, motivating fixed train-scale targets.

E.3 CANDIDATE-PRIOR DECONFOUNDING DOES NOT IMPROVE THE FINAL METHOD

Candidate-prior deconfounding reveals query-specific signal in some settings but does not improve the final raw Learned reranker and fails on Weather.

Deconfounded Learned still beats its Shuffled control by roughly 21% across ETTh1 horizons, indicating query-specific signal after prior subtraction.
The authors therefore treat the decomposition as a structural population view rather than imposing a G + H neural architecture.
A weighted combination of Pattern similarity, candidate prior, and a deconfounded residual remains worse than raw Learned; on Weather, validation often suppresses the query residual.
The financial robustness case is explicitly survivorship-biased and is not a point-in-time index backtest.

F.2 MULTI-HORIZON SAME-STOCK AND CROSS-STOCK RESULTS

The experiments show that learned relevance improves retrieval across confirmatory tasks, including cross-stock finance, but its advantage depends on the domain and relevance regime. Candidate-pool controls and ablations indicate that future supervision, rather than candidate access alone, explains the strongest query-specific gains.

Multi-horizon same-stock and cross-stock results: Cross-stock Learned improves both the cross-stock Pattern baseline and same-stock Learned retrieval.The finance case study reports 7.05%, 8.17%, and 7.28% Cross Pattern-to-Cross Learned ForecastMSE improvements at H = 1, 5, 20.
Multi-horizon same-stock and cross-stock results: Retrieval remains approximately 2.19%, 2.29%, and 2.49% behind the strongest direct predictor at the evaluated horizons.These results motivate framing the contribution around retrieval relevance rather than forecasting state of the art.
Alternative-similarity robustness: Last-value-anchored L2 remains significantly better than Learned on Electricity and Exchange, whereas Learned has its only significant gains over L2 on the three Solar horizons.Alternative similarities search the full admissible memory, while Learned reranks only the Pattern candidate pool.
Candidate-pool diagnostic: Holding the candidate pool fixed preserves the domain contrast: restricted L2 remains better on Electricity, while Learned remains better on Solar at H = 24, 48.On Electricity, restricting L2 causes only a 1.6–2.1% penalty; on Exchange, restriction slightly improves L2.
Scope of the relevance regime map: Relevance-regime coordinates are empirical diagnostics under fixed representations, horizons, generators, and controls rather than intrinsic dataset constants.The qualitative contrast is robust in these experiments: candidate priors dominate Weather/Exchange, correct correspondence matters for ETTh1/Traffic/Solar, and both coexist in Electricity.
Architecture and supervision ablations: Learned improves Pattern on all 12 confirmatory tasks, while correct future correspondence improves Shuffled on 9/12 tasks and Candidate Prior improves Pattern on all Electricity and Exchange horizons.Correct Learned is significantly better than Shuffled across all Electricity, Traffic, and Solar horizons, but not Exchange.

K.2 FORECAST AGGREGATION ABLATIONS

Forecast aggregation ablations improve retrieval-based forecasts but do not reliably surpass the strongest direct predictor. The results support keeping learned historical relevance as the paper’s primary claim rather than expanding into a complex fusion architecture.

Forecast aggregation ablations: Scalar shrinkage toward a no-change forecast substantially closes the forecasting gap, while gated neighbor aggregation improves uniform retrieval weighting without reliably surpassing the best direct predictor.These experiments reinforce the distinction between improving retrieval relevance and achieving direct-forecasting state of the art.
Forecast aggregation ablations: Shrinkage coefficients increase with horizon, from approximately 0 at H = 1 to 0.066 at H = 5 and 0.234 at H = 20.The selected coefficients remain conservative despite historical analogs becoming more useful for longer cumulative-return paths.