PaperlayerView exact revision on arXiv

Source-linked AI summary

JudgeCast: Time Series Forecasting with Experience-Informed Covariate Judgements

Donguk Kwon, Wooseok Jeong, Dongha Lee

arXiv:2609.36966v2cs.LGcs.AI

TL;DR

Covariate effects vary across contexts and shift over time, so forecasters must assess how to use covariates for each forecasting context. JudgeCast uses a frozen TSFM for a base forecast and a frozen LLM to form explicit covariate-wise judgments, adjust the forecast, and reconstruct decisions after observation for validated experience. Across diverse real-world datasets, JudgeCast outperforms the strongest baselines by 16.7% in MSE and 6.5% in MAE on average.

  • Problem

    Covariate effects vary across contexts and shift over time, so forecasters must assess how to use covariates for each forecasting context.

    Source: §1 · p. 1

  • Method

    JudgeCast uses a frozen TSFM for a base forecast and a frozen LLM to form explicit covariate-wise judgments, adjust the forecast, and reconstruct decisions after observation for validated experience.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Across diverse real-world datasets, JudgeCast outperforms the strongest baselines by 16.7% in MSE and 6.5% in MAE on average.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Revisiting forecast-time decisions after observation can make forecast feedback reusable for subsequent decisions about covariate use.

    Source: §5 · p. 9

    Validation does not establish that the selected judgment faithfully reflects the effects of the covariates.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Covariate effects vary across contexts and shift over time, requiring forecasters to assess how to use them for each forecasting context. As forecasting proceeds, observations for earlier forecasts become available, providing feedback on past covariate use for subsequent forecasts. However, when multiple covariates act together, the forecast error reveals the numerical discrepancy from the observation but not how the covariates should have been used. We introduce JudgeCast, an experience-based framework for time series forecasting with covariates. Following the judgmental adjustment practice, a frozen TSFM provides the base forecast, while a frozen LLM uses the current context and relevant experience to adjust it. Within the adjustment, assessing covariate effects and determining the numerical adjustment serve distinct roles, so JudgeCast first forms explicit covariate-wise judgments and then determines the adjustment. After observation, JudgeCast uses the observed residual of the base forecast to reconstruct alternative judgments and evaluates the original and alternatives through their resulting adjustments. The best-performing decision is selected and retained as validated experience for subsequent forecasts. Across diverse real-world datasets, JudgeCast outperforms strong baselines. Ablations show that explicit covariate-wise judgment can improve forecast-time adjustment, while residual-guided experience construction yields more reliable forecasting gains than retaining raw decisions as experience.

1 INTRODUCTION

Covariate effects vary across contexts, and errors from forecasts using multiple covariates do not reveal how each covariate should have been assessed. JudgeCast separates covariate-wise assessment from numerical adjustment and uses post-observation feedback to construct experience for later forecasts.

1 INTRODUCTION

  • Covariate effects vary across contexts and over time, while joint effects make forecast errors insufficient to identify how covariates should have been used.Forecast feedback could inform later forecasts without parameter updates, which can add computational cost and latency or be unavailable when parameters cannot be updated.Source: §1 · pp. 1–2
  • JudgeCast uses a frozen TSFM for a base forecast and a frozen LLM to form explicit covariate-wise judgments and determine the numerical adjustment.The judgments assess each covariate's expected effect relative to the base forecast; together with the adjustment, they constitute the forecast-time decision.Source: §1 · p. 2
  • After observation, JudgeCast uses the base-forecast residual to reconstruct alternative covariate-wise judgments and evaluates decisions through their resulting adjustments.The best-performing decision is retained as validated experience only when its adjusted forecast improves on the base forecast; this does not establish that its judgments faithfully reflect covariate effects.Source: §1 · p. 2
  • JudgeCast outperforms the strongest baselines across diverse real-world datasets by 16.7% in MSE and 6.5% in MAE on average.Ablations also show that explicit covariate-wise judgment can improve forecast-time adjustment, while residual-guided experience construction outperforms raw experience.Source: §1 · p. 2
  • JudgeCast's residual-guided experience construction reconstructs forecast-time decisions into validated experience, yielding more reliable gains than retaining raw decisions as experience.The introduction reports this comparison alongside the finding that explicit covariate-wise judgment can improve forecast-time adjustment.Source: §1 · p. 2

2 RELATED WORK

Prior work models covariates within forecast generation, reuses past cases or forecast errors, or automates forecast revision. JudgeCast distinguishes itself by explicitly separating covariate-effect assessment from numerical adjustment and by reconstructing forecast-time decisions after observation.

Covariate-Informed Time Series Forecasting

  • Covariate-informed forecasters model covariate effects within forecast generation, while post-hoc attribution quantifies contributions after prediction rather than as a forecast-time decision.The cited approaches include covariate conditioning (Lim et al., 2021; Wang et al., 2024b; Ansari et al., 2025; Yang et al., 2026) and post-hoc attribution.Source: §2 · p. 3

LLM-based Time Series Forecasting with Experience

  • Experience-based LLM forecasters reuse past cases, errors, reflections, guidance, or inference trajectories, but do not reconstruct how contextual information was used in a forecast-time decision.The passage describes MemCast as organizing prediction outcomes, inference trajectories, and temporal features into hierarchical forecasting experience.Source: §2 · p. 3

Judgmental Adjustment in Time Series Forecasting

  • Automated judgmental-adjustment approaches revise forecasts without explicitly separating covariate-effect assessment from numerical adjustment; JudgeCast represents covariate effects as explicit judgments.The passage describes agent-based revision and reflection (Liao et al., 2026), LLM forecast revision (Liu et al., 2026; You et al., 2026), residual learning, and retrieval-based revision (Liu et al., 2025).Source: §2 · p. 3

3 JUDGECAST

JudgeCast separates covariate-wise assessment from numerical adjustment: a frozen TSFM supplies a base forecast and an LLM adjusts it using context and experience. After observation, residual-guided evaluation selects decisions for validated memory only when adjusted forecasts improve on the base.

3.1 PROBLEM FORMULATION

  • JudgeCast uses a TSFM base forecast from the target’s past alone and an LLM that uses relevant experience to form covariate-wise judgments and determine a numerical adjustment.The framework operates over sequential forecasting windows.Source: §3 · p. 3, §3.1 · p. 3

3.2 FORECAST-TIME JUDGMENTAL ADJUSTMENT

  • Judgments assess each covariate’s expected effect relative to the base forecast, while the numerical adjustment is determined separately from those judgments collectively.Qualitative labels encode direction and strength: repeated signs indicate greater strength, and 0 means no meaningful effect.Source: §3.2 · p. 4
  • For current judgments, JudgeCast retrieves validated experience from preceding windows using similarity in normalized covariate values and the base forecast.For numerical adjustment, it retrieves matching-judgment experience using similarity in target context and base forecast; its adjustments serve as references.Source: §3.2 · p. 4

3.3 RESIDUAL-GUIDED EXPERIENCE CONSTRUCTION

  • After observation, the base-forecast residual guides generation of alternative covariate-wise judgments, excluding the original judgment and stored experience from alternative generation.The original judgment is included alongside the alternatives for subsequent evaluation.Source: §3.3 · pp. 4–5
  • JudgeCast evaluates original and alternative judgments through their resulting adjustments, selecting the judgment whose adjustment best matches the observed residual.The original judgment is re-evaluated without stored experience rather than reusing its forecast-time adjustment.Source: §3.3 · p. 5
  • The selected judgment and adjustment enter memory as validated experience only when the adjusted forecast improves on no adjustment; otherwise, memory remains unchanged.This validation does not establish that the selected judgment faithfully reflects covariate effects.Source: §3.3 · p. 5

4 EXPERIMENTS

Across diverse real-world forecasting tasks, JudgeCast outperforms strong baselines, while ablations and analyses examine its judgment and experience mechanisms. Results also cover covariate reliability, retrieval, and a PJM case study.

4.2 MAIN RESULTS

  • 16.7% lower MSE and 6.5% lower MAE on average than the strongest baseline; JudgeCast achieves the lowest values on all seven datasets and both metrics.The gap is particularly pronounced on long-term datasets, where inference-only LLM forecasters are less competitive.Source: Table 1 · p. 6, §4.2 · p. 6
  • All four tested TSFM–LLM backbone combinations outperform Table 1’s second-best method in MSE on NP and PJM, with comparable or better MAE.The tested combinations use TimesFM 2.5 or Toto 2.0-313M with Qwen3.5-27B or Gemini 3.5 Flash-Lite.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 ABLATION STUDY

  • On five short-term datasets, explicit covariate-wise judgments reduce MSE over direct adjustment by 3.4% without experience and 4.6% with raw experience.The improvement is less consistent on long-term tasks, where identical covariate judgments can require different numerical adjustments.Source: §4.3 · p. 7, Table 3 · p. 7
  • Validated experience reduces MSE and MAE by 12.1% and 9.7% on average versus no experience, outperforming both raw-experience settings across datasets.Unlike retaining only an improved original decision, validated experience reconstructs the forecast-time decision after observation.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 COVARIATE UTILIZATION ANALYSIS

  • On UK COVID, Rohlik Orders, and M5, JudgeCast improves target-only forecasts with the same covariates where direct conditioning provides no improvement.The results indicate that limited benefit from direct conditioning does not necessarily mean the covariates lack forecasting utility.Source: Table 4 · p. 7, §4.4 · p. 8
  • Under temporal misalignment, direct conditioning’s aligned-covariate advantage reverses at both tested offsets on NP and at 12 hours on DE.The 0 judgment rate rises mainly for the shifted covariate and remains relatively stable for the unchanged covariate.Source: Table 5 · p. 7, §4.4 · p. 8

4.5 EXPERIENCE ANALYSIS

  • As validated memory grows from 25% to 100%, average retrieval distance falls for both judgment retrieval and adjustment retrieval.Full memory yields the lowest MSE on all five short-term datasets and reduces MSE and MAE by 5.3% and 7.4% on average versus no experience; relevant retrieval at both stages yields the largest retrieval-setting gains.Source: §4.5 · pp. 8–9, Figure 3 · p. 8

4.6 CASE STUDY

  • Across PJM steps 9–16, JudgeCast reduces overestimation with a negative adjustment, while MemCast substantially overestimates; their MSEs are 1.4 and 39.6, respectively.JudgeCast uses retrieved covariate-wise COMED judgments from preceding weekend windows to form its downward adjustments.Source: §4.6 · p. 9, Figure 4 · p. 9

5 CONCLUSION

JudgeCast makes covariate-wise judgments explicit and uses accumulated experience to adjust a base forecast, then revisits decisions after observations to build validated experience. Across diverse real-world datasets, it improves on strong baselines.

5 CONCLUSION

  • Across diverse real-world datasets, JudgeCast consistently improves forecasting performance over strong baselines.Source: §5 · p. 9
  • At forecast time, JudgeCast uses accumulated experience to make explicit covariate-wise judgments and determine how to adjust the base forecast.Source: §5 · p. 9
  • After observations arrive, JudgeCast revisits forecast-time decisions to construct validated experience for subsequent forecasts.Source: §5 · p. 9

A.1 DATASET DETAILS

The experiments draw on short- and long-term real-world forecasting datasets and compare JudgeCast with statistical, neural, and LLM-based baselines. Dataset splits preserve a final chronological test portion.

A.1 DATASET DETAILS

  • The experiments reserve the final 20% of each dataset for testing and chronologically split the preceding 80% into 70% training and 10% validation when required.Dataset sizes match MemCast by taking the corresponding number of timestamps from each dataset's end.Source: §A.1 · p. 14
  • The short-term EPF datasets cover hourly prices in five regional markets, each paired with two market-specific forecast covariates.The markets are NP, PJM, BE, FR, and DE.Source: §A.1 · p. 14
  • Long-term and broader task coverage includes ENTSO-e Load and Rossmann Store Sales from fev-bench, alongside a fixed 10-store Rossmann subset.The Austrian ENTSO-e series uses weather covariates; fev-bench spans 100 tasks across seven domains.Source: §A.1 · p. 14

A.2 BASELINE DETAILS

  • The comparison also covers ARIMA, Prophet, DLinear, PatchTST, iTransformer, TimeXer, ConvTimeNet, and Time-LLM.Source: §A.2 · p. 15
  • Baselines include LSTPrompt, LLM-Time, TimeReasoner, and MemCast, spanning prompted LLM forecasting and accumulated-experience approaches.Source: §A.2 · p. 15

A.3 DATASET DETAILS FOR LIMITED DIRECT CONDITIONING ANALYSIS

The limited direct-conditioning analysis adds three daily forecasting tasks with varied targets and covariates. All use a context length of seven forecast horizons.

A.3 DATASET DETAILS FOR LIMITED DIRECT CONDITIONING ANALYSIS

  • Across the three tasks, the context length is L = 7H, and each multivariate target is evaluated separately.Each task retains its fev-bench forecast horizon H.Source: §A.3 · p. 15
  • UK COVID Nation covers daily measurements for four nations, forecasting cases, deaths, and admissions with five vaccination- and healthcare-related covariates at H = 28.Source: §A.3 · p. 15
  • Rohlik Orders covers daily order volumes from seven online-grocery warehouses, with 13 covariates and H = 61.Covariates describe calendar events, store operations, weather, and user activity.Source: §A.3 · p. 15
  • M5 1D forecasts daily Walmart item sales from price, event, and SNAP indicators at H = 28, excluding static attributes as separate covariates.Source: §A.3 · p. 15

B.1 TEMPORAL COVARIATE MISALIGNMENT RESULTS

The supplied passages describe the temporal covariate misalignment analysis and the measures reported, but do not provide its outcome values.

B.1 TEMPORAL COVARIATE MISALIGNMENT RESULTS

  • Table 7 reports forecasting MSE and MAE at 0-, 6-, and 12-hour offsets in the temporal covariate misalignment analysis.Source: §B.1 · p. 16, Table 7 · p. 16
  • Table 7 also reports the covariate-wise rate of 0 judgments for shifted and unchanged covariates across test-window horizon timestamps.Source: §B.1 · p. 16, Table 7 · p. 16
  • The analysis shifts grid load on NP and PV wind on DE while leaving the other covariate unchanged.Source: Table 7 · p. 16

B.2 EXPERIENCE ACCUMULATION AND RETRIEVAL BEHAVIOR RESULTS

The section tracks forecast accuracy and retrieval behavior as validated experience accumulates.

B.2 EXPERIENCE ACCUMULATION AND RETRIEVAL BEHAVIOR RESULTS

  • Experience accumulation is evaluated at 0%, 25%, 50%, 75%, and 100% of training windows, with MSE, MAE, and retained validated-experience counts and proportions reported.Retrieval behavior is tracked through average judgment- and adjustment-retrieval distances at each nonzero memory size.Source: §B.2 · p. 17

B.3 SELECTING RELEVANT EXPERIENCE RESULTS

The section describes how relevant-experience retrieval is compared and illustrates JudgeCast adjusting forecasts in opposite directions across two cases.

B.3 SELECTING RELEVANT EXPERIENCE RESULTS

  • The relevant-experience analysis compares similarity-based and random retrieval at judgment and adjustment stages, including all four combinations, using the full validated memory.It reports MSE, MAE, and average retrieval distance for each stage; no-experience is the reference, and relevant retrieval at both stages is full JudgeCast.Source: §B.3 · p. 17, Table 9 · p. 17

C.1 CASE VISUALIZATION

  • In two contrasting windows, JudgeCast adjusts upward for a sustained high price level and downward for an overestimated peak, achieving the lowest window-level MSE and MAE in both.It outperforms direct covariate conditioning with Chronos-2, the second-best method, in both cases.Source: §C.1 · pp. 17–18

C.2 FAILURE CASE ANALYSIS

The failure analysis identifies three ways forecast-time adjustments can miss the needed correction; the prompt appendix documents the three LLM roles.

C.2 FAILURE CASE ANALYSIS

  • An incorrect adjustment direction can increase error, especially when modest covariate departures coincide with opposite current residuals and inconsistent residual directions in similar windows.A substantial adjustment based on directional judgments can move the forecast considerably farther from the observation.Source: §C.2 · p. 18, Figure 7 · p. 20
  • An adjustment can move an already-close base forecast away from the observation when the current residual is unusually small and not evident at forecast time.Similar preceding covariate patterns may support the adjustment even though it is unnecessary in the current window.Source: §C.2 · p. 18, Figure 7 · p. 20
  • A within-horizon reversal can be missed when the needed correction changes direction but covariate judgments and applied adjustments remain largely one-sided.Similar preceding windows also do not consistently indicate a reversal in residual direction.Source: §C.2 · p. 18, Figure 7 · p. 20

D PROMPTS

  • Figures 8–10 provide prompts for forecast-time covariate-wise judgment, numerical adjustment determination, and post-observation alternative-judgment generation.The prompts are supplied for reproducibility.Source: D PROMPTS · p. 18, Figure 8 · p. 21, Figure 9 · p. 22

Related papers

Loading 2609.36966v2…