TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text

1 Introduction

• Forecast-time textual context remains underexplored in EPF, although official operational records can signal changing conditions before they appear in recent prices.A PJM Cold Weather Alert preceded nine COMED hourly prices exceeding the prior 14-day maximum.Source: §1 · p. 1
• Existing multimodal time-series benchmarks do not target EPF; concurrent NSW-EPNews reports limited gains but uses third-party commentary and GPT-4o-generated summaries.Those sources complicate forecast-time reconstruction and reproducibility.Source: §1 · pp. 1–2
• TRACE pairs prices from five PJM zones with official operational text reconstructed at each forecast cutoff, producing 7,300 zone–day instances without post-cutoff information leakage.Its construction applies unchanged as new official operational records arrive.Source: §1 · p. 2
• A median 7.4% reduction in upper-tail pinball loss across time-series foundation models accompanies semantic alignment strongest for upper-tail risk; cross-day text mismatch reverses gains below the no-text baseline.The benchmark also reports alignment with central movement and both tail risks.Source: §1 · p. 2

2 TRACE benchmark construction

• The forecast cutoff is 11:00 ET on the day before delivery, and numerical inputs comprise 336 hourly observations after daylight-saving-time normalization.Spring-forward gaps are interpolated and one fall-back observation is retained.Source: §2 · p. 2
• Publication time, cutoff-time cancellation status, and applicability intervals overlapping the target day determine which operational records enter TRACE, preventing post-cutoff information leakage.The text source is PJM’s Emergency Procedures archive; a 14-day lookback retains 87.6% of unlimited-lookback coverage while limiting stale context.Source: §2 · p. 2
• TRACE aligns preceding 14-day zone-specific prices with records published and uncancelled by the cutoff and applicable to the target day, yielding 7,300 instances across five zones and 1,460 days.The chronological train/validation/test split is 821/366/273; 702 records cover 400/146/131 days.Source: §2 · p. 2

3.1 Semantic alignment of textual context

• Assessors classify central movement and upper- and lower-tail risk from cutoff-available text, then compare those assessments with ground-truth price movements and risks.Central movement uses balanced accuracy; tail alignment uses Spearman correlation.Source: §3.1 · pp. 2–3
• The semantic-alignment construction is repeated with four-week same-weekday medians replacing previous-day references to account for recurring calendar patterns.Source: §3.1 · p. 3
• Positive, significant alignment occurs in 20/20 upper-tail settings, 17/20 central settings, and 16/20 lower-tail settings.Upper-tail alignment is most consistent, matching the strongest forecasting value reported for that risk.Source: Table 1 · p. 3, §3.1 · p. 3

3.2 Forecasting value of textual context

• Direct fusion combines encoded price history and official text, while structured fusion uses semantic assessments alongside baseline quantile forecasts and time-of-day features.Both are evaluated with pinball loss across quantiles; the figure distinguishes direct fusion (A) from structured semantic fusion (B).Source: §3.2 · pp. 3–4, Figure 3 · p. 4
• At τ=0.9, structured semantic fusion improves 70/70 settings over CNUM, with a median 7.4% reduction (range 1.4–19.3%).Direct text fusion reduces PB0.9 in all five zones by 3.7–9.1% versus NUMONLY; correct pairing beats mean re-pairings in all tested settings.Source: Table 2 · p. 4, §3.2 · p. 4

• TRACE is presented as the first reproducible EPF benchmark pairing prices with official operational text.Source: §4 · p. 4
• Semantic assessments align most strongly with upper-tail risk.Source: §4 · p. 4
• Forecasting gains are strongest for upper-tail risk.Source: §4 · p. 4

A.1 Official operational text source selection

• Under unlimited lookback, the latest publication before the forecast cutoff determines its age; “Within k days” measures how often that age is at most k days.The finite-window analysis applies publication and cancellation-status filters, and requires announced applicability intervals to overlap the delivery day.Source: §A.1 · p. 7, §A.2 · p. 7
• Emergency Procedures has a 1.15-day median publication age, versus 3.00 days for Inside Lines, and preserves timestamps needed to reconstruct earlier states.Inside Lines does not preserve superseded text, so its state at an earlier forecast cutoff cannot be reconstructed.Source: §A.1 · p. 7

A.2 Context window and coverage

• A 14-day window retains 677 of the full archive’s 773 context-bearing days (87.6%); extending to 28 days adds only 42 days while doubling maximum record age.The study therefore sets W = 14 days.Source: §A.2 · p. 7, Table 4 · p. 7

B.1 Frozen semantic assessment prompt

• Each day’s frozen assessment uses only reconstructed official text within the 14-day window, including record type, publication time, announced applicability start time, and body text.The assessment receives no price history, numerical forecasts, ground-truth prices, or previous-day or same-weekday reference values; outputs are frozen before alignment is computed.Source: §B.1 · pp. 7–8
• The prompt treats central direction and upper- and lower-tail risk as three independent assessments, using no-clear-signal labels when evidence is weak or conflicting.It asks assessors to consider event magnitude, timing, persistence, freshness, and operational relevance.Source: §B.1 · p. 8

B.2 Full five-zone semantic alignment

• Upper-tail price risk aligns positively and significantly for every zone–LLM pair under both references; central movement does so for 7/10 pairs and lower-tail risk for 6/10.Under the previous-day reference, central movement and lower-tail risk are nonsignificant for COMED, and lower-tail risk is also nonsignificant for AEP.Source: §B.2 · p. 8

C.1 Direct text fusion

• Correct day–context pairing beats the Shuffle mean in all five zones, with a median PB0.9 reduction of 20.7%; the Shuffle mean is worse than NUMONLY in all five.Across 131 held-out context-bearing test days, the aggregate result beats all 1,000 random re-pairings (p = 1/1001).Source: §C.1 · p. 9
• PB0.9 decreases in all five zones against NUMONLY, while PB0.1 increases in all five and PB0.5 increases in four of five.The reported pattern indicates that forecasting value is concentrated in upper-tail risk rather than broad distributional improvement.Source: §C.1 · pp. 8–9, Table 6 · p. 9
• The direct-text comparison uses a separately trained NUMONLY control with text features zeroed and covers five zones with 131 context-bearing test days each.Table 6 reports pinball loss at τ = 0.9 and relative reductions versus both NUMONLY and the shuffled-context control.Source: §C.1 · p. 8, Table 6 · p. 9

C.2 Structured semantic fusion

• The shuffled-context comparison tests whether gains depend on correctly matched forecast-time text rather than text-derived features alone.CNUM is the separately trained control with the semantic vector zeroed during training.Source: §C.2 · p. 9, Table 7 · p. 10
• At τ = 0.9, structured semantic fusion improves all 70/70 settings over CNUM, with a median reduction of 7.4% (range 1.4–19.3%).At τ = 0.5, 64/70 settings improve; at τ = 0.1, 24/70 improve.Source: §C.2 · p. 9, Table 7 · p. 10
• Correct forecast-time pairing beats the Shuffle mean in all 70/70 settings, with a median reduction of 13.9%; the aggregate beats all 1,000 random re-pairings (p = 1/1001).Numerical inputs and targets remain fixed while textual context is randomly re-paired across the 131 held-out context-bearing test days.Source: §C.2 · p. 9, Table 7 · p. 10