PaperlayerView exact revision on arXiv

Source-linked AI summary

GIFT-Eval-CTX: A Benchmark For Contextual Forecasting Model Evaluation

Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li

arXiv:2610.17794v1cs.LGcs.AI

TL;DR

Contextual forecasting benchmarks often lack broad domain coverage, standardized history–future splits, or text guaranteed to inform the future. GIFT-Eval-CTX uses structured effects derived from realized futures and an agentic annotation pipeline to evaluate numerical realization of that information. Structured effects lower aggregate relative MASE for both evaluated frontier LLMs, and MoiraiAgent achieves the best overall result.

  • Problem

    Contextual forecasting lacks reliable benchmarks with diverse domains, standardized history–future splits, and text guaranteed to contain future-relevant information.

    Source: §1 · p. 1

  • Method

    GIFT-Eval-CTX supplies portable Macro and Micro structured effects derived from realized futures, generated through agentic discovery and deterministic validation.

    Source: §1 · p. 2

  • Results

    Aggregate relative MASE decreases by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8 with structured effects versus history-only prompting, and MoiraiAgent achieves the best overall performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The analyses identify effect realization and coherent numerical dynamics as requirements for success, motivating stronger agentic forecasters or joint training on time series and structured effects.

    Source: §5 · p. 10

    Code-based editing can improve accuracy while suppressing local fluctuations or approximating recovery with an almost straight line.

    Source: §4.4 · p. 9

Abstract

from arXiv · show

Contextual forecasting seeks to improve time series forecasting by incorporating forward-looking information, such as planned events, external conditions, and expert expectations. However, reliable evaluation remains difficult: existing benchmarks often have limited domain coverage, lack standardized history-future splits, or provide text that is not guaranteed to inform the future. We introduce GIFT-Eval-CTX, a benchmark that isolates whether forecasting methods can translate verified future information into accurate numerical trajectories. Built upon GIFT-Eval, GIFT-Eval-CTX contains 10,000 instances across 20 datasets. Each instance provides structured effects derived from the realized future: Macro effects describe horizon-wide behavior, while Micro effects specify localized changes. We construct these annotations with an agentic pipeline combining LLM-based discovery and deterministic code validation. We benchmark TSFMs, frontier LLMs, and tool-using agentic systems. The results show that structured effects reduce aggregate relative MASE by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8 relative to history-only prompting. The evaluated MoiraiAgent system achieves the best overall result. We further analyze the contributions of structured effects and agentic systems to performance, and identify opportunities for future research.

1 INTRODUCTION

Contextual forecasting combines numerical histories with forward-looking information, but evaluating it is difficult when benchmarks lack domain diversity, standardized splits, or verified future-relevant text. GIFT-Eval-CTX evaluates whether models can turn verified future information into accurate trajectories, using structured effects and an agentic annotation pipeline; structured effects improve LLM results, and MoiraiAgent performs best overall.

Motivation

  • Existing contextual forecasting benchmarks often have limited domain diversity, lack standardized history–future splits, and do not ensure that text informs the future.Contextual forecasting seeks to combine numerical history with open-ended information such as planned events, external conditions, and expert expectations.Source: §1 · p. 1

Contribution

  • GIFT-Eval-CTX isolates forecasting from verified future information, rather than testing whether models extract or validate that information from raw sources.The benchmark builds on GIFT-Eval’s domain diversity and standardized training–test and history–future splits.Source: §1 · p. 2

Scope

  • The paper treats contextual understanding as an upstream capability handled by frontier LLMs and focuses on generating accurate numerical forecasts from verified future information.Contextual forecasting requires both interpreting diverse, domain-specific context and translating its implications into numerical trajectories.Source: §1 · pp. 1–2

Method

  • The benchmark represents context as portable key-value structured effects: Macro effects describe horizon-wide behavior, while Micro effects identify localized mechanisms and boundaries.An agentic pipeline proposes annotations that deterministic validators check before a Joint Critic reviews and commits them.Source: §1 · p. 2

Results

  • 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8: structured effects reduce aggregate relative MASE versus history-only prompting, while MoiraiAgent achieves the best overall performance.MoiraiAgent combines a frontier LLM with code-execution tools and a TSFM forecasting service.Source: §1 · p. 2

2 RELATED WORK

Prior multimodal time-series benchmarks cover understanding, question answering, pattern perception, and forecasting. GIFT-Eval-CTX complements forecasting benchmarks with broad domain coverage, standardized splits, and structured context tied to the realized future.

Multimodal time-series benchmarks

Multimodal forecasting benchmarks

  • Forecasting benchmarks pair series with manually constructed contexts, real text, metadata, images, or financial news, including Context is Key, Time-MMD (Liu et al., 2024a), and MoTime (Zhou et al., 2025).FinTexTS and FinMultiTime (Xu et al., 2025a) focus on financial forecasting with news matching and additional financial modalities.Source: §2.1 · p. 3

Benchmark gap

  • Existing forecasting benchmarks often lack broad domain coverage and standardized history–future splits, and their text is not guaranteed to describe the future.GIFT-Eval-CTX complements them with broad coverage, standardized splits, and structured context guaranteed to describe the realized future.Source: §2.1 · p. 3

3 GIFT-EVAL-CTX

GIFT-Eval-CTX pairs history and horizon information with structured future context, then compares forecasts with and without that context. It builds a filtered test corpus, represents future behavior through Macro and Micro effects, and generates annotations through agent discovery, validation, and review.

3.1 TASK DEFINITION

  • Paired evaluation compares the same forecaster given history and horizon alone with its forecast when structured future context is also supplied.The comparison assesses how effectively a model uses the supplied context.Source: §3.1 · p. 3

3.2 TIME SERIES CORPUS

  • The corpus uses held-out GIFT-Eval test regions, flattens multivariate series into univariate targets, and caps history length relative to forecast horizon.Windows are retained only when every evaluated TSFM exceeds an rMAE threshold, making the threshold comparable across different scales through future-range normalization.Source: §3.2 · p. 3

3.3 STRUCTURED EFFECTS SCHEMA

  • Structured effects use a domain-agnostic key-value schema to separate a single horizon-wide Macro description from localized Micro effects.Macro fields describe trend, seasonality, and history-normalized level range; Micro effects encode localized mechanisms, intervals, direction, and family-specific attributes.Source: §3.3 · pp. 3–4
  • Macro level range and Micro intensity use a shared scale normalized to the history minimum and maximum; future values may fall outside the 0–100 range.This normalization is intended to preserve relative scale across datasets with different units and magnitudes.Source: §3.3 · p. 4

3.4 STRUCTURED EFFECTS GENERATION PIPELINE

  • Macro and Micro discovery agents use numerical evidence and validators to propose and revise effect annotations.The Macro branch measures global trend, change points, and periodic structure; the Micro branch searches large TSFM residual regions and validates proposed effect types and boundaries.Source: §3.4 · pp. 4–5
  • A Joint Critic reviews discovery outputs and retains supported Micro effects, while uncertain candidates are excluded from the final annotation.The critic first forms an independent interpretation from raw history and future before inspecting the proposed effects.Source: §3.4 · pp. 4–5

4 EXPERIMENTS

The experiments evaluate TSFMs, frontier LLMs, and an agentic system on GIFT-Eval-CTX using structured effects and history-only inputs. Structured effects improve frontier-LLM forecasts, while MoiraiAgent achieves the best overall performance; analyses examine effect contributions and agent limitations.

4.1 EXPERIMENTAL SETUP

  • The benchmark contains 10,000 instances from 20 datasets, and experiments compare history-only TSFMs, frontier LLMs with and without effects, and a tool-using agent.Aggregate relative MASE is the geometric mean of normalized group-level MASE; lower is better, and seasonal naive scores 1.0 by construction.Source: §4.1 · p. 5

4.2 MAIN RESULTS

  • With history alone, both frontier LLMs underperform every TSFM overall, while all TSFMs outperform seasonal naive across all horizons.Both LLMs beat seasonal naive at short horizons, but not at medium and long horizons.Source: §4.2 · pp. 5–6
  • MoiraiAgent achieves the lowest overall MASE and leads at every horizon, while structured effects reduce overall MASE by 40.7% for GPT-5.6 Sol and 29.6% for Claude Opus 4.8.Relative to the best history-only TSFM overall, MoiraiAgent reduces MASE by 36.3%.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 ANALYSIS OF STRUCTURED EFFECTS

  • Micro effects contribute more individually than Macro effects, and combining both improves on Micro-only forecasts at every horizon for GPT-5.6 Sol and MoiraiAgent.Mean shifts yield the largest gains, excursions smaller gains, and seasonal changes only 6–9% overall; on long horizons, seasonal changes worsen GPT-5.6 Sol’s MASE from 1.070 to 1.119 but improve MoiraiAgent’s from 0.826 to 0.755.Source: §4.3 · pp. 6–7
  • Micro effects improve forecasts inside labeled regions and also improve trajectories outside them, while a seasonal-onset effect can preserve periodic dynamics despite worse sample-level MASE.Within-region MASE falls by 50.9% for GPT-5.6 Sol and 49.3% for MoiraiAgent; outside regions it improves from 0.890 to 0.802 and from 0.687 to 0.652, respectively. In the periodic example, MoiraiAgent’s MASE rises from 0.933 to 1.052 despite preserved oscillations.Source: §4.3 · pp. 7–8, Figure 2 · p. 8

4.4 FURTHER ANALYSIS OF MOIRAIAGENT

  • Direct LLMs often miss requested output lengths on longer horizons, whereas MoiraiAgent starts from an exact-length TSFM forecast and edits it with code.GPT-5.6 Sol produces the requested length for 38.9% of medium and 40.5% of long outputs; Claude Opus 4.8 does so for 2.3% and 1.2%. GPT-5.6 Sol follows the global level range on 97.41% of samples but every specified intensity on 81.57%.Source: §4.4 · pp. 8–9
  • Code-based edits can enforce event locations and intensities yet simplify local dynamics into plateaus or smooth transitions, even when sample-level MASE improves.The authors recommend pairing constraint enforcement with more flexible forecast editing, and identify joint training on time series and structured effects as another direction.Source: §4.4 · p. 9, Figure 3 · p. 9

5 CONCLUSION

GIFT-Eval-CTX isolates the numerical realization of verified future information through compositional Macro and Micro effects and agentic annotation with deterministic validation. Experiments show gains for frontier LLMs and best overall performance from MoiraiAgent, while the analyses motivate stronger agents or native effect modeling.

5 CONCLUSION

  • GIFT-Eval-CTX isolates the numerical realization of verified future information in time series forecasting.The benchmark represents context through compositional Macro and Micro effects and constructs annotations with an agentic pipeline and deterministic validation.Source: §5 · p. 9
  • MoiraiAgent achieved the best performance by combining LLM reasoning with forecasting and code-execution tools.The benchmark represents context as compositional Macro and Micro effects and uses an agentic pipeline with deterministic validation across diverse datasets.Source: §5 · pp. 9–10
  • Success requires realizing specified effects while maintaining coherent numerical dynamics throughout the forecast.The authors identify stronger agentic forecasters and joint training on time series and structured effects as future directions.Source: §5 · p. 10

AI USE STATEMENT

  • The authors used generative AI to generate annotations, assist code and results analysis, and support manuscript drafting and language polishing.They report manually inspecting annotation quality, testing code, verifying results, reviewing the manuscript, and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

A DETAILED BENCHMARK STATISTICS

GIFT-Eval-CTX contains 10,000 univariate forecasting instances from held-out regions of 20 source datasets. Its statistics describe dataset–frequency–horizon configurations and the scale and composition of the structured-effect annotations.

Instance distribution

Structured-effect distribution

  • 23,476 Micro effects appear across the benchmark, with a median of two and maximum of 15 effects per instance.Excursions account for 19,134 effects, seasonal changes 3,117, and mean shifts 1,225; Table 9 gives the subtype breakdown.Source: A DETAILED BENCHMARK STATISTICS · p. 13, Table 9 · p. 14

B.1 MACRO SCHEMA

The Macro schema describes the complete forecast horizon, while Micro entries encode localized mechanisms and their interval-specific properties. The taxonomy covers excursions, mean shifts, and seasonal changes, with examples illustrating their forms.

B.1 MACRO SCHEMA

  • Macro describes the complete forecast horizon, and Table 10 presents its complete schema.The supplied passages do not enumerate the Macro fields.Source: §B.1 · p. 14, Table 10 · p. 14

B.2 MICRO SCHEMA

  • Micro entries describe interval-specific mechanisms and can overlap when they represent different compatible mechanisms.Their fields include family, subtype, direction, amplitude trend, envelope change, inclusive boundaries, and a family-specific intensity.Source: §B.2 · p. 15
  • Micro excursion subtypes are impulse, sustained, and recurrent; mean shifts mark persistent level transitions, while seasonal changes describe altered periodic patterns.Excursion intensity uses an interval maximum or minimum for up or down events; seasonal-change intensity is defined separately by subtype.Source: §B.2 · pp. 15–16

B.3 MICRO EFFECT EXAMPLES

  • Figure 4 illustrates excursion subtypes, upward and downward mean shifts, and seasonal changes including amplitude, phase, period, onset, and disappearance.Its examples distinguish change relative to a preceding periodic region from amplitude evolution within the labeled interval.Source: §B.3 · p. 16

C.1 TIME SERIES CORPUS SELECTION

The benchmark selects non-overlapping GIFT-Eval windows that meet horizon, data-quality, and TSFM-difficulty criteria. An agentic pipeline proposes Macro and Micro effects, validates them against the series, and reviews annotations.

C.1 TIME SERIES CORPUS SELECTION

  • Candidates are GIFT-Eval test windows with horizons of 24–1,000 steps; multivariate collections become independent univariate targets, with history lengths set to 10H, 5H, or 3H by horizon configuration.Candidates with missing future values, over 20% missing history, or effectively constant finite history or future are discarded.Source: §C.1 · pp. 16–17
  • Windows are retained only when every reference TSFM has rMAE ≥ 0.10; retained windows do not overlap, and source caps limit dominance by large collections.Medium- and long-horizon configurations are capped at approximately 600 and 300 windows, respectively.Source: §C.1 · p. 18

C.2 STRUCTURED EFFECTS GENERATION

  • The Macro Agent proposes a horizon-wide object using deterministic measurements, while a validator checks its schema and replaces its level range with the value computed from the future.Trend and seasonality measurements are returned as evidence when they disagree with the proposal.Source: §C.2 · p. 18
  • Micro discovery targets regions with large residuals against an averaged TSFM reference, then proposes mechanisms and intervals from raw values and surrounding context.A deterministic family-specific validator checks numerical support, boundaries, and reference-forecast error, computes intensity, and can return evidence for revision; a blind-first critic reviews candidates before accepted effects enter the annotations.Source: §C.2 · p. 18

D.1 MODEL CONFIGURATIONS

The experiments compare univariate TSFMs, API-accessed frontier LLMs, and MoiraiAgent, which combines GPT-5.6 Sol with a Moirai-2-based agent harness. Separate prompts define history-only and history-plus-effects forecasting conditions.

D.1 MODEL CONFIGURATIONS

  • MoiraiAgent combines API-accessed GPT-5.6 Sol with a Moirai-2 tool and code-execution tools for inspecting and modifying forecasts.Runs allow at most ten agent steps and 16,000 output tokens, and require a finite, exact-length JSON forecast array.Source: §D.1 · pp. 18–19

D.2 LLM PROMPTS

  • The history-only prompt asks for forecasts from historical values, while the effects prompt adds structured Macro and Micro guidance alongside history.The effects prompt also supplies the Macro level range in raw-value units because direct LLMs sometimes miscalculate normalized-coordinate conversions.Source: §D.2 · pp. 19–20
  • Both prompting conditions require exactly H numerical forecast values in JSON, and the exact-length requirement is repeated because direct LLMs often returned the wrong number of values.The effects prompt additionally requires forecasts to remain within the supplied raw-value Macro range.Source: §D.2 · pp. 19–20

D.3 METRICS AND AGGREGATION

The evaluation computes MASE with a seasonal-naive fallback when the seasonal period is at least the history length, then aggregates results across dataset–frequency–term groups. The related-work discussion contrasts multimodal modeling approaches with this benchmark’s focus on frontier LLMs and agentic systems.

D.3 METRICS AND AGGREGATION

  • When the seasonal period m is at least the history length, the MASE denominator falls back to one-step naive error.Aggregation first averages per-sample MASE within each of 78 dataset–frequency–term groups, divides by the seasonal-naive group mean, then takes the geometric mean of ratios.Source: §D.3 · p. 20

E.1 TIME SERIES FOUNDATION MODELS AND MULTIMODAL MODELS

  • Prior multimodal time-series approaches use multimodal pretraining, language-model adaptation, or broader reasoning and unified understanding and generation.The discussion notes that many LLM-based approaches use small open-weight models, which may offer less world knowledge, reasoning, and agentic capability than frontier LLMs for complex contextual forecasting.Source: §E.1 · p. 20

Related papers

Loading 2610.17794v1…