TimeCAP: Learning to Contextualize, Augment, and Predict Time Series Events with Large Language Model Agents

• Real-world time series often include contextual information, such as geographical, climatic, and economic factors, that is crucial for comprehensive analysis and effective modeling.Source: §1 · p. 1
• Prior LLM-based time-series methods primarily fine-tune or prompt LLMs as predictors, often overlooking contextual understanding.Source: §1 · p. 1
• 21.98% improvement in F1 scores comes from contextualization alone, while 28.75% improvement follows the addition of augmentation on real-world event-prediction datasets.Source: Figure 1 · p. 2
• TIMECP uses two independent LLM agents: one generates a contextual text summary, and the other uses it to make event predictions.Source: §1 · p. 2, Figure 1 · p. 2
• TIMECAP adds a multi-modal encoder that uses time-series data and generated summaries, then retrieves relevant training summaries as in-context examples for prediction prompts.Source: §1 · p. 2, Figure 1 · p. 2
• TIMECAP supports black-box LMaaS APIs, provides interpretable prediction rationales, and releases seven real-world datasets with generated contextual summaries.Source: §1 · p. 2

• LLMs are large models trained on massive text corpora and demonstrate broad performance across language tasks, supported by domain knowledge and zero-shot capabilities.Source: §2 · pp. 2–3
• In-context learning enhances LLM performance by providing a few input-label pairs as demonstrations.Source: §2 · p. 3
• LLM applications extend beyond language into computer vision, tabular data analysis, and audio processing.Source: §2 · p. 3

LLMs and Time Series

• Time-series approaches include training LLMs or smaller language models from scratch, fine-tuning pretrained LLMs, and prompt tuning parameterized time-series inputs.Source: §2 · p. 3

TIMECP: Contextualize and Predict

• TIMECP assigns separate LLM agents to summarize contextual information from a time series and predict future events from that summary.The contextualizer produces sx from x, while the predictor uses sx to estimate the event outcome.Source: §3.2 · p. 3

TIMECAP: Contextualize, Augment, Predict

• TIMECAP extends TIMECP with a trainable multimodal encoder that jointly uses raw time series and generated text summaries.The encoder learns representations and event predictions from both modalities, complementing the LLM agents.Source: §3.3 · p. 4, Figure 2 · p. 4
• The encoder embeds text and segmented time-series patches, applies multi-head self-attention across modalities, and produces a fused prediction.Text is projected into the encoder space, while each channel is divided into nonoverlapping patches before cross-modal attention.Source: §3.3 · p. 4
• TIMECAP retrieves the k nearest training summaries in the encoder embedding space and supplies their text-outcome pairs as in-context examples to the predictor.The retrieved examples augment the prediction prompt so AP can compare the test summary with relevant training cases.Source: §3.3 · p. 4
• The framework combines the multimodal encoder prediction with the LLM prediction through a weighted linear fusion.The fusion parameter λ controls the relative contribution of the LLM and multimodal predictions.Source: §3.3 · p. 4

Interpretation

• TIMECAP supports implicit rationales and explicit selection of the most relevant retrieved example as forms of prediction interpretation.The LLM can provide a rationale for its prediction or identify a useful example from the in-context set for comparison.Source: §3.4 · p. 5

Experimental settings

• Experiments use seven real-world datasets spanning weather, finance, and healthcare event-prediction tasks.The default setup uses GPT-4 for LLM agents and BERT within the multi-modal encoder.Source: §4.1 · p. 5

Accuracy

• TIMECAP ranks first on average across seven datasets using F1 and AUROC, outperforming other methods on most datasets.Table 2 compares TIMECAP and TIMECP with conventional time-series and LLM-based competitors, including zero-shot methods.Source: Table 2 · p. 6, §4.2 · p. 6

Component effectiveness

• Every TIMECAP component contributes to F1 improvement, including contextualization, input augmentation, and prompt augmentation.TIMECP outperforms PromptCast across all datasets; textual summaries complement time-series inputs, while multi-modal selection provides more meaningful demonstrations.Source: Table 3 · p. 6, §4.3 · p. 6

Interpretability

• In-context examples affect both prediction and interpretation, while selected examples support post-hoc comparisons between the input and semantically similar cases.Without examples, the LLM predicts solely from the input and can make incorrect predictions; with examples, it uses prior text-outcome relationships.Source: §4.4 · pp. 6–7, Figure 3 · p. 7

Further analyses

• The multi-modal encoder selects more useful in-context examples than PatchTST, as shown by higher average F1 scores across domains.These selected examples lead to more accurate predictions by the prediction agent.Source: Table 4 · p. 7, §4.5 · p. 7

Conclusion

• TIMECAP uses independent LLM agents for contextualization and prediction, supported by a trainable multi-modal encoder that mutually enhances them.The framework leverages LLM contextual understanding for time-series event prediction.Source: §5 · pp. 7–8
• Experiments on seven real-world time-series datasets from various domains demonstrate TIMECAP’s effectiveness.The paper makes the datasets available and states that the code is available upon request.Source: §5 · p. 8