PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecasting Supply Chain Disruptions with Foresight Learning

Benjamin Turtel, Paul Wilczewski, Kris Skotheim

arXiv:2604.01298v1cs.LG

TL;DR

The paper addresses how to forecast infrequent, high-impact supply chain disruptions from noisy, unstructured news before they materialize. It trains LLMs end-to-end with realized disruption outcomes as supervision, producing calibrated probabilistic forecasts that outperform strong baselines and exhibit more structured reasoning without explicit prompting.

  • Problem

    Supply chain disruptions are costly and difficult to anticipate because conventional indicators are delayed, revised, or only partially informative, while reliable forecasting from unstructured text remains challenging.

  • Method

    The framework trains LLMs end-to-end on timestamped news and realized disruption outcomes to directly produce calibrated probabilistic forecasts.

  • Results

    Fine-tuned LLMs substantially outperform strong baselines in predictive accuracy and calibration, with higher precision and more structured probabilistic reasoning without additional prompting.

  • Takeaways & Limitations

    Foresight learning can produce calibrated, forward-looking supply chain disruption forecasts from unstructured news.

  • Takeaways & Limitations

    The noisy and incomplete relationship between news and realized disruptions places an upper bound on predictive performance, and the task covers only one-month-ahead binary events.

Abstract

from arXiv · show

Anticipating supply chain disruptions before they materialize is a core challenge for firms and policymakers alike. A key difficulty is learning to reason reliably about infrequent, high-impact events from noisy and unstructured inputs - a setting where general-purpose models struggle without task-specific adaptation. We introduce an end-to-end framework that trains LLMs to produce calibrated probabilistic forecasts using realized disruption outcomes as supervision. The resulting model substantially outperforms strong baselines - including GPT-5 - on accuracy, calibration, and precision. We also show that training induces more structured and reliable probabilistic reasoning without explicit prompting. These results suggest a general pathway for training domain-specific forecasting models that produce decision-ready signals. To support transparency we open-source the evaluation dataset used in this study. Dataset: https://huggingface.co/datasets/LightningRodLabs/supply-chain-predictions

1 Introduction

The paper frames supply chain disruption forecasting as a real-time probabilistic prediction problem using noisy, unstructured news. It introduces end-to-end foresight learning to train LLMs to produce calibrated forecasts and more structured reasoning.

  • Motivation: Delayed or incomplete conventional indicators create a forecasting gap for costly supply chain disruptions.News can provide earlier signals about geopolitical tensions, trade restrictions, labor disputes, and related developments.
  • Forecasting task: The task estimates next-month disruption probabilities from recent news and requires aggregating predictive signals from evolving unstructured text.The model predicts whether an entity’s disruption index will experience a large increase in the following month.
  • Approach: End-to-end training jointly identifies salient textual signals, reasons about them in a domain-adaptive way, and produces outcome-aligned likelihood estimates.Unlike prior pipelines, the model directly forecasts from raw news rather than first extracting structured signals for downstream models.
  • Approach: The framework extends Foresight Learning to one-month-ahead supply chain disruption forecasts using timestamped news and realized outcomes in a forward-looking setting.Forecasting examples are constructed from future disruption outcomes and information available before the prediction time.
  • Contributions: The trained approach achieves lower Brier scores, reduced calibration error, and higher precision than pretrained models and baselines.Training also induces improved uncertainty handling and signal prioritization without additional prompting.

2 Related Work

Prior supply chain research largely emphasizes detection, structured risk measures, extraction, or decision support rather than direct out-of-sample forecasting from raw text. This paper applies foresight learning to end-to-end, time-varying disruption prediction.

  • Text-based monitoring: Earlier text-based approaches use sentiment analysis, topic modeling, and supervised classification to detect disruptions and monitor risk signals.Surveys characterize this literature as emphasizing detection and visibility rather than out-of-sample prediction.
  • Structured forecasting: Structured-data, network, and text-derived-feature methods can improve forecasting but depend on engineered or structured inputs.Their reliance on task-specific features limits direct use of raw textual signals.
  • Disruption indices: Disruption indices provide systematic tracking of realized global supply chain stress but do not incorporate textual information.These measures are constructed from granular trade data.
  • LLM applications: LLM supply chain applications primarily support information extraction, reasoning, and decision support, often feeding extracted signals into downstream predictive models.Hybrid frameworks demonstrate textual organization but typically combine LLMs with statistical or graph-based models.
  • Research gap: Relatively little prior work studies end-to-end prediction of time-varying disruptions using LLMs conditioned directly on raw text.The paper positions its approach within foresight learning over temporally aligned text and future outcomes.

3 Data and Problem Setup

The dataset links timestamped supply-chain news with country- and product-level disruption indexes to form temporally aligned, one-month-ahead forecasting questions. Events are defined as unusually large index increases, with chronological splits designed to avoid look-ahead bias.

  • Data sources: The data combine timestamped news with monthly country- and product-level disruption indexes constructed from granular international trade data.The news covers logistics, manufacturing, trade policy, commodities, and geopolitical developments.
  • Forecasting questions: Each forecasting question specifies an entity, prediction month, news available by that month, the current index value, and a binary outcome for the following month.Questions are natural-language, entity-specific, and time-indexed.
  • Prediction task: The model estimates the probability of a disruption event in month t+1 using only information available at month t.The task produces a numerical probability and a news-grounded explanation.
  • News construction: Recent relevant articles are retrieved through time-aware search and restricted to those published at or before the prediction month.The resulting context is intended to reflect information available when the forecast is made.
  • Event definition and labeling: Disruption events are unusually large increases in the disruption index, with entity-specific volatility scaling based on training-data standard deviations.This threshold accounts for differences in volatility across countries or products.
  • Temporal design: Labels use future index realizations while thresholds use training data only, preventing look-ahead bias.The sample begins in 2022 to avoid the extreme volatility associated with the COVID-19 shock, and the split is chronological.

4 Model and Training

The model is trained under foresight learning with temporally masked inputs and future disruption outcomes as supervision. A frozen GPT-OSS-120B base is specialized through LoRA, while a log-score reward targets accurate and calibrated probabilities.

  • Training framework: Foresight learning excludes post-t information from inputs while deriving labels from future outcomes.This preserves the temporal structure of forecasting during offline training.
  • Base model: The base model is GPT-OSS-120B, an open-source 120B-parameter decoder-only transformer pretrained on general-domain and technical text.It is selected as a strong general-purpose model.
  • Adaptation: LoRA with rank 32 adapts the model using few trainable parameters while keeping the base model fixed.Specialization therefore avoids full-parameter updates.
  • Inputs: Inputs concatenate index values, time-restricted news context, and a forecasting prompt specifying the entity and event definition.The inputs fit within the model context window without truncation or sliding-window mechanisms.
  • Objective: The model outputs a probability pe,t+1, and GRPO-style reinforcement learning evaluates it with a log-score reward under the realized outcome.The objective also supports optimization over full outputs, including probabilities and associated reasoning.
  • Optimization: Optimization updates LoRA parameters with standard policy optimization methods while freezing the base model weights.Reported results use the final converged checkpoint.

5 Results

On held-out, temporally subsequent forecasting questions, the fine-tuned model outperforms pretrained, historical, and frontier general-purpose baselines across probabilistic accuracy, calibration, and precision. Its probabilities better match realized disruption frequencies, and its highest-confidence predictions more often identify true disruptions.

  • Calibration: Nearly 70% lower ECE than the pretrained model indicates substantially closer alignment between predicted probabilities and observed disruption frequencies.ECE measures calibration, with lower values indicating more reliable probability estimates.
  • Aggregate performance: The fine-tuned model outperforms pretrained, historical, and frontier general-purpose baselines across all reported metrics.The evaluation uses Brier score, Brier skill score, expected calibration error, and Precision@10%.
  • Aggregate performance: The fine-tuned model achieves markedly higher Precision@10%, so its highest-confidence predictions more often correspond to true disruption events.Precision@10% measures precision among predictions ranked in the top 10% by predicted probability.
  • Calibration: ECE decreases from 0.1740 for the pretrained model to 0.0525 after training, indicating closer alignment between predictions and realized outcomes.The historical baseline predicts a constant rate and has no discriminative power.
  • Calibration: The reliability diagram shows trained-model probabilities tracking empirical frequencies, with higher predicted risk corresponding to higher observed disruption rates.This provides a visual view of calibration on the test set.

6 Discussion

Foresight-oriented training improves both supply-chain disruption forecasting and the structure of probabilistic reasoning, while the approach remains bounded by noisy news–outcome links, a one-month binary task, and a single post-2022 regime.

  • Interpretation of Results: Task-specific adaptation improves discrimination and calibration over historical baselines and prompted general-purpose LLMs.General-purpose models do not naturally produce well-calibrated forecasts in out-of-domain settings, whereas targeted supervision improves alignment with realized outcomes.
  • Interpretation of Results: News provides anticipatory information about policy shifts, labor disputes, and geopolitical tensions that may precede measurable disruptions.The model leverages these heterogeneous textual signals without feature engineering.
  • Interpretation of Results: Calibration improvements make forecasts more suitable for decisions requiring probabilities that reflect true event frequencies.The reported reduction in calibration error indicates more reliable probability estimates for downstream use.
  • Improvements in Reasoning Behavior: Fine-tuned models produce more structured and forward-looking reasoning, including clearer links between developments and future disruptions.They more consistently connect identified developments to the event definition than pretrained models.
  • Improvements in Reasoning Behavior: The average reasoning rubric score increases from 2.76 to 5.17 out of 6 after training.Observed changes include base-rate anchoring, explicit forecasting models, and iterative uncertainty refinement that emerge without explicit prompting.
  • Limitations: Predictive performance is constrained by the noisy and incomplete relationship between news and realized disruptions.The formulation also abstracts from persistence and multi-period risk accumulation, and training and evaluation cover only the post-2022 period.

7 Conclusion

The paper constructs a temporally aligned news–outcome dataset and trains an LLM to produce calibrated, forward-looking supply-chain disruption forecasts. Fine-tuned models outperform strong baselines in predictive accuracy and calibration, while the framework points toward broader forecasting applications.

  • 7 Conclusion: The study links recent news to future supply-chain disruption events through a temporally aligned dataset.Forecasting examples pair recent information with subsequent realized disruption outcomes.
  • 7 Conclusion: Fine-tuned LLMs substantially outperform strong baselines in predictive accuracy and calibration.The authors report that predicted probabilities more closely reflect realized outcomes.
  • 7 Conclusion: The results indicate that textual signals can be effectively leveraged for supply-chain forecasting.The conclusion frames the findings as evidence that LLMs can support quantitative prediction under uncertainty.
  • 7 Conclusion: Future work should extend the framework to multi-horizon and multi-event forecasting, additional data sources, interpretability, and robustness under distributional shifts.The paper specifically names structured trade flows and firm-level disclosures as possible additional inputs.

A Forecasting Prompt Template

The forecasting prompt combines a prediction task, relevant context, and a standardized probability output. Its example specifies the entity, timing, shock threshold, news context, information constraints, and resolution format.

  • A Forecasting Prompt Template: Each forecasting example combines a prediction task, relevant context, and a standardized output format.This structure is used to make the forecasting question and response format consistent.
  • Prompt Structure (Example): The instruction asks a supply-chain analyst to estimate next-month disruption probability from the current index and relevant news.The prompt is framed around countries or products and a one-month forecasting horizon.
  • Prompt Structure (Example): The example question identifies the October 2025 date, residues_waste entity, current index, prior-month change, and disruption threshold.It asks whether the index will increase by more than one historical standard deviation in the following month.
  • Prompt Structure (Example): The example resolves Yes when the index rises by more than 0.46 from October to November 2025 and No otherwise.The resolution rule operationalizes the binary forecasting target.
  • Prompt Structure (Example): The news context includes supply, demand, infrastructure, waste-to-energy, and environmental developments relevant to residues_waste.The context is explicitly abridged rather than presenting the full article texts.
  • Prompt Structure (Example): The required response is a probability between 0 and 1 with a brief explanation grounded in the news context.The probability must be returned in the specified <answer>p</answer> format.
  • Prompt Structure (Example): Inputs use only information available at prediction time, with news filtered for relevance and entities spanning countries and product categories.Event thresholds are computed from historical training data.

B Model Reasoning Before and After Training

The before-and-after comparison uses an identical forecasting question and prompt structure, then contrasts the models’ reasoning traces. The examples show progression from threshold and trend analysis toward explicit probabilistic modeling, updating, and news integration.

  • B Model Reasoning Before and After Training: Model outputs are compared before and after training on an identical forecasting question and prompt template.Formatting is simplified, while the reasoning is otherwise unchanged for comparison.
  • B Model Reasoning Before and After Training: The forecast target is whether the next month’s index increase will exceed the 0.46 shock threshold.The example computes the required future index level as 1.43 from a current index of 0.97.
  • B Model Reasoning Before and After Training: The reasoning weighs news about feedstock demand, supply pressure, and potentially mitigating technologies as mixed evidence.The example does not treat the news as implying a disruption increase automatically.
  • B Model Reasoning Before and After Training: The initial reasoning assigns a low disruption probability of 0.2 after judging the threshold jump large.The answer is returned in the prescribed probability format.
  • B Model Reasoning Before and After Training: The trained-style reasoning explicitly formulates the shock definition and compares the recent change with the threshold.It identifies the observed +0.28 change as below the 0.46 threshold.
  • B Model Reasoning Before and After Training: A random-walk calculation estimates a 0.35 probability for exceeding the threshold under a mean change of 0.28 and standard deviation 0.46.An alternative zero-mean base-rate calculation gives approximately 0.16.
  • B Model Reasoning Before and After Training: The reasoning refines the estimate to roughly 0.25–0.3 before incorporating mixed news about feedstock demand and disruption pressure.The final synthesis characterizes the threshold exceedance as a tail event and assigns 0.30.

C Automated Evaluation of Reasoning Behavior

The automated evaluation detects predefined probabilistic reasoning behaviors in model traces using a strict binary rubric and aggregates their frequencies across held-out examples. Its results are scalable but coarse: they indicate pattern presence rather than reasoning correctness or effectiveness.

  • C Automated Evaluation of Reasoning Behavior: The evaluator detects concrete probabilistic reasoning patterns rather than assessing overall reasoning quality.It complements quantitative forecasting evaluation.
  • C Automated Evaluation of Reasoning Behavior: The fixed rubric contains six binary behaviors: base-rate anchoring, statistical modeling, explicit forecasting models, evidence–outcome linkage, probabilistic synthesis, and uncertainty refinement.Each behavior captures a distinct aspect of probabilistic reasoning.
  • C Automated Evaluation of Reasoning Behavior: Uncertainty refinement measures whether reasoning revises or updates an intermediate estimate after additional considerations.This behavior is one of the rubric’s six defined categories.
  • C Automated Evaluation of Reasoning Behavior: Each behavior is annotated as present or absent for an individual reasoning trace.The resulting indicators are binary.
  • C Automated Evaluation of Reasoning Behavior: The evaluator compares pretrained and fine-tuned traces generated from identical held-out inputs and prompt templates.A language model independently evaluates each trace against the rubric.
  • C Automated Evaluation of Reasoning Behavior: Deterministic decoding produces structured JSON indicators, which are aggregated into mean behavior frequencies for each model.The evaluator uses temperature 0.
  • C Automated Evaluation of Reasoning Behavior: The evaluator uses a strict structured prompt that marks behaviors only when they are clearly and explicitly demonstrated.The prompt presents the trace and requests binary annotations for every behavior.
  • C Automated Evaluation of Reasoning Behavior: Because the evaluator is language-model-based, its metrics are coarse and potentially sensitive to phrasing; they should be read as relative differences, not exact reasoning-quality measures.The rubric prioritizes precision over recall and does not measure whether detected patterns are correct or effective.
Loading 2604.01298v1…