PaperlayerView exact revision on arXiv

Source-linked AI summary

Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI

MAhdi Abolghasemi, Odkhishig Ganbold, Kristian Rotaru

arXiv:2312.06941v2cs.LGcs.CY

TL;DR

Systematic efforts in this area have been limited (Fildes et al., 2009; Lawrence et al., 2006). The study compares human and LLM forecasting across treatment conditions, including binary promotion presence. LLMs were not unequivocally superior to human forecasters.

  • Problem

    Systematic efforts in this area have been limited (Fildes et al., 2009; Lawrence et al., 2006).

    Source: §1 · p. 2

  • Method

    The study compares human and LLM forecasting across treatment conditions, including binary promotion presence.

    Source: §3.3 · p. 13, §5.1 · p. 27

  • Results

    LLMs were not unequivocally superior to human forecasters.

    Source: §5 · p. 27, §5.1 · p. 27

  • Takeaways & Limitations

    LLMs should not be assumed unequivocally superior when considering their integration into practical forecasting processes.

    Source: §5 · p. 27

    Graduate students were the primary human forecasters.

    Source: §5.3 · p. 30

Abstract

from arXiv · show

This study investigates the forecasting accuracy of human experts versus Large Language Models (LLMs) in the retail sector, particularly during standard and promotional sales periods. Utilizing a controlled experimental setup with 123 human forecasters and five LLMs, including ChatGPT4, ChatGPT3.5, Bard, Bing, and Llama2, we evaluated forecasting precision through Mean Absolute Percentage Error. Our analysis centered on the effect of the following factors on forecasters performance: the supporting statistical model (baseline and advanced), whether the product was on promotion, and the nature of external impact. The findings indicate that LLMs do not consistently outperform humans in forecasting accuracy and that advanced statistical forecasting models do not uniformly enhance the performance of either human forecasters or LLMs. Both human and LLM forecasters exhibited increased forecasting errors, particularly during promotional periods and under the influence of positive external impacts. Our findings call for careful consideration when integrating LLMs into practical forecasting processes.

1. Introduction

The study examines whether LLMs match human forecasters and how forecasting conditions shape accuracy. Its findings show that LLM performance varies and is not uniformly better than human performance.

1. Introduction

  • Controlled evidence on experts’ forecast modifications and LLM accuracy in complex promotional periods remains limited.The study identifies a need to examine advanced forecasting methods and qualitative information about past and upcoming promotions.Source: §1 · p. 2
  • The study compares human experts and LLMs under controlled conditions, varying statistical-model support, promotions, and external variables.Forecasters received no prior training or feedback on forecast accuracy.Source: §1 · p. 3
  • The study compared human forecasters and LLMs using forecasting accuracy measured by APE, finding that LLMs did not uniformly exceed humans; ChatGPT-4 and Bing performed comparably to humans.Source: §1 · p. 4
  • Human and LLM responses varied with sales data, forecast type, promotional periods, and external factors.The study frames these conditions as relevant to assessing forecasting performance.Source: §1 · p. 4
  • The comparison is intended to inform demand planning and forecasting-support systems that incorporate human expertise.The authors also identify collaborative human–AI decision support as a direction for further research.Source: §1 · p. 4

2. Background and hypothesis development

This section develops hypotheses for comparing human and LLM forecasting accuracy across baseline and advanced models, promotional periods, and external conditions. It also considers how these factors and their interactions may shape forecasting performance.

2.1. Background and literature review

  • The study compares baseline and advanced promotional forecasts and asks whether LLMs change understanding of judgmental forecasting performance, especially during promotions.This comparison also investigates whether forecast models support human analytical capabilities and judgment in complex promotional settings.Source: §2.1 · p. 6

2.2. Hypothesis development

  • Hypotheses assess humans and five LLMs across baseline versus advanced forecasts, promotional and external impacts, controlling for actual sales to isolate their effects on accuracy.The analysis considers each forecaster individually and controls for actual sales because sales fluctuations can skew absolute percentage error.Source: §2.2 · p. 7

2.2.1 Comparative Accuracy of LLMs and Human Forecasters (H1)

  • The H1 rationale contrasts human forecasters’ contextual knowledge and cognitive biases with LLMs’ data-processing strengths and potential hallucinations (Kahneman & Tversky, 1974; Chapman & Johnson, 1999; Lawrence et al., 2006).It motivates evaluating LLM forecasting accuracy against human forecasters without assuming either will perform better.Source: §2.2.1 · p. 7
  • LLMs were hypothesized to surpass humans in forecasting accuracy at comparable actual-sales levels, including in complex scenarios such as promotions.Source: §2.2.1 · pp. 7–8

2.2.2 Enhanced Forecasting Performance with Advanced Forecast Models (H2)

  • Advanced forecasts were hypothesized to improve both groups (H2a), with LLMs expected to use them more effectively than humans (H2b).Source: §2.2.2 · pp. 8–9

2.2.3 Variability in Forecast Accuracy During Promotional and Non-Promotional Periods (H3)

  • Accuracy was expected to differ between promotional and non-promotional periods for both groups (H3a), with a less pronounced difference for LLMs (H3b).Source: §2.2.3 · p. 9

2.2.4 Impact of External Conditions on Forecasting Disparities (H4)

  • Both groups were expected to vary in accuracy across negative and positive external impacts, with humans showing a greater difference than LLMs (H4a–H4b).The section attributes the predicted disparity to human sensitivity to emotional and cognitive biases.Source: §2.2.4 · p. 10

2.2.4 Variability in Forecast Accuracy under Complex Forecasting Scenarios (H5)

  • The complex-scenarios discussion frames accuracy as depending on interactions among advanced forecasts, promotion status, and external impacts, which LLMs may navigate systematically.The text describes six treatment conditions and suggests that multiple, sometimes conflicting inputs can strain human forecasters; the supplied passage ends before stating H5.Source: §2.2.4 · pp. 10–11
  • Human forecasters’ accuracy is hypothesized to vary across conditions involving advanced forecasts, promotions, and external impacts.This is stated as Hypothesis 5a.Source: §2.2.4 · p. 11
  • When the model incorporates these conditions, LLMs are hypothesized to forecast more accurately and consistently than humans.The passage characterizes this prediction as generally lower forecasting accuracy that fluctuates less across changing conditions.Source: §2.2.4 · p. 11

3. Method

The study compared 123 human forecasters with five LLMs on retail sales series, varying statistical forecast support, promotion status, and external impact across six experimental conditions.

3.1 Participants

  • 123 business-school graduate students with time-series forecasting experience were compared with five LLMs: ChatGPT3.5, ChatGPT4, Bard, Bing, and Llama2.The students had completed coursework applying forecasting models to stationary and non-stationary data.Source: §3.1 · pp. 11–12

3.2 Design

  • The mixed design varied statistical forecast type, promotion status, and external impact; advanced forecasts were used only when promotions were active.These factors produced six experimental conditions.Source: §3.2 · p. 12
  • Each participant evaluated the same 24 time series, with forecast accuracy measured by APE, where lower values indicate more accurate forecasts.APE compares forecasted values with actual sales and is expressed as a percentage.Source: §3.2 · p. 12

3.3 Materials and Apparatus

  • The case-study external-impact signal represented competitor-related sales increases or decreases of up to 10%; the authors note that other contexts may differ.The experimental series were genuine sales data from an Australian food manufacturer.Source: §3.3 · pp. 13–14
  • The dashboard supplied a baseline Holt-Winters forecast that excluded promotion and external impact, or an advanced dynamic linear regression forecast that accounted for promotion.The advanced forecast did not account for external impact.Source: §3.3 · p. 14

3.4 Procedures

  • Participants forecast Period 25 sales for 24 genuine product time series, which included promotion and non-promotion periods; immediate post-promotion periods were excluded.The series came from an Australian food manufacturer.Source: §3.4 · p. 14
  • The LLMs received the same information as human participants, and the same prompt was used for all LLMs.The LLM experiment was conducted during the same week as the human experiment in August 2023.Source: §3.4 · p. 15

4. Results

Mixed-effects regression found that forecasting accuracy varied by LLM: ChatGPT3.5 and Llama2 had higher APE than humans, while Bard, Bing, and ChatGPT4 did not differ significantly. The analysis also modeled forecaster and experimental-condition variation.

4. Results

  • The main analysis used linear mixed-effects models with restricted maximum likelihood to handle repeated measures, time-varying covariates, and correlated data.The authors describe these models as flexible and able to represent realistic variance and correlation patterns.Source: §4 · p. 15
  • The models treated APE as the outcome and included actual sales as a covariate, experimental conditions and their interactions as fixed effects, and forecaster as a random effect.Actual sales was included to mitigate the influence of sales-volume variation on APE.Source: §4 · pp. 15–16
  • For tests within each forecaster type, the authors used regression models because one observation per LLM forecaster did not support random-effects testing.The paper reports that fuller results for H2a–H5a are in Appendix J.Source: §4 · p. 16

4.1 Comparative Accuracy of LLMs and Human Forecasters (H1)

  • ChatGPT3.5 and Llama2 had significantly higher APE than human forecasters, while Bard, Bing, and ChatGPT4 showed no significant APE difference from humans.The mixed-effects results challenge the hypothesis that LLMs would surpass humans and indicate that performance varied by model.Source: §4.1 · p. 16, Table 2 · p. 17

Advanced Forecast Models (H2)

This section examines how advanced statistical forecasts relate to accuracy across forecasters and scenarios, including promotional periods and external impacts. It also covers model-specific estimates across treatment combinations and human forecasters’ views of advanced forecasts and promotional influence.

Advanced Forecast Models (H2)

  • Advanced forecasts did not significantly change accuracy for human forecasters or any individual LLM compared with baseline forecasts.The analysis tested humans with a mixed-effects regression and each LLM separately.Source: Advanced Forecast Models (H2) · pp. 17–18
  • When using advanced forecasts, Bard, ChatGPT3.5, and Llama2 exhibited significant APE increases of 10.73%, 10.25%, and 9.79%, respectively, compared with human forecasters.For Bard, the interaction term with the advanced forecast type significantly decreased APE by -14.64%, partially supporting H2b.Source: Advanced Forecast Models (H2) · p. 18

4.3 Variability in Forecast Accuracy During Promotional and Non-Promotional Periods (H3)

  • Promotional periods significantly increased APE for humans, ChatGPT3.5, and Llama2, while Bard, Bing, and ChatGPT4 showed no significant changes.The reported increases indicate lower forecasting accuracy during promotions for the affected forecasters.Source: §4.3 · p. 19

4.4 Impact of External Conditions on Forecasting Disparities (H4)

  • Llama2’s interaction effects suggested a complex response, including decreased APE under positive external impact.The findings suggest LLMs did not uniformly mitigate positive external influences.Source: §4.4 · p. 21
  • Positive external impacts were not significantly associated with an overall APE increase, but ChatGPT3.5 and Llama2 increased APE versus humans; Bard, Bing, and ChatGPT4 showed no significant changes.The results did not support the predicted smaller LLM–human APE difference.Source: §4.4 · p. 21, Table 5 · p. 22

4.5 Variability in Forecast Accuracy under Complex Forecasting Scenarios (H5)

  • For humans, APE increased under promotion with positive impact and an advanced forecast, and under promotion with negative impact and a baseline forecast; it decreased with no promotion, positive impact, and a baseline forecast.These condition effects show that forecast accuracy varied with the combination of promotion, external impact, and forecast method.Source: §4.5 · p. 23
  • Bard’s APE decreased significantly in almost all treatment conditions compared with the baseline, indicating improved accuracy in altered conditions, except for “Yes promotion, Negative external impact, Baseline forecast.”Source: §4.5 · p. 23
  • H5 estimates varied across forecasters and treatment combinations, with statistically significant entries for Llama2, Bard, and ChatGPT3.5.The reported combinations vary by promotion status, external impact, and baseline versus advanced forecast condition.Source: §4.5 · p. 25

4.5 Post-Experimental Questionnaire: Human Forecasters Perspective

  • Advanced statistical forecasts received a mean importance rating of 5.82, compared with 5.30 for baseline forecasts.Across the information factors assessed, importance ratings ranged from 5.11 to 5.68; participants showed mild to moderate agreement, with mean scores around the mid-5 range, that promotions influenced their judgment.Source: §4.5 · p. 26
  • Participants’ judgmental forecasting accuracy had a mean rating of 4.44 out of 7, with 7 labeled very high; the results indicated a modestly positive self-assessment.Source: §4.5 · p. 26

5. Discussion and conclusion

The discussion and conclusion synthesize comparisons of human and LLM retail forecasting across models and market conditions, emphasizing variation by model and context rather than a universal LLM advantage. They highlight context-specific, cautious integration, limits to generalizability, and priorities for further validation and forecasting systems attentive to human and AI capabilities and limitations.

5. Discussion and conclusion

  • LLMs did not uniformly outperform human forecasters: ChatGPT-4 and Bing were sometimes on par with humans, while ChatGPT3.5 and Llama2 lagged.The findings support choosing models with regard to their capabilities and the forecasting task rather than assuming general LLM superiority.Source: §5 · p. 27, §5.1 · p. 27

5.1 Discussion of main findings

  • Advanced forecasting models did not universally improve human forecasts, and both people and LLMs struggled to maintain accuracy across promotional and non-promotional periods.Errors increased for almost all forecasters during promotions, with the largest increase for ChatGPT3.5; humans and ChatGPT4 maintained relatively lower errors.Source: §5.1 · pp. 27–29
  • Under positive external impacts, most models except humans tended to have increased errors, while Llama2 reduced its APE and responded differently from humans, Bing, and ChatGPT.Humans, Bing, and ChatGPT made substantial positive adjustments under positive impacts and smaller adjustments under negative impacts, with accuracy remaining consistent across both conditions.Source: §5.1 · pp. 28–29
  • Humans generally achieved the lowest median and mean MdAPE across all categories of conditions, suggesting higher overall accuracy than other forecasters; ChatGPT3.5 had higher errors under promotional and advanced forecasting conditions.Source: §5.1 · p. 28

5.2 Implications

  • The study compares human and LLM forecasting under different conditions, challenges the assumption that LLMs are inherently superior, and highlights context and model specificity.Source: §5.2 · p. 30
  • Retail practitioners should avoid over-relying on LLMs and consider blending AI with human judgment, particularly during promotions when increased APE values may yield suboptimal forecasts.Source: §5.2 · p. 30
  • Policymakers and regulators should establish frameworks that recognize the strengths and limitations of humans and LLMs and encourage AI integration with human judgment.Source: §5.2 · p. 30

5.3 Limitations

  • Using graduate students rather than seasoned professionals as human forecasters may limit how well the findings generalize to professional forecasting settings.Source: §5.3 · pp. 30–31
  • The study used 24 real-world FMCG time series and examined only promotion status and binary positive or negative external impacts of up to 10% on sales.Source: §5.3 · p. 31
  • The selected LLMs represent only part of a rapidly evolving field, and model performance varies, motivating continuous evaluation of each model in forecasting contexts.Source: §5.3 · p. 31
  • LLM forecasting can be compromised by plausible but incorrect outputs and the absence of real-time data, so the authors call for validation, human oversight, and current domain-specific data.Source: §5.3 · pp. 31–32

5.4 Future research

  • Future studies should include experienced retail forecasters and broader, more realistic scenarios to test performance across market conditions.Source: §5.4 · p. 32
  • Future research should develop prompt-based forecasting models using companies’ in-house data to examine AI performance across forecasting tasks.The proposed work would also inform the integration of AI and human expertise in advanced forecasting.Source: §5.4 · p. 33
  • Future studies should test passive-to-highly-interactive LLM configurations to assess effects on prediction range and variance and whether they undermine the wisdom-of-the-crowd effect.The passage cites Schoenegger et al. (2024b) in connection with the wisdom-of-the-crowd effect.Source: §5.4 · p. 33
  • Research should examine whether repeated LLM use enhances or degrades human forecasting skills, to guide integration without diminishing human expertise.The passage cites Schoenegger et al. (2024a) in connection with preserving human expertise.Source: §5.4 · p. 33

5.5 Concluding remarks

  • ChatGPT4 and Bing achieved accuracies comparable to humans, while ChatGPT3.5 and Llama2 performed less well; LLM effectiveness varied by model and task.The study involved human forecasters, primarily business school graduate students, and five leading LLMs including ChatGPT4.Source: §5.5 · p. 33
  • Advanced forecasting models did not automatically improve performance, and both humans and LLMs showed limitations with them.The findings call for careful consideration when integrating sophisticated models into forecasting.Source: §5.5 · p. 33
  • Both human and AI forecasters had increased APE values during promotional periods, highlighting a shared difficulty with unpredictable market conditions.The study identifies a need for forecasting models better able to handle these complexities.Source: §5.5 · p. 33
  • The study calls for further research on combining human judgment and advanced AI models to optimize their complementary strengths.This recommendation follows the study’s findings on human and AI forecasting performance.Source: §5.5 · p. 33

Appendix A: Experiment guidelines to human forecasters

The human-forecaster guidelines and LLM prompt set up a shared retail sales task: forecast period 25 from 24 sales periods, with promotion, external-impact, and statistical-forecast information. The human interface presented time-series information and a forecast, while LLMs received tabular data and parallel instructions.

Appendix A: Experiment guidelines to human forecasters

  • Human participants forecast period 25 for 24 product sales series, with accuracy evaluated using Mean Absolute Percentage Error.The guidelines state that internal and external sales factors are assumed to be summarized in the provided information.Source: Appendix A · p. 39
  • The human dashboard supplied time-series observations, period-25 promotion and external-impact information, and baseline or advanced statistical forecasts.The guidelines describe Figure 1 as a screenshot of the forecasting interface.Source: Appendix A · pp. 39–40, Figure 1 · p. 39
  • The baseline forecast uses simple exponential smoothing and omits promotion and external-impact effects, whereas the advanced forecast uses dynamic linear regression and accounts for promotion.External impact summarizes competitors’ activity and its positive or negative effect on sales.Source: Appendix A · p. 40

Appendix B: LLM prompts.

  • LLMs received the sales history and forecasting inputs in a table, rather than the time-series plot shown to human participants, and were asked to predict period 25.The prompt describes the same 24 series and 24 observations as the human task.Source: Appendix B · p. 42

MdAPE

The appendix headings separate MdAPE summaries by adjustment direction and statistic.

MdAPE

  • The appendix identifies mean MdAPE for positive adjustments as a reporting category.No value is stated in the supplied passage.Source: MdAPE · p. 46
  • Median MdAPE for positive adjustments is listed separately from its mean.No value is stated in the supplied passage.Source: MdAPE · p. 46
  • Negative adjustments are likewise listed under mean and median MdAPE categories.The headings distinguish the adjustment direction and statistic.Source: MdAPE · p. 46

APPENDIX F: Mean, Median, and Standard Deviation of Responses to the Post-Experimental

The appendix reports participants’ ratings of forecasting information and their retrospective approaches to period-25 judgments.

APPENDIX F: Mean, Median, and Standard Deviation of Responses to the Post-Experimental

  • With baseline forecasts, participants rated forthcoming promotion type 5.50 and external impact 5.18 in importance on a 1–7 scale.Both medians were 6; standard deviations were 1.52 and 1.63, respectively.Source: Appendix F · p. 50
  • With advanced forecasts, participants rated forthcoming promotion type 5.68 and external impact 5.46 in importance on a 1–7 scale.Both medians were 6; standard deviations were 1.26 and 1.41, respectively.Source: Appendix F · p. 50
  • Participants rated their forecast accuracy after the experiment on a 1–7 scale and answered whether they relied on similar or all time-series periods.The appendix also asks whether earlier, later, or all promotions most influenced their period-25 judgment.Source: Appendix F · p. 50
Loading 2312.06941v2…