PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament

Philipp Schoenegger, Peter S. Park

arXiv:2310.13014v1cs.CYcs.AIcs.CLcs.LG

TL;DR

Whether LLMs can make accurate probabilistic predictions about future events remains an open research question, especially when benchmark answers may have appeared in training data. The paper evaluates GPT-4 in a real-world forecasting tournament, where answers are unknown at prediction time, and finds that it underperforms median human-crowd forecasts and does not outperform a 50% baseline.

  • Problem

    Evidence about LLM probabilistic forecasting is limited, and benchmark success may reflect memorized answers rather than understanding that generalizes.

    Source: §1 · p. 1

  • Method

    The authors entered GPT-4 in a forecasting tournament and compared its forecasts with human-crowd predictions.

    Source: §1 · p. 2

  • Results

    GPT-4 underperformed median human-crowd forecasts and did not outperform a no-information strategy assigning 50% to every question.

    Source: §3 · p. 5, §4 · p. 7

  • Takeaways & Limitations

    Forecasting tournaments may provide a fruitful setting for probing whether LLMs can generalize reasoning and prediction to future events.

    Source: §4 · p. 9

    Current models like GPT-4 may be unlikely to make worthwhile additions to forecast-aggregation algorithms until LLM capabilities improve.

    Source: §3 · p. 7, §4 · p. 9

Abstract

from arXiv · show

Accurately predicting the future would be an important milestone in the capabilities of artificial intelligence. However, research on the ability of large language models to provide probabilistic predictions about future events remains nascent. To empirically test this ability, we enrolled OpenAI's state-of-the-art large language model, GPT-4, in a three-month forecasting tournament hosted on the Metaculus platform. The tournament, running from July to October 2023, attracted 843 participants and covered diverse topics including Big Tech, U.S. politics, viral outbreaks, and the Ukraine conflict. Focusing on binary forecasts, we show that GPT-4's probabilistic forecasts are significantly less accurate than the median human-crowd forecasts. We find that GPT-4's forecasts did not significantly differ from the no-information forecasting strategy of assigning a 50% probability to every question. We explore a potential explanation, that GPT-4 might be predisposed to predict probabilities close to the midpoint of the scale, but our data do not support this hypothesis. Overall, we find that GPT-4 significantly underperforms in real-world predictive tasks compared to median human-crowd forecasts. A potential explanation for this underperformance is that in real-world forecasting tournaments, the true answers are genuinely unknown at the time of prediction; unlike in other benchmark tasks like professional exams or time series forecasting, where strong performance may at least partly be due to the answers being memorized from the training data. This makes real-world forecasting tournaments an ideal environment for testing the generalized reasoning and prediction capabilities of artificial intelligence going forward.

1 Introduction

The paper asks whether LLM benchmark performance reflects generalizable understanding or memorization of training data. It proposes testing prediction on future events whose answers are unknown at forecast time, and evaluates GPT-4 against human forecasters in a tournament.

1 Introduction

  • The study frames benchmark success as ambiguous because correct answers may reflect memorized training examples rather than understanding that generalizes to unseen questions.The authors note that genuine understanding is difficult to separate rigorously from memorization.Source: §1 · pp. 1–2
  • Future-event forecasting offers an evaluation context where the true answers are initially unknown, unlike benchmarks whose questions and answers may appear in training data.The paper distinguishes this form of forecasting from zero-shot extrapolation of time series.Source: §1 · p. 2
  • The study evaluates GPT-4 in a forecasting tournament and compares its probabilistic predictions with human-crowd forecasts.The tournament setting tests performance where answers are not known beforehand and supports comparison with human forecasters.Source: §1 · p. 2
  • Human crowds are presented as a strong comparison because tournament aggregates are often more accurate than individual forecasts, while outperforming them would be consistent with deep predictive understanding.The paper identifies probabilistic reasoning, generalization, and accurate prediction as relevant capabilities.Source: §1 · p. 2

2 Methods

The study entered GPT-4 in Metaculus’s Quarterly Cup and compared its initial forecasts with same-day crowd medians. The analysis focused on binary questions, using a prompt designed to elicit explicit probabilistic forecasts and rationales.

2 Methods

  • The study used Metaculus’s three-month Quarterly Cup, launched July 3, 2023, as a tournament setting for comparing GPT-4 and human forecasters.Metaculus hosts public forecasting competitions in which predictions are scored for accuracy.Source: §2 · p. 2
  • The analysis included 23 binary questions from a 51-question tournament with 843 participants, because binary questions were the types GPT-4 could answer without additional human input.The questions covered varied topics and were vetted by Metaculus moderators; Table 1 gives examples.Source: §2 · pp. 2–3, Table 1 · p. 3
  • The prompt asked GPT-4 to act as a superforecaster and provide numerical probabilities with rationales informed by base rates and comparison classes.The web interface was used at its default temperature; the prompt aimed to elicit explicit forecasts and improve their accuracy.Source: §2 · p. 3
  • GPT-4 received each question’s background information, resolution criteria, and question text; when the crowd median was not already visible, it was included in the description.The information was intended to mitigate the temporal gap between the model’s training-data cutoff and question launch.Source: §2 · p. 3
  • The researchers did not revise GPT-4’s initial forecasts because choosing when and how to update would require human involvement, while repeated updating is fundamental to crowd-aggregated forecasting.The design compared initial forecasts to avoid confounding from extended human updating and minimize bias in prompting model updates.Source: §2 · p. 4
  • GPT-4 forecasts were compared with the median human-crowd forecast recorded at the close of the same day, controlling for extended-period human updating.The comparison used initial forecasts; the dataset included 23 binary questions and a median of 37 human forecasters at the comparison point.Source: §2 · p. 4

3 Results

GPT-4 significantly underperformed median human-crowd forecasts, while its accuracy was not significantly different from the no-information baseline. Exploratory analyses found no significant midpoint-variance or duration effects.

3 Results

  • B = .20 (SD = .18) for GPT-4 versus B = .07 (SD = .08) for humans; only human-crowd accuracy significantly beat the 0.25 no-information baseline.GPT-4 did not differ significantly from that baseline, t(22) = −1.23, p = 0.23; the human-crowd comparison was t(22) = −10.69, p < .001.Source: §3 · p. 4
  • In 18 of 23 questions, median human-crowd forecasts were directionally closer to the truth than GPT-4 forecasts, χ2(1) = 12.52, p = .001.Figure 1 presents paired probability forecasts for each question, separated by outcome.Source: §3 · p. 4, Figure 1 · p. 5
  • ∆ = .13 in mean accuracy significantly favored median human-crowd forecasts over GPT-4, t(44) = 3.11, p = .003, Cohen’s d = .94.Figure 2 shows the Brier-score distributions by condition.Source: §3 · p. 5, Figure 2 · p. 6
  • 42.14% versus 57.59% CV around the probability midpoint indicated lower GPT-4 dispersion, but Levene’s test found no significant variance difference, F(1, 44) = .54, p = .47.Figure 3 displays forecast distributions; the authors do not conclude that GPT-4 forecasts differ in variance around the midpoint.Source: Figure 3 · p. 6, §3 · p. 6
  • ANCOVA found a method effect, F(1, 42) = 9.38, p = .004, but neither duration, F(1, 42) = 0.017, p = .898, nor its interaction with method, F(1, 42) = 0.75, p = .392, was significant.Figure 4 plots forecasting accuracy against question duration by condition.Source: Figure 4 · p. 7, §3 · p. 7
  • BrierBMA = .13 (SD = .09) significantly beat the random baseline, t(22) = −5.91, p < .001, though current GPT-4 may not make worthwhile additions to aggregation algorithms.The reported exploratory average weights were 0.467 for GPT-4 and 0.533 for the human crowd.Source: §3 · p. 7

4 Discussion

GPT-4 underperformed human-crowd forecasts in a real-world tournament and did not outperform a no-information baseline. The discussion considers knowledge-cutoff constraints, implications for forecasting-dependent applications, and directions for combining or improving human and LLM forecasts.

4 Discussion

  • GPT-4 was heavily outperformed by human forecasts and did not outperform a 50% no-information baseline; the results support the continuing value of human crowds and tournaments as probes of prediction.The questions spanned Big Tech, U.S. politics, viral outbreaks, and the Ukraine conflict.Source: §4 · pp. 7–9
  • GPT-4’s knowledge cutoff may impair forecasting of dynamic events; supplying Metaculus background information mitigated, but did not eliminate, the information gap with human forecasters.The authors suggest further study of ways to mitigate cutoff-related forecasting limits.Source: §4 · p. 8
  • The authors conclude that current state-of-the-art AI systems are not yet poised to replace human expertise in economically important forecasting fields such as business, policy, and law.They say accurate forecasting remains important for autonomous systems.Source: §4 · p. 8
  • Because long-term planning requires accurate forecasts, GPT-4’s poor forecasting supports the authors’ view that risks from AI planning against human interests remain presently low, not negligible.They call for continued monitoring as AI capabilities advance.Source: §4 · p. 8
  • The authors propose testing larger, more diverse LLM forecaster ensembles and internet-enabled systems that can update forecasts, while noting design choices may introduce bias.Ensembles could vary inputs, training data, prompts, and temperature values.Source: §4 · p. 8
  • Future work could combine human and machine forecasts through Bayesian model averaging or other aggregation methods, but such approaches may become worthwhile only after LLM foresight improves.The authors also suggest testing hybrid forecasters that use LLM outputs to augment human predictions.Source: §4 · pp. 8–9

Appendix A Forecasting questions

Table 2 is identified as the full list of forecasting questions used in the study.

  • Table 2 is identified as the full list of forecasting questions used in the study.Source: Table 2 · p. 12

Question

The question asks whether a vote on a Republican-introduced resolution to vacate the House Speaker would occur before October 1, 2023.

  • The question asks whether a vote on a Republican-introduced resolution to vacate the Speaker would be held before October 1, 2023.Source: Question · p. 12

Resolution Criteria

The question resolves Yes only if a Republican introduces a resolution to remove Kevin McCarthy and a vote occurs before October 1, 2023; otherwise it resolves No.

Background Information

The background describes McCarthy’s contentious election and weakened position, the House rules governing privileged resolutions, and a reported draft motion to vacate.

Background Information

  • McCarthy won the speakership on January 7, 2023, after 15 ballots, and concessions to the House Freedom Caucus weakened his position.Those concessions included allowing any House member to call for a vote to oust the Speaker by simple majority.Source: Background Information · p. 13
  • The 118th Congress adopted the 117th Congress’s House rules with amendments, including removal of subparagraph (3) of Rule IX, clause 2(a).Source: Background Information · p. 13
  • The quoted subparagraph (3) says a resolution causing a Speaker vacancy is not privileged unless offered at a party caucus’s or conference’s direction.Source: Background Information · p. 13
  • On September 19, 2023, journalist Matt Laslo reported finding a draft motion to vacate in a Capitol bathroom, naming Matt Gaetz as the member to submit it.Source: Background Information · p. 13
Loading 2310.13014v1…