AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval

1 INTRODUCTION

• Machine forecasting has moved beyond time-series toward news articles, while existing methods still lag behind human forecasters.The paper describes news as a critical modality for interpreting future events and identifies a gap between machine methods and human proficiency.Source: §1 · pp. 1–2
• The paper frames three open questions: how to select relevant articles, extract information effectively, and refine learning to use that information.These questions organize the proposed improvements to retrieval, article processing, and model learning.Source: §1 · p. 2
• The proposed components are task-aligned retrieval, a summarization-enhanced neural article reader, and a human-aligned loss function.The loss incorporates human preferences and judgments; the reader uses unsupervised distillation techniques, including summarization.Source: §1 · p. 2
• 48% for MCQ, 8% for TF, and 19% for numerical predictions are the reported improvements over baseline models.The introduction attributes these gains to the proposed components and notes that ablation studies support their contribution.Source: §1 · p. 2

Event Forecasting

• ForecastQA and the later dataset introduced by Zou et al. (2022) frame event forecasting; the latter is large-scale, balanced, and the study's primary evaluation setting.The passage notes shortcomings in ForecastQA, including nonsensical questions and potential leakage in retrospective human labels.Source: §2 · p. 2

Information Retrieval

• Information retrieval has progressed from bag-of-words BM-25r to language-model applications for retrieval and question answering.The passage cites both direct LLM-based retrieval and retrieval used to improve LLM question answering.Source: §2 · p. 2

LLMs for Text Summarization

• Large language models, including GPT and T5, have been used for text summarization alongside approaches dedicated to summarization.The passage describes LLMs as capable of condensing textual content.Source: §2 · pp. 2–3

Open-domain Question Answering

• Open-domain question answering typically pairs a retriever that selects documents with a reader that generates an answer, and UPR exemplifies unsupervised passage re-ranking.OpenQA generates factual answers from large document collections without explicit evidence pointers.Source: §2 · p. 3

3.1 PRELIMINARY

• The retriever selects news for each question, and a T5-based Fusion-in-Decoder reader generates a discrete forecasting outcome from the question and retrieved articles.The reader maximizes the likelihood of the actual outcome.Source: §3.1 · pp. 3–4

3.2 TASK-ALIGNED RETRIEVAL MODULE

• A zero-shot pretrained LLM scores question–article relevance, and the retriever re-ranks the initial top-K articles using those scores.Relevance is estimated through discrete bins rather than directly as a continuous score.Source: §3.2 · pp. 4–5
• Human-forecaster accuracy changes inform a recency score, which the method uses to prioritize more recent news alongside content relevance.The temporal pattern is estimated across questions and news articles, using normalized time for queries spanning multiple years.Source: §3.2 · p. 5

3.3 NEURAL READER WITH UNSUPERVISED DISTILLATION

• The reader replaces raw retrieved articles with concise abstractive summaries generated by a pretrained LLM without supervised signals.Question-news pairs are encoded separately, and the decoder attends across their combined representations to generate the answer.Source: §3.3 · pp. 5–6

3.4 TRAINING OBJECTIVES WITH HUMAN ALIGNMENT LOSS

• Numerical outcomes are discretized into categories for training, and inference maps the predicted category back to a value using its median.This retains the textual-token objective without adding a regression layer.Source: §3.4 · p. 6
• A causal self-attention layer estimates confidence over chronologically ordered news representations and regularizes them toward human forecaster accuracy at each time.The overall objective combines this alignment with decoder loss, using cross-entropy for both terms.Source: §3.4 · p. 6

4.1 DATASET AND METRICS

• Autocast covers TF, MCQ, and numerical questions across economics, politics, and technology; evaluation uses accuracy for TF and MCQ and absolute error for numerical prediction.The test questions span mid-2021 to mid-2022, with a mid-2021 dataset cut-off.Source: §4.1 · pp. 6–7

4.2 EXPERIMENTAL SETUP

• The setup uses GPT-3 for relevance evaluation and summarization, retrieves 50 BM25 articles before selecting 10, and uses LoRA on T5 for most optimization experiments.The study excludes newer models to avoid potential contamination from training data after the mid-2021 cut-off.Source: §4.2 · p. 7

4.3 MAIN RESULTS

• The 0.2B and 0.8B variants surpass FiD Static's 2.8B baseline, while FiD Temporal's 4.3B model is computationally intensive and performs less well on TF and MCQ.FiD Temporal processes news articles for each day in a question period, averaging 130 days in Autocast.Source: §4.3 · p. 7
• On 0.2B models, Autocast++ beats FiD Static by 48% on MCQ (43.8 versus 29.6), 8% on TF (66.7 versus 62.0), and 19% on numerical prediction (19.8 versus 24.5).Table 2 reports results across three model sizes and compares against baselines from Zou et al. (2022).Source: §4.3 · p. 7, Table 2 · p. 8

4.4 ABLATION STUDIES

• Zero-shot relevance re-ranking and summarization improve FiD Static and FiD Static∗ across question types, bringing MCQ performance close to the full model.These LLM-enhanced components are identified as the most effective proposed methods in the ablations.Source: §4.4 · p. 9
• Numerical-question binning converts regression into classification with cross-entropy, generally improving numerical performance without harming other question types.Its overall-metric impact is less significant, but it complements LLM enhancement.Source: §4.4 · p. 9
• Human-aligned loss helps the basic FiD Static baseline more than the full model, where relevance re-ranking and summarization are already used.The loss uses crowd forecasts to regulate text-encoder representations and simulate information progression over time.Source: §4.4 · p. 9

4.5 QUALITATIVE EXAMPLES

• Figure 3 shows summarized context from the third-highest-relevance article, with highlighted information supporting answer C about at least two Category 4 hurricanes.The figure is an expanded qualitative example of the retriever comparison in Figure 2.Source: Figure 3 · p. 9

5 CONCLUSION

• The conclusion identifies task-aligned retrieval, an enhanced neural article reader, and human-aligned loss as its three key components.Source: §5 · p. 9
• It reports improvements of 48% for multiple-choice questions, 8% for true/false questions, and 19% for numerical predictions.Source: §5 · p. 9
• The authors characterize the results as progress toward bridging the gap between theory and practical AI-driven applications in event forecasting.Source: §5 · p. 9

• FiD concatenates the question with its top-N retrieved articles, using [CLS] as the beginning token and [SEP] as the separator.Source: A IMPLEMENTATION DETAILS · p. 12
• The question input includes the question, its starting and ending dates, and its allowable choices.Source: A IMPLEMENTATION DETAILS · p. 12
• Each article input is prefixed with its title and date, and the article notation refers to the summarized article.Source: A IMPLEMENTATION DETAILS · p. 12

• LoRA reproduces the original paper’s outcomes at a virtually indistinguishable performance level.Source: §B.1 · p. 12
• LoRA is used to explore model design choices and search hyperparameters to save time.Source: §B.1 · p. 12
• The final experiments are fine-tuned without LoRA; Table 4 reports LoRA’s effects on FiD Static performance.Source: §B.1 · p. 12, Table 4 · p. 12

• The filtering threshold uses a retrieval score combining relevance and recency reranking.Source: §B.2 · p. 12
• The comparison uses vanilla FiD Static configurations without changing the retriever or reader, and noisy articles may confuse predictions.Source: §B.2 · p. 12
• Filtering less-relevant articles by retrieval score improves performance despite reducing the number of training instances.Source: Table 5 · p. 12, §B.2 · p. 12

B.3 SPLITTING NUMERICAL AND NON-NUMERICAL QUESTIONS

• Separating continuous numerical questions from discrete T/F and MCQ questions improves T/F and MCQ performance.The split ablation uses relevance-reranked, summarized news with the vanilla FiD Static retriever and reader.Source: §B.3 · p. 13
• Binning numerical questions consistently leads to better performance metrics than the results in Table 3.Source: §B.3 · p. 13

B.4 IMPACT OF CONTEXT SIZE

• Context size is the number of news articles input to the model, trading potential information gain against computation and noisier content.Source: §B.4 · p. 13
• Larger context windows do not necessarily improve performance because they may include less pertinent news and confuse model responses.The context-size ablation uses relevance-reranked, summarized news with the vanilla FiD Static retriever and reader.Source: Table 7 · p. 13, §B.4 · p. 13

C ADDITIONAL QUALITATIVE RESULTS

• GPT-3 (gpt-3.5-turbo API) is used for both relevance assessment and text summarization.The section presents qualitative results for these uses.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 14

C.1 QUESTION-NEWS RELEVANCE ASSESSMENT

• The relevance assessment selects semantically relevant news because such articles are described as contributing more to future event predictions.Figures 4 and 5 provide two examples of question-news relevance assessment.Source: §C.1 · p. 14, Figure 4 · p. 14, Figure 5 · p. 14

C.2 TEXT SUMMARIZATION

• Figures 6, 7, and 8 present three examples of text summarization.Source: Figure 6 · p. 15, Figure 7 · p. 16, Figure 8 · p. 17