PaperlayerView exact revision on arXiv

Source-linked AI summary

Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts

Cheng Peng, Ruixi Luo, Zhi Chen, Wei Tang

arXiv:2609.33257v1cs.LG

TL;DR

Access to the strongest LLM forecaster may be costly or constrained, motivating the question of whether weaker forecasters can be combined to match or beat it. The paper evaluates learned and fixed aggregation of weaker forecasters on ForecastBench and finds that learned linear pooling often matches or outperforms the strongest individual.

  • Problem

    The paper asks whether individually weaker LLM forecasters can be aggregated to match or outperform a stronger forecaster, especially when access to that strongest model is constrained.

    Source: §1 · p. 2

  • Method

    Using ForecastBench, the study evaluates aggregates composed only of weaker forecasters, comparing learned and fixed rules against the strongest individual in each group.

    Source: §1 · p. 3, §3.2 · p. 10

  • Results

    Learned linear pooling finds a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups.

    Source: §4.1 · p. 11, §5 · p. 14

  • Takeaways & Limitations

    The gains do not rely on a near-best constituent, are generally accompanied by good calibration, and adding more forecasters does not consistently improve performance.

    Source: §1 · p. 3, §5 · p. 14

    Benchmark and eligible-forecaster selection use test Brier score, making the comparison retrospective on the evaluation sample.

    Source: §3.2 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used to forecast real-world events, but access to the strongest individual forecaster may be costly or otherwise constrained. We study weak-to-strong forecast aggregation: can individually weaker LLM forecasters be aggregated to outperform a stronger forecaster? Using ForecastBench (Karger et al., 2025), we evaluate 70 LLM forecasters across 16 comparison groups, each with more than 1,000 shared subquestions, yielding 1,121 weaker-model pairs. Within each group, we identify the strongest individual by test Brier score and evaluate aggregates composed exclusively of weaker forecasters, with aggregation weights learned on separate training data. We find substantial evidence of weak-to-strong improvement. Learned linear pooling identifies a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups. We also find that these improvements do not rely on having a near-best constituent and are generally accompanied by good calibration. Additional analyses show that adding more models does not consistently improve performance, and competitive weaker-model aggregates also remain available under practical constraints.

1 Introduction

The paper asks whether weaker LLM forecasters can be aggregated to match or beat a stronger forecaster, a question relevant when access to the strongest model is constrained. It situates this question within event forecasting and prior work on forecast aggregation.

1 Introduction

  • Unlike classical forecasting from historical data, real-world event forecasts may require evidence distributed across heterogeneous textual, numerical, and contextual sources.The paper focuses on probabilistic judgmental forecasts of events rather than statistical time-series forecasting.Source: §1 · p. 2, §1.1 · p. 4
  • The central question is whether two or more individually weaker LLM forecasters can be aggregated to match or outperform a stronger forecaster, without using its forecast.This matters when the strongest model may be costly, available only through an external API, or incompatible with open-weight or local deployment.Source: §1 · pp. 2–3
  • Learned linear pooling finds a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16 groups.The study also reports that gains do not rely on a near-best constituent and are generally accompanied by good calibration.Source: §1 · p. 3

1.1 Further Related Work

  • LLM forecasting systems have also incorporated retrieval, reasoning, and multiple forecasts, while statistical time-series applications form a distinct research area.The paper’s focus is real-world event forecasts informed by heterogeneous sources, not forecasts from a well-defined historical time series.Source: §1.1 · pp. 3–4
  • ForecastBench and other evolving benchmarks support live evaluation of LLM predictions; this paper shifts attention from individual forecasters to what aggregation can gain.The related-work discussion distinguishes these benchmarks from earlier event datasets and describes extensions to agentic and modularized evaluation.Source: §1.1 · p. 4
  • Prior work on LLM forecasting aggregation examines crowd performance, model diversity, and learned pooling; this paper poses the more stringent comparison against a stronger individual.Aitchison et al. (2026) discuss complementary information from weaker, more diverse models, while Douven (2026) studies learned aggregation.Source: §1 · p. 2, §1.1 · pp. 4–5
  • The paper compares data-driven rules with robust aggregation approaches that address limited knowledge of information dependence and overlap.The broader aggregation literature also motivates comparisons between linear and log-odds pooling.Source: §1.1 · p. 5

2 Preliminaries

The preliminaries define the event-forecasting inputs, accuracy and calibration measures, and forecast-pooling rules. The main data-driven rules learn pool-specific weights, alongside fixed classical and robust alternatives.

2 Preliminaries

  • The paper studies whether aggregation of individually weaker forecasters can outperform a stronger forecaster, using a shared evaluation pool and defined aggregation rules.It introduces the task, performance metric, and rules considered in the study.Source: §2 · p. 5

2.1 Forecasting Questions and Forecasts

  • Each ForecastBench question contains binary Yes/No subquestions, and each forecaster assigns a probability to each subquestion’s outcome.A standard question has one subquestion; combination questions may have several.Source: §2.1 · pp. 5–6

2.2 Evaluation Metrics

  • Accuracy is measured by Brier score and calibration by expected calibration error (ECE); lower Brier score means better accuracy, and the two metrics are not equivalent.The overall Brier score weights questions equally, while ECE pools predictions into probability bins and weights evaluated subquestions equally.Source: §2.2 · pp. 6–7

2.3 Aggregation Rules

  • An aggregation rule maps a pool’s probability forecasts for the same subquestion to one forecast; the main analysis emphasizes learned linear and log-odds pooling.The learned rules adapt each forecaster’s contribution to the candidate pool, alongside parameter-free classical and robust rules.Source: §2.3 · p. 7

2.3.1 Learned Aggregation Rules

  • Learned linear and log-odds pooling assign nonnegative, pool-specific coefficients fitted by minimizing training Brier score, then evaluate frozen coefficients on test Brier score and ECE.Linear pooling combines probabilities directly, whereas log-odds pooling combines forecasts in log-odds space and converts back to probability.Source: §2.3.1 · p. 7

2.3.2 Classical Aggregation Rules

  • Fixed alternatives include arithmetic, harmonic, geometric-odds, and median pooling, plus four robust rules for pairwise aggregation.These classical rules do not fit parameters to historical data.Source: §2.3.2 · pp. 7–8

3 Experimental Setups and Dataset Statistics

The experiments select representative forecasters, form groups with adequate shared forecast coverage, and compare individual and aggregated forecasts on common subquestions. Training and test data are split within groups, with benchmark selection and aggregation rules specified for evaluation.

3 Experimental Setups and Dataset Statistics

  • The experimental setup selects representative forecasters, forms groups with sufficient shared coverage, and compares individual and aggregated forecasts on the same subquestions.The section then describes training procedures and evaluation criteria.Source: §3 · p. 8

3.1 Data and Comparison Groups

  • ForecastBench supplies the probability forecasts and resolved outcomes; within each model family, the representative is the configuration covering the most distinct subquestions.The selected representatives are used to construct comparison groups.Source: §3.1 · p. 8
  • Comparison groups require at least three forecasters and at least N shared subquestions; maximal eligible groups are retained and evaluated separately.Shared coverage consists of subquestions forecast by every group member, and N is a minimum rather than a fixed sample size.Source: §3.1 · p. 9
  • At N = 1,000, maximal-group retention reduces the candidate groups from 567,316 to 16; increasing the threshold to 4,000 reduces evaluated forecasters from 70 to 22.Table 1 summarizes forecaster counts, retained groups, group-size ranges, and shared-subquestion ranges by threshold.Source: §3.1 · p. 9, Table 1 · p. 9

3.2 Experimental Protocol and Evaluation

  • Groups are split approximately equally into training and test sets; learned rules fit pool-specific coefficients on training Brier score, while fixed rules require no training.Test comparisons use common group data and report benchmark matching or improvement, within-5% performance, and improvement over pool members.Source: §3.2 · pp. 9–10
  • Within each group, the lowest-test-Brier-score forecaster is the benchmark and is excluded from candidate pools, which contain only forecasters with higher test Brier scores.The authors note that using test Brier score to select both benchmark and eligible forecasters makes this a retrospective comparison.Source: §3.2 · p. 10

4 Main Experiment Results

Across 16 comparison groups, learned linear pooling can match or outperform the group benchmark using only two weaker forecasters, and increasing pool size does not consistently improve results. Aggregation gains vary by question category, with learned linear pooling particularly effective in Finance.

4 Main Experiment Results

  • 11 of 16 groups have a learned-linear pair matching or improving on the benchmark, and all 16 have one within 5%.The best learned-linear pair is within 3% in all 16 groups; Figure 1 compares each group benchmark with its best learned pair under each rule.Source: §4.1 · pp. 11–12, Figure 1 · p. 12

4.1 Pairwise Aggregation Results

  • Learned linear pooling improves on the benchmark for 28.10% of eligible pairs, versus 6.24% for the simple mean.Learned linear and log-odds pooling improve on both constituents for 95.72% and 91.17% of pairs, versus 65.66% and 61.46% for their equal-weight counterparts.Source: §4.1 · p. 12
  • 15 of 16 groups have a no-near pair within 5% of the benchmark, compared with 9 groups containing a near pair.A near-best forecaster has individual test BS at most 5% above the group benchmark; a pair is near if it contains at least one such forecaster.Source: §4.1 · pp. 12–13
  • Learned linear pooling finds a pair matching or improving on benchmark ECE in all 16 groups; 68.87% of eligible pairs meet that criterion.The BS-best benchmark, BS-defined candidate pairs, and BS-trained coefficients are held fixed for this ECE evaluation.Source: §4.1 · p. 13

4.2 Multi-Model Aggregation

  • For learned linear pooling, groups with a successful pool rise from 68.75% with two forecasters to 75.00% with three, then return to 68.75% with four or five.In G13, the best learned-linear pool reaches 0.848 with two members and remains approximately 0.848 for pool sizes three through five.Source: §4.2 · p. 13

4.3 Question-Category-Based Analysis

  • Aggregation gains differ substantially by category, and learned linear pooling is particularly effective in the Finance comparisons.Finance benchmark BS ranges from 0.210 to 0.247, versus 0.075 to 0.111 in Politics; the paper suggests this may leave more room for aggregation in Finance.Source: §4.3 · pp. 13–14

5 Conclusions

The paper studies whether weaker LLM forecasters can be aggregated to match or outperform a stronger forecaster, and finds substantial evidence that they can.

5 Conclusions

  • Learned linear pooling finds a weaker pair that matches or outperforms the strongest individual in 11 of 16 groups and comes within 5% of its Brier score in all 16.The gains do not rely on a near-best constituent and are generally accompanied by good calibration.Source: §5 · p. 14
  • Adding more forecasters does not consistently improve performance, while competitive weaker-model aggregates remain available under practical cost and access constraints.The weak-to-strong pattern also persists at a higher shared-coverage threshold.Source: §5 · p. 14

AI Disclosure

The paper discloses limited generative-AI use and responsibility for its content, and describes additional fixed aggregation rules drawn from robust forecast aggregation literature.

AI Disclosure

  • Generative AI was used for proofreading and readability, not for research methodology, experiments, implementation, translation, dataset processing, or analysis.The authors take responsibility for the final content, including AI-assisted text, claims, or artifacts.Source: AI Disclosure · p. 14

A Additional Aggregation Rules

  • The paper adds four fixed robust-aggregation baselines: two prior plug-in rules, a precision rule, and a prior-agnostic robust rule.The literature addresses combining forecasts when the forecasters’ underlying information structures are unknown to the aggregator.Source: A Additional Aggregation Rules · p. 23
  • The average-prior and heuristic-prior rules plug their respective estimated priors into g, while the precision aggregator gives more weight to forecasts closer to 0 or 1.Source: A Additional Aggregation Rules · p. 23

B Additional Experimental Details and Group Statistics

The appendix reports the composition of the overall comparison groups and the evaluated forecasters, then introduces robustness checks and controls.

B Additional Experimental Details and Group Statistics

C Robustness Checks and Controls

  • The robustness appendix tests whether results persist with stricter shared coverage and whether individual recalibration can explain the aggregation improvement.It identifies these as the aims of Sections C.1 and C.2, respectively.Source: C Robustness Checks and Controls · p. 26

C.1 Higher Shared-Coverage Threshold N = 4000

At N = 4,000, weaker-pair aggregation remains competitive with the group benchmark; a separate control finds individual recalibration less often matches the best aggregation pair.

C.1 Higher Shared-Coverage Threshold N = 4000

  • Learned linear pooling finds a pair matching or improving on the benchmark in 8 of 9 groups (88.89%) and within 5% in all 9; log-odds pooling does so in 5 (55.56%) and 6 (66.67%) groups, respectively.Table 5 reports group-level and pair-level rates and the share of pairs improving on both members; Figure 2 visualizes group benchmarks against the best pair for each learned rule.Source: §C.1 · p. 26, Table 5 · p. 26, Figure 2 · p. 27
  • At N = 4,000, Table 6 reports each group benchmark and the best weak-pair aggregate under each learned rule, selected by test Brier score.Its Brier-score intervals are symmetric 95% bootstrap CI half-widths conditional on the test-selected pairs.Source: §C.1 · p. 27, Table 6 · p. 27

C.2 Individual Recalibration as a Control

  • The best calibrated individual matches or improves on the benchmark in 7 of 16 groups (43.75%) and on the best aggregation pair in 2 (12.50%).The control fits separate two-parameter Platt scaling on training data, then evaluates calibrated forecasts on the common test set; recalibration therefore does not reproduce the best pair’s test BS in most groups.Source: §C.2 · p. 28, Table 7 · p. 28

D Additional Pairwise Aggregation Results

Across comparison groups, learned linear pooling more often yields a successful weak-model pair than learned log-odds pooling. Learning also lowers Brier score for most pairs relative to equal-weight pooling, while the appendix examines near-best constituents and calibration.

Best pairs across comparison groups

  • Learned linear pooling finds a successful pair in 11 groups, versus 8 for learned log-odds pooling; which rule succeeds varies across groups.Linear pooling succeeds where log-odds does not in G9, G12, G15, and G16, while the reverse occurs in G11.Source: D Additional Pairwise Aggregation… · pp. 28–29

Learned versus equal-weight pooling

  • 84.66% of pairs have lower Brier score under learned linear pooling than under the equal-weight mean, compared with 76.81% under learned log-odds pooling.Learning improves most pairs, though some perform worse than their equal-weight counterparts.Source: D Additional Pairwise Aggregation… · p. 30, Figure 3 · p. 30

Near-best and no-near pairs

Calibration results

E Case Study: When Learned Linear Pooling Succeeds

In G13, learned linear pooling succeeds across nearly all eligible pairs, including pairs without top-ranked constituents. Holding the pair fixed, linear pooling also outperforms log-odds pooling in every comparison.

G13 pairwise results

  • 96.43% of G13 pairs match or improve on the benchmark under learned linear pooling, versus 7.14% under learned log-odds pooling.Linear-pooling gains span almost the entire pair-rank matrix, including lower-ranked forecasters; only one pair has a negative gain.Source: E Case Study: When Learned Linear… · p. 33, Figure 5 · p. 33

Pooling-rule comparison

Calibration

F Additional Multi-Model Aggregation Results

Across exact pool sizes from two to five, learned linear pooling remains among the strongest rules, while larger pools do not consistently improve results. The best pool is selected separately at each size, rather than tracked as models are added.

Across pool sizes

  • Across pool sizes k ∈{2, 3, 4, 5}, learned linear pooling remains among the strongest rules, while classical-rule success rates do not generally rise as pools grow.The best-performing pool is selected separately for each rule and exact size; the comparisons do not track one pool as forecasters are added.Source: F Additional Multi-Model Aggregation… · pp. 34–35, Figure 8 · p. 35

G13 results

G Practical Constraints

The practical-constraint analyses test whether weak-model aggregates remain competitive when pairs must be cheaper than the benchmark or use open-weight models. Such alternatives occur in some groups, with price–accuracy gains especially visible in G13.

Group-level results

  • Under lower input-price constraints, learned linear pooling finds a pair matching or improving on the benchmark in 2 of 16 groups and within 5% in 5; under open-weight constraints, it does so in 3 of 10 and 4 groups, respectively.For log-odds pooling, the corresponding counts are 1 and 3 groups under lower input price, and 1 and 3 under open weights.Source: G Practical Constraints · pp. 35–36, Table 12 · p. 36

Price-accuracy trade-offs

  • G6 has one lower-cost pair that beats its benchmark, while G13 has several alternatives across different price levels.Pair cost is the sum of the members’ published input prices; the analysis uses 1,000-coverage thresholds in both groups.Source: G Practical Constraints · p. 36, Figure 9 · p. 37
  • In G13, Claude-3-5-Sonnet with Mistral-Large costs $5 per million tokens versus $15 for the Claude-3-Opus benchmark and reduces Brier score by 15.23% under learned linear pooling.Under learned log-odds pooling, the same pair reduces Brier score by 3.58%.Source: G Practical Constraints · p. 36, Figure 9 · p. 37

Best-performing constrained pairs

  • The best open-weight learned-linear pair, Mistral-Large with Qwen3-235B-A22B, reduces Brier score by 9.72% relative to the Claude-3-Opus benchmark.The best open-weight learned log-odds example, Llama-3.3-70B with Llama-3.1-405B, reduces Brier score by 4.87% relative to O4-Mini.Source: G Practical Constraints · p. 36, Table 13 · p. 37

H Additional Category-Specific Results

The category-specific analysis finds more consistent learned-linear-pooling performance in Finance than Politics, with calibration patterns broadly similar to Brier-score comparisons. The category groups differ in both questions and forecasters, so these contrasts are descriptive rather than causal.

Category-specific data and comparison groups

  • At the shared-coverage threshold N = 1,000, Politics and Finance have 12 and 9 eligible maximal groups, while Climate has two and the other three categories have none.The six categories do not cover all subquestions because some cannot be categorized properly.Source: H Additional Category-Specific Results · p. 38, Table 14 · p. 38

Group composition

  • Because Politics and Finance groups do not hold the forecaster set fixed, their differences should be interpreted descriptively rather than as a causal effect of question category.Groups are constructed separately by category.Source: H Additional Category-Specific Results · pp. 38–39

Best weak-pair aggregates by group

Aggregate performance within categories

  • Learned linear pooling performs close to or better than the benchmark across Finance groups, while Politics results are more heterogeneous; log-odds performance varies more in both categories.Figure 10 compares test Brier scores for the group benchmark and the lowest-test-BS pair under each learned rule.Source: H Additional Category-Specific Results · p. 40, Figure 10 · p. 41

Calibration of the BS-selected aggregates

  • Learned linear pooling’s pair-level ECE success rate is substantially higher in Finance than Politics, broadly mirroring the Brier-score pattern.The same BS-selected benchmarks, pairs, and fitted coefficients are evaluated without reselection or refitting.Source: H Additional Category-Specific Results · p. 40, Figure 11 · p. 41

Related papers

Loading 2609.33257v1…