PaperlayerView exact revision on arXiv

Source-linked AI summary

Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction

Nirupam Chetlapalli, Yiming Liao, Min-Chun Chen, Keke Chen

arXiv:2608.24001v1cs.AI

TL;DR

LLM future prediction raises the question of how to build a diverse crowd without letting redundant model behaviors inflate crowd size and cost. The paper characterizes models from reasoning traces on independent tasks, clusters behavioral similarity, and selects representatives. A three-model K-means++ medoid crowd outperforms 25-model voting on both prediction benchmarks while using 88% fewer model calls and approximately 80% lower inference cost.

  • Problem

    Different LLMs can produce redundant judgments, leaving open which models should constitute an effective and efficient prediction crowd.

  • Method

    The framework embeds reasoning traces from independent development tasks, forms model-level behavioral signatures, clusters similar models, and constructs representative crowds.

  • Results

    A three-model K-means++ medoid crowd outperforms conventional voting over all 25 models on both benchmarks, reducing model calls by 88% and inference cost by approximately 80%.

  • Takeaways & Limitations

    Representative behavioral diversity, rather than simply maximizing dissimilarity, is important for constructing effective LLM crowds.

  • Takeaways & Limitations

    The evaluation is limited to 25 LLMs and two future-prediction benchmarks with 100 sampled questions each.

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for future prediction, motivating the use of multiple models as a wisdom-of-the-crowd mechanism. However, simply increasing crowd size does not guarantee effective diversity, as different LLMs may exhibit redundant behaviors. We propose a behavior-aware framework for constructing diverse LLM crowds. The framework characterizes models using their reasoning traces on independent development tasks, clusters models by behavioral similarity, and selects representatives for collective prediction. We evaluate 25 LLMs using seven development benchmarks for behavioral diversity modeling and two future-prediction benchmarks for evaluating diverse crowds' performance. Our results show that crowd composition can matter more than crowd size: a three-model medoid crowd based on K-means++ behavioral clustering outperforms conventional voting over all 25 models on both prediction benchmarks, while reducing model calls by 88% and inference cost by approximately 80%. The results further suggest that representative behavioral diversity, rather than simply maximizing diversity, is important for constructing effective LLM crowds

I. INTRODUCTION

The paper frames LLM forecasting as a crowd-construction problem: model redundancy can make simply enlarging a crowd inefficient. It proposes selecting representatives by behavioral diversity and finds that a small behavior-aware crowd can outperform all-model voting on two benchmarks.

  • Behaviorally similar models may contribute redundant judgments, so crowd composition matters alongside crowd size.
  • The framework characterizes models from reasoning traces on independent development tasks, clusters behavioral signatures, and selects representative models for prediction.The behavioral characterization uses tasks separate from the future-prediction benchmarks.
  • A three-model K-means++ medoid crowd outperforms conventional voting over all 25 models on both future-prediction benchmarks.The comparison uses FutureX-Past and Bench to the Future.
  • 88% fewer model calls and approximately 80% lower inference cost accompany the stronger three-model crowd result.
  • The work extends behavior-based crowd construction to LLM reasoning traces while complementing approaches that improve aggregation or model interaction.

III. APPROACH

The approach infers LLM diversity from observable reasoning behavior rather than model metadata or final predictions. It represents, clusters, and uses these behavioral signatures to construct crowds without using future-prediction labels.

  • The framework has three stages: behavioral representation, behavioral clustering and crowd construction, and prediction aggregation.Development questions are separate from future-prediction questions.
  • Each model’s behavioral signature is the normalized average of embeddings of its reasoning traces across common development questions.Normalization facilitates cosine similarity and reduces the influence of embedding magnitude.
  • Averaging heterogeneous tasks emphasizes persistent observable behavior, but does not faithfully describe a model’s internal reasoning process.
  • The method changes crowd composition using behavior observed independently of prediction tasks while retaining simple aggregation rules.
  • Behavioral clustering approximates behavioral modes so similar models do not receive proportionally greater representation in the crowd.The clusters are not interpreted as intrinsic or universal LLM types.

C. Behavior-Aware Crowd Construction

After clustering models by behavioral similarity, the paper tests three crowd-construction strategies: representative medoids, maximally dissimilar representatives, and cluster subcommittees. These strategies distinguish representative coverage from maximal diversity and vote balancing.

  • Medoid Representatives: Medoid selection chooses one actual LLM per behavioral cluster whose behavior is most representative of that cluster.
  • Medoid Representatives: One medoid per cluster preserves coverage while reducing the crowd from N models to K and limiting redundant models’ influence.
  • Diverse Representatives: The medoid strategy emphasizes representative coverage, whereas the k-dissimilar strategy maximizes pairwise behavioral differences.
  • Cluster Subcommittees: Cluster subcommittees retain all models but aggregate first within clusters and then across cluster decisions.
  • Cluster Subcommittees: Subcommittee voting tests balanced influence across behavioral groups without reducing model calls.A cluster with many similar models does not automatically receive more influence than a smaller cluster.

D. Prediction Aggregation

The prediction stage uses simple, answer-type-aware aggregation rules so that comparisons primarily reflect crowd construction. The evaluation separates behavioral characterization from prediction and examines effectiveness, diversity, and efficiency.

  • Selected medoid and diverse-representative models vote over the prediction answers, with R denoting the selected representative set.
  • Aggregation is type-aware: categorical answers use plurality voting, numeric answers use the median, and structured outputs use answer-specific similarity.
  • Cluster subcommittees vote hierarchically, first within each cluster and then over the K cluster decisions.
  • Simple aggregation rules isolate crowd-construction effects and enable direct comparison with standard all-model voting.
  • The evaluation asks whether behavioral diversity exists, improves prediction, and preserves quality with fewer model calls.

A. Experimental Setup

The study uses a heterogeneous crowd of 25 LLMs, selected to avoid training-data leakage, and characterizes their behavior from independent development tasks. Seven heterogeneous benchmarks provide 350 shared questions whose reasoning traces are embedded into model-level behavioral signatures.

  • Model population: 25 LLMs from different model families and providers form the heterogeneous experimental population.Models are included only when their training data ends before the questions they will predict.
  • Model population: 16 February 2026 is the latest training cutoff or release date in the crowd, and every evaluation question resolves afterward.This temporal separation prevents models from having seen the outcomes they are asked to predict.
  • Development benchmarks: 350 development questions are formed by randomly sampling 50 questions from each of seven heterogeneous benchmarks.Equal sampling prevents any single benchmark from dominating the behavioral representation by size.
  • Development benchmarks: The development benchmarks probe reasoning, mathematics, instruction following, data analysis, scientific reasoning, planning, and program execution.They function as behavioral probes rather than capability evaluations of the benchmarks themselves.
  • Behavioral characterization: All 25 models receive the same development questions, and their reasoning traces are embedded and aggregated into model-level behavioral signatures.The resulting representations are used to characterize systematic behavioral differences among models.

3) Future-Prediction Benchmarks:

The evaluation uses two future-prediction benchmarks with distinct answer formats and scoring rules, while enforcing temporal separation from model training data. It samples 100 questions per benchmark and treats method-generated abstentions as misses.

  • Temporal separation: Every evaluation question resolves after 16 February 2026, the latest training cutoff among the 25 models.FutureX-Past resolves from 7 March to 28 July 2026, while BTF-v3 resolves from 5 May to 1 July 2026.
  • Benchmarks: Two benchmarks, FutureX-Past and Bench to the Future v3, evaluate the constructed crowds on 100 sampled questions each.The future-prediction questions are separate from the development questions used for behavioral signatures.
  • Scoring: BTF-v3 elicits binary-event probabilities and reports accuracy after thresholding p ≥0.5 as “yes” and lower probabilities as “no”.FutureX-Past instead combines four difficulty-level scores using benchmark weights of 10, 20, 30, and 40.
  • Scoring: FutureX-Past uses exact match for level 1, F1 for set-valued level 2 answers, and ordered-list scoring for levels 3 and 4.The study adopts FutureX-Past’s scoring function without modification.
  • Abstention handling: Crowd abstentions remain in the denominator as misses, unlike unusable model-question cells, which are dropped under the FutureX-Past protocol.At levels 2–4, the medoid rule always returns an observed answer, so those crowds do not abstain.

5) Behavioral Representation and Clustering:

Models are represented by normalized embeddings of their reasoning traces on development questions, then clustered by behavioral similarity. Prediction strategies compare all-model voting, individual or random baselines, representative crowds, and hierarchical cluster voting.

  • Behavioral representation: Each model’s 350 reasoning traces are embedded, averaged into a model-level vector, and normalized to form its behavioral signature.Question-level embeddings use BAAI/bge-m3 with a default dimensionality of 1024.
  • Clustering: Candidate cluster counts K ∈{2, . . . , 8} are evaluated using silhouette score to identify the supported clustering structure.Both K-means++ with 50 restarts and agglomerative hierarchical clustering are applied.
  • Baselines: All-model voting has all 25 models participate equally as the conventional wisdom-of-the-crowd baseline.The performance-based expert instead selects one model using selection-fold scores without future-prediction labels.
  • Representative crowds: Random representatives select the same number of models as a behavior-aware method and aggregate their predictions by majority vote.Random selection is repeated 1000 times.
  • Representative crowds: Medoid representatives select one medoid from each behavioral cluster, whereas k-dissimilar representatives maximize within-cluster diversity with fewer than half the population.Cluster subcommittees instead aggregate equal-weighted cluster decisions after within-cluster voting.

7) Evaluation Protocol:

The evaluation compares all methods on identical held-out questions using five-fold cross-validation for each benchmark. Behavioral crowds remain fixed because their construction uses only independent development-task behavior, while the expert baseline uses labels for selection.

  • Cross-validation: Each benchmark’s 100 questions is partitioned into five seeded folds of exactly 20 questions, with the same split used for every method.Four folds form each selection set and one fold is held out, pairing all comparisons on identical questions.
  • Behavior-aware methods: Behavioral clusters and representatives are constructed only from 350 development questions and remain independent of future-prediction labels.These components therefore remain fixed across all five evaluation folds.
  • Scoring aggregation: After five folds, held-out predictions are pooled so every question contributes exactly once to each benchmark-level score.FutureX-Past tier weights are applied once to the pooled 100 predictions rather than separately within folds.
  • Scoring aggregation: The procedure produces 100 held-out predictions per method and benchmark instead of averaging five scores based on 20 questions each.All LLM calls use OpenRouter APIs.

B. RQ1: Behavioral Diversity among LLMs

Reasoning traces from heterogeneous development tasks reveal structure used to compare clustering approaches and construct representative model groups. The selected K-means++ structure supports evaluation of whether behavior-aware diversity transfers to future prediction.

  • Clustering structure: K-means++ clustering sweeps K from 2 to 8 and selects K = 3 among nontrivial solutions.K = 3 achieves silhouette score 0.237 while retaining clustering stability of 0.767.
  • Clustering structure: K = 3 balances cluster separation, reproducibility, and the need for multiple representative perspectives.
  • Clustering robustness: 500 bootstrap replicates resample the 350 development questions to assess robustness of the identified clustering structure.
  • Candidate crowds: K-means++ and hierarchical clustering generate alternative candidate diverse groups from the model representations.The 25 model representations are visualized in a two-dimensional t-SNE projection.
  • Transfer evaluation: The behavioral clusters and representatives are determined without future-prediction questions, testing transfer from general reasoning behavior to a different prediction domain.
  • Future prediction: Medoid voting from K-means++ behavioral clusters achieves 0.302 on FutureX-Past and 0.810 on BTF-v3 while querying K = 3 instead of all 25 models.The three-model crowd outperforms all-model voting, which scores 0.296 and 0.800 respectively.

1) When Does Behavioral Diversity Help?:

Across 498 committees, behavioral disagreement tracked higher FutureX-Past scores after controlling for committee size, but this association did not reliably provide a selection rule. Tie-breaking conventions substantially affected FutureX-Past results without yielding a principled improvement over random resolution.

  • Analysis design: 498 candidate committees spanning sizes 2 to 25 were evaluated using label-free behavioral disagreement and consensus measures.The measures came from behavioral representations alone and were correlated with benchmark scores using a tie-aware Spearman coefficient.
  • Behavioral diversity and performance: +0.211 within-stratum correlation links committee disagreement with FutureX-Past score, while consensus correlates −0.276.The positive disagreement direction held in all seven committee-size strata, whereas consensus was negative in every stratum.
  • Behavioral diversity and performance: +0.024 pooled and −0.010 within-stratum correlations on BTF-v3 show no comparable disagreement signal.BTF-v3 probabilities are aggregated by median, which averages disagreement before it can affect the result; FutureX-Past set answers allow differing views to change the selected candidate.
  • Selection boundary: Behavioral disagreement accompanied stronger committee performance but did not reliably select committees that beat same-size random committees.The authors conclude that medoid representatives were the only tested construction converting representative diversity into a consistent gain.
  • Tie-break evaluation: +0.0192 was the best joint tie-break gain, yet every tested strategy had p ≥0.19 against the arbitrary-rule null.Three strategies exceeded random on all six committees, but accounting for shared questions, overlapping members, and arbitrary-rule sweeps gave p = 0.103 for a clean sweep.
  • Tie-break evaluation: Random resolution is the defensible tie-break default because deterministic strategies did not separate from the arbitrary-rule null.Its expected score is the mean over tied candidates and can be computed in closed form without a seed.

D. RQ3: Prediction Accuracy versus Crowd Cost

The analysis evaluates whether behavior-aware crowd construction improves the accuracy–cost tradeoff, rather than merely reducing crowd size. Medoid voting achieves higher prediction scores than all-model voting on both benchmarks at substantially lower measured cost.

  • Call efficiency: 88% fewer model calls: the representative-crowd comparison measures reductions relative to all-model voting.All-model voting requires 25 model calls per prediction, whereas medoid voting requires approximately K calls.
  • Experimental control: Random subsets of the same size are included to test whether smaller crowds alone, rather than preserved behavioral diversity, explain the observed performance.The comparison targets the accuracy–cost tradeoff of behavior-aware selection against conventional and randomly constructed crowds.
  • Measured cost: 0.21× the cost on BTF-v3 and 0.19× on FutureX-Past: medoid voting scores above all-model voting on both benchmarks while querying three rather than 25 models.The measured costs are $4.51 versus $21.26 on BTF-v3 and $2.10 versus $10.96 on FutureX-Past.
  • Cost drivers: Cost is dominated by model choice rather than crowd size, because per-call prices span roughly two orders of magnitude across the evaluated models.The single most expensive model costs $6.46 per 100 questions on BTF-v3, exceeding the entire three-model medoid committee's cost.
  • Measured cost: About 20% of the outlay: medoid voting achieves a higher future-prediction score than all-model voting on both benchmarks.Costs are the amounts the provider actually charged over the 100 questions of each benchmark.

V. CONCLUSION

The paper constructs prediction crowds from LLM reasoning behavior on independent development tasks and finds that crowd composition can matter more than crowd size. Its results favor representative behavioral diversity, while identifying limits from behavioral averaging, static crowds, and a narrow evaluation scope.

  • Conclusion: A three-model medoid crowd based on K-means++ behavioral clustering achieves the best performance on both future-prediction benchmarks, outperforming voting over all 25 models.The crowd uses only a small fraction of the model calls and inference cost, and size-matched random-crowd comparisons support behavior-aware selection.
  • Limitations: Behavioral signatures average embeddings of observable reasoning traces, which may obscure task-dependent behavioral patterns.The framework also constructs a single static crowd, although model complementarity may vary across prediction questions.
  • Conclusion: Useful crowd diversity is not simply maximized behavioral difference, because maximally dissimilar representatives and cluster-level voting do not consistently improve prediction.The supported conclusion is that effective construction should consider diversity, representativeness, and predictive competence.
  • Limitations: The evaluation is limited to 25 LLMs and two future-prediction benchmarks with 100 sampled questions each.Future work is proposed across larger, evolving model populations and broader prediction domains.
Loading 2608.24001v1…