Source-linked AI summary
When is a crowd wise?
Clintin Davis-Stober, David Budescu, Jason Dana, Stephen Broomell
TL;DR
The paper asks when aggregated judgments are wiser than individual judgments, especially under bias and dependence. It develops a general theoretical framework and finds that averaging is usually more accurate, with maximal wisdom arising from strongly differing judgments.
Problem
Existing crowd-wisdom definitions provide limited generality because they mainly compare aggregates with average individual accuracy rather than arbitrary selection rules.
Method
The authors define crowd wisdom through expected squared error and analyze arbitrary linear aggregates, individual-selection probabilities, biases, and judgment correlations.
Results
Crowd wisdom is robust: weighted aggregation usually outperforms selecting the best member, and accuracy is maximized when members’ judgments are as negatively correlated as possible.
Takeaways & Limitations
Judgment diversity, rather than independence alone, is central to maximizing the accuracy advantage of crowds.
Takeaways & Limitations
The framework assumes variable individual judgments and permits correlations among crowd members, while its conclusions depend on using average squared error as the accuracy metric.
Abstract
from arXiv · showhide
Numerous studies and anecdotes demonstrate the "wisdom of the crowd," the surprising accuracy of a group's aggregated judgments. Less is known, however, about the generality of crowd wisdom. For example, are crowds wise even if their members have systematic judgmental biases, or can influence each other before members render their judgments? If so, are there situations in which we can expect a crowd to be less accurate than skilled individuals? We provide a precise but general definition of crowd wisdom: A crowd is wise if a linear aggregate, for example a mean, of its members' judgments is closer to the target value than a randomly, but not necessarily uniformly, sampled member of the crowd. Building on this definition, we develop a theoretical framework for examining, a priori, when and to what degree a crowd will be wise. We systematically investigate the boundary conditions for crowd wisdom within this framework and determine conditions under which the accuracy advantage for crowds is maximized. Our results demonstrate that crowd wisdom is highly robust: Even if judgments are biased and correlated, one would need to nearly deterministically select only a highly skilled judge before an individual's judgment could be expected to be more accurate than a simple averaging of the crowd. Our results also provide an accuracy rationale behind the need for diversity of judgments among group members. Contrary to folk explanations of crowd wisdom which hold that judgments should ideally be independent so that errors cancel out, we find that crowd wisdom is maximized when judgments systematically differ as much as possible. We re-analyze data from two published studies that confirm our theoretical results.
1. INTRODUCTION
The paper defines crowd wisdom as a linear aggregate having lower expected squared error than a probabilistically selected individual, then derives testable conditions for when this holds. Its results indicate that crowd wisdom is robust to bias, correlated judgments, and aggregation or sampling choices, while re-analyses support the effect in prior data.
- Definition and framework: Crowd wisdom holds when a linear aggregate has lower expected squared error than an individual selected from the crowd according to a prespecified probability distribution.This definition generalizes comparisons with the average individual’s accuracy and supplies an explicit, testable condition.
- Definition and framework: The framework allows members to have different means and variances, systematic bias, and arbitrary covariances with one another.Judgments and criterion values are modeled as random variables, permitting correlated predictions without imposing restrictions on members’ means or variances.
- Definition and framework: Accuracy is defined by average squared prediction error, enabling distribution-free results without assuming normality or a particular distributional shape.Crowd predictions are formed by linearly combining judgments with fixed nonnegative weights that sum to one.
- Main results: Crowd wisdom is robust to aggregation and sampling rules, and even a simple average is usually wiser than an individual judgment.The framework examines trade-offs between interdependence and bias and considers many crowd weights and individual-selection probabilities.
- Main results: A weighted aggregate is almost always preferable to selecting the single best member unless that member can be selected deterministically, because aggregation reduces prediction variance.This advantage can persist even when crowd members are biased.
- Empirical application: Re-analyses of data from Vul and Pashler (2008) and Simmons et al. (2011) found evidence supporting the crowd-wisdom effect.The first analysis extended the original study by examining a group of individuals, while the second supported the overall conclusion.