Peer Truth Serum: Incentives for Crowdsourcing Measurements and Opinions

1 Introduction

• Crowdsourcing lets centers aggregate independent reports about measurements, product ratings, and human-computation tasks without direct supervision.Examples include community sensing, reputation systems, online surveys, and prediction markets.Source: §1 · pp. 1–2
• Inaccurate answers can result from minimized effort, incompetence, or sloppiness, so rewards should maximize payment for reporting privately believed accurate values.Source: §1 · p. 2
• Proper scoring rules incentivize truthful predictions when a ground truth eventually becomes public and can score the initial reports.Weather forecasting and prediction markets are examples.Source: §1 · p. 2
• Peer consistency mechanisms instead score agents against randomly selected peer reports when ground truth does not exist or will remain unknown.Accurate peer reports make accurate reporting an equilibrium.Source: §1 · p. 2
• Earlier mechanisms often assume identical belief responses to measurements, whereas this paper studies a general non-binary setting without that assumption.Source: §1 · pp. 2–3

2 Related Work

• Peer prediction and Bayesian truth serum use Bayesian-game assumptions about agents’ information and belief formation, with BTS eliciting posterior-belief information.Source: §2 · p. 3
• Subsequent methods improve robustness to collusion, small populations, non-binary information, redundant elicitation, and absent common priors.Source: §2 · p. 3
• Mechanisms for multiple similar phenomena reduce the need for common priors but are mostly designed for binary information, while other mechanisms require trusted peers or structured priors.Source: §2 · p. 3
• This paper studies stochastic arrivals observing one phenomenon and seeks minimal mechanisms that elicit only private information without knowing agents’ beliefs.It builds on helpful reporting to obtain a unique mechanism under mild belief-update assumptions.Source: §2 · p. 3

3 Contributions

• The paper introduces Peer Truth Serum as a mechanism for truthful information elicitation.Source: §3 · p. 4
• Peer Truth Serum replaces common posterior beliefs with a common prior and a restriction requiring belief updates to reflect observations.The authors argue common priors are more applicable in practical settings than common posteriors.Source: §3 · p. 4
• Peer Truth Serum is unique up to a linear transformation under its specified belief-update restriction.Source: §3 · p. 4
• The paper derives impossibility theorems that characterize achievable properties of minimal elicitation mechanisms under different belief-update assumptions.Source: §3 · p. 4

4.1 Peer consistency mechanisms

• The center elicits reports from M > 1 independent agents observing the same phenomenon, pays through a peer-based function, and uses each agent only once.The payment may depend on one or more peer reports, with no collusion or repeated game.Source: §4.1 · p. 4, §4 · p. 4
• Peer consistency payments take the agent’s report, a same-round reference report, and a distribution over possible values as inputs.The center publishes a continuously updated public distribution derived from a report histogram.Source: §4.1 · pp. 4–5

4.2 Reporting Strategies

• A reporting strategy maps an observation, prior belief, and public distribution to a report, with truthful reporting mapping every observation to itself.Source: §4.2 · p. 5

4.3 Agent Beliefs

• Agents share a common prior but privately update beliefs after observing signals; the paper distinguishes self-dominating updates from weaker self-predicting updates.Self-predicting updates permit correlated values and include Dirichlet-based Bayesian updating.Source: §4.3 · pp. 5, 7

4.4 Solution concepts

• Because agents may have different private belief updates, the paper models them as private types in a subjective game and uses ex-post subjective equilibrium for the main belief structures.Source: §4.3 · p. 6, §4.4 · p. 8

4.3 Agent Beliefs

• The strongest results assume self-predicting updates with a public distribution close to the common prior, while initial-stage settings with unknown priors require additional restrictions on the value space.Source: §4.3 · pp. 7–8

4.4 Solution concepts

• When only one value is underreported, including common binary cases, there is only one possible non-truthful equilibrium.Source: §4.4 · pp. 8–9

5.1 Minimal Elicitation

• Minimal elicitation transmits only the elicited information, avoiding the larger belief reports required by mechanisms such as Bayesian Truth Serum.This reduces the information agents must provide and can reduce their effort.Source: §5.1 · p. 9

5.2 Arbitrage-Free

• Arbitrage-free mechanisms give agents with only the public histogram the same expected payment for every report, allowing their expected payment to be reduced to zero.Such agents add no information about accuracy and therefore should not be incentivized to participate.Source: §5.2 · p. 9

5.3 Unconstrained-prior

• Unconstrained-prior mechanisms impose no restrictions on agents’ prior beliefs, reducing reliance on assumptions that may not hold across agents.The property is considered alongside informed priors.Source: §5.3 · p. 9

5.4 Heterogeneous Update Tolerant

• Heterogeneous-update tolerance allows agents to respond differently to measurements, including self-dominating and self-predicting update restrictions.This permits diversity in how agents react to their observations.Source: §5.4 · p. 9

5.5 Truthful

• Truthfulness means truthful reporting is a strict equilibrium, and rewards can be scaled so truthful reporting offsets the effort required to obtain accurate answers.Strictness excludes indifference and implies pure equilibrium strategies.Source: §5.5 · p. 10

5.6 Asymptotically Accurate

• Any truthful mechanism is asymptotically accurate, but asymptotic accuracy alone does not require truthful equilibria and can arise from impractical mechanisms that already know the true distribution.With continuous payments independent of the true distribution, asymptotic accuracy forces truthful reporting whenever the private belief is sufficiently close to the public histogram; discontinuities are excluded by consistency.Source: §5.6 · pp. 10–11

6 Output Agreement

• The output agreement mechanism rewards agreement with a peer using a report-independent positive constant, making truthful reporting optimal under self-dominating updates.For observation x, the expected reward for reporting x is Pr[x|x], which exceeds Pr[y|x] for every alternative y.Source: §6 · p. 11
• Output agreement mechanisms are truthful independently of prior distributions and permit heterogeneous prior beliefs and heterogeneous self-dominating updates.They are also widely proposed as peer consistency mechanisms.Source: §6 · p. 11
• No unconstrained-prior payment function can make truthful reporting a strict equilibrium, even with common informed priors and self-predicting updates.Different agents’ posterior beliefs can coincide after different observations, forcing identical expected rewards for reports that cannot both be truthful.Source: §6 · p. 12

7 Helpful reporting strategies

• Helpful misreports can move the public histogram toward the true distribution faster than truthful reports when the public prior is far from an agent’s prior.The idea is conceptually similar to prediction markets, although the ground truth is not revealed here.Source: §7 · pp. 12–13
• A ρ-helpful strategy is truthful when public and private beliefs are ρ-close and otherwise avoids reporting values already over-represented publicly.The definition allows reports of under-represented values even when they differ from the observation.Source: §7 · p. 12
• When agents have informed common priors, every equilibrium in helpful strategies is asymptotically accurate, despite multiple possible equilibria.This permits analysis of helpful incentives without characterizing each equilibrium in detail.Source: §7 · p. 13

8 Peer Truth Serum

• The Peer Truth Serum induces ρ-helpful equilibria when agents have ρ-informed common priors and self-predicting belief updates.It is truthful when the public distribution is sufficiently close to private beliefs and otherwise supports helpful reports that avoid over-represented values.Source: §8 · pp. 13–15
• PTS rewards agreement with a peer by a term inversely proportional to the public probability of the reported value, while disagreement receives no report-dependent reward.Its simplest form sets C = 1 and f(rr) = 0; choosing f(rr) = −C yields zero expected payment for agents relying only on the public distribution.Source: §8 · pp. 13–14
• When private beliefs are not close enough for truthfulness, agents can select under-represented reports, and the posterior best-response set remains within the helpful-strategy space.With multiple under-represented values, multiple equilibria may exist; a single under-represented value yields a unique non-truthful singleton strategy.Source: §8 · pp. 15–16
• PTS is asymptotically accurate even though it is not always truthful, because truthful and helpful reports both make the report histogram approximate the true distribution.When private beliefs are close to the public distribution, truthful reporting is optimal; otherwise, singleton or other helpful equilibria may arise.Source: §8 · pp. 15–16
• Theorem 2 identifies PTS as the unique arbitrage-free mechanism that tolerates heterogeneous self-predicting updates, is truthful near a common prior, and is asymptotically accurate with informed priors.The result follows because only consensus can be rewarded and the payment function must have the PTS form.Source: §8 · p. 16

• Asymptotic accuracy requires agents’ updates to satisfy a self-predicting restriction rather than allowing arbitrary belief-updating functions.Source: §9 · p. 17
• No incentive scheme allows unrestricted heterogeneous updates while remaining asymptotically accurate.Source: §9 · p. 17
• Asymptotic accuracy also requires informed priors because agents with uninformed priors cannot distinguish whether their report would move the public prior toward or away from the true distribution.Source: §9 · p. 17
• No asymptotically accurate mechanism exists with unconstrained priors when updates are self-predicting and heterogeneous.Source: §9 · p. 17
• Theorems 3 and 4, together with Theorems 1 and 2, characterize feasible minimal peer-prediction mechanisms across prior and update restrictions.Source: §9 · p. 17

• A non-binary construction makes reports of x exceed Q[x] and reports of z fall below Q[z], preventing convergence of the public distribution to Q.Source: §10 · p. 18
• For answer spaces with |X| ≥3, no asymptotically accurate mechanism exists for self-predicting heterogeneous updates with informed, different priors.Source: §10 · p. 18
• Binary indicative observations satisfy the self-predicting condition, but self-prediction alone does not establish asymptotic accuracy.Source: §10 · p. 18
• The Peer Truth Serum is asymptotically accurate in a binary setting with informed, possibly non-common priors and indicative observations.Source: §10 · pp. 18–19
• In the binary equilibrium, agents report honestly or report the under-represented value relative to the public prior.Source: §10 · p. 18
• The equilibrium argument uses informed priors to make reporting the observed binary value yield a strictly higher expected payoff than misreporting it.Source: §10 · p. 19

11 Optimality of the Peer Truth Serum

• A report shifts the public histogram toward its reported value by ϵ = 1/(t+1), enabling a first-order approximation when t is large.The update increases the reported value's mass and proportionally decreases the mass of every other value; additive Laplace smoothing is ignored in this derivation.Source: §11 · p. 19
• Because the center cannot sample the true distribution directly, PTS uses an honest peer report as a sample for the logarithmic scoring rule.Under honest peers, their reports correspond to samples from the true distribution, so the score incentivizes the report that maximizes the center's expected gain.Source: §11 · p. 20
• The logarithmic-scoring derivation yields the PTS form with f(rr) = 0 and C = 1, while scaling and shifting payments gives a more general form.Analogous normalization can also be applied to the payment function derived from quadratic scoring.Source: §11 · p. 20
• The PTS mechanism uniquely maximizes the expected quality of the report distribution when quality is evaluated with logarithmic scoring and a peer report.This result assumes the center updates R as the histogram of received reports.Source: §11 · pp. 19–20
• Using quadratic scoring instead leads to an alternative PTS version whose self-predicting condition requires the observed value to have the highest relative probability increase.The corresponding uniqueness and impossibility results are conjectured rather than established for this scheme.Source: §11 · p. 20

12.1 Community Sensing

• In community sensing, actual measurements receive higher rewards than fictitious reports, accuracy increases rewards, and gaming requires collusion by a majority of agents.These findings come from simulations using an air-pollution model based on actual measurements.Source: §12.1 · p. 21

12.2 Human Computation

• In human computation, a PTS-derived bonus for uncommon answers eliminated systematic bias in Amazon Mechanical Turk experiments.The scheme addresses settings where aggregated answers otherwise remain biased because workers respond differently to atypical instances.Source: §12.2 · p. 21

12.3 Online Polls

• Online polling attracts participants with ulterior motives, while prediction markets counter this by requiring participants to risk their own money but depend on eventual ground truth.PTS addresses the latter constraint by comparing predictions against one another instead.Source: §12.3 · p. 21
• In a randomized platform comparison, PTS and a classical prediction market achieved about the same prediction accuracy.The comparison was conducted by randomly distributing users between the two schemes.Source: §12.3 · p. 22
• PTS can reward online predictions by comparing them with peers, allowing payment before the true outcome is known or when it will never be known.A Swissnoise platform implemented this approach for questions about elections, sports, breakthroughs, and hypothetical votes or product quality.Source: §12.3 · pp. 21–22

• Peer Truth Serum is the unique mechanism class shown to satisfy the paper’s intuitive practical constraints while remaining optimal.Peer consistency provides incentives without independent ground-truth verification.Source: §13 · p. 22
• Peer Truth Serum requires a common prior belief but not common posterior beliefs, unlike earlier peer prediction mechanisms such as.Source: §13 · p. 22
• Agents without information receive no reward, so the mechanism does not require restricting participation.Source: §13 · p. 22
• With informed prior beliefs, the mechanism is asymptotically accurate and guaranteed to converge to the true distribution.Source: §13 · p. 22
• Experiments demonstrate applicability to community sensing, human computation, and online opinion polls, with encouraging results.A crowdsourcing variant averaging reports from similar phenomena is more robust to collusive behaviour.Source: §13 · p. 22

14.1 Proof of Proposition 2

• Proposition 2 assumes homogeneous agents and shows that if truthful reporting fails near a matching prior and public histogram, misreporting cannot preserve convergence across nearby asymmetric priors.With the true distribution equal to the common prior, convergence permits either honest reporting, contradicting misreporting, or a prior-specific misreporting pattern that fails under another prior.Source: §14.1 · p. 25

14.2 Proof of Proposition 3

• Proposition 3 constructs two distinct priors yielding identical posterior beliefs for different observations, so any payment based on reports, the public histogram, and expected reference reports can incentivize honesty in at most one scenario.The argument remains valid when payments depend on agent identities if the center does not know which agents use which priors.Source: §14.2 · p. 26

14.3 Proof of Proposition 4

• Proposition 4 proves convergence in probability to Q because informed agents never non-truthfully report an over-represented value, while updates occur in fixed batches of M reports.The proof bounds overestimation probabilities using a worst-case reporting process and Chebyshev’s inequality.Source: §14.3 · p. 27

14.4 Proof of Proposition 5

• Proposition 5’s Dirichlet-prior argument shows that satisfying truthful-reporting constraints for all priors with the same public expectation imposes the proposition’s derived payment condition.The proof varies concentration parameters while preserving the expected signal distribution and shrinking posterior updates.Source: §14.4 · pp. 29–30

14.5 Proof of Proposition 6

• Proposition 6’s lemma shows that, with at least 3 values and priors bounded away from 0 and 1, bounded helpful payment functions are independent of the reported value when the reference report is fixed.The proof derives τ(r, rr, R) = f(rr, R) for r ≠ rr by contradiction and symmetry.Source: §14.5 · pp. 30–31

14.6 Proof of Theorem 3

• Theorem 3 derives a contradiction when the self-predicting condition holds for one answer but fails for another: truthful incentives require C > 0 in one case and C < 0 in the other.The proof reduces report comparisons to the mechanism’s g component because the f component is report-independent.Source: §14.6 · p. 32

14.7 Proof of Theorem 4

• Theorem 4 constructs an asymmetric three-answer setting with uninformed priors where agents can report honestly for x and z, but observation y yields specified rewards for reporting y or z.The construction uses Pr[x] = R[x], Pr[y] = R[y] − ϵ, and Pr[z] = R[z] + ϵ, with posterior parameters involving δ.Source: §14.7 · pp. 32–33
• When agents use symmetric and consistent pure strategies, report frequency for y differs from Rt[y], causing histogram R to diverge from its current value.If the prior Pr is uninformed and Q = Rt, the payment scheme might not be asymptotically accurate.Source: §14.7 · p. 33

14.8 Proof of Proposition 8

• The peer truth serum does not converge when agents’ priors differ, demonstrated with Q[x] = 0.5, Q[y] = 0.2, and Q[z] = 0.3.The construction considers agents who have priors different from R while believing other agents’ priors equal R.Source: §14.8 · p. 33
• With heterogeneous priors, agents misreport under peer truth serum: they report x after observing y when R[y] > Q[y], and report y instead when R[y] ≤ Q[y].They report y after observing z when R[y] < Q[y], but report z after observing z when R[y] > Q[y].Source: §14.8 · p. 34
• The reported frequency of z is strictly below 0.3 and that of x strictly above 0.5, so R does not converge in probability to Q.This occurs when Rt[x] = 0.7, Rt[y] = 0.2, and Rt[z] = 0.1.Source: §14.8 · p. 34