Source-linked AI summary
Quizz: Targeted crowdsourcing with a billion (potential) users
Panagiotis G. Ipeirotis, Evgeniy Gabrilovich
TL;DR
Crowdsourcing needs reliable participation and access to qualified contributors, while paid marketplaces may not provide specialized expertise. Quizz combines gamified quizzes, expertise assessment, targeted advertising, and contribution feedback to acquire knowledge from unpaid users. Experiments report effective targeting of niche expertise and lower costs than paid crowdsourcing, with scope limitations in some tasks.
Problem
Crowdsourcing struggles to reproduce successful engagement and to access users with specialized expertise, while paid incentives attract a different participant population.
Method
Quizz uses gamified calibration and collection questions, targeted online advertising, information-based contribution evaluation, and feedback to optimize ad placement.
Results
50% conversion rate per ad click and systematically declining cost per answer are reported, while the approach achieves knowledge-curation accuracy superior to paid crowdsourcing at the same or lower cost.
Takeaways & Limitations
Advertising can identify niche audiences of competent unpaid users and combine the reach of unpaid participation with more predictable crowdsourcing deployment.
Takeaways & Limitations
The advertising system failed to identify users knowledgeable across a highly diverse music quiz covering artists and songs from multiple genres.
Abstract
from arXiv · showhide
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement. Our experiments, which involve over ten thousand users, confirm that we can crowdsource knowledge curation for niche and specialized topics, as the advertising network can automatically identify users with the desired expertise and interest in the given topic. We present controlled experiments that examine the effect of various incentive mechanisms, highlighting the need for having short-term rewards as goals, which incentivize the users to contribute. Finally, our cost-quality analysis indicates that the cost of our approach is below that of hiring workers through paid-crowdsourcing platforms, while offering the additional advantage of giving access to billions of potential users all over the planet, and being able to reach users with specialized expertise that is not typically available through existing labor marketplaces.
1. INTRODUCTION
Quizz addresses the difficulty of building reliable crowdsourcing communities and finding qualified contributors by using advertising to target users, assess their expertise, and collect new knowledge. Its experiments report stronger quality and lower cost than paid crowdsourcing while using engagement mechanisms to sustain valuable participation.
- Motivation: Crowdsourcing platforms struggle to reproduce successful communities and to attract users with specialized expertise.Paid crowdsourcing provides predictable monetary incentives, but labor marketplaces often lack experts such as Swahili speakers or maxillofacial surgeons.
- Approach: Quizz uses targeted advertising to attract unpaid users who match the expertise needed for a task.The approach leverages existing advertising systems and their targeting capabilities to identify suitable user communities.
- Approach: Users answer calibration questions with known answers to assess expertise and collection questions with unknown answers to acquire new information.Quizz identifies correct answers to collection questions from responses provided by competent participants.
- Evaluation: Quizz experiments examine how engagement mechanisms can retain high-quality users while discouraging low-quality participation.The paper emphasizes short-term goals and rewards as incentives for contributing useful answers.
- Evaluation: Quizz achieves superior knowledge-curation accuracy at the same or lower cost than paid crowdsourcing mechanisms.The comparison covers both paid hourly and paid piecemeal crowdsourcing, evaluating user quality and total monetary cost.
2. ADVERTISING FOR TARGETING USERS
Quizz uses online advertising to reach unpaid contributors with relevant interests and expertise, then feeds contribution outcomes back to the advertising system. This feedback shifts optimization from attracting clicks toward attracting users who participate and provide valuable answers.
- Motivation: Crowdsourcing must attract inherently motivated participants, but successful engagement strategies remain difficult to reproduce.Paid crowdsourcing addresses participation through monetary rewards, although extrinsically motivated users differ from intrinsically motivated users.
- Targeting: Online advertising lets Quizz reach unpaid users and potentially scale participation to billions of people.Unlike paid marketplaces, advertising can expand the reachable population by increasing the campaign budget, subject to budgetary constraints.
- Targeting: Topic-specific keywords and relevant ad contexts help advertising platforms target users with expertise in the task domain.The study uses advertising systems such as Google AdWords to place ads in search and display contexts related to the target topic.
- Optimization: Quizz records conversions after ad clicks and sends the signal back so campaigns optimize for participation rather than clicks.A conversion occurs when a user clicks an ad and then answers at least one quiz question.
- Optimization: Fine-grained feedback about contribution quantity and quality enables advertising systems to find users more likely to participate and provide high-quality answers.In one campaign, conversion rate increased from 20% to over 50% within one month after optimization.
3. MEASURING USER CONTRIBUTIONS
Quizz measures user contributions with an information-theoretic metric that rewards both answer quality and volume. It extends this metric with Bayesian estimation to account for uncertainty about user competence.
- Information Gain: Information Gain combines answer quality and quantity by measuring information transmitted per answer and multiplying it by the user's total answers.The user is treated as a noisy channel, with competence q representing the probability of answering a randomly selected quiz question correctly.
- Information Gain: The entropy H(q, n) is zero for perfect answers and log(n) for random selection among n possible answers.These boundary cases represent no uncertainty and maximum uncertainty, respectively.
- Bayesian Information Gain: A Bayesian formulation treats competence q as a distribution rather than a point estimate because early answer data are sparse.With a Beta(1, 1) prior, a user with a correct answers and b incorrect answers has posterior Beta(a + 1, b + 1).
- Bayesian Information Gain: The expected information gain integrates information gain over the posterior distribution of user competence.This allows the metric to incorporate uncertainty in the estimated quality of each user.
- Assumptions: The default model assumes a constant latent competence across questions and uniformly distributes errors across the n − 1 incorrect answers.The paper notes Item Response Theory and mixtures of Beta priors as possible future refinements.
- Bayesian Information Gain: The system also computes information-gain variance and uses these measurements to support decisions about assessing users versus collecting new judgments.The expected information gain varies with the number of answers and user competence when each question has n = 2 possible answers.
4. EXPLORATION / EXPLOITATION
Quizz frames question selection as an exploration–exploitation problem: calibration questions assess competence, while collection questions acquire new knowledge. An MDP uses the user's current state, expected information, future participation, and dropout risk to choose actions.
- Exploration / Exploitation: Calibration questions explore user competence, whereas collection questions exploit that competence to obtain answers to questions with unknown answers.The system alternates between learning how competent a user is and collecting new information from them.
- Decision Process: The MDP represents a user by correct answers a, incorrect answers b, and collection questions c, then selects the next action from this state.The implementation returns utilities for all actions and the optimal next action.
- Participation and Computation: The policy accounts for dropout by favoring states with higher survival probability, while bounding recursive utility through a survival probability below one and a finite execution horizon.The horizon is increased iteratively to assess convergence of actions and utilities.
- Action Utilities: A collection question yields immediate expected information gain plus future utility, while the system discounts uncertainty-related value to learn more about competence.Its utility includes the state reached after asking one more collection question.
- Action Utilities: A calibration question updates expected information gain and its variance, thereby revising the utility estimates for previously asked collection questions.Correct and incorrect responses lead to different forward paths in the utility calculation.
5. ENGAGEMENT INCENTIVES
Because short participation limits both competence assessment and knowledge collection, Quizz continuously tests engagement mechanisms designed to retain competent users and discourage low-quality contributions.
- Engagement Incentives: Quizz experiments with incentives to prolong competent users' participation while discouraging non-knowledgeable users from submitting low-quality answers.The platform treats continuous experimentation with incentive mechanisms as a key component of its design.
- Feedback: Feedback experiments compare no feedback, correctness feedback, and display of the correct answer.The underlying hypothesis is that immediate performance feedback can motivate continued participation and improvement.
- Scores: Score experiments vary whether users see correct-answer percentages, total correct answers, information-gain scores, or combinations of these measures.
- Social Comparison: The system also tests crowd-performance displays and leaderboards as mechanisms intended to increase effort or alter behavior through social comparison.The leaderboard hypothesis is that users strive to reach top positions.
6. EVALUATION
Quizz evaluates engagement, contribution quality, advertising targeting, incentives, and costs across multiple experiments. The results show that feedback-driven targeting and unpaid-user self-selection can improve answer yield and quality, while some incentive and topic conditions limit performance.
- Recruitment and evaluation metrics: 34.60% average conversion engaged 4,091 of 11,825 ad-clicking users, with conversion reaching 51.25% by September 30.Conversion rate rose from about 20% at the beginning of the month.
- Capacity and cost analysis: $0.16 per fact at 99% accuracy and $0.09 at 95% accuracy were the average validation costs across eight quizzes.The Artist and Song quiz was a cost outlier because advertising failed to identify users knowledgeable across its diverse music topics.
- User contributions and paid comparison: Unpaid users self-selected by quality: knowledgeable users submitted more answers, while low-quality users dropped out.Paid workers continued submitting low-quality answers, and their information gain remained significantly below that of unpaid users.
- Advertising targeting: Targeted advertising produced 9.2x more answers and 11.4x higher total information gain than untargeted advertising.The targeted campaign had a 34.62% versus 13.43% conversion rate, 2,866 versus 279 answers, and 7,560 versus 610 bits of information gain.
- Advertising targeting: 30% higher conversion, 42% more submitted answers, and 63% higher total information gain resulted from the conversion optimizer.Conversion rose from 29% to 39%, answers from 1,183 to 1,683, and information gain from 2,870 to 4,690 bits.
- Engagement incentives: Showing the correct answer most strongly increased participation across answer counts, correct answers, and total information gain.Knowing whether an answer was correct did not have a similarly strong effect.
- Engagement incentives: Competitive information increased participation, whereas displaying leaderboards generally reduced participation, answer counts, and information gain.Leaderboards were strongly positive early, when users could more easily reach top positions.
- User contributions and paid comparison: Paid crowdsourcing workers were usually not topic experts, and matching unpaid-user performance would require paying about $0.05 per question or $3 per hour.The best paid worker achieved 68% quiz quality across 40 answers.
7. RELATED WORK
Quizz builds on crowdsourcing and engagement research while addressing how to assess user competence and acquire knowledge from users. It also distinguishes its approach from paid crowdsourcing by targeting specialized expertise and combining assessment with knowledge acquisition.
- Engagement mechanisms: Prior work models badges, leaderboards, and engagement metrics, while Quizz empirically evaluates engagement mechanisms and web-analytics measures.Its engagement analysis focuses mainly on web analytics measurements.
- Quality inference: Quizz explicitly assesses competence with calibration questions, whereas related methods estimate participant quality without known ground truth through redundancy-based techniques.The authors identify combining such quality inference with exploration and exploitation as future work.
- Planning under limited participation: Unlike industrial acceptance sampling, Quizz must plan dynamically because users have limited lifetimes rather than sustained production capacity.This difference makes the crowdsourcing setting more dynamic than most acceptance-sampling use cases.
8. CONCLUSIONS
The conclusions present advertising-based targeting and engagement as a way to identify and retain competent unpaid contributors at scale. Experiments with over ten thousand users show improving acquisition efficiency, practical incentive effects, and quality-adjusted costs comparable to paid crowdsourcing.
- Targeting and acquisition: Quizz uses existing advertising platforms to identify niche audiences of competent users and provides feedback that improves targeting effectiveness.The application reports a 50% conversion rate per ad click, while cost per answer decreases as the advertising system learns.
- Engagement: Controlled experiments with over ten thousand users show how incentive mechanisms can engage unpaid users and extend their lifetime in the system.The conclusions frame these experiments as guidance for setting up incentive mechanisms in practice.
- Cost-quality trade-off: Quality-adjusted costs are on par with paid crowdsourcing despite the advertising costs.The comparison concerns quality-adjusted costs rather than raw monetary costs alone.
- Practical implications: The approach combines unpaid-user engagement with the predictability associated with paid crowdsourcing through an ad-based deployment model.The authors describe this as a foundation for more predictable deployment and engagement of unpaid users.
A. VARIANCE OF INFORMATION GAIN
This appendix analyzes uncertainty in information gain when user quality is unknown. Under a constant correctness probability across questions and a uniform prior, it expresses the variance using quantities defined from the quiz and the user's answer history.
- Assumptions: When user quality q is uncertain, the information gain for each question is also uncertain.The variance is derived under the assumption that q is constant across questions and the prior is uniform.
- Variance expression: The variance expression combines terms involving a, b, and the quantities J_a, J_b, I_ab, and K_ab.The supplied appendix passages provide the component expressions for I_ab and K_ab and an additional variance term.
- Parameter definitions: Here, n is the number of answer options, while a−1 and b−1 count the user's correct and incorrect submitted answers.These definitions connect the variance parameters to the user's observed answer history.