Source-linked AI summary
Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, Mohammad Allahbakhsh
TL;DR
Crowdsourcing relies on workers with diverse and often unknown abilities, creating a need for systematic quality control. This survey synthesizes a quality model, assessment methods, and assurance actions, then analyzes platform support and identifies open research issues. It finds broad platform coverage of assurance actions but limited support for many quality attributes and substantial room for improvement in assessment.
Problem
Crowdsourcing quality control is difficult because workers have unknown or varied skills and motivations, while existing platforms are not robust against quality problems.
Method
The survey synthesizes a quality model and reviews methods for assessing quality attributes and actions for enforcing expected quality levels.
Results
The survey finds that platforms support many assurance actions, while support for identified quality attributes is limited and assessment methods still have substantial room for improvement.
Takeaways & Limitations
Quality control requires understanding where quality arises in the crowdsourcing process and selecting suitable assessment and assurance methods and actions.
Takeaways & Limitations
Current crowdsourcing instruments commonly neglect factors affecting people’s attitudes and satisfaction, and assessment methods require significant improvement.
Abstract
from arXiv · showhide
Crowdsourcing enables one to leverage on the intelligence and wisdom of potentially large groups of individuals toward solving problems. Common problems approached with crowdsourcing are labeling images, translating or transcribing text, providing opinions or ideas, and similar - all tasks that computers are not good at or where they may even fail altogether. The introduction of humans into computations and/or everyday work, however, also poses critical, novel challenges in terms of quality control, as the crowd is typically composed of people with unknown and very diverse abilities, skills, interests, personal objectives and technological resources. This survey studies quality in the context of crowdsourcing along several dimensions, so as to define and characterize it and to understand the current state of the art. Specifically, this survey derives a quality model for crowdsourcing tasks, identifies the methods and techniques that can be used to assess the attributes of the model, and the actions and strategies that help prevent and mitigate quality problems. An analysis of how these features are supported by the state of the art further identifies open issues and informs an outlook on hot future research directions.
1. INTRODUCTION
Crowdsourcing assigns work to unknown and varied workers, making output quality multifaceted and difficult to control. This survey organizes quality control into a taxonomy covering quality models, assessment, assurance, platform support, and future challenges.
- Crowdsourcing context: Crowdsourcing outsources work through an open call, with requesters submitting tasks and workers producing outputs that may be evaluated and rewarded.Rewards can include money, gifts, or reputation badges, and quality control may be delegated to the platform.
- Quality problem: Quality depends on workers, task processes, selection, coordination, incentives, and defenses against low-quality or malicious outputs.Existing platforms are reported to be insufficiently robust against quality problems and attacks such as cheating or manipulation.
- Quality problem: No general framework robustly and flexibly equips crowdsourcing services with quality controls, while existing techniques are often embedded in proprietary platforms and difficult to generalize.Designing, building, and maintaining robust quality controls therefore remains challenging and partly unsolved.
- Survey scope: The survey classifies quality-control research through a taxonomy covering quality dimensions and attributes, assessment methods, and assurance actions.Assessment measures attribute values, while assurance actions aim to achieve expected quality levels and intervene when quality drops.
- Survey scope: The survey analyzes how state-of-the-art crowdsourcing platforms and services support quality control, then discusses shortcomings, limitations, challenges, and future research opportunities.The analysis spans practical platform support and the outlook for quality-control research and development.
2. QUALITY IN CROWDSOURCING
The survey frames crowdsourcing quality as a multifaceted problem requiring a holistic taxonomy, a quality model, assessment methods, and assurance actions. It defines task quality by how well outputs meet or exceed requester expectations.
- Quality challenges: Crowdsourcing quality control must address novel issues arising from human involvement and workers with varied skills and motivations.Quality also depends on workers, task processes, coordination, and quality-control activities.
- Quality definition: The survey defines crowdsourced-task quality as the extent to which output meets and/or exceeds requester expectations.
- Quality-control taxonomy: The taxonomy organizes quality control around a quality model, assessment methods, assurance strategies, and concrete actions.Assessment methods measure attributes; strategies determine what to improve and actions prevent or fix quality problems.
- Quality model and assessment: The quality model captures literature-identified quality dimensions and attributes, while assessment methods analyze how those attributes can be measured.Dimensions are not directly measurable; attributes may be concrete or abstract, with abstract values derived from concrete attributes.
3. CROWDSOURCING QUALITY MODEL
The quality model groups crowdsourcing quality attributes into dimensions covering data, tasks, people, and performance-related concerns. It combines general quality properties with task-specific attributes identified across the literature.
- Model structure: The survey’s quality model groups crowdsourcing attributes into dimensions identified from the literature.
- Task quality: Task quality includes description, user interface, incentives, terms and conditions, and performance, each affecting worker participation or output quality.Clear instructions influence worker performance; interface quality affects task ease, and incentives relate to output quality or execution speed.
- People quality: People quality covers requesters, workers, and groups, including fairness, communication, credentials, experience, diversity, availability, and non-collusion.Low worker availability can reduce output quality or slow execution, while group properties include diversity and non-collusiveness.
- Data quality: Data quality centers on output accuracy, while consistency measures similarity among workers’ outputs and timeliness concerns availability for further processing.Task-specific attributes may also be needed, such as coverage and conciseness for textual summaries.
4. QUALITY ASSESSMENT
The survey classifies quality assessment methods by who performs the assessment: individuals, groups, or computation-based mechanisms. It reviews methods ranging from ratings and qualification tests to voting, consensus, expert review, and implicit feedback.
- Assessment techniques: Assessment methods are organized into individual, group, and computation-based techniques according to the assessment actor.Individual methods use workers, experts, or requesters; group methods use multiple participants; computation-based methods run automatically.
- Individual techniques: Rating assigns an item a value from a scale, which may be unary, binary, ordinal, or continuous.Ratings are used for output quality, text similarity, worker confidence, task quality, and requester reputation.
- Individual techniques: Qualification tests use questionnaires with known correct answers to grant task access when workers meet a minimum correctness threshold.They are suited to tasks requiring prior knowledge or skills such as language or programming.
- Group techniques: Voting aggregates preferences to rank candidates, whereas group consensus aggregates ratings to infer the most representative label for one item.Consensus can identify cheaters through disagreement with the majority, but divergent outputs may also have value.
- Feedback methods: Implicit feedback infers evaluator judgments from behavior rather than explicit feedback forms.In WikiTrust, preserving a user’s content changes increases that user’s reputation.
5. QUALITY ASSURANCE
Quality assurance in crowdsourcing comprises reactive and proactive strategies that prevent or address quality problems. The survey organizes these strategies into improving data, selecting and incentivizing people, and actions such as training, task redesign, and runtime control.
- 5. QUALITY ASSURANCE: Reactive assurance responds to identified quality issues, whereas proactive assurance is applied without a triggering event.Filtering verified low-quality outputs is reactive; usability guidelines for task implementation are proactive.
- 5. QUALITY ASSURANCE: Assessment can itself improve worker, reviewer, requester, and overall quality, although the survey focuses on first-order assurance actions.The survey notes positive effects from reviewing and rating but does not further study these assessment side effects.
- 5. QUALITY ASSURANCE: The identified strategies improve data quality, select people, incentivize people, train workers, improve task design, and control execution.Runtime control includes reallocating tasks when it becomes evident that some workers will not produce outputs.
- 5. QUALITY ASSURANCE: Incentivizing people draws on extrinsic drivers such as rewards and intrinsic drivers such as entertainment or social comparison.These sub-strategies leverage distinct motivations that can encourage better performance.
- 5. QUALITY ASSURANCE: Concrete assurance actions can target preparation, task usability and understandability, or execution while workers are active.Training may teach workers or provide feedback, while task redesign can address issues such as empty input fields.
5.1. Improve Data Quality
Improving data quality addresses defects in inputs and outputs before or during crowdsourced work. The survey describes preprocessing, cleansing, iterative improvement, redundancy, aggregation, and filtering as relevant actions.
- 5.1. Improve Data Quality: Good-quality input is a precondition for good-quality output, and requesters remain responsible for input accuracy.Inaccurate or visibly poor inputs may also make workers reluctant to perform a task.
- 5.1. Improve Data Quality: Input preprocessing can assemble, reshape, or filter data, while computer vision can improve pictures before workers process them.Data cleansing can also be applied to output data.
- 5.1. Improve Data Quality: Aggregating redundant responses can improve output quality, but it requires paying more workers.Worker responses may also be weighted according to reliability to increase the influence of more reliable answers.
- 5.1. Improve Data Quality: Filtering removes outputs assessed as bad so that only outputs judged good are retained.Reported bases for filtering include self- and expert reviews, output agreements, arbitration, and peer review.
- 5.1. Improve Data Quality: Iterative improvement lets workers directly revise submitted work rather than merely evaluate it.Examples include collaboratively improving text and repeatedly deciphering blurred text.
5.2. Select People
Selecting people aims to match workers to task requirements and remove unsuitable or adversarial participants. Approaches include filtering, targeted assignment, recommendations, recruitment, and team formation.
- 5.2. Select People: Worker filtering can use profiles, skills, expertise, badges, demographics, personality, and reliability to identify suitable participants.These attributes are used as indicators of workers’ fit for particular tasks.
- 5.2. Select People: Rejecting fake, malicious, colluding, or otherwise unsuitable workers requires detecting adversarial behavior without excluding good workers.Examples include CAPTCHAs and start-time or end-time signals, with thresholds requiring careful tuning.
- 5.2. Select People: Automatically assigning stronger workers to harder tasks uses defined task requirements and worker skills.Priority queues and matching mechanisms support proactive task assignment.
- 5.2. Select People: Task recommendations use identified task-worker matches while leaving workers free to accept or reject suggested tasks.Recommendations can be delivered through notifications and based on workers’ browsing history.
- 5.2. Select People: Promotion and team-based recruitment enlarge or organize the worker pool for task-specific needs.Teams can match profiles to requirements and help address complex tasks in specialized communities.
5.3. Improve Extrinsic Motivation
Extrinsic motivation is addressed through reward design, bonuses, and promotions. The surveyed literature emphasizes matching rewards to desired performance and adjusting them to support quality and sustained engagement.
- 5.3. Improve Extrinsic Motivation: Reward form and amount are important for crowdsourced task success, and studies examine how to tune them for good results.Dynamic payment adjustment based on output quality is reported as effective.
- 5.3. Improve Extrinsic Motivation: Bonuses add discretionary rewards to the base payment when workers meet predefined goals, milestones, or performance conditions.Examples include correct answers, rapid responses, and milestone achievement.
- 5.3. Improve Extrinsic Motivation: Promotion prospects can increase motivation over longer periods in environments with sustained worker engagement.Promotions may provide higher rewards or access to new task types, especially under deferred payment schemes.
5.4. Improve Intrinsic Motivation
Intrinsic motivation strategies improve participation and, in some settings, output quality by engaging workers beyond direct monetary reward. The survey covers purpose, competition, transparency, gamification, priming, teaching, feedback, and teamwork.
- Purpose-driven tasks can attract more motivated workers, including volunteers who contribute for the purpose rather than monetary reward.
- Performance comparison, ratings, and leaderboards can encourage workers to enter a competition mode and perform better.
- Social transparency can build trust, bonds, shared standards, and quality-control mechanisms among workers and requesters.
- Gamification improves results for complex tasks compared with paid conditions, while simple tasks show no difference.
- Teaching, requester feedback, peer review, and teamwork support quality improvement, skill development, or collaborative task completion.
5.6. Improve Task Design
Task design affects crowdsourcing quality through complexity, decomposition, role separation, input validation, usability, worker reflection, and pacing. The reviewed strategies simplify work, constrain errors, and support sustained attention.
- Output accuracy typically decreases as task complexity increases, so designs should limit cognitive complexity and choose manageable task granularity.
- Decomposing complex tasks into parallel subtasks can lower complexity before merging the resulting outputs.
- Separation of duties assigns finding, fixing, and verifying errors to different workers to help prevent fraud or errors.
- Validating worker inputs against formal requirements prevents malformed entries that can lower output quality.
- Usable interfaces, examples, rationales, breaks, and speed-oriented designs address output quality, verification, retention, or completion-time priorities.
5.7. Control Execution
Execution-control strategies manage worker availability, task visibility, monitoring, dependencies, ordering, and workflow automation. They aim to reduce waiting, detect problems during execution, avoid unnecessary work, and coordinate complex tasks.
- Maintaining a ready worker pool minimizes waiting time when results must be collected quickly, including in real-time settings.
- Repeatedly posting tasks increases their visibility and attractiveness by keeping them near the top of platform listings.
- Monitoring execution enables requesters to identify incomplete or low-quality work while tasks are still running.
- Controlling task order avoids useless work when task outputs are interdependent and can reduce the number of deployed tasks.
- Workflow automation organizes complex composite tasks through event-condition-action approaches, scripting, or visual modeling.
6. ANALYSIS OF STATE-OF-THE-ART CROWDSOURCING PLATFORMS
The survey classifies heterogeneous crowdsourcing platforms using a taxonomy of quality attributes, assessment methods, and assurance actions, then compares their support qualitatively and through heat maps. Current platforms prioritize output accuracy, rewards, and relatively simple assessment, while assurance actions are broader but important gaps remain.
- Platform selection and taxonomy: The analysis compares 14 heterogeneous platforms selected from industrial systems and academic prototypes, without claiming completeness or popularity.
- Platform selection and taxonomy: The taxonomy classifies platform support for quality attributes, assessment methods, and assurance actions, with heat maps summarizing support intensity.
- Quality model: 13/14 platforms support output accuracy and extrinsic rewards; worker selection also uses location (9/14), skills (8/14), reliability (6/14), and reputation (6/14).
- Open issues: State-of-the-art instruments commonly neglect worker and requester behavior, while group work and collaboration remain insufficiently studied and supported.
- Quality assessment: Only about half of identified assessment methods are implemented; rating (9/14) and voting (6/14) are the most prominent, while complex computation-based methods remain limited.
- Open issues: Assessment is mostly applied after execution, leaving room for during-execution analytics and preemptive interventions targeting accuracy, timeliness, and cost efficiency.
- Quality assurance: Assurance support spans most identified actions, led by reward tailoring (11/14), output filtering (8/14), worker filtering (6/14), and rationale prompting (8/14).
- Quality assurance: Research prototypes contribute strongly to inter-task coordination and task-order control, whereas commercial platforms emphasize broader cross-task actions such as intrinsic motivators and team building.
7. CONCLUSION AND OUTLOOK
The survey synthesizes a quality model for crowdsourcing and reviews methods for assessing and assuring quality, concluding that quality problems remain unresolved. It identifies domain-specific services and shared regulation and ethics as critical future directions for sustainable crowdsourcing.
- Conclusion: The survey analyzes literature from 2009 onward to synthesize a quality model covering attributes studied in crowdsourcing.It also discusses methods for assessing quality and actions for enforcing it.
- Conclusion: Quality control remains crucial, but crowdsourcing still does not reliably deliver human intelligence without quality issues.
- Future research: Domain-specific service providers are needed to assist less-skilled requesters throughout crowdsourcing processes.The paper distinguishes quality assurance for image labels from assurance for text translations.
- Future research: Crowdsourcing platforms commonly manage crowds and tasks abstractly, requiring requesters to possess intimate crowdsourcing expertise.
- Future research: Shared rules, legal protections, work ethics, and standards for crowd work remain underdeveloped across the ecosystem.The passage mentions taxation, pensions, dispute resolution, and related issues.
Online Appendix to: Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques and Assurance Actions
The online appendix identifies the paper’s five authors and their institutional affiliations.
- Authors: Florian Daniel and Cinzia Cappiello are affiliated with Politecnico di Milano.
- Authors: Pavel Kucherbaev is affiliated with Delft University of Technology.
- Authors: Boualem Benatallah is affiliated with the University of New South Wales, while Mohammad Allahbakhsh is affiliated with the University of Zabol.