Source-linked AI summary
CDAS: A Crowdsourcing Data Analytics System
Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, Meihui Zhang
TL;DR
Computer systems struggle with complex tasks such as image tagging and natural language processing, motivating human assistance. The paper presents CDAS and its quality-sensitive answering model, which plans worker participation and verifies answers while supporting online refinement. The evaluated human-assisted methods achieve higher accuracy, while answer-arrival order can make early approximate results unreliable.
Problem
Complex tasks such as image tagging and natural language processing remain difficult for computers, while crowdsourcing results can be degraded by worker errors and costly replication.
Method
CDAS combines a prediction model for estimating required workers with a verification model based on worker performance, plus online result processing.
Results
Human-assisted CDAS methods achieve much higher accuracy than some traditional approaches across the implemented Twitter sentiment and image-tagging jobs.
Takeaways & Limitations
Embedding quality-sensitive answering into the crowdsourcing query engine supports accuracy-aware job planning and human-assisted data analytics.
Takeaways & Limitations
Online approximate accuracy varies with answer-arrival order, so confidence must be updated dynamically because early termination can degrade accuracy.
Abstract
from arXiv · showhide
Some complex problems, such as image tagging and natural language processing, are very challenging for computers, where even state-of-the-art technology is yet able to provide satisfactory accuracy. Therefore, rather than relying solely on developing new and better algorithms to handle such tasks, we look to the crowdsourcing solution -- employing human participation -- to make good the shortfall in current technology. Crowdsourcing is a good supplement to many computer tasks. A complex job may be divided into computer-oriented tasks and human-oriented tasks, which are then assigned to machines and humans respectively. To leverage the power of crowdsourcing, we design and implement a Crowdsourcing Data Analytics System, CDAS. CDAS is a framework designed to support the deployment of various crowdsourcing applications. The core part of CDAS is a quality-sensitive answering model, which guides the crowdsourcing engine to process and monitor the human tasks. In this paper, we introduce the principles of our quality-sensitive model. To satisfy user required accuracy, the model guides the crowdsourcing query engine for the design and processing of the corresponding crowdsourcing jobs. It provides an estimated accuracy for each generated result based on the human workers' historical performances. When verifying the quality of the result, the model employs an online strategy to reduce waiting time. To show the effectiveness of the model, we implement and deploy two analytics jobs on CDAS, a twitter sentiment analytics job and an image tagging job. We use real Twitter and Flickr data as our queries respectively. We compare our approaches with state-of-the-art classification and image annotation techniques. The results show that the human-assisted methods can indeed achieve a much higher accuracy. By embedding the quality-sensitive model into crowdsourcing query engine, we effectiv...[truncated].
1. INTRODUCTION
Crowdsourcing extends computer analytics by assigning difficult subtasks to human workers, but worker errors create accuracy and cost challenges. CDAS addresses these challenges with quality-sensitive prediction, verification, and adaptive processing.
- Motivation: Crowdsourcing uses human participation to supplement computer tasks that remain difficult for automated methods.Examples include image tagging and natural language processing, with applications deployed through platforms such as AMT.
- Challenges: Human workers can submit random or uninformed answers, making crowdsourced results arbitrarily inaccurate.Replication provides conflicting answers for comparison but does not fully resolve answer diversity.
- Challenges: A required accuracy creates a trade-off between assigning enough workers for reliable inference and limiting worker cost.The paper illustrates this with a 95% image-tag precision target and a $0.01 worker cost per HIT.
- Approach: CDAS proposes a quality-sensitive answering model that aims to improve query-result quality while reducing processing cost.The model is the core of CDAS and supports analytics jobs including image tagging and sentiment analysis.
- Approach: The model uses prediction to estimate required workers and verification to select among returned answers using worker performance and probability.Prediction uses historical worker-performance distributions; verification combines vote distributions with worker accuracies.
- Approach: Adaptive processing provides an approximate result with confidence and refines it as asynchronous worker answers arrive.This online-style strategy is intended to reduce waiting compared with waiting for all results.
- Evaluation: CDAS is evaluated with Twitter sentiment analytics and Flickr image tagging jobs using real data.The paper compares these human-assisted applications with existing classification and image-annotation approaches.
2. OVERVIEW
CDAS combines computer processing with human workers through a crowdsourcing engine, using prediction and verification models to target required accuracy. Its architecture supports privacy-aware task processing and applications such as Twitter sentiment analytics, which outperforms traditional approaches in reported accuracy.
- Architecture: CDAS combines a job manager, crowdsourcing engine, and program executor to use human workers alongside computer-based analytics.Its processing mechanism differs from conventional analytics systems that rely solely on computers.
- Crowdsourcing engine: The crowdsourcing engine generates platform-compatible tasks, predicts the required number of workers, and verifies returned answers probabilistically.Processing proceeds in two phases: task generation and answer combination or ambiguity removal.
- Privacy: A privacy manager adaptively changes question formats and may reject workers when tasks involve sensitive data.
- Applications: CDAS validates its models through Twitter sentiment analytics and image tagging applications.The system is presented as a framework for deploying different crowdsourcing applications.
- Reported result: CDAS can achieve much higher accuracy than some traditional machine-learning and information-retrieval approaches for Twitter sentiment analytics.
- Twitter sentiment analytics: A TSA query specifies keywords, required accuracy, answer domain, timestamp, and time window, with example outputs including opinion percentages and reasons.The example query concerns public opinions about iPhone4S from Oct-14-2011 to Oct-23-2011 and requires 95% accuracy.
3. PREDICTION MODEL
The prediction model links required accuracy to the number of workers, while accounting for AMT costs. It combines probability-based accuracy estimation with tighter search and worker sampling to choose worker counts more efficiently.
- 3.1 Economic Model in AMT: AMT charges CDAS (mc+ms)n for a HIT with n workers, making worker-count selection directly affect query cost.Each worker receives mc, and AMT receives ms per worker.
- 3.1 Economic Model in AMT: The query cost is (mc +ms)wK ×g(C), where g relates the required accuracy C to the number of workers.For TSA, K tweets are available per time unit and w is the query window.
- 3.2 Voting-based Prediction: The voting strategy accepts an answer when at least ⌈n/2⌉ workers agree, but does not determine how many workers n should be used.This leaves the accuracy-cost trade-off unresolved without a prediction model.
- 3.2 Voting-based Prediction: The voting-based prediction model derives g for each query from an accuracy requirement and provides a bound for probability-based verification.Its analysis uses workers’ accuracy probabilities and their mean value μ.
- 3.2 Voting-based Prediction: Chernoff-based estimation can be conservative for small worker counts, so CDAS uses it as an upper bound and binary-searches odd n for a tighter estimate.The binary search finds the minimum odd n satisfying the expected-accuracy requirement.
- 3.3 Sampling Worker Accuracy: CDAS samples worker accuracy by injecting αB testing samples into each HIT; its implementation uses α=0.2 and B=100.The collected statistics feed both the prediction and verification models, and experiments report acceptable estimation even at low sampling rates.
4. VERIFICATION MODEL
CDAS replaces simple voting with probability-based verification that combines answer distributions with workers’ historical accuracies. The model estimates answer confidence, supports quality bounds, prunes unlikely answer domains, and can terminate online when an answer meets the required condition.
- Motivation: Voting can fail when no answer exceeds 50% agreement and incorrectly treats workers as equally reliable.Worker accuracies vary, and malicious or colluding workers can make majority-based decisions unreliable.
- Probability-based Verification: The verification model applies Bayesian analysis to estimate each answer’s probability of being correct from worker-performance distributions and observed answers.It extends data-fusion techniques to integrate conflicting worker results.
- Answer Confidence: CDAS accepts the answer with the highest confidence, effectively weighting votes by worker confidence rather than counting all votes equally.Higher-accuracy workers receive larger confidence values and therefore greater influence.
- Quality Bound: The model guarantees that, under its stated conditions, the correct answer is returned with probability no less than C.The proof uses a quality bound on the expected number of workers providing the correct answer and assumes independent, identically distributed worker accuracies.
- Answer-Domain Estimation: The system estimates the effective answer-domain size m to prune low-probability answers that would otherwise reduce the correct answer’s weight.The verification sets ϵ to 0.05 and uses the resulting bound to estimate m.
- Early Termination: In online processing, CDAS stops when an answer meets the termination condition; otherwise, it updates answer confidences as new worker results arrive.This avoids waiting for every worker before accepting an adequately confident result.
5. PERFORMANCE EVALUATION
CDAS is evaluated on Twitter sentiment analytics and image tagging, with experiments examining accuracy, worker estimation, verification, online termination, sampling, and comparisons against automated methods. Across these settings, the quality-sensitive crowdsourcing approach generally achieves high accuracy while reducing worker usage or processing cost.
- 5.1.1 Crowdsourcing vs. SVM Algorithm: Crowdsourcing achieved higher Twitter sentiment accuracy than LIBSVM in most tested cases, even with one worker.The comparison used five movies with 200 tweet reviews each and varied the number of workers from one to five.
- 5.1.2 Accuracy Analysis: The refined worker estimate was less than half the conservative estimate as required accuracy varied from 0.65 to 0.99.Binary search refined the prediction-model estimate and was used in subsequent experiments.
- 5.1.2 Accuracy Analysis: With 29 workers, probability-based verification reached 0.99 accuracy and outperformed Half-Voting and Majority-Voting as worker counts increased.The model incorporates workers’ historical performance when selecting answers.
- 5.1.2 Accuracy Analysis: The probability-based verification model consistently satisfied user-required accuracy in the tested Twitter sentiment experiments.The real accuracy was computed against ground truth and compared with the requester’s specified requirement.
- 5.1.3 Online Processing: ExpMax saved more than 50% of workers while MinMax and ExpMax satisfied the required accuracy in all tested cases.MinMax reduced worker usage by 20%, whereas MinExp missed the requirement at several points.
- 5.1.4 Effect of Sampling: Sampling rates above 10% produced stable mean accuracy and average error approaching 0; 20% sampling met all required accuracies with a small gap from 100% sampling.The system therefore used a 20% sampling rate in verification experiments.
- 5.2 Application 2: IT: For image tagging, crowdsourcing exceeded 80% accuracy with one worker, while ALIPR reached 30% at best and 12.6% on the apple tag.The comparison used five groups of the top 20 Flickr images returned by a tag.
- 5.2 Application 2: IT: The image-tagging model always satisfied the user’s specified accuracy requirement in the reported experiment.The evaluation compared achieved accuracy with the user-required-accuracy line.
6. RELATED WORK
Crowdsourcing has expanded from web platforms to complex analytics and database applications, but quality control remains a central barrier. CDAS addresses this barrier by balancing monetary cost and accuracy while integrating conflicting human answers.
- Crowdsourcing applications: Web 2.0 platforms and services such as Amazon Mechanical Turk enable users and programmers to distribute complex tasks among online workers.Crowdsourcing supports tasks including image annotation, information retrieval, and natural language processing that computers may not perform satisfactorily.
- Crowdsourcing databases: Qurk and CrowdDB integrate crowdsourcing into database systems, while hQuery extends this approach with query-language support.These systems partially answer queries through the Amazon Mechanical Turk platform.
- Quality control: Quality control is a major obstacle because human workers are unpredictable and may provide arbitrarily bad answers.Prior work addresses incentives, worker-quality ranking, and abusive-content identification.
- CDAS positioning: CDAS differs from previous efforts by designing a feasible model that balances monetary cost and accuracy while integrating conflicting worker results.The query engine combines quality control with crowdsourcing-supported database processing.
7. CONCLUSION
CDAS introduces a quality-sensitive answering model that guides crowdsourcing query plans and supports high-quality analytics results. Its prediction, verification, and online-processing components were implemented for Twitter sentiment analysis and image tagging.
- 7. CONCLUSION: CDAS uses prediction and verification sub-models to estimate required workers and select the best answer under user accuracy requirements.The model also updates answers gradually during verification through online processing.
- 7. CONCLUSION: CDAS implemented Twitter sentiment analytics and image tagging using real Twitter and Flickr data with Amazon Mechanical Turk.