PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowdsourcing for Bioinformatics

Benjamin M. Good, Andrew I. Su

arXiv:1302.6667v1q-bio.QMcs.CYcs.SIphysics.soc-ph

TL;DR

Bioinformatics problems often need human input, but guidance on how to use crowdsourcing effectively in science has been limited. This review classifies crowdsourcing systems by task type and surveys bioinformatics examples. It concludes with a framework for matching problems to these approaches while noting that worker conditions require attention.

  • Problem

    Bioinformatics tasks from annotation to protein structure can benefit from human input, but resources describing how crowdsourcing systems work and how to use them effectively in science are limited.

    Source: §1 · p. 1

  • Method

    The review organizes crowdsourcing into high-volume microtasks and high-difficulty megatasks, then surveys system types and examples in bioinformatics.

    Source: §1 · p. 1

  • Results

    The review presents successful examples spanning sequence alignment, protein-folding algorithms, parasite counting, and biomarker discovery through contests.

    Source: §1 · p. 1, §3.1 · p. 5, §3.2 · p. 5

  • Takeaways & Limitations

    Matching crowdsourcing approaches to task difficulty and volume provides a practical framework for applying human effort to biological problems.

    Source: §1 · p. 1, §4 · p. 6

    Some microtask-market workers earn less than $2 per hour, making worker conditions a concern for responsible and sustained use.

    Source: §5 · p. 7

Abstract

from arXiv · show

Motivation: Bioinformatics is faced with a variety of problems that require human involvement. Tasks like genome annotation, image analysis, knowledge-base construction and protein structure determination all benefit from human input. In some cases people are needed in vast quantities while in others we need just a few with very rare abilities. Crowdsourcing encompasses an emerging collection of approaches for harnessing such distributed human intelligence. Recently, the bioinformatics community has begun to apply crowdsourcing in a variety of contexts, yet few resources are available that describe how these human-powered systems work and how to use them effectively in scientific domains. Results: Here, we provide a framework for understanding and applying several different types of crowdsourcing. The framework considers two broad classes: systems for solving large-volume 'microtasks' and systems for solving high-difficulty 'megatasks'. Within these classes, we discuss system types including: volunteer labor, games with a purpose, microtask markets and open innovation contests. We illustrate each system type with successful examples in bioinformatics and conclude with a guide for matching problems to crowdsourcing solutions.

1 INTRODUCTION

Crowdsourcing harnesses distributed human intelligence for bioinformatics problems ranging from large-volume annotation to difficult algorithmic challenges. This review classifies systems by task type and describes how their incentives, work, and quality controls can be matched to scientific problems.

1 INTRODUCTION

  • Crowdsourcing has produced bioinformatics outcomes including improved sequence alignments, a protein-folding algorithm, and parasite counts from tens of thousands of blood-cell images.Thirty sequence-alignment algorithms reportedly improved on the state of the art in two weeks for $6,000.Source: §1 · p. 1
  • The review focuses on directed work requiring human intelligence, with discrete tasks and clear endpoints, rather than wiki-style open-ended work.The described systems allow top-down control over the work conducted.Source: §1 · p. 1
  • Distributed computing and online health research rely on crowds but not crowd intelligence, so the review treats them as conceptually distinct.The authors exclude both from the crowdsourcing applications considered here.Source: §1 · p. 1
  • The review examines crowdsourcing as an alternative to algorithms that mimic human abilities, aiming to offer practical guidance for tackling difficult biological problems.The authors call this approach “artificial artificial intelligence.”Source: §1 · p. 1
  • The framework separates high-volume, low-difficulty microtasks from individually challenging megatasks, with systems varying in incentives, work, and quality controls.Table 1 arranges systems from easier, more numerous tasks toward harder tasks requiring fewer individual solutions.Source: §1 · p. 1, Table 1 · p. 2

2 CROWDSOURCING MICROTASKS

Microtask systems distribute large volumes of simple, short tasks across workers and aggregate their responses to produce annotations. Their principal differences lie in the incentives used to attract workers.

2 CROWDSOURCING MICROTASKS

  • Microtasks typically take seconds and require only the ability to follow brief instructions; bioinformatics examples often involve image or text annotation.Workers act in parallel to label large volumes of data quickly.Source: §2 · p. 2
  • Redundancy and aggregation, such as voting across multiple workers, can yield annotation quality typically as good as or better than expert annotators.Each task is presented to multiple workers, whose contributions are integrated into a final solution.Source: §2 · p. 2
  • Microtask systems in this category vary primarily in the incentives they use to attract workers.The article next examines the systems in turn.Source: §2 · p. 2

2.1 Volunteer (Citizen Science)

Volunteer projects enlist the public for large-scale image classification, including biomedical image annotation. Galaxy Zoo and CellSlider illustrate the volume such efforts can process.

2.1 Volunteer (Citizen Science)

  • Within ten days of launch, Galaxy Zoo had collected 8 million classifications of distant-galaxy images.After nine months, more than 100,000 people had classified over 1,000,000 images, with an average of 38 volunteers viewing each image.Source: §2.1 · p. 2
  • CellSlider was launched by Zooniverse in collaboration with Cancer Research UK as a volunteer initiative in the biomedical domain.The project followed Galaxy Zoo’s public image-classification effort.Source: §2.1 · p. 2
  • CellSlider volunteers label cell types and quantities in stained cancer-biopsy images, including irregular yellow-stained cells associated with estrogen-receptor expression.Quantifying these “cancer core” cells can help assess treatment response and personalize therapy; the initiative reported analyzing 550,000 images in its first three months, before publishing findings.Source: §2.1 · p. 2

2.2 Casual Games

Casual games align game objectives with real-world microtasks, rewarding players with points and levels as they produce annotations. Bioinformatics examples include sequence alignment, malaria-image classification, and neuron tracing.

2.2 Casual Games

  • Games with a purpose embed microtasks in simple, typically web-based games whose objectives align with the underlying task.Players earn points and advance through levels, while winning requires solving real-world problems at high quality and in large quantities.Source: §2.2 · p. 3
  • Phylo players solved alignment puzzles by moving nucleotide blocks until pre-computed alignment sections reached a target score; their reassembled blocks improved more than 70% of original alignments.In seven months, more than 12,000 players completed over 254,000 puzzles.Source: §2.2 · p. 3
  • MOLT and MalariaSpot aggregated multiple players’ responses and benchmarked them on images with known annotations, achieving expert-level performance on their respective malaria-image tasks.The systems were envisioned for medical-image annotation in regions without local pathologists, as well as pathologist and computer-vision training.Source: §2.2 · p. 3
  • EyeWire players identify the structure and connectivity of individual retinal neurons in three-dimensional microscopy images.The images were collected using serial block face scanning electron microscopy.Source: §2.2 · p. 3

2.3 Microtask Markets

Microtask markets use cash rewards for image and text annotation, but published bioinformatics applications were few. Examples show crowd workers handling specialized tasks, while platform services address common market-management problems.

2.3 Microtask Markets

  • Cash-reward markets have been used in bioinformatics for image and text annotation, though published examples were few.The authors describe the AMT as the first and leading microtask market, with other platforms emerging.Source: §2.3 · p. 3

2.3.1 Image annotation.

  • $53.60 bought 5,360 polyp-image judgments in 3.5 days; crowd and machine-learning AUCs were both close to 0.85, while a later integrated system reached 0.91.The crowd system used 150 workers, with 20 independent assessments for each of 268 candidates; the integrated system combined automated predictions and crowdsourcing.Source: §2.3.1 · p. 3

2.3.2 Text annotation.

  • Biomedical text crowdsourcing was just beginning, with examples covering clinical-trial annotations, gene-mutation relations, and linked medication attributes (Yetisgen-Yildiz, et al., 2010; Burger, et al., 2012; Zhai, et al., 2012).For gene-mutation verification, majority voting achieved 83.8% overall accuracy across 1,733 candidate relationships; the best-scoring worker achieved 90.5% accuracy on more than 1,000 HITs (Burger, et al., 2012).Source: §2.3.2 · pp. 3–4

2.3.3 Microtask platforms.

  • The AMT and emerging alternatives provide market infrastructure; meta-services such as Crowdflower address spammer identification, worker rating, and response aggregation, generally with less requester control.The passage also names open-source projects Turkit and CrowdForge.Source: §2.3.3 · p. 4

2.4 Forced Labor (Workflow Sequestration)

Workflow sequestration embeds unpaid data work in tasks people already want to complete. In one biomedical application, clinicians created medication-problem links at scale, with high specificity but lower sensitivity.

2.4 Forced Labor (Workflow Sequestration)

  • ReCAPTCHA pairs a known word that verifies a human with scanned text to digitize, making website access the incentive for free OCR work.The passage says the task is difficult to accomplish computationally and engages hundreds of millions of web users.Source: §2.4 · p. 4
  • 867 clinicians created 239,469 problem-medication links in one year; after filtering, 11,166 distinct pairs remained, with 99.6% specificity and 42.8% sensitivity.Compared with expert review, the links had high specificity and lower sensitivity.Source: §2.4 · p. 4
  • The authors suggest this approach may spread in biomedicine as electronic health record adoption continues; in bioinformatics, workflow sequestration was most common in education.Source: §2.4 · p. 4

2.5 Crowdsourcing and education

Educational projects make genome annotation part of undergraduate coursework, while language-learning translation illustrates how the work itself can generate worker value.

2.5 Crowdsourcing and education

  • Undergraduate annotation projects combine genome-sequence work with curriculum, addressing a large labor need while training students.Using redundancy, aggregation, and expert-curator review, these initiatives generated thousands of high-quality annotations (Brister, et al., 2012).Source: §2.5 · p. 4
  • In the described model, learners generate the capital to pay workers through knowledge acquired while using the system.The passage presents this as simultaneously accomplishing desired work and generating the capital needed for payment.Source: §2.5 · p. 4
  • DuoLingo applies the learning-as-payment pattern at scale: millions of students translate web documents while learning foreign languages.Source: §2.5 · p. 4

3 CROWDSOURCING MEGATASKS

Megatasks use crowdsourcing to recruit a few capable solvers for exceptionally difficult problems, through hard games and open innovation contests (p08).

3.1 Hard Games

  • Foldit players outperformed some leading automated structure-prediction systems, helped solve a retroviral structure that had eluded specialists, and optimized engineered enzymes [Khatib, et al., 2011; Eiben, et al., 2012].The review reports hundreds of thousands of players since 2008, though only some achieved these advances.Source: §3.1 · p. 5
  • Foldit players used its scripting system to collaboratively write protein-folding algorithms that rivaled professionally designed solutions, while automatic folding remains a greater challenge [Khatib, et al., 2011].Scripts combine optimization widgets and can be used alongside direct manipulation.Source: §3.1 · p. 5
  • Foldit trains players through increasingly complex levels, where they manipulate protein structures directly or run optimization tools; EteRNA instead invites designs for predefined RNA shapes, tested in weekly laboratory contests.EteRNA has recruited tens of thousands of players, including star players producing designs and identifying rules of RNA behavior.Source: §3.1 · p. 5
  • Hard games recruit and enable a small fraction of players with exceptional skills; their different perspectives can open new ways to solve problems.The review frames these games as portals for discovering latent scientists as well as making scientific discoveries.Source: §3.1 · p. 5

3.2 Open Innovation Contests

  • Open innovation contests set a challenge, invite public submissions, evaluate solutions against criteria, and reward the best with prizes that can also confer social prestige.The upfront cost is comparatively small relative to games, and prize amounts can be tuned to the difficulty and importance of the problem.Source: §3.2 · p. 5
  • In a two-week TopCoder contest costing $6,000, 30 of 122 submitted sequence-alignment solutions were more accurate and substantially faster than NCBI’s MegaBlast; none came from professional computational biologists.The contest drew 733 participants, mostly software developers.Source: §3.2 · p. 5
  • A successful million-dollar contest led to discovery of a novel biomarker for amyotrophic lateral sclerosis.Groups including Life Technologies and the U.S. Government’s Defense Threat Reduction Agency were running million-dollar bioinformatics contests.Source: §3.2 · p. 5
  • Contest participants spent an estimated 2,684 hours on the sequence-alignment problem in two weeks for $6,000, illustrating the skilled labor recruited in a short period.The estimate is reported by Lakhani and colleagues.Source: §3.2 · pp. 5–6

4 CHOOSING A CROWDSOURCING APPROACH

The review offers general guidance for matching crowdsourcing approaches to bioinformatics tasks, while emphasizing that problem characteristics and system design shape their performance.

4 CHOOSING A CROWDSOURCING APPROACH

  • For highly granular, repetitive tasks, volunteers, casual games, workflow sequestration, and microtask markets are options, each with distinct trade-offs in labor certainty, upfront cost, targeting, and scale.Volunteer initiatives and games may lack guaranteed labor and require substantial upfront investment; per-unit payment limits microtask-market scalability.Source: §4 · p. 6
  • For extended work requiring specialized skills, innovation contests and hard games are options; contests are more popular and generalizable, while their solutions have addressed difficult problems at comparatively tiny costs.Hard games offer potential scale, diversity, and collaboration, but these benefits are not guaranteed and require high upfront development time.Source: §4 · p. 6
  • Hard games best suit tasks with scoring functions that link in-game performance to the problem, giving players feedback to learn the problem space.The review notes that many important tasks may not be gamifiable.Source: §4 · p. 6
  • The microtask–megatask distinction may blur as integrated systems emerge, but problem-solving communities remain difficult to generate and sustain.Such future systems would need to form extended communities with varied skills and proclivities around shared goals.Source: §4 · pp. 6–7

5 SOCIAL IMPACT

Crowdsourcing redistributes scientific work beyond traditional institutions, widening public participation while raising worker-welfare concerns. It can also build public awareness of science and expand scientific problem-solving, provided system architects foster that awareness.

5 SOCIAL IMPACT

  • Crowdsourcing shifts scientific work from in-house teams to an open pool, creating both worker-exploitation concerns and opportunities for societal benefits.The authors say these issues matter alongside economic considerations of work and cost.Source: §5 · p. 7
  • Some microtask-market workers earn less than $2/hour, prompting calls for fair treatment, better pay, and longer-term worker relationships.The authors argue that worker satisfaction can support more effective crowdsourcing systems.Source: §5 · p. 7
  • Foldit expanded the number of people working on protein folding from perhaps a few hundred to hundreds of thousands.The authors contrast this with the small fraction of society that previously had direct input into scientific progress.Source: §5 · p. 7
  • Crowdsourcing participants become more aware of scientific problems, and that awareness may improve public perceptions, voting, and support for future generations if fostered effectively.The proposed effects are conditional on system architects fostering this awareness.Source: §5 · p. 7
  • Crowdsourcing systems open avenues for scientific exploration, but taking advantage of them requires openness and, in some cases, humility.Researchers must be willing to share difficult problems and let others help solve them.Source: §5 · p. 7
Loading 1302.6667v1…