PaperlayerView exact revision on arXiv

Source-linked AI summary

Crowdsourcing in Computer Vision

Adriana Kovashka, Olga Russakovsky, Li Fei-Fei, Kristen Grauman

arXiv:1611.02145v1cs.CVcs.HC

TL;DR

Collecting large, high-quality annotated datasets for computer vision is costly and challenging. This survey synthesizes crowdsourcing approaches for efficient data collection, quality assurance, and adaptive annotation, providing a starting point for designing strategies for new vision tasks.

  • Problem

    Computer vision needs large manually labeled datasets, but collecting annotations involves substantial researcher and annotator costs and difficult tradeoffs between effort and benefit.

  • Method

    The survey synthesizes crowdsourcing applications, interface and workflow designs, quality-assurance strategies, and methods for selecting data to annotate.

  • Results

    The surveyed literature provides a solid starting point for designing data-collection strategies, while common noise-prevention approaches work reasonably well.

  • Takeaways & Limitations

    Researchers can use the survey’s synthesis to devise data-collection strategies for new computer vision tasks.

  • Takeaways & Limitations

    With small exceptions, computer vision methods have not accounted for the large number of human factors affecting annotation.

Abstract

from arXiv · show

Computer vision systems require large amounts of manually annotated data to properly learn challenging visual concepts. Crowdsourcing platforms offer an inexpensive method to capture human knowledge and understanding, for a vast number of visual perception tasks. In this survey, we describe the types of annotations computer vision researchers have collected using crowdsourcing, and how they have ensured that this data is of high quality while annotation effort is minimized. We begin by discussing data collection on both classic (e.g., object recognition) and recent (e.g., visual story-telling) vision tasks. We then summarize key design decisions for creating effective data collection interfaces and workflows, and present strategies for intelligently selecting the most important data instances to annotate. Finally, we conclude with some thoughts on the future of crowdsourcing in computer vision.

Introduction

The introduction frames large, manually labeled datasets as essential but costly for computer vision, motivating crowdsourcing and algorithmic selection to optimize annotation cost against community benefits. The survey is organized around what, how, and which data should be annotated.

  • Role of Data: Annotated benchmark datasets focus research, evaluate algorithms quantitatively, and provide training data for learning visual-world statistics.
  • Motivation: Large manually labeled datasets drive computer vision progress but require costly, innovative collection and annotation techniques.The costs include researcher time, crowdsourcing workflow design, and annotator time.
  • What Data to Annotate: Annotation design requires choosing the media, annotation type, and scale, balancing sparse labels across more images against detailed labels across fewer images.
  • How to Collect Data: Crowdsourcing platforms such as Amazon Mechanical Turk outsource suitable low-cost tasks to non-expert online workers.
  • Which Data to Annotate: Algorithms can select challenging data instances so annotation effort focuses on scenarios current vision systems handle poorly, reducing human effort.
  • Survey Organization: The survey organizes its discussion around what, how, and which data should be annotated.

4 Introduction

This survey reviews how crowdsourcing has been used in computer vision and helps researchers design effective, high-quality, efficient annotation strategies. It focuses mainly on recent, low-cost, short-task annotation while also covering emerging approaches and algorithmically guided data selection.

  • Data selection: Section 4 examines which data should be annotated and how data collection can be integrated directly with algorithmic development.This shifts attention from annotation execution toward selecting data and coordinating collection with model development.
  • Survey scope: The survey overviews crowdsourcing applications in computer vision to help researchers devise data collection strategies and improve annotation quality and expedience.It also provides a broader view for crowdsourcing researchers examining applications in computer vision.
  • Annotation setting: Most discussed Mechanical Turk-style annotation uses payments on the order of cents and relatively small, often simple tasks requiring minutes.The survey asks readers to imagine this common annotation scenario, while acknowledging longer and more complex requester–provider interactions.
  • Research coverage: The survey focuses on crowdsourcing research from the past 5-10 years and covers subjective judgments, visual abstractions, similarity, annotation translation, and grouped labeling.These approaches illustrate how crowdsourcing can capture varied forms of visual knowledge.

What annotations to collect

The annotations to collect are determined by the computer vision tasks being addressed, spanning image and video tasks whose labels support both model training and quantitative evaluation.

  • Task-driven annotation: Annotation requirements follow the target task, such as binary scene labels for distinguishing parks from living rooms.Different tasks motivate different annotation types and collection goals.
  • Task-driven annotation: The section covers several computer vision tasks on images and videos and summarizes efforts to collect their corresponding annotations.The task overview is illustrated in Figures 2.1, 2.3, and 2.6.
  • Annotation use: The described annotations can be used for both training models and quantitatively evaluating what they have learned.The section does not distinguish annotations collected for training from those collected for test time.
  • Core visual tasks: Core image-understanding tasks include scene classification, object classification, and object detection.These tasks require annotations for understanding core visual building blocks of images.

2.1 Visual building blocks

Visual building-block annotations range from image-level class labels to object locations, pixel-level regions, parts, and attributes. Crowdsourcing pipelines scale these annotations while using verification, worker qualification, and procedures that address ambiguity and annotator disagreement.

  • Image classification: Image classification assigns one or more concept labels, commonly using web proposals followed by manual verification.ImageNet scaled this approach to 14,197,122 annotated images using query expansion and consensus-based verification.
  • Image classification: Two verification rounds improve scene-label quality by selecting positive images first and filtering false positives through negative-image selection.Fine-grained categories such as violin versus viola additionally require worker training and evaluation.
  • Object detection: Object detection localizes every object instance with an axis-aligned bounding box, enabling large-scale datasets for decomposing scenes into constituent objects.An iterative workflow alternates drawing a box, verifying it, and checking for additional instances, and was more efficient than majority-voting annotation.
  • Segmentation: Segmentation adds pixel-level detail through instance outlines for each target object or semantic outlines for contiguous regions sharing a property.Instance segmentation annotates each object separately, whereas semantic segmentation gives all pixels of a category one label.
  • Parts and attributes: Part annotations capture object spatial configuration, supporting inference about bird species and human actions from localized parts and their appearance.Attribute annotations benefit recognition and retrieval but remain subjective, with disagreement over both presence and relative labels.

2.2 Actions and interactions

This section examines crowdsourced annotation of actions and interactions in images and videos, moving beyond object identities and locations to capture relationships, behavior, and temporal extent. It highlights large-scale datasets and cost-saving strategies such as temporal interpolation, active learning, and proposal/verification workflows.

  • Images: Annotating actions and interactions provides a more comprehensive view of images than object or part locations alone.Image-based efforts include human actions, object interactions, and motivations explaining behavior.
  • Images: Crowdsourced image datasets scale interaction annotation to 58,808 human-object instances, more than 10,000 images with interaction boxes, and 47,774 images across 600 interaction categories.These datasets include TUHOI, UT Interactee, and HICO, respectively.
  • Videos: Video annotation is challenging because 10 minutes contain 18K-36K frames, but temporal redundancy enables labeling key frames and interpolating between them.Interfaces also use active learning to query frames expected to change estimated object tracks most.
  • Videos: Despite cost-saving interfaces, video object annotation remains costly and scarce, with datasets covering limited classes, frames, or short videos.Examples include YouTube-Objects, SegTrack, and ILSVRC video.
  • Videos: Crowdsourcing has produced large-scale action datasets through proposal/verification and fully crowdsourced scripting, filming, and annotation workflows.ActivityNet contains 27,801 untrimmed videos across 203 activities, while Hollywood in Homes created Charades through worker-generated videos.
  • Abstract representations: Abstract art and cartoons help crowdsource high-level interactions by suppressing low-level visual variability and emphasizing semantic configurations.INTERACT combines 3,172 photographs of two-person interactions with 3,000 cartoons depicting the same 60 fine-grained actions.

2.3 Visual story-telling

Visual story-telling extends computer vision beyond documentary image description toward questions, subjective reactions, implicit messages, and perceptual similarity. Crowdsourced collection therefore requires task-specific strategies to elicit challenging, high-quality judgments and account for substantial variation in human perception.

  • Visual question answering: Visual question answering asks systems to answer questions about image content, making it an AI-complete task requiring collaboration across several artificial-intelligence fields.Questions may require free-form text, image regions, or multiple-choice answers.
  • Visual question answering: Researchers collect challenging visual questions and answers using worker-engagement prompts, multiple independent answers, high-level movie questions, journalism-style question-answer pairs, and quality ratings.Antol et al. collect ten answers per question from ten annotators; other studies use text summaries, hourly payment, or ratings to shape data quality.
  • Visual question answering: Question-collection workflows must prevent trivial, irrelevant, or weakly image-grounded questions, because workers may optimize monetary gain per minute.The cited studies address this challenge through different strategies for obtaining high-quality questions and answers.
  • Subjective visual judgments: Crowdsourcing also captures subjective visual responses, including aesthetics, artistic quality, popularity, emotions, annoyance, and implicit messages conveyed by photographs.These tasks shift attention from physical image content toward how viewers perceive and react to visual media.
  • Subjective visual judgments: Subjective judgments are difficult to denoise by majority vote, so researchers often collect many labels per image and try to avoid annotator bias.The goal is to capture variability in human perceptions of aesthetics, emotion, and related responses.
  • Perceptual similarity: Perceptual-similarity studies use pairwise or set-based comparisons and adaptive querying, while quality controls include task caps and gold-standard questions.These methods learn similarity structures or embeddings from relatively focused comparisons rather than direct property labels.

2.4 Annotating data at different levels

Researchers typically collect annotations that match the target vision task, but auxiliary annotations can also support learning. Because annotation types serve different tasks and pose distinct quality challenges, systems can translate between them with human verification while the field seeks robust collection strategies.

  • Direct and auxiliary annotations: Researchers usually collect annotations that directly match the task, such as object labels for systems predicting image object labels, but auxiliary data can help learn the main task.Donahue and Grauman collect annotator rationales explaining why a label is present in an image.
  • Translating annotation types: Branson et al. automatically translate between annotation types, such as segmentations and object part locations, with humans verifying the translated annotations.The system presents each estimated annotation to a human for verification.
  • Quality and future strategies: Different annotation types create distinct quality challenges and require related but different techniques, motivating robust and commonly agreed-upon data-collection strategies.The paper next discusses data collection and quality assurance.

How to collect annotations

Effective crowdsourcing requires workflows and interfaces that make annotation efficient despite limited budgets, while structuring tasks to reduce worker effort and preserve accuracy. Researchers can use concept grouping, interactive or game-based designs, and established quality-control strategies, although project-specific needs often require custom systems.

  • Motivation: Limited annotation budgets make efficient crowdsourcing frameworks essential for producing useful large-scale datasets.The section frames collection design as potentially determining whether researchers can annotate a large dataset or only a small handful of images.
  • Workflows and interfaces: Workflow-management systems support complex iterative tasks, but researchers often need custom workflows and interfaces because project requirements are unique.Examples include task pipelines with sequential annotation and verification, active-learning integration, video querying, and specialized image or video labeling interfaces.
  • Efficient task design: Grouping concepts, objects, or images can reduce annotation effort by using high-level decisions, interactive semantic grouping, or multi-image similarity judgments.Hierarchical questions can propagate negative labels, while interactive systems let users confirm predicted group labels and redesigned interfaces compare multiple images at once.
  • Game-based annotation: Consensus-based annotation games can lower crowdsourcing costs while deriving likely-correct labels from independent agreement or successful partner guessing.The ESP game collects matching object names, whereas Peekaboom uses partner guessing to obtain spatial annotations; game development and maintenance can nevertheless be expensive.
  • Quality control: Three basic quality-control strategies are trusted gold-standard items, worker grading, and multiple annotations for every input.These approaches verify work through secretly injected reference images, peer assessment, or annotation redundancy.

Which data to annotate

After choosing annotation types and designing collection methods for quality and efficiency, researchers must actively or interactively select which data instances to label.

  • Which data to annotate: Data selection determines which instances receive annotations, even when the annotation type is already fixed.The section frames instance selection as a distinct decision from choosing annotation types and collection procedures.

4.1 Active learning

Active learning selects the most informative unlabeled samples for annotation so a classifier learns a target concept efficiently. Common criteria include classifier uncertainty, expected model or confidence change, and disagreement among ensemble members.

  • Active learning: Active learning iteratively selects a sample or batch of unlabeled data for human labeling instead of choosing a random batch.Selection is designed to help the classifier learn the desired concept as quickly as possible.
  • Active learning: Uncertainty sampling queries samples with the highest classifier uncertainty or the smallest peak in class-label probabilities.These criteria are illustrated in Figure 4.1(a).
  • Active learning: Other methods select instances expected to produce the largest model change or the largest change in classifier confidence.Reported applications include Gaussian processes, conditional random fields, and estimated labels.
  • Active learning: Ensemble-based methods query samples whose possible labels generate the greatest disagreement among ensemble members.This provides an alternative to uncertainty- and model-change-based selection criteria.

4.1. Active learning

Active learning accelerates learning by selecting informative instances for annotation, achieving higher accuracy than passive labeling as labels are iteratively provided. Selection methods also account for uncertainty, data density, label format, annotation level, and labeling cost.

  • 4.1. Active learning: Query-by-committee requests streaming instances on which randomly sampled hypotheses disagree or for which at least one hypothesis predicts an unknown class.This extends disagreement-based selection to streaming data.
  • 4.1. Active learning: Li and Guo select images that are both uncertain and located in dense feature regions with high mutual information to other unlabeled samples.The method combines classifier uncertainty with feature-space density.
  • 4.1. Active learning: Joshi et al. make multi-class annotation efficient by decomposing questions into binary answers and approximating the selection formulation.The approximation relies on an optimistic assumption and considers only a small set of possible labels.
  • 4.1. Active learning: Active learning selects labels iteratively so vision methods learn categories faster and achieve higher accuracy than passive methods.The passive method does not actively solicit labels.
  • 4.1. Active learning: When annotation types differ from the modeled task, active selection chooses both which images to label and the annotation level, including whole-image, segment, or fully detailed labels.These alternatives range from image-level object labels to segment labeling and full annotation.
  • 4.1. Active learning: Vijayanarasimhan and Grauman combine Mechanical Turk timing data with classifier risk and labeling cost when selecting among fully labeled, partially labeled, and unlabeled data.Their formulation models the time required to provide each label type on each image.

4.1. Active learning

Active learning in computer vision selects informative annotations to improve learning efficiency across object recognition, 3D reconstruction, and relative-attribute modeling. Approaches adapt the queried modality, simplify annotation tasks, batch instances strategically, and operate in realistic live-learning settings.

  • Adaptive query selection: Active learning can adaptively choose which annotation modality to examine, including context, 3D relationships, and attribute-based questions.Li and Guo select within one modality per iteration before choosing which modality to examine, while related approaches capture contextual and 3D relationships.
  • 3D active learning: For 3D reconstruction, annotators mark coplanar, connected, or occluding regions with colored scribbles instead of performing more complex tasks.Konyushkova et al. additionally model uncertainty from geometric constraints alongside feature-based uncertainty.
  • Batch active learning: Ordering 4 images yields 6 ordered pairs at roughly the cost of ordering 2 images, enabling faster relative-attribute model learning than labeling pairs explicitly.The full ordering provides six implicit pairwise comparisons while collecting annotations more efficiently.
  • Live active learning: Live active learning can learn an object detector from only a category name by combining crowd annotations with scalable active selection.This setting moves beyond constrained sandbox demonstrations toward independently learning detectors for specified categories.

4.2 Interactive annotation

Interactive annotation combines human feedback with computer vision models or contextual information to reduce costly labeling and improve performance on difficult tasks. These systems can also enable crowd workers to complete complex annotations that would otherwise require extensive training.

  • Object annotation: Interactive annotation proposes object hypotheses for rapid human verification, reducing the cost and time of exhaustively labeling cluttered images.One framework iteratively generates hypotheses, incorporates user corrections, and updates the detector.
  • Segmentation: Active segmentation selects images likely to improve propagation, while models choose faster annotation modalities such as bounding boxes or sloppy contours when sufficient.Segmentation-quality prediction helps determine whether computer output is adequate or human feedback is needed.
  • Image retrieval: Interactive retrieval engages users in relative attribute questions so the system can identify the image they are seeking.The interaction is framed as a visual relative 20-questions game.
  • Fine-grained recognition: Attribute-based human feedback combined with computer vision classifiers lets workers perform fine-grained recognition by answering simpler visual questions.Workers can answer questions about bird attributes even when they cannot directly identify the bird type.
  • Overall impact: Interactive methods enable crowd workers to perform complex image annotation that would otherwise be impossible without extensive training.They can support recognition tasks that are challenging for non-domain experts.

Conclusions

The survey summarizes crowdsourced annotation tasks, efficiency strategies, and methods for evaluating annotation quality and annotator skill in computer vision. It also identifies unresolved challenges involving human factors and capturing human knowledge beyond fixed labels.

  • Contributions: The survey covers computer vision tasks benefiting from crowdsourcing, efficient data-collection strategies, and evaluation of annotation quality and annotator skill.Crowdsourcing distributes labeling among non-expert annotators to obtain massive datasets inexpensively.
  • Limitations: Prior literature offers a starting point for designing data-collection strategies and reducing noisy annotations through gold-standard questions and label reconciliation.These approaches help prevent noisy data from entering the annotation pool.
  • Limitations: Computer vision crowdsourcing has largely not accounted for human factors such as attention, fatigue, and miscommunication that affect data quality.The passage identifies these factors as an unresolved limitation, with only small exceptions in existing methods.
  • Future directions: Researchers have not yet found a way to capture vast human knowledge without reducing it to a set of labels.The survey anticipates closer integration of computer vision with knowledge representation and language processing and generation.
Loading 1611.02145v1…