PaperlayerView exact revision on arXiv

Source-linked AI summary

Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective

Steven Euijong Whang, Yuji Roh, Hwanjun Song, Jae-Gil Lee

arXiv:2112.06409v3cs.LG

TL;DR

Deep learning increasingly depends on large amounts of data, yet real-world datasets are often small, dirty, biased, or poisoned, making data preparation a major part of machine learning. This survey synthesizes data collection and quality techniques for deep learning, including validation, cleaning, sanitization, integration, robust training, and fairness, and concludes that data-centric AI requires coordination across these areas.

  • Problem

    Deep learning needs substantial data, but preparing real-world data is costly and datasets may be small, dirty, biased, or poisoned.

  • Method

    The survey organizes research on data acquisition, labeling, data improvement, validation, cleaning, sanitization, integration, robust training, noisy or missing labels, and fairness.

  • Results

    The survey finds that data-centric AI spans complementary techniques from data management, machine learning, security, and fairness, rather than a single solvable subproblem.

  • Takeaways & Limitations

    Data-centric AI ultimately needs holistic orchestration across data collection, quality, robustness, and fairness, with the data management community positioned to contribute.

  • Takeaways & Limitations

    Methods for noisy labels have trade-offs: loss correction can accumulate false corrections, while sample selection can ignore useful hard samples.

Abstract

from arXiv · show

Data-centric AI is at the center of a fundamental shift in software engineering where machine learning becomes the new software, powered by big data and computing infrastructure. Here software engineering needs to be re-thought where data becomes a first-class citizen on par with code. One striking observation is that a significant portion of the machine learning process is spent on data preparation. Without good data, even the best machine learning algorithms cannot perform well. As a result, data-centric AI practices are now becoming mainstream. Unfortunately, many datasets in the real world are small, dirty, biased, and even poisoned. In this survey, we study the research landscape for data collection and data quality primarily for deep learning applications. Data collection is important because there is lesser need for feature engineering for recent deep learning approaches, but instead more need for large amounts of data. For data quality, we study data validation, cleaning, and integration techniques. Even if the data cannot be fully cleaned, we can still cope with imperfect data during model training using robust model training techniques. In addition, while bias and fairness have been less studied in traditional data management research, these issues become essential topics in modern machine learning applications. We thus study fairness measures and unfairness mitigation techniques that can be applied before, during, or after model training. We believe that the data management community is well poised to solve these problems.

1 Overview

Deep learning shifts software engineering toward data-centric AI, where data preparation and quality are central across the machine learning process. The survey connects techniques for collection, quality, robustness, and fairness into a broad workflow.

  • Motivation: Data preparation takes 45% or even 80–90% of the entire machine learning process.The reported estimates cover collecting, cleaning, and preparing data for training.
  • Motivation: Data-centric AI prioritizes improving data preprocessing for better model accuracy rather than improving model training algorithms.
  • Data Challenges: Deep learning requires large amounts of training data because feature engineering is less of a concern than in traditional machine learning.
  • Data Challenges: Real-world data may be unavailable, dirty, or poisoned, and not all data-cleaning techniques directly benefit deep learning accuracy.
  • Survey Scope: Data-centric AI requires orchestration across related areas rather than solving only one data issue, so the survey connects them at a high level.Table 1 provides a taxonomy, while Figure 2 shows how techniques connect within one workflow.
  • Survey Scope: The survey covers data collection, validation, cleaning, integration, robust training, and fair training across the deep learning process.Its paper-selection criteria emphasize breadth, prominent work, and emerging techniques.

2 Data Collection

The survey organizes data collection into acquisition, labeling, and improvement of existing data and models. It keeps this coverage brief while adding updates from a tutorial.

  • Data Collection: Data collection comprises data acquisition, data labeling, and improving existing data and models.The three approaches address obtaining datasets, adding annotations, and improving available resources.
  • Data Collection: Data labeling includes semi-supervised learning, crowdsourcing, and weak supervision because labeling is expensive.

2.1 Data Acquisition

Data acquisition addresses insufficient training data through discovery, augmentation, or generation. The survey presents systems for finding datasets and methods for creating realistic or domain-specific data.

  • Data Acquisition: Data acquisition includes data discovery, data augmentation, and data generation when suitable training data are insufficient.
  • Data Discovery: Goods indexes and searches tens of billions of datasets in Google’s data lake using metadata such as size, provenance, and schema.It also provides search, monitoring, and dataset annotations.
  • Data Discovery: Juneau finds related tables by comparing records, schemas, and provenance, supporting efficient joins or unions in data lakes.It is built on the Jupyter Notebook data science platform.
  • Data Augmentation: GANs generate realistic fake data from random noise using adversarially trained generator and discriminator components.GANs cannot generate data completely different from existing data, so domain-expert transformations can complement them.
  • Data Augmentation: Mixup augments data by mixing pairs of examples from different classes, effectively regularizing predictions for in-between training points.Model patching augments specific class subgroups so accuracy is similar across subgroups.
  • Data Generation: Crowdsourcing platforms and domain-specific simulators or generators can create new data, including demographic face images, mobility data, and driving data.

2.2 Data Labeling

Data labeling uses existing labels, human workers, or weak supervision to produce training annotations. Active learning reduces crowdsourcing cost by prioritizing uncertain examples likely to improve model accuracy.

  • Data Labeling: Labeling techniques use existing labels or manually or automatically generate labels when examples lack annotations.
  • Semi-Supervised Labeling: Semi-supervised learning trains on available labeled data and adds high-confidence predictions for unlabeled examples.Self-training is presented as the simplest form.
  • Manual Labeling: Crowdsourcing services provide workers, interfaces, and tools for labeling data when no labels exist initially.
  • Manual Labeling: Active learning reduces crowdsourcing cost by asking labelers to annotate uncertain examples expected to improve model accuracy most.Selection methods identify uncertain examples or use decision-theoretic analyses.
  • Weak Supervision: Weak supervision generates imperfect labels automatically at scale, using methods such as crowdsourcing, distant supervision, and data programming.Larger label volume may compensate for lower label quality.
  • Weak Supervision: Snorkel combines user-provided labeling functions probabilistically, then uses generated labels and unlabeled data to train a discriminative model.

2.3 Improving Existing Data

When relevant external datasets are unavailable or additional data no longer improves accuracy because of low quality, improving existing data may be preferable. Relabeling is one approach for improving existing data.

  • Improving existing data is useful when the target application is novel or non-trivial and relevant external datasets do not exist.
  • Improving existing data may be preferable when collecting more data no longer benefits model accuracy because the available data is low quality.
  • Relabeling is an effective approach for improving existing data labels.

3 Data Validation, Cleaning, and Integration

Data validation detects errors, while cleaning and sanitization address different forms of imperfect data. For deep learning, these techniques increasingly target model accuracy and robustness rather than only conventional data correctness.

  • Data Validation: Training data commonly contains errors, motivating validation components that detect problems before model training.
  • Data Validation: Data visualization supports rapid human sanity checks, while automatic visualization generation can identify statistically interesting views.
  • Data Validation: TFDV generates a schema from previous datasets, validates future datasets against feature statistics, and alerts users to anomalies.
  • Data Cleaning: Traditional cleaning repairs integrity-constraint violations, but CleanML finds that cleaning does not necessarily improve downstream accuracy and can sometimes harm models.Cleaning methods and parameters must be adapted to the noise type and tuned carefully.
  • Data Cleaning: ActiveClean iteratively samples likely-dirty data, cleans selected examples, and updates the model because cleaning the entire dataset is presumed expensive.
  • Data Sanitization: Data poisoning changes a fraction of training data to alter model behavior, and sanitization uses outlier detection to identify and discard poisoned points.Carefully designed attacks remain difficult to detect, and no evaluated technique adequately defends against them.

4 Robust Model Training

Robust model training addresses dirty, missing, noisy, and poisoned data when clean information cannot be fully recovered. The survey covers defenses based on architectures, losses, input data, regularization, and approaches for informative missingness and missing labels.

  • Motivation: Real-world datasets remain dirty or erroneous after cleaning, with flaws involving noisy or missing features and labels.These data-quality problems motivate robust training during model learning.
  • Data Poisoning: Poisoning attacks contaminate training data with maliciously designed examples intended to deceive the training procedure.Features, labels, or both may be targeted through adversarial perturbations or label changes.
  • Missing Features: GRU-D exploits informative missingness through masking and time intervals, applying decay terms to inputs and recurrent states during joint training.The model uses missing patterns alongside temporal observations to improve prediction results.
  • Limitations: Loss correction can accumulate false corrections, whereas sample selection may discard useful hard samples classified as unclean.These limitations expose different trade-offs in robustness to noisy labels.
  • Noisy Labels: DivideMix models per-sample loss distributions with a GMM and alternates mostly clean labeled data with mostly noisy unlabeled data between two networks.Its semi-supervised training can reduce clean-sample loss while keeping most noisy-sample losses larger under 40% asymmetric CIFAR-10 noise.

5 Fair Model Training

Fair model training addresses biased data through multiple fairness definitions and mitigation stages, while recognizing that fairness and robustness interact without being interchangeable.

  • Fairness Measures: Fairness cannot be captured by one notion; representative criteria include independence, separation, sufficiency, individual fairness, and causality fairness.Independence, separation, and sufficiency correspond to demographic parity, equalized odds, and predictive parity, respectively.
  • Fairness Measures: No two of independence, separation, and sufficiency can be fully satisfied together.The paper identifies this as an impossibility result for the three major statistical criteria.
  • Unfairness Mitigation: FairBatch improves fairness by adjusting sensitive-group ratios within training batches when an intermediate model shows poor fairness.This reweights batch composition rather than requiring extensive modification of model training.
  • Convergence with Robustness Techniques: Fairness and robustness techniques are converging because both address data flaws, but robust cleaning can worsen bias by disproportionately removing underrepresented-group data.The paper distinguishes fairness-oriented, robust-oriented, and equal-merger approaches.

6 Overall Findings and Future Directions

The survey finds that data collection, quality management, robust training, and fair training are interconnected areas that increasingly need to be used together. It identifies future convergence across these research directions while noting that its trend list is not exhaustive.

  • Overall Findings: The survey covers data collection, validation, cleaning, sanitization, integration, robust training, and fair training across multiple research communities.Validation uses visualizations and schemas; integration is especially challenging for multimodal data.
  • Convergence Directions: Fairness and robustness mitigation can be categorized as fair-robust, robust-fair, or equal-merger techniques.The paper presents these categories as convergence directions for addressing data flaws and training behavior.
  • Limitations: The survey’s proposed list of converging research trends is not exhaustive.This is the paper’s stated scope boundary for its future-direction overview.
  • Future Directions: Data cleaning and robust training should continue integrating because data cleaning is considered less effective than robust training.The survey describes recent cleaning methods as increasingly machine-learning oriented.
  • Future Directions: Data collection should increasingly consider fairness and robustness rather than assuming the input data is already given.The authors identify data collection for accurate machine learning as an active data-management research direction.
  • Concluding Remarks: The authors expect data techniques and robust and fair training techniques to converge as data-centric AI matures.They also suggest that privacy and explainability should eventually become part of data-centric AI.
Loading 2112.06409v3…