PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-centric Artificial Intelligence: A Survey

Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, Xia Hu

arXiv:2303.10158v3cs.LGcs.AIcs.DB

TL;DR

AI progress depends heavily on abundant, high-quality data, motivating data-centric AI as a complement to model-focused development. This survey synthesizes tasks and methods across the data lifecycle, organizing them by goals, automation, and human participation while identifying open directions. It concludes that cross-task automation and data-model co-design remain important areas for future work.

  • Problem

    AI research needs a systematic focus on data because major progress depends on abundant, high-quality data, while data can contain missing values, incorrect labels, and anomalies.

  • Method

    The survey organizes data-centric AI into training data development, inference data development, and data maintenance, and reviews methods through automation, collaboration, benchmark, and goal-driven perspectives.

  • Results

    The survey synthesizes data-centric AI research across tasks and domains, identifies current progress, and highlights cross-task automation and data-model co-design as open directions.

  • Takeaways & Limitations

    Data-centric AI is an interdisciplinary field spanning multiple data-lifecycle goals, with future work needed to unify automation across them and coordinate data with models.

  • Takeaways & Limitations

    Data quality assessment and improvement remain difficult because suitable metrics depend on domain knowledge, may not suffice alone, and improvement is laborious.

Abstract

from arXiv · show

Artificial Intelligence (AI) is making a profound impact in almost every domain. A vital enabler of its great success is the availability of abundant and high-quality data for building machine learning models. Recently, the role of data in AI has been significantly magnified, giving rise to the emerging concept of data-centric AI. The attention of researchers and practitioners has gradually shifted from advancing model design to enhancing the quality and quantity of the data. In this survey, we discuss the necessity of data-centric AI, followed by a holistic view of three general data-centric goals (training data development, inference data development, and data maintenance) and the representative methods. We also organize the existing literature from automation and collaboration perspectives, discuss the challenges, and tabulate the benchmarks for various tasks. We believe this is the first comprehensive survey that provides a global view of a spectrum of tasks across various stages of the data lifecycle. We hope it can help the readers efficiently grasp a broad picture of this field, and equip them with the techniques and further research ideas to systematically engineer data for building AI systems. A companion list of data-centric AI resources will be regularly updated on https://github.com/daochenzha/data-centric-AI

1 INTRODUCTION

Data-centric AI emerges from the central role of abundant, high-quality data in AI progress and shifts attention toward systematically improving data. The survey organizes this field around data development, automation, human participation, and future research directions.

  • Motivation: Abundant, high-quality data enable major AI breakthroughs across domains, including language processing, vision, healthcare, biology, and finance.Examples include ImageNet, annotated protein sequences for AlphaFold, and large text datasets for large language models.
  • Motivation: Data-centric AI complements model advancement by emphasizing systematic improvement of data quality and quantity while models are relatively more fixed.The model-centric paradigm can overlook missing values, incorrect labels, and anomalies in data.
  • Related directions: The field includes established methods such as data augmentation and feature selection alongside newer directions including data programming, algorithmic recourse, and prompt engineering.Research also explores automated data augmentation, automated pipeline discovery, and human-machine collaboration in data creation.
  • Survey scope: The survey provides a holistic, goal-driven view of data-centric AI, covering definitions, tasks, algorithms, challenges, benchmarks, automation, and human participation.Its research questions address necessary tasks, the significance of automation, essential human involvement, and current research progress.
  • Survey scope: Its central task taxonomy spans training data development, inference data development, and data maintenance across the data-centric AI lifecycle.The survey structure separately develops these three goals before discussing benchmarks, global perspectives, and future directions.

2 BACKGROUND OF DATA-CENTRIC AI

Data-centric AI shifts attention from fixed-data model advancement toward systematically improving data across the AI lifecycle. The survey organizes tasks by goals and methods by automation or human participation.

  • Need for Data-centric AI: Data-centric AI addresses data quality issues that model-centric AI can overlook, including missing values, incorrect labels, and anomalies.
  • Tasks in Data-centric AI: The survey organizes data-centric AI into training data development, inference data development, and data maintenance.
  • Tasks in Data-centric AI: Training data development covers collection, labeling, preparation, reduction, and augmentation.
  • Tasks in Data-centric AI: Inference data development creates evaluation sets or engineered inputs for granular model insights, capability triggering, or desired language-model predictions.
  • Automation and Human Participation: Methods are categorized by automation or collaboration, reflecting automated processing levels or degrees of human participation.
  • Overview: The survey’s contribution is a holistic, goal-driven organization of tasks, methods, benchmarks, challenges, and future directions.

3 TRAINING DATA DEVELOPMENT

Training data development creates and processes data for machine learning by encoding human intentions and making data suitable for learning. It includes data creation and data processing stages.

  • Training data development aims to create and process training data that supports machine learning models.
  • Data creation: Data creation encodes human intentions into datasets through data collection and data labeling.
  • Data processing: Data processing makes data suitable for learning through preparation, reduction, and augmentation.

3.1 Data Collection

Data collection gathers data from diverse sources and fundamentally determines dataset quality and quantity. Efficient strategies leverage existing datasets, but collection remains technically, legally, ethically, and logistically challenging.

  • Data collection gathers and acquires data from various sources, fundamentally determining data quality and quantity.
  • Domain knowledge is critical for collecting relevant, representative data and synthesizing realistic data.
  • Efficient Data Collection Strategies: Dataset discovery identifies useful datasets from a data lake using a human-specified query.
  • Efficient Data Collection Strategies: Data integration combines datasets by matching columns and transforming source values into a unified target dataset.
  • Efficient Data Collection Strategies: Raw data synthesis can create desirable patterns more efficiently than real-world collection when target phenomena are rare.
  • Challenges: Collection challenges include dataset misalignment, difficult synthesis, unavailable sources, and legal, ethical, logistical, privacy, and security constraints.

3.2 Data Labeling

Data labeling supplies the supervision needed for models to reflect human intentions, while strategies trade off label quality, quantity, human effort, and cost. The survey reviews manual, semi-supervised, active, weakly supervised, and externally supervised approaches alongside persistent noise and ethical challenges.

  • Data labeling enables algorithms to learn from labeled data and helps trained models reflect human intentions.
  • Researchers use crowdsourcing, semi-supervised labeling, active learning, data programming, and distant supervision to improve labeling efficiency.These strategies can also be combined as hybrid approaches.
  • Active learning: Active learning iteratively selects uncertain unlabeled samples for human annotation, using newly labeled data to choose subsequent queries.Existing work mainly differs in its query-selection strategy.
  • Data programming: Data programming infers labels from human-designed heuristic functions, enabling rapid generation of many labels with minimal or partial human participation.
  • Distant supervision: Distant supervision assigns labels from external sources, but automatically generated labels can be noisy.Relation extraction is a prominent application using external knowledge such as Freebase.
  • Labeling must balance label quality, quantity, and financial cost while addressing subjective judgments, labeling noise, privacy, and bias.

3.3 Data Preparation

Data preparation converts raw data into training-ready representations through cleaning, feature extraction, and feature transformation. Learning-based and collaborative methods can improve preparation, but dataset-specific search spaces and computational demands remain challenging.

  • Data preparation cleans and transforms raw data into a format appropriate for model training.The process traditionally requires substantial engineering effort and trial and error.
  • Raw data may contain noise, inconsistencies, irrelevant information, and sensitive attributes that can reduce generalizability or produce biased predictions.
  • The survey organizes preparation methods around data cleaning, feature extraction, and feature transformation.
  • Data cleaning: Data cleaning identifies and corrects dataset errors using heuristics, learning-based prediction, search-based optimization, or hybrid human-machine workflows.Examples include imputing missing values, estimating duplicates, correcting labeling errors, and presenting similar pairs to humans.
  • Feature extraction: Feature extraction derives relevant representations using domain knowledge for traditional models, whereas deep learning automatically extracts features.Image features may include color, texture, and intensity; time-series features may be temporal, statistical, or spectral.
  • Feature transformation: Feature transformation converts original features into new representations through operations such as normalization, standardization, logarithms, and polynomial transformations.Reinforcement learning can search transformation graphs using model feedback.
  • Preparation remains difficult because datasets differ substantially, and automated search requires an appropriate search space and non-trivial time.

3.4 Data Reduction

Data reduction seeks to simplify datasets while preserving essential information, either by reducing features or samples. Its methods can improve efficiency and interpretability, but information loss, computational cost, and amplified bias remain concerns.

  • Data reduction decreases dataset complexity while retaining essential information through feature-size or sample-size reduction.
  • Reducing samples can ease memory and computation constraints and alleviate class imbalance, while reducing features can remove irrelevant or redundant information.
  • Data reduction can help models focus on essential information, enhancing accuracy, efficiency, and interpretability.
  • Feature reduction: Feature selection chooses task-relevant subsets using filter, wrapper, or embedded methods with different relationships to feature interactions and model performance.
  • Feature reduction: Dimensionality reduction maps high-dimensional features into lower-dimensional spaces while preserving representative information through linear or nonlinear techniques.Principal Component Analysis is a common linear method based on orthogonal combinations of original features.
  • Sample reduction: Instance selection retains representative samples using filter or wrapper methods, including strategies that select boundary instances or optimize model performance.
  • Reduction methods face information-loss and computational challenges, and may amplify bias by selecting features associated with protected attributes.Fairness-aware data reduction remains under-explored.

3.5 Data Augmentation

Data augmentation enlarges and diversifies datasets by modifying existing samples or synthesizing new ones, supporting accuracy, generalization, robustness, and class balance. The survey covers manipulation, generative synthesis, and imbalance-focused methods while noting that suitable strategies depend on the data and task.

  • Data augmentation increases dataset size and diversity by artificially creating variations of existing data.It can complement data reduction by removing redundancy while increasing diversity.
  • Augmentation exposes models to more training examples, improving accuracy, generalization, and robustness when data is limited or classes are imbalanced.The survey highlights expensive, well-annotated medical data as one motivating setting.
  • Basic manipulation: Basic manipulation creates augmented samples through modifications such as scaling, rotation, flipping, blurring, or interpolation with Mixup.
  • Augmentation data synthesis: Generative augmentation learns the existing data distribution to synthesize new samples, commonly using a generator and discriminator in GAN-based methods.
  • Methods tailored for class imbalance: For class imbalance, SMOTE interpolates between minority instances and neighbors, while ADASYN emphasizes difficult points and AutoSMOTE searches oversampling strategies with reinforcement learning.
  • No single augmentation strategy suits every scenario because optimal methods vary across data types and even across datasets sharing a data type.Vanilla Mixup, for example, cannot be directly applied to irregular graph data.

3.6 Pipeline Search

Pipeline search jointly explores data-processing and modeling choices because pipeline steps can interact, but its search cost grows rapidly with pipeline complexity.

  • Pipeline steps can interact, so optimizing tasks independently may miss the best overall pipeline configuration.The preferred augmentation strategy may depend on the selected features.
  • AutoSklearn searches preprocessing modules, models, and associated hyperparameters together to optimize validation performance.Its preprocessing search space remains very small.
  • Pipeline search faces high computational overhead because repeated module-combination trials produce exponential search-space growth as modules increase.More efficient search strategies are required for broader real-world application.

4 INFERENCE DATA DEVELOPMENT

Inference data development creates evaluation or input data that reveals model behavior within and beyond the training distribution. The survey covers fine-grained subgroup evaluation, recourse, adversarial samples, and distribution shifts, alongside their construction challenges.

  • Inference data development: Inference data development creates novel evaluation sets for granular model insights or engineered inputs that trigger specific capabilities.It includes in-distribution evaluation, out-of-distribution evaluation, and prompt engineering.
  • In-distribution evaluation: In-distribution evaluation can expose failures on specific sub-populations and support inspection of decision boundaries and model ethics.These concerns are especially relevant in high-stakes or risky applications.
  • In-distribution evaluation: Data slicing partitions datasets into sub-populations, while automated methods such as SliceFinder search for interpretable slices with poor model performance.SliceFinder uses tree-based and lattice-based search methods.
  • In-distribution evaluation: Algorithmic recourse generates hypothetical samples that flip decisions toward preferred outcomes, helping users understand model decision boundaries.Methods vary between white-box and black-box access requirements.
  • In-distribution evaluation: Constructing in-distribution evaluation sets is difficult because possible data subsets expand exponentially and finding closest recourse can require substantial effort.The challenge concerns identifying targeted samples effectively and efficiently.
  • Out-of-distribution evaluation: Out-of-distribution evaluation uses samples differing from training data to assess generalization to unexpected scenarios and model transferability.Distribution shifts can be natural or synthetically generated through covariate, label, or general shifts.
  • Out-of-distribution evaluation: Adversarial samples intentionally modify inputs to cause incorrect predictions and can be used to examine model robustness.Automated perturbation methods include white-box, physical-world, black-box, and poisoning attacks.
  • Out-of-distribution evaluation: Out-of-distribution generation is challenged by poor training-data representativeness, mode collapse, and difficulty evaluating generated-data quality.Mode collapse can limit sample diversity and omit parts of the target distribution.

5 DATA MAINTENANCE

Data maintenance keeps data reliable and available as production data and processing infrastructure change continuously. It spans data understanding, quality assurance, and storage and retrieval, with visualization and valuation supporting human analysis.

  • Data maintenance: Data maintenance is needed because production data is continuously updated while AI systems require a reliable and instant data supply.The survey organizes maintenance into data understanding, quality assurance, and data storage and retrieval.
  • Data understanding: Data understanding addresses the difficulty of analyzing large, complex, and high-dimensional real-world data.Visualization helps summarize and present such data in more accessible forms.
  • Data visualization: Visual summarization condenses raw data into graphical diagrams, but choosing a faithful and user-friendly format often requires human participation.Linear charts may perform better for many analytical tasks, whereas radial charts can be more natural or memorable.
  • Data visualization: Dimensionality reduction and clustering organize complex data into two- or three-dimensional, grouped, and often color-coded visualizations.
  • Data visualization: Visualization recommendation ranks chart candidates using programmatic rules or learning-based methods such as DeepEye.DeepEye optimizes NDCG based on the match between data statistics and chart choice.
  • Data understanding: Data valuation estimates each data point’s contribution, including through Shapley values, to inform stakeholders and data-market transactions.
  • Data quality assurance: Maintaining data quality requires continuous monitoring because data and processing infrastructure undergo frequent updates.Quality assurance covers quality assessment and improvement.
  • Data quality assurance: Quality assessment uses objective metrics or human-centered subjective evaluations, while quality improvement applies programmatic and machine-learning-based strategies.Objective metrics include accuracy, timeliness, consistency, and completeness; subjective metrics include trustworthiness, understandability, and accessibility.

6 DATA BENCHMARK

The survey compiles benchmarks that evaluate data rather than models, then analyzes their coverage across domains, modalities, and data-centric goals. It finds broad interdisciplinary activity but uneven task coverage.

  • Benchmark collection: Data benchmarks evaluate techniques aimed at achieving data excellence using standard and agreed-upon metrics, rather than evaluating models.
  • Benchmark collection: The collection process searched benchmark literature, manually filtered out model-focused evaluations, and screened papers by citations and venue reputation.
  • Collected benchmarks: 36 benchmarks were collected, including 23 with open-source code.
  • Collected benchmarks: No benchmark was found for generating distribution-shift samples because existing related benchmarks mainly assess model performance rather than data-generation methods.
  • Meta-analysis: The benchmarks span AI, databases, computer graphics, human-computer interaction, biomedicine, security, and healthcare, indicating interdisciplinary data-centric AI research.
  • Meta-analysis: Tabular data is the most frequently benchmarked modality, followed by image, time-series, text, audio, and graph data.
  • Meta-analysis: Training data development has more benchmarks than inference data development and data maintenance.

7 DISCUSSION AND FUTURE DIRECTION

The survey frames data-centric AI as a broad field spanning training-data development, inference-data development, and data maintenance, with automation and human participation both central. It connects the field to foundation models and reinforcement learning, then identifies cross-task automation, data-model co-design, debiasing, and unified benchmarks as future directions.

  • Data-centric AI includes training-data development, inference-data development, and data maintenance across tasks such as cleaning, labeling, evaluation-data generation, prompt tuning, and data debugging.
  • Automation is needed to manage rapidly increasing data, and surveyed methods span programmatic, learning-based, and pipeline automation levels.
  • The survey cannot cover every aspect of data-centric AI despite broadly examining its tasks, techniques, needs, and challenges.
  • Foundation models and reinforcement learning both blur data-model boundaries while continuing to depend on or generate data within their learning processes.
  • Future directions: Future research should pursue unified automation across data-centric goals and co-design data with models as their boundaries increasingly blur.
  • Future directions: More unified data benchmarks are needed because existing efforts such as DataPerf support only a limited number of tasks.

8 CONCLUSION

The survey synthesizes data-centric AI through its definition, goal-driven taxonomy, techniques, automation and collaboration perspectives, benchmarks, and global discussion. It concludes that data will become increasingly important in building AI systems, while substantial challenges remain.

  • The survey reviews data-centric AI techniques across purposes and analyzes them through automation and collaboration perspectives.
  • It collects benchmarks from different domains and analyzes them at a meta-level while discussing blurred boundaries between data and model.
  • Data will play an increasingly important role in building AI systems, but numerous challenges still need to be addressed.
Loading 2303.10158v3…