PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective

Yuji Roh, Geon Heo, Steven Euijong Whang

arXiv:1811.03402v2cs.LGstat.ML

TL;DR

Machine-learning data collection is a bottleneck because new applications may lack labeled data and deep learning may require larger training sets. The survey unifies machine learning and data management research across acquisition, labeling, and improving existing data or models, then provides selection guidelines and research challenges. Its conclusion emphasizes that these techniques complement one another, while systematic tradeoff evaluation and model-feedback analysis remain open challenges.

  • Problem

    Data collection is a critical machine-learning bottleneck because new applications often lack training data, while deep learning may require larger amounts of training data.

  • Method

    The survey synthesizes machine learning and data management research on data acquisition, data labeling, and improving existing data or models, and provides guidelines for choosing techniques.

  • Results

    The paper presents a research landscape and guidelines showing how data acquisition, labeling, and data or model improvement techniques complement one another.

  • Takeaways & Limitations

    Choosing among data-collection techniques requires considering their combinations and the application’s available data, labels, budget, and human effort.

  • Takeaways & Limitations

    The survey identifies unresolved tradeoffs between labeling accuracy and scalability and challenges in using model feedback to improve data as models become more complicated.

Abstract

from arXiv · show

Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough labeled data. Second, unlike traditional machine learning, deep learning techniques automatically generate features, which saves feature engineering costs, but in return may require larger amounts of labeled data. Interestingly, recent research in data collection comes not only from the machine learning, natural language, and computer vision communities, but also from the data management community due to the importance of handling large amounts of data. In this survey, we perform a comprehensive study of data collection from a data management point of view. Data collection largely consists of data acquisition, data labeling, and improvement of existing data or models. We provide a research landscape of these operations, provide guidelines on which technique to use when, and identify interesting research challenges. The integration of machine learning and data management for data collection is part of a larger trend of Big data and Artificial Intelligence (AI) integration and opens many opportunities for new research.

1 INTRODUCTION

Data collection has become a critical machine-learning bottleneck because new applications often lack training data, while deep learning can require more labeled data despite reducing feature-engineering effort. This survey bridges machine learning and data management perspectives to organize acquisition, labeling, data or model improvement, decision guidelines, and research challenges.

  • Data collection is a critical bottleneck because end-to-end machine-learning workflows spend substantial time preparing data, including collecting, cleaning, analyzing, visualizing, and engineering features.
  • New machine-learning applications often have little or no training data, making manual labeling potentially expensive and dependent on domain expertise.
  • Deep learning automatically generates features and reduces feature-engineering effort, but may require larger amounts of training data to perform well.
  • It organizes data collection into acquisition, labeling, and improving existing data or models, while noting that these operations can be combined.
  • The survey bridges machine learning and data management literature to help users select relevant data-collection techniques across disciplines.
  • The paper combines these techniques into decision guidelines and identifies future research challenges, including how to choose methods under application-specific conditions.

2 DATA ACQUISITION

Data acquisition finds or creates datasets for machine learning through discovery, augmentation, and generation. These approaches can also be combined, such as generating data while augmenting an existing dataset.

  • Data acquisition aims to find datasets that can be used to train machine learning models.
  • The literature organizes data acquisition into data discovery, data augmentation, and data generation.
  • Data discovery supports sharing or searching for datasets, augmentation enhances existing datasets with external data, and generation addresses cases without available external datasets.
  • Some acquisition techniques can be used together, including generating data while augmenting existing data.

2.1 Data Discovery

Data discovery covers publishing, sharing, searching, and extracting datasets across collaborative platforms, the Web, corporate data lakes, and structured online sources. Key concerns include discoverability, metadata, scalability, provenance, and expressive search.

  • Data discovery involves indexing and publishing generated data for sharing, followed by searching datasets for machine learning tasks.
  • Collaborative Analysis: Collaborative systems such as DataHub support dataset version control, search, cleaning, integration, visualization, and merging across analysts’ dataset versions.
  • Web: Web-based systems publish structured datasets for search engines, while marketplaces provide ways to buy, sell, or find public datasets.
  • Data Lake: Corporate data-lake systems address datasets that are internally generated but difficult for other teams to discover, often using post-hoc processing.
  • Data Lake: GOODS catalogs metadata from tens of billions of datasets and infers ownership, provenance, and content information through a central catalog.
  • Data Lake and Web: DATA CIVILIZER complements scalable keyword search by emphasizing dataset discovery through a linkage graph, while WebTables extracts structured tables from online HTML.

2.2 Data Augmentation

Data augmentation enriches existing datasets with external data, embeddings, latent semantics, missing information, or integrated tables. It can improve the information available for model training while avoiding unnecessary joins when they add little accuracy.

  • Data augmentation enhances existing datasets with external data, and data integration can serve this role when newly acquired datasets are added.
  • Latent Semantics: Embeddings represent words, entities, or knowledge as vectors and are widely used to increase features for machine learning.
  • Latent Semantics: Word2vec trains word vectors with Continuous Bag-of-Words or Skip-gram models, capturing linguistic context through prediction tasks.
  • Latent Semantics: Latent Dirichlet Allocation derives latent topics by explaining similarities in data through unobserved groups.
  • Missing Information: Systems such as Octopus and InfoGather fill missing values or features by finding and joining relevant Web-table information.
  • Data Integration: Hamlet and Hamlet++ predict when key-foreign key joins are necessary for accuracy and can safely avoid joins to reduce runtime.

2.3 Data Generation

Data generation creates datasets manually or automatically when existing data is insufficient, using crowdsourcing, synthetic-data methods, transformations, and domain-specific generation. The surveyed techniques span collection, preprocessing, quality control, and multiple data modalities.

  • Manual data generation commonly uses crowdsourcing, while automatic generation creates synthetic datasets; generation may also function as data augmentation.
  • Crowdsourcing: Crowdsourced data generation consists of gathering data and preprocessing it, with tasks categorized as procedural or declarative and under closed-world or open-world assumptions.
  • Crowdsourcing: Crowdsourcing systems gather records or partial structured data and preprocess it through curation, entity resolution, and dataset joining.
  • Crowdsourcing: Quality control must address interface design, worker skill differences, spammers, task decomposition, and aggregation.
  • Synthetic Data: Synthetic data generation includes probability-distribution sampling, GANs, human-defined policies, and application-specific techniques.
  • Synthetic Data: GANs train generative and discriminative networks so the generator produces candidates that resemble the true distribution, including synthetic images, videos, and relational records.
  • Data-Specific Generation: Data-specific methods generate or select synthetic images and text using motion assumptions, varied typography, noisy-image filtering, and paraphrasing.

3 DATA LABELING

Data labeling organizes techniques around whether labels already exist, are collected from crowds, or are generated imperfectly at scale. These approaches vary with the learning task and data type.

  • Use existing labels: Existing-label techniques exploit available labels to predict labels for remaining examples.
  • Crowd-based: Crowd-based techniques range from labeling individual examples to active learning and worker-support methods.
  • Weak labels: Weak-label methods generate imperfect labels in large quantities when producing fully correct labels is too expensive.
  • Further categorization: Labeling approaches can be classified by learning task, data type, and whether they use existing, crowd-based, or weak labels.Tasks include classification and regression, while data types include text, images, and graphs.

3.1 Utilizing existing labels

Semi-supervised labeling uses limited labeled data with larger unlabeled sets, through self-training, ensembles, multiple algorithms, multiple views, or graph structure. Surveyed methods achieve similar average accuracies across inductive and transductive settings, but some require restrictive assumptions.

  • Overview: Semi-supervised learning exploits a small labeled set and a larger unlabeled set to make predictions.Transductive learning predicts for available unlabeled data, whereas inductive learning predicts on unseen data.
  • Single-model methods: Self-training repeatedly adds the most confident unlabeled predictions to the labeled examples.
  • Multiple classifiers: Tri-training uses three models and adds unlabeled examples when the other two models agree on their predictions.Final unlabeled labels are assigned by majority voting requiring agreement from at least two models.
  • Multiple algorithms and views: Democratic Co-learning combines predictions from classifiers trained with different learning algorithms, while Co-training exchanges labels across conditionally independent feature views.
  • Comparison and limitations: Similar transductive or inductive accuracies were reported when these algorithms were averaged across 55 UCI and KEEL datasets.Co-training requires sufficient and redundant views, while Democratic Co-learning requires three different algorithms.
  • Graph-based methods: Graph-based label propagation infers unlabeled examples from similarity structure, with methods reducing per-node space complexity from O(m) to O(log m) or O(1).MAD-Sketch achieves O(log m) under certain conditions, while EXPANDER further reduces space complexity per node to O(1).

3.2 Crowd-based techniques

Crowd-based labeling uses human input, especially for selected or uncertain examples, and addresses worker interaction, quality control, and scalability. Active-learning variants extend from uncertainty and disagreement criteria to decision-theoretic and regression settings.

  • Active learning: Manual labeling is accurate but can be costly, motivating active learning to select informative unlabeled examples for human annotation.The key challenge is choosing which examples to ask about under a limited budget.
  • Selection criteria: Uncertainty sampling selects examples with the least confident predictions, while Query-by-Committee selects examples where model disagreement is greatest.
  • Selection criteria: Density weighting reduces the risk of selecting distributional outliers by favoring examples that are both uncertain or disputed and representative.
  • Decision-theoretic approaches: Decision-theoretic active learning chooses examples to optimize objectives such as estimated model accuracy or reduced generalization error.
  • Regression: Regression active learning uses prediction variance for uncertainty sampling or committee disagreement, with committee methods reported to work well under small model bias and resist overspecification.
  • Combined methods: Semi-supervised and active learning are complementary: one adds high-confidence predictions, while the other identifies low-confidence cases for labeling.Combined methods can select representative documents with committee disagreement and further exploit unlabeled data through EM.
  • Crowdsourcing: Crowdsourcing research addresses worker errors through interaction design, worker evaluation, bias reduction, and label aggregation.
  • User interaction: Revolt structures crowdsourcing as voting, explanation, and categorization to support post-hoc judgments of label decision boundaries.

3.3 Weak supervision

Weak supervision generates large quantities of imperfect labels when labeled data is scarce. Data programming combines labeling functions to produce weak labels for training discriminative models.

  • Weak supervision semi-automatically generates large quantities of labels that are less accurate than manual labels but sufficient for model training.
  • Data programming can improve accuracy and usability because many weak labels may outperform fewer manual labels, while labeling functions can be more intuitive than feature engineering.
  • Generative models reduce the influence of correlated or outlier labeling functions instead of treating every function equally.
  • Data programming converts annotations into labeling functions, combines them with a generative model or majority voting, and trains a discriminative model on generated weak labels.
  • If labeling functions are reasonably accurate, theoretical analysis shows that generative-model predictions become arbitrarily close to true labels.
  • Data programming primarily supports classification; regression requires real-valued labeling functions and a continuous label distribution.

4 USING EXISTING DATA AND MODELS

When acquiring or labeling new data is difficult or no longer improves accuracy, existing data and models can be improved through cleaning, noise-robust training, or transfer learning.

  • Improving existing data or model training is useful when applications are novel, additional data has diminishing benefits, or cleaning existing data is faster.
  • Data cleaning: Data-cleaning methods increasingly target machine-learning outcomes by selecting, repairing, or relabeling examples to improve model accuracy.
  • Data cleaning: ActiveClean prioritizes samples using expected accuracy improvement and dirtiness, while BoostClean ensembles detection-and-repair functions to maximize accuracy.
  • Robust against noise and bias: Robust-training methods use many noisy labels alongside fewer clean labels rather than discarding the noisy data.
  • Transfer learning: Transfer learning starts from a model trained for a source task and incrementally trains it for a related target task when data or training time is limited.
  • Transfer learning: Transfer-learning research considers what knowledge to transfer, how to transfer it, and when transfer is appropriate without negative effects.

5 PUTTING EVERYTHING TOGETHER

The survey’s workflow selects data-collection techniques according to available data, labels, budget, expertise, and model tolerance for weak supervision. These choices involve an accuracy–scalability tradeoff and application-specific effort assessment.

  • Data acquisition: With little initial data, practitioners can search existing datasets, generate data with equipment, or augment product metadata with external information.
  • Data labeling: Labeling choices depend on existing labels, budget, and available expertise: semi-supervised learning, crowd-based methods, active learning, or weak supervision may be appropriate.
  • Improving data and models: Existing noisy or biased labels can be improved with data cleaning, while existing models can support further training through transfer learning.
  • Practical considerations: Technique selection is difficult because data sufficiency, distribution learnability, human effort, and programming difficulty cannot be captured by simple yes-or-no questions.
  • Accuracy and scalability: Manual labeling is most accurate but least scalable, whereas semi-supervised learning scales best and data programming can scale beyond active learning when implementation costs are reasonable.

6 FUTURE RESEARCH CHALLENGES

Future research must improve data evaluation, model-informed data improvement, labeling tradeoffs, human collaboration, empirical comparison, and generalization across applications and task types.

  • Data Evaluation: Data evaluation must determine whether collected datasets have sufficient quantity and quality, while supporting dynamic dataset selection and automatic metadata extraction.Too many datasets can harm model training, and dataset quality may change over time.
  • Data Improvement: Model performance should guide data augmentation and cleaning, but increasingly complex models make data-level feedback harder to analyze.The survey identifies fairness and accuracy improvements as examples of model-informed data improvement.
  • Performance Tradeoff: Researchers need systematic ways to balance labeling accuracy against scalability, since more weak labels do not necessarily yield perfect model accuracy.Additional human labeling or transfer learning may become worthwhile after weak-label gains plateau.
  • Crowdsourcing: Crowdsourcing remains difficult because effective data collection depends on task and interface design, worker quality, pricing, and usable labeling-function programming.Libraries or templates could make collaborative programming of labeling functions easier.
  • Empirical comparison of techniques: Technique-selection flowcharts remain incomplete because effectiveness and human effort depend on application-specific data, expertise, and labeling requirements.The survey calls for more empirical research rather than relying solely on theoretical effort models.
  • Generalizing and integrating techniques: Many collection techniques are data-type or application-specific, with labeling research concentrated on classification rather than regression, motivating broader generalization.The survey highlights extending classification methods to regression and applying continuous extraction ideas across data sources.

7 CONCLUSION

The survey frames data acquisition, labeling, and improvement as increasingly important as machine learning expands, especially for neural networks. It synthesizes contributions across machine learning and data management, offers technique-selection guidance, and identifies challenges for broader Big data–AI integration.

  • Conclusion: Large-scale data acquisition and labeling are increasingly important as machine learning expands, especially for state-of-the-art neural networks.The conclusion emphasizes the growing importance of obtaining and labeling large amounts of data.
  • Conclusion: The survey maps how machine learning and data management techniques complement one another across data acquisition, data labeling, and improvement of existing data.It presents the integration of these communities as part of Big data and AI integration.
  • Conclusion: The paper provides guidelines for choosing techniques and identifies unresolved data collection challenges, with future integration expected across machine learning more broadly.The conclusion extends the Big data–AI integration outlook beyond data collection.
Loading 1811.03402v2…