Source-linked AI summary
Data Valuation using Reinforcement Learning
Jinsung Yoon, Sercan O. Arik, Tomas Pfister
TL;DR
Data valuation is difficult at the individual-sample level, especially for large datasets and complex models. DVRL jointly learns data values and a target predictor using reinforcement signals from a small validation set, and it outperforms competing methods across diverse applications, while requiring target-distribution validation data.
Problem
Evaluating the value of individual training samples is difficult, particularly for complex models trained on large-scale datasets, although data quality and usefulness vary across samples.
Method
DVRL is a meta learning framework that jointly optimizes data valuation and the target predictor, training a neural data value estimator with rewards from a small validation set.
Results
DVRL significantly outperforms competing data valuation methods across image, tabular, and language datasets and diverse applications.
Takeaways & Limitations
DVRL produces efficient, high-quality data-value rankings useful for domain adaptation, corrupted sample discovery, and robust learning.
Takeaways & Limitations
DVRL requires a validation dataset from the target distribution, potentially with noise-free labels, domain matching, or high quality, which can be costly to acquire.
Abstract
from arXiv · showhide
Quantifying the value of data is a fundamental problem in machine learning. Data valuation has multiple important use cases: (1) building insights about the learning task, (2) domain adaptation, (3) corrupted sample discovery, and (4) robust learning. To adaptively learn data values jointly with the target task predictor model, we propose a meta learning framework which we name Data Valuation using Reinforcement Learning (DVRL). We employ a data value estimator (modeled by a deep neural network) to learn how likely each datum is used in training of the predictor model. We train the data value estimator using a reinforcement signal of the reward obtained on a small validation set that reflects performance on the target task. We demonstrate that DVRL yields superior data value estimates compared to alternative methods across different types of datasets and in a diverse set of application scenarios. The corrupted sample discovery performance of DVRL is close to optimal in many regimes (i.e. as if the noisy samples were known apriori), and for domain adaptation and robust learning DVRL significantly outperforms state-of-the-art by 14.6% and 10.8%, respectively.
1 INTRODUCTION
Data valuation addresses the difficulty of identifying useful, high-quality samples when data collection and labeling are costly, distributions differ, and training data may contain noise. DVRL jointly learns data values and a target predictor through meta learning, outperforming competing methods across diverse applications and datasets.
- Large, high-quality datasets improve machine-learning models, but collecting and correctly labeling them is costly and challenging.
- Data valuation can prioritize samples affected by incorrect labels, distribution shifts, noise, or differing usefulness for the target task.
- Data valuation supports data collection, value-based pricing, and cheaper construction of large-scale training datasets by filtering less valuable data.
- DVRL jointly optimizes a meta-learning data valuation framework with the target task predictor model.
- DVRL significantly outperforms competing methods across multiple use cases and image, tabular, and language datasets.
- DVRL is scalable to large datasets and complex models, with computational complexity not directly dependent on training-set size.
2 RELATED WORK
Prior data-valuation methods rely on costly leave-one-out, influence-function, or Shapley-value computations, while related meta-learning methods adapt sample weights for specific applications. These approaches face computational, approximation, or application-specific limitations described in the related work.
- Leave-one-out data valuation measures the performance difference caused by removing a specific training sample.Its cost scales linearly with the number of training samples, and it can undervalue equivalent samples.
- Influence Function approximates leave-one-out using loss gradients and small perturbations, but neural-network Hessian computations are prohibitively expensive.Hessian approximations can reduce cost but generally introduce performance limitations.
- Data Shapley assigns values from marginal performance improvements across all possible training subsets.Exact computation is exponential in the number of samples, motivating Monte Carlo and gradient-based approximations.
- Related meta-learning methods adapt sample weights or identify corrupted data for robust learning, domain adaptation, and corrupted sample discovery.Examples use output correlations, prediction uncertainty, synthetic noise, low-loss selection, validation performance, or label-correction strategies.
3 PROPOSED METHOD
DVRL jointly learns a predictor and a data value estimator, using stochastic sample selection and validation performance to adapt sample values during training.
- The estimator maps each training sample to a selection probability, and selected samples alone train the predictor model.The selection vector is binary, with si indicating whether sample (xi, yi) is included.
- Predictor loss on a small validation set is compared with a moving average of previous losses to determine the reward and update hφ.The moving average baseline is used to improve training stability.
- DVRL comprises a target task predictor fθ and a data value estimator hφ that outputs sample weights determining selection likelihood.The predictor minimizes a weighted training loss, while hφ is trained to assign values to training samples.
- DVRL uses reinforcement learning because stochastic sample selection is non-differentiable, while the predictor can use standard stochastic gradient descent.The reinforcement signal updates the estimator based on the reward from predictor performance.
- DVRL’s training-time overhead is twice conventional training rather than exponential in dataset size, and it scales to CIFAR-100 with complex models.The required iterations and per-iteration complexity dominate runtime; pre-trained predictor initialization can reduce overhead.
4 EXPERIMENTS
DVRL is evaluated across diverse datasets and data-valuation use cases, including sample removal, corrupted-sample discovery, robust learning, and domain adaptation. It generally identifies valuable training samples effectively while requiring a target-distribution validation set.
- Experimental setup: DVRL is evaluated against Random, LOO, Data Shapley, and, in some experiments, Learning to Reweight, MentorNet, and Influence Function across 12 tabular, image, and language datasets.The predictor models include LightGBM, XGBoost, multilayer perceptrons, and Inception-v3, supporting a model-agnostic evaluation.
- Removing high/low value samples: Removing 10%-20% of the highest-valued samples significantly degrades performance, while removing low-valued samples causes the slowest degradation in most cases.DVRL shows stronger separation between high- and low-value samples than LOO and Data Shapley.
- Corrupted sample discovery: DVRL consistently outperforms Data Shapley, LOO, and Influence Function for corrupted-sample discovery with 20% noisy labels across six datasets.Its discovery trend is close to optimal on Adult, CIFAR-10, and Flower.
- Robust learning with noisy labels: DVRL outperforms other robust-learning methods in all reported cases, losing 5.06% and 7.99% accuracy under Uniform noise on CIFAR-10 and CIFAR-100, respectively.Learning to Reweight loses 7.16% and 13.21% in the corresponding comparisons.
- Domain adaptation: DVRL improves prediction in every Rossmann domain-adaptation setting, with the largest gains under substantial domain mismatch.With XGBoost for store type B in Train on Rest, DVRL reduces error by more than 50% relative to Baseline; it also prioritizes samples from the target store type.
- Validation-set requirements: DVRL achieves reasonable corrupted-sample valuation with 100-400 validation samples, while 10 samples suffice for reasonable quality on Adult.The method requires a validation dataset from the target distribution, which may need noise-free labels, domain matching, or high quality.
5 CONCLUSIONS
DVRL adaptively learns data values jointly with a target predictor, using a validation-set reward to train a data value estimator. At inference, fixed estimator and predictor separately value samples and make predictions.
- DVRL jointly learns data values and a target task predictor through a meta learning framework.
- A neural data value estimator predicts how likely each datum is used to train the predictor.
- A small validation set representing target-task performance supplies the reinforcement-learning reward for training the estimator.
- At inference, the fixed data value estimator maps a sample set to corresponding data values, while the fixed predictor maps a sample to its prediction.
- DVRL training on CIFAR-100 with ResNet-32 takes less than 8 hours after pre-training, or less than 12 hours including pre-training, on one Tesla V100 GPU.
C.1 ADDITIONAL RESULTS ON ROBUST LEARNING WITH NOISY LABELS
DVRL is evaluated for robust learning with noisy labels across three image datasets and noise levels up to 50%. It maintains high prediction performance despite substantial label noise.
- The noisy-label evaluation uses CIFAR-10, Flower, and HAM 10000 with noise levels ranging from 0% to 50%.
- DVRL remains highly robust with noisy labels at noise ratios up to 50%.
- Noisy labels significantly degrade prediction performance relative to training only on clean labels.
- DVRL can sometimes outperform the Clean case even at 0% noise by prioritizing some clean samples over others.
C.2 ADDITIONAL RESULTS ON ROBUST LEARNING WITH NOISY FEATURES
DVRL is tested for robustness to noisy input features on tabular datasets, alongside sample-removal and corrupted-sample discovery analyses under 20% label noise. The supplied passages report robustness to feature noise but provide no numeric table results.
- DVRL is robust to noisy input features on the Adult and Blog tabular datasets.
- Feature-noise performance gains over Baseline are higher at larger Gaussian-noise levels.
- Table 5 reports testing accuracy under noisy features, with σ denoting the standard deviation of added Gaussian noise.
- Under 20% label noise, Figure 7 compares prediction performance after removing the most versus least important samples by estimated data value.
- Under 20% noisy labels, Figure 8 compares corrupted-sample discovery across Adult, Fashion-MNIST, and Flower against Optimal and Random references.
D LEARNING CURVES OF DVRL
DVRL learning curves compare validation log loss with and without DVRL during estimator training on six datasets with 20% label noise.
- The curves cover Adult, Blog, Fashion-MNIST, Flower, HAM 10000, and CIFAR-10.
- Figure 9 plots validation log loss against data value estimator training iterations for six datasets with 20% noisy labels.
- Orange denotes validation log loss without DVRL, while blue denotes validation log loss with DVRL.
E CONFIDENCE INTERVALS OF DVRL PERFORMANCE ON CORRUPTED
The corrupted sample discovery experiments evaluate DVRL’s estimated data values under 20% label noise across six datasets, reporting 95% confidence intervals from 10 independent runs.
- The experiments assess corrupted sample discovery using data values estimated by DVRL.
- 20% label noise is evaluated on Adult, Blog, Fashion-MNIST, Flower, HAM 10000, and CIFAR-10.
- 95% confidence intervals are computed from 10 independent runs.
F ROSSMANN DATA STATISTICS & T-SNE ANALYSIS
This section presents Rossmann data statistics and t-SNE analyses of final-layer representations for each store type.
- Rossmann Data Statistics: Rossmann data statistics report the 25th, 50th, and 75th percentiles for sales and customers.The table uses # to represent the number.
- t-SNE Analysis: The t-SNE analysis examines final-layer representations for each store type in the Rossmann dataset.
G FURTHER ANALYSIS ON ROSSMANN DATASET IN Train on All SETTING
In the Rossmann Train on All setting, DVRL’s estimated data values prioritize training samples from the target store type.
- Further Analysis: Training samples are sorted in decreasing order according to their data values estimated by DVRL.The analysis illustrates how target-store samples are distributed across the sorted values.
- Further Analysis: DVRL prioritizes training samples originating from the same target store type.Figure 12 shows the distribution using data-value percentiles on the x-axis and sample ratios on the y-axis.