Source-linked AI summary
The Effects of Data Quality on Machine Learning Performance on Tabular Data
Sedir Mohammed, Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Felix Naumann, Hazar Harmouch
TL;DR
The paper addresses limited systematic evidence on how data quality affects machine learning performance across tasks, algorithms, and pipeline stages. It experimentally varies six quality dimensions across 19 algorithms and three tasks under differing training and test quality scenarios. The results show that quality issues affect performance in distinct ways, while the evaluation remains bounded by deliberate scope limitations.
Problem
Incomplete, erroneous, or inappropriate data can undermine machine learning applications, motivating systematic evidence about how data quality affects model performance.
Method
The study varies six data quality dimensions across 19 algorithms and classification, regression, and clustering tasks under polluted training, test, or both-data scenarios.
Results
Data quality issues affect machine learning performance in distinct ways across tasks, algorithms, and training/test quality scenarios.
Takeaways & Limitations
The findings offer data scientists insights for developing machine learning pipelines in the presence of data quality issues.
Takeaways & Limitations
The evaluation uses synthetic pollution and individual quality dimensions, omits full hyperparameter optimization, and includes a basic unoptimized clustering autoencoder.
Abstract
from arXiv · showhide
Modern artificial intelligence (AI) applications require large quantities of training and test data. This need creates critical challenges not only concerning the availability of such data, but also regarding its quality. For example, incomplete, erroneous, or inappropriate training data can lead to unreliable models that produce ultimately poor decisions. Trustworthy AI applications require high-quality training and test data along many quality dimensions, such as accuracy, completeness, and consistency. We explore empirically the relationship between six data quality dimensions and the performance of 19 popular machine learning algorithms covering the tasks of classification, regression, and clustering, with the goal of explaining their performance in terms of data quality. Our experiments distinguish three scenarios based on the AI pipeline steps that were fed with polluted data: polluted training data, test data, or both. We conclude the paper with an extensive discussion of our observations.
1 DATA QUALITY AND AI
The paper studies how six data quality dimensions relate to the performance of 19 machine learning algorithms across classification, regression, and clustering. It models differing training and test data quality through systematic pollution experiments, while limiting the evaluation to synthetic pollution and individual dimensions.
- The study examines six data quality dimensions across 19 machine learning algorithms and three tasks: classification, regression, and clustering.The algorithms range from simple models to complex deep learning models.
- The study limits its scope to traditional and deep models, synthetic pollution, and individual data quality dimensions within a much larger experimental space.The authors identify broader model, error-type, and multi-dimension variations as future work.
- The experiments distinguish scenarios in which training data, test data, or both have lower quality.Training and testing data may originate from different sources or collection processes and therefore have different quality.
- For each quality dimension, parameterized polluters introduce corresponding errors into real-world datasets, sometimes using manually created clean baselines.Performance is reported for both original and baseline datasets when a clean version had to be created.
- The paper presents a comprehensive empirical study intended to understand how data quality relates to machine learning model performance and to provide practical insights for data scientists.Its contributions include systematic experimentation, realistic pipeline scenarios, learned lessons, and future research directions.
2 RELATED WORK
Prior work addresses selected data errors, validation, cleaning, and classification settings, but this paper positions itself as a systematic study spanning data quality dimensions, algorithms, tasks, and training/test scenarios. It also supplies explicit quality definitions and systematic pollution methods.
- Validation systems check traditional dimensions such as consistency and completeness, as well as machine-learning-dependent properties such as robustness and privacy.Some systems specifically validate serving data or test robustness under serving-data errors.
- Prior task-dependent work argues that data quality should be assessed in relation to the task, and this paper reports different effects of the same low-quality data across models.The cited framework evaluates task performance on datasets with systematically generated noise.
- The paper distinguishes its approach from data-cleaning benchmarks by systematically polluting data to study the effects of quality issues rather than primarily improving data quality.The resulting observations are intended to help experts judge insight robustness and prioritize problems.
- Earlier research studied label noise, missing values, data cleaning, validation, and selected classification settings, but coverage was fragmented across error types and tasks.The related work includes both machine learning and data management approaches.
- The authors present the first systematic empirical study covering training and test data quality across classification, regression, and clustering, with multiple algorithms and scenarios.They also define each quality dimension and provide a corresponding systematic pollution method.
3 DATA QUALITY DIMENSIONS AND DATA POLLUTION
The paper defines six data-quality dimensions and introduces systematic pollution procedures to study their effects across machine-learning pipelines. The dimensions cover representation, completeness, accuracy, uniqueness, and target-class balance, with explicit assumptions and scenario-specific constructions.
- Representation and uniqueness: Consistent representation requires semantically equivalent values to share one representation, while uniqueness requires that duplicate records be absent.For example, New York, NYC, and NY violate representation consistency, whereas duplicate rows violate uniqueness.
- Representation and uniqueness: Consistency is defined through the minimum replacements needed to transform a categorical feature into a consistent state, with numerical features assigned zero inconsistency.The measure also depends on how many alternative representations exist, so consistency cannot generally be computed as one minus inconsistency.
- Pollution procedures: Pollution is parameterized separately for each dimension by injecting alternative representations, missing values, inaccurate values, duplicates, or class-count imbalance.For target class balance, the constructed worst case assigns samples to half the classes and none to the remainder.
- Completeness: Completeness is the proportion of non-missing feature values, excluding the target because samples missing target values are usually removed for training.A completeness of 1 means no missing values, while 0 means all non-target feature values are missing.
- Accuracy: Feature and target accuracy measure deviations from ground-truth values, with target accuracy treated separately because the target directly influences prediction performance.Categorical quality is one minus the erroneous-value ratio; numerical accuracy uses average absolute distance scaled by the ground-truth mean.
4 MACHINE LEARNING TASKS
The paper evaluates 19 machine-learning algorithms across classification, regression, and clustering, covering diverse algorithm families and model complexities.
- The study covers three machine-learning tasks: classification, regression, and clustering.
- Classification: Classification includes linear, tree-based, nearest-neighbor, neural-network, and transformer-based models.The selected models include Logistic regression, SVM, decision trees, gradient boosting, KNN, MLP variants, and TabNet.
- Regression: Regression compares seven widely used approaches from three regression-algorithm categories.
- Clustering: Clustering uses algorithms from distribution-based, centroid-based, hierarchical, density-based, and deep-learning families.The selection includes Gaussian mixture, k-means, k-prototypes, agglomerative clustering, OPTICS, and a deep autoencoder.
5 EXPERIMENTAL SETUP
The experimental setup systematically varies tasks, pollution types, datasets, scenarios, algorithms, and repeated runs to assess machine-learning performance under differing data-quality conditions.
- 4 905 experimental runs combine tasks, pollution types, datasets, scenarios, task-specific approaches, and repeated runs.The setup includes three tasks, six pollution types, four datasets, three pollution scenarios, seven approaches per task or five for clustering, and five runs, with one run for TabNet.
- Scenarios: The experiments evaluate polluted training data, polluted test data, and polluted training and test data for classification and regression.Clustering has no separate training and test set, so these scenarios do not apply to it.
- Pollution parameterization: Pollution ratios range from 0 to 1 in increments of 0.1, with additional parameter variation for representation consistency and uniqueness.Consistent representation uses k_v values of 2 and 5, while uniqueness varies ρ from 1 to 5.
- Preprocessing: The experimental dataset preparation includes task-specific preprocessing, including discretization for selected regression polluters and removal of sparse classes for class-balance experiments.The class-balance procedure also removes the same sparse classes from the corresponding original dataset to preserve comparability.
- Datasets: The study uses datasets spanning classification, regression, and clustering, selected for varied domains, sample sizes, characteristics, and task suitability.The datasets include Telco, Contraceptive, COVID, Houses, IMDB, Cars, and Letter examples, among others described in the setup.
- Performance measures: Regression performance is measured with R^2, which represents the fraction of target variance explained by the model.An R^2 of 1 explains all variance, 0 matches always predicting the target mean, and negative values indicate worse performance than that baseline.
6 RESULTS
Across classification, data-quality pollution affected algorithms differently by dimension, dataset, task stage, and model. Training-data label noise and test-data quality degradation were especially consequential, while some matching or favorable distribution shifts improved performance.
- Consistent Representation: Consistent-representation pollution usually had limited impact, although mismatched polluted test representations reduced performance and matching pollution across training and test data could improve it.MLP-1 was more susceptible than MLP-5 and MLP-10 in Scenario 2, while both larger variants remained nearly constant after 50% pollution.
- Completeness: Completeness pollution caused limited training-data degradation but faster test-data degradation that ended below the majority-class baseline.When both training and test data were incomplete, the risk from incomplete test data appeared lower than when only test data were incomplete.
- Feature Accuracy: Feature-accuracy pollution degraded classification performance in a dataset-dependent manner, with test quality often declining linearly and quality of 0.5 causing about a 10%pt F1-score drop for linear models.For some datasets, training models remained robust until a quality threshold before degrading more steeply.
- Target Accuracy: Training-data target accuracy below 1 divided by the number of classes reduced every classifier below the class-ratio baseline.Up to 20% of training labels could be flipped with no more than a 10%pt F1-score loss for most algorithms.
- Target Accuracy: 20% more incorrectly labeled test samples decreased prediction accuracy by up to 10%pt across datasets.The results followed a largely linear trend, underscoring the importance of carefully labeled test data.
- Target Class Balance: Target-class imbalance sometimes improved classification performance, but performance approached the majority-class baseline once minority examples became scarce.This pattern occurred in training-pollution scenarios, while models were generally robust when test distributions shifted toward greater balance.
7 DISCUSSION
The discussion identifies distinct effects of data-quality dimensions across classification, regression, and clustering, while showing that pollution can sometimes produce dataset- or algorithm-specific exceptions. It also emphasizes practical evaluation choices and the need to interpret isolated improvements cautiously.
- Cross-task trends: Missing values can reduce performance substantially, with average degradation exceeding 25%pt when missingness is especially high.Completeness was therefore classified as having a high effect across classification, regression, and clustering.
- Classification: Classification is least affected by uniqueness, consistent representation, and moderate target-class imbalance, whereas missing values and inaccurate labels or features can substantially degrade performance.Training with less than 40% missing values did not significantly reduce performance, but test-label errors can make a model appear worse than a baseline.
- Regression: Regression is most affected by completeness, feature accuracy, and target accuracy, with missing or inaccurate test data especially harmful when the model was not trained on comparable pollution.Uniqueness and target-class balance generally had little impact, while random forest was robust across most cases.
- Clustering: Clustering is most affected by completeness and feature accuracy, while centroid-based methods were generally the most robust across the six dimensions.Gaussian mixture benefited from slight numerical noise, and k-Prototypes had the best degradation-to-initial-performance trade-off for mixed-type data.
- Qualitative examples: The qualitative examples show that pollution can alter predictions even for unpolluted records, although isolated improvements were exceptions rather than the general pattern.The study selected fixed algorithm–quality-dimension combinations and tracked records at specified pollution levels.
8 CONCLUSION AND FUTURE WORK
The conclusion frames data quality as central to data-centric AI and summarizes an evaluation of six dimensions across three tasks, 19 algorithms, and ten datasets. Future work should broaden clustering evaluation, add quality dimensions, and examine individual models more deeply.
- Data quality is increasingly important as AI shifts from a model-centric to a data-centric perspective.
- The study evaluates six data-quality dimensions across classification, regression, and clustering using 19 algorithms and ten diverse datasets.Experiments vary the quality of training and test data across multiple scenarios.
- Future work should evaluate clustering with additional metrics, add more data-quality dimensions, and conduct deeper analyses of individual machine-learning models.
A ADDITIONAL PLOTS
The appendix adds supplementary plots for classification, regression, and clustering experiments, including focused views of clustering results and cluster-count relationships.
- The appendix includes classification F1-score plots for consistent representation with k_v=2.
- It includes regression R2 plots for consistent representation and for uniqueness when duplicate counts follow a normal distribution.
- Cluster-count relationships are examined for OPTICS and Autoencoder, while Agglomerative and k-Means/k-Prototypes return the expected count through an input parameter.
- Figure 27 reports average numbers of clusters identified under completeness, target class balance, and feature accuracy dimensions.
- The appendix provides AMI-score plots for the Bank clustering results and a focused version of those plots because shared y-axes made the original figures difficult to read.