SDQM: Synthetic Data Quality Metric for Object Detection Dataset Evaluation

1 Introduction

• Repeated testing and fine-tuning to bridge synthetic and real data require resources and pose hurdles to wider adoption.The paper says these iterations help synthetic data reflect real-world characteristics and variability.Source: §1 · p. 1
• SDQM is intended to predict object-detection model performance from dataset quality without exhaustive training and testing.The paper expects higher SDQM scores to correlate with improved model performance.Source: §1 · p. 2
• The study explores existing data-quality metrics and their effects on predicting object-detector performance from synthetic data.The paper also notes possible extensions to instance segmentation, object localization, and image classification.Source: §1 · p. 2
• The paper develops SDQM by evaluating domain gaps across pixel, spatial, frequency, and feature spaces using AI techniques and nonparametric distribution comparisons.The listed comparisons include K-S and A-D tests, KL divergence, energy and Wasserstein distances, Bhattacharyya distance, and JS divergence.Source: §1 · p. 2
• r = 0.87 between SDQM values and synthetic-data-trained object detectors’ mAP50 scores, with consistent performance gains across RarePlanes, DIMO, and WASABI.The paper presents this as evidence that SDQM can reduce reliance on exhaustive training-validation cycles.Source: §1 · p. 2

2.1 Existing Data Quality Metrics

• MAUVE compares real and synthetic distributions using quantized embeddings and divergence measures; precision, recall, and authenticity assess fidelity, diversity, and novelty.The paper adapted MAUVE for image data as an SDQM submetric; α-precision, β-recall, and authenticity are described as measures of fidelity, diversity, and authenticity, respectively.Source: §2.1 · pp. 2–3
• Existing image and distribution metrics do not directly establish dataset quality as defined by correlation with model performance; SSIM and LPIPS are content-invariant for this use case.IS evaluates generated-image quality using a pretrained Inception model, while FID compares real and generated feature distributions.Source: §2.1 · pp. 2–3

2.2 Dataset Interpretability

• V-usable information assesses task-specific useful information and dataset difficulty, while Data Maps categorize examples by training difficulty and consistency.The paper draws on the V-usable information framework and notes Data Maps can help identify problematic training examples.Source: §2.2 · p. 3

2 Related Work

• The reviewed work provides comparison benchmarks and methods that the authors aim to extend and integrate into a metric for synthetic object-detection data.Their stated goal is a metric that quantifies dataset quality and guides high-quality synthetic-data generation.Source: §2.2 · p. 3

3.1 Individual Metrics for Dataset Quality Evaluation

• The component set compares object-label distributions, spatial occupancy, bounding-box geometry, and pixel-intensity distributions across real and synthetic data.When available, informative metadata is incorporated into the object-count and category distributions; spatial heatmaps are mean-pooled by a factor of eight before RMSE comparison.Source: §3.1 · p. 5
• V-information extends V-usable information to object detection by measuring how easily a model learns from synthetic data and applies it to the target domain.The conditional-entropy variant fine-tunes YOLO11n on synthetic data for 10 epochs before validation on real data; predictive entropy uses mapped class-label matches from a pretrained YOLO11n model.Source: §3.1 · p. 6
• Extended V-information has the strongest individual correlation with mAP50 among the discussed metrics, while dataset separability and clusterability correlate moderately.Dataset separability combines classifier evaluation accuracy with model parameter count; clusterability compares real and synthetic embeddings jointly clustered by k-means.Source: §3.1 · pp. 5–6

3.2 Evaluating Feature Extraction Models

• GroundingDINO-tiny provides the most robust feature representations among the three evaluated extractors on DIMO real-synthetic image pairs.Models are assessed using cosine similarity and Euclidean distance between feature representations.Source: §3.2 · p. 7, Table 1 · p. 7

3.3 SDQM: Integrated Metric Development

• The integration process selects statistical tests for component measures and uses multivariate regression to examine other components’ relationships and weights.Backward feature reduction removed MAUVE, authenticity, and clusterability without reducing linear-regression performance and with only a slight random-forest performance reduction.Source: §3.3 · p. 9
• V-information, β-recall, Dataset Separability, and Pixel Intensity Match have the highest correlations with mAP50; Random Forest integration produces SDQM.Label Overlap correlates least with model performance, and the selected submetrics are integrated through a random forest.Source: §3.3 · pp. 8–9

4 Experiments

• The study uses mAP50 under Train on Synthetic, Test on Real as its measure of task utility, while noting its dependence on how well the real test set reflects deployment.Source: §4 · p. 9

4.1 Datasets

• Three synthetic-real dataset pairs support evaluation across settings, including RarePlanes, DIMO, and WASABI.WASABI experiments focus exclusively on vehicle detection.Source: §4.1 · pp. 9–10

4.2 Dataset Partitioning

• After splitting data to prevent scene leakage, the procedure selects α-precision, β-recall, and authenticity values to create 364 training and 64 evaluation subsets.The selection varies three embedding models and 11 equally spaced thresholds.Source: §4.2 · p. 10

4.3 Model Training

• Selected subsets yielded training mAP50 values of 0.31 to 0.79 and evaluation values of 0.06 to 0.40; random pairs broadened evaluation mAP50 to 0.06 to 0.79.Source: §4.3 · p. 10

4.4 SDQM Development

• Random Forest regression predicts YOLO11n mAP50 from submetric values, with Pixel Intensity Analysis, β-recall, and V-information identified as the most significant predictors.Source: §4.4 · p. 11
• SDQM shows strong correlation with mAP50, unlike existing metrics’ moderate correlations; authenticity’s negative association is negligible and unreliable.The comparison is reported in Table 3.Source: Table 3 · p. 11, §4.4 · pp. 11–12
• On the DIMO Small benchmark, SDQM takes 6 min 21 s versus over 11 h for 500-epoch YOLO11n training, an approximately two-order efficiency improvement.Source: §4.4 · p. 12

• SDQM combines coverage, diversity, domain similarity, and interpretability to predict synthetic datasets’ object-detection efficacy; its Pearson correlation with downstream YOLO11 performance is 0.87, surpassing baselines such as MAUVE.Source: §5 · p. 12

• DIMO and its synthetic counterpart have the highest α-precision scores, suggesting a close match to the real-data distribution.Source: §6 · p. 12
• β-recall is low across tests, suggesting that substantial portions of real data fall outside the synthetic distributions.Source: §6 · p. 12
• For these physics-based synthetic datasets, authenticity indicates that synthetic datapoints were not copied from real datapoints.Source: §6 · p. 13

• Scores for the same dataset pairs vary across feature extractors, highlighting the need to tailor extractors to the data and standardize evaluation pipelines.Source: §7 · p. 13
• Table 5 reports MAUVE and FI scores, including smoothed variants, for synthetic-real dataset pairs evaluated with different feature extractors.Source: Table 5 · p. 13, §7 · p. 13

8 Appendix C: Comparison of Regression Methods

• Random forest yielded the best correlation with the data; linear regression, ridge, XGBoost, and decision tree, but not lasso, had Pearson correlations within 0.05 of random forest.Source: §8 · p. 14
• Linear regression performed well and is presented as a viable, simpler, more interpretable way to calculate SDQM.Its validation datapoints are plotted against YOLO11n mAP50 in Fig. 5.Source: §8 · p. 14, Figure 5 · p. 14
• Ridge regression matched standard linear regression, indicating that its L2 regularization had negligible effect on the fitted curve.The authors interpret this consistency as evidence that SDQM does not suffer from high variance or overfitting.Source: §8 · p. 14, Figure 6 · p. 14
• XGBoost matched random forest on Spearman correlation and had a Pearson correlation only 0.01 lower, but the authors chose random forest for simplicity and robustness.Fig. 7 plots XGBoost validation datapoints against YOLO11n mAP50.Source: §8 · p. 14, Figure 7 · p. 15
• Decision tree regression performed adequately, while added randomness in random forest may explain the decision tree’s performance deficit.Fig. 8 plots decision tree validation datapoints against YOLO11n mAP50.Source: Figure 8 · p. 15, §8 · p. 16
• Lasso did not meet the authors’ performance needs because L1 regularization zeroed out dense submetric contributions, leading to underfitting.Lasso showed promise over linear regression with an added regularization term; its validation datapoints are plotted in Fig. 9.Source: Figure 9 · p. 15, §8 · p. 16

9 Appendix D: Submetric Regression Analysis

• The analysis combined submetrics into subcomponents through quadratic regression, removing terms with P values above 0.05 to reduce the number of terms.Source: §9 · p. 16
• For Bounding Box Match, retained terms include energy distances for aspect ratio, size, and area, their aspect-ratio and size squares, and their interaction.Energy distance was selected as the most effective statistical comparison based on P values; terms with P values above 0.05 were removed.Source: §9 · p. 16
• Pixel Intensity Match retained green pixel value, squared red pixel value, and their interaction; blue-pixel terms were removed.The authors suggest the removal indicates that blue-pixel value match is less important for predicting mAP.Source: §9 · p. 16
• Label Overlap’s most useful variable was the squared K-S Statistic value.Label Overlap contains only one subcomponent.Source: §9 · p. 16
• MAUVE was dropped from final SDQM and combined its MAUVE and smoothed MAUVE* subcomponents as (MAUVE*)^2 + (MAUVE)(MAUVE*); Frontier Integral used only its smoothed variant as a linear term.The authors report that Frontier Integral correlated highly with MAUVE.Source: §9 · p. 16
• Clusterability was removed from final SDQM after regression of its logarithmic and non-logarithmic subcomponents.The passage gives the reduced regression equation as c2 + l2 + cl + c + l.Source: §9 · p. 16
• V-information, the submetric with the highest correlation with mAP50, reduced to predictive confidence, conditional fusion, and their interaction with IoU V-information.Source: §9 · p. 16
• Submetrics with one component used a linear term for α-precision and linear-plus-quadratic terms for spatial distribution difference, β-recall, and authenticity.Authenticity was removed from final SDQM.Source: §9 · p. 16

Disclosures

• The authors declare no conflicts of interest.Source: Disclosures · p. 16