PaperlayerView exact revision on arXiv

Source-linked AI summary

Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Zhenyu Tao, Wei Xu, Xiaohu You, Petar Popovski, Osvaldo Simeone

arXiv:2608.27996v1cs.AI

TL;DR

Synthetic augmentation may not improve real-world performance because simulated and real data distributions differ, so its usefulness must be tested with scarce real evaluations. This paper develops aeSFT, which adaptively allocates both sign-flip computation and real test data while controlling Type-I error over time. Across three tasks, aeSFT detected useful synthetic data earlier than mean-based sequential testing and maintained comparable power to fixed-sample alternatives.

  • Problem

    Synthetic data may not improve population-level performance because the sim-to-real gap makes synthetic and real distributions differ, while reliable real test data are expensive.

  • Method

    aeSFT adaptively processes real test data in batches and adapts Monte Carlo sign-flip rounds while compounding evidence across batches.

  • Results

    Across three tasks, aeSFT detected useful synthetic data earlier than aMT, matched fixed-sample power, and kept the empirical false-positive rate below α = 0.1.

  • Takeaways & Limitations

    Adaptive testing can focus additional real evaluations on unresolved synthetic datasets while avoiding a prespecified test-set size.

Abstract

from arXiv · show

Digital twins (DTs) and learned world models are increasingly used to generate synthetic data that augment the scarce real datasets available for training artificial intelligence (AI) models in engineering systems. Owing to the inevitable simulation-to-reality (sim-to-real) gap, however, augmentation may fail to improve the performance of the trained model on the real data distribution. This paper addresses the resulting decision problem: Given a real dataset, a candidate synthetic dataset, and a fixed learning algorithm, decide whether training on the augmented dataset improves the true, population-level performance, while consuming as few real test data points as possible. Two formulations are considered: a direct test on the mean loss difference between the two trained models, and a symmetry-based test on the paired loss difference, which trades a stronger null assumption for faster evidence accumulation. For the latter, we introduce the {adaptive e-process sign-flip test} (aeSFT), a doubly adaptive procedure that adapts both the number of Monte Carlo sign-flip rounds, and hence the computational cost, and the amount of real test data consumed. aeSFT yields anytime-valid Type-I error control, with no need to pre-specify the test-set size. Experiments on a synthetic-data classification task, a DT-aided wireless packet-scheduling task, and a radio-map prediction task show that aeSFT identifies useful synthetic data using substantially fewer real test samples than mean-based sequential testing, matches the power of fixed-sample sign-flip testing and the paired $t$-test, while keeping the false-positive rate below the target level.

I. INTRODUCTION

Synthetic data can expand scarce engineering datasets, but the sim-to-real gap prevents simulator output alone from establishing whether augmentation improves population-level performance. The paper formulates this decision as statistical testing with minimal real test data and introduces aeSFT as an adaptive sign-flip procedure with anytime-valid error control.

  • Motivation and Context: Deployment-specific data are expensive to collect because they require measurements, human labels, physical-system episodes, or high-fidelity emulator runs.This makes limiting real evaluations central to deciding whether synthetic augmentation is useful.
  • Motivation and Context: Digital twins and learned world models generate synthetic data, but simplifications create a sim-to-real gap that can bias training toward the wrong population loss.Synthetic and real distributions may differ even when the simulator is high fidelity.
  • Motivation and Context: Whether a candidate synthetic dataset helps a fixed learning algorithm must therefore be evaluated using real data drawn from the target distribution.The target quantity is population loss, not simulator output alone.
  • Problem Formulation: The paper frames usefulness as sequential binary testing and studies direct mean-loss and symmetry-based paired-loss formulations.The goal is to decide whether augmenting a real training set improves real-world performance while consuming few real test points.
  • Main Contributions: aeSFT adapts both Monte Carlo sign-flip computation and real-data batches, compounding evidence across batches until sufficient evidence is collected.The procedure uses independent batches and can close an unpromising batch early.
  • Main Contributions: Theoretical analysis gives anytime-valid Type-I error control, while experiments report higher early true-positive rates than mean-based testing and power comparable to fixed-sample procedures.The experiments cover binary classification, wireless packet scheduling, and radio-map prediction.

A. Direct Synthetic Data Test

The direct test compares models trained with and without synthetic data through their population mean loss difference. It is broadly applicable, but heavy-tailed sample-level differences can make sequential evidence accumulation inefficient when test data are limited.

  • Direct Test: The direct formulation compares the loss of a model trained on real data with one trained on real and synthetic data.It evaluates whether augmentation lowers the population loss.
  • Direct Test: Synthetic augmentation is useful when the population loss gap ∆ = L(f) − L(f_syn) is positive.The baseline model loss is subtracted from the augmented-model loss, so lower augmented loss yields ∆ > 0.
  • Hypotheses: The direct binary test treats non-positive mean loss improvement as the null and positive improvement as the alternative.Rejecting the null provides sufficient evidence that the synthetic data are useful for training.
  • Limitations: Mean-based sequential testing can be sample-inefficient when sample-level objective differences are heavy-tailed under the alternative.In that regime, evidence against the null accumulates slowly with limited test data.
  • Symmetry-Based Alternative: The symmetry-based formulation instead tests paired sample-level loss differences whose expectation equals the population loss gap.Its specialized null requires additional distributional assumptions beyond the direct mean test.
  • Symmetry-Based Alternative: The symmetry-based null assumes synthetic data cannot harm average performance and that paired loss differences are symmetric around zero when augmentation is unhelpful.This stronger assumption can support faster evidence accumulation but narrows the interpretation of rejection.
  • Interpretation: Rejecting the specialized symmetry null supports improved synthetic augmentation, but does not entail rejection of the original direct null.The stronger null is therefore not interchangeable with the mean-based formulation.

C. Error Control

The sequential procedures control the probability of incorrectly declaring synthetic data useful at a user-defined level while allowing decisions before the maximum test-set size. They seek to minimize real test data consumption when synthetic data improve model performance.

  • The test processes real test data sequentially and may stop when a final decision is reached, subject to a maximum collection size M.If no rejection occurs through time M, the null cannot be rejected.
  • Type-I error is defined as the probability of incorrectly declaring synthetic data useful, with the worst-case probability required to be at most α.The probability is taken over the data used for testing.
  • The testing objective is to minimize the number of test points needed to reject the null when synthetic data actually improve model performance.This efficiency objective is considered under the relevant alternatives H1 or Hsyn.
  • Direct test: For the direct test, bounded sample-level losses imply a bounded loss difference, allowing the mean-loss null to be normalized as E[eδ] ≤ 0.5.The direct null is originally E[δ] ≤ 0, with δ ∈ [−Lmax, Lmax].
  • Direct test: The betting statistic is an e-process whose threshold γ = 1/α ensures Type-I error control, but heavy-tailed paired loss differences can make evidence accumulate slowly.Under the alternative, this may require many test points before rejecting the direct null.
  • Sign-flip test: In fixed-sample sign-flip testing, both the test-set size M and the number of random sign-flip sequences B are fixed in advance.The basic SFT is reviewed as a non-sequential procedure.

IV. FIXED-SAMPLE SYMMETRY-BASED SYNTHETIC DATA TEST

The fixed-sample symmetry-based test uses sign-flipped paired loss differences to assess synthetic-data usefulness, with Monte Carlo randomization and e-process variants providing evidence against the null. SFT fixes both test-data and sign-flip budgets, whereas eSFT adapts sign-flip rounds but retains a fixed test-set size.

  • Sign-Flip Test: The test evaluates whether paired loss differences support synthetic-data usefulness under a sign-flip-invariant null.For each test sample, δ_i is the loss of the real-only model minus the loss of the augmented model.
  • Sign-Flip Test: Monte Carlo SFT draws B independent random binary sign sequences and computes the corresponding sign-flipped statistics.Each sign is independently sampled from Unif{−1,+1}.
  • Sign-Flip Test: A small one-sided p-value indicates that few sign-flipped statistics match the observed improvement, providing evidence that synthetic data are useful.The test rejects when the p-value is at most α.
  • Limitations of SFT: SFT must choose the sign-flip budget B in advance, creating a power–computation trade-off and making post hoc increases invalid.Increasing B after inspecting the p-value would constitute p-hacking and compromise Type-I error control.
  • E-Process Sign-Flip Test: eSFT converts sign-flip testing into a betting process that updates wealth across rounds and rejects when W_b reaches 1/α.Its betting function is constrained to preserve a fair game under the null, while an optimized strategy can accelerate wealth growth under alternatives.
  • E-Process Sign-Flip Test: With the deterministic choice η̂, a fixed-round p-value satisfying p ≤ η̂ implies that some round no later than B crosses the e-process rejection threshold.This connects the fixed-sample Monte Carlo test to the e-process stopping rule.

V. ADAPTIVE SYMMETRY-BASED SYNTHETIC DATA TEST

The adaptive symmetry-based test addresses eSFT’s fixed test-data budget by adapting both sign-flip computation and real-data consumption across independent batches. Its validity depends on stronger symmetry and interpretive assumptions than the direct mean-based test.

  • Assumptions: The direct mean test needs a finite loss bound, whereas SFT, eSFT, and aeSFT allow unbounded losses under i.i.d. symmetric paired differences.The methods therefore exchange boundedness for stronger distributional assumptions.
  • Assumptions: The symmetry-based route additionally assumes that synthetic data cannot increase population loss when rejection is interpreted as usefulness.It therefore does not strictly dominate the direct test.
  • Motivation for aeSFT: Unlike SFT, eSFT adapts sign-flip rounds but still requires the test-set size M to be fixed in advance.Thus, eSFT adapts computational complexity without adapting real-data consumption.
  • Motivation for aeSFT: A fixed eSFT data pool cannot acquire additional evidence for difficult tests or stop early when fewer real samples would suffice.This fixed-budget constraint is the central data-consumption bottleneck.
  • aeSFT: aeSFT adapts both sign-flip rounds and test-data consumption by moving across independent batches and stopping unpromising batches early.An intra-batch e-process tracks within-batch wealth, while an inter-batch e-process aggregates wealth across batches.

A. Intra-Batch E-Process

The intra-batch component processes fresh, non-overlapping real-data batches and adaptively accumulates sign-flip evidence over a batch-specific number of rounds. It can stop early based only on information observed within the current batch.

  • A. Intra-Batch E-Process: Each batch draws fresh i.i.d. test data from the ground-truth distribution and computes paired objective differences and a batch statistic.
  • A. Intra-Batch E-Process: The sign-flipped statistics update an intra-batch e-process whose betting function determines the batch’s wealth growth.
  • A. Intra-Batch E-Process: A batch may close early using only accumulated within-batch information, without rejecting the null, when its evidence becomes unpromising.
  • A. Intra-Batch E-Process: Within each batch, aeSFT adaptively selects the number of Monte Carlo sign-flip rounds and generates independent sign sequences.

B. Inter-Batch E-Process

The inter-batch e-process compounds evidence across fresh batches while permitting adaptive sign-flip budgets and test-data budgets. Predictable batch adaptation preserves the e-process property and supports anytime-valid rejection.

  • B. Inter-Batch E-Process: The inter-batch e-process multiplies intra-batch e-process values, reinvesting inherited wealth so evidence accumulates across fresh batches.This product preserves anytime-valid continuation under predictable adaptation.
  • B. Inter-Batch E-Process: aeSFT remains adaptive in both the total number of sign-flip rounds and the total test-dataset size.
  • B. Inter-Batch E-Process: Under the symmetry null, independent batches, predictable batch sizes, and stopping-time sign-flip budgets establish the theorem’s validity conditions.
  • B. Inter-Batch E-Process: aeSFT closes an unpromising batch when wealth falls below ω, while retaining a change-based rejection rule rather than using only a fixed wealth ceiling.
  • B. Inter-Batch E-Process: The procedure preserves the e-process property when intra-batch stopping rules depend only on the sign-flip history observed so far.
  • B. Inter-Batch E-Process: Adaptive batch sizes begin small and enlarge after unpromising batches, balancing data economy against the risk that small batches mask moderate improvements.The next batch size is selected from information available before drawing that batch.
  • B. Inter-Batch E-Process: If the accumulated wealth reaches 1/α, aeSFT rejects; otherwise it continues while data remain available and ends without rejection when the maximum budget is exhausted.

VI. EXPERIMENTS

Experiments evaluate aeSFT against adaptive mean testing, fixed-sample sign-flip testing, and a fixed-sample t-test across classification, wireless scheduling, and radio-map prediction. Across these settings, aeSFT improves early detection while maintaining low false-positive rates.

  • VI. EXPERIMENTS: The evaluation covers binary classification, DT-aided wireless packet scheduling, and radio-map prediction, using aMT, eSFT, and a fixed-sample t-test as baselines.
  • A. Binary Classification: In a useful-synthetic-data classification instance, aeSFT rejects after 680 test samples, whereas aMT requires approximately 1,410 samples and fixed-sample methods use 2,000.
  • A. Binary Classification: Under useful-data alternatives, aeSFT places substantially more stopping-time probability at small sample sizes than eSFT and the t-test, which decide only at prespecified fixed sizes.
  • A. Binary Classification: The comparison uses different formal nulls: aMT controls Type-I error under the direct mean null, whereas SFT-based methods use the symmetry null.
  • A. Binary Classification: At target level α = 0.1, aeSFT achieves higher TPR than aMT at small average consumption while its empirical FPR remains below 0.1.
  • A. Binary Classification: The selected aeSFT setting has mean TPR 0.612 ± 0.026, mean FPR 0.027 ± 0.007, and EH1[mstop] = 1,024 samples.

B. Wireless Network Packet Scheduling

The wireless scheduling experiment uses a digital twin with perturbed traffic parameters while preserving the system’s bandwidth, rewards, state, and action structure. aeSFT detects useful synthetic trajectories earlier than baselines while keeping false-positive rates below the target.

  • B. Wireless Network Packet Scheduling: The three-class task models bursty packet arrivals with class-dependent rates and learns a policy maximizing long-term average network reward.
  • B. Wireless Network Packet Scheduling: The digital twin perturbs real arrival rates and deadlines while keeping bandwidth requirements, packet rewards, shared capacity, states, and actions unchanged.
  • B. Wireless Network Packet Scheduling: Offline deep Q-learning trains on trajectories from fixed real and synthetic datasets without further environment interaction.
  • B. Wireless Network Packet Scheduling: The paired loss difference equals the difference in normalized model returns, with bounded loss Lmax = 1 enabling aMT.
  • B. Wireless Network Packet Scheduling: Among 200 held-out real-environment episodes, 99 satisfy H1 and 101 satisfy H0; usefulness is defined through the direct population gap between the two models.
  • B. Wireless Network Packet Scheduling: After 16 episodes, aeSFT reaches TPR 0.657 while aMT remains near zero; aeSFT reaches 0.778 at 88.0 episodes, versus aMT’s 0.717 at 143.6.
  • B. Wireless Network Packet Scheduling: All methods remain below FPR α = 0.1, while aeSFT combines low false-positive rates, earlier detection, and continued testing with fresh data.

C. Radio-Map Prediction

The radio-map experiment varies ray-tracing fidelity across 196 synthetic datasets and tests whether augmented training improves prediction using limited high-fidelity maps. aeSFT detects useful synthetic data adaptively while maintaining false-positive control and performance close to fixed-sample baselines.

  • Setting: Radio maps predict received signal strength across a 128 × 128 grid from building-height maps and base-station information.The scene covers a 512 m × 512 m area imported from OpenStreetMap.
  • Setting: Increasing rays per transmitter and reflection depth improves non-line-of-sight coverage but increases computational complexity.Low ray budgets produce sparse maps, whereas higher-fidelity settings resolve more propagation paths.
  • Experimental design: The experiment compares 196 low-fidelity synthetic datasets against high-fidelity real maps, isolating the effect of ray-tracing fidelity.High-fidelity maps use depth 20 and 10^9 rays/Tx; synthetic settings span depths 1–14 and 10^4–10^5 rays/Tx.
  • Results: aeSFT reaches TPR 0.463 after 50 maps, then 0.480 at 135.9 maps and 0.512 at 227.4 maps under H1.Its empirical FPR is 0.014 initially and 0.096 with further continuation under H0, remaining below α = 0.1.
  • Results: Fixed-sample eSFT and t-test points require prespecified dataset sizes, whereas aeSFT and aMT permit statistically valid continuation.Continuation avoids reusing a test after inspecting results and thereby preserves the Type-I error guarantee.
  • Results: At 40 maps on average for the IoU metric, aeSFT attains TPR 0.925 versus 0.957 for eSFT and 0.968 for the t-test, with zero empirical FPR for all methods.The IoU loss is bounded and less heavy-tailed than the MSE-based loss, favoring mean-based sequential testing in this experiment.

VII. CONCLUSION

The paper addresses whether synthetic data improve population-level performance and introduces aeSFT for adaptive testing with limited real data. Across three tasks, aeSFT detects useful synthetic data earlier than aMT, matches fixed-sample power, and maintains false-positive control.

  • Contribution: aeSFT adapts both Monte Carlo sign-flip rounds and real test-data consumption through intra-batch and inter-batch e-processes.Under the symmetry null, stopping when evidence exceeds 1/α controls Type-I error without a prespecified test-set size or known finite loss bound.
  • Findings: Across binary classification, wireless packet scheduling, and radio-map prediction, aeSFT consistently detects useful synthetic data earlier than aMT.It also matches the power of fixed-sample procedures without prespecifying a sample size.
  • Findings: In packet scheduling, aeSFT reaches TPR 0.657 after 16 real test episodes, with mean stopping time 88.0 episodes versus 143.6 for aMT.In radio-map prediction, it reaches TPR 0.463 after 50 maps and 0.512 with continuation while keeping empirical FPR below α = 0.1.
  • Implications: aeSFT applies wherever paired loss differences can be evaluated on real test data, especially when trusted real evaluations are expensive.The paper discusses wireless communication evaluations and limited-data assessment of synthetic datasets for large language models.

APPENDIX A PROOF OF THEOREM 1

The appendix proves validity of aeSFT by establishing intra-batch and inter-batch e-process properties under the symmetry null. These results justify optional stopping and the rejection threshold 1/α.

  • Validity guarantee: Rejecting when accumulated wealth reaches 1/α controls Type-I error at level α.This is the validity guarantee used by aeSFT's stopping rule.
  • Intra-batch e-process: Within each batch, the betting process is a nonnegative e-process under the null, with adaptive sign-flip rounds treated as a stopping time.The fair-game property prevents a single null bet from increasing expected wealth.
  • Intra-batch e-process: Optional stopping makes each stopped intra-batch wealth a valid e-value under the symmetry null.The proof invokes the optional stopping theorem for nonnegative test martingales.
  • Inter-batch e-process: Across batches, independent fresh observations and batch sizes selected from past data preserve the conditional law needed for the e-process argument.The inter-batch process compounds wealth across the testing history of all batches.
  • Inter-batch e-process: The inter-batch wealth growth process is a nonnegative test martingale and therefore an e-process under the stated conditions.The appendix derives this property from the intra-batch result and cross-batch conditioning.
Loading 2608.27996v1…