Should I Use This Synthetic Dataset for Training? How to Test with Minimal Real Data

Motivation and Context

• Deployment-specific data are expensive to collect because they require measurements, human labels, physical-system episodes, or high-fidelity emulator runs.This makes limiting real evaluations central to deciding whether synthetic augmentation is useful.Source: A. Motivation and Context · pp. 1–2
• Digital twins and learned world models generate synthetic data, but simplifications create a sim-to-real gap that can bias training toward the wrong population loss.Synthetic and real distributions may differ even when the simulator is high fidelity.Source: A. Motivation and Context · p. 1
• Whether a candidate synthetic dataset helps a fixed learning algorithm must therefore be evaluated using real data drawn from the target distribution.The target quantity is population loss, not simulator output alone.Source: A. Motivation and Context · pp. 1–2

Problem Formulation

• The paper frames usefulness as sequential binary testing and studies direct mean-loss and symmetry-based paired-loss formulations.The goal is to decide whether augmenting a real training set improves real-world performance while consuming few real test points.Source: C. Main Contributions · pp. 2–3, §II · p. 3

Main Contributions

• aeSFT adapts both Monte Carlo sign-flip computation and real-data batches, compounding evidence across batches until sufficient evidence is collected.The procedure uses independent batches and can close an unpromising batch early.Source: C. Main Contributions · p. 3
• Theoretical analysis gives anytime-valid Type-I error control, while experiments report higher early true-positive rates than mean-based testing and power comparable to fixed-sample procedures.The experiments cover binary classification, wireless packet scheduling, and radio-map prediction.Source: C. Main Contributions · p. 3

Direct Test

• The direct formulation compares the loss of a model trained on real data with one trained on real and synthetic data.It evaluates whether augmentation lowers the population loss.Source: A. Direct Synthetic Data Test · p. 3
• Synthetic augmentation is useful when the population loss gap ∆ = L(f) − L(f_syn) is positive.The baseline model loss is subtracted from the augmented-model loss, so lower augmented loss yields ∆ > 0.Source: A. Direct Synthetic Data Test · p. 4

Hypotheses

• The direct binary test treats non-positive mean loss improvement as the null and positive improvement as the alternative.Rejecting the null provides sufficient evidence that the synthetic data are useful for training.Source: A. Direct Synthetic Data Test · p. 4

Limitations

• Mean-based sequential testing can be sample-inefficient when sample-level objective differences are heavy-tailed under the alternative.In that regime, evidence against the null accumulates slowly with limited test data.Source: B. Symmetry-Based Synthetic Data Test · p. 4

Symmetry-Based Alternative

• The symmetry-based formulation instead tests paired sample-level loss differences whose expectation equals the population loss gap.Its specialized null requires additional distributional assumptions beyond the direct mean test.Source: B. Symmetry-Based Synthetic Data Test · p. 4
• The symmetry-based null assumes synthetic data cannot harm average performance and that paired loss differences are symmetric around zero when augmentation is unhelpful.This stronger assumption can support faster evidence accumulation but narrows the interpretation of rejection.Source: B. Symmetry-Based Synthetic Data Test · p. 4

Interpretation

• Rejecting the specialized symmetry null supports improved synthetic augmentation, but does not entail rejection of the original direct null.The stronger null is therefore not interchangeable with the mean-based formulation.Source: B. Symmetry-Based Synthetic Data Test · pp. 4–5

• The test processes real test data sequentially and may stop when a final decision is reached, subject to a maximum collection size M.If no rejection occurs through time M, the null cannot be rejected.Source: C. Error Control · p. 5
• Type-I error is defined as the probability of incorrectly declaring synthetic data useful, with the worst-case probability required to be at most α.The probability is taken over the data used for testing.Source: C. Error Control · p. 5
• The testing objective is to minimize the number of test points needed to reject the null when synthetic data actually improve model performance.This efficiency objective is considered under the relevant alternatives H1 or Hsyn.Source: C. Error Control · p. 5

Direct test

• For the direct test, bounded sample-level losses imply a bounded loss difference, allowing the mean-loss null to be normalized as E[eδ] ≤ 0.5.The direct null is originally E[δ] ≤ 0, with δ ∈ [−Lmax, Lmax].Source: §III · p. 5
• The betting statistic is an e-process whose threshold γ = 1/α ensures Type-I error control, but heavy-tailed paired loss differences can make evidence accumulate slowly.Under the alternative, this may require many test points before rejecting the direct null.Source: §III · pp. 5–6

Sign-flip test

• In fixed-sample sign-flip testing, both the test-set size M and the number of random sign-flip sequences B are fixed in advance.The basic SFT is reviewed as a non-sequential procedure.Source: Figure 3 · p. 6

Sign-Flip Test

• The test evaluates whether paired loss differences support synthetic-data usefulness under a sign-flip-invariant null.For each test sample, δ_i is the loss of the real-only model minus the loss of the augmented model.Source: A. Sign-Flip Test · p. 6, §IV · p. 6
• Monte Carlo SFT draws B independent random binary sign sequences and computes the corresponding sign-flipped statistics.Each sign is independently sampled from Unif{−1,+1}.Source: A. Sign-Flip Test · p. 6
• A small one-sided p-value indicates that few sign-flipped statistics match the observed improvement, providing evidence that synthetic data are useful.The test rejects when the p-value is at most α.Source: A. Sign-Flip Test · p. 6

Limitations of SFT

• SFT must choose the sign-flip budget B in advance, creating a power–computation trade-off and making post hoc increases invalid.Increasing B after inspecting the p-value would constitute p-hacking and compromise Type-I error control.Source: B. E-Process Sign-Flip Test (eSFT) · p. 6

E-Process Sign-Flip Test

• eSFT converts sign-flip testing into a betting process that updates wealth across rounds and rejects when W_b reaches 1/α.Its betting function is constrained to preserve a fair game under the null, while an optimized strategy can accelerate wealth growth under alternatives.Source: B. E-Process Sign-Flip Test (eSFT) · p. 7
• With the deterministic choice η̂, a fixed-round p-value satisfying p ≤ η̂ implies that some round no later than B crosses the e-process rejection threshold.This connects the fixed-sample Monte Carlo test to the e-process stopping rule.Source: B. E-Process Sign-Flip Test (eSFT) · p. 7

Assumptions

• The direct mean test needs a finite loss bound, whereas SFT, eSFT, and aeSFT allow unbounded losses under i.i.d. symmetric paired differences.The methods therefore exchange boundedness for stronger distributional assumptions.Source: §V · p. 8
• The symmetry-based route additionally assumes that synthetic data cannot increase population loss when rejection is interpreted as usefulness.It therefore does not strictly dominate the direct test.Source: §V · p. 8

Motivation for aeSFT

• Unlike SFT, eSFT adapts sign-flip rounds but still requires the test-set size M to be fixed in advance.Thus, eSFT adapts computational complexity without adapting real-data consumption.Source: §V · p. 8
• A fixed eSFT data pool cannot acquire additional evidence for difficult tests or stop early when fewer real samples would suffice.This fixed-budget constraint is the central data-consumption bottleneck.Source: §V · p. 8

aeSFT

• aeSFT adapts both sign-flip rounds and test-data consumption by moving across independent batches and stopping unpromising batches early.An intra-batch e-process tracks within-batch wealth, while an inter-batch e-process aggregates wealth across batches.Source: §V · p. 8

A. Intra-Batch E-Process

• Each batch draws fresh i.i.d. test data from the ground-truth distribution and computes paired objective differences and a batch statistic.Source: A. Intra-Batch E-Process · p. 8
• The sign-flipped statistics update an intra-batch e-process whose betting function determines the batch’s wealth growth.Source: A. Intra-Batch E-Process · p. 8
• A batch may close early using only accumulated within-batch information, without rejecting the null, when its evidence becomes unpromising.Source: A. Intra-Batch E-Process · p. 8
• Within each batch, aeSFT adaptively selects the number of Monte Carlo sign-flip rounds and generates independent sign sequences.Source: A. Intra-Batch E-Process · pp. 8–9

B. Inter-Batch E-Process

• The inter-batch e-process multiplies intra-batch e-process values, reinvesting inherited wealth so evidence accumulates across fresh batches.This product preserves anytime-valid continuation under predictable adaptation.Source: B. Inter-Batch E-Process · p. 9
• aeSFT remains adaptive in both the total number of sign-flip rounds and the total test-dataset size.Source: B. Inter-Batch E-Process · p. 9, C. Adaptive Budget Schedule · p. 9
• Under the symmetry null, independent batches, predictable batch sizes, and stopping-time sign-flip budgets establish the theorem’s validity conditions.Source: B. Inter-Batch E-Process · p. 9, C. Adaptive Budget Schedule · p. 10
• aeSFT closes an unpromising batch when wealth falls below ω, while retaining a change-based rejection rule rather than using only a fixed wealth ceiling.Source: C. Adaptive Budget Schedule · pp. 9–10
• The procedure preserves the e-process property when intra-batch stopping rules depend only on the sign-flip history observed so far.Source: C. Adaptive Budget Schedule · p. 10
• Adaptive batch sizes begin small and enlarge after unpromising batches, balancing data economy against the risk that small batches mask moderate improvements.The next batch size is selected from information available before drawing that batch.Source: C. Adaptive Budget Schedule · p. 10
• If the accumulated wealth reaches 1/α, aeSFT rejects; otherwise it continues while data remain available and ends without rejection when the maximum budget is exhausted.Source: C. Adaptive Budget Schedule · p. 10

VI. EXPERIMENTS

• The evaluation covers binary classification, DT-aided wireless packet scheduling, and radio-map prediction, using aMT, eSFT, and a fixed-sample t-test as baselines.Source: §VI · p. 10

A. Binary Classification

• In a useful-synthetic-data classification instance, aeSFT rejects after 680 test samples, whereas aMT requires approximately 1,410 samples and fixed-sample methods use 2,000.Source: A. Binary Classification · pp. 10–11
• Under useful-data alternatives, aeSFT places substantially more stopping-time probability at small sample sizes than eSFT and the t-test, which decide only at prespecified fixed sizes.Source: A. Binary Classification · pp. 11–12
• The comparison uses different formal nulls: aMT controls Type-I error under the direct mean null, whereas SFT-based methods use the symmetry null.Source: A. Binary Classification · p. 12
• At target level α = 0.1, aeSFT achieves higher TPR than aMT at small average consumption while its empirical FPR remains below 0.1.Source: A. Binary Classification · p. 12
• The selected aeSFT setting has mean TPR 0.612 ± 0.026, mean FPR 0.027 ± 0.007, and EH1[mstop] = 1,024 samples.Source: A. Binary Classification · p. 12

B. Wireless Network Packet Scheduling

• The three-class task models bursty packet arrivals with class-dependent rates and learns a policy maximizing long-term average network reward.Source: B. Wireless Network Packet Scheduling · p. 13
• The digital twin perturbs real arrival rates and deadlines while keeping bandwidth requirements, packet rewards, shared capacity, states, and actions unchanged.Source: B. Wireless Network Packet Scheduling · p. 13
• Offline deep Q-learning trains on trajectories from fixed real and synthetic datasets without further environment interaction.Source: B. Wireless Network Packet Scheduling · p. 13
• The paired loss difference equals the difference in normalized model returns, with bounded loss Lmax = 1 enabling aMT.Source: B. Wireless Network Packet Scheduling · p. 14
• Among 200 held-out real-environment episodes, 99 satisfy H1 and 101 satisfy H0; usefulness is defined through the direct population gap between the two models.Source: B. Wireless Network Packet Scheduling · p. 14
• After 16 episodes, aeSFT reaches TPR 0.657 while aMT remains near zero; aeSFT reaches 0.778 at 88.0 episodes, versus aMT’s 0.717 at 143.6.Source: B. Wireless Network Packet Scheduling · p. 14
• All methods remain below FPR α = 0.1, while aeSFT combines low false-positive rates, earlier detection, and continued testing with fresh data.Source: B. Wireless Network Packet Scheduling · p. 14

Setting

• Radio maps predict received signal strength across a 128 × 128 grid from building-height maps and base-station information.The scene covers a 512 m × 512 m area imported from OpenStreetMap.Source: C. Radio-Map Prediction · pp. 14–15
• Increasing rays per transmitter and reflection depth improves non-line-of-sight coverage but increases computational complexity.Low ray budgets produce sparse maps, whereas higher-fidelity settings resolve more propagation paths.Source: C. Radio-Map Prediction · p. 15, Figure 13 · p. 15

Experimental design

• The experiment compares 196 low-fidelity synthetic datasets against high-fidelity real maps, isolating the effect of ray-tracing fidelity.High-fidelity maps use depth 20 and 10^9 rays/Tx; synthetic settings span depths 1–14 and 10^4–10^5 rays/Tx.Source: C. Radio-Map Prediction · p. 15

Results

• aeSFT reaches TPR 0.463 after 50 maps, then 0.480 at 135.9 maps and 0.512 at 227.4 maps under H1.Its empirical FPR is 0.014 initially and 0.096 with further continuation under H0, remaining below α = 0.1.Source: C. Radio-Map Prediction · p. 15
• Fixed-sample eSFT and t-test points require prespecified dataset sizes, whereas aeSFT and aMT permit statistically valid continuation.Continuation avoids reusing a test after inspecting results and thereby preserves the Type-I error guarantee.Source: C. Radio-Map Prediction · pp. 15–16
• At 40 maps on average for the IoU metric, aeSFT attains TPR 0.925 versus 0.957 for eSFT and 0.968 for the t-test, with zero empirical FPR for all methods.The IoU loss is bounded and less heavy-tailed than the MSE-based loss, favoring mean-based sequential testing in this experiment.Source: C. Radio-Map Prediction · p. 16

Contribution

• aeSFT adapts both Monte Carlo sign-flip rounds and real test-data consumption through intra-batch and inter-batch e-processes.Under the symmetry null, stopping when evidence exceeds 1/α controls Type-I error without a prespecified test-set size or known finite loss bound.Source: §VII · pp. 16–17

Findings

• Across binary classification, wireless packet scheduling, and radio-map prediction, aeSFT consistently detects useful synthetic data earlier than aMT.It also matches the power of fixed-sample procedures without prespecifying a sample size.Source: §VII · p. 17
• In packet scheduling, aeSFT reaches TPR 0.657 after 16 real test episodes, with mean stopping time 88.0 episodes versus 143.6 for aMT.In radio-map prediction, it reaches TPR 0.463 after 50 maps and 0.512 with continuation while keeping empirical FPR below α = 0.1.Source: §VII · p. 17

Implications

• aeSFT applies wherever paired loss differences can be evaluated on real test data, especially when trusted real evaluations are expensive.The paper discusses wireless communication evaluations and limited-data assessment of synthetic datasets for large language models.Source: §VII · p. 17

Validity guarantee

• Rejecting when accumulated wealth reaches 1/α controls Type-I error at level α.This is the validity guarantee used by aeSFT's stopping rule.Source: Appendix A · p. 17

Intra-batch e-process

• Within each batch, the betting process is a nonnegative e-process under the null, with adaptive sign-flip rounds treated as a stopping time.The fair-game property prevents a single null bet from increasing expected wealth.Source: Appendix A · p. 17
• Optional stopping makes each stopped intra-batch wealth a valid e-value under the symmetry null.The proof invokes the optional stopping theorem for nonnegative test martingales.Source: Appendix A · p. 17

Inter-batch e-process

• Across batches, independent fresh observations and batch sizes selected from past data preserve the conditional law needed for the e-process argument.The inter-batch process compounds wealth across the testing history of all batches.Source: Appendix A · p. 17
• The inter-batch wealth growth process is a nonnegative test martingale and therefore an e-process under the stated conditions.The appendix derives this property from the intra-batch result and cross-batch conditioning.Source: Appendix A · p. 17