Source-linked AI summary
Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
Yanran Wu, Sana Lakdawala, Renzo Tassara Miller, Chongyang Bai, Sharath Ciddu, Shivendra Pratap Singh, Kungang Li, Sandeep Pandey, Chunwei Liu
TL;DR
Synthetic data can scale supervision, but noise and distribution mismatch complicate selecting examples that help real-task performance as the learner evolves. FROST uses real-training gradient feedback and history-calibrated batch routing to filter synthetic data online. It improves public-benchmark performance with fewer synthetic samples and reports gains in industrial ads re-ranking.
Problem
Synthetic data can contain noise or distribution mismatch, while selection methods may not track the learner’s evolving needs when choosing supervision for real-task performance.
Source: §1 · pp. 1–2Method
Results
With 20–30% fewer synthetic samples, absolute accuracy improves 0.58–1.14% across 3 image generators and 0.9% on text-to-SQL over full pools; industrial relative NE improves 0.096% over real-only training.
Source: §1 · p. 2Takeaways & Limitations
Results across public benchmarks and industrial ads re-ranking indicate FROST’s effectiveness across different domains and a large-scale production setting.
Source: §5 · p. 10The batch-calibration assumption is that candidate utilities are i.i.d. conditional on a fixed model and reference, with finite, nonzero variance.
Source: §3.2 · p. 5
Abstract
from arXiv · showhide
Synthetic data can scale training supervision when real-world data are limited, but noise and distribution mismatch can reduce its value. Existing synthetic data selection methods often emphasize fidelity or diversity rather than the learner's evolving needs. We propose FROST, an online framework that estimates synthetic-data utility through gradient feedback anchored in real training data. It calibrates batch utility against recent history to determine when filtering is needed and filters samples only in out-of-band batches to determine what to retain, without an external verifier or held-out validation set. Experiments on two public benchmarks for image classification and LLM fine-tuning for text-to-SQL show that FROST filters out around 20--30% of the synthetic data while improving real-task performance compared with training on the full synthetic data pool. We further apply FROST during training in a large-scale industrial ads re-ranking system, achieving significant performance gains over a highly optimized production baseline, demonstrating its effectiveness and generalizability.
1 Introduction
Real-world data constraints motivate synthetic supervision, but noisy or mismatched synthetic examples may not help real-task performance. FROST estimates utility from real-training gradients and filters conditionally, improving performance with fewer synthetic samples and in industrial ads re-ranking.
1 Introduction
- Synthetic data can expand supervision when real-data collection or annotation is costly or constrained, including for vision, LLMs, and recommendation systems.Synthetic data can also contain semantic errors, artifacts, missing variations, or distribution mismatch that are difficult to identify from individual samples.Source: §1 · p. 1
- The central challenge is selecting synthetic data that provides useful supervision and improves performance on real data.Existing selection approaches often emphasize fidelity and diversity rather than the learner’s evolving needs, and full-pool feature processing can be resource-intensive.Source: §1 · pp. 1–2
- With 20–30% fewer synthetic samples, FROST improves absolute accuracy by 0.58–1.14% across 3 image generators and 0.9% on text-to-SQL over full synthetic pools.In industrial ads re-ranking, relative NE improves 0.096% over real-only training, reversing the 0.211% regression from the full synthetic pool.Source: §1 · p. 2
- FROST estimates utility through gradient feedback anchored in real training data and conditions sample-level filtering on batch-level utility.It calibrates batch utility against recent history, retaining moderate-utility batches directly and applying IQR-based filtering only to out-of-band batches.Source: §1 · p. 2
- FROST demonstrates effectiveness across image classification and LLM fine-tuning, including 3 synthetic image generators, and reports significant gains over a highly optimized production baseline.The industrial application is a large-scale ads re-ranking system.Source: §1 · p. 2
2 Related Work
Prior work uses synthetic data to expand supervision across vision, language, and recommendation tasks. Selection methods emphasize data properties or training dynamics, while FROST targets online selection of synthetic supervision as the model evolves.
2 Related Work
- Synthetic data expands supervision in vision, LLM pretraining and instruction tuning, and recommendation, where it can enrich sparse interactions or incomplete side information.Examples include generated images and text-to-SQL data, synthetic instruction–response pairs, and generated or transferred recommendation interactions.Source: §2 · p. 2
- Fidelity- and diversity-based selection uses semantic filtering, feature-space coverage, or distribution matching, typically with offline feature extraction and preprocessing.FROST instead estimates online whether a synthetic update would reduce current real-data loss, addressing scores that remain fixed as the model evolves.Source: §2 · p. 3
- Training-dynamics methods use losses or gradients for subset selection, but some select offline or use held-out target examples as the reference.FROST retains all real data during mixed training and uses a smoothed real-training gradient reference, with history-calibrated batch utility deciding when to filter synthetic samples.Source: §2 · p. 3
3 Method
FROST selects synthetic examples alongside real batches to reduce expected loss on real test data, using the current model, training data, and past statistics without quality labels or a held-out validation set. Its method estimates sample utility, routes batches by calibrated utility, and filters samples within out-of-band batches.
3 Method
- FROST selects a subset of each synthetic mini-batch to train alongside a real mini-batch, aiming to reduce expected loss on real test data.Selection uses the current model, available training data, and past training statistics, without data-quality labels or a separate held-out validation set.Source: §3 · p. 3
3.1 Estimating synthetic sample utility
- Synthetic-sample utility is estimated by aligning its gradient with a smoothed reference from real training batches; positive alignment predicts a first-order reduction in real-data loss.The reference is a bias-corrected EMA, and gradient alignment is approximated on a selected parameter block: the prediction head for image models and LLMs, or the final shared-backbone layer for ads re-ranking.Source: §3.1 · pp. 3–4
3.2 Batch-level calibration and routing
- The method calibrates aggregate batch utility against a rolling history and directly retains batches within a bounded moderate-utility band.The batch-utility normal approximation relies on i.i.d. candidate utilities with finite, nonzero variance conditional on a fixed model and reference; mixed training and filtering begin after an approximately 5% real-only warmup.Source: §3.2 · pp. 4–5
3.3 Sample-level IQR filtering
- For batches outside the utility band, FROST applies sample-level filtering using IQR-scaled thresholds based on empirical quartiles.The thresholds scale with the central spread of within-batch utilities, and retained samples train alongside the real batch.Source: §3.3 · p. 5
4 Experiments
FROST is evaluated on image classification, text-to-SQL, and industrial ads re-ranking, with comparisons to selection baselines and analyses of its two-stage design. Across the tasks, filtering synthetic data improves real-task performance over using the full synthetic pool, while ablations examine when batch routing and sample filtering help.
4.1 Image classification
- Absolute accuracy improves by 0.58%, 0.65%, and 1.14% over full synthetic-pool training on FLUX, SANA, and SD1.4, respectively.FROST also outperforms dynamic and synthetic-data selection baselines at matched drop ratios across all three generators.Source: Table 1 · p. 6, §4.1 · p. 6
4.2 LLM fine-tuning for text-to-SQL
- FROST reaches 65.7% EX while dropping 20.4% of synthetic data, outperforming selection methods at the same drop ratio and improving EX by 0.9% over the full pool.EX is execution accuracy on Spider 1.0 dev.Source: Table 2 · p. 7, §4.2 · p. 7
- The text-to-SQL ablations show that batch-only rejection drops 80% of synthetic data, while filtering every batch also underperforms full FROST.Out-of-band batches can contain useful samples for sample-level IQR filtering to retain, and batches that do not need cleanup can lose useful supervision when filtered.Source: Figure 3 · p. 8, §4.2 · p. 8
4.3 Ads re-ranking on real-world data
- FROST turns the full synthetic pool’s 0.211% NE regression into a 0.096% improvement over real-only training while dropping 36% of synthetic data.Lower relative NE is better; all real data are retained.Source: §4.3 · pp. 8–9, Table 3 · p. 9
- At similar drop ratios of 69% and 70%, the asymmetric band yields +0.074% relative NE change versus +0.198% for the symmetric band.Narrowing the symmetric band from [−1, 1] to [−0.25, 0.25] raises the drop ratio from 26% to 69% but does not consistently improve NE.Source: Figure 4 · p. 9, §4.3 · p. 9
- On ads, full FROST reaches −0.096% relative NE change at a 36% drop, compared with +0.074% at a 70% drop for batch gating alone and +0.036% at a 54% drop for sample IQR alone.Full FROST retains moderate-utility batches in full and applies sample-level filtering only to batches flagged by routing.Source: §4.3 · pp. 9–10
5 Conclusion
FROST filters synthetic data using utility estimated from gradients anchored in real training data, improving downstream performance with fewer synthetic samples. The conclusion reports gains across image classification, LLM fine-tuning, and industrial ads re-ranking.
5 Conclusion
- Figure 5 presents a two-stage ablation, with percentages denoting the synthetic-data drop ratio.Source: Figure 5 · p. 10
- The framework estimates synthetic-data utility from real-anchored gradient feedback without an external verifier or held-out validation set.It uses conditional sample-level filtering to select useful synthetic data.Source: §5 · p. 10
- FROST improves absolute accuracy with fewer synthetic samples on image classification and LLM fine-tuning, and gains over a production baseline in industrial ads re-ranking.In ads re-ranking, it flips the regression from the full noisy synthetic pool into an improvement over real-only training.Source: §5 · p. 10
A.1 Derivation of the Utility Approximation
The derivation relates a synthetic-gradient update to local real-loss change, while noting that utility scores alone do not guarantee the ordering of training benefits. A separate short-horizon experiment finds that the highest-utility band is not best at the examined checkpoint.
A.1 Derivation of the Utility Approximation
- A hypothetical SGD step using the synthetic-batch gradient is evaluated against the empirical loss on the current real mini-batch.The loss is viewed as a function of the scoring block, with other parameters held fixed.Source: §A.1 · p. 14
- Taylor’s theorem expresses the local loss change with a higher-order correction, while the current real gradient differs from its EMA reference.The derivation substitutes the current gradient as the EMA reference plus a deviation and separates reference-error and higher-order terms.Source: §A.1 · p. 14
- Utility alone need not preserve the ordering of local training benefits because reference-error and higher-order terms can increase or decrease benefit.A larger utility increases the first-order term but does not determine the other terms.Source: §A.1 · p. 14
- At the examined checkpoint, the High utility band has a higher early accuracy peak but lower mean real-test accuracy than Mid at the end of the 30-step window.The comparison concerns subsequent full-model training, whereas Equation 16 concerns a single update in the scoring block.Source: §A.1 · p. 14
A.2 Theoretical Guarantee for EMA Reference Tracking
The appendix bounds expected squared error between the bias-corrected EMA reference and the population real gradient under stated assumptions. The bound separates mini-batch noise from tracking lag, exposing a trade-off as the EMA history grows.
A.2 Theoretical Guarantee for EMA Reference Tracking
- The bound assumes unbiased mini-batch gradients, bounded conditional variance, an L-smooth population objective, and bounded parameter updates.The population real gradient is defined at step t in the scoring block.Source: §A.2 · pp. 14–15
- Under the stated assumptions, the expected squared tracking error of the bias-corrected EMA reference is bounded for fixed 0 ≤ β < 1 and t ≥ 1.Source: §A.2 · p. 15
- The proof unrolls the bias-corrected EMA as a weighted sum of past mini-batch gradients and decomposes its error into mini-batch noise and tracking lag.Source: §A.2 · p. 15
- The noise contribution is bounded using zero conditional mean, while smoothness and bounded updates bound the tracking lag.The lag bound is LSβ/[(1 − β^t)(1 − β)].Source: §A.2 · pp. 15–16
- In the long-history regime, high β suppresses the mini-batch noise contribution but introduces a tracking-lag term proportional to L^2S^2[β/(1 − β)]^2.Small parameter updates control this lag, supporting EMA use when the real-gradient trajectory changes slowly.Source: §A.2 · p. 16
B Implementation Details
The rolling-buffer window differs by task: image classification and ads re-ranking use a longer history than LLM fine-tuning.
B Implementation Details
- The rolling-buffer window is W = 200 batch scores for image classification and ads re-ranking, and W = 50 for LLM fine-tuning.Source: B Implementation Details · p. 16
B.1 Image classification
The image-classification pipeline builds synthetic CIFAR-100 examples from captions and rewrites, then evaluates training and feature-based selection procedures. The text-to-SQL setup specifies fine-tuning and an efficient token-level gradient-scoring computation.
B.1 Image classification
- CIFAR-100 training uses a CIFAR-style ResNet-18 for 200 epochs and evaluates accuracy on the 10,000 real test images.Training uses real and synthetic images, with SGD, momentum 0.9, and batch size 1,024.Source: §B.1 · p. 16
- LLaVA captions the 50,000 real training images, and Qwen produces three short rewrites that preserve the subject while varying scene details.The rewrites request a large, centered subject and simple background to keep the labeled object recognizable at 32 × 32 resolution.Source: §B.1 · p. 16
- Each of FLUX.1 [schnell] and SANA-1.6B generates four images per real image from a base caption and three rewrites, yielding 200,000 images per generator.A separate Stable Diffusion v1.4 pool contains 1,800 images per class and was not generated through this caption pipeline.Source: §B.1 · pp. 16–17
- CovMatch greedily matches selected synthetic-feature covariance to real-data covariance, while DS3 samples from synthetic clusters nearest to real examples.CovMatch projects features to 32 dimensions; DS3 uses k-means clusters, eligibility based on nearest real references, and uniform sampling to meet the selection budget.Source: §B.1 · pp. 17–18
B.2 LLM fine-tuning for text-to-SQL
- Text-to-SQL fine-tunes Qwen2.5-Coder-0.5B-Instruct on schema, knowledge when available, and questions, supervising gold SQL tokens while masking prompt tokens.Prompts are left-truncated as needed to preserve the target within 2,048 tokens.Source: §B.2 · pp. 18–19
- For text-to-SQL utility scoring, token-level output-projection gradients are computed without materializing per-example head-gradient matrices.Supervised tokens are processed in chunks of at most 256, bounding temporary storage while sharing one real-reference matrix.Source: §B.2 · p. 19
B.3 Computational overhead
FROST scores synthetic examples using head-level gradient computations and routes batches before applying sample-level filtering. The added computations are small relative to model training, while measured throughput reductions are modest.
B.3 Computational overhead
- For a linear prediction head, scoring m synthetic examples costs O(mCd), while reference aggregation costs O(|Rt|Cd) and the EMA update costs O(Cd).Head-level scoring uses O(Cd + (|Rt| + m)(C + d)) storage rather than an O((|Rt| + m)Cd) per-example gradient tensor.Source: §B.3 · p. 19
- Batch aggregation and calibration cost O(m + W), while sorting-based IQR filtering costs O(m log m) and is used only for out-of-band batches.Utility scores are computed for all synthetic candidates before routing.Source: §B.3 · p. 19
- For the CIFAR-style ResNet-18, head-level matrix arithmetic is much smaller than regular forward/backward training, though this comparison is not end-to-end runtime.The stated comparison is for convolution/linear arithmetic, not implementation overheads.Source: §B.3 · p. 20
- Training throughput falls by approximately 6.8% in image classification and 5.7% in industrial ads re-ranking with FROST.The measured overhead is dominated by memory traffic and synchronization rather than FLOPs.Source: §B.3 · p. 20
C.1 Two-stage ablation on Spider 1.0, by difficulty
On Spider 1.0 dev, full FROST improves overall execution accuracy over both single-stage variants while dropping less synthetic data. Its advantage holds across difficulty categories except that Sample IQR only scores higher on Hard.
C.1 Two-stage ablation on Spider 1.0, by difficulty
- On Spider 1.0 dev, Full FROST improves overall execution accuracy over Batch gate only and Sample IQR only by 5.0 and 2.6 percentage points, respectively.Table 5 defines Drop as the synthetic-data drop ratio (%) and EX as execution accuracy (%); all variants retain the real training data.Source: §C.1 · p. 20, Table 5 · p. 21
- On Extra Hard, full FROST gains 9.1 and 6.7 points over Batch gate only and Sample IQR only, respectively; Sample IQR only leads full FROST by 1.7 points on Hard but trails it in the other three difficulty categories.The comparison is the per-difficulty breakdown of the Spider 1.0 dev ablation.Source: §C.1 · p. 21, Table 5 · p. 21