PaperlayerView exact revision on arXiv

Source-linked AI summary

Training-Aware Target Coverage for Synthetic Data Selection

Yang Ba, Michelle V. Mancenido, Rong Pan

arXiv:2610.00814v1cs.LGcs.AI

TL;DR

Synthetic examples can add useful target-relevant information, but redundancy and response errors make their value dependent on the existing training set. The paper develops a linear theory and TATC, which selects examples by training effect and complementary target coverage; experiments validate the theory and report TATC outperforming competing methods on GSM8K.

  • Problem

    Synthetic examples can be redundant or introduce harmful errors, and existing data-selection perspectives do not jointly determine an example’s contribution to the training set and target task.

    Source: §1 · pp. 1–2

  • Method

    The paper develops a linear theory of information-error tradeoffs and introduces TATC, which filters for beneficial training effects and selects complementary target coverage.

    Source: §1 · p. 2

  • Results

    Text and image experiments validate the linear theory; on GSM8K, 87.34% exact match accuracy at B=128 exceeds OptimSyn (86.07%) and TATC (Coverage) (85.19%).

    Source: §3.3 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    An example’s value depends on the information and errors already present, supporting selection in the context of existing data and downstream tasks.

    Source: §6 · pp. 9–10

    The experiments do not establish suitability for high-stakes or safety-critical applications, and TATC does not replace factual verification, safety evaluation, or appropriate data governance.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Synthetic data are increasingly used to scale LLM training, yet more synthetic data do not necessarily produce better models. Useful synthetic data must add information relevant to the target task without introducing errors that offset their benefit, and the value of an example can change as the training set grows. We develop a linear theory that characterizes this tradeoff and determines where synthetic data are useful, how much should be added, and the marginal value of adding one example to an existing set. The analysis shows the conditions when input coverage alone is sufficient and when synthetic errors must also be considered. Guided by these results, we introduce \emph{Training-Aware Target Coverage} (TATC), a synthetic data selection method for LLM fine-tuning. TATC identifies candidates whose training effects are beneficial to the target task and selects among them to expand coverage of target-relevant directions not already represented by the available data. Experiments on text and image data verify the linear theory. With mathematical reasoning tasks, TATC selects synthetic solutions for fine-tuning Qwen2.5-Math-1.5B-Instruct and outperforms alternative synthetic-data selection methods on GSM8K across selection budgets. In summary, we provide a principled approach to synthetic data selection by quantifying and maximizing its value to the target task.

1 INTRODUCTION

Synthetic data selection must account for both the information an example adds and the training effects of its response, with value depending on the existing dataset. The paper develops a linear theory and introduces TATC, which combines beneficial training effects with complementary target coverage; experiments validate the theory and show TATC outperforming competing methods on GSM8K.

1 INTRODUCTION

  • Synthetic examples can be redundant or harmful, so their value depends on which examples are already in the training set.Existing selection approaches capture complementary properties but do not by themselves answer how an example contributes to the existing set and target task.Source: §1 · pp. 1–2
  • The linear theory makes synthetic-data value directional, amount-dependent, and marginal, jointly balancing added information against introduced errors.Useful data may be absent, finite, or continue to increase along a direction; the marginal benefit depends on real data and previously selected synthetic examples.Source: §1 · p. 2
  • TATC first retains candidates with beneficial training effects, then selects those adding target-relevant coverage absent from real and previously selected data.A small labeled real probe set supports this set-dependent selection.Source: §1 · p. 2
  • TATC outperforms competing synthetic-data selection methods on GSM8K across selection budgets.The experiments fine-tune Qwen2.5-Math-1.5B-Instruct using synthetic mathematical solutions.Source: §1 · p. 2
  • Experiments on text and image data validate the theory, while TATC combines error-aware training effects with complementary target coverage.The theoretical framework characterizes downstream benefit jointly through information added by synthetic inputs and errors they introduce.Source: §1 · p. 2

2 RELATED WORK

Prior work selects synthetic data using quality, diversity, distribution matching, experimental design, or estimated training utility. This paper distinguishes its approach by jointly considering input information, response error, and set-dependent marginal value.

Synthetic inputs and generated response error

  • Recursive synthetic training can amplify errors (Alemohammad et al., 2024), and this work jointly models input information, systematic response error, and excess response noise.Source: §2 · pp. 2–3

Quality and validation-guided data selection

  • AlpaGasus filters by model-based quality scores (Chen et al., 2024), while DEITA combines complexity, quality, and diversity (Liu et al., 2024); this work separates training effect from set-dependent feature coverage.Source: §2 · p. 3

Diversity and experimental design

Gradient and influence selection

  • LESS uses optimizer-aware gradient similarity (Xia et al., 2024), while OptimSyn estimates training influence (Fan et al., 2026); this work analyzes input information with response error to assess marginal value.Source: §2 · p. 3

3 DOWNSTREAM RISK THEORY IN SYNTHETIC DATA SELECTION

The linear theory evaluates synthetic data by balancing target-relevant information against label errors. It characterizes value at the dataset, directional, and candidate levels.

3.1 RISK OF TRAINING WITH SYNTHETIC DATA

  • Synthetic-set value is downstream information gain minus the costs of random label noise and systematic label error.The risk decomposition shows that inputs affect risk through the information matrix, while synthetic-label errors contribute through both systematic error and random variation.Source: §3.1 · p. 4, §3.2 · p. 5

3.2 VALUE OF SYNTHETIC DATA

  • A direction’s benefit depends on its downstream relevance, added information, and label errors, so its optimal synthetic amount can be zero, finite, or unbounded.When a downstream-relevant direction is absent from real training data, synthetic labels alone cannot distinguish target signal from systematic error; additional labeled real data covering it can resolve the ambiguity.Source: §3.2 · pp. 5–6

3.3 MARGINAL VALUE OF A SYNTHETIC EXAMPLE

  • A candidate’s marginal value depends on the selected set, incorporating variance changes, label error, and its interaction with existing bias.Input coverage alone is sufficient only with clean labels; otherwise candidate value also depends on label error.Source: §3.3 · p. 6
  • On SetFit/subj, the complete marginal closely predicts refitting’s risk reduction while coverage alone misses harmful candidates; on CIFAR-10, probe-selected amounts track the test optimum as label reliability falls.The input-only rule selects the full CIFAR-10 candidate pool.Source: §3.3 · p. 7, Figure 2 · p. 7

4 TRAINING-AWARE TARGET COVERAGE IN SUPERVISED FINE-TUNING

TATC carries the linear theory’s distinction between target-relevant input coverage and response-dependent training effects into nonlinear LLM fine-tuning. It uses a real labeled probe to rank candidate effects and select complementary coverage, with a fixed-checkpoint approximation.

4 TRAINING-AWARE TARGET COVERAGE IN SUPERVISED FINE-TUNING

  • TATC combines target coverage with a local estimate of each complete synthetic example’s training effect because input coverage alone may miss response errors.The coverage score measures target-weighted prediction-variance reduction; the training-effect score estimates the example’s effect on loss over a labeled real probe.Source: §4 · pp. 7–8
  • The real training set sets the checkpoint and representation geometry, while a disjoint labeled real probe measures target relevance and predicted training effects.Synthetic candidates are represented at the real-data-trained checkpoint.Source: §4 · p. 7
  • Target coverage measures how much a candidate input covers probe-relevant directions not already covered by the real training data and selected examples.Because the score depends on the input, it does not account for the effect of the synthetic response.Source: §4 · p. 8
  • TATC uses the training-effect score to shortlist candidates, then selects within that shortlist by the candidate’s additional coverage given the selected set.The scores have different roles and are not on a common scale: the training-effect score is computed once at the initial checkpoint, while coverage changes with the selected set.Source: §4 · p. 8
  • At each step, TATC chooses the highest-coverage candidate among the top min{κs, N} unselected examples ranked by training effect, and stops after budget B selections.The shortlist expands with step s; experiments use κ = 2.Source: §4 · p. 8

5 EXPERIMENTS

On GSM8K, TATC outperforms OptimSyn at budgets 64, 128, and 256, while their means are effectively tied within run-to-run variation at 512. At B = 128, it also exceeds the coverage-only ablation.

Experimental setup

  • The evaluation uses Qwen2.5-Math-1.5B-Instruct on GSM8K, with a real-data-trained checkpoint shared by every method and exact match measured on a holdout set.A disjoint set of 512 labeled real examples serves as the target probe, and 1,024 examples form the holdout; each selected subset is trained for one epoch from the shared checkpoint.Source: §5 · pp. 8–9

Results

  • 87.34% exact match accuracy for TATC at B = 128 exceeds OptimSyn (86.07%) and TATC (Coverage) (85.19%).The 2.15-point gap over the coverage-only ablation persists at larger budgets, reaching 2.99 points at B = 512; TATC is higher than OptimSyn at 64, 128, and 256, while their means are effectively tied within run-to-run variation at 512.Source: Figure 3 · p. 9, §5 · p. 9

Setup and baselines

  • Methods are compared at budgets B ∈ {64, 128, 256, 512} against Random, NLL, diversity-, influence-, covariance-matching, and information-based baselines.TATC (Coverage), which removes the training-effect component, is included as an ablation.Source: §5 · p. 9

6 CONCLUSION

The paper argues that synthetic-example value depends on the data already available and the information an example adds. It presents TATC as a practical selection approach and reports effectiveness for synthetic mathematical reasoning data.

6 CONCLUSION

  • A synthetic example’s usefulness depends on the information and errors already present in the training data, so the same example can be valuable, redundant, or harmful.The paper notes that increasing synthetic data can eventually yield diminishing benefit or become detrimental.Source: §6 · pp. 9–10
  • TATC applies this context-dependent view to LLM fine-tuning by considering both an example’s training effect and how it complements available data.The experiments demonstrate its effectiveness for selecting synthetic mathematical reasoning data.Source: §6 · p. 10
  • The authors argue that synthetic-data pipelines should evaluate examples in the context of existing data and the downstream tasks for which they are used.Source: §6 · p. 10

AI USE STATEMENT

The authors report using generative AI tools for manuscript and research-workflow assistance, while retaining responsibility for the final content.

AI USE STATEMENT

  • Generative AI tools assisted with manuscript drafting and editing and with literature organization.Source: AI USE STATEMENT · p. 10
  • The tools also assisted with code development and debugging, figure and table preparation, and review of theoretical and methodological exposition.Source: AI USE STATEMENT · p. 10
  • The authors independently checked derivations, sources, code, and results, did not treat AI output as evidence or validation, and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement limits the work’s claims about application scope, while the appendix proofs derive directional and marginal-value results under stated linear-model assumptions. The proofs also show that without labeled real evidence in a task-relevant direction, synthetic-data value can be ambiguous.

ETHICS STATEMENT

  • Experiments do not establish TATC’s suitability for high-stakes or safety-critical applications, and selection does not replace factual verification, safety evaluation, or data governance.The work uses public mathematical-reasoning and standard classification datasets and released model weights.Source: ETHICS STATEMENT · p. 10

A MATHEMATICAL PROOFS IN LINEAR THEORY

  • The linear proofs assume finite second moments, a real and synthetic linear label model, zero-mean random noise, and nonrandom systematic synthetic-label error.The proof conditions also restrict selection to fixed subsets or choices independent of realized noise innovations.Source: A MATHEMATICAL PROOFS IN LINEAR THEORY · p. 13

A.1 PROOF OF PROPOSITION 1

  • The Proposition 1 proof expresses conditional downstream risk as a squared-bias term plus a noise-variance trace term.The cross term vanishes under the conditional zero-mean noise assumption.Source: §A.1 · p. 13

A.2 PROOF OF COROLLARY 2

  • The Corollary 2 proof specializes the noise matrix under isotropic real and synthetic noise, then obtains the selected-set value by subtracting its risk from real-only risk.The synthetic noise covariance includes an additional τ^2 component.Source: §A.2 · p. 14

A.3 PROOF OF PROPOSITION 3

  • The Proposition 3 proof decomposes information and excess-noise terms by direction and evaluates systematic error, without requiring alignment between the relevant matrices.The trace decompositions rely on diagonal factors and use only the diagonal entries of the transformed matrix.Source: §A.3 · p. 14

A.4 PROOF OF COROLLARY 4

  • The Corollary 4 proof determines whether a directional contribution is positive by rearranging its sign expression under positive direction and target weights.The supplied proof passages do not state the resulting threshold explicitly.Source: §A.4 · p. 15

A.5 PROOF OF COROLLARY 5

  • The directional optimum can be zero, finite, or approached only as synthetic information grows without bound, depending on the noise and error regime.When the finite-maximizer conditions hold, the objective increases up to a unique global maximum and then decreases.Source: §A.5 · p. 15

A.6 PROOF OF PROPOSITION 6

  • Without labeled real evidence in a downstream-relevant direction, observationally equivalent models can assign different target signals and opposite values to synthetic data.The sign-reversal construction is existential and requires additional conditions; observational equivalence alone does not imply sign reversal for every fixed observed distribution.Source: §A.6 · pp. 15–16

A.7 PROOF OF THEOREM 7

Theorem 7 derives a single-candidate marginal from changes in bias and variance, with a clean coverage-only specialization. Controlled tests support the theory, and in the text experiment the complete marginal predicts candidate value better than coverage alone.

A.7 PROOF OF THEOREM 7

  • Under the clean homoscedastic specialization, the proof recovers the coverage marginal by setting bias and systematic error to zero and candidate noise variance to σ^2.The stated specialization also assumes N = σ^2A.Source: §A.7 · p. 17
  • Theorem 7’s single-candidate marginal combines changes in squared bias and variance, mirroring the information-gain, excess-noise, and bias-cost tradeoff.Its rank-one update assumes fresh candidate noise uncorrelated with existing label noises and a positive-definite A.Source: §A.7 · pp. 16–17

B CONTROLLED TESTS OF THE LINEAR THEORY

  • The controlled tests hold candidate inputs fixed while varying training targets, using frozen Qwen2.5 and ResNet-50 representations with ordinary least squares.Figure 2 compares exact rank-one marginal predictions with before–after fits, and probe-selected synthetic-to-real ratios with held-out test optima.Source: B CONTROLLED TESTS OF THE LINEAR THEORY · p. 17

B.1 CANDIDATE VALUE IN TEXT DATA

  • The text experiment compares an input-only clean coverage marginal with Equation (5), then checks both scores against risk reduction after refitting.Each context averages 3,000 independent label-noise trials.Source: §B.1 · p. 18
  • Coverage alone barely distinguishes beneficial from harmful text candidates, whereas the complete marginal predicts the sign and magnitude of observed risk reduction.Table 2 presents agreement with observed risk reductions in the text experiment.Source: Table 2 · p. 18, §B.1 · p. 18

B.2 SYNTHETIC-DATA AMOUNT IN IMAGE DATA

The image experiment evaluates synthetic-data amount selection across pseudo-label reliability conditions using CIFAR-10 feature representations. A probe-based estimate tracks the test-optimal amount across those conditions.

B.2 SYNTHETIC-DATA AMOUNT IN IMAGE DATA

  • CIFAR-10 images are represented by frozen 2,048-dimensional ResNet-50 features, with PCA fitted on real training images and applied unchanged to other partitions.Each of four partitions assigns images to real training, probe, label-model fitting, candidate, and unused sets; the official test split remains separate.Source: §B.2 · p. 19
  • Five fitted label models vary pseudo-label reliability through smaller training subsets or removal of task-relevant PCA coordinates; their pseudo-labels are deterministic.For the coordinate-removed condition, directions are ranked by explained variance weighted by reference-classifier coefficients, concentrating errors in task-relevant directions.Source: §B.2 · p. 19
  • The probe-based amount rule minimizes the sum of estimated bias and variance risks, while dropping the bias term makes the comparator select all 24,000 candidates.Source: §B.2 · p. 19
  • As label reliability falls, the probe estimate tracks the decrease in useful synthetic-data amount.For the most reliable label model, the test optimum on the evaluated grid occurs at the largest ratio.Source: §B.2 · p. 19
  • Kendall τ = 1.00 across the five conditions, and every predicted amount falls within the 1% test-optimal plateau.The predicted and test-optimal amounts have the same ordering; for the most reliable label model, the test optimum on the evaluated grid is the largest ratio.Source: §B.2 · p. 19, Table 3 · p. 20

C BASELINE IMPLEMENTATIONS

The baselines use prompt or gradient representations and implement different selection criteria, including quality ranking, diversity, target alignment, and coverage.

C.1 REPRESENTATIONS AND SCORES USED BY THE SELECTORS

  • Prompt features average final-layer hidden states across prompt tokens, then use a 64-dimensional PCA projection fitted on real-training representations.The same projection is used by Embedding Vendi, Covariance Matching, TATC (Coverage), and TATC's coverage stage.Source: §C.1 · p. 20
  • Gradient features are CountSketched to 512 dimensions and reduced to 64 by PCA; G-Vendi and BAIT use projected features, while LESS uses the sketches before PCA.Source: §C.1 · p. 20
  • Candidate negative log-likelihood is the response's mean token loss under the real-data checkpoint, with lower values indicating responses that checkpoint considers more likely.Source: §C.1 · p. 20

C.2 SELECTION RULES

  • Random samples uniformly, NLL returns candidates in increasing response loss, and Covariance Matching targets the real-training input second moment without labeled probe responses.Source: §C.2 · pp. 20–21
  • Embedding Vendi and G-Vendi optimize diversity through one-swap rounds on prompt or gradient features, while BAIT uses an A-optimal trace criterion in forward–backward selection.Vendi methods test 256 proposed replacements in each of 32 rounds; BAIT grows to 2B candidates before removing examples until B remain.Source: §C.2 · pp. 20–21
  • LESS ranks target-gradient alignment, OptimSyn scores first-order probe effects of AdamW updates, and TATC (Coverage) removes the training-effect filter.TATC (Coverage) isolates the response-aware training-effect component beyond target coverage.Source: §C.2 · pp. 20–21

D.1 FROM THE LINEAR ANALYSIS TO TATC

TATC combines target coverage with a local score for each complete example's training effect. The section specifies its scoring rationale, sequential selection procedure, and GSM8K evaluation setup.

D.1 FROM THE LINEAR ANALYSIS TO TATC

  • TATC uses a training-effect score to form a shortlist, then selects within it by set-dependent target coverage; the scores are computed at a fixed checkpoint and are not on a common scale.The coverage criterion is an experimental-design approximation to downstream fine-tuning performance, even though the stated algebraic identity is exact within the fixed prompt representation.Source: §D.1 · pp. 21–22

D.2 QUADRATIC TRAINING SCORE

  • The quadratic training score rewards update alignment with decreasing probe loss and penalizes displacement; saved AdamW moments affect the candidate-versus-zero update difference.Common weight decay cancels from the update difference, while the saved optimizer moments still affect it.Source: §D.1 · p. 22, §D.2 · p. 23
  • The displacement reduces probe loss whenever its score exceeds the stated bound, whose error terms depend on curvature approximation and the Taylor remainder.The curvature term reflects error from the surrogate, and the cubic term controls the Taylor remainder.Source: §D.2 · p. 23

D.3 SEQUENTIAL TATC SELECTION

  • One sequential run yields nested selected-set prefixes for each budget, but every prefix is trained independently from the same real-data checkpoint and optimizer state.Source: §D.3 · p. 24

D.4 GSM8K PROTOCOL

  • The GSM8K protocol uses Qwen2.5-Math-1.5B-Instruct, OpenR1-Math-220k trajectories, four selection budgets, and three independent training repetitions.The synthetic pool contains 4,096 deduplicated trajectories, and TATC does not use verifier status; absolute exact-match values across panels are not comparable because generation allowances differ.Source: §D.4 · p. 24
Loading 2610.00814v1…