PaperlayerView exact revision on arXiv

Source-linked AI summary

Effective Synthetic Data Curation Requires Group-Level Signals

Cathy Jiao, Chenyan Xiong

arXiv:2610.00779v1cs.CL

TL;DR

Synthetic-data training can degrade generation, making careful curation important, yet individual-level signals do not capture interactions among samples. The paper evaluates group-level curation and proposes a low-cost diagnostic for prioritizing groups to score; group-aware methods improve over baseline, especially in generative capability.

  • Problem

    Synthetic data can degrade LLM generation at scale, while individual-level curation estimates each sample’s utility in isolation and does not account for sample interactions.

    Source: §1 · p. 1

  • Method

    The paper compares individual- and group-level influence for synthetic-data curation across training settings and uses within-group gradient diversity to prioritize group-level scoring under a budget.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Results

    Group-level curation improves downstream performance, particularly generative capability; on free-form generation, group oracle selection achieves a +16.9% relative gain over the base model.

    Source: §1 · p. 2, §3.3 · p. 5

  • Takeaways & Limitations

    Within-group gradient diversity can help practitioners direct limited compute toward groups where individual-level signals are likely to mislead.

    Source: §1 · p. 2, §5 · p. 8

Abstract

from arXiv · show

Synthetic data now is essential to LLM training, used to strengthen advanced capabilities such as autonomous and long-horizon task execution. Yet recent work shows that training on it at scale can degrade model generation, making it important to decide what synthetic data is worth training on. While current data curation practices do so with individual-level signals (i.e., estimates of each data sample's training utility in isolation), across pre-training and post-training settings we show that this is insufficient for synthetic data, and that group-level signals (i.e., estimates of utility that account for interactions among data samples) are necessary for effective data curation. First, we show that individual-level signals are blind to how samples jointly affect training: synthetic datasets with different compositions can be indistinguishable under individual-level influence yet differ sharply under group-level influence, and curating by the latter yields better downstream performance, particularly in generative capability. Second, we find that group-level signals matter more as training pipelines become increasingly synthetic: among widely used data curation methods, only those incorporating them improve over baseline, with gains increasing when weights capturing relations among samples are amplified. Finally, we translate these findings into practice -- for model developers under a compute budget, we offer a cheap diagnostic that prioritizes which groups of synthetic data most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.

1 INTRODUCTION

Synthetic data can advance LLM capabilities but requires careful selection because large-scale training on it can degrade generation. The paper argues that sample-level utility estimates miss interactions, and presents group-level signals and a budget-conscious diagnostic for curation.

1 INTRODUCTION

  • Synthetic data is used across LLM training, from pre-training corpora to post-training for reasoning, and supplies targeted signal for capabilities difficult to learn from organic web data alone.Examples include long-horizon coding autonomy, competition-level mathematics, and reliable tool use.Source: §1 · p. 1
  • Training on synthetic data at scale can degrade LLM generation because synthetic data has narrower coverage than organic web data and inherits generator biases.Recovering from collapsed training runs wastes substantial compute.Source: §1 · p. 1
  • Group-level influence can distinguish synthetic datasets that look alike under individual-level influence, and selecting by group-level influence improves downstream performance, especially generative capability.The paper also finds that group-aware curation methods improve over baseline as training pipelines become more synthetic, with further gains when relational weights are amplified.Source: §1 · pp. 1–2
  • Within-group gradient diversity offers a cheap way to prioritize groups for group-level scoring, recovering much of its benefit at a fraction of the compute cost.The diagnostic targets groups where individual-level signals are likely to mislead.Source: §1 · p. 2

2 PRELIMINARIES

The paper frames data influence as oracle scoring against a reference set, then contrasts individual scores with group-level value. Synthetic-data curation is challenging because samples can be homogeneous while dataset value remains non-additive.

2 PRELIMINARIES

  • Oracle scoring measures how including or excluding a training set changes model performance on a reference set; for one example, it is the individual oracle score.The group oracle score applies to a set of examples, while the individual score is its single-example case.Source: §2 · p. 2
  • A reference set is chosen to represent a target capability, with its loss serving as a measurable proxy for that capability.Prior work found that selecting training data to reduce reference loss yields gains on corresponding downstream tasks.Source: §2 · p. 2
  • Individual-level curation scores each sample in isolation, while the individual proxy oracle approximates group value by aggregating individual oracle scores.Cheaper approximation methods such as LESS and LoGRA make the proxy tractable compared with the true group-level oracle.Source: §2 · pp. 2–3
  • Synthetic samples show lower variance than organic samples in token-level, information-density, and individual-level influence measures, making useful samples harder to distinguish.The paper attributes synthetic-data homogeneity in part to generation from a small set of seed prompts.Source: §2 · p. 3
  • A dataset’s value is non-additive because sample interactions can cancel or amplify contributions, effects invisible to individual scorers.The individual proxy oracle consequently may rank datasets differently from the group oracle.Source: §2 · p. 3

3 GROUP-LEVEL SIGNALS FOR SYNTHETIC DATA INFLUENCE

Across pre-training and post-training, controlled experiments show that group-level influence distinguishes synthetic data groups that individual-level scores treat similarly, and selecting by group oracle improves downstream performance. The advantage is strongest on generative tasks; the experiments compare matched training setups across both settings.

3.1 TASK SETTINGS

  • The study tests synthetic data pre-training and RLVR post-training, evaluating eight pre-training benchmarks and six reasoning, math, and coding benchmarks.The settings use Repro-400M with a FLAN reference set for pre-training, and Qwen2.5-1.5B-Instruct with a GPQA-extend reference set for post-training.Source: §3.1 · p. 4

3.2 EXPERIMENT SETUP

  • Candidate groups hold mean individual-proxy scores near a target while varying composition across 1 to 20 semantic clusters.The pool spans groups with similar proxy scores but different compositions; Figure 2 describes semantic and topic diversity.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The three selection methods are compared under matched budgets, with the same starting model, hyperparameters, and training steps.This design attributes downstream performance differences to selection alone.Source: §3.2 · p. 5

3.3 RESULTS

  • Across task settings and group sizes, group-oracle selection outperforms both random selection and individual-proxy selection.Table 1 reports per-task accuracy for pre- and post-training selection.Source: §3.3 · p. 5, Table 1 · p. 5
  • Groups with near-identical individual-proxy scores span a much wider range of group-oracle scores.Figure 3 compares group-oracle and individual-proxy scores for groups of 5120 and 128 samples.Source: §3.3 · p. 5, Figure 3 · p. 5
  • On free-form generation, group-oracle selection yields +16.9% relative gain over the base model, versus +6.3% for individual proxy and +4.7% for random selection.On multiple-choice tasks, all three methods are comparable, at +3.8% to +4.8%.Source: §3.3 · p. 5, Figure 4 · p. 6
  • During GRPO training, Pass@8 increases under group-oracle selection, while individual-proxy selection ends below its starting value.Figure 4b shows Pass@8 over post-training steps.Source: §3.3 · p. 6, Figure 4 · p. 6

4 GROUP-LEVEL SIGNALS FOR SYNTHETIC DATA CURATION

The section tests whether group-aware data selection helps as synthetic training data becomes more prevalent. Group-level methods outperform individual-level methods on generative tasks, and stronger relational weighting helps in a highly synthetic stress test.

4.1 EXPERIMENT SETUP

  • The experiment recycles noisy web text into clean synthetic pools, varying rephrase factor r across 1, 10, and 50 at a fixed pool size of 716,800 sequences.The study compares individual- and group-level selection methods and trains the same Repro-400M base model under identical optimization.Source: §4.1 · pp. 6–7

4.2 RESULTS

  • Group-level selection methods outperform individual-level methods on generative tasks, while all methods plateau on multiple-choice tasks.The comparison is shown as rephrase factor increases.Source: Figure 5 · p. 7, §4.2 · p. 7
  • Tripling GMRel’s relational weight at r = 50 improves relative downstream performance by 4.7%, surpassing random selection and baseline.This r = 50 setting is a stress test of highly synthetic training data.Source: §4.1 · p. 6, Figure 5 · p. 7, §4.2 · p. 7
  • At r = 1, GMRel and GREATS form more diverse groups at a slight cost to group oracle value, while GradSim selects high-value but least-diverse groups.The authors suggest that balancing value and diversity is important for effective synthetic data selection.Source: §4.2 · p. 7
  • As training pipelines rely more on synthetic and self-generated data, the authors argue that selection methods should model group effects directly to avoid suboptimal selection.Source: §4.2 · p. 7

5 DIAGNOSTICS FOR GROUP-LEVEL DATA SELECTION

The section characterizes when individual proxy scores may misrepresent group scores and uses within-group gradient diversity to direct limited group-oracle compute. Gradient-diversity diagnostics recover much of the group oracle’s ranking and recall with lower memory use.

Characterizing the oracle gap

  • As within-group gradient diversity increases, the gap between the individual proxy oracle and true group oracle widens.The section bounds this gap by within-group gradient diversity, the spread of per-sample gradients within a group.Source: Figure 6 · p. 8, §5 · p. 8

Gradient diversity as a budget allocator

  • Because high-diversity groups are where the proxy can be far off and misrank groups, the proposed strategy audits the B groups with the highest cheap diagnostic scores.The diagnostic score ν assigns larger values to groups with a larger predicted oracle gap.Source: §5 · p. 8
  • At a 20% group-oracle-call budget, GradDivfull recovers 75% of top-10% groups, compared with 20% under random allocation.Vendi and compression ratio reach 0.44 and 0.40 recall, respectively; token-level distinct-3 falls below random.Source: Figure 7 · p. 9, §5 · p. 9

Setup

  • The evaluation compares diagnostic rankings with the group oracle using top-k% recall and Spearman ρ; alternatives include gradient subsampling, gradient-free scores, and random allocation.GradDivq estimates gradient diversity from a uniform subsample of each group’s examples.Source: §5 · pp. 8–9

Gradient diversity as a budget allocator

  • At q = 0.1, GradDivq recovers 74% of group-oracle recall using 27% of its memory at a 20% budget.It gives similar ranking benefit to GradDivfull: 0.69 versus 0.75 recall and 0.40 versus 0.44 Spearman correlation at that budget.Source: §5 · p. 9

6 CONCLUSION

The conclusion argues that synthetic-data curation needs group-level signals because sample interactions affect utility and downstream performance. The included related-work and supplementary passages describe prior curation approaches and experimental setup details.

6 CONCLUSION

  • The paper concludes that group-level signals, which capture interactions among samples, are needed to identify synthetic data worth training on.Its findings motivate scalable curation methods that model relational structure directly.Source: §6 · p. 9

A RELATED WORK

  • Prior work uses synthetic data across pre-training, mid-training, post-training, and task or environment construction, while synthetic-data selection often scores generations individually.A smaller line of work scores generations against one another to promote diversity or limit redundancy.Source: A RELATED WORK · pp. 16–17
  • Group influence is not simply the sum of sample influences: samples may amplify or cancel one another, and some approaches model groups directly or include relational information.Other approaches aggregate per-sample scores or use cooperative-game formulations.Source: A RELATED WORK · p. 17

B SUPPLEMENTARY MATERIALS

B.1 ORACLE SCORING CONFIGURATION

  • Oracle scoring configuration uses one AdamW step on a candidate group from a trained model checkpoint and resets weights between candidate groups.The configurations cover pre-training and post-training settings.Source: §B.1 · p. 17, Table 3 · p. 17

B.2 PROXY-MATCHED DIVERSITY SAMPLING

  • Proxy-matched diversity sampling constructs groups within an individual proxy-oracle score range while varying diversity.Member selection mixes examples on both sides of a target score so the subset mean matches that target.Source: §B.2 · pp. 17–18, Table 4 · p. 18

B.3 DATA INFLUENCE SETUP

The pre-training and post-training experiments use small- and large-group configurations with different candidate-pool and selection sizes.

B.3 DATA INFLUENCE SETUP

  • Small-group settings use 1,000 candidate groups and select more groups than large-group settings, which use 500 candidate groups.Selection counts are 40 versus 20 for pre-training and 25 versus 7 for post-training; groups span whole numbers of optimizer steps, and the base model serves as both scoring and training initialization.Source: Table 5 · p. 19, §B.3 · p. 19

B.4 DATA CURATION SETUP

The appendix specifies the experimental settings, selection procedures, group construction, diversity measures, and computational accounting used in the curation experiments. It distinguishes sample-wise scorers from methods that rescore candidates in relation to the partially formed group.

B.4 DATA CURATION SETUP

  • The pool size is fixed across the data-curation settings, while rephrase factor r controls near-duplicate redundancy.Table 6 organizes settings by the three experimental phases.Source: Table 6 · p. 20, §B.4 · p. 20

B.5 DATA SELECTION METHODS

  • FineWeb-Edu scores educational value from 0 to 5, BM25 ranks lexical relevance to a reference set, and GradSim scores cosine similarity between candidate and reference gradients.FineWeb-Edu averages chunk scores per sequence; GradSim computes gradients at the oracle-scoring base checkpoint.Source: §B.5.1 · p. 21, §B.5.2 · p. 21, §B.5.3 · p. 21
  • GMRel learns relational influence weights from oracle rollouts, while GREATS discounts candidates aligned with gradients of samples already selected.GMRel’s relational model combines individual influence with relationships among group members; GREATS iteratively re-scores candidates against the selected group.Source: §B.5.4 · p. 21, §B.5.5 · pp. 22–23

B.6 GROUP FORMATION

  • Groups are formed from sampled clusters: individual-level methods take top-scored samples, whereas GMRel and GREATS repeatedly rescore candidates against the partial group.The procedure embeds sequences, partitions the pool into 30 clusters, samples 20 clusters, and forms one group of size 5120 per sampled cluster.Source: §B.6 · p. 23

B.7 DIVERSITY METRICS

  • The evaluated diversity measures cover gradient directions, embedding-based effective sample counts, compression redundancy, and unique pooled token n-grams.Higher metric values indicate more diverse groups; Vendi accounts for higher-order redundancy, while Distinct-2 and Distinct-3 capture repetition between samples.Source: §B.7 · pp. 23–24
  • Gradient diversity is one minus mean pairwise cosine similarity of normalized projected gradients, using directions rather than magnitudes.The cosine form prevents a few high-magnitude samples from dominating the metric.Source: §B.7 · p. 23
  • Vendi measures the effective number of distinct samples from the embedding-kernel eigenspectrum, while compression ratio measures redundancy exploitable by compression.Compression ratios near 1 indicate little shared material; values well below 1 indicate substantial restatement.Source: §B.7 · pp. 23–24

B.8.1 FLOPS ANALYSIS

  • The budget analysis compares group-oracle calls with per-sample gradient passes, and a gradient pass over fraction q of a group costs q times the full gradient-pass FLOPs.A per-sample gradient requires one forward and one backward pass; GradDivq samples a fraction q of each group.Source: §B.8.1 · p. 24

B.8.2 CUMULATIVE MEMORY ANALYSIS

The appendix models cumulative GPU memory for allocations that audit groups with the group oracle and optionally compute gradients across the pool. It also sets up the proof framework comparing averaged individual oracle scores with joint group scoring.

B.8.2 CUMULATIVE MEMORY ANALYSIS

  • Cumulative memory combines oracle-call memory with gradient-pass memory, whose relative resident-memory cost is κ = (2W + A)/(4W + A).A group-oracle call includes AdamW moments, while a gradient pass does not.Source: §B.8.2 · p. 25
  • The allocations share the same cost expression and differ in q, the fraction of each group receiving a gradient pass.The comparison includes group-oracle, random, GradDivfull, and GradDivq allocations.Source: §B.8.2 · p. 25
  • When R is much smaller than 3|G|, the cumulative-memory ratio is approximately H/B + κq and does not depend on absolute model size or GPU.Table 8 reports ratios for the stated pre-training pool configuration.Source: §B.8.2 · p. 25, Table 8 · p. 26

C.1 SETUP

  • The proof setup represents the proxy score as an average of loss changes at individual gradient steps and the group score as the loss change at the mean group gradient.The reference-set loss-change function is evaluated after the individual or group parameter displacement.Source: §C.1 · pp. 26–27

C.2 LEADING-ORDER CHARACTERIZATION OF THE GAP

The section characterizes the difference between scoring a group jointly and averaging its individual proxy scores. The leading discrepancy is second-order and depends on reference-loss curvature and within-group gradient covariance.

C.2 LEADING-ORDER CHARACTERIZATION OF THE GAP

  • The derivation expands the loss-change function around zero displacement and subtracts the individual-proxy and group-oracle expansions.The section presents the leading-order characterization before establishing an exact bound in Theorem 5.1.Source: §C.2 · pp. 27–28
  • The leading-order gap between the individual proxy oracle and group oracle is a second-order term involving reference-loss curvature and within-group gradient covariance.The linear terms cancel, so the two scores agree to first order in the gradient-step size.Source: §C.2 · pp. 27–29
  • The gap vanishes when the group has zero within-group gradient covariance or the reference loss is locally flat.The discrepancy grows with both curvature and within-group gradient diversity.Source: §C.2 · pp. 27, 29

C.3 PROOF OF THE EXACT BOUND

The appendix proves an exact bound on the individual-versus-group oracle gap under β-smoothness and supplies additional experimental materials. The included captions identify full curation results, group-size discrepancy sweeps, cluster-topic composition, and diversity plots.

C.3 PROOF OF THE EXACT BOUND

  • Under β-smoothness, the leading-order gap characterization becomes an exact bound, with the proof using a second-order Jensen-gap argument and within-group gradient diversity.The bound is exact in the gradient-step size; first-order agreement follows separately.Source: §C.3 · pp. 29–31

D ADDITIONAL FIGURES AND TABLES

D.2 GROUP VERSUS INDIVIDUAL PROXY ORACLE GAP ACROSS GROUP SIZES

  • The group-versus-individual oracle discrepancy is shown as a full sweep across group sizes, complementing the main-body summary.Figure 8 compares scores across varying group sizes, and Figure 9 shows their ranges across constructed groups.Source: §D.2 · p. 33, Figure 8 · p. 33, Figure 9 · p. 33

D.3 CLUSTER TOPIC COMPOSITION

  • Figure 10 encodes the topic and corpus share of each of the 20 semantic clusters, with color indicating topic family.Source: Figure 10 · p. 33

D.4 ADDITIONAL DIVERSITY PLOTS

  • Figures 11 and 12 compare diversity metrics on constructed groups with randomly formed groups in pre-training and post-training, respectively.Source: Figure 11 · p. 34, Figure 12 · p. 34

D.5 GROUP ORACLE BUDGET ANALYSIS PLOTS

This section presents group-oracle budget-allocation plots for pre-training and post-training data, covering diagnostic choice and gradient subsampling.

D.5 GROUP ORACLE BUDGET ANALYSIS PLOTS

  • The full set of group-oracle budget-allocation plots is summarized in Figure 7 of the main body.This section contains the full set of plots.Source: §D.5 · p. 35
  • Figure 13 compares GradDiv with cheap diversity heuristics for choosing which groups receive a group-oracle call.The comparison covers pre-training and post-training data, with group sizes of 5120 and 512 samples, respectively.Source: §D.5 · p. 35, Figure 13 · p. 35
  • Figure 14 compares the full GradDiv diagnostic with gradient-subsampled variants at q = 0.1 and q = 0.01.The comparison covers pre-training and post-training data, with group sizes of 5120 and 512 samples, respectively.Source: §D.5 · p. 35, Figure 14 · p. 35
Loading 2610.00779v1…