High-Value Training Data from Noisy, Diverse Sources

• Heterogeneous training data can include low-quality or missing examples, distribution shifts, and mismatches between training objectives and downstream evaluation.Source: §1 · p. 1
• Heuristic filtering, label synthesis, and data mixing often require problem-specific design and potentially unreliable human judgments.Source: §1 · p. 1
• The method uses metadifferentiation to learn dataweights and parametric rules that transfer to new data and substantially larger models.Source: §1 · p. 2
• Learned dataweights improve performance by upweighting helpful examples and downweighting harmful ones, including outliers and adversarial examples.Source: §1 · p. 2
• Dataweights identify examples relevant to specialized targets such as coding or STEM question answering within diverse training data.Source: §1 · p. 2
• With mixed human and synthetic labels, dataweights improve performance over human-only data and equally weighted human-plus-synthetic data.They adapt to synthetic-label quality by upweighting helpful labels and downweighting harmful ones.Source: §1 · p. 2
• For reward modeling, dataweighting can optimize model-ranking accuracy while retaining the standard Bradley–Terry training objective.Source: §1 · p. 2

2.1 Problem statement

• The objective is to choose weights that make training on heterogeneous data perform well on a small, curated gold-standard target set, but direct optimization is computationally infeasible.Source: §2.1 · pp. 2–3

2.2 Scalable dataweighting

• Metagradients measure how training-example weights affect the target objective through the training process, enabling metagradient descent with nonnegative clipping.Source: §2.2 · pp. 3–4
• A parametric weighting map reduces optimization dimensionality and assigns weights to unseen or newly collected examples.The linear parameterization optimizes ψ ∈ R^(d+1) rather than one weight per training example.Source: §2.2 · p. 4
• Segmenting training lets the method update weights more frequently, while increasing the segment size reduces optimization cost linearly and can harm performance near full-run segments.The experiments use K = 1, 10, or 20 depending on the setting.Source: §2.2 · p. 4
• Weights learned with a less costly proxy algorithm, such as LoRA fine-tuning a 7B model, can be used to train a larger model, such as a 70B model.Source: §2.2 · p. 5
• The target objective can encode differentiable downstream behavior, provided evaluating and differentiating it costs less than T/K training steps.Source: §2.2 · p. 5

3.1 Experimental setup

• The experiments use Arena human, bot, and synthetic preference data, with Expert and STEM target sets split into optimization targets and held-out tests.Reward models are generally fine-tuned from Qwen2.5-0.5B-Instruct, with matched seeds for weighted and unweighted comparisons.Source: §3.1 · pp. 5–6

3.2 Establishing the efficacy of dataweighting

• 0.578 held-out STEM loss with dataweighting beats TRAK’s 0.619, and dataweighting reduces test loss 15% versus unweighted Human-155k training.Dataweighting reweights the full pool, while selection baselines retain a fraction of it.Source: Figure 1 · p. 6, §3.2 · p. 6

3.3 Revisiting our motivating challenges

• Dataweighting lowers Expert loss across human-data sizes, recovers performance with bot-contaminated data in almost all cases, and reduces STEM test loss from 0.6631 to 0.5572.The STEM target uses an annotation process substantially different from Arena’s crowdsourced data.Source: §3.3 · pp. 8–9
• With dataweighting, mixed human and synthetic supervision performs better than human-only training, whereas unweighted mixing does not beat the human subset alone.Under unweighted training, synthetic-only data performs poorly; weighting brings its performance closer to human data.Source: §3.3 · p. 9

3.4 Parametric dataweights and larger-scale transfer

• Linear dataweights match per-sample losses with roughly tenfold fewer parameters and improve over unweighted training when transferred across model scales and to unseen Human-155k examples.The weighting rule learned on a smaller base can be applied to new examples using embeddings and a forward pass, without further metagradient optimization.Source: §3.4 · pp. 9–10

3.5 Optimizing a complex target objective

• Rank-targeted dataweights with standard Bradley–Terry training improve Rank MAE and Spearman ρ over unweighted, BT-targeted, and direct soft-Spearman training.The results suggest that Bradley–Terry fits local reward signals while the rank surrogate selects a trajectory for downstream ordering.Source: §3.5 · p. 11, Table 1 · p. 11
• 0.915 held-out Spearman ρ with a map learned on 50k pairs approaches 0.925 from non-parametric weights, versus 0.648 unweighted, across 92k training pairs.The map predicts weights for all training pairs, and larger meta-training subsets improve the result.Source: Figure 6 · p. 11, §3.5 · p. 11

Correcting biased labels

• As source data grows, dataweighting risk converges to zero while pretraining followed by target fine-tuning remains bounded away from zero under the stated biased-label construction.The construction allows arbitrarily small source-label errors; the target set is too small to fully remove the resulting spurious dependence.Source: §4 · p. 12

Setting and intuition

• Upweighting examples where the labeling functions agree yields zero asymptotic risk, whereas unweighted source training retains spurious features that target fine-tuning cannot unlearn.A finite validation set can select agreeing examples, or examples with comparable population risk.Source: §4 · pp. 12–13

Filtering severe corruption

• As label corruption grows, dataweighting selects an entirely clean subset with probability tending to one, conditional on enough clean examples existing.The probability of error is exponentially small in the source-pool size.Source: §4 · p. 13

Adapting to covariate shift

• Even with correctly labeled source data, selecting examples can favor target-relevant features when source and target covariances differ.The stated construction uses an orthogonal change of basis.Source: §4 · p. 13

5 Related work

• Unlike methods that use attribution scores or metagradients to select training examples, this approach optimizes continuous example weights directly in the training loss using exact metagradients of clean-target performance to downweight noisy and irrelevant data.Source: §5 · p. 14
• Rather than seek invariant representations or tune domain-level mixture weights, the method uses a small, high-quality target set to reweight heterogeneous examples toward the target distribution.Prior language-model work tunes mixture weights over domains [XPD+23].Source: §5 · p. 14
• The approach uses target examples to determine each training example’s influence, differing from robust methods that detect corruption through internal inconsistency.It is also applied to noisy, adversarial, synthetic, and mismatched preference data.Source: §5 · p. 14

A Full algorithm

• The full algorithm chooses a proxy training algorithm, target objective, segmentation, and parametric weighting rule, then iteratively updates weights through training runs.It can compute weights for all training samples and return a final model trained with them.Source: A Full algorithm · p. 18

B Dataset construction and reward model training

• Human-155k contains 155,577 single-turn, non-tie human preference rows, and Bot-10k contains 10,000 examples flagged for bot or adversarial activity.The human rows come from arena-human-preference-100k and arena-human-preference-140k [CZS+24].Source: B Dataset construction and reward… · p. 18
• Gemini-155k replaces Human-155k labels with Gemini 3 Flash judgments that achieve 0.6812 accuracy against the original human labels.The model selects the preferred response for each prompt and response pair.Source: B Dataset construction and reward… · p. 18
• Expert evaluates in-distribution performance on a niche, heavily filtered Arena subset, while STEM evaluates out-of-distribution performance using expert-labeled or verified STEM preferences.Expert target/test splits contain 3,000/547 examples; STEM splits contain 6,796/6,796.Source: B Dataset construction and reward… · pp. 18–19
• Reward models are trained with the Bradley-Terry objective, usually from Qwen2.5-0.5B-Instruct, with a separate 100-example set used for early stopping.The authors leave alternatives such as cross-validation to future work.Source: B Dataset construction and reward… · p. 19

B.1 LLM-as-a-judge prompt

• The synthetic-label prompt asks Gemini to choose which response better satisfies user intent and to judge helpfulness, relevance, accuracy, depth, and creativity when applicable.The prompt requests the winner as either A or B and instructs impartial evaluation based on response quality and user needs.Source: §B.1 · p. 19

C.1 Ranking performance on held-out models

• Across nine holdout splits, Spearman-targeted meta-optimization shifts the distribution toward higher held-out Spearman ρ and lower Rank MAE than unweighted Bradley-Terry training or BT-targeted meta-optimization.Figure 8 compares the three approaches across model-disjoint splits.Source: Figure 8 · p. 20

C.2 Alternative rank surrogates

• Smooth footrule matches or slightly exceeds soft-Spearman on held-out ρ on one split (0.965 vs. 0.957), while soft-Kendall-τ is consistently weaker.The paper uses soft-Spearman by default.Source: §C.2 · p. 20

D.1 Leveraging heterogeneous data

• For Human-155k training targeted to Expert, the Human-155k example in Figure 9 improves test loss initially, but further meta-gradient steps overfit the target.Figures 10 and 11 show test and target loss by segment for Human-45k and Human-77k models targeting Expert, without a stated outcome in their captions.Source: Figure 9 · p. 21, Figure 10 · p. 21, Figure 11 · p. 22

D.2 Downweighting noisy and adversarial data

• Bot dataweights shift downward throughout the dataweighting process, as shown in the human-versus-bot weight distributions.Figure 12 shows KDE ridges across checkpoints; Figure 13 gives the corresponding KDE for STEM-targeted weights.Source: §D.2 · p. 22, Figure 12 · p. 22, Figure 13 · p. 23
• Expert-targeted trajectories report notable overfitting with Human-60k + Bot-10k and Human-100k + Bot-10k.The figures show test and target loss by segment for each training pool.Source: Figure 17 · p. 25, Figure 18 · p. 25

D.3 Customizing to specific domains

• Figure 19 plots target loss by segment for a Human-155k model targeting STEM.The caption identifies the pool, target domain, and plotted quantity but does not state a comparative outcome.Source: Figure 19 · p. 26

D.4 Incorporating synthetic data

• Figures 20–25 show synthetic-only and mixed-data loss trajectories targeting Expert, including Gemini-110k and human-plus-Gemini pools.The captions identify test and target loss by segment for several pools; Figure 20 also includes Human-77k as a reference.Source: Figure 20 · p. 26, §D.4 · p. 26, Figure 21 · p. 27
• With uniform weights, test loss does not decrease significantly; as metagradient steps learn weights, test loss begins decreasing.The target and test curves are not perfectly aligned, suggesting some overfitting may occur.Source: §D.4 · p. 27
• Figure 23 compares test loss for equal-sized human, synthetic, and mixed training sets targeting Expert.The caption identifies the comparison but does not state which set performs best.Source: Figure 23 · p. 28

D.5 Transferring dataweights to larger models

• Figure 26 compares STEM target and test loss for a 1.5B reward model with and without dataweights learned at 0.5B.Figures 27 and 28 also show transfer of learned dataweights to 1.5B models trained on human or Gemini data and targeting Expert.Source: Figure 26 · p. 29, Figure 27 · p. 30, Figure 28 · p. 30

E Selection-baseline comparison: protocol and additional results

• The appendix supplies compute-cost accounting and additional two-pool trajectories for the main-paper cost–performance comparison.Its stated scope excludes source-checkpoint generation and keep-fraction search from the selection-cost accounting.Source: E Selection-baseline comparison:… · p. 31, §E.2 · p. 31

E.1 Evaluation protocol and scorer configurations

• Each scorer retains its top-scoring examples at each keep fraction, tiles them to the original pool size, and uses a selection set to choose a checkpoint before held-out test evaluation.The protocol includes random-drop controls; keep-fraction points are single-seed, while the unweighted baseline uses three seeds.Source: §E.1 · p. 31

E.2 Cost–performance tradeoffs

• 58.3 GPU-hours and test loss 0.635 on Human-77k, and 48.8 GPU-hours and test loss 0.637 on Human-30k+Bot-10k, are the first observed frontier points.These occur at meta-epochs 3 and 4, respectively; dataweighting requires no separate retraining stage.Source: Figure 29 · p. 31, §E.2 · p. 31

E.3 Response-length audit

• At 10% retention, shorter-response preferences are 80.6% for TracIn, 57.2% for influence functions, and 45.3% for TRAK, versus 39.9% for random selection and 40.3% in the full pool.The audit describes selection-induced distribution shift, not whether length skew causes generalization degradation.Source: §E.3 · p. 32, Figure 30 · p. 32

E.4 Comparisons across training pools

• Dataweighting attains the lowest test loss on all four pools, followed by TRAK and influence functions.Its margins over the strongest baseline are 0.041, 0.028, 0.025, and 0.033 for 155k, poisoned 40k, clean 30k, and 77k pools, respectively.Source: Figure 31 · p. 33

F Experimental setup

• Each inner repeat replays a segment from its starting model state, computes a metagradient, and updates dataweights; I = 1 is one meta-optimization traversal, not one model-optimizer step.A traversal may contain multiple optimizer steps; forward-only training uses I = 0.Source: F Experimental setup · p. 34

G.1 Shared model and learning procedures

• Dataweighting selects fixed-size source subsets by target-set loss, while baselines use all or random subsets followed by target-set fine-tuning.The formal results concern global subset optimization, not whether a particular metagradient algorithm finds the optimum.Source: §G.1 · p. 35, G Stylized theoretical results for… · p. 35

G.2 Reduction to two dimensions and validation control

• After nuisance risk vanishes, target predictions depend on two signal coefficients, and validation selection has an expected-risk bound independent of target covariance T.The selection guarantee compares dataweighting with a source-only reference subset; its bound uses a constant depending on n′.Source: §G.2 · p. 36

G.3 Correcting small synthetic-label bias

• For systematic synthetic-label bias, both full-data and random-subset fine-tuning retain limiting risk at least cλ,n′δ^2, while a growing clean subset gives dataweighting vanishing expected risk.The result assumes fixed λ, δ > 0 and k = ⌊αn⌋ for 0 < α < 1/2.Source: §G.3 · p. 36

G.4 Filtering an unknown corrupted set

• As corruption size δ grows, dataweighting selects an entirely clean subset with probability tending to one, conditional on a clean k-subset existing.The theorem considers fixed finite n, n′, and D, with k = ⌊αn⌋.Source: §G.4 · p. 37