Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle

1 Introduction

• Training data affects both downstream accuracy and efficiency, and data strategy determines which examples fill a fixed token budget.Figure 1 traces the expansion from pretraining mixtures to SFT, RL, and mid-training.Source: §1 · p. 2
• Pretraining strategies commonly reweight domains during stable training, while empirical annealing approaches typically select high-quality data.Domain-level optimisation is tractable but trades away sample-level flexibility and assumes a useful corpus partition.Source: §1 · p. 2
• Post-training methods span task-level mixing, data selection, and curriculum design, with approaches varying across SFT, preference optimisation, and RL.SFT methods balance transfer against interference and forgetting or select and order instructions; other approaches prioritize lagging tasks or learn mixtures for generalisation.Source: §1 · p. 2
• Data quality is training-dependent: an example is high-quality when allocating budget to it improves the target objective from the current model state.Its utility depends on the target capability, base model, and training state; fixed filters and external scorers use model-independent proxies.Source: §1 · p. 2
• Inconsistent terminology obscures the shared structure of data strategies and complicates compute comparisons, motivating a lifecycle-wide review and unified framing.Existing surveys cover different parts of the landscape, including broad data management, pretraining selection, and instruction tuning.Source: §1 · p. 2
• The paper casts selection, reweighting, mixing, and curricula as policy optimisation over the training trajectory, then compares their assumptions, costs, and empirical evidence.It also compares partitions induced by three signals and reconstructs auxiliary-training compute for representative data-mixture strategies.Source: §1 · p. 2

2.1 Data Strategies as Policy Optimisation

• A data policy maps the training state and candidate pool to feasible example allocations, which are sampled into batches to update the model under an explicit token budget.The objective evaluates the final model, while the state update takes an optimiser step using the sampled batch gradient.Source: §2.1 · p. 3

2.2 Disambiguating Related Concepts

• A data policy changes the current training distribution, whereas Adam transforms accumulated gradients through coordinate-wise coefficients.The policy assigns scalar probabilities to current examples; Adam applies coefficients to historical batch gradients.Source: §2.2 · p. 4
• Data selection assigns equal probability to retained examples and zero to the rest, while reweighting permits any simplex allocation and domain reweighting ties examples within each domain.Domain reweighting reduces n example probabilities to K mixture proportions, improving tractability at pretraining scale but preventing isolation of one example within a multi-example domain.Source: §2.2 · p. 4
• The feasible sets for selection and domain reweighting are generally incomparable, although both lie within the reweighting simplex; domain reweighting equals the full simplex only when every domain is a singleton.Selecting one member of a multi-example domain violates the domain constraint, while unequal positive probabilities across domains violate selection's uniform-support constraint.Source: §2.2 · p. 4

2.3 Curricula as Structured Policies

• A curriculum constrains the allocation trajectory through a signal and scheduling rule, and can be combined with selection or reweighting.Static curricula fix the signal and schedule before training; adaptive curricula update one or both from the current state.Source: §2.2 · p. 4, §2.3 · p. 5

2.2 Disambiguating Related Concepts

• Pretraining's massive, largely static corpora favor coarse domain decisions or infrequent rescoring, while smaller post-training pools and richer feedback can make fine-grained, model-conditioned policies more practical.Post-training pools may be fixed for supervised fine-tuning and offline preference optimisation or evolve through online reinforcement-learning rollouts.Source: §2.2 · p. 4

2.3 Curricula as Structured Policies

• Curriculum methods schedule different units—including examples, clusters, categories, prompts, and replayed prompt–output pairs—using a signal and rule that govern exposure.SPARD also adapts reward-objective weights, a decision outside the data allocation.Source: §2.3 · p. 5
• Curricula need not order data easy-to-hard: methods use state-dependent learning-value signals, with examples prioritising uncertainty, intermediate difficulty, or dimensions showing reliable recent gains.Curriculum denotes temporal control of exposure, not a particular difficulty measure or mandatory direction.Source: §2.3 · p. 5

3.1 A Taxonomy of Methods

• Pretraining data strategies differ in whether they retain individual documents or set sampling weights for domains, trading finer control for tractability at scale.Domain reweighting uses a shared weight within each domain, while document selection makes individual retention decisions.Source: §3.1 · p. 5, §3.1.1 · p. 6, §3.1.2 · p. 9

3.1.1 Data Selection

• Rule-based filters use metadata, document-level properties, or corpus comparisons, while external scorers rank documents using reference signals, rated properties, or direct usefulness judgments.Rules are cheap and transparent but reflect surface properties; external scorers may encode quality that does not match the target model.Source: §3.1.1 · p. 6
• Target-model selection ranges from forward-pass signals to per-sample gradients and Hessian-based curvature, with higher-order signals capturing interactions at greater computational expense.Examples include loss-based AttentionInfluence, gradient-based GREATS, and Hessian-based BLISS and D3.Source: §3.1.1 · p. 7

3.1.2 Domain Reweighting

• Domain partitions use human labels, semantic embeddings, or model-gradient directions, but existing comparisons establish no generally preferable definition.Gradient-based partitions require per-sample gradient computation and storage; their transfer across model scales is empirical rather than theoretically guaranteed.Source: §3.1.2 · p. 8
• Fixing a partition before optimizing mixture weights assumes examples within each domain can share one weight, so erased distinctions cannot be recovered by a better optimizer.Most studies optimize weights over a fixed taxonomy rather than revising partitions and mixtures jointly.Source: §3.1.2 · p. 8
• Domain reweighting methods either fit a surrogate over candidate mixtures or reformulate the objective to avoid repeated full pretraining runs.Surrogate approaches transfer from observed to unobserved mixtures and from proxy runs to target scale; methods do not guarantee transfer beyond sampled regimes.Source: §3.1.2 · pp. 8–9

3.2 Summary and Discussion

• Transfer across models, the appropriate unit of decision, and cost accounting that separates reusable setup from per-run computation remain open questions.Evidence for proxy or gradient-partition transfer across scales is empirical, and studies vary in model scale, token budget, and hardware.Source: §3.2 · p. 9

4.2 Summary and Discussion

• Preference optimisation and RL receive less coverage than SFT, leaving open how to allocate rollout budgets across heterogeneous environments and select coherently across long-horizon trajectories.The discussion also identifies selection efficiency as an open question when feature recomputation over large candidate pools may be costly.Source: §4 · p. 9, §4.2 · p. 12

4 Data Strategies for Post-training

• Post-training adapts pretrained LLMs through SFT, preference optimisation, and RL, and a small, well-chosen subset can match or exceed training on the full set.The cited evidence for this subset comparison is attributed to Dong et al. (2024) and Li et al. (2024b).Source: §4 · p. 9

4.1 A Comparison of Methods

• Post-training methods use distributional profiles or gradient information to estimate data value, with gradient rankings tied to a checkpoint and target set and distributional profiles dependent on a predefined taxonomy.Recomputing gradient scores can track training changes but adds recurring cost; neither signal directly estimates marginal utility after full training.Source: §4.1 · pp. 10–11
• Methods judge policies using final-model objectives such as average or worst-group risk, base-capability preservation, or proxies such as coverage and instruction-following difficulty.These criteria can rank the same candidate pool differently, and proxy-based gains do not establish that a proxy is calibrated to the final objective.Source: §4.1 · p. 11, Table 5 · p. 11

4.2 Summary and Discussion

• Post-training policies range from data-only and checkpoint-based approaches to stateful policies that control inclusion, sampling frequency, and order.Greater adaptivity makes scores more specific to the current model but shortens their validity; comparisons that ignore rescoring frequency can conflate signal quality with additional model queries.Source: §4.2 · p. 11

5 Conclusion

• Selection, domain reweighting, and curricula are framed as policies that allocate training budget over examples.The paper distinguishes how these strategies and optimisers constrain those policies.Source: §5 · p. 12
• Coarse decisions require a useful partition, while model awareness and adaptivity add scoring, proxy-training, or rollout costs.Auxiliary FLOPs depend on model size, tokens per run, and run count.Source: §5 · p. 12
• Future evaluations should report utility and end-to-end cost together, including for agentic RL policies.The analyses find that partitions depend on how their construction signal is chosen.Source: §5 · p. 12

Limitations

• The analysis covers text-only LLM data strategies and FLOPs, while multimodal selection, reweighting, and curricula remain less studied.For multimodal LLMs, utility depends on modality quality and cross-modal alignment.Source: Limitations · p. 12

A Related Surveys

• Albalak et al. (2024) organise language-model data selection by utility function and decision mechanism, while Wang et al. (2023) take a broader data-management view and Zhang et al. (2025a) focus on instruction-tuning subset construction.The paper frames selection, domain reweighting, and curricula as restrictions on a common time-varying weighting policy.Source: A Related Surveys · p. 18
• The related-survey comparison tracks coverage of stable pretraining, annealing, supervised fine-tuning, and preference optimisation.The table caption identifies these stage abbreviations as PT, AN, SFT, and PO, respectively.Source: Table 6 · p. 18
• Bai et al. (2024) survey the multimodal data pipeline, whereas Qin et al. (2025b) study data–model co-development, including models' roles as data generators, annotators, and evaluators.These surveys cover broader data construction; this work asks which existing training units to expose, at what weight, and at which state.Source: A Related Surveys · p. 18

• The analyses compare source-, semantic-, and gradient-based partitions of the same documents and reconstruct auxiliary accelerator compute for representative domain-reweighting methods.Source: §B.1 · p. 18

• The analysis asks whether source metadata, semantic similarity, and model gradients induce compatible partitions of the same documents.Source: §B.2 · p. 18
• Semantic clusters follow source boundaries more closely than gradient clusters, while semantic and gradient partitions have the lowest agreement.The same ordering holds at K = 25 and K = 100, so it is not specific to matching the number of source categories.Source: Figure 4 · p. 19, §B.2 · p. 19
• The comparison uses 115,839 SlimPajama documents, clustering 50 principal components with KMeans and comparing at K = 7 using NMI and ARI.Semantic vectors come from stella_en_400M_v5; gradient representations use projected, L2-normalised gradients from Qwen2.5-0.5B-Instruct.Source: §B.2 · p. 19
• Source, semantic, and gradient signals induce substantially different partitions of the same documents, making a domain a method-dependent modelling choice rather than a corpus-intrinsic attribute.Source: §B.2 · p. 19

B.3 RQ2: What Is the Auxiliary Training Cost?

• The common target has 1.035B non-embedding parameters and trains on 5.822B SlimPajama tokens; the estimates are approximate lower bounds.CPU, I/O, and validation passes with unreported token counts are excluded, and the methods retain method-specific adaptation budgets and public run counts.Source: §B.3 · p. 20
• Run count alone does not determine auxiliary compute: model size, tokens per run, and run count must be reported together.The analysis does not measure mixture fidelity or downstream utility, so it does not establish a quality–cost frontier.Source: §B.3 · p. 20
• When proxy runs or a fitted surrogate support R target runs, amortised overhead is Caux/(RCtarget); target-dependent scores may require fresh computation.Changes in target distribution, corpus partition, or scoring checkpoint can invalidate cached artefacts.Source: §B.3 · p. 20
• In the DIDS-inspired local adaptation, influence estimation and periodic evaluation account for the largest logged compute shares, followed by embedding extraction.Published DIDS clusters per-example gradients, so these component shares characterize only the adaptation.Source: §B.3 · p. 20
• For this adaptation, evaluation frequency and reuse of gradients or embeddings are the main efficiency variables to ablate.Source: §B.3 · p. 20

C Notation

• Table 8 collects notation used in the common formulation and its special cases.Source: Table 8 · p. 21