Source-linked AI summary
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
Shaobo Wang, Xuan Ouyang, Tianyi Xu, Yuzheng Hu, Jialin Liu, Guo Chen, Tianyu Zhang, Junhao Zheng, Kexin Yang, Xingzhang Ren, Dayiheng Liu, Linfeng Zhang
TL;DR
As high-quality public text becomes scarce, existing static filters ignore training dynamics and dynamic methods often use raw-gradient criteria mismatched with modern optimizers. OPUS defines utility in optimizer-induced update space, using proxy-guided scoring, scalable estimation, and diversity-preserving sampling. Across model scales, optimizers, and corpus-quality settings, it consistently improves compute-matched pre-training.
Problem
Existing static filters are training-agnostic, while prior dynamic methods score raw gradients despite modern LLM training using adaptive optimizer geometries.
Method
OPUS projects optimizer-induced effective updates onto a target direction from a stable proxy and estimates utilities efficiently while using Boltzmann sampling.
Results
Across model scales, optimizers, and corpus-quality settings, OPUS consistently improves compute-matched pre-training.
Takeaways & Limitations
The results suggest that accounting for how the optimizer moves parameters strengthens dynamic data selection.
Takeaways & Limitations
Exact virtual-gradient computation would require an additional validation backward pass after every selection step, so OPUS uses a first-order linearization.
Abstract
from arXiv · showhide
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either rely on heuristic static filters that ignore training dynamics, or use dynamic yet optimizer-agnostic criteria based on raw gradients. We propose OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework that defines utility in the optimizer-induced update space. OPUS scores candidates by projecting their effective updates, shaped by modern optimizers, onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, we employ Ghost technique with CountSketch for computational efficiency, and Boltzmann sampling for data diversity, incurring only 4.7\% additional compute overhead. OPUS achieves remarkable results across diverse corpora, quality tiers, optimizers, and model scales. In pre-training of GPT-2 Large/XL on FineWeb and FineWeb-Edu with 30B tokens, OPUS outperforms industrial-level baselines and even full 200B-token training. Moreover, when combined with industrial-level static filters, OPUS further improves pre-training efficiency, even with lower-quality data. Furthermore, in continued pre-training of Qwen3-8B-Base on SciencePedia, OPUS achieves superior performance using only 0.5B tokens compared to full training with 3B tokens, demonstrating significant data efficiency gains in specialized domains.
1 Introduction
OPUS addresses data-selection limits by adapting utility to evolving training dynamics and the optimizer-induced update geometry. It combines proxy-guided scoring with scalable estimation and diversity-preserving sampling.
- High-quality public text is becoming scarce, making the efficiency and quality of each pre-training token increasingly important.
- Static filters ignore model evolution, while prior dynamic methods score raw gradients and implicitly assume SGD-like dynamics.
- OPUS scores candidates by projecting optimizer-shaped effective updates onto a descent direction derived from a high-quality proxy distribution.
- OPUS estimates utilities without materializing full gradients, using lightweight projections for scalable scoring.
- Boltzmann sampling preserves diversity rather than repeatedly selecting only the highest-scoring candidates.
2 Related Work
Prior data-selection methods include static filtering, dynamic heuristics, and influence-based attribution. Static approaches are scalable and effective for removing noise but remain training-agnostic.
- Static filtering uses classifiers or rules to remove low-quality web data before training.
- Static approaches assume sample utility is time-invariant and do not adapt to the model’s evolving optimization needs.
- Dynamic selection chooses samples during training using estimated utility, including loss, perplexity, online sampling, and importance-based criteria.
- Influence-function and attribution methods estimate how training samples affect model behavior or validation loss, often using scalable approximations.
3 Background
The paper formalizes autoregressive language modeling, separates training and validation data, and distinguishes static from dynamic selection. Dynamic methods adapt scores to the current model and optimizer state.
- 3.1 LLM Pre-training: An autoregressive language model predicts each next token from its preceding context and is trained using sequence negative log-likelihood.
- 3.1 LLM Pre-training: The corpus is partitioned into training data for parameter updates and held-out validation data used only to guide selection.
- 3.2 Data Selection in Pre-training: Static selection filters the candidate pool before training using fixed sample scores and thresholds or top-k retention.
- 3.2 Data Selection in Pre-training: Dynamic selection chooses a subset from a candidate buffer at each step using scores or sampling distributions dependent on the current model and proxy signal.
- 3.2 Data Selection in Pre-training: Modern optimizers transform raw gradients through state-dependent operators, so effective update directions can differ from raw-gradient directions.
4 Optimizer-induced Preconditioners
The paper represents optimizer-specific training dynamics through preconditioners that reshape gradients before utility scoring. It derives tractable fixed-step approximations for SGD, AdamW, and Muon.
- SGD: SGD provides an identity-like reference geometry in which utility is naturally measured in raw-gradient space.
- Muon: Under fixed-step scoring, OPUS freezes Muon’s state and orthogonalization operator, yielding an approximately linear map for candidate comparison.
- Muon: Muon’s preconditioner can be plugged into the same utility machinery used for AdamW, preserving a common optimizer-aware selection framework.
- AdamW: AdamW’s momentum and adaptive second-moment terms motivate holding optimizer state fixed and freezing RMS geometry during candidate scoring.
- Muon: Muon applies Newton–Schulz orthogonalization to matrix-shaped directions, inducing a dense layerwise preconditioner beyond raw-gradient space.
- AdamW: The AdamW effective data-dependent update is obtained by removing an S-independent shift that cannot change relative candidate rankings.
5 Methodology: OPUS
OPUS defines dynamic data utility in the optimizer-induced update space and estimates it efficiently at LLM scale. It combines proxy-aligned scoring, ghost and CountSketch estimation, and Boltzmann sampling to select useful yet diverse training data.
- End-to-end selection: At each step, OPUS builds a candidate buffer, constructs an optimizer-state preconditioner, generates proxy features, and selects K = ⌊ρN⌋ samples.The proxy pool supplies a target direction for scoring candidate updates.
- Optimizer-induced utility objective: OPUS evaluates candidate batches by their expected one-step validation-loss reduction under the optimizer-induced update geometry.The objective aligns selection with optimizer trajectories rather than raw gradient space.
- Stable proxy construction: BENCH-PROXY retrieves benchmark-aligned samples to provide a stable, in-distribution proxy direction while avoiding the noise and distribution shift of alternative proxies.The proxy signal is intended to align selected updates with benchmark-relevant regions of the optimization landscape.
- Scalable utility estimation: Ghost factors and CountSketch estimate projected optimizer-shaped update inner products without materializing per-sample gradients or full high-dimensional matrices.CountSketch streams over outer-product-gradient coordinates, while ghost computation uses rank-1 gradient structure from standard passes.
- Diversity-aware sampling: Boltzmann sampling favors high-utility candidates while preserving non-zero probability for complementary samples, reducing sensitivity to local proxy noise.This sampling rule supports diversity instead of always selecting the current top-k candidates.
6 Experiments
Across pre-training and continued pre-training, OPUS improves compute-matched performance across datasets, optimizers, model scales, and quality settings. It also accelerates convergence, generalizes beyond proxy-aligned benchmarks, and adds only 4.7% computational overhead.
- FineWeb pre-training: OPUS achieves the best compute-matched average on FineWeb across model scales and optimizer settings, often matching or exceeding random sampling trained for twice as long.GPT-2 models are trained with a fixed budget of 30B update tokens, while Random 60B provides a longer-training reference.
- FineWeb-Edu robustness: 44.99: OPUS obtains the best compute-matched average for GPT-2 XL with Muon on FineWeb-Edu despite selecting from score-3 data while baselines use scores 4 and 5.The evaluation deliberately gives baselines access to a higher-quality partition.
- Optimizer robustness: 40.29 → 41.75: OPUS substantially outperforms Random for GPT-2 XL with Muon on FineWeb, supporting optimizer-induced selection over raw-gradient selection.The advantage is also reported under AdamW for both GPT-2 Large and GPT-2 XL.
- Generalization and continued pre-training: OPUS achieves the best performance on out-of-distribution benchmarks and improves domain-level continued-pre-training results across scientific domains.On SciencePedia, OPUS reaches the best specialized-benchmark performance using 0.5B tokens and outperforms Random trained for 3B tokens; domain gains are broadly distributed.
- Convergence: 17B update tokens: OPUS reaches the validation loss of Random trained for 60B tokens on GPT-2 XL, while maintaining lower loss than representative baselines across training.The same convergence trend appears for GPT-2 Large under the FineWeb-Edu setup.
7 Conclusion and Future work
OPUS aligns training-time data selection with the optimizer’s effective update geometry and improves compute-matched pre-training across model scales, optimizers, and corpus-quality settings. The paper identifies extending this approach to richer training regimes, such as data mixtures, as a next step.
- OPUS aligns dynamic data selection with the optimizer’s effective update geometry.
- Across model scales, optimizers, and corpus-quality settings, OPUS consistently improves compute-matched pre-training.
- The paper proposes extending optimizer-aligned selection to richer training regimes, including data mixtures.
Random
The supplied passages cover varied material, including HVAC maintenance, political elections, historical events, education, science, and consumer topics. They do not provide a coherent account of a research-paper section titled “Random.”
- HVAC filters are described as important components of heating, ventilation, and air-conditioning systems.
- Political parties are presented as established parts of modern mass democracy, with elections in India largely dependent on their behavior.
- The passages also include educational and scientific explanations concerning classroom design, solar radiation, earthquakes, and rotavirus.
OPUS
The supplied passages comprise unrelated explanatory and promotional excerpts rather than a coherent description of OPUS. They span education, sustainability, history, health, and consumer content.
- The passages describe KS2 Maths as covering numbers, calculations, problems, and measures for classroom or home learning.
- Other excerpts discuss origami, conduct disorder, the Wannsee Conference, earthquakes, and political parties.
- They describe sustainability education as available through independent, university, and noncredit learning resources.
High-PPL
The supplied passages contain unrelated excerpts about films, education, consumer products, history, recycling, science, and health rather than a coherent account of “High-PPL.”
- One excerpt characterizes Basic as a plot-twisting whodunnit rather than a war movie.
- Other passages discuss classroom design, decodable readers, sneaker-production technologies, recycling access, and scientific topics.
- Health-related excerpts address rotavirus immunization and the replacement of helmets after crashes.
GREATS
This section combines background and explanatory passages spanning HVAC maintenance, education, history, science, health, and environmental topics.
- HVAC filters are presented as an important but overlooked part of maintaining heating and cooling systems.
- Several passages introduce educational resources, including sustainability courses, decodable readers, and fraction worksheets.
- The collection also covers historical developments, political institutions, literature, and environmental change.
- Science and health explanations address galaxy motion, rotavirus illness, earthquakes, and solar radiation.
- Additional passages discuss ancient DNA reconstruction, helmet replacement, origami, and a biblical film narrative.
QuRating
This section presents miscellaneous passages on education, science, history, health, recycling, deforestation, and other informational topics.
- Educational materials address decodable reading, kindergarten project-based learning, mathematics, classroom design, and sustainability study.
- Science passages explain earthquakes and solar radiation, while another discusses challenges in reconstructing genomes from ancient DNA.
- Historical and civic content covers the Wannsee Conference, the Civil Rights Act of 1864, and political developments in India.
- Other entries concern rotavirus, conduct disorder, recycling access, helmets, entertainment, and hospitality venues.
- One environmental passage reports an estimated 522,000 acres of forest cover lost in Oregon since 2000.
FineWeb-Edu
This section contains repeated explanatory and informational passages spanning science, history, education, health, environmental issues, and cultural topics.
- Science explanations describe galaxy motion, earthquakes, solar radiation, and the effects of dark matter on observed stellar behavior.
- Historical and civic topics include Italian and German unification, the Wannsee Conference, Indian elections, and civil-rights legislation.
- Health and environmental entries discuss rotavirus, conduct disorder, Oregon deforestation, and recycling access.
- Educational passages address mathematics, decodable readers, classroom design, kindergarten projects, and academic biographies.
- Other passages concern ancient DNA, origami, HVAC filters, helmets, film, and literature.
Ultra-FineWeb
This section assembles diverse informational and cultural passages, with recurring material on education, science, history, health, and environmental topics.
- Historical material addresses the Civil Rights Act of 1864, the Wannsee Conference, and a quotation about the Boston Tea Party.
- Health and practical topics include conduct disorder, rotavirus, HVAC filters, and skate-helmet replacement.
- Additional entries concern the Central Limit Theorem, a biblical film, sneaker design, literature, recycling, and Oregon deforestation.
- Science passages discuss galaxy motion, earthquakes, solar radiation, and ancient DNA reconstruction challenges.
- Educational content covers decodable reading, fraction worksheets, mathematics curricula, classroom design, kindergarten projects, and sustainability learning.
DCLM-FastText
The supplied passages cover assorted educational, historical, scientific, health, and lifestyle topics rather than a coherent DCLM-FastText section.
- A passage discusses galaxies, stars, and dark matter.
- Several passages address historical and political subjects, including civil rights, national unification, elections, and the Wannsee Conference.
- Other passages cover health, science, education, recycling, classroom design, sneakers, exercise, and origami.
DSIR
The supplied passages combine unrelated excerpts on history, science, education, health, entertainment, and recycling rather than presenting a coherent DSIR section.
- Historical and cultural excerpts discuss the Boston Tea Party, civil rights, and a biblical film plot.
- Scientific and educational excerpts concern solar radiation, ancient DNA, classroom design, mathematics, and decodable readers.
- Additional excerpts address deforestation, accounting education, helmets, HVAC filters, conduct disorder, and women’s underwear.