PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling

Zailin Ma, Quzhe Huang, Yujun Li, Congyuan Rao, Yaodong Yang

arXiv:2609.32484v1cs.AI

TL;DR

Quality filtering can collapse diversity by favoring narrow references, while direct geometric diversification can be computationally costly. Lev selects high-leverage samples to expand representation volume efficiently. It outperforms diversification baselines across diversity, downstream performance, and runtime, with a 72× speedup over DiSF.

  • Problem

    Quality filtering can induce dimensional collapse by favoring narrow reference data, while direct geometric-diversity methods require costly covariance recomputation.

    Source: §1 · p. 2

  • Method

    Lev iteratively selects samples with high leverage scores, which quantify their contribution to expanding the determinantal volume of the selected representation subspace.

    Source: §1 · p. 2

  • Results

    Lev outperforms diversification baselines across diversity metrics, downstream task performance, and runtime efficiency, with a 72× speedup over DiSF.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The findings underscore the complementary role of diversity-aware selection to quality filtering and motivate tighter integration of diversity and quality objectives in data curation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Data selection for language model pretraining faces a fundamental tension between quality and diversity. While quality filtering is empirically effective, it often induces diversity collapse: by favoring texts similar to high-quality reference corpora (e.g., educational or QA-style data), it systematically excludes valuable data from underrepresented domains. In contrast, diversified selection preserves domain balance and encourages robust downstream performance, yet existing methods either focus on coverage-oriented objectives that indirectly enhance diversity, or directly optimize for diversity via costly covariance matrix recomputation that limits scalability. To address these issues, we introduce \textbf{Leverage Score Sampling (Lev)}, which iteratively selects samples that maximally expand the determinantal volume of the embedded data via leverage scores, a computationally efficient criterion that eliminates matrix recomputation and enables scalable selection. Empirically, Lev delivers up to $72\times$ speedup and improves dataset diversity, measured by the Vendi score, by $9.2\%$ over the strong diversification baseline \textbf{DiSF}. On CommonCrawl (CC) web data selection, Lev improves accuracy across seven downstream tasks by up to $1.31\%$ over existing baselines. For domains where robust quality criteria are inherently difficult to define (e.g., code), Lev serves as an effective unsupervised curation alternative: on StarCoderData, the selected subset reduces bits-per-byte by $3.08\%$ over DiSF. Notably, we uncover a cross-domain collapse of quality filtering: CC data filtered by DCLM-fastText fail to retain sufficient code-related content, yielding inferior code performance relative to Lev-selected data. These findings advocate for integrating diversity-aware practices into quality filtering for more effective data curation in language model pretraining.

1 Introduction

The introduction argues that quality filtering can collapse diversity by favoring narrow reference corpora, while existing direct diversification methods are costly. It presents Leverage Score Sampling (Lev) as a scalable alternative and reports gains in diversity, runtime, and downstream performance.

1 Introduction

  • Quality classifiers trained on narrow high-quality references can induce dimensional collapse and degrade performance on underrepresented domains.The introduction describes the resulting embeddings as spanning a low-dimensional subspace.Source: §1 · p. 2
  • Domain balancing and coverage-based sampling address diversity indirectly, while DiSF directly optimizes geometric diversity but recomputes covariance for each candidate.That repeated computation imposes overhead that limits scalability.Source: §1 · p. 2
  • Lev selects samples with the highest leverage scores, which quantify their contribution to determinantal-volume expansion using quadratic expressions and one shared Gram matrix.This avoids DiSF's per-candidate covariance recomputation.Source: §1 · p. 2, Figure 2 · p. 2
  • Lev improves Vendi score by 9.2% over DiSF, achieves 72× speedup, and improves CommonCrawl accuracy by up to 1.31% across seven downstream tasks.On StarCoderData, Lev reduces bits-per-byte by 3.08% over DiSF.Source: §1 · p. 2
  • DCLM-fastText filtering leaves insufficient code-related content in CommonCrawl, with code performance inferior to diversity-selected data.The authors argue that integrating diversity-aware practices into quality filtering can support more robust curation.Source: §1 · pp. 2–3

2 Preliminaries

The preliminaries formulate diverse data selection under a fixed selection ratio and distinguish indirect diversity signals from direct set-level objectives. They explain why coverage-based uniformity may not ensure geometric diversity and why direct covariance-based optimization can be costly.

2.1 Problem statement

  • The problem is to select a diverse subset of corpus documents under a target selection ratio, with Diversity(S) quantifying subset diversity.Documents are represented as unit vectors produced by an embedding model.Source: §2.1 · p. 3

2.2 Recent progress

  • Prior heuristic-guided methods use domain organization, attribute balancing, or coverage-based sampling to encourage diversity indirectly.Coverage-based approaches promote uniform coverage through clustering or local density estimates.Source: §2.2 · p. 3
  • Uniform density need not span the representation space; DiSF directly optimizes set-level diversity but requires costly iterative covariance updates.DiSF's objective encourages a more uniform covariance eigenspectrum.Source: §2.2 · p. 3

3 Method

The method uses leverage scores as a geometric diversity signal, selects high-scoring candidates through an incrementally maintained Gram matrix, and explains their preference for underrepresented semantic directions.

3.1 Leverage score as diversity signal

  • A candidate's leverage score quantifies the relative volume expansion from adding it to the selected pool, making it a signal of geometric diversity.The paper interprets the determinant of the regularized Gram matrix as squared volume in representation space.Source: §3.1 · pp. 3–4

3.2 Algorithm

  • Lev processes the corpus in batches, repeatedly adding the highest-scoring candidates and updating a shared Gram matrix rather than recomputing covariance for each candidate.Cholesky solves compute scores in a numerically stable and efficient form equivalent to the quadratic expression; incremental updates improve efficiency over DiSF.Source: §3.2 · p. 4

3.3 How do leverage scores identify underrepresented semantics?

  • Leverage scores weight candidate projections more heavily along semantic directions that occur less frequently in the selected pool.When a direction is nearly absent, its inverse-frequency weight becomes large; the paper reports that λ = (100 × d)^-1 is sufficient to highlight uncovered semantic signals.Source: §3.3 · p. 5

4 Experiments

Experiments compare Lev with diversity, distribution-matching, random, and quality-filtering baselines on CommonCrawl and code data. Lev improves diversity and downstream performance while reducing selection runtime relative to DiSF.

4.1 Setup

  • Experiments use a 1.1B-parameter Llama 3.1 configuration trained for 60B tokens, evaluating seven language-understanding tasks and seven code benchmarks.The experiments include CommonCrawl and six StarCoderData programming languages.Source: §4.1 · pp. 5–6

4.2.1 Lev outperforms existing diversity-based selection methods

  • At a 50% selection ratio, Lev improves average accuracy by 1.31% over DSIR across seven CommonCrawl tasks.Lev achieves the best performance among diversity-based methods.Source: Table 1 · p. 6, §4.2.1 · p. 7
  • Lev improves Vendi score by 9.2% over DiSF (186.63 vs. 203.81) and performs best across all seven diversity metrics.The study compares normalized distance-, spectrum-, and combined-perspective diversity metrics.Source: Figure 4 · p. 6, §4.2.1 · p. 6
  • At a 50% code selection ratio with quality-filtered web data, Lev reduces average code BPB by 3.08% over DiSF (65.68 to 63.66).Lev attains the lowest code BPB across all four tested settings.Source: Figure 5 · p. 7, §4.2.1 · p. 7

4.2.2 Revisiting dimensional collapse: quality filtering demands diversity

  • DCLM-fastText-filtered CC underrepresents code-related content relative to Lev-selected CC, and the code-performance gap persists after both are augmented with Lev-selected StarCoderData.The authors also report consistent results with two additional code datasets.Source: Table 2 · p. 7, §4.2.2 · p. 8
  • Combining quality-filtered CC with Lev-selected code yields lower code BPB (63.66 vs. 65.96) and higher text accuracy (51.53% vs. 50.46%) than Lev-selected CC mixed with unfiltered code.The authors identify deeper integration of diversity and quality objectives as future work.Source: Figure 6 · p. 8, §4.2.2 · p. 8

4.2.3 Lev enables efficient large-scale diversity selection

4.3 Ablation Study

  • Lev remains effective across tested step sizes, batch sizes, and selection ratios; higher web selection ratios lower diversity and performance, while code remains effective from 30% to 60%.The authors use B = 1024 to balance efficiency and performance.Source: §4.3 · pp. 9–10

5 Conclusion

Lev is presented as a scalable diversity-oriented selection method, and the conclusion reports cross-domain dimensional collapse from quality filtering. The related work places pretraining selection among quality, diversity, and training-aware approaches, while later-stage selection emphasizes controlled optimization over smaller, cleaner datasets.

5 Conclusion

  • Lev iteratively selects high-leverage samples to expand the representation subspace, and the conclusion reports a 72× speedup over DiSF alongside gains across diversity metrics and downstream performance.The conclusion also identifies systematic exclusion of code-related content from CommonCrawl as an example of cross-domain dimensional collapse induced by quality filtering.Source: §5 · p. 10

A Related work

  • Pretraining selection spans heuristic cleaning, target-distribution matching, model-based quality filtering, training-aware methods, rewriting, and set-level diversity or mixture optimization.The related-work discussion frames these developments as motivating corpus-level selection that balances quality, redundancy, and coverage under a fixed training budget.Source: A Related work · p. 16
  • Later-stage selection covers instruction-tuning subset choice, alignment and post-training objectives, and dynamic scheduling or mixture adaptation.These methods typically operate on smaller, cleaner datasets and emphasize annotation quality, formatting consistency, diversity, preference structure, curriculum, safety, and capability balance.Source: A Related work · p. 16

B.1 Proof of Theorem 3.1

The appendix passages outline proof steps and interpretations related to the theorems, occurrence frequencies, and regularization. They connect principal-component variance with sample occurrence frequency and motivate a small positive regularization parameter.

B.1 Proof of Theorem 3.1

  • Theorem 3.1’s proof invokes the matrix determinant lemma and substitutes z for both vectors in that relation.The supplied proof steps do not state the resulting equality.Source: §B.1 · pp. 16–17

B.2 Proof of Theorem 3.2

  • Theorem 3.2’s proof uses an SVD to form a regularized Gram matrix and derive its inverse before applying the definition.The supplied steps do not give the final derived expression.Source: §B.2 · p. 17

B.3 Interpretation of the occurrence frequencies

  • For normalized embeddings, the variance of a principal component can be interpreted as its occurrence frequency in the dataset.The appendix illustrates this interpretation using repeated orthonormal basis samples with frequencies p_i.Source: §B.3 · p. 18

B.4 The choice of the regularization parameter

  • The regularization parameter λ should be positive and much smaller than d^-1; rare or uncovered semantic directions motivate keeping it small.In a perfectly diverse distribution, the appendix states that this choice gives each sample a leverage score approximately d.Source: §B.4 · p. 19

C.1 Datasets

The experimental appendix describes the web and code datasets, embeddings, and selection baselines. It specifies how corpus embeddings and baseline selection criteria are constructed.

C.1 Datasets

  • The web corpus is a CommonCrawl dump processed with FineWeb-like heuristic filtering and deduplication, while quality filtering is excluded to preserve diversity.The code experiments use StarCoderData and restrict the study to six programming languages: Python, Java, JavaScript, Go, PHP, and Ruby.Source: §C.1 · p. 19

C.2 Embedding model

  • Qwen3-Embedding-0.6B supplies embeddings for all diversity-based methods, using the first 512 tokens for CommonCrawl and 1,500 tokens for StarCoderData.The appendix describes the model as supporting flexible embedding dimensions and long-context inputs.Source: §C.2 · p. 20

C.3 Baselines

  • DiSF encourages a uniform covariance-eigenvalue spectrum and selects greedily within batches using a score based on the covariance Frobenius norm.The appendix interprets its overall objective as approximately minimizing spectral concentration.Source: §C.3 · p. 20
  • Density sampling estimates embedding-space density with signed random projections and assigns larger sampling weights to samples in sparse regions.Its randomized hashing scheme builds a histogram sketch, and subset sampling uses inverse-density weights.Source: §C.3 · p. 21
  • DSIR matches samples to target domains such as Wikipedia and books using n-gram features, whereas DCLM-fastText scores web text with a lightweight classifier trained on positive and negative examples.Random sampling is also included as a non-selective baseline.Source: §C.3 · pp. 21–22

C.4 Pretrained model configurations and training details

The appendix specifies the pretraining and evaluation setup, describes the included language and code-generation benchmarks, and defines diversity metrics based on local distances, spectral structure, and combined signals.

C.4 Pretrained model configurations and training details

  • All 1.1B models are trained for 60B pre-tokenized tokens with batch size 1,024 and context length 2,048, then evaluated using OLMES.The training configuration totals 28,610 iterations; Table 4 is identified as containing detailed model configurations.Source: §C.4 · p. 22, Table 4 · p. 22

C.5 Benchmarks

  • The benchmark suite includes seven commonsense or reasoning tasks and two code-generation benchmarks, with the latter evaluating HumanEval and multilingual MBPP.The appendix describes the code benchmarks as unit-test-based or executable-test-based evaluation.Source: §C.5 · pp. 22–23

C.6 Diversity metrics

  • The diversity metrics cover local nearest-neighbor distances, pairwise cosine distances, cluster dispersion, spectral entropy, Vendi score, and log-determinant volume.The appendix says spectral and distributional metrics characterize global Gram-matrix eigenstructure, while distance-based metrics emphasize pairwise or local separation.Source: §C.6 · pp. 24–25
  • NovelSum combines proximity-aware weighting with density-aware scaling to estimate each sample’s unique contribution to dataset diversity.The appendix characterizes it as integrating local proximity, inter-sample differences, and heterogeneous information density.Source: §C.6 · p. 25

D.1 Diversity metrics

Lev consistently performs best across the reported diversity metrics, while the appendix also provides full downstream results for CommonCrawl and StarCoderData selections at 40% and 50% ratios.

D.1 Diversity metrics

  • Lev consistently achieves the best performance across the diversity metrics; Table 5 reports the comparison at a 40% selection ratio.Figure 4 normalizes the metrics to compare relative values; the original values are reported in Table 5.Source: §D.1 · p. 26, Table 5 · p. 26

D.2 Downstream performance on CommonCrawl subset selection

  • Full downstream results cover seven language-understanding tasks on CommonCrawl subsets selected at 40% and 50% ratios.The appendix reports results across all baseline methods and Lev.Source: §D.2 · p. 26, Table 6 · p. 26

D.3 Downstream performance on StarCoderData subset selection

  • Full code bits-per-byte results cover seven code-related tasks on StarCoderData subsets selected at 40% and 50% ratios.The appendix reports results across the diversity-based selection methods.Source: §D.3 · p. 27, Table 7 · p. 27

D.4 Quality-filtered CC data shows inferior code performance relative to Lev-selected CC data

Quality-filtered CommonCrawl underperforms Lev-selected CommonCrawl on code evaluation, and adding dedicated code data does not eliminate the gap.

D.4 Quality-filtered CC data shows inferior code performance relative to Lev-selected CC data

  • Quality-filtered CommonCrawl yields higher code bits-per-byte than Lev-selected CommonCrawl, and remains inferior after both subsets are augmented with StarCoderData.The comparison is reported for code evaluation; Table 8 compares DCLM-fastText and leverage-score sampling under different code data.Source: §D.4 · p. 28, Table 8 · p. 28

E.1 Why does Lev achieve better diversity optimization results than DiSF?

The analysis compares Lev’s leverage-score objective with DiSF’s Frobenius objective and argues that Lev is more sensitive to samples carrying rare semantics.

E.1 Why does Lev achieve better diversity optimization results than DiSF?

  • Lev is more sensitive than DiSF to samples that capture rare semantics, the proposed explanation for its stronger diversity optimization.The analysis frames this as a theoretical comparison of the leverage-score and Frobenius objectives.Source: §E.1 · p. 29
  • The analysis represents the selected data with a Gram matrix and evaluates how a candidate sample changes DiSF’s criterion.The passage defines candidate projections onto principal vectors and the corresponding eigenvalues.Source: §E.1 · p. 29
  • When a principal component is extremely rare, samples assigning it more weight offer greater diversity gains under the stated entropy-based rationale.The example uses eigenvalues (3/4, 1/4 − 10^-2, 10^-2) and considers maximizing Shannon entropy, such as the Vendi Score.Source: §E.1 · p. 29
  • In the example, DiSF’s score ranks S1 above S2 even though S2 places more weight on the third, rare dimension.The text states that DiSF therefore mistakenly selects S1.Source: §E.1 · p. 29
  • Lev’s leverage weighting amplifies S2’s rare-semantic contribution, and the example is said to represent a broader case where DiSF can miss diverse samples.As the selected set grows, samples with rare semantics may also contain semantics already common in the set.Source: §E.1 · p. 30

E.2 Insufficient recall of code data is a general problem among open-source data scorers

Across the examined open-source scorers, code-related recall is associated with code modeling ability, while Lev retrieves more code content than the quality filters described.

E.2 Insufficient recall of code data is a general problem among open-source data scorers

  • DCLM-fastText includes code data in its positive training examples, but its code-data recall is not as strong as Lev’s.Code makes up 18.2% of the approximately 1M OpenHermes-2.5 samples used as positive examples.Source: §E.2 · p. 30
  • Code-related document recall is positively correlated with code modeling ability across the compared subsets.The comparison considers the number of code-related documents retrieved and code bits-per-byte after training.Source: §E.2 · p. 30
  • Lev retrieves the most code-related content and its pretrained model shows the best code modeling ability, while quality filters generally retrieve less code content and perform worse.Two quality filters retain even less code-related content than Random.Source: §E.2 · pp. 30–31
  • Several quality filters include code-domain data in their training sets yet do not necessarily attain Lev’s code-data recall, supporting the authors’ call for diversity enhancement.Source: §E.2 · p. 31

E.3 Detailed analysis of the time complexity of Lev and DiSF

Lev and DiSF use the same greedy batch-selection framework, but differ in how they score unselected samples. Lev avoids DiSF’s repeated covariance computations, giving it a lower time complexity.

E.3 Detailed analysis of the time complexity of Lev and DiSF

  • Both methods divide data into batches, repeatedly score and select the top-k unselected samples, and stop when the selection budget is reached; only score computation differs.The comparison of score computation is the focus of the time-complexity analysis.Source: §E.3 · p. 31
  • Lev computes all unselected samples’ leverage scores together after one Cholesky decomposition, making greedy Step 2(a) O(Bd^2) when B ≥ d.The passage identifies this substep as dominant under that condition.Source: §E.3 · pp. 31–32
  • DiSF’s Step 2(a) costs O((B − n)nd^2), with its dominant cost coming from recomputing the covariance matrix for each unselected sample.Its scoring procedure also normalizes every column before computing the covariance matrix.Source: §E.3 · p. 31
  • DiSF’s time complexity is one order over B higher than Lev’s, whose total Step 2 complexity is O(αB^2d^2) = O(B^2d^2).The comparison sums Lev’s per-selection cost across n; the conclusion compares the algorithms’ time complexity.Source: §E.3 · p. 32

Related papers

Loading 2609.32484v1…