PaperlayerView exact revision on arXiv

Source-linked AI summary

Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Models

Zhenghao Zhao, Gaowen Liu, Zhiling Lan, Yan Yan

arXiv:2610.02593v1cs.LG

TL;DR

In continual pre-training, data selection must support target-domain learning while preserving capabilities encoded in the pretrained model. The paper selects data using Fisher-aware gradient components and a log-determinant objective; on TinyLlama-1.1B, 1B selected tokens outperformed replay trained with 10B tokens on adaptation and forgetting.

  • Problem

    In CPT, selecting target-domain data must balance learning with retaining pretrained capabilities, but parameter-agnostic selection does not account for Fisher-sensitive directions.

    Source: §1 · pp. 1–2

  • Method

    The selector decomposes candidate gradients into Fisher-guided anchor and frontier components, then aggregates them with a log-determinant objective optimized by single-pass Sieve-Streaming.

    Source: §1 · p. 2

  • Results

    1B selected tokens outperformed replay trained with 10B tokens on adaptation and forgetting on TinyLlama-1.1B, a ≥10× token-efficiency advantage.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For medical-domain CPT, selected data can outperform forgetting-aware replay on adaptation and forgetting with a much smaller token budget.

    Source: §1 · p. 2

    The scalable formulation uses diagonal Fisher estimates in the LoRA adapter subspace and evaluates gradients at a short warmup operating point θ⋆.

    Source: E LIMITATIONS · p. 21

Abstract

from arXiv · show

Data selection is already a central bottleneck in large-language-model training, where web-scale corpora are noisy and token budgets are finite. In continual pre-training (CPT), it becomes a forgetting-control problem: a poorly chosen target-domain corpus can overwrite capabilities encoded in the pretrained checkpoint. Existing CPT practice either scores candidates with parameter-agnostic scalars such as perplexity, or mitigates forgetting by spending many extra general-domain replay tokens. Neither strategy directly asks how training on a candidate will move the model parameters. We show that loss-based selection causes the post-CPT Fisher diagonal to drift downward on exactly the high-Fisher coordinates the pretrained model had committed to, while leaving low-Fisher coordinates largely untouched. This asymmetry exposes a parameter-space mechanism for catastrophic forgetting. Motivated by this observation, we propose a Fisher-aware CPT selector that decomposes each candidate's gradient into an anchor component, which measures perturbation along committed parameter directions, and a frontier component, which measures update capacity in unconstrained low-Fisher subspaces. We aggregate these signals with a log-determinant submodular objective and optimize it in a single pass using a scalable streaming data selection pipeline. On TinyLlama-1.1B and Llama-3.1-8B CPT over medical data, our selector improves target-domain quality while bounding forgetting on held-out pretraining benchmarks. Most importantly, it is substantially more token-efficient than forgetting-aware replay. 1B selected tokens already outperform the replay strategy trained with 10B tokens on both adaptation and forgetting, giving a 10x token-efficiency advantage.

1 INTRODUCTION

CPT data selection must acquire target-domain knowledge while preserving pretrained capabilities, yet common scalar scores do not reveal how training moves parameters and replay costs additional tokens. The paper uses Fisher geometry to guide candidate selection and reports improved adaptation-retention trade-offs with greater token efficiency than replay.

1 INTRODUCTION

  • Replay mitigates forgetting by adding general-domain data, but incurs many additional tokens and leaves target-domain selection unresolved.Source: §1 · p. 1
  • Parameter-agnostic CPT is associated with downward Fisher-diagonal drift on committed high-Fisher coordinates, while low-Fisher coordinates remain largely untouched.The paper presents this asymmetry as a parameter-space mechanism for catastrophic forgetting.Source: §1 · p. 2
  • Each candidate gradient is decomposed into an anchor component for perturbation along high-Fisher directions and a frontier component for update capacity in low-Fisher subspaces.The two components jointly address interference with existing capability and acquisition of new, non-redundant information.Source: §1 · p. 2
  • A log-determinant objective combines the two signals to favor high-value, mutually non-redundant candidates; Sieve-Streaming maximizes it in one pass.The objective is monotone submodular, and the scalable pipeline uses a LoRA adapter subspace, diagonal Fisher, TRAK gradient projections, and Sherman–Morrison updates.Source: §1 · p. 2
  • For medical CPT on TinyLlama-1.1B, a 1B-token selected subset outperforms replay trained with 10B tokens on both adaptation and forgetting, yielding a ≥10× token-efficiency advantage.On CPT of TinyLlama-1.1B and Llama-3.1-8B over medical data, the selector improves target-domain quality while bounding degradation on held-out pretraining benchmarks.Source: §1 · p. 2
  • The selector also improves target-domain quality at a fraction of the tokens used by random and perplexity baselines, while bounding forgetting on held-out benchmarks.Source: §1 · p. 2

2 RELATED WORK

Related work covers continual pre-training, language-model data selection, and subset-selection methods. The paper distinguishes its approach as Fisher-guided selection for CPT, using a token-budgeted unlabeled setting.

Continual pre-training of LLMs

Data selection for language-model training

  • Language-model selection includes mixture optimization, resampling, learned scorers, pruning, and small-set instruction-tuning selectors; this method targets CPT with anchor and frontier components.The related-work passage contrasts those approaches with a distinction between existing-capability directions and uncommitted directions.Source: §2 · p. 3

Coreset selection and submodular subset methods

  • Prior coreset methods include geometry, uncertainty, decision-boundary, gradient-matching, and bi-level approaches; this paper frames selection for token-budgeted, unlabeled language modeling.The paper describes its formulation as providing a constant-factor approximation with per-sample cost independent of candidate-pool size.Source: §2 · p. 3

3 METHOD

The method selects CPT data by measuring how candidate gradients interact with retained capabilities and underused parameter directions, while favoring complementary updates. Its experiments report stronger adaptation with bounded forgetting, including a 1B-versus-10B-token advantage over replay.

3.1 PROBLEM FORMULATION; 4.1 EXPERIMENTAL SETUP

  • Experiments use a medical candidate stream formed from PMC full-text articles (38B tokens) and PubMed abstracts (7B tokens), with TinyLlama-1.1B as primary backbone and Llama-3.1-8B for scale-up validation.Source: §4.1 · p. 7

3.2 OBSERVATION; 3.3 FISHER DECOMPOSITION

  • The anchor view emphasizes high-Fisher directions tied to retained behavior, while the frontier view emphasizes low-Fisher directions available for lower-interference acquisition.Parameter-agnostic CPT instead decreases Fisher mass on committed coordinates while underusing low-Fisher coordinates.Source: §3.2 · p. 4, §3.3 · p. 5

3.4 SUBMODULAR SUBSET SELECTION

  • A two-geometry log-determinant objective rewards diversity in both Fisher-weighted gradient views, giving redundant candidates smaller marginal gains than candidates that add new directions.The objective is monotone submodular, with an offline greedy (1 − 1/e) approximation guarantee.Source: §3.4 · p. 5

3.5 PRACTICAL PIPELINE

  • The streaming pipeline warms up a LoRA operating point, estimates a diagonal Fisher on a proxy set, reweights and projects candidate gradients, then evaluates marginal gains.The implementation uses a one-pass Sieve-Streaming process over candidates.Source: §3.5 · pp. 5–6, Figure 2 · p. 6

4.2 MAIN RESULTS

  • On TinyLlama-1.1B and Llama-3.1-8B, the selector improves adaptation while bounding forgetting; on 8B medical QA, it achieves the best adaptation and lowest forgetting among CPT methods.It also surpasses Galactica across the medical-QA suite and matches or exceeds Mistral-7B on medical QA while reaching general-suite parity.Source: Table 1 · p. 7, §4.2 · p. 8

4.3 ABLATION STUDIES

  • Removing the anchor term inflates forgetting, removing the frontier term stalls adaptation, and replacing Fisher weights with identity removes the curvature signal.Log-determinant selection on raw gradients also shows that diversity without Fisher geometry is insufficient.Source: §4.3 · p. 8
  • Unlike per-sample top-B rules, subset-level log det favors high-magnitude, mutually orthogonal anchor gradients; Sieve-Streaming recovers most of its value at much smaller cost.Per-sample addition cannot suppress mutual redundancy.Source: §4.3 · pp. 8–9

4.4 ANALYSIS

  • At 1B tokens, Ours exceeds Replay at 10B tokens on both adaptation A and forgetting Φ, a 10× token-efficiency advantage.The comparison is against Replay across budgets {1, 2, 4, 10}B, since other baselines do not reach Ours’ adaptation at any budget.Source: §4.4 · p. 9, Table 4 · p. 9

5 CONCLUSION

The paper presents a Fisher-guided submodular CPT selector that balances target-domain acquisition with preservation of pretraining capabilities, while noting limitations in its Fisher approximation and checkpoint dependence.

5 CONCLUSION

  • The selector decomposes candidate gradients into Fisher-weighted anchor and frontier components, then uses a log-determinant objective and Sieve-Streaming to select a high-utility, non-redundant subset.Low-rank approximations support efficient selection.Source: §5 · p. 9
  • The method strictly Pareto-dominates perplexity- and replay-based baselines across 1.1B and 8B LLMs, with at least a 10× token-efficiency gain.Source: §5 · p. 9
  • The framework is currently bounded by a diagonal Fisher approximation and reliance on a warmup checkpoint θ⋆.Source: §5 · p. 9

A IMPLEMENTATION DETAILS

The appendix describes reproducible implementation choices for warmup and Fisher estimation, emphasizing consistent use of a single LoRA checkpoint across pipeline stages.

A IMPLEMENTATION DETAILS

  • The same LoRA adapter, tokenizer, Fisher estimate, and sample-level data stream are used across warmup, Fisher estimation, selection, and final CPT.The appendix presents these choices as necessary for reproducibility.Source: A IMPLEMENTATION DETAILS · p. 16

A.1 WARMUP ADAPTER AND FISHER ESTIMATE

  • For TinyLlama, LoRA modules are attached to query, key, value, and output attention projections with rank 128 and scale 512, then warmed up on target-domain text to produce θ⋆.Warmup makes the LoRA coordinates non-degenerate and provides the point at which candidate gradients are evaluated.Source: §A.1 · p. 16
  • The warmup checkpoint is reused as the source of adapter weights and cached optimizer state because zero-product initialization can otherwise make adapter-gradient signals vanish and Fisher estimates ill-conditioned.Source: §A.1 · p. 16
  • The diagonal Fisher is estimated in the same LoRA subspace and at θ⋆ from squared gradients on held-out reference-distribution proxy documents, with the base model frozen and dropout disabled.Workers average local squared-gradient sums; near-zero Fisher fractions are recorded as a health check for checkpoint mismatch.Source: §A.1 · p. 16

A.2 CANDIDATE STREAM AND SAMPLE-LEVEL GRADIENTS

The pipeline preserves candidate document boundaries and computes sample-level LoRA gradients, then conditions and projects Fisher signatures for streaming selection.

A.2 CANDIDATE STREAM AND SAMPLE-LEVEL GRADIENTS

  • The candidate corpus is consumed as a document-level stream without packing, using the warmup model’s tokenizer family and a fixed selection cutoff length.Unique increasing stream indices support calibration replay, sieve tracking, and reconstruction of the selected set.Source: §A.2 · p. 16
  • A broad perplexity interval filters candidates before backward passes, removing documents that are too easy for θ⋆ or so out-of-distribution that their gradients are noise-dominated.Survivors receive individual flattened LoRA gradients from the retained computation graph.Source: §A.2 · p. 16

A.3 CONDITIONED FISHER SIGNATURES

  • A fixed Adam-style preconditioner rescales raw LoRA-gradient coordinates using warmup moments that are not updated as the stream progresses.This fixed local rescaling means earlier accept/reject decisions do not affect candidate ordering.Source: §A.3 · pp. 16–17
  • A median-based Fisher floor suppresses unstable near-zero coordinates, and fixed Rademacher projections are applied after Fisher reweighting without ℓ2 normalization.The floor preserves relative contrast between high- and low-Fisher directions; retaining projected norms preserves singleton gains.Source: §A.3 · p. 17

B BENCHMARK DECONTAMINATION

The appendix identifies potential overlap between biomedical benchmarks and the target-domain corpus, then documents source blocking, n-gram filtering, and near-duplicate removal.

B BENCHMARK DECONTAMINATION

  • PubMedQA and BioASQ can overlap lexically with the PMC/PubMed candidate pool because their questions or answers draw on PubMed articles.MedQA-USMLE and MedMCQA derive from exams rather than PubMed source documents, so the passage identifies no such lexical path for them.Source: B BENCHMARK DECONTAMINATION · p. 17
  • All methods use a filtered target-domain pool after three ordered steps: source-ID blocking, 13-gram overlap removal, and MinHash near-duplicate removal.The source blocklist covers PubMedQA and BioASQ records; duplicate pairs are dropped at Jaccard similarity of at least 0.8 on 5-gram shingles.Source: B BENCHMARK DECONTAMINATION · p. 17
  • The held-out general-domain shard used to estimate Λ is disjoint from both the candidate stream and every evaluation set.Source: B BENCHMARK DECONTAMINATION · p. 17

C COMPARISON WITH DATA-SELECTION METHODS.

Under matched CPT conditions, four adapted data-selection baselines improve adaptation over Random but forget more than the proposed selector, which performs best on every metric.

C COMPARISON WITH DATA-SELECTION METHODS.

  • All four baselines improve adaptation over Random, but each forgets more than our selector, which is best on every metric.The comparison uses a shared enlarged, decontaminated candidate pool and matched CPT settings.Source: C COMPARISON WITH DATA-SELECTION METHODS · pp. 17–18

C.1 BASELINE SETUP

  • DSIR resamples documents by hashed n-gram importance weights without using the model or a retention term.Held-out medical text defines the target sample, and the candidate pool supplies the raw distribution.Source: §C.1 · p. 18
  • DoReMi is adapted with pretrained TinyLlama as its proxy, held-out medical text as its Group-DRO reference, and four corpus sources as domains.CPT supplies neither the proxy nor the reference required by the original setup.Source: §C.1 · p. 18
  • FisherSFT's port uses one mean-pooled final hidden state per document, making each document a rank-one design update.Its log-determinant lower-bounds the linearized last-layer log-likelihood Hessian's log-determinant.Source: §C.1 · p. 18
  • The SPICE port shares our per-candidate LoRA gradients and TRAK projection, while retaining a conflict penalty against the selected set's mean gradient.Its original method uses instruction-response gradients from a smaller proxy and periodic selection to reduce greedy-search cost.Source: §C.1 · p. 18

D.1 END-TO-END COMPUTE COST

Including selection, the 1B-token pipeline costs less than Replay at 10B tokens and outperforms it on adaptation and forgetting, though its token advantage is smaller in end-to-end compute.

D.1 END-TO-END COMPUTE COST

  • The 1B-versus-10B token budgets count CPT training tokens, including Replay's general-domain replay tokens, not total pipeline compute.At those budgets, Replay trains on 7B medical and 3B general-domain tokens, while our method trains on 1B selected medical tokens; selection also processes discarded candidates.Source: §D.1 · pp. 18–19
  • CPT is the largest single cost at 58.4% of GPU-hours; Fisher estimation and candidate-gradient stages dominate selection, while streaming takes 0.6%.The measured streaming step excludes signature generation, and the perplexity gate avoids candidate backward passes for rejected examples.Source: §D.1 · p. 19
  • At 1B tokens, our pipeline uses 5.7× fewer FLOPs and 3.9× fewer GPU-hours than Replay at 10B, while improving adaptation and forgetting.Adaptation is A = +3.0 vs. −2.0, and forgetting is Φ = 0.6 vs. 0.8; totals include selection.Source: §D.1 · p. 19, Table 8 · p. 20

D.2 ADDITIONAL ABLATIONS

The ablations identify the Fisher floor and balanced default settings as important to preserving adaptation while limiting forgetting.

D.2 ADDITIONAL ABLATIONS

  • Without an effective Fisher floor, frontier-norm inflation reaches 3.1 × 10^4 and performance falls to A = +3.0, Φ = 4.5.A 5th-percentile floor reduces Φ to 1.9; floors from the 25th through 75th percentiles stay within 0.3 of default A and 0.1 of default Φ.Source: §D.2 · p. 20
  • A constant Fisher floor removes frontier Fisher reweighting and yields A = +3.4, Φ = 1.2, though the row also changes the effective scale.The comparison therefore mixes loss of Fisher contrast with a scale change.Source: §D.2 · p. 20
  • Default hyperparameters are robust and near-optimal, while removing warmup collapses adaptation, reducing projection dimension increases forgetting, and small term scales weaken the log-determinant benefit.The default setting has the strongest adaptation gain while keeping forgetting low.Source: §D.2 · p. 20

E LIMITATIONS

The scalable formulation relies on approximations, and the empirical evaluation covers medical-domain CPT on two model backbones.

E LIMITATIONS

  • Curvature is approximated by a diagonal Fisher in the LoRA adapter subspace, with gradients evaluated at a short-warmup operating point.The approach does not use full-parameter curvature matrices.Source: E LIMITATIONS · p. 21
  • Random projection preserves the subset-level log-determinant geometry instead of using full-parameter covariance matrices.The paper identifies richer block-diagonal or low-rank curvature models as natural extensions.Source: E LIMITATIONS · p. 21
  • Empirical claims are scoped to medical-domain CPT on two LLM backbones; other domains, instruction-tuning regimes, and multimodal streams remain for future validation.Source: E LIMITATIONS · p. 21

F EXTENDED RELATED WORK

Prior work addresses CPT forgetting through optimizer stabilization, replay, and model-level approaches, while language-model data selectors span pretraining and instruction tuning. This work distinguishes itself by using Fisher curvature to select CPT data and combining that signal with global subset selection.

Continual pre-training of large language models

  • CPT research includes optimizer stabilization and replay, forgetting analyses, and mitigation through routing or model merging across language, domain, and model settings.The cited work also extends CPT to knowledge-intensive and multimodal models and to architecture-level adaptation.Source: F EXTENDED RELATED WORK · pp. 21–22
  • Unlike EWC, which uses diagonal Fisher as a post-selection parameter regularizer, this selector uses Fisher upstream to choose CPT tokens without an auxiliary penalty.The passage states that adaptation can then proceed with an unconstrained optimizer.Source: F EXTENDED RELATED WORK · p. 22

Data selection for language-model training

  • Language-model selection spans pretraining scorers and mixture methods, instruction-tuning curation, and broader loss- or difficulty-based pruning.The related work also covers coreset methods using geometry, uncertainty, decision boundaries, gradient matching, submodular guarantees, and bilevel selection.Source: F EXTENDED RELATED WORK · pp. 22–23
  • The selector targets post-pretraining CPT, where prior competence must be protected, and decomposes sample contributions along high-Fisher anchor and low-Fisher frontier directions.The passage contrasts this curvature-weighted decomposition with loss, perplexity, and feature-similarity surrogates, which do not weight directions by Fisher mass.Source: F EXTENDED RELATED WORK · p. 22
  • The method differs from prior selectors by deriving candidate utility from pretrained-model Fisher curvature and offering a constant-factor approximation with per-sample cost independent of pool size.Its setting is post-pretraining, rather than pretraining from scratch or small-budget instruction tuning.Source: F EXTENDED RELATED WORK · pp. 22–23
Loading 2610.02593v1…