PaperlayerView exact revision on arXiv

Source-linked AI summary

Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models

Bing Wang, Changchun Li, Xin-Qiang Cai, Lin Yuanbo Wu, Ximing Li, Gang Niu, Masashi Sugiyama

arXiv:2609.30935v1cs.CLcs.AIcs.LG

TL;DR

Continual fine-tuning can help LLMs adapt to emerging tasks, but existing methods may not preserve general-purpose knowledge when pre-training data and gradients are unknown. EOUPCT estimates those gradients from differentiable pseudo data and jointly optimizes the soft prompt and LLM parameters; experiments report improved task performance and reduced forgetting across multiple LLMs.

  • Problem

    Existing continual fine-tuning methods may not preserve pre-trained LLMs’ general-purpose knowledge because their original pre-training data and gradients are unknown and diverse.

    Source: §1 · p. 2

  • Method

    EOUPCT uses a learnable soft prompt and Gumbel-Softmax relaxation to generate pseudo data for estimating pre-training gradients, then jointly optimizes the prompt and LLM parameters with a first-order Pareto optimizer.

    Source: §1 · p. 2

  • Results

    Across six pre-training LLMs, EOUPCT reports a 3.5% improvement in average accuracy and a 3.7% reduction in forgetting against SOTA results.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments report that EOUPCT preserves both task-specific proficiency and general-purpose knowledge while mitigating catastrophic forgetting.

    Source: §1 · p. 2

    Differentiable pseudo-data generation adds training latency relative to standard LoRA, and pseudo-data capacity is bounded by soft-prompt length and generation steps.

    Source: F Limitations · p. 22

Abstract

from arXiv · show

Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs' general-purpose knowledge. Although existing methods, such as orthogonal gradient projection, mitigate the forgetting across various fine-tuning tasks, they fundamentally fail to preserve pre-training LLMs' inherent general-purpose knowledge because the original data and gradients of off-the-shelf pre-training LLMs required by these methods are strictly unknown and highly diverse. To bridge this critical gap, we propose EoupCT, a novel framework designed to Estimate and Orthogonalize Unknown Pre-training gradients for Continual LLM fine-Tuning. Specifically, EoupCT estimates pre-training gradients by dynamically generating pseudo data that is most susceptible to forgetting for new tasks through a learnable soft prompt equipped with Gumbel-Softmax relaxation. Furthermore, we formulate a multi-objective optimization problem and introduce a first-order efficient Pareto optimizer that jointly optimizes LLM parameters and the soft prompt, rigorously enforcing orthogonality between new task updates and the estimated pre-training gradients. Extensive experiments across multiple LLMs demonstrate that EoupCT effectively preserves both task-specific proficiency and inherent general-purpose knowledge, successfully mitigating the catastrophic forgetting.

1 Introduction

Continual fine-tuning helps pre-trained LLMs adapt to new tasks, but can erode both earlier task performance and general-purpose knowledge. EOUPCT estimates unknown pre-training gradients for orthogonal optimization, and experiments report improved task proficiency and knowledge preservation.

1 Introduction

  • Continual fine-tuning is presented as a way for pre-trained LLMs to adapt to the continuing influx of knowledge across emerging real-world tasks.The introduction points to scientific question answering and mathematical reasoning as examples of downstream tasks.Source: §1 · p. 1
  • Existing orthogonal-gradient methods mitigate interference among downstream tasks but do not preserve general-purpose knowledge when pre-training data and gradients are unknown and diverse.The paper identifies commonsense knowledge and logic reasoning as examples of general-purpose knowledge at risk.Source: §1 · pp. 1–2
  • Across six pre-training LLMs, EOUPCT reports a 3.5% improvement in average accuracy and a 3.7% reduction in forgetting against SOTA results.Evaluation samples 15 SuperNI tasks, measures task proficiency on SuperNI and general-purpose knowledge on MMLU.Source: §1 · p. 2
  • EOUPCT estimates unknown pre-training gradients and orthogonalizes them with continual fine-tuning updates to address the preservation gap.It generates pseudo data with a learnable soft prompt and Gumbel-Softmax relaxation, then jointly optimizes the prompt and task-specific parameters using a first-order Pareto optimizer.Source: §1 · p. 2

2 Related Work

Related work groups continual LLM fine-tuning methods into orthogonal gradient projection, data rehearsal, and parameter isolation. The paper focuses on orthogonal projection, noting that existing approaches may neglect pre-training gradients and leave general-purpose knowledge vulnerable.

2 Related Work

  • Orthogonal gradient projection, data rehearsal, and parameter isolation are described as three approaches to mitigating catastrophic forgetting in continual LLM fine-tuning.Examples include OLoRA [31] and InfLoRA [14] for orthogonal or subspace constraints, self-synthesized rehearsal [8], and LoRAMoE [19] [40].Source: §2 · pp. 2–3
  • Parameter isolation decouples pre-trained weights from new-task parameters but requires retaining historical models for inference, while integration can still erode general-purpose knowledge.LoRAMoE [19] [40] is given as an example of low-rank adaptation-based mixture-of-experts.Source: §2 · p. 3
  • Existing orthogonal methods may constrain weights or hidden-feature subspaces rather than gradients, and typically omit unknown pre-training gradients.The paper argues this omission leaves general-purpose knowledge vulnerable during continual adaptation.Source: §2 · p. 3

3 Our Proposed EOUPCT

EOUPCT addresses continual fine-tuning by estimating unknown pre-training gradients and optimizing updates to protect both earlier tasks and general-purpose knowledge. Its method generates differentiable pseudo data, formulates joint objectives for task learning and knowledge stability, and uses a first-order optimizer to avoid second-order computation.

3 Our Proposed EOUPCT

  • Continual fine-tuning adapts a pre-trained LLM across downstream tasks while seeking to retain both its general-purpose knowledge and knowledge from earlier tasks.Orthogonal gradient projection computes the current-task gradient and projects it into the null space of gradients from previous tasks.Source: §3 · p. 3

3.1 Method Overview

  • EOUPCT estimates unknown pre-training gradients and jointly optimizes the soft prompt and LLM parameters to keep new-task gradients orthogonal to those estimates.The method overview describes pre-training gradient estimation and Pareto orthogonal optimization as its two modules.Source: §3.1 · pp. 3–4

3.3 Multi-Objective Formulation of Pareto Orthogonal Optimization

  • The multi-objective formulation combines new-task plasticity, stability on pseudo data, vulnerability targeting, and orthogonality constraints across task and estimated pre-training gradients.The prompt targets general-purpose knowledge most susceptible to forgetting, while estimated pre-training gradients are encouraged to represent distinct pre-training tasks.Source: §3.1 · p. 4, §3.3 · p. 5

3.2 Pre-training Gradient Estimation

  • A learnable soft prompt steers the pre-trained LLM to generate pseudo data, using Gumbel-Softmax relaxation to make the generated token sequence differentiable.The method uses a knowledge-distillation loss on the pseudo data to estimate pre-training gradients.Source: §3.2 · pp. 4–5, §3.1 · p. 4

3.4 Efficient Pareto Orthogonal Optimizer

  • The first-order Pareto optimizer avoids Hessian computation by identifying vulnerable knowledge with a virtual task-gradient step, projecting updates away from historical task gradients, and removing conflicts with estimated pre-training gradients.The projected update is described as having zero inner product with previously learned tasks; the conflict-removal step targets the estimated pre-training gradient.Source: §3.4 · p. 6

4 Experimental Evaluation

Experiments evaluate continual fine-tuning on SuperNI and general-purpose knowledge on MMLU across multiple task orders and model families. EOUPCT is compared with continual fine-tuning baselines, alongside sensitivity, ablation, and forgetting-dynamics analyses.

4.1 Main Results

  • Across model families and task orders, EOUPCT reports the highest average SuperNI ROUGE and lower forgetting, alongside higher MMLU accuracy and lower forgetting across categories.On Qwen3-4B under Order 1, ROUGE versus OLoRA rises from 48.9 to 50.7 and forgetting falls from 9.0 to 6.0; on Gemma2-9B, MMLU STEM accuracy rises from 42.9 to 52.3 and forgetting falls from 10.3 to 0.9.Source: §4.1 · p. 8

4.2 Sensitivity Analysis

  • On Qwen3-4B and Llama3-3B, increasing soft prompt length L and sequence length M generally improves ROUGE until excessive lengths, such as 32, stop or slightly reverse gains.The sensitivity experiments use SuperNI ROUGE under Order 1 and test L and M values of 4, 8, 16, and 32.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 Ablation Experiments

  • Removing any EOUPCT component degrades task-specific and general-knowledge performance; removing vulnerability or orthogonality objectives produces additional forgetting.For Qwen3-4B, removing C2 lowers ROUGE from 50.7 to 47.6 and increases forgetting from 6.0 to 10.3; for Llama3-3B, removing P&PO lowers ROUGE from 44.0 to 42.1 and raises forgetting from 10.5 to 12.2. Removing N&PO increases forgetting across settings, while removing N&NO produces the most severe degradation.Source: §4.3 · pp. 8–9, Table 2 · p. 9

4.4 Forgetting Dynamics

  • As SuperNI tasks accumulate, EOUPCT’s MMLU accuracy decreases more slowly and remains higher than CLoRA’s, sometimes exceeding the initial pretrained model’s accuracy.The forgetting-dynamics analysis tracks accuracy in three MMLU categories during sequential learning.Source: §4.4 · p. 9, Figure 3 · p. 9

5 Conclusion

The conclusion presents EOUPCT as a framework for estimating unknown pretraining gradients and enforcing orthogonality during continual fine-tuning. The appendix frames theoretical proofs around estimating the pretraining distribution, resolving forgetting, and preserving pretraining knowledge.

5 Conclusion

  • EOUPCT combines soft-prompt-generated pseudo-data with a first-order Pareto optimizer to jointly optimize the prompt and LLM parameters while enforcing gradient orthogonality.Experiments across multiple LLMs are reported to maintain task-specific proficiency and broad general-purpose capabilities.Source: §5 · p. 10

A Theoretical Proofs

  • The appendix frames proofs around whether continuous Gumbel-Softmax relaxation estimates discrete pretraining distributions and whether orthogonal projection can theoretically resolve forgetting.These are posed as theoretical research questions.Source: A Theoretical Proofs · p. 13
  • A further proof question asks whether retaining only a soft prompt and its generated surrogate sequences can preserve the repository of pretrained knowledge.Source: A Theoretical Proofs · p. 13

A.1 Can Gumbel-Softmax Relaxation Estimate Discrete Pre-training Sequences?

The theoretical analysis bounds the soft-to-discrete relaxation gap, characterizes the Pareto projection as the constrained closest update, and relates worst-case prompt protection to global forgetting under stated approximation conditions.

A.1 Can Gumbel-Softmax Relaxation Estimate Discrete Pre-training Sequences?

  • Under an L-Lipschitz distillation loss, the expected soft-versus-discrete loss gap is bounded by O(τ), and annealing τ toward zero makes the approximation gap vanish.The bound uses the linear-in-τ expected distance between the Gumbel-Softmax probabilities and the discrete one-hot vector.Source: §A.1 · p. 13

A.2 Can EOUPCT Theoretically Resolve the Catastrophic Forgetting?

  • The conflict-case update is the closest direction to the task gradient that is orthogonal to the protection gradient, enforcing zero interference with estimated pre-training knowledge.The derivation formulates a constrained distance minimization and identifies the projection as its solution.Source: §A.2 · pp. 13–14

A.3 Can Maintaining Only a Soft Prompt Preserve Pre-trained Knowledge?

  • If the adversarial solver finds a prompt whose worst-case loss is bounded by ε, the theorem bounds expected forgetting over the true pre-training distribution.The theorem’s statement conditions this conclusion on successful worst-case prompt identification and the Pareto-orthogonal update’s loss bound.Source: §A.3 · p. 14
  • The accompanying remark says global prompt-space coverage is unnecessary, relying instead on a task-specific adversarial coreset for the vulnerable local manifold.Its claimed bound applies within the task-specific projection subspace, with local approximation error δ_local ≪ δ_global.Source: §A.3 · pp. 14–15

B Complexity Analysis of EOUPCT

The analysis replaces a Hessian-requiring prompt optimization with first-order backpropagation, then compares EOUPCT’s iteration cost with standard LoRA fine-tuning.

B Complexity Analysis of EOUPCT

  • The analysis frames EOUPCT as addressing the second-order cost of optimizing its multi-objective formulation with a first-order Pareto optimizer.The appendix introduction points to a comparison of baseline and EOUPCT time complexity and empirical computational budgets.Source: B Complexity Analysis of EOUPCT · p. 15, §B.1 · p. 16

B.1 Efficient First-Order Pareto Optimization

  • EOUPCT’s virtual-step approximation treats virtual weights as fixed, bypassing the Hessian and reducing spatial and temporal complexity from O(N^2) to O(N).The direct prompt-gradient objective requires a mixed second-order derivative whose computation is described as intractable for LLMs.Source: §B.1 · pp. 15–16

B.2 Time Complexity Compared with Standard LoRA Baselines

  • With surrogate length M held constant, EOUPCT scales linearly with LLM size like standard LoRA, at a small constant-factor computational overhead.Its per-step cost is O((M + 3)FPLLM + 2BPLoRA + BPPrompt).Source: §B.2 · p. 16

B.3 Empirical Computational Budget

On 8×GPU runs, EOUPCT used 2.0× LoRA’s training time, saving 23% versus CLoRA and 37% versus LoRAMoE. The section also outlines its optimization workflow and the LLM families used for evaluation.

B.3 Empirical Computational Budget

  • EOUPCT took 2.0× standard LoRA’s time, saving 23% versus CLoRA and 37% versus LoRAMoE across evaluated model scales.The comparison reports empirical training time on 8×GPUs.Source: §B.3 · pp. 16–17, Table 3 · p. 17

C Algorithm and Notations

  • EOUPCT uses a first-order bilevel training paradigm: a virtual step targets vulnerable pre-training data, then projections reduce gradient conflicts.The workflow applies null-space projection and Pareto-optimal orthogonal projection before updating model parameters.Source: C Algorithm and Notations · p. 17

D Experimental Setups

  • The experiments evaluate continual fine-tuning on multi-task benchmarks across open-source pre-training LLMs.The section describes the models, datasets, baselines, metrics, and implementation details used in the experiments.Source: D Experimental Setups · p. 17

D.1 Pre-training LLMs

  • Evaluation spans Qwen3, Llama3, and Gemma2, using two model scales from each family to cover varied architectures and capacities.The selected scales are Qwen3-4B/8B, Llama3-3B/8B, and Gemma2-2B/9B.Source: §D.1 · p. 18

D.2 Benchmark Datasets

The evaluation pairs sequential task learning on SuperNI with repeated MMLU assessment of general-purpose knowledge. It compares EOUPCT with representative LoRA and continual fine-tuning baselines.

D.2 Benchmark Datasets

  • SuperNI evaluation uses 15 tasks across five categories, trains on each task once, and tests all tasks after each update under three task orderings.The categories include question answering, information extraction, sentiment analysis, summarization, and dialogue generation.Source: §D.2 · p. 19, Table 5 · p. 19, Table 6 · p. 19
  • MMLU evaluation uses a subset spanning broad domains, with 5-shot contexts and test-set accuracy assessed after every task update.The development split constructs the contexts; the test split measures performance.Source: §D.2 · p. 19, Table 7 · p. 19

D.3 Baselines Methods

  • The comparison includes LoRA, LoRAMoE, O-LoRA, GainLoRA, and CLoRA as representative continual fine-tuning baselines.The set covers standard LoRA variants and state-of-the-art continual fine-tuning strategies.Source: §D.3 · p. 20

D.4 Evaluation Metrics

The section defines forgetting measures for task-specific performance on SuperNI and pre-training knowledge on MMLU. It also specifies implementation settings and memory-conscious adaptations for gradient estimation.

D.4 Evaluation Metrics

  • SuperNI uses ROUGE-L for generation quality and defines forgetting as the drop in task t’s ROUGE score from immediately after learning it to after all T tasks.Overall SuperNI forgetting is the average across learned tasks; a lower rate indicates better retention.Source: §D.4 · p. 20
  • MMLU uses pooled accuracy and defines general-purpose knowledge forgetting as the original model’s accuracy minus its accuracy after all T tasks.The measure quantifies the trade-off between downstream skill acquisition and retaining broad pre-training knowledge.Source: §D.4 · p. 20

D.5 Implementation Details

  • Implementation uses frozen pre-training backbones with LoRA fine-tuning, AdamW at 2 × 10^-4, and evaluation after each task.Inputs are truncated to 1024 tokens, generation is capped at 50 tokens, and MMLU follows a 5-shot protocol with a 2048-token context limit.Source: §D.5 · p. 21
  • To stabilize gradient estimation, EOUPCT limits surrogate sequences to moderate lengths and computes Gumbel-Softmax over the top 50 tokens at each step.The adaptations address memory use and gradient instability from differentiating over large vocabularies across long sequences.Source: §D.5 · p. 21

E Performance Dynamics on SuperNI

EOUPCT outperforms baselines in most reported SuperNI task scenarios. Table 9 identifies a Qwen3-4B comparison with GainLoRA.

E Performance Dynamics on SuperNI

F Limitations

EOUPCT’s limitations include added training latency and a prompt-bounded pseudo-data capacity. The checklist response connects the paper’s motivation to estimating and orthogonalizing unknown pre-training gradients.

F Limitations

  • Gumbel-Softmax pseudo-data generation adds computational overhead and increases forward- and backward-pass latency compared with standard LoRA.The authors say the method avoids storing historical pre-training datasets and describe refinements to reduce overhead.Source: F Limitations · p. 22
  • Pseudo-data capacity is bounded by soft-prompt length L and generation steps M, and may saturate for extremely long, heterogeneous task sequences.The authors identify dynamically expanding prompts or hierarchical prompt routing as possible future directions.Source: F Limitations · p. 22

NeurIPS Paper Checklist

  • The checklist response identifies unknown pre-training gradients as the motivation for EOUPCT’s estimation and orthogonalization approach.The authors refer readers to the Abstract and Introduction for details of these contributions.Source: NeurIPS Paper Checklist · p. 23

2. Limitations

The authors report discussing limitations, providing experimental and reproducibility details, and supporting key results with trials across distinct task orders.

Limitations

  • The authors say Appendix F discusses limitations, focusing primarily on computational efficiency as an area for future investigation.The checklist also identifies computational efficiency as a limitation topic warranting discussion.Source: §2 · p. 23

Theory assumptions and proofs

  • The authors state that Appendix A provides theoretical proofs addressing the paper’s three primary theoretical questions and underlying assumptions.Source: §2 · p. 24

Open access to data and code

  • The authors report providing implementation code and documentation for its use.Source: §2 · pp. 25, 28

Experimental setting/details

  • Appendix D provides experimental details, while a sensitivity analysis examines selection of key hyperparameters L and M.The checklist response says those details are intended to support understanding of the results.Source: §2 · p. 25

Experiment statistical significance

  • Table 1 results aggregate three independent trials with distinct random seeds and task orders, which the authors report consistently demonstrate statistical significance.Source: §2 · p. 25

Crowdsourcing and research with human subjects

  • The authors report using existing publicly available benchmarks without additional crowdsourced data collection.Source: §2 · p. 28
Loading 2609.30935v1…