PaperlayerView exact revision on arXiv

Source-linked AI summary

DRIVE: Fine-tuning via Data Contribution- and Diversity-aware Weighting with Prior Regularization

Qing Liu, Xinrui Chen, Yi Du, Ou Wu, Weiyao Zhu

arXiv:2610.32501v1cs.LG

TL;DR

When pre-training data is unavailable, fine-tuning can forget prior knowledge, and existing weighting methods do not jointly model retention, adaptation, diversity, and sample characteristics. DRIVE combines these signals in a differentiable weighting objective. Across vision and language benchmarks, it consistently surpasses SOTA methods.

  • Problem

    Existing sample-weighting methods emphasize either retention or task adaptation and lack a unified differentiable objective for contribution, diversity, and sample characteristics.

    Source: §1 · pp. 1–2

  • Method

    DRIVE combines a Shapley-additivity decomposition of anti-forgetting and adaptation contributions with covariance-determinant diversity and a KL-anchored sample-characteristic prior.

    Source: §1 · p. 2

  • Results

    Across vision and language benchmarks, DRIVE consistently surpasses SOTA methods.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    In linear and local nonlinear regimes, the unified modeling tightens the stability–plasticity trade-off, while diversity accelerates convergence and bolsters estimation robustness.

    Source: §6 · p. 10

    The setting excludes access to pre-training data and continual task sequences, and formal guarantees rely on linear or locally linearized regimes with smoothness and local-approximation assumptions.

    Source: §3 · p. 3, G Limitations · p. 38

Abstract

from arXiv · show

Single-stage fine-tuning is pervasive, yet catastrophic forgetting persists when the source pre-training data is inaccessible. Data weighting affords a natural lever to modulate sample influence. Yet existing weighting methods lack a principled mechanism to explicitly arbitrate knowledge retention against new-task adaptation, and provide no unified differentiable objective to accommodate auxiliary signals such as sample diversity and sample characteristics. This is because current schemes privilege one goal over the other, and no explicit gradient-compatible encoding exists for such signals. To address this, we propose DRIVE, a principled and efficient weight assignment framework. It derives per-sample weights via a unified differentiable objective where three coupled signals co-evolve: Shapley additivity splits contribution into anti-forgetting and adaptation components, which are reshaped by a covariance-based diversity term and stabilized by a sample-characteristic prior. Far from a mere combination, this yields an end-to-end gradient-based scheme balancing stability against plasticity. Theoretically, we prove that DRIVE enjoys superior stability-plasticity guarantees over existing weighting schemes and enhances parameter estimation robustness. Empirically, it attains superior overall performance, faster convergence, and stronger robustness. Code at https://github.com/luciabcd/DRIVE.

1 Introduction

Fine-tuning can cause catastrophic forgetting when pre-training data is unavailable, while existing weighting methods emphasize either retention or adaptation and lack a unified differentiable objective for contribution, diversity, and sample characteristics. DRIVE combines these signals in an end-to-end weighting framework.

1 Introduction

  • Prior fine-tuning methods address forgetting through parameter-space constraints or sample-space weighting, selection, and augmentation.The source pre-training data is typically unavailable in practice.Source: §1 · p. 1
  • Existing weighting methods measure sample contribution mainly through new-task learning, while FLOW measures its role in preserving pre-trained knowledge.Neither perspective alone explicitly balances stability and plasticity.Source: §1 · pp. 1–2
  • Diversity and sample characteristics are complementary signals, but existing weighting methods incorporate only parts of them and lack a unified differentiable objective.Diversity methods such as clustering and DPP are difficult to encode as continuous weighting terms, while sample characteristics do not directly measure fine-tuning contribution.Source: §1 · p. 2
  • DRIVE combines a Shapley-additivity decomposition, covariance-determinant diversity term, and KL-anchored sample-characteristic prior in one differentiable weighting framework.The decomposition separates anti-forgetting from task adaptation and supports a tunable stability–plasticity trade-off.Source: §1 · p. 2

2 Related Work

Prior work addresses forgetting through parameter- and sample-space strategies, and studies data weighting and diversity through separate approaches. DRIVE instead proposes a unified weighting objective that models contribution, diversity, and sample prior together.

Mitigating Catastrophic Forgetting in Fine-tuning

  • Forgetting mitigation has targeted parameter or sample spaces, including regularization, orthogonal updates, parameter isolation, and replay.The paper positions data weighting as a way to optimize the trade-off between anti-forgetting and task adaptation.Source: §2 · pp. 2–3

Data Weighting in Fine-tuning

  • Data-weighting studies use training-dynamics signals, quality proxies, or analyses of fine-tuning contributions, while DRIVE jointly models contribution, diversity, and sample prior.The cited approaches include loss alignment, meta-learning, reward scores, and distribution-shift analyses.Source: §2 · p. 3

Data Diversity in Fine-tuning

  • Existing diversity strategies use discrete selection, including clustering, coreset selection, DPP, and complexity-aware filtering; DRIVE integrates a covariance-determinant regularizer into weighting.Other cited work studies response-level diversity for alignment.Source: §2 · p. 3

3 Methodology

DRIVE infers sample weights for data-oblivious, single-task fine-tuning by combining contribution, diversity, and sample-characteristic prior signals in a differentiable objective. A projected iterative procedure makes the weighting practical for fine-tuning.

3 Methodology

  • DRIVE considers single-task fine-tuning without access to pre-training data or continual task sequences, then infers optimal weights for model fine-tuning.Weights are nonnegative and normalized to sum to one.Source: §3 · p. 3

3.1 Data Contribution and Diversity Measures

  • Shapley additivity decomposes each sample’s contribution into anti-forgetting and task-adaptation components, combined with a tunable trade-off coefficient.The two components correspond to separate cooperative games over the dataset.Source: §3.1 · p. 3
  • Because validation-based contribution measures are unavailable in the data-oblivious setting, DRIVE uses a loss-based proxy that correlates with validation-based Shapley contributions at Spearman’s ρ=0.83.The estimates match gradient-based variants within 0.3% in accuracy while running about 2× faster.Source: §3.1 · p. 3
  • DRIVE makes set-level diversity differentiable by adapting D-optimal design to continuous data weighting through a covariance-determinant term.The method contrasts this with discrete clustering and notes limitations of combinatorial coverage, pairwise-repulsion, and discrete-subset DPP approaches.Source: §3.1 · pp. 3–4

3.2 Data Weight Optimization Objective via Prior Regularization

  • A KL-anchored prior encodes task-agnostic sample characteristics, while the objective combines contribution, feature-volume expansion, and the prior reference distribution.The iterative fixed-point solver normalizes weights to the simplex; for τ>0 the objective is strictly concave and the iteration converges linearly under the stated contraction condition.Source: §3.2 · p. 4
  • DRIVE combines the uniform prior with diversity regularization, upweighting peripheral samples while anchoring weights to the prior to preserve coverage.The uniform prior alone over-concentrates on dense regions, while diversity alone favors peripheral samples at the majority’s expense.Source: §3.2 · p. 5, Figure 2 · p. 5

3.3 Efficient Data Weighting for Fine-tuning

  • To reduce covariance inversion from O(d^3) to O(k^3), DRIVE sketches features with a Gaussian random matrix and uses a FIFO buffer of 128 recent samples.The projected fixed-point iteration obtains weights for the local support set, restricts and renormalizes them within the batch, and uses them in the weighted loss.Source: §3.3 · p. 5

4 Theoretical Analysis

The analysis studies DRIVE in linear and locally nonlinear regimes, alongside convergence of its iterative solver. It derives stability–plasticity and robustness properties under stated assumptions.

4 Theoretical Analysis

  • The theoretical analysis examines DRIVE in linear and locally nonlinear regimes and establishes linear convergence of its fixed-point solver.The reported empirical results are described as consistent with these theoretical insights.Source: §4 · p. 5

4.1 Linear and Local Nonlinear Regimes

  • In the linear regime, lowering λ reallocates weight toward task-relevant conflicting samples, while log|Σ_w| enlarges weighted feature volume and improves conditioning.Under controlled local approximation error, the analysis says these effects improve adaptation toward θ* and optimization robustness.Source: §4.1 · p. 6
  • In the local nonlinear regime, shifting λ toward task adaptation reallocates weight to conflicting directions, while diversity emphasizes under-covered feature directions.The analysis states these effects reduce the local residual gradient and support better-conditioned local optimization when the nonlinear remainder is controlled.Source: §4.1 · p. 6

4.2 Convergence Analysis of the Iterative Solver for Eq. (4)

  • When ηL_h<τ, the fixed-point map is contractive and its iterates approach the unique maximizer at rate O(ξ^T).The objective is τ-strongly concave for τ>0 under the stated ridge-regularized covariance setting.Source: §4.2 · p. 6

5 Experiments

Experiments compare DRIVE with data-oblivious fine-tuning baselines across vision and language tasks, then examine its components, hyperparameters, convergence, robustness, and contribution scores. Across the evaluated settings, DRIVE improves overall performance while retaining task adaptation and reducing forgetting.

5.2 Comparison on Vision and Language Tasks (Q1)

  • With ResNet-50, DRIVE gains 2.0%, 1.54%, and 3.16% average accuracy over FLOW on CIFAR-10, CIFAR-100, and Flowers102, respectively.The reported gains come with higher target accuracy while largely retaining ImageNet-1K performance.Source: §5.2 · p. 8
  • As a plug-and-play module, DRIVE boosts target accuracy by 3.0% over ℓ2-Regularization and beats LoRA+FLOW by 0.76% in overall average.These comparisons are on Llama-3.1-8B; FLOW improves target accuracy by 1.6% over the ℓ2-Regularization baseline.Source: §5.2 · p. 8, Table 4 · p. 8
  • DRIVE exceeds the strongest baseline by 1.68, 1.43, and 1.57 percentage points overall on Gemma-2-2B, Llama-3.2-3B, and Llama-3.1-8B, respectively.All three improvements remain statistically significant after Holm correction across model-wise comparisons (adjusted p≤0.0152).Source: Table 3 · p. 7, §5.2 · p. 8

5.3 Ablation Study (Q2)

  • The entropy-informed prior gives slightly better general and overall performance than the uniform prior while maintaining comparable target accuracy.The prior assigns more mass to lower-predictive-entropy samples; the uniform prior remains a strong default without extra estimators.Source: §5.3 · p. 9, Table 6 · p. 9
  • On Llama-3.1-8B, DRIVE surpasses the discrete DRIVE-Cluster variant by 2.27%, supporting a more adaptive mechanism than rigid cluster balancing.DRIVE-Cluster uses seven semantic groups with balanced aggregate cluster weights.Source: §5.3 · p. 9, Table 7 · p. 9

5.4 Hyperparameter Sensitivity Analysis (Q3)

  • On ResNet-50, the static balance parameter λ favors plasticity at lower values and retention at higher values; λ=0.8 is the best static balance, while a 1→0 decay performs better.Small τ favors target adaptation, moderate τ gives the best overall trade-off, and overly large τ makes weights nearly uniform; increasing η helps target performance up to 1.0, after which generalization degrades.Source: §5.4 · p. 9, Figure 3 · p. 9

5.5 Analysis of Convergence and Robustness (Q4)

  • DRIVE stabilizes significantly earlier than baselines without diversity constraints and reduces average pairwise parameter distance by 43.46% across four seeds.The distance analysis uses ResNet-50 on CIFAR-100; on the most challenging 10% of samples, DRIVE maintains satisfactory performance while FLOW performs poorly and Standard FT suffers catastrophic forgetting on the retention set.Source: §5.5 · pp. 9–10, Figure 6 · p. 10, Figure 4 · p. 10

5.6 Validation of Loss-Based Contribution Scores (Q5)

  • On ResNet-50/Flowers102, loss-based contribution scores correlate with validation-gradient Shapley approximations (Spearman’s ρ=0.83; Pearson’s r=0.81; both p<0.001).In a single-seed Llama-3.1-8B/MetaMathQA diagnostic, shuffling or reversing score rankings reduces Overall by 2.17 and 7.06 percentage points, respectively.Source: §5.6 · p. 10

6 Conclusion

DRIVE combines contribution, diversity, and a sample prior in one differentiable objective for data-oblivious fine-tuning. The appendix explains why certain contribution measures are unavailable in this setting and reports that its loss-based proxies align with validation-based references.

6 Conclusion

  • DRIVE recasts contribution into retention and adaptation with a tunable Shapley-additive decomposition, captures diversity with a differentiable covariance-determinant term, and anchors weights with a prior regularizer.The paper reports tighter stability–plasticity trade-offs in linear and local nonlinear regimes, linear convergence of the solver, and consistent superiority over state-of-the-art methods across vision and language benchmarks.Source: §6 · p. 10

A.1 Inapplicable Contribution Measures

  • Exact Shapley values require repeated retraining over exponentially many subsets, making them infeasible for large-scale fine-tuning and usable only as a theoretical reference.Validation-based approximations and meta-learning also require auxiliary retain or target validation sets, which the strict data-oblivious setting does not assume.Source: §A.1 · p. 18

A More Details Supporting Our Contribution Function C

  • The adopted loss-based proxy is computable from the model and sample alone, avoiding the validation supervision unavailable in the strict data-oblivious setting.The paper evaluates proxy agreement with validation-based Shapley-style references using trend, top-sample overlap, ranking, and linear correlation measures.Source: §A.2 · pp. 18–19

A.2 Statistical Support for the Loss-Based Contribution Proxies

  • Loss-based contribution scores show strong agreement with validation-based Shapley-style references, including for the unified score used to determine DRIVE’s sample weights.The reported agreement covers the decomposed anti-forgetting and task-adaptation components as well as their unified score.Source: §A.2 · p. 19

A.3 Alternative Validation-Free Metrics

DRIVE can use validation-free contribution surrogates, with loss-based scores as its practical default. Alternatives trade accuracy, computational cost, robustness, and alignment with optimization dynamics.

Alternative validation-free metrics

  • The alternative validation-free surrogates average their scores over L prediction positions, set to sequence length for LLMs and 1 for vision classification.The main text uses loss-based contributions.Source: §A.3 · p. 19

Gradient-based metrics

  • Gradient norms measure sample-level update influence more directly than scalar loss, but require per-sample backward computation and are significantly more expensive at scale.The gradient-based variant is theoretically appealing and related to GraNd-style sample valuation.Source: §A.3 · p. 19

Error-based metrics

  • EL2N-based scores are bounded and relatively robust, but are less directly aligned with optimization dynamics than loss-based scores.The alternative sets a_i to negative EL2N under the pretrained model and t_i to EL2N under the current model.Source: §A.3 · p. 19

Uncertainty-based metrics

  • Entropy scoring emphasizes uncertainty but may over-focus on ambiguous or difficult samples and can be less stable for balancing retention and adaptation.The appendix identifies prediction entropy as an alternative motivated by semantic uncertainty.Source: §A.3 · p. 19

Margin-based metrics

  • Margin-based scoring prioritizes target samples near the decision boundary while favoring prior knowledge the model has confidently mastered.Source: §A.3 · p. 20

Replacement study

  • Gradient-based DRIVE-GRA scores highest overall, but takes roughly twice the training time of standard DRIVE; loss-based DRIVE remains competitive and most efficient.The comparison is on Llama-3.2 3B.Source: §A.3 · p. 20, Table 11 · p. 20

B Solution for Objective (4) and Convergence Analysis

Appendix B analyzes DRIVE's weight objective, update rules, convergence, and randomized covariance approximation. It establishes uniqueness and conditional solver convergence, and clarifies the approximation's effective-rank boundary.

B Solution for Objective (4) and Convergence Analysis

B.1 Concavity and Uniqueness of Eq. (4)

  • For τ > 0, the objective is strictly concave on the simplex and therefore has a unique global maximizer.The ridge term keeps the diversity term continuous on the closed simplex.Source: §B.1 · pp. 20–21

B.2 Derivation of the Iterative Update Rule

  • The stationarity condition yields weights determined by contribution, diversity, and the prior; under a uniform prior, the update simplifies to Softmax.The uniform prior assigns p_k = 1/N to every sample.Source: §B.2 · p. 21

B.3 Optimization for the No-Prior Ablation

  • Without the KL prior, the no-prior objective has an implicit stationarity condition and no Softmax-form closed update, so it uses simplex-constrained ascent instead.The ablation retains DRIVE's iterative covariance-refresh framework and projects updates onto the simplex.Source: §B.3 · p. 22

B.4 Convergence of the Weight Solver

  • When ηL_h < τ, the fixed-point solver is contractive and converges linearly to the unique maximizer; bounded features provide a Lipschitz condition.Source: §B.4 · p. 22

B.5 Justification for Randomized Dimensionality Reduction

  • The Gaussian sketch reduces covariance inversion from O(d^3) to O(k^3), but k = 128 is a low-rank approximation, not a full-rank guarantee when local rank exceeds k.Approximation quality depends on the effective intrinsic rank of centered features.Source: §B.5 · p. 23

C Detailed Theoretical Proofs

  • The detailed proofs cover linear-regime contribution reallocation, adaptation, and diversity-induced conditioning, then extend the analysis to local nonlinear fine-tuning with Hessian remainder control.Source: C Detailed Theoretical Proofs · p. 23

C.1 Proof for the Linear Regime in Sec. 4.1

The linear-regime proof links contribution reallocation to adaptation along conflicting directions and diversity regularization to better conditioning and robustness. The local nonlinear analysis gives related guarantees under local assumptions and explicit remainder control.

C.1 Proof for the Linear Regime in Sec. 4.1

  • DRIVE’s unified contribution score reallocates weight toward task-relevant conflicting samples as λ decreases, improving adaptation toward θ* along directions stability-oriented weighting can underweight.In the linear regime, increasing the effective eigenvalue along a conflicting direction reduces its fitting gap; the diversity term lifts under-covered directions, improves conditioning, and accelerates convergence under the stated step-size condition.Source: §C.1 · p. 24

C.2 Local Nonlinear Analysis with Hessian Remainder Control

  • In the local nonlinear regime, contribution reallocation reduces the residual gradient toward θ*, while diversity can improve local curvature and conditioning when the nonlinear remainder is controlled.The guarantee is local, not global; descent and perturbation robustness depend on local curvature and the remainder being non-dominant.Source: §C.2 · pp. 25–26

D More Experimental Details

  • The experimental-details appendix lays out implementation, datasets, evaluation metrics, and training configurations for vision and language-model experiments.The supplied overview specifies that baseline details, vision datasets, LLM benchmarks, and implementation settings are covered.Source: §D.1 · p. 26, §D.2 · p. 27, §D.3 · p. 27

D.1 Baseline Details

  • The baseline comparison spans parameter-space methods, parameter-efficient fine-tuning, sample selection, sample reweighting, and token-level loss reweighting in a data-oblivious setting.FLOW weights samples inversely to pre-trained loss, whereas DEFT-UCS uses discrete clustering-based subsampling.Source: §D.1 · pp. 26–27

D.2 Vision Datasets and Evaluation Details

  • Vision evaluation uses target-task accuracy and ImageNet-1K retention accuracy, with their arithmetic mean reported as Overall Avg.The downstream datasets include CIFAR-10, CIFAR-100, Flowers102, and additional fine-grained and domain-specific tasks.Source: §D.2 · p. 27

D.3 LLM Datasets and Evaluation Details

  • LLM evaluation covers commonsense reasoning, general capabilities, mathematical target tasks, and code generation, with Accuracy or Pass@1 reported by task.Overall Avg. averages the general-capability score and target-task accuracy; benchmark settings include both zero-shot and few-shot evaluation.Source: §D.3 · p. 27

D.4.1 Vision Model Implementation Details

The appendix specifies training configurations for vision and language models, then reports additional vision results. Across six downstream vision datasets, DRIVE achieves the highest average accuracy on most datasets for both ResNet architectures.

D.4.1 Vision Model Implementation Details

  • Vision experiments search learning rates across ResNet-18 and ResNet-50 on nine downstream datasets, selecting settings on a held-out validation split before one final test evaluation.All vision baselines use consistent optimizer and schedule configurations, with SGD, momentum 0.9, and weight decay 5 × 10−4.Source: §D.4.1 · p. 28

D.4.2 LLMs Implementation Details

  • LLM runs use uniform training configurations across Gemma-2-2B, Llama-3.2-3B, and Llama-3.1-8B, and report means and standard deviations over five runs.Dataset-specific batch sizes and sequence limits differ between MetaMathQA and Magicoder; learning rates are tuned by method–model combination.Source: §D.4.2 · p. 28

E Additional Experimental Results and Analysis

  • The additional-results appendix covers detailed vision and LLM evaluations, sensitivity analyses, component ablations, and computational efficiency.The listed analyses include the balance parameter, prior threshold, diversity coefficient, and DRIVE components.Source: E Additional Experimental Results and… · p. 28

E.1 Analysis of Vision Tasks

  • DRIVE achieves the highest average accuracy on most of six downstream datasets for both ResNet-18 and ResNet-50, including Food101 and EuroSAT.The average reflects representation preservation and target-task adaptation; DRIVE also maintains higher ImageNet-1K retention than Standard FT while achieving higher target-task accuracy than ℓ2-Regularization.Source: §E.1 · p. 29, Table 13 · p. 29

E.2 Experiments on Vision Transformers

The appendix evaluates DRIVE on ViT, Magicoder, parameter sensitivities, component ablations, and efficiency, and tests the statistical reliability of its main LLM gains. It also describes a Shapley-style analysis for separating sample contributions to retention and adaptation.

E.2 Experiments on Vision Transformers

  • Nearly 18% of general accuracy is recovered by DRIVE versus Standard FT on ViT-B/16 Food101, with negligible adaptation cost and competitive target accuracy.DRIVE also exceeds Linear Probe by more than 7.5% on the target task and FLOW by approximately 1.6% in target accuracy.Source: §E.2 · pp. 29–30

E.3 Experiments on Language Models

  • DRIVE achieves the highest overall average across evaluated Magicoder models; on Llama-3.2-3B, it improves the score by 2.86% over LoRA and 0.98% over FLOW.TALR on Llama-3.2-3B obtains slightly higher general capability but lower HumanEval accuracy.Source: §E.3 · p. 30, Table 16 · p. 31

E.4 Impact of Balance Parameter λ on Language Tasks

  • λ=0.8 achieves the best overall balance on Llama-3.2-3B, while the reverse increasing-λ schedule yields sub-optimal results.Under the reverse schedule, Overall Avg falls to 54.92% for Gemma-2-2B and 66.75% for Llama-3.1-8B; the authors report that early high λ provides initial anchoring.Source: §E.4 · p. 30, Table 17 · p. 31, §E.5 · p. 31

E.6 Impact of Prior Regularization τ

  • The 10th-percentile τ gives the best overall performance on Llama-3.2-3B, while η sensitivity is non-monotonic and overall performance peaks at η=1.00.The authors report that overly large τ smooths weights toward a uniform prior, while excessive diversity emphasis can reduce general-capability metrics.Source: §E.6 · p. 31, Table 19 · p. 32, §E.7 · p. 32

E.8 Ablation Study on Vision Tasks

  • The full method achieves 72.34% average accuracy on CIFAR-100 and 83.85% on Flowers102; removing KL regularization costs approximately 9.5% on CIFAR-100.With λ=0, target accuracy rises to 86.40% on CIFAR-100 and 93.67% on Flowers102, while ImageNet performance falls to 57.49% and 71.98%, respectively.Source: §E.8 · pp. 32–33, Table 21 · p. 33

E.9 Computational Efficiency Analysis

  • DRIVE adds 11.8%–14.7% GPU time per epoch on vision tasks and roughly 17% on MetaMathQA, with projection reducing high-dimensional geometric calculations to linear scaling.The reported costs are for CIFAR-100 with ResNet models and MetaMathQA with Gemma-2B or Llama-3-8B.Source: §E.9 · p. 33, Table 22 · p. 33, Table 23 · p. 33

E.10 Paired Statistical Analysis of the Main LLM Results

  • All three five-seed paired comparisons favor DRIVE, with confidence intervals excluding zero and Holm-adjusted p≤0.0152.Hedges’ g_z ranges from 1.45 to 2.00; the reported analysis pairs DRIVE with the strongest baseline for each model.Source: §E.10 · pp. 33–34, Table 24 · p. 34

F.1 Shapley Analysis of Anti-Forgetting vs. Task Adaptation

  • The Shapley-style analysis uses gradient inner products to estimate each sample’s contribution to preserving pre-trained knowledge and learning the new task.It sums marginal contributions along the optimization trajectory using full validation-set gradients at each step.Source: §F.1 · p. 34, §F.1.1 · pp. 34–35

F.1.2 Empirical Results and Analysis

The analyses test loss-based contribution proxies, examine the conflict between retention and adaptation, and compare diversity and covariance-approximation strategies. They report that a single random projection performs nearly as well as an ensemble, while DRIVE’s diversity objective preserves covariance structure that a diagonal approximation omits.

F.1.2 Empirical Results and Analysis

  • Pearson’s r=0.6493 for anti-forgetting and r=0.8679 for task adaptation support the loss-based proxies’ correlations with their respective Shapley contributions.The authors use these efficient proxies for scalable DRIVE training.Source: §F.1.2 · p. 35
  • Synergistic high-loss samples maintain ImageNet performance comparable to the pre-trained baseline while achieving competitive target accuracy; the conflicting mixture triggers severe forgetting.The comparison partitions the top 40% high-loss Flowers102 samples by gradient inner product with the retention task.Source: §F.1.2 · p. 35, Table 25 · p. 35
  • Pearson’s r=−0.628 and only 8.7% co-contributing samples indicate an empirical conflict between normalized adaptation and anti-forgetting contributions.The scatter plot shows these normalized contributions; 44.8% of samples are adaptation-biased and 31.4% retention-biased.Source: §F.1.2 · p. 35, Figure 7 · p. 35

F.2 Sample Diversity

  • Figure 7 plots normalized anti-forgetting contributions against task-adaptation contributions for samples.The figure caption identifies the plotted quantities but does not state a result.Source: Figure 7 · p. 35

F.2.1 Diversity in Fine-tuning

  • DRIVE integrates diversity into its continuous weighting objective through a differentiable covariance determinant rather than relying on selection-based or discrete metrics.The section contrasts this unified objective with existing fine-tuning diversity strategies.Source: §F.2.1 · p. 36

F.2.2 Details of DRIVE-Cluster

  • DRIVE-Cluster uses a cluster-balanced prior to down-weight samples from dense clusters in global competition, without hard cluster-level constraints.The prior gives each semantic cluster equal initial probability mass; the variant sets η=0 and encodes diversity in the prior.Source: §F.2.2 · p. 36

F.3.1 Comparison with Diagonal Approximation

  • A diagonal covariance approximation cannot distinguish diverse orthogonal features from redundant collinear ones with identical marginal variances, so it omits structure DRIVE uses to penalize redundancy.The analysis motivates preserving off-diagonal interactions through random projection; the ensemble comparison finds only marginal gains from multiple projections.Source: §F.3.1 · p. 37, §F.3 · p. 37, §F.3.2 · p. 37

F.3.2 Empirical Analysis of Single vs. Ensemble Random Projections

  • 0.04% on CIFAR-100 and 0.03% on Flowers102: increasing projection ensembles from M=1 to M=10 yields only marginal average-accuracy gains.The authors therefore use one Gaussian projection in the main experiments.Source: §F.3.2 · p. 37, Table 26 · p. 37

F.4 Empirical Verification of Dynamic Adaptation on Hard Samples

On a fixed probe set of difficult samples, DRIVE’s unweighted loss decreases steadily while FLOW’s remains high and unstable. The projection-dimension study identifies k=128 as a practical fidelity–efficiency trade-off for Gemma-2-2B on MetaMathQA.

F.4 Empirical Verification of Dynamic Adaptation on Hard Samples

  • ≈2.0: DRIVE’s final loss on the hard-sample probe set, compared with FLOW’s nearly flat loss at ≈6.0.DRIVE’s trajectory is smooth and decreasing, while FLOW fluctuates and stalls; Figure 8 depicts the trajectories.Source: §F.4 · p. 38, Figure 8 · p. 38
  • DRIVE’s determinant-based diversity regularization stabilizes weighted-covariance condition number κ(Σw) at order 10^3, three orders below FLOW’s order 10^6, contrasting better-conditioned geometry with FLOW’s spectral collapse and ill-conditioned landscape.Source: §F.4 · p. 38

F.5 Sensitivity to the Projection Dimension 𝑘

  • k=128 is the practical elbow point for Gemma-2-2B on MetaMathQA: smaller sketches reduce performance, while larger ones bring marginal gains at sharply rising training cost.The study varies sketch dimension while holding other settings fixed.Source: §F.5 · p. 38

G Limitations

The stated limitations concern the scope of the theoretical guarantees, approximations and alignment assumptions in the method, and computational overhead. The paper also notes potential broader impacts and describes its datasets, models, and released assets.

G Limitations

  • The formal guarantees apply under linear or locally linearized nonlinear assumptions, within a bounded neighborhood of pre-trained weights; support beyond that regime is empirical.Extending the results to fully non-convex global dynamics of large transformers remains open.Source: G Limitations · pp. 38–39
  • The JL projection may attenuate diversity signals at high intrinsic dimension, the FIFO buffer captures local rather than global diversity, and effectiveness relies on reasonable pre-trained/target representation alignment.These are stated constraints on approximation, data-stream coverage, and domain alignment.Source: G Limitations · p. 39

Broader Impacts

  • Diversity estimation adds covariance and determinant overhead, reported as approximately 17% for LLMs; the experiments use public datasets and benchmarks without user data or human-subject studies.The paper also says released assets are anonymized code, configurations, and reproduction scripts, not new datasets or model checkpoints.Source: G Limitations · p. 39, Broader Impacts · p. 39, Licenses and Assets · p. 39

Related papers

Loading 2610.32501v1…