PaperlayerView exact revision on arXiv

Source-linked AI summary

Which and When to Admit: Gradient Admission for Data-Centric Small Language Model Finetuning

Hongyu Cao, Yanchi Liu, Kunpeng Liu, Xujiang Zhao, Wei Cheng, Zhengzhang Chen, Yanjie Fu, Haifeng Chen

arXiv:2610.07553v1cs.LGcs.AI

TL;DR

Heterogeneous supervision can create conflicting gradients, stale selection scores, and overwriting updates in LoRA’s constrained subspace. GRADE combines state-aware gradient-aligned sample selection with a self-calibrating step gate. Across three backbones, it is the only method with positive worst-case and highest mean gain over LoRA.

  • Problem

    Under heterogeneous supervision, LoRA faces conflicting gradients, static selection that cannot track changing model state, and later updates that can overwrite useful directions.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    GRADE continually selects samples aligned with an evolving multi-task direction and admits an update only when its probe loss is below a running EMA baseline.

    Source: §1 · p. 2

  • Results

    GRADE is the only method with strictly positive Worst-∆ (+0.18 pp) and achieves the highest Mean-∆ (+0.99 pp) across three backbones versus LoRA.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    GRADE’s selection improves gradient coherence, while its gate suppresses destructive updates near subspace saturation.

    Source: §5.2 · p. 8, §6 · p. 10

    GRADE uses forward-probe approximations, a fixed LoRA parameterization, and evaluation on 8–9B instruction-tuned backbones.

    Source: A Limitations · p. 12

Abstract

from arXiv · show

LoRA fine-tuning adapts small language models (SLMs) to heterogeneous instruction data within a low-rank update subspace, making it vulnerable to three structural problems: conflicting gradients that cancel, static data selection that cannot track evolving learning dynamics, and subspace saturation that causes later updates to overwrite useful directions. We argue that effective adaptation therefore requires controlling which data-induced gradients enter the LoRA subspace and when. We propose GRADE (GRadient-Aligned Data-centric rEcipe), a data-centric framework combining two mechanisms: a state-aware selector that continually admits samples aligned with the evolving multi-task gradient field, and a self-calibrating step-level gate that rejects updates likely to cause destructive overwrite near saturation. Across three current-generation backbones and a heterogeneous seven-dataset instruction pool, GRADE outperforms strong data-selection and PEFT-stabilization baselines in accuracy and robustness. It is the only method to improve consistently over standard LoRA on every architecture, while producing more coherent gradient trajectories and less destructive overwrite. These results show that successful SLM adaptation depends not only on which data are selected, but also on which gradients are allowed to enter and persist in the constrained update subspace.

1 Introduction

Heterogeneous instruction data can produce unstable LoRA fine-tuning because conflicting gradients meet in a restricted update subspace. GRADE couples state-aware sample selection with a gate that decides whether updates are committed.

Motivation

  • Heterogeneous instruction data can make fine-tuning unstable: adding data may not improve performance, mixtures can affect backbones differently, and continued updates can overwrite earlier gains.The introduction frames the problem as deciding which data-induced gradients enter the constrained LoRA subspace and when updates are committed.Source: §1 · p. 1

Our Perspective

  • In LoRA’s narrow subspace, incompatible gradients can cancel, static selection scores become stale as the model changes, and later updates can overwrite useful directions.The paper attributes these fragilities to uncontrolled gradient admission under restricted update capacity.Source: §1 · p. 2

Proposed Framework

  • GRADE continually re-scores candidate samples against an evolving multi-task reference direction and commits an update only when its probe loss is below a running EMA baseline.When the update does not improve relative to the model’s recent trajectory, the optimizer step is skipped.Source: §1 · p. 2

Contributions

  • Across Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B, GRADE alone achieves the highest cross-architecture mean gain and a strictly positive worst-case gain over LoRA.The framework couples gradient-aligned sample admission with a self-calibrating step-level update gate.Source: §1 · p. 2

2 Related Work

Related work addresses data composition, LoRA optimization, and modular adaptation through distinct mechanisms. GRADE is positioned as controlling whether candidate data and the resulting updates are admitted.

Data-centric fine-tuning

  • Gradient-aligned selection includes GRAD-MATCH, LESS [31], and ClusterUCB [30], while other approaches use difficulty, forgetting, or diversity signals.These alternatives target gradient approximation, influence-aware similarity, scoring cost, hardness, instability, or coverage.Source: §2 · p. 3

LoRA fine-tuning under restricted and adaptive subspaces

  • Adaptive LoRA methods allocate low-rank capacity [19] [35], while stability-aware methods regularize updates after batches form [7].The paper distinguishes GRADE by asking whether candidate data should enter training and whether each step remains constructive.Source: §2 · p. 3

Routed and modular LoRA fine-tuning

  • Mixture-of-adapters methods route inputs across specialized modules, whereas GRADE gates training-time updates rather than routing requests.GRADE compares admitted-batch loss with its running EMA during optimization.Source: §2 · p. 3

Forward-only and gradient-free adaptation

  • MeZO uses forward passes for memory-efficient optimization, but GRADE uses forward probes for compatibility scoring and retains standard LoRA optimization.The probes are not presented as a standalone optimizer.Source: §2 · p. 3

3 Background and Problem Setup

The setup treats heterogeneous supervision as a gradient-compatibility problem within LoRA’s shared low-rank space. It formalizes admission at both the sample and optimizer-step levels.

Problem setup

  • Heterogeneous datasets induce gradients that may be weakly aligned or conflicting across tasks, so updates that help one task can interfere with another.The training pool spans different task families, and the evaluation distribution aggregates multiple benchmarks.Source: §3 · p. 3

Restricted adaptation under heterogeneous gradients

  • Because LoRA restricts updates to a narrow low-rank subspace, incompatible gradients can cancel, and near saturation later updates may overwrite useful directions.The same update space is shared across datasets, tasks, and optimization stages.Source: §3 · p. 3

Observed fragility

  • The paper identifies gradient conflict, stale static-selection scores, and subspace overwrite as recurring failure modes.These are framed as consequences of uncontrolled admission under heterogeneous supervision.Source: §3 · p. 3

Problem formulation

  • At each step, the policy must select samples compatible with a shared multi-task direction and decide whether to commit the aggregated optimizer update.The stated goal is improved downstream utility while remaining stable under heterogeneous supervision and restricted adaptation capacity.Source: §3 · p. 4

Design implication

  • The required compatibility, evolving aggregate direction, and subspace saturation state are not directly observable at scale, motivating control over which gradients enter and when updates remain constructive.The paper contrasts gradient admission with asking only which examples should be selected.Source: §3 · p. 4

4 The GRADE Recipe

GRADE is a closed-loop LoRA recipe that couples state-aware sample selection with a self-calibrating decision about whether to commit each update. The updated adapter informs future compatibility scoring.

4 The GRADE Recipe

  • GRADE re-scores candidate samples against an evolving multi-task reference direction, then commits the admitted-batch update only when its probe loss is below a running EMA.The updated adapter changes the reference direction used at the next iteration.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Gradient-Aligned, State-Coupled Data Admission

  • The selector uses a forward-probe proxy for gradient alignment and admits the top-scoring samples; its magnitude-weighted projection favors samples that are both compatible and able to produce meaningful updates.The reference direction is recomputed at the current model state, so the criterion tracks what the model still needs.Source: §4.1 · pp. 4–5
  • The reference set comes from the heterogeneous training pool, and its direction changes with the current LoRA state even though the set itself stays fixed.A single-task reference would make admission task-specific rather than target the aggregate evaluation panel.Source: §4.1 · p. 5
  • The keep fraction is set from the pool’s gradient geometry: it approaches 1 when datasets are nearly orthogonal and 1/T when they are strongly aligned.The keep fraction remains fixed during training while admitted samples change as the reference direction evolves.Source: §4.1 · p. 6

4.2 Self-Calibrating Step-Level Update Admission

  • The step-level gate uses admitted-batch probe loss against a running EMA baseline to skip updates that are not constructive relative to the recent trajectory.The gate is armed after the probe-loss trajectory reaches a plateau; before then, GRADE commits all steps like standard LoRA.Source: §4.2 · pp. 6–7
  • Sample admission makes the batch direction more coherent before optimization, while step admission prevents updates from being committed unconditionally after saturation.The two decisions are coupled through model state: admitted updates change future sample-compatibility scores.Source: §4.2 · p. 7

5 Experiments

GRADE jointly controls which sample gradients enter LoRA training and which updates are committed, addressing instability under heterogeneous supervision. Across three backbones, it achieves positive Worst-∆ and the highest Mean-∆, while selection and step admission each contribute distinct stability effects.

5.1 Experimental Setting

  • The pool comprised seven instruction datasets with ∼40% quality-degraded samples (n ≈21K), evaluated on ARC-Challenge, HellaSwag, and Winogrande.The pool spans reasoning, QA, summarization, dialogue, and code tasks.Source: §5.1 · p. 7
  • Experiments compared three current-generation backbones against data-selection, LoRA-importance, stability-aware, and gradient-conflict baselines under a unified LoRA setting.The backbones were Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B; LoRA used rank r = 16 and α = 32.Source: §5.1 · p. 7

5.2 Experimental Results

  • GRADE alone achieved positive Worst-∆ (+0.18 pp) across all three backbones and the highest Mean-∆ (+0.99 pp); every baseline regressed on at least one architecture.Methods controlling only selection or only fine-tuning-side updates did not avoid per-architecture regression.Source: §5.2 · p. 8
  • Gradient-aligned selection improved the intra/inter coherence ratio by ∼5×, preserved dataset proportions better than loss-based selection, and varied by dataset across epochs.The results support directional compatibility and changing model state as relevant to selection.Source: §5.2 · p. 8, Figure 3 · p. 8
  • With the gate enabled, accuracy increased by +0.04 to +0.38 pp across backbones, while the steady-state SKIP fraction approached the 0.5 loss-decrement prediction.The gate filters updates near the boundary between constructive and destructive regimes.Source: §5.2 · p. 8, Figure 4 · p. 9
  • Selection improves input-signal quality, whereas step admission controls update persistence; removing either mechanism reintroduces instability.Selection-only improves coherence but still suffers post-saturation degradation; the gate alone leaves a residual cross-architecture gap.Source: §5.2 · p. 9
  • Sample-only admission scored −0.22 versus −0.15 pp for plain LoRA on Mean-∆; step-only still regressed on 2/3 backbones, while joint admission avoided regression.The admission lattice compares the four sample/step admission combinations at matched lr=2e−5.Source: §5.2 · p. 9, Figure 5 · p. 9

6 Conclusion and Limitations

GRADE treats adaptation as controlled admission: it selects samples aligned with the evolving gradient field and gates updates near saturation. The appendix details its scope, implementation choices, and operating regime.

6 Conclusion and Limitations

  • GRADE couples state-aware gradient-aligned sample selection with a self-calibrating step-level gate to suppress destructive updates near subspace saturation.The conclusion frames both mechanisms as controlling which data-induced updates enter and persist in the LoRA adapter.Source: §6 · p. 10

A Limitations

  • GRADE uses forward-probe approximations, a fixed LoRA parameterization, and 8–9B instruction-tuned backbones, leaving richer estimators and broader regimes for future work.The authors identify adaptive parameterizations and broader model/data regimes as natural next steps.Source: A Limitations · p. 12

B Formal Problem Setup

  • The deployment objective is aggregate downstream utility on a test distribution, optimized only over trainable parameters in the frozen-backbone LoRA subspace.The formal setup allows the training and test distributions to differ.Source: B Formal Problem Setup · p. 12

C Boundary Handling of the Closed-Form Keep-Rate

  • All three reported backbone measurements have positive inter-dataset cosine, leaving the keep-rate safeguards inactive and the raw keep rates within the non-binding [0.1, 1.0] clamp.The reported raw r⋆ values lie in [0.378, 0.779]; Table 2 covers the three main-table backbones, while Figure 6 includes eleven measurements overall.Source: C Boundary Handling of the Closed-Form… · p. 13, Table 2 · p. 13, Figure 6 · p. 13

D Runtime and FLOPs Overhead

  • GRADE adds forward-only selection computation, while its gate adds no compute savings; the stated anticipated wall-clock cost is approximately 1.3–1.5× vanilla LoRA.The appendix describes this as a trade of extra forward-only compute for higher per-step quality.Source: D Runtime and FLOPs Overhead · p. 14

E Hyperparameter Sensitivity Plan

  • The planned sensitivity analysis varies probe scale, EMA decay, plateau threshold, reference-set size, and cosine-normalized versus magnitude-weighted projection.The plan identifies EMA decay as the dominant axis because smaller decay delays gate latching.Source: E Hyperparameter Sensitivity Plan · p. 14

F Probe-Direction Sampling Across LoRA Layers

  • A single globally normalized random probe direction is shared across candidates, preserving an unbiased inner-product estimator without per-layer rescaling that would bias it.The expected per-layer perturbation contribution is proportional to that layer’s LoRA parameter count.Source: F Probe-Direction Sampling Across LoRA… · p. 14

G–J Evaluation, Controls, Positioning, and Reference Set

K Broader Benchmarks and Pool-Composition Stress Plan

The appendix plans broader benchmark and pool-composition checks while distinguishing generality from ranking. It also outlines tests of the stability of single-probe sample rankings.

K Broader Benchmarks and Pool-Composition Stress Plan

  • The planned broader suite tests non-regression on MMLU, BBH, GSM8K, and TruthfulQA, while clean- and noisy-pool variants test whether the keep rate tracks heterogeneity.The three-task evaluation surface is rank-stable and disjoint from the seven-dataset training pool; the authors explicitly limit the broader-benchmark claim to non-regression, not superiority.Source: K Broader Benchmarks and… · pp. 15–16

L Single-Probe Variance and Multi-Direction Probes

  • The shared single-probe direction cancels dominant common-mode score noise in within-batch rankings; planned checks compare admitted-set overlap across probes and multi-direction variants.The appendix says absolute-score variance scales as Θ(1/q), while pairwise ranking differences benefit from shared-direction cancellation.Source: L Single-Probe Variance and… · p. 16

M Mathematical Derivations

The derivations connect forward-probe directional derivatives to gradient alignment and use a loss-decrement identity to interpret the gate. They also qualify what the plateau and routing proxy can establish.

M Mathematical Derivations

  • A shared random unit-vector probe estimates each sample’s LoRA-subspace gradient inner product with the reference gradient in expectation, up to the known 1/q factor.The alignment score is used for ranking, so its common 1/q scaling is absorbed by the top-k selection rule.Source: M Mathematical Derivations · p. 16
  • A first-order expansion relates the admitted-batch loss decrement to the LoRA-projected update, with an O(η^2) residual that is negligible in the stated small-step regime.The appendix says the default step size and observed gradient magnitudes meet the stated condition with margin; mini-batch noise can dominate when the LoRA gradient norm is small.Source: M Mathematical Derivations · pp. 16–17
  • A plateau in selected-batch loss is interpreted as signal moving toward LoRA-orthogonal directions, but the identity alone cannot distinguish this from convergence on the admitted batch.The authors rely on positive intra-dataset gradient-flow coherence and the selection mechanism to argue against global convergence as the interpretation.Source: M Mathematical Derivations · p. 17
  • The gate’s SKIP action withholds base-LoRA updates after the plateau trigger, when weak admitted gradients would otherwise be forced into a saturated subspace.The probe uses an EMA-smoothed loss because mini-batch noise can dominate the decrement signal at small gradient norms.Source: M Mathematical Derivations · p. 17
  • The routing statistic’s gradient denominator is a single-layer language-model-head proxy, not a full-network gradient norm; it can be biased downward when gradient energy concentrates in shallow layers.The LoRA gradient term is divided by the number of modified transformer layers for scale matching; without this normalization, the trigger reportedly never fires.Source: M Mathematical Derivations · p. 17

N GRADE Training Algorithm

  • GRADE’s procedure combines top-k alignment selection with a probe-loss plateau trigger, then either skips or performs the base-LoRA update.The gate state latches open once the relative-slope test falls below its tolerance; selection determines what to learn, while the gate determines whether to commit a step.Source: N GRADE Training Algorithm · pp. 18–19

O.4–O.5 Forward Probe and Reference Gradient

  • The forward perturbation estimates alignment without full gradients, and the reference direction is recomputed from a small representative reference set as the model state evolves.The appendix describes a shared random LoRA-subspace direction and a perturbation scale for computing directional derivatives.Source: §O.4 · p. 20, §O.5 · p. 20

O.6–O.7 Routing Signal and Stability Constraint

  • The routing trigger uses an EMA-smoothed probe loss and dimensionless relative-slope tolerance; its cooldown is derived from the EMA response time, while update-magnitude bounding is a safeguard.The reported EMA smoothing is α = 0.1, with a cooldown derived to align with the EMA settling time.Source: §O.6 · p. 21, §O.7 · p. 21

O.8 Datasets

The dataset pool is deliberately heterogeneous so the paper’s gradient-conflict diagnostics can be observed. The appendix specifies how it is assembled and how the ablations isolate selection and routing.

O.8 Datasets

  • The pool spans reasoning, knowledge-intensive QA, summarization, dialogue, and code-like instructions, with heterogeneity treated as a precondition for observing gradient conflict.The authors explain that a homogeneous pool would not yield the same interpretable intra/inter-cosine contrast or per-dataset diagnostics.Source: §O.8 · pp. 21–22
  • Seven source datasets are capped at 3,000 samples before merging and shuffling, then split into 90% training and 10% in-pool validation; the reference set is drawn from that validation split.Six datasets reach the cap, while ARC-Challenge-train contributes 1,119 samples; the resulting pool is approximately 19,119 examples.Source: §O.8 · p. 22
  • The paper separates in-pool validation used for monitoring and routing from the external ARC-Challenge, HellaSwag, and Winogrande test splits used for headline accuracy.The appendix states that the headline evaluation is disjoint from the training pool and is not computed on the in-pool validation set.Source: §O.8 · pp. 21–22

O.9 Figure 1 setup details

  • Figure 1’s cosine heatmap uses 50 samples per dataset from the heterogeneous pool and reports mean per-sample LoRA-gradient cosine within or between datasets.The specified snapshot is Gemma-2-9B at the default-LoRA, learning-rate 2 × 10^-5, epoch-2 checkpoint.Source: §O.9 · pp. 22–23

O.10 Ablation and Ablation-Variant Configurations

  • The selection-disabled variant admits all data but retains routing; routing-disabled retains selection without triggering the plateau detector, and frozen-selection reuses epoch-0 scores.These variants isolate compatibility-aware filtering, capacity allocation, and the re-scoring feedback loop, respectively.Source: §O.10 · p. 23

P Single-Backbone Mechanism Diagnostics (LLaMA-2-7B)

On the single legacy LLaMA-2-7B diagnostic, adaptive selection turns over the admitted population as criterion sharpness rises; direct mechanism validation on the three main backbones remains pending.

Scope

  • The mechanism diagnostics use one legacy backbone; matched per-epoch GFC ablations on Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B are left for future work.Cross-architecture transferability is supported indirectly by the main-table Mean-∆ and Worst-∆ rankings, not direct GFC trajectories.Source: P Single-Backbone Mechanism… · p. 23

Adaptive vs. frozen selection: population-level dynamics

  • Adaptive and frozen selection share the geometric interference reduction, but adaptive GRADE turns over about two-thirds of its admitted population each epoch (J≈0.33 vs. ≈0.67).The population-dynamics diagnostic contrasts adaptive selection with frozen selection, which reorders cached scores.Source: P Single-Backbone Mechanism… · p. 23, Figure 7 · p. 23
  • Criterion sharpness rises from 1.00× to 1.20× while consecutive-epoch Jaccard stays near 1.00×, indicating the churn is not explained by noise.The reported sequences are 1.00× → 1.05× → 1.20× for sharpness and 1.00× → 1.00× → 1.05× for Jaccard.Source: P Single-Backbone Mechanism… · p. 23, Figure 7 · p. 23

Routing-statistic diagnostic

  • The probe statistic is the unique candidate in Figure 8’s desirable lower-right region, while ρ̂(t) and its log variant lie in the upper-left.The figure frames the comparison as monotonicity with respect to step versus within-epoch noise.Source: P Single-Backbone Mechanism… · p. 24, Figure 8 · p. 24

Q Diagnosis and Replacement of the Capacity Ratio

The appendix diagnoses why the original capacity ratio fails, validates a plateau-loss replacement, and reports component, dataset-level, and estimator checks on the gate and selection mechanisms.

Diagnosis and replacement

  • The original ρ̂(t) moves opposite its intended capacity-exhaustion signal: its step correlation is +0.39 (+0.71 smoothed), as its numerator rises and denominator falls.The numerator’s Spearman correlation is +0.20 and the last-layer VJP denominator’s is −0.10; EMA-smoothed probe loss is more monotone and less noisy than ρ̂(t).Source: Q Diagnosis and Replacement of the… · p. 24

Plateau detector

  • EMA-smoothed probe loss scores mean |Spearman| 0.606 and CV 0.049, versus 0.266 and 0.246 for ρ̂(t), making it 2.3× more monotone and 5× less noisy.Its trigger is a relative-slope plateau detector using dimensionless εrel rather than a hand-tuned threshold.Source: Q Diagnosis and Replacement of the… · p. 24

End-to-end validation

  • Plateau-triggered routing matches hand-tuned τ=0.7 within ∆=−0.00015 on the 4-task metric, with zero hand-tuned routing hyperparameters.Per-task changes are ARC +0.94, HellaSwag +0.15, Winogrande −0.32, and GSM8K −0.83.Source: Q Diagnosis and Replacement of the… · p. 24

Dataset-aggregate footprint

  • Gate gain rises with residual gradient norm across datasets, but Dolly is a high-norm, low-gain deviation because it produces few SKIP actions.The gate’s dataset-selective gain is consistent with its EMA-thresholded filter rather than a uniform metric improvement.Source: Q Diagnosis and Replacement of the… · pp. 24–25

ARC-C-excluded sensitivity check

  • Removing ARC-C attenuates the Figure 9 slope from α≈0.0022 to α≈0.0012, but leaves the relationship positive; Dolly remains the high-norm, no-trigger deviation.ARC-C supplies the high-end point in the original fit, while the remaining colored datasets retain a positive-slope arrangement.Source: Q Diagnosis and Replacement of the… · p. 25, Figure 9 · p. 25

Latching continuity and cascade prevention

  • The latch changes only future step decisions, so parameters remain unchanged at the transition; its one-shot design prevents cascade re-triggering.Before latching, a three-detection-window cooldown debounces EMA noise.Source: Q Diagnosis and Replacement of the… · p. 25

Estimator comparison

  • Figure 10 compares forward-probe and exact vmapped gradients by held-out loss against wall-clock cost, and by seed-paired held-out loss across three seeds.The caption reports mean ±σ for the cost–loss Pareto panel.Source: Figure 10 · p. 26

Component magnitude breakdown

  • Under matched conditions, LoRA-only scores 0.6145, LESS-only 0.6154, routing-only 0.6180, and full GRADE 0.6235.Routing-only exceeds LESS-only by +0.26 points, and full GRADE exceeds routing-only by +0.55 points; LESS is a selection-only yardstick, not a GRADE component.Source: Q Diagnosis and Replacement of the… · p. 26

R Gradient Estimator Variants and Selection-as-Regularization

The estimator sweep finds that the forward probe nearly matches the exact gradient end to end despite weak per-sample ranking correlation. The authors interpret selection as closer to structured regularization than precise gradient estimation.

Variants

  • The tested alternatives are K-direction random probes, antithetic SPSA with symmetric ±u pairs, and a last-layer lm_head proxy equivalent to GraNd/EL2N.The sweep also compares the forward probe with the exact vmapped gradient.Source: R Gradient Estimator Variants and… · p. 26

End-to-end results

  • The exact gradient is best; the forward probe is within +0.35 pp, while K=4 probing is worse by +0.21 pp, antithetic SPSA by +0.27 pp, and the last-layer proxy by +0.93 pp.At feasible K, variance reduction does not close the gap to the exact gradient; matching it would require K=O(D) directions, computationally comparable to the exact gradient.Source: R Gradient Estimator Variants and… · p. 26

Per-sample ranking versus end-to-end utility

  • Per-sample score correlations with the exact gradient are −0.22 for the forward probe and +0.29 for the last-layer proxy, yet the proxy performs worst end to end.The forward probe has near-zero correlation but matches the exact gradient within 0.35 pp.Source: R Gradient Estimator Variants and… · pp. 26–27

Selection-as-regularization

  • The authors interpret selection as closer to structured regularization than precise gradient-alignment estimation at the studied pool scales and model sizes.The accompanying Phase-1 figure locates per-sample alignment against loss on a noisy heterogeneous pool, with reasoning-task anchors distinguished from five instruction-style datasets.Source: R Gradient Estimator Variants and… · p. 27, Figure 11 · p. 27

S Selection Dynamics Detail

The selection diagnostics show that alignment-based selection distinguishes samples by gradient direction rather than loss, while the loss-based rule’s performance depends on whether the pool is noisy. The appendix also reports that GRADE’s learning-rate choice is backbone-specific, with different rates preferred across the three backbones.

S Selection Dynamics Detail

  • 91% of gsm8k+arc anchors fall below the loss median, so loss-top-50% retains only ~9%; alignment-top-50% retains 49–54% across datasets.Within-dataset Spearman correlations between loss and alignment are |ρ|<0.04 on every dataset.Source: S Selection Dynamics Detail · p. 27
  • 8.68→10.38 (+19.6%) is the alignment-score gap from epoch 0 to 2, while training loss falls from 1.006 to 0.852.The growing gap indicates that the selection criterion becomes more discriminative as the model improves.Source: S Selection Dynamics Detail · p. 27
  • On the noisy pool, gradient-aligned selection (+0.0182) and random selection (+0.0160) are tied-best, while loss-based selection is negative (−0.0055); on the clean pool, the three strategies are near-tied.Loss-based improvement changes from −0.0055 on the noisy pool to +0.0006 on the clean pool, indicating a noise-specific pathology.Source: Figure 12 · p. 28, S Selection Dynamics Detail · p. 28
  • Dolly15k’s selection rate falls from 0.549 to 0.495, while WizardLM’s rises from 0.500 to 0.557; adaptive-selection Jaccard overlap is 0.329 and 0.333, versus 0.664 and 0.683 when frozen.The changing rates track progressive difficulty, while frozen-selection variation comes solely from shuffled mini-batch order.Source: S Selection Dynamics Detail · p. 29

T Per-Backbone Learning-Rate Selection for GRADE

  • GRADE’s learning rate is chosen from 2×10−4 and 2×10−5 using validation loss at the end of the first half-epoch, before routing or selection-fraction effects.The rule is monotone and uses two prespecified candidates, not a continuous sweep.Source: T Per-Backbone Learning-Rate Selection… · p. 29
  • 2×10−5 gains +0.34 pp on Llama-3.1-8B and +1.36 pp on Gemma-2-9B, but loses 0.39 pp on Qwen3-8B versus 2×10−4.A shared rate would forgo the Gemma gain or accept a −0.21 pp Qwen3-8B regression.Source: T Per-Backbone Learning-Rate Selection… · p. 29

U Cross-Architecture Protocol Details

The cross-architecture appendix describes a keep-rate rule tied to measured gradient covariance and reports a three-task average for one configuration. Its extended related-work discussion positions GRADE alongside alternative parameterizations, optimization methods, data proxies, routing approaches, and gradient-free adaptation.

U Cross-Architecture Protocol Details

  • All three backbones use the same ARC-C, HellaSwag, and Winogrande ranking surface; per-architecture calibration is limited to GRADE hyperparameters tied to initial gradient geometry.GSM8K is reported as an auxiliary metric rather than used to switch the ranking surface between architectures.Source: U Cross-Architecture Protocol Details · pp. 30–31
  • The closed-form keep rate r⋆ represents the within-task fraction of pool covariance signal, retaining samples according to measured within- and across-dataset gradient cosines.The rule treats keeping more as admitting cross-task interference and keeping less as discarding within-task signal.Source: U Cross-Architecture Protocol Details · p. 30
  • When datasets are nearly orthogonal, r⋆→1; when they are aligned, r⋆→1/T, with intermediate pruning increasing as cosintra/cosinter shrinks.The keep rate is a functional of pool gradient covariance rather than a tuned hyperparameter.Source: U Cross-Architecture Protocol Details · pp. 30–31
  • 0.6589 versus 0.6571 for LoRA (+0.18 points) is the reported three-task average under the stated configuration.The average is reported for ARC-C, HellaSwag, and Winogrande.Source: U Cross-Architecture Protocol Details · pp. 30–31

V Extended Related Work

  • GRADE takes a different position from parameterization and optimization-geometry methods: it withholds updates when the active subspace is saturated, while allowing alternative LoRA parameterizations.The discussion names DoRA, LoRA+, and PiSSA as possible parameterizations for the LoRA block inside GRADE.Source: V Extended Related Work · p. 31
  • Routed-adapter methods select experts at inference, whereas GRADE gates training-time updates; the approaches can coexist in one stack.Source: V Extended Related Work · pp. 31–32
  • GRADE’s forward-probe score is presented as a practical compatibility proxy, not an exact influence estimator or uniquely correct data-importance measure.The discussion identifies coverage- or influence-based auxiliaries as a possible extension.Source: V Extended Related Work · p. 32
  • GRADE uses forward probes for compatibility and routing decisions but retains standard backward LoRA optimization for admitted updates.Its use of probes is narrower than replacing the main optimizer with zeroth-order updates throughout training.Source: V Extended Related Work · p. 32
Loading 2610.07553v1…