PaperlayerView exact revision on arXiv

Source-linked AI summary

AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning

Yue Xie, Zhi Zheng, Yunpeng Ba, Xuyang Wu, Xialiang Tong, Zhichao Lu, Tao Zhong, Zhenkun Wang

arXiv:2609.35257v1cs.LG

TL;DR

Billion-parameter LLM fine-tuning with zeroth-order optimization can waste forward evaluations on weakly informative random directions, while low-dimensional subspaces may miss useful update directions. AIM-ZO accumulates forward activations into a broad maintained subspace and uses smaller shared-and-sampled active subspaces for perturbations. Across five models and 11 tasks, it reports improvements over ZO baselines, including gains on OPT-2.7B and OPT-30B.

  • Problem

    In billion-parameter LLMs, random full-space ZO perturbations can provide weak optimization signals, while low-dimensional subspaces risk missing useful gradient information.

    Source: §1 · p. 1

  • Method

    AIM-ZO accumulates forward activation information across training in a width-K subspace and activates smaller shared-and-sampled direction sets for each perturbation.

    Source: §1 · p. 2

  • Results

    Across five models and 11 tasks, AIM-ZO improves over the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and over MeZO by 2.85 percentage points on OPT-30B.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    AIM-ZO's maintained subspace can cover more candidate directions over time without increasing the dimension of every perturbation estimate.

    Source: §1 · p. 2

    The analysis assumes post-burn-in tracking conditions and Gaussian perturbations, and leaves implemented rank-one updates and changing active subspaces only partially characterized.

    Source: §6 · p. 10

Abstract

from arXiv · show

Zeroth-order (ZO) optimization offers a memory-efficient alternative for LLM fine-tuning by estimating updates only from forward evaluations of perturbed parameters, without backpropagation or activation storage. However, in billion-parameter LLMs, isotropic perturbations often waste many forward evaluations on weakly informative directions. To make these evaluations more informative, existing ZO methods restrict perturbations to low-dimensional subspaces. Yet the quality of these subspaces is critical: overly compressed or poorly maintained spaces can miss useful update directions. To obtain a high-quality subspace for ZO updates, this paper proposes AIM-ZO, a ZO fine-tuning method based on Activation-Informed Subspace Maintenance. AIM-ZO uses forward activations as local directional information and continuously integrates them into a broad, evolving subspace over training. To access broader gradient-relevant structure while keeping individual perturbations low-dimensional, AIM-ZO activates only a smaller set of shared and sampled directions, decoupling the maintained width from the active width. We evaluate AIM-ZO across 5 LLMs and 11 downstream tasks under matched forward-evaluation budgets; its six-task average exceeds the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and MeZO by 2.85 percentage points on OPT-30B. Our code is available at https://github.com/EkkoXy/AIM-ZO

1 INTRODUCTION

AIM-ZO addresses the tension between broad gradient coverage and low-dimensional ZO estimation by maintaining activation-informed directions across training while activating smaller subsets for updates. The introduction reports stronger overall performance than existing ZO baselines across its evaluations.

Motivation

  • ZO fine-tuning avoids backpropagation and activation storage, but random full-space perturbations can yield weak optimization signals in billion-parameter models.MeZO estimates updates from forward loss evaluations at perturbed parameter points.Source: §1 · p. 1

Research question

  • Low-dimensional ZO methods can miss useful gradient information when their subspaces are random or overly compressed, while widening them makes finite-sample estimation harder.The paper poses obtaining a high-quality subspace for low-dimensional ZO fine-tuning as its central question.Source: §1 · p. 1

Limitations of prior methods

  • AGZO rebuilds its subspace from current activations at each step, adding computational cost and leaving it sensitive to stochastic variation.Using one subspace for perturbations also couples gradient coverage to estimation variance.Source: §1 · p. 2

Results

  • AIM-ZO improves over the strongest fully evaluated ZO baseline by 1.26 percentage points on OPT-2.7B and over MeZO by 2.85 percentage points on OPT-30B.The evaluation covers 5 LLMs and 11 downstream tasks.Source: §1 · p. 2

Proposed method

  • AIM-ZO accumulates forward activation information into a broad, evolving subspace and activates shared and sampled directions for each low-dimensional perturbation.Separating maintained and active widths lets perturbations access different parts of the broader space over time.Source: §1 · p. 2

2 RELATED WORK

Prior ZO fine-tuning work reduces estimation or computational burdens through coordinate selection, structured perturbations, or activation-informed subspaces. AIM-ZO instead accumulates activation information while keeping each perturbation smaller than the maintained space.

Zeroth-order fine-tuning

  • MeZO estimates updates from forward evaluations and regenerates random perturbations from seeds to avoid storing full perturbation vectors.ZO fine-tuning estimates parameter updates without backpropagation.Source: §2 · pp. 2–3

Curvature-Guided Scaling and Coordinate Selection

  • HiZOO, CurvZO, and Sparse MeZO use curvature proxies or parameter magnitudes to scale perturbations or select coordinates.HiZOO-L reduces the memory required for HiZOO's inverse-Hessian state.Source: §2 · p. 3

Structured Perturbations and Random Subspaces

  • LoZO, SubZero, and ZO-Muon impose low-rank or random-basis structure, but their random bases do not explicitly target gradient-relevant directions.Their approaches reuse, refresh, or orthogonalize structured perturbations in different ways.Source: §2 · p. 3

Activation-Informed Subspaces

  • AGZO extracts subspaces from current minibatch activations, whereas ZO-Act reuses an initial-activation subspace that cannot track changing representations.Both truncate the activation span; widening it can improve coverage but increases basis storage and, when fully activated, estimation dimension.Source: §2 · p. 3

AIM-ZO

  • AIM-ZO accumulates activation information across steps in a width-K space but activates only k < K directions per perturbation.Directions omitted from one perturbation remain available to others without increasing every estimate's dimension.Source: §2 · p. 3

3 METHODOLOGY: AIM-ZO

AIM-ZO maintains a broad activation-informed subspace and uses smaller active subspaces for forward-only ZO updates. Its procedure updates the maintained basis, samples active directions for perturbations, and aggregates their evaluations into parameter updates.

3.1 NOTATION AND SETUP

  • AIM-ZO defines a perturbation population whose members evaluate perturbed parameter points on the same minibatch to estimate an update.The setup indexes trainable matrices by layer and specifies their dimensions.Source: §3.1 · p. 3, §3 · p. 3

3.2 ACTIVATION-INFORMED SUBSPACE MAINTENANCE

  • An unperturbed centre pass collects layer activations, and Oja's rule updates an orthonormal basis to track prominent activation directions across iterations.One basis update serves all N perturbations, and the updated basis is stored for the next iteration.Source: §3.2 · p. 4

3.3 ACTIVE-SUBSPACE SELECTION AND PARAMETER UPDATES

  • Each perturbation uses h shared basis directions and k − h directions sampled from the remaining maintained space to form its width-k active subspace.Sampling is uniform without replacement and independent across population members and layers; the perturbation magnitude is normalized to match MeZO's layer-wise RMS Frobenius norm.Source: §3.3 · pp. 4–5
  • AIM-ZO evaluates a population of perturbations and aggregates the one-sided estimates into a parameter update, using one unperturbed and N perturbed forward passes per iteration.The update applies learning rate η.Source: §3.3 · p. 5

4 ANALYSIS OF AIM-ZO COMPONENTS

The analysis bounds how activation-informed subspaces track and capture shared gradient structure, then gives a condition under which partial activation improves expected directional alignment. It also finds that one-sided versus two-sided estimator alignment gain increases with gradient capture.

4.1 MAINTAINING GRADIENT-RELEVANT INFORMATION

  • Figure 2(a,b) reports smaller shared-gradient residuals in middle and late layers and activation-alignment residuals that decrease with K.The diagnostics use exact training-set mean gradients over 16 consecutive steps.Source: §4.1 · p. 5, Figure 2 · p. 6
  • Under stated initialization and post-burn-in conditions, Theorem 1 bounds tracking error; larger eigengap reduces lag and stochastic error, while drift and observation second moment increase them.Within the admissible range, increasing the Oja step size reduces lag but increases stochastic error.Source: §4.1 · pp. 5–6
  • 52.2%–95.1% gradient capture by the maintained subspace is reported on RTE.Figure 2(c) reports maintained-subspace capture for K = 128.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 DECOUPLING MAINTAINED AND ACTIVE WIDTHS

  • Full activation retains all maintained signal at variance factor pK + 1, while increasing K at fixed h and k lowers tail inclusion ρ.The comparison motivates separating maintained width from the number of directions activated per estimate.Source: §4.2 · pp. 6–7
  • When active capture A_K,k exceeds β_{pK}/β_{pk}, partial activation has higher expected cosine alignment than full activation.The condition applies when k < K.Source: §4.2 · p. 7

4.3 ONE-SIDED ESTIMATION IN A HIGH-CAPTURE SUBSPACE

  • Online training and offline ablations favor the population-centered one-sided RLOO estimator over two-sided estimation.The comparison is reported for the estimator used by AIM-ZO.Source: §4.3 · p. 7
  • At matched active width and evaluation budget, OP–TP alignment gain increases with gradient capture, and actual-loss results closely follow the linear control.OP16 and TP8 each use 16 perturbed loss evaluations at k = 64.Source: §4.3 · p. 7, Figure 3 · p. 7

5 EXPERIMENTS

Experiments compare AIM-ZO with zeroth-order baselines under matched forward-evaluation budgets, test its design choices, and measure peak memory. AIM-ZO leads on reported six-task averages, while memory overhead over MeZO is modest.

5 EXPERIMENTS

  • Across the reported experiments, models include OPT-2.7B, OPT-13B, OPT-30B, and Qwen3-8B, with OPT-2.7B covering all 11 tasks; main tables report six-task comparisons.Methods use BF16 and approximately 40,000 training forward evaluations; AIM-ZO uses 16 evaluations per update and 2,500 updates.Source: §5 · pp. 7–8, §5.1 · p. 8

5.1 COMPARISON WITH ZEROTH-ORDER METHODS

  • 70.94% on OPT-2.7B and 72.42% on OPT-13B are AIM-ZO’s highest six-task averages among methods with complete results; it also leads the subspace baselines at both scales.On OPT-2.7B, ZO-Muon’s 69.44% rises to 70.94%; on OPT-13B, its 68.08% rises to 72.42%.Source: Table 1 · p. 8, §5.1 · p. 8
  • On OPT-30B, AIM-ZO improves over MeZO on all six tasks and raises the average by 2.85 percentage points; on Qwen3-8B, it raises the average by 0.93 points.On Qwen3-8B, it improves on five of six tasks; SQuAD has the largest gain while SST-2 declines.Source: §5.1 · pp. 8–9, Table 2 · p. 9

5.2 ABLATION STUDIES

  • On OPT-2.7B RTE, updating the maintained subspace every step outperforms a fixed activation subspace or less frequent updates; per-perturbation resampling improves over a common active subspace across four tasks.The population-centered one-sided estimator also outperforms the two-sided alternative at both best and final OPT-2.7B RTE checkpoints.Source: §5.2 · p. 9

5.3 PEAK GPU MEMORY

  • AIM-ZO uses 0.242–0.254 GiB more peak memory than MeZO and less than AGZO and full-parameter SGD on measured Qwen3-0.6B-Base/DROP configurations.SGD runs out of memory at the largest tested tensor lengths or batch sizes, and AGZO at batch size 64; OOM marks failures, not memory values.Source: §5.3 · p. 9, Figure 4 · p. 9

6 CONCLUSION

AIM-ZO maintains an activation-informed subspace across training and selects smaller active subspaces for forward-only ZO updates. The paper reports strong average benchmark performance, while noting limits in the analysis assumptions and characterization of implemented updates.

6 CONCLUSION

  • Across five models and 11 tasks, AIM-ZO attains the best six-task average among fully evaluated ZO baselines on OPT-2.7B and improves over MeZO on all six OPT-30B tasks.The conclusion also reports smaller gains on Qwen3 and modest peak-memory overhead.Source: §6 · p. 10
  • The analysis assumes post-burn-in tracking conditions and Gaussian perturbations, leaving implemented rank-one updates and changing active subspaces only partially characterized.Adapting preset widths and tail sampling using loss feedback remains future work.Source: §6 · p. 10

KEY NOTATION

  • The notation treats each basis matrix as spanning the subspace of its columns; P_Q denotes its projector, and unindexed Q is the maintained basis.K, k, h, and s denote maintained, active, prefix, and sampled-tail widths, with k = h + s ≤ K.Source: KEY NOTATION · p. 14

A.1 CONDITIONS AND STATEMENT

  • The tracking result uses an initialized subspace and post-burn-in conditions including spectral separation and bounded activation observations.The conclusion characterizes its analysis as bounding gradient capture by the maintained subspace.Source: §6 · p. 10, §A.1 · p. 15

A.2 ONE-STEP EXPANSION

  • A QR–Oja step updates the basis from S, and its projector change is expressed as first-order terms plus a remainder.The cited expansion applies when η_q∥S∥_2 ≤ 1/8.Source: §A.2 · pp. 15–16

A.3 GLOBAL INEQUALITY AND LOCAL CONTRACTION; A.4 CONCENTRATION AND INVARIANT-REGION INDUCTION

  • The tracking proof combines a local contraction with concentration and an invariant-region induction to control projector error over the finite trajectory.The induction is established on a high-probability event.Source: §A.3 · p. 16, §A.4 · p. 17

A.6 PROOF OF MAINTAINED-SUBSPACE CAPTURE

  • The maintained-subspace capture argument relates tracking error to gradient capture over a window, while noting that a window-average assumption alone does not imply a pointwise bound.The target basis is defined by the conditional covariance, not a covariance averaged over future observations.Source: §A.6 · pp. 17–18

B.1 FIXED-SUBSPACE IDENTITY; B.2 PROOF OF THE ESTIMATION-ERROR DECOMPOSITION

  • The active-subspace analysis samples shared prefix directions and a tail without replacement, then decomposes estimation error into orthogonal components.The active basis has width k and includes h shared directions; the conditional mean of the estimator is GP_QS.Source: §B.1 · p. 19, §B.2 · p. 19

B.3 PROOF OF PROPOSITION 2

The proof bounds how much gradient energy and directional alignment the sampled active subspace captures. It also compares width choices under stated concentration conditions.

B.3 PROOF OF PROPOSITION 2

  • The capture identity weights shared-direction energy fully and non-shared energy by its inclusion probability ρ = (k − h)/(K − h).The selected columns are orthonormal.Source: §B.3 · p. 19
  • When shared directions capture at least 1 − ε of the energy, the capture bound is at least 1 − ε(K − k)/(K − h).This follows from the expression for A_K,k.Source: §B.3 · p. 19

B.4 SUPPLEMENTARY DIRECTIONAL-ALIGNMENT BOUNDS

  • The supplementary alignment bounds establish J_K,k ≥ L_K,k, and under Proposition 2’s sufficient condition the lower bound exceeds the exact full-activation value.At k = K, the fixed-subspace identity gives J_K,K = βp_K C_Q.Source: §B.3 · p. 19

B.5 SEPARATING MAINTAINED AND ACTIVE WIDTHS

  • For admissible integer widths, the inclusion probability decreases with K and increases with k.At k = K, every column participates and ρ = 1.Source: §B.5 · p. 20

B.6 ADDITIONAL COMPARISON UNDER SHARED-ENERGY CONCENTRATION

  • If shared energy at k0 is at least 1 − ε and βp_k1/βp_k0 < √(1 − ε), then expected cosine at k0 exceeds that at k1.This comparison uses a lower bound at k0 and an upper bound at k1.Source: §B.6 · p. 20

C EXPERIMENTAL SETUP AND EVALUATION PROTOCOL

The section specifies the benchmark and diagnostic protocols, then reports evidence on gradient capture and active-width trade-offs. The diagnostics find stronger capture from maintained subspaces in most measured layers, while increasing active width trades capture against single-direction alignment.

C EXPERIMENTAL SETUP AND EVALUATION PROTOCOL

  • The main comparisons span five LLMs and 11 tasks, with approximately 40,000 training forward evaluations per run and task-specific splits and evaluation schedules.The six-task main tables report accuracy for classification and F1 for extractive question answering.Source: C EXPERIMENTAL SETUP AND EVALUATION… · p. 21

D SUPPLEMENTARY EXPERIMENTS FOR SECTION 4

  • The supplementary diagnostics test shared gradient structure, activation alignment, Oja maintenance, and active-width behavior using held-out gradients and checkpoint-based protocols.The width scan fixes shared rank r = 8 and varies K; the active-width scan compares capture and single-direction alignment.Source: D SUPPLEMENTARY EXPERIMENTS FOR… · p. 22, §D.1 · p. 22, §D.2 · p. 24

D.1 GRADIENT STRUCTURE AND MAINTAINED-SUBSPACE CAPTURE

  • Oja improves mean independent-batch capture in five of six measured layers, while exact current-batch SVD has higher capture in all six.Middle and late layers show concentrated shared mean-gradient structure, whereas early layers require a larger rank.Source: §D.1 · pp. 23–24
  • Oja maintenance takes 0.124 s versus 0.229 s for current-batch PI basis construction, but total step times are 0.774 s versus 0.760 s, respectively.These are reported diagnostic timings.Source: §D.1 · p. 24

D.2 ACTIVE-WIDTH DIAGNOSTICS

  • As active width grows, capture increases while single-direction estimate alignment decreases; at width 64, sampled-tail capture is .8526 on Qwen and .8928 on OPT.Top-64 capture is .8624 on Qwen and .8975 on OPT.Source: §D.2 · p. 24

D.3 ONE-SIDED AND TWO-SIDED ESTIMATION

The section compares one-sided and two-sided estimators under matched perturbation-evaluation budgets, then reports ablations of active-subspace sampling and maintenance choices. One-sided estimates show lower MSE in the stated tests, while ablation outcomes favor independent resampling and per-step maintenance.

D.3 ONE-SIDED AND TWO-SIDED ESTIMATION

  • In FP32 fixed-objective Gaussian tests, one-sided population estimates have lower mean MSE in all 15 settings per model; at N = 16 and ϵ = 8 × 10−5, learned-space MSE reductions are 45.8% on Qwen and 48.1% on OPT.Each comparison matches the number of perturbed loss evaluations.Source: §D.3 · p. 25
  • In the matched normalized Gaussian control, one-sided estimation lowers MSE in learned, random, and full spaces; the reported learned-minus-random cosine-gap interactions are 1.001×10−3 on Qwen and 8.756×10−4 on OPT.The test matches perturbation evaluation counts between OP16 and TP8.Source: §D.3 · p. 25, Table 11 · p. 25
  • With finite-training-set gradients, one-sided mean cosine favors learned spaces at small tested scales, while at 10−3 the learned-space gain disappears on Qwen and shrinks on OPT.One-sided estimation has lower MSE in every tested space and scale.Source: §D.3 · p. 25, Table 12 · p. 25, Table 13 · p. 26
  • In the BF16 implementation-level control, learned-space directions have higher cosine alignment, and the mean OP–TP gain is larger in both models.The comparison uses rank-one BF16 perturbations and dense Gaussian perturbations.Source: §D.3 · p. 26, Table 14 · p. 26

E.1 SHARED VERSUS INDEPENDENTLY RESAMPLED ACTIVE SUBSPACES

  • Independent resampling scores higher on all four OPT-2.7B tasks; per-step Oja refresh has the highest mean RTE development accuracy among the tested refresh schedules.Both ablations use matched forward-evaluation budgets.Source: §E.1 · p. 27, Table 15 · p. 27, §E.2 · p. 27

E.3 MAINTAINED WIDTH AND FIXED-PREFIX SELECTION

  • The fixed-prefix width-64 and width-128 variants are compared with the five-seed AIM-ZO main-result reference on official OPT-2.7B RTE accuracy.The fixed-prefix variants use three seeds and separately initialized maintained spaces.Source: §E.3 · p. 27, Table 17 · p. 27

E.4 MAINTAINED WIDTH WITH SAMPLED TAILS

These ablations vary maintained width, shared width, population size, and perturbation-coefficient rank on OPT-2.7B RTE. The stated comparisons use matched forward-evaluation budgets, with the larger population reporting the higher mean accuracy.

E.4 MAINTAINED WIDTH WITH SAMPLED TAILS

  • The maintained-width scan fixes k = 64 and h = 48, varies K, and reports OPT-2.7B RTE official-validation accuracy.At K = 64 all 16 tail columns are selected; at larger widths they are sampled from the maintained tail.Source: §E.4 · p. 27, Table 18 · p. 28

E.5 SHARED WIDTH

  • The shared-width scan varies h at fixed K = 128 and k = 64, reporting three-seed OPT-2.7B RTE official-validation accuracy.The h = 48 and h = 64 rows reuse earlier controls.Source: §E.5 · p. 28, Table 19 · p. 28

E.6 POPULATION SIZE

  • The larger one-sided population has the higher mean OPT-2.7B RTE accuracy under a matched 40,000-forward-evaluation budget.The comparison is between N = 8 and N = 15, each run with three seeds.Source: §E.6 · p. 28, Table 20 · p. 28

E.7 PERTURBATION COEFFICIENT RANK

  • Rank-one, rank-four, and full Gaussian coefficients have similar mean OPT-2.7B RTE accuracies; the table also reports step time and peak CUDA memory.Each variant uses three seeds and 40,000 training forward evaluations.Source: §E.7 · p. 28, Table 21 · p. 28

F.1 ONLINE ESTIMATOR ABLATION

The appendix examines estimator and subspace quality, reports memory and runtime measurements, evaluates restoration caching, and adds fine-tuning evaluation details.

F.1 ONLINE ESTIMATOR ABLATION

  • OP15-RLOO’s mean development accuracy exceeded raw TP8 by 1.36 points at the best-accuracy checkpoint and 1.84 points after the final update.The comparison used five matched seeds and 40,000 versus 42,500 total loss evaluations, respectively.Source: §F.1 · p. 29, Table 22 · p. 29

F.2 SPACE QUALITY AT FIXED ACTIVE WIDTH

  • At fixed active width k = 64, OP’s alignment gain rose with capture in exact-linear and real-loss controls; all 336 real-loss mean-MSE comparisons favored OP.The exact-linear control’s checkpoint-level correlations were .970 for Qwen3-4B and .965 for OPT-2.7B.Source: §F.2 · pp. 29–30, Table 23 · p. 29, Table 24 · p. 30

G.1 PEAK-MEMORY SCALING ON DROP

  • AIM-ZO used .242–.254 GiB more memory than MeZO across measured Qwen3-0.6B DROP configurations, but less than AGZO and FO-SGD.The measurements used rank-one perturbations, K = 128, k = 64, and h = 48.Source: §G.1 · p. 31

G.2 COMPLETE RUNS AT A MATCHED EVALUATION BUDGET

  • Completed 40,000-evaluation runs are summarized by training-step hours and AIM-ZO/MeZO ratios, with comparisons interpreted within rows because tasks and conditions vary.Validation and checkpoint I/O are excluded.Source: Table 25 · p. 31, §G.2 · p. 31

G.3 LARGE-MODEL SHORT-RUN MEASUREMENTS

  • Large-model short runs report mean step time over steps 6–10 and peak allocated memory; time per evaluation divides mean step time by forward-evaluation count.Each method and model ran ten BF16 updates in an independent process with the first batch matched.Source: Table 26 · p. 31, §G.3 · p. 31

G.4 RESTORATION CACHING

  • Factor caching reduced mean step time by 20.6% for an additional .036 GiB, while full-noise caching yielded little further speedup; all variants produced identical updates.The comparison used ten matched-batch Qwen3-0.6B SQuAD updates.Source: §G.4 · p. 31, Table 27 · p. 32

H ADDITIONAL FINE-TUNING RESULTS

  • Additional-results notes specify seed counts and checkpoint and validation choices for selected OPT-2.7B, OPT-13B, and SQuAD comparisons.SQuAD results use an independent 1,000-example official-validation subset; the MeZO entry uses full-parameter generation CE.Source: H ADDITIONAL FINE-TUNING RESULTS · p. 32

H.1 COMPLETE OPT-2.7B TASK COVERAGE

AIM-ZO has the higher five-seed mean on most OPT-2.7B tasks, while the supplied Qwen3-0.6B SQuAD EM scores favor MeZO. The appendix also compares ZO methods’ subspace designs and their information sources.

H.1 COMPLETE OPT-2.7B TASK COVERAGE

  • AIM-ZO had the higher five-seed mean on 9 of 11 OPT-2.7B tasks, with 43 wins, 1 tie, and 11 losses across 55 paired comparisons.COPA was approximately tied, while CB favored MeZO under accuracy and macro-F1.Source: Table 28 · p. 32, §H.1 · p. 32
  • On OPT-30B SQuAD, MeZO and AIM-ZO scored 70.10±1.36 and 72.88 ± 0.88 EM, respectively, over five seeds.Source: §H.1 · p. 32

H.2 QWEN3-0.6B RESULTS

  • On Qwen3-0.6B SQuAD, MeZO scored 74.57±1.12 EM and AIM-ZO scored 72.87 ± 0.59 EM.Source: §H.2 · p. 33

H.3 ADDITIONAL QWEN3-8B METRICS

  • Qwen3-8B additional metrics use the same three seeds and development-selected checkpoints for MeZO and AIM-ZO; the AGZO main-table row uses BF16 and 39,999 training forwards.AGZO checkpoints were selected from five development milestones.Source: §H.3 · p. 33, Table 30 · p. 33

I ADDITIONAL RELATED WORK

  • ZO-Act uses a fixed activation-derived basis from an initial snapshot, whereas AIM-ZO maintains layer-wise activation bases as training proceeds.ZO-Act avoids repeated basis construction but does not refresh its basis as parameters change.Source: I ADDITIONAL RELATED WORK · p. 34
  • Sparse MeZO and static sparsity restrict perturbations in parameter coordinates; matrix subspaces can represent combinations of coordinates through basis vectors.Source: I ADDITIONAL RELATED WORK · p. 34
  • ASEBO estimates an adaptive space from covariance of full-dimensional evolution-strategy estimates; AIM-ZO instead samples smaller active subspaces within activation-maintained bases.Source: I ADDITIONAL RELATED WORK · p. 34
  • Block Oja supports incremental leading-eigenspace estimation, but classical streaming guarantees typically assume stationary covariance; AIM-ZO uses a maintained activation space for perturbations.Source: I ADDITIONAL RELATED WORK · p. 34

Related papers

Loading 2609.35257v1…