PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning

Yuyang Zhang, Xiaoyin Chen, Chunlin Ren, Qihuang Zhang

arXiv:2610.04670v1cs.AI

TL;DR

Replay-free fine-tuning constraints use one shared input subspace across update rows, even though many ReLU token–unit pairs are closed. This paper derives a gate-aware per-unit projector from an exact decomposition of output change. It reports less forgetting than Adam-NSCL across matched OPT-1.3b budgets, with additional results on OPT-6.7b, token selection, and pruning repair.

  • Problem

    Replay-free methods such as Adam-NSCL and GPM constrain one shared input subspace in every update row, although closed ReLU token–unit pairs contribute zero until their pre-activation crosses the threshold.

    Source: §1 · p. 1

  • Method

    The paper derives a gate-aware per-unit projector from an exact decomposition of ReLU output change and a budget-separation analysis of open-token constraints.

    Source: §1 · p. 2

  • Results

    On OPT-1.3b, the per-unit projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 directions per row; on OPT-6.7b, it matches Adam-NSCL’s forgetting while learning more.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The exact identity links the ReLU geometry to per-unit constraints in fine-tuning and identifies the block left unpriced by curvature objectives in pruning repair.

    Source: §7 · p. 10

Abstract

from arXiv · show

Fine-tuning a language model on new text degrades what it already does. Replay-free projectors such as Adam-NSCL and GPM forbid one shared subspace of a layer's inputs in every row of the update. The tropical geometry of a ReLU layer shows why this is too coarse. In data space, the units' walls are tropical hypersurfaces whose cells are dual to the upper vertices of a zonotope; in weight space, each old token is a hyperplane, and the tokens cut out a polyhedron, the closure of the weights that keep every token on its side. An exact identity joins the two pictures: the squared change of the layer's output under any weight change splits into in-cell, open-to-closed and closed-to-open terms, and the first two live on the tokens each unit fires on. The identity names a gate-aware per-unit projector, and a budget-separation theorem prices exact protection: it costs a unit the rank of its own open tokens, while a shared subspace pays at least the rank of their union in every row. On OPT-1.3b, where 96% of (token, unit) pairs are closed, the projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 constrained directions per row, the gap widening from $1.1\times$ to $4.3\times$; with 1/5.5 of the directions it halves the forgetting of Adam-NSCL at GPM's energy threshold. On OPT-6.7b, it matches Adam-NSCL's forgetting at matched budget while learning more. As the theory predicts, the open/closed partition is the operative variable: open tokens beat random, sign-blind and anti-gate token sets on 18 of 18 seed-pairs. In pruning repair, every derivative-based local model of the output error at the dense weights is blind to pairs that open: the minimisers of the gate-weighted objective can leave the polyhedron, the objective's closed-form solution is 1.94 nats worse than no repair on OPT-1.3b, and a convex one-sided penalty bounds the escape.

1 INTRODUCTION

The introduction argues that shared replay-free constraints are too coarse for ReLU layers because closed token–unit pairs contribute no output until their gates open. It presents a geometry-derived per-unit projector and reports lower forgetting than Adam-NSCL across matched OPT-1.3b budgets, alongside results on larger models and pruning repair.

1 INTRODUCTION

  • Shared replay-free methods constrain one input subspace in every update row, although closed ReLU token–unit pairs contribute zero until their pre-activation crosses the threshold.In OPT-1.3b, 96% of token–unit pairs are closed.Source: §1 · p. 1
  • The paper derives a gate-aware per-unit projector from an exact output-change identity and prices exact protection by each unit’s open-token rank rather than the rank of their union in every row.The identity separates in-cell, open-to-closed, and closed-to-open terms; the first two involve tokens each unit fires on.Source: §1 · p. 2
  • On OPT-1.3b, the per-unit projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 directions per row, with the gap widening from 1.1× to 4.3×.With 1/5.5 of the directions, it halves Adam-NSCL’s forgetting at GPM’s energy threshold; on OPT-6.7b it matches Adam-NSCL’s forgetting while learning more.Source: §1 · p. 2
  • Open tokens outperform random, sign-blind, and anti-gate token sets on 18 of 18 seed-pairs, supporting the open/closed partition as the operative selection criterion.This comparison holds at equal per-unit dimension.Source: §1 · p. 2
  • In pruning repair, derivative-based local models at dense weights miss the closed-to-open term, while a convex one-sided penalty bounds escape from the activation-preserving region.The gate-weighted objective’s closed-form solution is 1.94 nats worse than no repair on OPT-1.3b.Source: §1 · p. 2

2 TWO DUAL PICTURES OF A RELU LAYER

The section connects ReLU activation patterns in data space to constraints on weights in weight space. An exact decomposition of output change then distinguishes within-cell changes from gate crossings, informing which movements to constrain.

2 TWO DUAL PICTURES OF A RELU LAYER

  • In weight space, each old token defines a hyperplane, and the activation-preserving polyhedron is the closure of weights that keep every token on its original side.Crossing a token’s hyperplane changes its gate; the polyhedron’s facets correspond to token hyperplanes.Source: §2 · pp. 2, 4
  • The figure links the dual views: moving a unit’s wall across a token in data space is the same gate-change event as its weight point crossing that token’s hyperplane.Its toy example also shows that some bit patterns can be unrealized.Source: Figure 1 · p. 3, §2 · p. 4
  • In data space, ReLU walls divide inputs into gate-pattern cells dual to upper vertices of a zonotope; the tropical form maps realized gate patterns to exposed zonotope vertices.A gate pattern that does not occur in the input space need not correspond to an exposed vertex.Source: §2 · pp. 3–4, Figure 1 · p. 3
  • The exact decomposition partitions tokens into four old/new gate cases and expresses squared output change as in-cell, open-to-closed, and closed-to-open contributions, with no remainder.The pattern-changing terms are exact sums over the corresponding token sets.Source: §2 · p. 4
  • The pairwise case split matches measured output changes to relative gaps of at most 5.8 × 10^-5 on 27 pruned OPT cells and 9.0 × 10^-5 on 18 cells across two 7B ReLU-family models.The measurements are reported in Appendix J.Source: §2 · p. 4
  • Within the activation-preserving region, output change is only the in-cell term; each separating token adds its corresponding crossing term.The open-to-closed and closed-to-open sets consist of tokens whose hyperplanes separate the old and new weights.Source: §2 · p. 4

3 THE CONSTRAINT THE IDENTITY NAMES, AND WHAT IT PREDICTS

The section uses the exact output-change decomposition to design per-unit constraints around each unit’s open tokens, while leaving closed tokens free unless they open. It derives budget and selection predictions from that partition.

3 THE CONSTRAINT THE IDENTITY NAMES, AND WHAT IT PREDICTS

  • The projector selects up to C of each unit’s open tokens and projects its cumulative row displacement; at zero ridge, selected pairs remain unchanged and closed pairs contribute only if they open.When C covers all open tokens, the in-cell and open-to-closed terms vanish at the base inputs; positive ridge shrinks rather than removes directions.Source: §3 · p. 5
  • Exact protection costs unit i the rank of its own open-token inputs, while a shared constraint costs at least the rank of the union in every row.With B total directions, per-unit subspaces can protect units whose open-token ranks sum to at most B; freezing whole rows spends d + 1 directions per unit.Source: §3 · pp. 5–6
  • In OPT-1.3b, the median unit fires on 11 of 2,048 old tokens, the mean on 72.1, and 19.5% of units never fire.The section uses this sparsity to motivate the potential budget advantage of per-unit constraints over a shared subspace.Source: §3 · p. 5
  • A closed token can open only if its update crosses its threshold margin, so only near-wall closed tokens can contribute to closed-to-open change.The theory predicts constrained updates free less closed-to-open mass than unconstrained ones, but does not order the two constrained methods.Source: §3 · pp. 5–6
  • For differentiable downstream losses, row gradients lie in the row space of a unit’s open-token inputs, and the row Fisher annihilates their null space.The stated first-order criterion therefore reads open pairs, while diagonal EWC Fisher terms are indexed by open pairs.Source: §3 · pp. 5–6
  • The shared null-space constraint is exact for ungated matrices; for a ReLU-gated GLU, constraining gate and up rows preserves every open pair’s output.The comparison of methods is scoped to threshold-activated matrices such as OPT fc1.Source: §3 · p. 6

4 EXPERIMENTS

The experiments compare the per-unit projector with shared-subspace and other replay-free methods across matched budgets, token-selection criteria, model scales, and downstream accuracy. The reported results show where its forgetting and learning outcomes differ from those baselines.

4 EXPERIMENTS

  • The setup uses pretrained OPT models, MiniPile to represent retained knowledge, and WikiText-103 or Python as the new task; forgetting is measured by old-probe NLL rise and learning by new-task held-out NLL change.Projectors are built from 2,048 support tokens and evaluated on a disjoint 1,024-token probe.Source: §4 · p. 6

4.1 MATCHED BUDGET, CAPACITY AND SCALE (P1, P4)

  • On OPT-1.3b, the per-unit projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 dims/row, with the gap growing from 1.1× to 4.3×.At 17 dims/row and above on OPT-125m, it forgets less on 14 of 18 seed-pairs and learns more by 0.023 nats at 17 dims/row and 0.008 at 42.Source: §4.1 · pp. 6–7, Figure 2 · p. 7
  • At 47.3 dims/row, cap 256 forgets 2.3× less than Adam-NSCL at 259 dims/row and learns 0.052 nats more.Cap 256 removes 95.9% of unconstrained forgetting on OPT-1.3b and 97.7% on OPT-125m.Source: §4.1 · p. 7
  • On OPT-6.7b, cap 128 matches Adam-NSCL's forgetting at 68.7 versus 69 dims/row and learns 0.011 nats more.Both methods remove 85% of unconstrained forgetting.Source: §4.1 · p. 7

4.2 WHAT THE CRITERION READS (P2, P5)

  • At equal per-unit dimensions, open tokens beat random, sign-blind, and most-closed selectors on all 18 seed-pairs; 99.6% of gradient-selected pairs are open.Random tokens within each unit's open set protect as well as its most active tokens, despite covering 35.2% versus 59.6% of active energy.Source: §4.2 · p. 7

4.3 AGAINST THE PUBLISHED METHOD FAMILIES

  • Across replay-free method families on OPT-1.3b, the per-unit projector compares favorably with several alternatives, while higher-learning configurations of LoRA incur 12–16× more forgetting.LoRA-Null still forgets 3.6–5.1× more than cap 128; Adam-NSCL at the GPM energy threshold reaches similar forgetting with 1/7.5 of the dimensions for cap 128.Source: Figure 3 · p. 8, §4.3 · p. 8

4.2 WHAT THE CRITERION READS (P2, P5)

  • The comparison reports 0.025 and 0.039 nats less learned; freezing enough rows to hold the same 59.6% of active energy costs 69 dims/row and still forgets 0.172.Source: §4.2 · p. 8

4.3 AGAINST THE PUBLISHED METHOD FAMILIES

  • On OPT-1.3b, cap 128 costs 1.02 points of mean zero-shot accuracy, versus 2.94 points for unconstrained WikiText-103 fine-tuning.At matched budget, the two projectors agree within 0.1 points.Source: §4.3 · p. 8

5 SECOND APPLICATION: REPAIR AFTER PRUNING

The pruning-repair analysis identifies a blind spot in local output-error objectives and introduces a one-sided penalty to constrain escape from activation-preserving weights.

5 SECOND APPLICATION: REPAIR AFTER PRUNING

  • Every derivative-based local model of post-repair output error at dense weights matches the gate-weighted objective nearby and is blind to tokens that change activation state.The result assumes no calibration token lies on the activation wall.Source: §5 · p. 9
  • When the active-token design fails to constrain inactive-token directions, gate-weighted minimisers can have unbounded escape while the objective remains constant.For sufficiently many i.i.d. calibration tokens under the stated moment and rank conditions, the minimiser is almost surely unique and its per-token escape converges to a positive value exactly when some tokens open.Source: §5 · p. 9
  • For λ > 0, adding the convex one-sided escape penalty yields a minimiser and bounds escape by ∥ζ∥2/λ + E0 for solutions no worse than the masked starting point.As λ tends to infinity, escape tends to zero if and only if the feasible masked set is nonempty.Source: §5 · p. 9
  • The closed-form gate-weighted objective is 1.94 nats worse than no repair on OPT-1.3b at sparsity 0.7; pricing both crossing directions orders the objectives on 46 of 47 probe cells.The author reports the same ordering on all 15 WikiText-2 cells and 20 of 20 cells on two 7B ReLU-family models.Source: §5 · p. 9

6 RELATED WORK

The related-work discussion connects the paper to projection-based continual learning, other continual-learning methods, layer repair, and tropical geometry.

Projection-based continual learning

  • Projection-based continual-learning methods constrain updates using shared subspaces, while this paper changes which input-unit pairs are constrained.The discussion includes GPM, Adam-NSCL, and AdNS, as well as methods that relax the subspace or use its complement for adapters.Source: §6 · p. 9

Other continual-learning methods

  • Other approaches include orthogonal task adapters, per-unit movement decisions, importance-weighted constraints, and replay.Examples named include O-LoRA, MIGU, HAT, AGS-CL, EWC, LDIFS, and replay.Source: §6 · p. 9

Activation patterns and repair

  • Prior pruning and quantisation repair fits pre-activations or curvature-weighted outputs; the paper relates its local curvature result to Gauss–Newton blocks and its penalty to inactive-side activation-boundary loss.The discussion also connects activation-preserving inequalities to provable repair methods.Source: §6 · p. 9

Tropical geometry of ReLU networks

  • The paper uses the data-space subdivision and weight-space dual of tropical geometry for ReLU networks.The cited background describes ReLU networks as tropical rational maps and prior bounds on their linear regions.Source: §6 · p. 10

7 CONCLUSION

The conclusion frames an exact identity as the link between two views of a ReLU layer and its applications to fine-tuning and pruning repair.

7 CONCLUSION

  • An exact identity joins a ReLU layer's dual pictures and supplies the basis for a per-unit constraint in fine-tuning and a curvature-blind block in pruning repair.The conclusion identifies these as the identity's two applications.Source: §7 · p. 10

A PRELIMINARIES: A RELU UNIT AS A TROPICAL BINOMIAL

  • The summed output of a ReLU layer is represented in max-plus notation as the tropical product of its units' binomials.Multiplying the product gives one monomial for each subset of units.Source: A PRELIMINARIES: A RELU UNIT AS A… · p. 14
  • The lifted-point convex hull is a zonotope whose projected upper faces form a subdivision with vertices corresponding to realised gate patterns and cells dual to wall intersections.For real weights the construction is a tropical signomial, and the stated duality remains that of regular subdivisions.Source: A PRELIMINARIES: A RELU UNIT AS A… · p. 14

B PROOFS FOR SECTIONS 2 AND 3

These proofs connect ReLU gate patterns to zonotope and weight-space geometry, characterizing the regions and boundaries that preserve unit-specific token gates. They formalize how gate-aware protection, post-update loss, and first-order update statistics depend on each unit’s open-token subspace, including when closed tokens can open and how the protection condition extends to gated units.

Proposition 2: the tropical form and the dual subdivision

  • A ReLU layer’s summed output is a maximum over subset sums, and off-wall gate patterns map injectively to exposed zonotope vertices.The map is onto those vertices when no unit is constant at the threshold.Source: Figure 4 · p. 16, B PROOFS FOR SECTIONS 2 AND 3 · p. 17
  • When two off-wall inputs differ by one firing unit, the corresponding zonotope vertices are joined by an edge directed by that unit’s weights; the wall is dual to the edge.The edge direction is (w_i, b_i − θ), whose first d coordinates give the wall’s normal.Source: Figure 4 · p. 16, B PROOFS FOR SECTIONS 2 AND 3 · pp. 17–19

Proposition 4: the data–weight duality

  • The weights preserving a unit’s base token gates form the closure of a convex chamber in the token-hyperplane arrangement.Its interior keeps open tokens strictly above threshold and closed tokens strictly below it; the closure permits closed tokens at threshold.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 19
  • Every boundary point of the gate-preserving polyhedron lies on a facet, and each facet is the intersection with a token hyperplane.A facet’s token label is active when that token was open at the base weights and inactive otherwise.Source: B PROOFS FOR SECTIONS 2 AND 3 · pp. 17, 19, 20

Theorem 5

  • Theorem 5’s figure illustrates the exact output-change decomposition: in-cell motion, closing, and opening contribute separately, while a token closed at both weights contributes zero.For the four illustrated tokens, the three nonzero terms sum to 1.82 with no remainder.Source: Figure 5 · p. 20

Full form of Corollary 6

  • The post-update loss is at most quadratic on any fixed-gate region, equals the in-cell quadratic on KX, and at θ = 0 has that same form throughout PX.Tokens that change sides between the base and new weights are precisely those whose hyperplanes separate the two weights.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 20

Proposition 8: exactness and the ridge residual

  • With C ≥ |Oi| and ρ = 0, the gate projector confines a row to a flat parallel to active hyperplanes, and every path from KX to outside it meets the polyhedron boundary at inactive facets.Pi(0) is the orthogonal projector onto ker Ai; at base inputs, selected open tokens stay open, while selected closed tokens preserve their pre-activation.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 21
  • For any selected token set, the update preserves those tokens’ base pre-activations, and a closed token with sufficient margin contributes zero loss whether selected or not.The sufficient margin is θ − zi(xt) ≥ ||u′|| ||x̃t||.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 21

Theorem 9: budget separation

  • A per-unit protecting subspace must lie in that unit’s open-token kernel, while a shared protecting subspace pays at least the rank of the union of open-token inputs in every row.With enough budget, the per-unit projector removes exactly rank(˜X_Oi) directions; under a total codimension budget B, it can protect the largest set of units whose ranks sum to at most B.Source: B PROOFS FOR SECTIONS 2 AND 3 · pp. 22–23

Proposition 10: the budget-limited optimum

  • For fixed codimension q, the expected unit-level quadratic error is minimized by excluding that unit’s top-q open-token Gram eigenspace, and per-unit allocation is no worse than one shared subspace.The per-unit minimum is σ^2 sum_{j>q} λj(Ξi), attained when U⊥ is a top-q eigenspace; the gate projector reaches zero error when its selected tokens span the open-token row space.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 24

Per-unit allocation and protection

  • When selected tokens span a unit’s open-token inputs, nulling those inputs gives exact protection at the minimum per-unit distortion.The resulting subspace has codimension equal to the open-input rank; equal codimension q per unit is admissible and uses the same total codimension as a shared subspace of codimension q per row.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 25

Proposition 11: the freed term is a near-wall term

  • A closed token opens only when its update exceeds its margin, so only near-wall closed tokens can open.For margin γ_t and update a_t, opening requires a_t > γ_t; the closed-to-open contribution is (θ + a_t − γ_t)^2.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 25

Proposition 12: first-order criteria inherit the partition

  • The Fisher matrix for unit i annihilates the nullspace of its open-token inputs, and its gradient and per-token terms depend only on open pairs.A unit open on no token has zero gradient and Fisher matrix.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 25

Lemma 21 (Gated units)

  • For a gated unit, updates that vanish on its open inputs preserve outputs on open tokens; a closed token’s output can change only if its gate opens.The condition applies to both gate and up-weight updates.Source: B PROOFS FOR SECTIONS 2 AND 3 · p. 26

C FINE-TUNING SETUP AND IMPLEMENTATION

The experiments fine-tune OPT models on new text while building projectors from old-support inputs and scoring forgetting on a disjoint old probe. All constrained methods reduce closed→open mass on held-out text, and paired tests quantify method differences.

C FINE-TUNING SETUP AND IMPLEMENTATION

  • Each run updates OPT fc1 weights and bias; 2,048 MiniPile tokens build the projectors, while disjoint old-probe and new-task text provide evaluation data.Runs use 512 fresh 128-token windows, four per AdamW step, for 128 steps; measurements are taken at step 128.Source: C FINE-TUNING SETUP AND IMPLEMENTATION · p. 26
  • Forgetting is old-probe NLL minus its base value, while learning is held-out new-task NLL minus its base value; more negative values indicate greater improvement.Methods share the same seed-specific data order, and pairwise comparisons use per-seed differences.Source: C FINE-TUNING SETUP AND IMPLEMENTATION · p. 26, §C.2 · p. 28

C.1 THE FREED TERM ON HELD-OUT TEXT

  • On the held-out old probe, every constrained method reduces closed→open mass to 16–25% of unconstrained fine-tuning’s mean level across all three seeds.The share of post-activation change attributable to this mass also falls.Source: §C.1 · p. 27

C.2 STATISTICS

  • Paired t-tests compare methods, using one-sided tests for predicted directions and two-sided tests otherwise; equivalence uses a 0.0068-nat margin.The margin corresponds to the probe noise floor described for repair experiments.Source: §C.2 · p. 28

D MATCHED BUDGET AND CAPACITY IN FULL

At matched constrained dimensions per row, the per-unit projector reduces forgetting relative to Adam-NSCL on OPT-1.3b, while the comparison is equivalent on OPT-6.7b and learning is greater. The section also reports capacity comparisons and the open-token limit on reachable budget.

D MATCHED BUDGET AND CAPACITY IN FULL

  • On OPT-1.3b, the forgetting ratio rises from 1.1 at 9 dimensions per row to 4.3 at 60, while ours drops from 0.047 to 0.011 nats between 15 and 47 dimensions.On OPT-6.7b, at cap 128 (68.7 dimensions per row) versus Adam-NSCL at 69, forgetting was equivalent (TOST at δ = 0.0068; 90% CI [−0.0001, +0.0018]) and learning was 0.0109 nats greater on all three seeds.Source: D MATCHED BUDGET AND CAPACITY IN FULL · p. 28
  • Pooled OPT-1.3b forgetting is lower on 24 of 27 seed-pairs, with a random-effects difference of −0.0182 and 95% interval [−0.0279, −0.0086].For OPT-125m, it is lower on 14 of 18 seed-pairs, with pooled difference −0.0049 and interval [−0.0102, +0.0005].Source: D MATCHED BUDGET AND CAPACITY IN FULL · p. 28

D.1 CAPACITY

  • At cap 256 on OPT-1.3b, the projector forgets 0.0110 versus Adam-NSCL’s 0.0255 on WikiText-103, while learning 0.0518 nats more.The cap uses 47.3 mean dimensions per row, compared with Adam-NSCL’s uniform 259.Source: §D.1 · p. 29

D.2 THE REACHABLE BUDGET

  • The mean per-unit dimension is bounded by mean open-set size: at 2,048 support tokens, the reported reachable budgets are 72.1 on OPT-1.3b and 111.3 on OPT-125m.The cap binds for a heavy-tailed distribution of unit activity.Source: §D.2 · p. 29

E WHAT THE CRITERION IS: THE SELECTOR CONTROLS IN FULL

Selector controls hold each unit’s budget fixed while changing which tokens receive protection. Open-token selection outperforms off-partition selectors, while freezing whole rows performs worse than the per-unit projector at equal and larger budgets.

E WHAT THE CRITERION IS: THE SELECTOR CONTROLS IN FULL

  • The controls vary selected tokens while keeping each unit’s budget fixed: gate chooses its largest activations, openrand samples open tokens, and other controls include gradient, sign-blind, closed-only, and random selection.Figure 6 depicts the token-unit pairs selected by the constraint families and the selector controls.Source: E WHAT THE CRITERION IS: THE SELECTOR… · p. 29, Figure 6 · p. 30
  • With identical per-unit dimensions, the gate selector forgets less than each of three off-partition selectors on all 18 seed-pairs; openrand and grad are equivalent to the gate on-partition.The open share separates the selectors on and off the partition, while preserved energy does not order them.Source: E WHAT THE CRITERION IS: THE SELECTOR… · pp. 29, 31

E.1 FREEZING WHOLE UNITS

  • At equal budget, freezing whole units forgets about eight times as much as the per-unit projector and learns 0.036–0.039 nats less; at three times the budget, it still forgets five times as much.A frozen row uses all its directions, unlike a per-unit allocation.Source: §E.1 · p. 31

F COMPARISON WITH OTHER METHOD FAMILIES

The section compares the projector with alternative method families and reports zero-shot and repair analyses. Results vary across competitors, while the repair analysis distinguishes curvature-based local models from objectives that penalize closed tokens opening.

F.1 EWC, MIGU, LORA AND LORA-NULL

  • Against EWC, MIGU, LoRA, LoRA-Null, and Adam-NSCL, outcomes depend on the configuration: EWC reaches the cap-128 frontier at λ = 4 × 10^5, while several LoRA and MIGU settings trade greater forgetting for learning.L2-to-init at λ = 0.3 is equivalent in forgetting to cap 256 but learns 0.049 nats less; Adam-NSCL at GPM energy 0.95 forgets about as much as cap 128 and learns 0.014 nats more.Source: §F.1 · p. 33

F.2 WISE-FT AND ZERO-SHOT ACCURACY

  • On five-task zero-shot evaluation, OPT-1.3b fine-tuned on WikiText-103 with cap 128 achieved 1.92 points higher mean accuracy than no constraint, positive on all three seeds (paired one-sided p = 0.0125).Source: Table 9 · p. 33
  • At zero forgetting after WiSE-FT composition, cap 64 learns 0.0074 nats more than matched Adam-NSCL and cap 128 learns 0.0084 more, though neither difference is significant.The interpolation parameter is swept on evaluation sets, so the curves are trade-offs rather than deployable choices.Source: §F.2 · p. 34

G.1 NOTATION AND CURVATURE BLINDNESS

  • Pre-activation, Gauss–Newton, and Fisher repair objectives charge only open-token in-cell error locally, whereas the one-sided objective adds a penalty for closed tokens that open.The exact post-activation error stops growing when an open token closes and charges only opening for a token initially closed.Source: §G.1 · p. 35
  • In the toy pruned row, curvature fitting moves to 4.13 and opens token E, producing exact error 52.8, 22 times the no-repair error; the one-sided fit at λ = 10 stays near no-repair error.The gate constraints for A and C conflict with E, so no refit on the displayed line keeps every gate.Source: §G.1 · p. 35
  • At the dense weights, post-activation error and its gate-weighted counterpart have identical derivatives of every order, so local curvature models do not capture closed tokens opening.The stated Hessian is twice the sum of outer products for tokens open at the dense weights.Source: §G.1 · pp. 35–36

G.2 PRICING THE FACES BACK

The section analyzes a one-sided penalty that prices escape across inactive constraints and bounds that escape under stated feasibility conditions. It also reports how repair objectives differ in activation-crossing patterns on OPT-1.3b.

G.2 PRICING THE FACES BACK

  • The objective Lgated h adds a convex, continuously differentiable penalty for squared violations of inactive constraints, zero exactly on Q_M.Q_M contains the masked weights that satisfy the activation-preserving polyhedron’s inactive constraints.Source: §G.2 · p. 36
  • Every point no worse than the starting mask has escape at most ||ζ||^2/λ + E0; if Q_M is nonempty, escape tends to zero as λ grows, and only then.When Q_M is empty, the minimum escape is positive, so increasing λ cannot drive escape to zero.Source: §G.2 · p. 36
  • As λ increases, the minimisers’ escape is non-increasing while their gated loss is non-decreasing.Both quantities take single values across the minimiser set.Source: §G.2 · p. 36
  • For θ = 0, the escape penalty is exactly the closed→open term of Eq. (2), evaluated at the mask-only weights when δ = 0.For θ > 0, the implemented penalty remains an upper bound on escape, but need not be convex or continuous at the threshold.Source: §G.2 · pp. 36, 38

G.3 PREDICTED ACTIVATION-PATTERN SIGNATURES

  • Against the dense pattern on OPT-1.3b at s = 0.7, closed→open crossings account for 0.372 (37.2%) of Lgated’s residual, versus 0.171 (17.1%) for Lgated h(1).Lpre’s crossings run the other way, open→closed by 0.52% to 0.34% of pairs.Source: §G.3 · p. 38

H PROOF OF THEOREM 14

The theorem characterizes the gate-weighted fitting objective using active calibration tokens, then examines what its minimisers imply for inactive-token escape. Under sampling and rank assumptions, the fitted point converges while escape depends on the distribution’s closed-cell restriction too.

H PROOF OF THEOREM 14

  • Lgated depends on calibration data only through active rows A and residual ζ, and its minimisers form A⁺ζ + ker A.Changing inactive rows or inactive pre-activations leaves the objective and its minimiser set unchanged.Source: H PROOF OF THEOREM 14 · pp. 38–39
  • When ker A is not contained in ker ˜A, every minimiser lies on a ray of equally good fits whose escape grows without bound.This condition holds when the kept-input matrix has full column rank and there are fewer active tokens than kept columns.Source: H PROOF OF THEOREM 14 · p. 39
  • For i.i.d. calibration tokens with finite second moments, a fixed mask, and a nonsingular active second moment, the unique fitted point converges almost surely and average escape converges to e⋆.The limiting fit depends on the distribution only through its open-cell restriction; e⋆ also depends on its closed-cell restriction and is positive iff Pr[zw(x) ≤ θ < zw⋆(x)] > 0.Source: H PROOF OF THEOREM 14 · pp. 39–40
  • At OPT-1.3b with s = 0.7, 39.2% of rows have fewer active tokens than kept columns, so both the escape-ray and sampling regimes occur.The closed-form analysis uses 57,344 calibration tokens; a row with gate density 0.035 has about 2,007 active tokens and 614 kept columns.Source: H PROOF OF THEOREM 14 · p. 40

I EXPERIMENTAL SETUP AND NOISE FLOORS

The setup specifies the models, pruning masks, calibration data, and evaluation protocols used for repair experiments. It also defines noise floors and verifies the identity underlying the analysis on pruned layers.

I EXPERIMENTAL SETUP AND NOISE FLOORS

  • The setup spans ReLU, GELU, thresholded FATReLU, and SwiGLU models, with the wall matrix identified as the activation-output matrix modified during repair.The mask is Wanda-style, ranked within each output row; 2:4 masking keeps two of every four consecutive input channels.Source: Table 10 · p. 40, I EXPERIMENTAL SETUP AND NOISE FLOORS · p. 40
  • Calibration uses MiniPile, with an in-domain held-out probe from the same file and an out-of-domain WikiText-103 probe; WikiText-2 is evaluated at context 2048.The 256-token probe floor is 0.0068 nats, while the WikiText-2 floor is 0.0150 nats.Source: I EXPERIMENTAL SETUP AND NOISE FLOORS · p. 41

J VERIFICATION OF THE IDENTITY

  • The Eq. (2) identity is checked on every wall at every evaluation, and its implementation gap is zero on OPT.On pruned layers, the case-split residual also matches at floating-point level; Table 11 reports the relative-gap measure.Source: J VERIFICATION OF THE IDENTITY · p. 41, Table 11 · p. 41

K REPAIR EXPERIMENTS

The repair experiments compare objectives under a shared frozen mask and common evaluation conditions. They report a closed-form failure for the gate-weighted fit, alongside differences in post-activation loss and crossing patterns.

K REPAIR EXPERIMENTS

  • Lpost is below Lpre at every sparsity on OPT-1.3b, and by more than the 0.0150-nat WikiText-2 noise floor at three of four sparsities.The comparison uses WikiText-2 perplexity at context 2048 with one frozen mask per row.Source: Table 12 · p. 41, K REPAIR EXPERIMENTS · p. 41
  • +1.9416 nats against not repairing is the cost of the closed-form Lgated solution on OPT-1.3b at s = 0.7.It adds 237% of the error it was asked to remove; the result is attributed to objective degeneracy rather than the solver.Source: K REPAIR EXPERIMENTS · p. 42
  • Lpost reaches −0.1098 nats at relative weight movement 0.255, while prematch reaches +0.2894 and pre pval +0.2258 on OPT-1.3b at s = 0.7.All three results use the 3000-step budget.Source: K REPAIR EXPERIMENTS · p. 42
  • 37.2% of Lgated’s residual is closed→open, 2.2 times the next-largest share, while its open→closed mass is lowest at 2.7%.The comparison is across repair objectives on OPT-1.3b at s = 0.7.Source: K REPAIR EXPERIMENTS · p. 42, Table 13 · p. 42

L THE λ SWEEP AND THE Lpost < Lpre COUNTS

Across comparable configurations, λ > 1 often yields lower held-out probe loss than λ = 1, and Lpost < Lpre counts are high across several evaluation groups. Comparisons are read within runs because protocol differences can substantially spread Lpost − Lpre.

L THE λ SWEEP AND THE Lpost < Lpre COUNTS

  • Across comparable plain-FFN ReLU, GELU, and thresholded-ReLU SwiGLU configurations, λ > 1 yields lower held-out loss in 9 of 11 cases and beats the noise floor in 7 of 11 (Table 14).Five of nine rows are bold, with gaps 5 to 27 times the floor; the second seed reproduces the sign in 6 of 6 cases.Source: Table 14 · p. 42, L THE λ SWEEP AND THE Lpost < Lpre… · p. 42
  • Across runs measuring one configuration under different protocols, Lpost − Lpre spreads by up to 1.04 nats, so orderings are read within runs.Table 14 compares λ > 1 and λ = 1 within shared configuration, no-repair anchor, calibration seed, and budgets.Source: Table 14 · p. 42, L THE λ SWEEP AND THE Lpost < Lpre… · p. 43
  • Lpost < Lpre holds in 46 of 47 probe cells at 256-token evaluation, 23 of 24 under every protocol, 15 of 15 WikiText-2 cells at 2048 tokens, and 20 of 20 7B ReLU-family cells.The probe counts include an OPT-125m s=0.3 configuration at +0.0058.Source: L THE λ SWEEP AND THE Lpost < Lpre… · p. 43
  • Across the 7B cells, gaps are 0.0086–0.1594 nats, exceed the 0.0061-nat seed-noise floor, and remain negative across 10 independent configurations (sign test p = 9.8 × 10^-4).The cells cover ReluLLaMA-7B and ProSparse-7B at four unstructured sparsities and a 2:4 mask, with two calibration seeds.Source: L THE λ SWEEP AND THE Lpost < Lpre… · p. 43
Loading 2610.04670v1…