PaperlayerView exact revision on arXiv

Source-linked AI summary

FedAlphaEdit: Null-Space-Aligned Merging for Collaborative Knowledge Editing

Sota Sugawara, Yukihiko Okada

arXiv:2610.11033v1cs.CLcs.LG

TL;DR

Institutions need to combine private knowledge edits in a shared model, but AlphaEdit’s preservation guarantee covers a single editing party. FedAlphaEdit aligns client editing and server merging under a null-space principle, bringing editing and preservation close to centralized editing across two architecture families.

  • Problem

    Collaborative knowledge editing must combine institutions’ private edits without sharing raw requests, but AlphaEdit’s preservation guarantee is established only for a single editing party.

    Source: §1 · p. 1

  • Method

    FedAlphaEdit aligns local AlphaEdit updates and server merging under one null-space principle, aggregating projected covariance statistics to recover global editing exactly in a one-shot solve.

    Source: §1 · pp. 1–2

  • Results

    Naive merging loses 4.2 points in neighborhood specificity and 17.4 points in edit success; FedAlphaEdit restores specificity to centralized level and closes most edit-success gap.

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    Institutions unable to share raw edit requests can jointly maintain a shared model whose edits approximate centralized editing.

    Source: §1 · p. 1, §6 · p. 8

    The exactness result assumes one-shot solves and a common model state; under genuine sequential editing, recovery is approximate and exhibits measured drift.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multiple institutions may each hold their own private knowledge edits and wish to integrate them into a single large language model without sharing raw edit requests. Null-space-constrained editing methods such as AlphaEdit mathematically guarantee that each update leaves unrelated knowledge intact, while collaborative frameworks such as CollabEdit aggregate edits from multiple clients without data sharing. Combining the two appears trivial. However, we show that this naive combination fails structurally, and we identify its cause. Guided by this analysis, we propose FedAlphaEdit. To our knowledge, this is the first collaborative knowledge editing framework that aligns both local editing and the server-side merging rule under a single null-space principle for preserving existing knowledge. FedAlphaEdit builds on null-space-aligned merging, in which clients share projected statistics and the server provably recovers the result of editing everything in one place under a one-shot idealization. Empirically, the proposed method repairs the collapse and brings edit success and preservation simultaneously close to the level of centralized editing across two architecture families. FedAlphaEdit thus lets institutions that cannot share raw edit data, such as hospitals and financial firms, jointly maintain a shared model that closely approximates editing all facts in one place.

1 Introduction

Institutions need to consolidate knowledge edits without sharing raw requests, while preserving unrelated knowledge. AlphaEdit guarantees preservation for local edits, but its behavior under multi-client merging had not been examined.

Introduction

  • CollabEdit supports collaborative editing by aggregating key statistics with a covariance-weighted rule, while keeping requests local is a data-locality property, not a privacy guarantee.The setting matters for institutions such as hospitals and financial firms that cannot share raw edit requests.Source: §1 · p. 1
  • Naive merging loses 4.2 neighborhood-specificity points and 17.4 edit-success points, while pushing up to 14% of the merged update outside the null space.Both degradations grow with client count at a fixed total edit budget and vanish at N = 1.Source: §1 · p. 1, Table 1 · p. 1, Figure 1 · p. 2
  • The collapse occurs because CollabEdit’s merge weights cancel the MEMIT-type normalizer, but not AlphaEdit’s differently structured normalizer.Lemmas 2 and 4 show that the local null-space constraint alone does not survive covariance-weighted merging.Source: §1 · pp. 1–2
  • FedAlphaEdit aligns server-side merging with the null-space principle, using projected covariance statistics and matching weights to recover global AlphaEdit editing exactly in the one-shot regime.For genuine sequential editing the identity holds only approximately, with resulting drift quantified empirically; the method shares no raw keys and includes CollabEdit’s merge as its P0 = I instance.Source: §1 · p. 2
  • FedAlphaEdit restores neighborhood specificity to the centralized level, brings oons to the measurement floor, and closes most of the edit-success gap on GPT-2 XL and Qwen2.5-1.5B.Figure 1 evaluates both methods as the same 5000-edit workload is partitioned across growing client counts.Source: §1 · p. 2, Figure 1 · p. 2

2 Related Work

Prior work addresses sequential knowledge preservation and collaborative editing largely as separate problems; FedAlphaEdit aligns local editing and server-side merging under one null-space preservation principle.

2.1 Locate-then-Edit Knowledge Editing and Sequential Degradation

  • ROME and MEMIT established locate-then-edit methods with closed-form, batched updates to feed-forward weights (Meng et al. 2022, 2023).Source: §2.1 · p. 2
  • AlphaEdit constrains updates to the preserved-key null space, while later work studies approximate-projector leakage and extensions to sequential editing (Fang et al. 2025; Liu, Liu, and Li 2026; Lyu et al. 2026; Eskandar et al. 2026).The guarantee is exact only for an exact null space; finite key samples yield an approximate projector whose leakage can accumulate.Source: §2.1 · p. 2

2.2 Collaborative and Federated Knowledge Editing

  • CollabEdit merges covariance-weighted key statistics without raw edit requests, but its non-destructiveness depends on editor-specific cancellation and its combination with null-space-constrained editing was unexamined (Zheng et al. 2025; McMahan et al. 2017).FedAlphaEdit is presented as the first collaborative framework aligning local editing and server-side merging under a shared null-space preservation principle.Source: §2.2 · p. 2
  • Task-vector merging and merge-stage null-space projections address task-performance interference or task-vector and LoRA fusion, not preservation guarantees for aggregated closed-form key-covariance updates (Ilharco et al. 2023; Yadav et al. 2023; Qiu et al. 2026, 2025).Source: §2.2 · p. 3

3 Method

The method diagnoses why combining AlphaEdit with CollabEdit breaks preservation and edit success, then derives a null-space-aligned merge that recovers pooled AlphaEdit editing under stated one-shot conditions.

3.1 Preliminaries and Notation

  • The method defines P0 from low-eigenvalue directions of preserved-key statistics and each client’s update from its edit residual, key covariance, and ridge.Preservation is the condition ΔK0 = 0; the released CollabEdit merge adds shared covariance to each client’s weight but only once to the total, creating a surplus.Source: §3.1 · p. 3

3.2 A Right-Projected Form of the Update

  • Each AlphaEdit update factors as Δ = DP0, so it preserves K0 exactly when P0K0 = 0, including after cross-round accumulation.With an approximate null space, the update inherits leakage bounded by the approximation error.Source: §3.2 · p. 3

3.3 Merging Breaks Preservation

  • Merging client updates can break preservation because merge weights that do not commute with P0 create a structural leak; scalar weights, including simple averaging, preserve it.Cross-covariance between preserved and orthogonal subspaces generally makes weights fail to commute, and the merged update places up to 14% of its norm share outside the null space.Source: §3.3 · p. 3

3.4 The Cancellation Identity and the Edit-Success Collapse

  • AlphaEdit’s normalizer does not cancel against CollabEdit’s merge weights, so one mismatch causes both out-of-null-space leakage and distortion within the null space.The shared-covariance weight cannot generally fix the mismatch; its per-client addition creates an (N −1)λC surplus, while at N = 1 the distortion is zero.Source: §3.4 · p. 4

3.5 Null-Space-Aligned Merging

  • Under a one-shot, common-state idealization, null-space-aligned merging exactly recovers pooled AlphaEdit editing and adds no out-of-null-space component.Clients share their updates and projected statistics P0AiP0, not raw keys; the rule requires the ridge to enter the global normalizer exactly once. CollabEdit is its P0 = I special case.Source: §3.5 · pp. 4–5

4 Experiments

In single-round collaborative editing, naive covariance-weighted merging undermines both edit success and preservation. FedAlphaEdit aligns local updates and server merging with the null-space constraint, recovering most editing while bringing preservation close to centralized editing across models and datasets.

4.1 Experimental Setup

  • The primary MCF evaluation uses 5,000 edits across 10 clients on GPT-2 XL and Qwen2.5-1.5B, with single-round C1=0 and three-seed client partitions.The main results use a single-round multi-layer regime where the common-state premise of Theorem 1 does not hold exactly; reported measures include edit success, neighborhood specificity, GLUE, and oons.Source: §4.1 · p. 5, §4 · p. 5

4.2 The Naive Combination Breaks Both Axes

  • The naive combination loses 17.39 edit-success points and 4.19 neighborhood-specificity points versus centralized AlphaEdit, while merged updates leak 14.0–14.1% of their norm outside the null space.As client count grows from N=2 to 5 to 10, neighborhood-specificity drops 1.15 → 2.38 → 4.19 points and edit success falls 93.80 → 87.28 → 79.69; degradation persists under fixed-condition controls.Source: §4.2 · p. 5, Table 2 · p. 6

4.3 Null-Space-Aligned Merging Recovers Preservation, and Most of Editing

  • 94.67 ± 0.18 edit success, 66.38 ± 0.19 neighborhood specificity, and 3.21×10−4% oons show FedAlphaEdit recovers most editing while matching preservation.Its edit success is 2.41 points below centralized AlphaEdit, versus a 17.39-point gap for the naive combination; neighborhood specificity is 1.95 points above centralized.Source: Table 2 · p. 6, §4.3 · p. 6

4.4 Ablation: Both Constraints Are Necessary

  • Only FedAlphaEdit simultaneously achieves editing and specificity: merge-side projection alone sacrifices 12.22 edit-success points, while scalar merges fail to recover editing.The 1/N average reaches edit success 31.23; unnormalized sum reaches 51.17 with neighborhood specificity 51.07, and γ=0.5 reaches only 55.16. FedAlphaEdit holds its single-round level across rounds; the five-round test is outside Theorem 1’s bookkeeping guarantee.Source: §4.4 · pp. 6–7, Table 2 · p. 6

4.5 Generality beyond MCF: zsRE

  • On zsRE, null-space-aligned merging restores every metric to or within 0.4 points of centralized results on GPT-2 XL and Qwen2.5-1.5B, while the naive structural leak recurs.The Qwen×zsRE behavioral neighborhood degradation for the naive method is not resolvable (−0.26 points), although its structural leak is clear.Source: §4.5 · p. 7, Table 4 · p. 7

4.6 Diagnostics

  • The 2.41-point edit-success residual reflects layer-wise state divergence, not uniform shrinkage: a single-layer control reduces relative weight deviation from 0.4547 to 0.0176 and matches centralized behavioral metrics within variation.The residual error lies within the editor’s writable subspace, while the orthogonal component remains at the oons floor; GLUE is 0.3732 versus 0.3728 for centralized AlphaEdit.Source: §4.6 · p. 7

5 Limitations •

FedAlphaEdit’s guarantees and empirical performance depend on one-shot, common-state conditions, and its current evaluation leaves important questions about scale, shared preservation requirements, and privacy unresolved. The protocol avoids sharing raw keys but does not itself guarantee privacy or provide evaluated privacy protections.

Idealization of Theorem 1

  • Theorem 1’s exactness requires one-shot solving and common-state statistics; sequential edits drift, and GPT-2 XL edit success remains 2.41 points below centralized editing.A 0.0176 weight-space deviation persists even in the single-layer control; the 2.41-point residual is attributed to layer-wise state divergence.Source: §5 · p. 8

Scale

  • Experiments cover two 1.5B-parameter families under a single-GPU budget; naive-collapse magnitude varies from 17.39 points on GPT-2 XL to 4.75 on Qwen.Larger models, non-IID sharding, imbalanced or conflicting edits, client dropout, and low-rank or sketched compression remain untested; dense d0 × d0 statistics incur substantial communication.Source: §5 · p. 8

Shared projector

  • Theorem 1 assumes one shared projector P0, leaving open how clients with heterogeneous preservation requirements would agree on it or ensure public statistics cover private needs.A common projector is considered natural when clients have similar preservation requirements, such as hospitals maintaining one clinical model.Source: §5 · p. 8

Privacy

  • Projected covariance sharing is not a privacy guarantee: low-rank statistics may reveal projected key directions, while Δi may reveal residual information; the protocol assumes an honest-but-curious server without a formal threat model.Reconstruction-attack evaluation and differentially private transmission are left to future work.Source: §5 · p. 8
  • Dense-statistic sums are locally computable and compatible in principle with additive secure aggregation, but this was not evaluated; factorized transmission lacks this property.Secure aggregation was not implemented, and the factorized form requires the server to combine each client’s individual factor; both are left for future work.Source: §5 · p. 8

6 Conclusion

FedAlphaEdit repairs the structural failure of naively merging null-space-constrained edits by aligning local editing and server-side merging under a shared null-space principle. It restores edit success and preservation toward centralized editing, with exact recovery in the one-shot regime and approximate recovery under sequential editing.

6 Conclusion

  • FedAlphaEdit restores neighborhood specificity to the centralized level and closes most of the edit-success gap across both architecture families.The passage also reports preservation at the measurement floor, though its wording is unclear.Source: §6 · p. 8
  • Naively merging independent clients’ null-space-constrained edits causes editing and preservation to collapse because the covariance-weighted merge conflicts with AlphaEdit’s projected normalizer.Source: §6 · p. 8
  • FedAlphaEdit shares projected statistics and provably recovers global AlphaEdit editing in the one-shot regime, while sequential editing yields approximate recovery with disclosed drift.The remaining edit-success residual is attributed to layer-wise state divergence, which Theorem 1’s idealization excludes.Source: §6 · p. 8

A Proofs

The proofs show that null-space-aligned merging matches centralized AlphaEdit and preserves unrelated knowledge, with the argument extending to commuting positive-definite regularizers.

Proof of Lemma 1

  • Lemma 1 rewrites each update as Δ=DP0 by commuting P0 through the β-regularized inverse, for β>0.Source: A Proofs · p. 10

Proof of Theorem 1

  • Theorem 1 shows the merged update equals centralized AlphaEdit with pooled residuals and a pooled normalizer, yielding exact preservation.The proof obtains exact preservation because right-multiplication by (I-P0) annihilates the update’s trailing P0.Source: A Proofs · p. 10
  • Because the identity holds for every P0K0, merged leakage matches centralized leakage for every ε, with no structural leakage term.Source: A Proofs · p. 10

General regularizer

  • The proofs extend to symmetric positive-definite Λ when P0Λ=ΛP0; block decomposition ensures invertibility, allowing both proofs to carry over.Source: A Proofs · p. 10

B Released Implementation Form and Notation Reconciliation

This section reconciles the released implementation with the paper’s notation, distinguishing the code’s one-sided solve operator from the paper’s symmetric second-moment matrix and identifying an over-counted ridge-weighting term.

B Released Implementation Form and Notation Reconciliation

C Experimental Details

The diagnostics show that exact one-shot recovery depends on evaluating key statistics at a common model state, while sequential multi-layer editing introduces drift. That residual changes edit strength within the edited subspace, while the orthogonal error remains at the measurement floor.

Unit test of Theorem 1

  • The unit test gives 3.4 × 10−15 one-shot single-client error, but two-client divergence is 2.4%, rising to 11.8% when the ridge is over-counted.The two-client divergence follows a 0.6% operator difference; the negative control indicates the test detects the forbidden (N−1)-fold β over-counting.Source: C Experimental Details · p. 11

Experimental controls

  • Three-seed conditions use shard seeds 1, 2, and 3, while editing hyperparameters come from released AlphaEdit and CollabEdit configurations without independent tuning.The shared covariance is fixed at C1 = 0; Dynamic Covariance is omitted because no working public implementation exists. Qwen-specific settings are held common across Qwen runs.Source: C Experimental Details · pp. 10–11

Non-IID sharding

  • Under relation-based non-IID sharding, the naive combination reaches ES 80.06, NS 61.94, and oons 12.0%; the proposed method was not measured in this setting.The naive collapse persists at a magnitude comparable to IID sharding.Source: C Experimental Details · p. 11

Drift diagnostics

  • Sequential multi-layer evaluation has mean drift 0.4547, and removing it reduces deviation to 0.0176, a 26-fold drop as the behavioral gap closes.Operator divergence increases monotonically across edited layers and peaks at the last; its five-layer mean is 0.1879. Theorem 1 is exact only when key statistics share a common model state.Source: C Experimental Details · p. 11, Figure 2 · p. 11

Residual error

  • The error lies entirely in the edited subspace: EP0 accounts for 100.0% across layers and seeds, while orthogonal error remains at the oons floor of 3.2×10−4%.The residual therefore perturbs edit strength inside the subspace the editor writes into, leaving preserved knowledge untouched.Source: C Experimental Details · p. 11

Sensitivity and measurement controls

  • Across λ values 10000, 20000, and 40000, no naive-combination setting recovers both edit success and specificity toward centralized levels; scalar-merge oons stays at the measurement floor.Only three scalar-merge γ levels were tested, and finer tuning is untested; the floor result follows by construction. The pipeline also has a shared +1.8 to +2.0 point paraphrase shift excluded as a bf16 artifact.Source: C Experimental Details · pp. 10, 12, Table 7 · p. 12, Table 8 · p. 12
Loading 2610.11033v1…