Source-linked AI summary
Tracing Knowledge Lineage for Federated Unlearning under Mutual Distillation
Zihao Ding, Jun Huang, Guodong Wang
TL;DR
Mutual distillation can copy removed-data effects into students, whose uploads may restore them to a corrected teacher. HERD uses a typed lineage graph to select holders for correction and guard those left uncorrected or offline; on TOFU, it leaves less exposure than UNDO at a fraction of its correction cost.
Problem
Under mutual distillation, correcting only the teacher does not remove effects copied into students, which can return through their uploads.
Source: §I · p. 1Method
HERD builds a transfer-weighted lineage graph, selects the cheapest holders to correct within a cumulative-residual bound, and guards uncorrected or offline holders.
Source: §I · pp. 1–2Results
On TOFU, HERD leaves 18% to 52% less exposure than UNDO, the strongest baseline, at 6% to 12% of its cost.
Source: B. Experimental Results · p. 6Takeaways & Limitations
Partial correction suffices only when the lineage graph's copy number is below one, and the selected holder set remains valid under link outages.
Source: §I · p. 2
Abstract
from arXiv · showhide
An edge server and nearby devices can train language models of different sizes together by mutual distillation. Federated unlearning (FU) and unlearning for large language models (LLMs) execute a removal request on one model. Under mutual distillation, however, the effect of the removed data has been copied into many students, and their uploads teach it back to the corrected teacher. We propose Holder-level Erasure via Replication-weighted Delta-closure (HERD), which decides which holders must be corrected. HERD records a knowledge lineage graph from the transfers that the server already performs and weights each edge by its transfer type. It computes how many times the residual at each holder will be copied and selects the cheapest set of holders whose correction bounds the cumulative residual after the request. A transfer guard and a rejoin rule cover the holders left uncorrected or offline. We prove that partial correction suffices only when the copy number of the lineage graph is below one, that averaging alone never reaches one, and that the correction set remains valid under link outages. On TOFU, HERD leaves 18% to 52% less cumulative residual than the strongest baseline at 6% to 12% of its correction cost.
I. INTRODUCTION
Mutual distillation spreads removed-data effects beyond the teacher, allowing student uploads to restore them after correction. HERD uses transfer-weighted lineage to select holders for correction and guard those left uncorrected or offline.
I. INTRODUCTION
- Mutual distillation copies removed-data effects into students, whose uploads can teach those effects back to the corrected teacher.Some students may also be offline when the request arrives and later rejoin with their old copies.Source: §I · p. 1
- Existing approaches either correct every descendant, cover only teacher-to-student distillation, or resend the full distillation set, leaving holder selection and offline clients unresolved.The cited approaches are lineage-based unlearning, distillation unlearning, and noising descendants before redistillation.Source: §I · p. 1
- Transfer types determine whether residuals are merely spread by averaging or copied by distillation, so correction depends on expected copies rather than distance from the source.The server can estimate copies from its transfer records without sending removed data to devices.Source: §I · p. 1
- HERD builds a typed lineage graph, ranks holders by copy weight, and chooses the cheapest correction set that keeps cumulative residual below δ.A transfer guard screens later transfers, and a rejoin rule corrects offline holders before their first transfer.Source: §I · pp. 1–2
- Partial correction suffices only when the lineage graph's copy number is below one; averaging alone never reaches one, and the selected set remains valid under link outages..Source: §I · p. 2
- On TOFU, HERD leaves 18% to 52% less cumulative residual than the strongest baseline at 6% to 12% of its correction cost.Every model also stays within one point of its retrained counterpart in the forget score.Source: §I · p. 2
II. SYSTEM MODEL AND PROBLEM FORMULATION
The system pairs an edge-server teacher with smaller client students, using mutual distillation across architectures and averaging among same-family clients. Wireless link availability governs which clients participate in each round, while the server tracks holders that may carry learned knowledge.
A. System Model
- The server holds a large teacher, while each client holds a smaller student sized for its hardware; different model architectures exchange knowledge through mutual distillation.Students are same-family when they share the teacher's base model and cross-family otherwise.Source: A. System Model · p. 2
- Each client link has bandwidth B, and the system models its rate in round t.The passage introduces the rate expression but does not state its form in the supplied text.Source: A. System Model · p. 2
- A client link is down when its rate falls below bmin, the rate needed to deliver that round's transfers.The link model uses transmit power pk, noise power σ2, and Rayleigh fading around the pathloss gain.Source: A. System Model · p. 2
- The server denotes the clients with up links in round t as A[t], while βk is client k's per-round outage probability, including leaving server coverage.Each client fine-tunes its student on its local dataset Dk.Source: A. System Model · p. 2
- In each round, the server broadcasts teacher outputs for public prompts, receives student outputs, trains the teacher, averages same-family students, and replays cached outputs.A down client misses that round's transfers and rejoins with its own student when its link returns.Source: A. System Model · p. 2
B. Problem Formulation
- The teacher, students, and cached corpus are holders because each can carry learned knowledge; a removal request identifies a target and its data.The target may be one client's data, entities spread across clients, or poisoned samples, corresponding to client-, class-, or sample-level unlearning.Source: A. System Model · p. 2
B. Problem Formulation
Mutual distillation leaves residuals in the teacher, students, and cached corpus, so correcting the teacher alone does not complete removal. The formulation chooses holders to correct while bounding residuals accumulated across later rounds.
B. Problem Formulation
- Correcting only the teacher lets students upload outputs shaped by the removed data, which the teacher can learn again.Residuals may remain in the teacher, students, and cached corpus when the request arrives.Source: B. Problem Formulation · p. 2
- The residual vector tracks the fraction of the removed data’s effect remaining in each holder, and transfers pass residuals between holders.The inheritance matrix records how much residual each holder inherits from another; a down client link removes that round’s off-diagonal transfer.Source: B. Problem Formulation · p. 2
- The correction set minimizes holder-correction cost while bounding cumulative residual across every holder and later round.The cumulative constraint counts residuals that return through later transfers, even after the teacher is cleared.Source: B. Problem Formulation · p. 2
- Averaging spreads a residual, whereas distillation copies it.This contrast motivates accounting for transfer type when assessing residual propagation.Source: Figure 2 · p. 3
- The optimization cannot be solved directly because residuals are unobservable without querying holders with the removed data, and inheritance varies with transfer type and prompt overlap.Holders with down links at the request time can neither be measured nor corrected.Source: B. Problem Formulation · p. 3
III. HERD: HOLDER-LEVEL ERASURE VIA REPLICATION-WEIGHTED DELTA-CLOSURE
HERD is designed to solve the holder-correction problem in Eq. (4) through three modules. The paper uses figures to illustrate the modules’ timing and the motivating ideas.
III. HERD: HOLDER-LEVEL ERASURE VIA REPLICATION-WEIGHTED DELTA-CLOSURE
- HERD is designed to solve the holder-selection problem in Eq. (4) through three modules.The modules are described in Sections III-A, III-B, and III-C.Source: §III · p. 3
- Figure 4 shows when each HERD module runs.Source: §III · p. 3
- Figures 2 and 3 illustrate ideas that the equations leave implicit.Source: §III · p. 3
A. Typed Knowledge Lineage Graph
HERD builds a typed knowledge-lineage graph from transfers the server already records. Transfer type determines how edges represent residual inheritance, while distillation edge weights reflect how much removed-data content transmitted outputs carry.
A. Typed Knowledge Lineage Graph
- The server records delivered transfers as a holder graph using broadcast acknowledgments, uploads, averaging weights, and additions to the cached corpus.Graph construction adds no transmission, and the server stores outputs and weights rather than model checkpoints.Source: A. Typed Knowledge Lineage Graph · p. 3
- The inheritance matrix factors into holder retention, averaging, and distillation components, encoding their different effects on residuals.The retention fraction is below one; averaging rows sum to one, while distillation edges represent senders retaining what they send.Source: A. Typed Knowledge Lineage Graph · p. 3
- Distillation edge weights depend on how much removed-data content the transmitted outputs carry, with rates fit using harmless canary facts.The model distinguishes cross-family and same-family absorption rates and includes a floor for traits passing between same-family models without appearing in outputs.Source: A. Typed Knowledge Lineage Graph · p. 3
- The Fig. 3 comparison includes a return edge, where hop distance misses the path back to the teacher.Source: Figure 3 · p. 3
B. Copy-Weighted δ-Closure
HERD estimates residual exposure from recorded transfers and uses copy weights to select a correction set. Unlike hop distance, copy-weighted closure accounts for return paths to the teacher.
B. Copy-Weighted δ-Closure
- HERD reconstructs residuals at the request round by propagating recorded inheritance from holders that received the removed data, without querying holders.Source: B. Copy-Weighted δ-Closure · p. 3
- Under random link failures, the server scales edges to and from each client by its link-availability probability when forming expected inheritance.The probability is expressed as 1 − β_k for client k.Source: B. Copy-Weighted δ-Closure · p. 3
- The copy weight φ_j measures the total exposure created by one unit of residual at holder j across later transfers, including returns to the teacher.When the spectral radius of the expected inheritance matrix is below one, Theorem 1 converts the cumulative constraint into a copy-weighted closure condition.Source: B. Copy-Weighted δ-Closure · p. 3
- Hop distance can miss a return path because a student can teach the teacher again, whereas copy-weighted closure captures that exposure.Figure 3 illustrates the contrast on a graph with one return edge.Source: B. Copy-Weighted δ-Closure · p. 3
- With unit costs, correcting holders in decreasing φ_j r_j[t_u] order gives the smallest closure; general costs yield a covering knapsack solved by dynamic programming.The dynamic-programming method uses discretized exposure and has the guarantee of Proposition 3.Source: B. Copy-Weighted δ-Closure · pp. 3–4
- Student correction sends corrected-teacher outputs on recorded prompts that carried the removed data, making slow-link students more costly to correct.The removed data themselves remain at the server.Source: B. Copy-Weighted δ-Closure · p. 4
C. Transfer Guard and Rejoin
HERD guards transfers to limit residuals in holders it leaves uncorrected and handles offline clients when they rejoin. It adds no transmission beyond student corrections.
C. Transfer Guard and Rejoin
- The transfer guard screens broadcasts, replays, and uploads for outputs carrying the removed content, setting their excitation to zero.Only the floor of same-family edges survives.Source: C. Transfer Guard and Rejoin · p. 4
- Offline clients are corrected before their first upload after rejoining if their residual would exceed the remaining budget; otherwise, they rejoin under the guard.The server uses each client's last-transfer round to track its residual.Source: C. Transfer Guard and Rejoin · p. 4
- HERD stores each round's transmitted outputs and screens later transfers, adding no transmission beyond corrections of selected students.Source: C. Transfer Guard and Rejoin · p. 4
IV. THEORETICAL ANALYSIS
The analysis relates cumulative residual to the lineage graph's copy number and gives conditions for partial correction, a transfer guard, and offline-client handling. It also provides procedures for selecting a correction set.
IV. THEORETICAL ANALYSIS
- When the mean copy number is below one, expected cumulative residual equals the bound in Eq. (8); at or above one, it is infinite if any residual remains.With no distillation, the copy number is strictly below one.Source: §IV · p. 4
- Distillation can return removed content, whereas averaging alone does not; the guard bounds the copy number using same-family distillation edges.The guard brings it below one when within-family spread is sufficiently weak, as measured by ν.Source: §IV · p. 4
- The correction set remains valid with offline clients if each is corrected before its first transfer after rejoining.A client's expected return delay, 1/(1 − β_j), contributes to the residual budget even if it is never corrected.Source: §IV · p. 4
- With unit costs, selecting holders in decreasing order of a_j gives a minimum-size closure; with general costs, dynamic programming finds a feasible correction set.The general-cost procedure takes O((N + 2) δ/ϵ) time and costs at most the optimum with δ replaced by δ − (N + 2)ϵ.Source: §IV · p. 4
- Choosing ϵ = δ/(10(N + 2)) loosens the budget by one tenth while keeping the running time quadratic in N.Source: §IV · p. 5
A. Experimental Setup
The experiments evaluate HERD in a simulated edge federation and on a two-client hardware testbed. They measure unlearning quality, residual exposure, correction cost, and scalability.
A. Experimental Setup
- The simulation uses 50 clients uniformly placed within a 500 m-radius cell around the edge server, with a Qwen2.5-7B-Instruct teacher and equal halves using Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct students.Each round distills on 256 public prompts.Source: A. Experimental Setup · p. 5
- Client, class, and triggered-sample removal requests test unlearning across three types of forget sets.The hardware testbed uses a Jetson AGX Thor server and two Jetson AGX Orin clients over Wi-Fi.Source: A. Experimental Setup · p. 5
- FRL and model utility assess forget-set behavior and retained-author utility, while forget quality, membership-inference accuracy, resurrection, exposure, and cost capture additional outcomes.The forget-score evaluation compares unlearned holders with models retrained without the removed data.Source: A. Experimental Setup · pp. 5–6
- HERD is compared with SimNPO, FIUn, DU, UNDO, and PURGE; the figures examine closure behavior, outages, inheritance, and scalability.The scalability plots track closure time and storage as client count grows, and closure size and correction traffic as the request round changes.Source: A. Experimental Setup · p. 6, Figure 6 · p. 6, Figure 7 · p. 6
B. Experimental Results
HERD reduces cumulative exposure relative to the strongest baseline at a fraction of its correction cost. The conclusion frames it as targeted correction of high-copy-weight holders with safeguards for the rest.
B. Experimental Results
- 18% to 52% less exposure than UNDO, the strongest baseline, at 6% to 12% of its cost; the hop-based closure exceeds δ at seven of ten budgets, and closure takes at most 121 ms for N = 200.Source: B. Experimental Results · p. 6
VI. CONCLUSION
- HERD corrects holders with the largest copy weights in a typed lineage graph and guards the remaining holders.Source: §VI · p. 6