Source-linked AI summary
Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints
Junlong Shen Xingyu Li
TL;DR
The paper asks whether unlearning verdicts are affected by undocumented batch-normalization fitting conventions in released checkpoints. It refits normalization state on retained data at unchanged weights across released artifacts and finds checkpoint-specific shifts, with narrow but real verdict consequences and no crossings in a deliberately near-criterion trained population.
Problem
Unlearning audits read published quantities from unlearned and retrained models, but releases do not record the batch-normalization fitting convention that produced their deployed states.
Method
The audit refits each checkpoint’s deployed normalization state on retained data at byte-identical trainable weights, anchors reproduced releases, and tests shifts across independently redrawn replicates.
Results
47 of 221 released checkpoints moved beyond their release seed spread, while exchanging kept for removed records barely moved published cells and state drift tracked displacement.
Takeaways & Limitations
Published unlearning numbers and some verdicts are properties of individual shipped checkpoints, so batch-normalized releases should name the fitting convention beside each number.
Takeaways & Limitations
The census is limited to convolutional, batch-normalized vision releases, and the trained near-criterion population produced no verdict crossings.
Abstract
from arXiv · showhide
An unlearning audit reads its verdict off numbers that an unlearned model and its retrained reference each publish, and both also ship batch-normalization statistics that no gradient step wrote and no release records. Refitting them on kept data at bit-identical weights moves 47 of 221 released checkpoints past the spread their own release's seeds show, several inside a method whose average does not move: what moves is the checkpoint's property, not its method's. What does the moving is not the removed data surviving in the state: exchanging kept records for removed ones inside a fixed fitting pool moves a published cell by almost nothing, while how far a checkpoint's shipped state has drifted from any refit does track it. The consequence for a published decision is real but narrow: twelve verdicts cross, four clear a measured recalibration budget, two clear it on every replicate, and a population we trained and sited near its own criterion yields none. A release should therefore name the fitting convention beside the number, on the batch-normalized vision models where this channel exists.
1 INTRODUCTION
Unlearning audits compare an unlearned checkpoint with a retained-data retrained reference using published accuracy cells, but batch-normalization state is fitted under undocumented conventions. This audit refits that state at unchanged weights and finds that published numbers shift per checkpoint, mainly with state staleness rather than removed-data provenance.
- Motivation: Published quantities can depend on whether the deployed normalization state was shipped, retrained online, or refit afterward, yet releases record none of these conventions.The test refits only the deployed state on retained data while keeping trainable tensors byte-identical.
- Scope: The audit addresses both whether a release’s arms use the same state convention and whether any published cell is sensitive to that convention.The first question requires a release shipping the retrained reference; the second is measurable from any release.
- Method: A refitted-minus-shipped shift is defined per checkpoint, and checkpoints are anchored only after reproducing their own published cells.The audit reports accuracies on removed, retained, and test subsets and uses a fixed ±1.2 percentage-point margin for displacement.
- Findings: The median checkpoint shift was 2.247 pp, while exchanging kept records for removed ones changed published forget accuracy by only +0.015 pp, or +0.057 pp with every removed record included.State drift from a retain-only refit, rather than removed-data provenance, tracks the observed displacement.
2 DEPLOYED STATE AND THE AUDITS THAT OMIT IT
Unlearning audits often evaluate released outputs and parameters while omitting deployed batch-normalization state. The paper measures this documentation gap and distinguishes it from prior work on training history and normalization-statistics estimation.
- Existing audits: Unlearning evaluations compare an unlearned artifact with an omission-retrained reference using retained and removed accuracy, often alongside membership-inference scores.The literature varies the readout, attack, and enforcement setting used by the audit.
- Positioning: This paper studies a fitted deployed state absent from the released weights, whereas related work studies training history or the estimator used for normalization statistics.The authors adopt exact-statistics recomputation as a rebuild rather than proposing it as a new operation.
- Documentation gap: Only 4 of 19 reviewed papers using batch-normalized convolutional backbones mention the deployed state, or 3 excluding concurrent work focused on it.The count describes what papers document, not necessarily what their authors did.
3 A RECONSTRUCTION TEST FOR RELEASED ARTIFACTS
The reconstruction test holds trainable weights fixed and refits deployed normalization state on retained data, then measures changes in published quantities and audit verdicts. It is applied to reproducible checkpoint populations using anchoring and family-aware equivalence testing.
- Test definition: The test replaces deployed normalization state ϕ with a retained-data refit ˆϕ while keeping trainable tensors θ byte-identical.The reported shift is m(θ, ˆϕ) − m(θ, ϕ), and displacement requires clearing the stated margin on every replicate.
- Populations: The primary population is MU-Bench’s CIFAR-100 ResNet-50 release, with 180 checkpoints and 72 admitted by |df −dt| ≤0.05.The population spans three families, three procedure variants, five removal ratios, and five seeds.
- Anchoring: Anchoring reproduces published df, dr, and dt within 0.02 for 71 of 72 primary artifacts, while all 8 exclusions in the second population are SalUn.SalUn fails to anchor on forget accuracy in 54 of 55 CIFAR-10 cases, while its other two cells reproduce normally.
- Equivalence analysis: Equivalence is tested with two one-sided tests at the 90% level against a ±1.2 percentage-point margin, using family-aware intervals.The interval structure accounts for dependence among variants, removal ratios, and seeds rather than treating the grid as independent observations.
- Illustrative reconstruction: At unchanged tensors, one bad-teaching checkpoint moves from |df −dt| = 0.0414 to 0.0573 after retained-data state refitting, crossing the 0.05 criterion bound.The shipped checkpoint reproduces its published cells before refitting.
4 WHAT THE REBUILD MOVES, AND WHAT IT LEAVES ALONE
The rebuild shows that published unlearning measurements can shift at the checkpoint level, while provenance of removed records has little effect and state staleness tracks displacement. These shifts alter some published decisions, but the effect is bounded by release, metric, and population.
- 4.1 HOW FAR A PUBLISHED NUMBER MOVES IS INDEXED BY THE CHECKPOINT: 47 of 221 checkpoints are individually displaced on forget accuracy, 39 on retain, and 40 on test, while a family equivalent to zero still contains 8 of 45 displaced checkpoints.All displacements are certified at q = 0.05; the primary population uses clustered family intervals, so checkpoint-level effects need not appear in family means.
- 4.3 WHAT INDEXES IT IS A STALE STATISTIC, NOT THE REMOVED DATA: The published number moves little when kept records are exchanged for removed records: the deployment-share contrast is +0.015 pp, versus +0.057 pp with every removed record in the fitting pool.No checkpoint exceeds 0.5 pp on either provenance contrast, whereas the census margin is 1.2 pp.
- 4.3 WHAT INDEXES IT IS A STALE STATISTIC, NOT THE REMOVED DATA: A checkpoint’s shipped-state distance from a retain-only refit clears the reference band on all twenty checkpoints and is 5.43 times larger at the median than provenance contrasts.The distance also tracks the displacement, supporting stale shipped statistics as the measured index rather than removed-data provenance.
- 4.3 WHAT INDEXES IT IS A STALE STATISTIC, NOT THE REMOVED DATA: The intervention isolates buffer-update trajectory rather than fitting-pool provenance, because paired models share bit-identical weights while differing in buffer state.The paired arms remain within the metric’s 12.93 pp detection floor, so admissibility and resolvability do not overlap for this producer.
- 4.4 PUBLISHED VERDICTS CROSS THE THRESHOLD IN BOTH DIRECTIONS: The rebuild changes release verdicts in both directions: eight of 71 natively admitted MU-Bench checkpoints fall out, while four of 42 excluded checkpoints enter.On the second release, two of four natively failing checkpoints pass without a trade-off.
- 4.5 HOW FAR THE ACCOUNT REACHES, AND WHERE IT STOPS: The result is bounded across populations: SVHN shows +0.473 pp against a 0.323 pp margin, while a criterion-sited CIFAR-100 population has no crossings among 45 checkpoints.The CIFAR-100 population nevertheless has 20 of 45 checkpoints displaced past its 1.868 pp margin, including one at +13.26 pp; none clears its 0.085 pp recalibration budget.
5 LIMITATIONS
The evidence is limited in scope and interpretability: family-level differences remain unresolved, some populations and effects come from narrow release settings, and conclusions apply to convolutional batch-normalized vision models rather than language models.
- Scope of evidence: Family-level differences are not established, and one family’s displacement cannot be resolved under the study’s rebuild design.The second-population contrast reflects displacement under the authors’ rebuild rather than a procedure difference, while family means on the primary release cannot distinguish planted zero from +3.0 pp effects.
- Interpretation: The census is a floor rather than an estimate, uses false-discovery rather than familywise multiplicity control, and leaves procedural causes of stale state unresolved.The classifier separates an implementation property and buffer-update mode, not the removal algorithm itself; the matched interpretation remains unresolved.
- Scope of evidence: The forgetting-quality result comes from one release family of ten classes sharing a single base training, with the effect-concentrating split chosen after the first draw.The reported shift is +0.0167 against an across-class spread of 0.0119.
- Generalizability: The study remains confined to convolutional batch-normalized vision models across three codebases, three datasets, and four architectures; it does not establish how language-model cells move.Layer-normalized models are not included, and other pipeline quantities such as calibration temperatures, feature scalers, and class priors remain untouched.
6 CONSEQUENCES FOR DELETION AUDITS
The audit finds that published cells depend on how batch-normalization state is fitted, so releases should report the fitting convention rather than only the shipped number. The evidence concerns reproducibility of measurements, not deletion safety.
- Population and scope: A released arm’s forget cell is not recoverable without its own forget-set definition, while other families anchor at 71 of 72, 42 of 43, and 40 of 40.The failure is narrower than claiming the arm’s full published table is unreproducible.
- Population and scope: 263 anchored checkpoints yield a census of 221 because the second release enters through a paired contrast, while excluded and quantized strata are counted separately.Anchored checkpoints reproduce their own published cells as shipped; census checkpoints are decided on ten re-keyed replicates.
- State rebuild: Changing the displacement margin from 1.2 to 1.0 pp raises the forget count from 47 to 56, while 1.5 pp lowers it to 31 and 2.0 pp leaves 23 uncertified.The census explicitly treats the margin and retained-pool size as swept design parameters rather than estimated quantities.
- State rebuild: The rebuild replaces batch-normalization running statistics at frozen weights, using sequential layerwise fitting from retained data and leaving trainable tensors untouched.The exact refit computes means and unbiased variances from forward activations; training-mode accumulation would depend on batch partition.
C THE CONTROLS, COLLECTED
The controls compare the observed displacement with measurement rivals, replicate sensitivity, checkpoint scatter, and shipped-state drift. They support a state-rebuild association while retaining unresolved or withdrawn components where the evidence does not identify them.
- Detection and rival controls: A planted 1.5 pp effect is recovered on 134 of 150 CIFAR-10 checkpoints, while 2.0 pp is recovered on all 150 and effects at or below threshold recover none.The detection floor is measured at the geometry used by the census.
- Detection and rival controls: The controls measure four rival explanations, with three smaller than the effect or pointing the wrong way and native accuracy remaining unresolved.The table includes two known-answer negative controls, one per population.
- Detection and rival controls: Checkpoint scatter cannot explain the smallest displaced artifact at the census margin without a 0.746 pp standard deviation, estimated as 6.2× and 10.3× larger than relevant upper bounds.This prices checkpoint-to-checkpoint variability rather than assuming it away.
- Calibration limitation: The withdrawn bootstrap-t calibration returned size 0.0643 instead of 0.050 and critical value 2.499 instead of 2.132, so final counts rely on the exact sign test.The authors state that the bootstrap-t on five centred residuals was anti-conservative.
D THE SECOND METRIC FAMILY
An independent zero-retrain-forgetting metric reads the same deployed batch-normalization state through a different statistic. It corroborates the family pattern without extending the channel beyond that state.
- Independent forgetting measure: The recomputed metric anchors 70 of 71 artifacts at a median absolute error of 0.0036, while 71 of 71 reproduce the frozen accuracy wave with worst difference 0.0 pp.The wave covers all 72 artifacts admitted by the benchmark for the new metric.
- Independent forgetting measure: The displacement correlation with native accuracy is −0.30 pooled and −0.56 and −0.65 within two families, so the result licenses no causal claim.This readout widens the statistic but not the batch-normalization channel’s reach.
- Independent forgetting measure: The zero-retrain-forgetting measure separates the same families more sharply than accuracy, using three re-keyed replicates and both the release margin M and wider sensitivity margin M_s.The no-removal base checkpoint serves as the metric’s negative control.
E THE SECOND DEPLOYED STATE: QUANTIZATION SCALES
A second deployed state, static 8-bit activation quantization, tests whether calibration-set changes move published measurements. The observed scale shifts are small or decay with calibration size, and the accuracy effect does not reach an audit cell.
- Quantization scales: Static 8-bit activation scales are refitted on a retain-only calibration set, changing calibration records while leaving trainable tensors and normalization buffers untouched.The quantized convolutional group contains 52 sites, and the method uses one scale per tensor.
- Quantization scales: The displacement contrast compares each shipped checkpoint with its retain-only refit, while the identification contrast exchanges removed and retained records within a shared calibration core.The contrasts use the release’s ±1.2 pp margin and thirteen re-keyed draws for the accuracy census.
- Quantization scales: A plumbing control failed because batch-size-dependent kernel selection produced a 0.0333 pp cross-wave difference, but every reported displacement remains a within-wave contrast.The authors retain the failed bar rather than round it to a pass.
- Quantization scales: Absmax scale influence decays slowly with calibration size, whereas percentile and mse influence decay as O(1/n); median absmax log ratios fall from 0.031 to 0.015 between 128 and 2048 records.The predicted accuracy pattern passes on the convolutional group but is unevaluable on the transformer group because its largest-size median displacement is exactly zero.
- Quantization scales: The quantized-state census returns zero on both architectures, including the architecture without normalization statistics.Table 9 reports the same census on the second deployed state using thirteen re-keyed draws per checkpoint.
F A THIRD POPULATION, BUILT FROM SCRATCH
A self-trained SVHN/VGG-16-BN population tests whether the observed displacement generalizes beyond released CIFAR convolutional networks. Recalibration moves one arm but controls remain near zero, while frozen-buffer variants expose a narrower state-level mechanism with substantial utility costs.
- How it was built: The population is self-trained on SVHN with VGG-16-BN, using five seeds and ten independently re-keyed retained-subset draws per artifact.
- What it returns: +0.473 pp is the average retain-only rebuild shift for four admissible masked-relabelling artifacts, with 2 of 4 clearing the margin.The corresponding per-artifact means are 0.240, 0.280, 0.431 and 0.942 pp.
- What it returns: The three same-instrument controls return near zero: −0.006 pp for retain-only fine-tuning, −0.010 for the no-removal base, and +0.024 for omission retraining.
- What the frozen twin shows: Freezing buffer updates leaves weights bitwise identical but can reduce test accuracy by −9.50 to −75.81 points, so the twin intervention has a major utility cost.A retain-only rebuild recovers +11.119 pp of forget accuracy on the fine-tuning arm against −0.006 for its default twin.
- What the frozen twin shows: The frozen-twin evidence shows that stale deployed buffers can carry behavior not determined by the released weights, but does not establish that the cost comes from removed data.Exposure and weight movement are confounded across the two usable arms.
- What it returns: Two planned predictions were not evaluable: default gradient ascent was inert or collapsed test accuracy, and frozen-buffer variants fell outside the admissible utility window.
G A FOURTH POPULATION, SITED WHERE A VERDICT COULD CROSS
A fourth, self-trained CIFAR-100/VGG-16-BN population was deliberately placed near its own criterion to make verdict crossings reachable. Although displacement reappeared at released scale, none of 45 checkpoints crossed because refitting widened rather than narrowed the criterion gap.
- How it was built, and how it was sited: The population uses CIFAR-100, VGG-16-BN, the same public implementation, five seeds, and independently redrawn retained subsets and augmentation draws.
- What the anchors and controls return: Its five base checkpoints were sited between 72.00% and 72.42% test accuracy, within a pre-fixed band, and the instrument matched the third population to 0.000 pp across shared cells.
- What it returns: +13.26 pp is the largest per-checkpoint displacement, with 20 of 45 checkpoints displaced beyond the population’s own margin.The released-scale magnitude reproduces, but the verdict effect does not.
- What it returns: None of the 45 checkpoints crosses its criterion because the rebuild lifts the forget cell farther than the test cell and widens the gap on 33 of 45.
- Corrections: The first aggregate analysis was corrected because it included non-default frozen-buffer crossings and ignored the effect’s shift direction.
H MATCHING WEIGHT MOVEMENT BETWEEN AN EXPOSED AND AN UNEXPOSED PROCEDURE
Matched exposed and unexposed procedures isolate buffer-trajectory differences while holding trainable weights identical, but the primary matched-pair reading remains unresolved. Curve-based evidence separates exposure groups, although most frozen arms fail the utility floor and the family-level ordering test is limited.
- The primary reading: Only one of three matched pairs is admissible on all five seeds, and its largest arm-seed contrast is 0.533 pp against a 12.93 pp detection floor.The other two pair contrasts are +13.04 and +51.37 pp, but their frozen arms remain admissible on 3 and 0 of 5 seeds.
- Curve reading: +33.80 pp separates exposed from unexposed twin gaps across five seed clusters, while a near-matched pair differs by +51.37 pp with the same sign on all seeds.The unmatched operating point gives +65.31 pp with five of five seeds agreeing in sign.
- What it isolates, and what it does not: The paired procedures differ in buffer accumulation trajectories, not in whose data enters the fitting pool, because their input streams largely share images and normalization statistics never read labels.
- What it isolates, and what it does not: Most frozen arms fail the utility floor, so the state-level result concerns deployed-state content rather than an artifact anyone would ship.
- Family-level reading: +1.899 pp versus +0.034 pp separates HIGH and ZERO families, but the ordering test returns p = 0.0556, its design floor with three and four families.
- Family-level reading: The family prediction concerns averages rather than which individual checkpoint moves, and an unseen prospective population’s zero-exposure cells return −0.006 and +0.024 pp.
P EXPOSURE VERSUS ESTIMATION
A fixed-pool provenance test exchanges retained and removed records while holding the fitting setup constant. The resulting cell shifts are tiny, whereas shipped-state distance from a retain-only refit is consistently large, supporting an estimation/convention explanation rather than removed-data provenance at these shares.
- Design: The design fixes tensors, estimator, training transform, and an 8000-record fitting pool, varying only which records fill the pool.
- What the cell-level arms return: +0.015 pp at the deployment share and +0.057 pp at the ceiling are the pooled forget-accuracy contrasts from exchanging retained and removed records.No checkpoint exceeds 0.5 pp on either contrast; the largest per-checkpoint mean is +0.271 pp.
- Which kind of null it is: The provenance contrasts are 0.00061 and 0.00101, with 0 and 1 of 20 checkpoints above the 0.0019 null band, while every checkpoint’s shipped-to-refit distance clears it.The retain-only refit distance is 0.00538, 5.43 times larger at the median, ranging from 1.82 to 15.00.
- Which kind of null it is: The result supports the second null branch: at these removal shares, the removed data does not measurably change the fitted state for the audit to detect.The provenance distances remain conservative upper bounds because finite-sample replacement can perturb estimates even when provenance is irrelevant.
Q FORGETTING QUALITY: TWO READOUTS, ONE USABLE
Of the two forgetting-quality readouts, only the second release’s oracle-based comparison was usable. Under that comparison, SSD’s refit shifted one family toward the un-unlearned base, while the weight-movement rival did not explain the shift.
- Primary release: The primary release’s membership attack was unreadable, while its published privacy metric moved by −0.789 pooled and up to 3.633 under the convention.The attack’s median cross-validation accuracy was 0.518 against a 0.55 floor, with no certified count at any depth.
- Oracle-based readout: SSD moved +0.0167 toward the un-unlearned base on 10 of 10 classes, with the sign holding across three draws and both pole conventions.Five classes passed the preregistered materiality bar; the other three families remained at noise with inconsistent signs.
- Oracle-based readout: The effect was larger on shippable checkpoints: +0.0215 across seven classes clearing the release’s retain bar versus +0.0054 across three failing it.All seven qualifying classes had the same sign, and the split was confirmed on later draws.
- Weight-movement rival: Weight movement was an inadequate rival: SalUn moved farther than SSD between families but drifted less, and SSD’s within-family correlation was significantly negative at ρ = −0.794.The permutation test gave p = 0.0085, opposite the predicted positive relation.
R NEGATIVE AND UNRESOLVED RESULTS
The negative tests do not establish removed-data provenance or a universal checkpoint-readable predictor. Several proposed explanations fail, while one weight-distance signal works only within the CIFAR-10 groups and a preregistered identity holds only as an unpromoted observation.
- Unresolved mechanisms: The removed-data provenance channel was not established: four of eight artifacts rejected a retain-only swap control, exceeding the preregistered allowance of one.The readout itself appeared sound, with a no-removal checkpoint reaching at worst p = 0.402 across removal ratios.
- Unresolved mechanisms: The removed-fraction identity failed every preregistered bar on the only moving family, including a shift correlation of −0.211 with p = 0.81.Leave-one-ratio-out prediction also lost to the family mean.
- Unresolved mechanisms: The two zero-equivalent families fit the removed-fraction identity as an unregistered observation, but the paper withholds it as a claim pending preregistered confirmation.Their correlations ranged from +0.56 to +0.58 across three metrics, with intervals containing zero for the intercepts.
- Predictors: Native accuracy did not explain the census overall: displacement correlations were −0.137 in the primary population and −0.010 in the CIFAR-10 groups.Some within-family correlations were stronger, but the large primary families were near zero and the headroom prediction was not recovered.
- Predictors: No predictor transferred universally: weight distance was informative in all three CIFAR-10 families but recovered nothing in the primary CIFAR-100 population.The authors therefore support per-artifact measurement rather than a universal screening rule.