PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Retraining-Free MoE Compression: A Cost-Normalized Study of Post-Compression Adjustment

Sieun Hyeon, Jaeyoung Do

arXiv:2609.06076v2cs.LG

TL;DR

MoE compression reduces deployment memory, but compressed checkpoints are often treated as final artifacts despite residual performance gaps. This paper compares small post-compression adjustment strategies across compressed models; with 3,000 C4 examples and one epoch, Full FT recovers 37.3% of the original-to-compressed performance gap on average.

  • Problem

    Although retraining-free MoE compression reduces memory, compressed checkpoints can retain routing shifts, expert-function distortion, and router–expert interaction errors.

    Source: §1 · p. 1

  • Method

    The study compares LM fine-tuning and teacher-based KD, along with trainable parameter scopes, as post-compression adjustment under a small data budget and measured GPU costs.

    Source: §1 · pp. 1–2

  • Results

    37.3% of the original-to-compressed performance gap is recovered on average with Full FT after one epoch over 3,000 C4 examples; LM fine-tuning is more cost-effective than standard token-level KD in this setting.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support treating compressed checkpoints as initializations and pairing retraining-free compression with a small adjustment stage for recovery.

    Source: §7 · p. 9

    The GPU-cost comparison is controlled but not fully system-optimized, and the study does not evaluate how specialized system optimizations would change the cost–recovery trade-off.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

Retraining-free MoE compression reduces deployment memory by pruning or merging experts, but often treats the compressed checkpoint as the final artifact. We argue that this view is incomplete: compressed MoE checkpoints are better understood as compressed initializations that benefit from a tiny post-compression adjustment stage. Across two MoE LLM backbones, four pruning/merging methods, three expert-retention ratios, and 28 benchmarks, we compare LM fine-tuning and teacher-based KD under matched small-data budgets and measured GPU costs. Using only 3,000 C4 examples and a single epoch of adjustment, Full FT recovers 37.3% of the original-to-compressed performance gap on average. Moreover, LM fine-tuning is more cost-effective than standard token-level KD, and full-parameter adjustment gives the strongest cost--recovery trade-off among the tested scopes. These results suggest that retraining-free compression should be paired with small post-compression adjustment to recover a substantial portion of the performance lost during compression.

1 Introduction

MoE reduces computation by activating only some experts, yet deployment still requires all expert parameters in memory. This paper treats compressed checkpoints as initializations for a small adjustment stage and compares adjustment objectives and parameter scopes across compressed models.

1 Introduction

  • The authors frame pruning and merging as changing the expert landscape and leaving residual routing shifts, expert-function distortion, and router–expert interaction errors.They therefore treat post-compression adjustment as a practically important recovery stage for near-original performance.Source: §1 · p. 1
  • The study asks which compression-and-adjustment pipeline gives the best recovery under the same small data budget and measured GPU cost.Its stated goal is to evaluate post-compression adjustment as a design axis, not to introduce another compressor.Source: §1 · pp. 1–2
  • With one adjustment epoch over 3,000 C4 examples, compressed MoE LLMs recover 37.3% of the original-to-compressed performance gap on average.The experiments span two backbones, four pruning and merging methods, three expert-retention ratios, and 28 downstream benchmarks.Source: §1 · p. 2
  • Under a small general-domain adjustment budget, causal LM fine-tuning is more cost-effective than standard token-level teacher KD in this setting.LM fine-tuning improves observed next-token likelihood without teacher forward passes.Source: §1 · p. 2
  • Full-parameter causal LM adjustment is the strongest default among tested scopes under the measured GPU-cost protocol.It can be GPU-time competitive despite updating more parameters because it avoids expert-selection overhead and yields larger recovery per run.Source: §1 · p. 2

2 Related Works

Prior MoE compression work includes expert pruning and merging, and some studies add recovery procedures. The paper identifies a gap in systematic comparison of post-compression objectives, trainable scopes, and GPU-cost trade-offs under a tiny calibration budget.

2 Related Works

  • Expert pruning removes selected redundant experts using importance, reconstruction, routing, output, token-variation, trajectory, or layer-level criteria.Examples include reconstruction loss (Lu et al., 2024), differentiable selection (Bai et al., 2025), and router magnitudes (Lasby et al., 2025).Source: §2 · p. 2
  • Expert merging synthesizes experts using output-similarity clustering, selective dualmasks, or compression matrices.The cited approaches include Chen et al. (2025a), Zhao et al. (2025), and Miao et al. (2025).Source: §2 · p. 2
  • Existing recovery work does not systematically compare objectives, trainable scopes, and GPU-cost trade-offs across deployable expert-compressed checkpoints under a tiny calibration budget.Prior examples include expert-wise KD after pruning (Xie et al., 2024), condensed-layer fine-tuning, and LoRA-based recovery after hybrid compression.Source: §2 · pp. 2–3

3 Causes of Performance Degradation

Compression can alter both the router’s mixture weights and the selected expert functions or paths. The section analyzes these coupled residual sources and explains why router-only adjustment may not address the typical compressed path.

3 Causes of Performance Degradation

  • At a common reference input, the analysis separates residuals into shared-path router reweighting and active-path replacement.It defines T as shared active experts, D as experts active only in the original, and R as experts active only in the compressed model.Source: §3 · p. 3
  • When the original and compressed active expert sets match, the local residual can contain only shared-path router reweighting.This best case requires the original active set to be retained; with unchanged router and gate implementation, the same-input residual is zero.Source: §3 · p. 3
  • Router-only adjustment changes mixture weights on shared experts, but frozen expert parameters prevent it from generally reconstructing a missing expert contribution.That limitation does not apply when available replacement experts already provide an adequate functional approximation.Source: §3 · p. 3
  • The dominant partial-overlap scenario permits both router and expert/path contributions, motivating comparison of router-only, expert-inclusive, and full-parameter adjustment.Table 1’s frequency analysis does not determine which residual source is larger in norm.Source: §3 · p. 4, Table 1 · p. 4
  • Even when compression is limited to MoE experts, its local source enters the residual stream and is transformed by subsequent model modules.The propagation analysis assumes the same initial input for the original and compressed models and treats end-to-end state shifts separately.Source: §3 · pp. 3–4
  • Router-plus-top-k adjustment preserves router directions and adds only directions associated with selected trainable experts.These are first-order reachable-space inclusions, not evidence of optimization success or the parameter groups’ relative causal importance.Source: §3 · p. 4

4 Experiment Setups

The experiments compare post-compression adjustment objectives and trainable scopes under a matched small-data budget and measured GPU costs. They evaluate compressed checkpoints from two backbones across pruning and merging methods, retention ratios, and downstream benchmarks.

4 Experiment Setups

  • The setup asks whether small post-compression adjustment recovers performance and which strategy and objective are most efficient under a matched adjustment budget.The comparison spans compression paradigms, ratios, objectives, and trainable parameter scopes.Source: §4 · p. 4

4.1 Backbones and Compression Baselines

  • Two MoE backbones are paired with representative expert-pruning and expert-merging methods.The backbones are Qwen3-30B-A3B-Instruct-2507 and gemma-4-26B-A4B-it; the methods are REAP and HC-SMoE for Qwen3, and AIMER and M-SMoE for gemma.Source: §4.1 · p. 5
  • Each compression method is evaluated at 50%, 62.5%, and 75% expert-parameter retention, with non-expert parameters unchanged.In pruning, 50% retention means half the expert parameters are removed.Source: §4.1 · p. 5

4.2 Benchmarks and Evaluation Protocol

  • Models are evaluated on 28 downstream benchmarks spanning five categories, using harness-provided task metrics and greedy decoding.The categories are general reasoning/QA, mathematical reasoning, coding, chain-of-thought reasoning, and multiple-choice question answering; evaluations use temperature 0 and seed 42.Source: §4.2 · p. 5
  • Adjustment uses the same C4 subset and hyperparameters across conditions, with one epoch over 3,000 examples and batch size 2.Each run uses two NVIDIA H200 GPUs; benchmark evaluation is separate from the adjustment runs.Source: §4.2 · p. 5

4.3.1 Objectives

  • The objectives are causal LM loss on general-domain text and token-level KD from the original uncompressed model.LM loss improves observed-text likelihood without a teacher forward pass; DRLM is included separately as a diagnostic router-alignment baseline.Source: §4.3.1 · pp. 5–6

4.3.2 Trainable parameter scopes.

  • Trainable scopes range from router-only to full-parameter adjustment, with router-plus-top-k experts updating the most frequently activated experts selected using adjustment data.The top-k values are 8, 16, and 50; selecting experts adds calibration inference before training. FULL FT and FULL KD update all compressed-checkpoint parameters using LM and KD losses, respectively.Source: §4.3.2 · p. 6, §4.3.3 · p. 6

4.3.3 Adjustment strategies.

  • The adjustment strategies combine LM or KD objectives with trainable scopes, while DRLM is reported as a diagnostic baseline.The unadjusted compressed model is the baseline for recovery gain.Source: §4.3.3 · p. 6

5 Experiment Results

Across the tested compression settings, full-parameter LM fine-tuning recovers the most performance, while its cost and gains vary with adjustment scope and benchmark family. The results also show that adjustment beyond MoE modules and accounting for expert-selection overhead matter when comparing strategies.

Overall recovery

  • FULL FT gains +3.49 pp over the unadjusted compressed baseline, raising mean score from 39.01% to 42.50% and recovering 37.3% of the original-to-compressed gap.It leads ROUTER+ALL-EXPERT FT (+2.31 pp) and FULL KD (+2.19 pp).Source: §5 · p. 6, Figure 1 · p. 7

Adjustment objectives

  • Under matched trainable scopes, causal LM fine-tuning consistently outperforms standard token-level KD under the study’s tiny general-domain adjustment budget.The comparison includes KD’s additional teacher-forward cost, so it does not establish that distillation is generally ineffective.Source: §5 · p. 7

Router-only adjustment

  • Router-only FT and KD yield small gains of +0.23 pp and +0.17 pp, respectively, while direct router logit matching yields +0.06 pp.Router reweighting addresses only part of the error when expert-path and expert-function errors remain.Source: §5 · p. 7

GPU cost–performance trade-off

  • FULL FT has the highest recovery and best recovery per end-to-end run among high-recovery strategies, although it is not the cheapest in absolute GPU cost.Expert-subset strategies add selection inference while recovering less; Figure 2 compares recovery against GPU-hours, memory occupancy, and energy.Source: §5 · p. 7, Figure 2 · p. 7

Benchmark-family recovery

  • FULL FT yields the largest recovery on chain-of-thought and MCQA benchmarks, while math and coding show only marginal changes.Compression also produces generation failures such as repetition and premature termination; FULL FT restores fluent continuation without task-specific supervision, consistent with gains on general, CoT, and MCQA-style tasks.Source: Figure 3 · p. 8, §5 · p. 8

GPU cost–performance trade-off

  • Top-k expert strategies require a separate selection stage, increasing GPU cost while recovering less than ROUTER+ALL-EXPERT FT or FULL FT.Their cost efficiency therefore depends on identifying selected experts at negligible cost.Source: §5 · p. 8

Trainable parameter scope

  • For both FT and KD, recovery increases as the trainable scope expands from router-only through expert-inclusive adjustment to full-parameter adjustment.The large jump from ROUTER+ALL-EXPERT to FULL adjustment indicates that updating dense/shared parameters adds recovery beyond updating MoE modules alone.Source: Figure 4 · p. 8, §5 · pp. 8–9

6 Robustness Check

The main strategy ranking persists with fewer C4 examples and across two calibration domains. The cross-domain aggregate trend is strong, but does not imply invariant rankings in every individual setting.

Reduced calibration budget

  • With 1,024 C4 examples, Full FT remains the strongest strategy (+2.78 pp), followed by Router All-Expert FT (+1.86 pp) and Full KD (+1.77 pp).Causal LM fine-tuning also remains ahead of KD at each matched trainable scope.Source: §6 · p. 9

Calibration-domain robustness

  • Full FT, Full KD, and Router All-Expert FT form the same top-three set across the two calibration domains, but rankings are not invariant in every setting.The domain comparison uses six matched settings and 28 tasks; setting-level variation is reported separately.Source: §6 · p. 9, Figure 6 · p. 9
  • Full FT has the largest recovery averaged across C4 and OpenR1-Math calibration, at 4.04 pp versus 3.40 pp for Full KD.Across adjustment strategies, aggregate gains are strongly associated between domains (Spearman ρ = 0.973; Kendall τb = 0.897).Source: Figure 6 · p. 9, §6 · p. 9

7 Conclusion

The conclusion reframes retraining-free MoE compression as producing an initialization for adjustment, not a final artifact. It reports substantial recovery with a tiny adjustment budget and the strongest tested cost–recovery trade-off for causal LM fine-tuning, especially full-parameter adjustment.

7 Conclusion

  • Compressed MoE models recover a substantial portion of the original-to-compressed performance gap with a tiny general-domain adjustment budget.The conclusion characterizes the compressed checkpoint as an initialization rather than the final artifact.Source: §7 · p. 9
  • Causal LM fine-tuning, especially full-parameter adjustment, offers the strongest cost–recovery trade-off among the tested strategies.Source: §7 · p. 9
  • Retraining-free compression is a strong start, with recovery toward the original model substantially improved through a small post-compression adjustment stage.Source: §7 · p. 9

Limitations

The reported cost comparison is controlled but not system-optimized, and the study’s conclusions are bounded by its compression paradigms, model scale, and tiny general-domain adjustment regime.

Limitations

  • GPU-cost comparisons use a unified implementation and hardware, but omit system-level optimizations that could change the cost–recovery trade-off.The authors cite optimized serving and distributed-training systems as possible improvements but do not evaluate their effects.Source: Limitations · pp. 9–10
  • Expert Editing is not primary evidence because adjusting reconstructed checkpoints is not parameterization-equivalent to updating native factorized editing models.Source: Limitations · p. 10
  • Models above 100B parameters are unevaluated, and their routing, expert redundancy, and cost trade-offs may differ.Source: Limitations · p. 10
  • Conclusions apply to a small-budget, general-domain adjustment regime; task-specific, multilingual, or safety data and longer schedules may change the optimal objective or trainable scope.Source: Limitations · p. 10

Ethical Considerations

The paper reports its data and safety boundaries, implementation settings, model targets, adjustment procedures, and evaluation setup. The experiments use two instruction-tuned MoE backbones and compare adjustment objectives and trainable scopes under shared settings.

Ethical Considerations

  • The study uses no human-subject, private-user, or personally identifiable data; behavioral proxies are diagnostics, not safety certification, and adjustment is not a safety intervention.Deployment should follow original model and dataset restrictions and include safety evaluation for the intended domain.Source: Ethical Considerations · p. 10
  • The authors disclose Gemini assistance for debugging and writing polish, while stating that authors verified experimental designs, analyses, and scientific claims.Source: Ethical Considerations · p. 10

A Experiment Settings

  • Retention ratios are 50%, 62.5%, and 75%, corresponding to 64, 80, and 96 retained or physical routed experts per layer for pruning and merging.Source: A Experiment Settings · p. 15

A.1 Backbones and Compression Targets

  • The study uses Qwen3-30B-A3B-Instruct-2507 and gemma-4-26B-A4B-it; pruning and merging reduce routed expert capacity while preserving dense components.Qwen3 has 48 sparse MoE blocks and Gemma4 has 30 routed MoE layers, each with 128 routed experts per layer.Source: §A.1 · p. 15

A.2 Post-Compression Adjustment

  • Adjustment uses the first 3,000 nonempty C4 examples, truncated to 512 tokens, and compares causal LM loss with token-level KL distillation from the uncompressed backbone.Source: §A.2 · p. 15
  • Trainable scopes range from router-only to full-parameter adjustment; router-plus-top-k scopes select frequently activated physical experts, with k=8 in main experiments and k=16 or 50 in ablations.Source: §A.2 · pp. 15–16
  • Adjustment runs use two H200 GPUs and report effective GPU-hours, memory GiB-hours, and energy; monitored KD costs include both teacher and student passes.Source: §A.2 · p. 16

A.3 Evaluation with lm-evaluation-harness

  • Downstream evaluations use lm-evaluation-harness with vLLM; Gemma4 disables thinking across categories, while Qwen3 explicitly disables it for AIME.Source: §A.3 · p. 16

A.4 Benchmarks

This section specifies the benchmark and diagnostic setup, develops residual decompositions for pruning and merging, and reports adjustment results by compression paradigm. For Expert Editing, it presents reconstructed-checkpoint adjustment as an unconstrained materialized-weight-space test and treats its results as supplementary evidence about the edited function, not native factorized optimization.

A.4 Benchmarks; A.5 Direct router logit matching; B Original MoE (before Compression)

  • Evaluation spans 28 downstream benchmarks across reasoning, coding, chain-of-thought, and multiple-choice QA; DRLM is a diagnostic router-alignment baseline when compression provides a router-coordinate mapping.The appendices use a common local hidden-state input to isolate local compression effects, with end-to-end input shifts handled separately.Source: §A.4 · pp. 16–17, §A.5 · p. 17, B Original MoE (before Compression) · p. 19

C Expert Pruning (N →N −α)

  • With pure pruning and an unchanged router, preserving the original active set makes the same-input residual exactly zero, although upstream hidden-state shifts can still alter deeper end-to-end paths.With partial active-set overlap, both residual components may contribute; router-only adjustment can change shared-path weights or the top-k boundary but cannot directly change expert functions.Source: C Expert Pruning (N →N −α) · p. 20

D Expert Merging (N →M, where M < N)

  • For merging, complete projected-path coverage leaves router reweighting and merged-function approximation as residual sources, while insufficient physical routing capacity can make part of the original path unrepresentable.Router-only adjustment may alter selected clusters and their gate masses but cannot directly make merged functions equal the input-dependent original cluster outputs; expert-inclusive adjustment can change merged functions.Source: D Expert Merging (N →M, where M < N) · p. 21

E Experiment Results by Compression Paradigm

E.1 Expert Pruning

  • For pruning, FULL FT is especially effective on CoT and MCQA, while math and coding decrease slightly; recovery rises with scope, with LM fine-tuning stronger than KD at each matched scope.Router-only adjustment is a low-cost but low-recovery option and cannot fully repair pruning-induced degradation through router reweighting alone.Source: §E.1 · p. 22, Figure 9 · p. 23, Figure 10 · p. 23

E.2 Expert Merging

  • For merging, recovery gains exceed the pruning-only breakdown; router+top-k does not dominate router+all-experts or full adjustment when recovery and expert-selection overhead are considered together.FULL FT recovers strongly on CoT, MCQA, general, and math benchmarks, while coding is essentially unchanged; recovery increases with scope, including a large final jump to full adjustment.Source: §E.2 · p. 23, Figure 13 · p. 24, Figure 14 · p. 24

F.1 Compatibility-Mode Analysis for Expert Editing

  • Compatibility-mode Expert Editing tests adjustment of reconstructed standard-MoE checkpoints, not parameterization-equivalent expert-inclusive or full adjustment of native factorized models.The reconstructed checkpoint matches the native edited model function at initialization, and router-only adjustment is equivalent up to numerical differences; expert-parameter updates are not generally equivalent.Source: §F.1 · p. 25
  • Native and direct materialized updates are equivalent only under a restrictive fixed-point condition; expert-inclusive and full reconstructed-checkpoint updates are unconstrained and need not match native factorized optimization.Shared bases or tensor factors can couple native expert updates, while direct reconstructed-weight adjustment can move materialized expert matrices more freely.Source: §F.1 · p. 26
  • Small post-compression adjustment improves reconstructed Expert Editing checkpoints; causal LM fine-tuning tends to outperform teacher KD, router-only gains are limited but nonzero, and recovery increases with trainable scope.Expert-inclusive and full updates do not preserve the native factorized constraint, so these trends are supplementary evidence about the materialized edited function rather than deployable native artifacts.Source: §F.1 · p. 26

F.2 Active-Set Formulation for Expert Editing

  • The active-set formulation preserves the routed-expert index set while replacing expert functions, and organizes the local residual into routing, expert-function, and active-path components.It compares original and edited layer maps at a common local hidden-state input; end-to-end state shifts are handled separately by propagation analysis.Source: §F.2 · p. 26

F.3 Best Scenario: Preserved Active Path

  • Preserving the selected expert indices does not guarantee zero residual: shared-index router reweighting and expert-function approximation can remain, though unchanged routing and a common input eliminate the routing term.Without further assumptions, the formulation implies no ordering between the remaining terms.Source: §F.3 · p. 27

F.4 Most Common Scenario: Partial Active-Set Overlap

  • With partial active-set overlap, all three residual terms may be nonzero, separating shared-index routing disagreement, shared-expert function mismatch, and active-path replacement.The path term compares original contributions that leave the active path with newly activated edited-expert contributions.Source: §F.4 · p. 27

F.5 Worst Scenario: Disjoint Active Paths

  • Disjoint active paths constitute complete active-path replacement, but the “worst” label describes the scenario and does not establish maximal residual norm for every input.Router-only adjustment can change routing and active-set membership but cannot alter expert functions; expert-inclusive and full adjustment can alter them in unconstrained materialized weight space.Source: §F.5 · p. 27, §F.6 · p. 27

F.7 Expert Editing Experiment Results

In compatibility-mode Expert Editing, post-compression adjustment improves reconstructed checkpoints, with FULL FT producing the largest recovery. Qualitative examples show more fluent output after adjustment, while the results remain supplementary because they use reconstructed standard-MoE weights.

F.7 Expert Editing Experiment Results

  • FULL FT yields the largest recovery; LM fine-tuning beats KD at expert-inclusive and full scopes, while router-only KD exceeds FT (+0.36 vs +0.25 pp).Router-only adjustment gives limited but nonzero gains, and recovery increases as the trainable scope expands.Source: §F.7 · p. 27, Figure 15 · p. 28
  • These compatibility-mode results use reconstructed standard-MoE weights, not native factorized Expert Editing parameters, so they are supplementary rather than primary compressed-artifact evidence.Source: §F.7 · p. 28

G Qualitative Examples of Generation-Level Recovery

  • The qualitative examples contrast degenerate compressed generations, including repetition and broken fragments, with more fluent, coherent explanations after FULL FT.The examples are illustrative rather than a comprehensive evaluation of generation quality.Source: G Qualitative Examples of… · p. 28

After FULL FT

H Related Works

  • The paper situates parameter-level MoE compression within broader efforts to reduce deployment costs for individual models and multi-agent systems.Source: H Related Works · p. 30

H.1 Compression Operators and Their Consequences

  • Pruning removes experts, merging synthesizes a smaller expert set, and editing changes expert functions while preserving routing slots, potentially disrupting router alignment.Source: §H.1 · p. 30

H.2 Post-Compression Adjustment and Recovery Training

  • Prior recovery studies are often tied to one compressor, objective, or trainable scope; this paper compares objectives and scopes under matched small-data and GPU-cost measurements.Source: §H.2 · p. 31

I Additional Calibration Robustness

The calibration robustness checks preserve the leading strategy pattern under a smaller C4 budget and show strong aggregate association between C4 and math-domain calibration. However, setting-level rankings vary across the domain-shift comparison.

Smaller calibration budget

  • With 1,024 C4 examples, Full FT gives the largest mean recovery (+2.78 pp), followed by Router All-Expert FT (+1.86 pp) and Full KD (+1.77 pp).FT remains ahead of KD at every matched trainable scope across 12 settings and 28 evaluation benchmarks.Source: I Additional Calibration Robustness · p. 31, Table 5 · p. 32

Calibration-domain shift

  • The domain comparison uses 1,024 examples per corpus, holds the unadjusted checkpoints and 28-task evaluation suite fixed, and covers six matched settings.Source: I Additional Calibration Robustness · pp. 31–32
  • Across C4 and OpenR1-Math-220k, strategy-level ordering is strongly associated (Spearman ρ = 0.973; Kendall τb = 0.897), and Full FT has the largest two-domain mean recovery (4.04 pp).Setting-level Spearman correlations average 0.672 and range from 0.071 to 0.962, so rankings are not invariant for every checkpoint.Source: I Additional Calibration Robustness · p. 32

J Behavioral Proxies

Behavioral proxies show compression as the larger aggregate source of change on IFEval, TruthfulQA, and ToxiGen, while adjustment shifts other proxy means toward or beyond original levels. These proxies do not establish a causal mechanism or show that adjustment improves safety.

Compression and subsequent adjustment

  • Compression changed IFEval accuracy from 0.853 to 0.761, TruthfulQA MC1 from 0.421 to 0.365, and ToxiGen classification accuracy from 0.812 to 0.708.Across 13 adjusted strategies, additional changes relative to Compressed Only averaged −0.47 pp, −0.24 pp, and −0.23 pp, respectively.Source: J Behavioral Proxies · p. 32, Table 7 · p. 34

Restoration has a dual character

  • Adjustment moves strategy-mean WMDP, CrowS-Pairs, and WinoGender scores toward or beyond Original levels, but may also recover hazardous knowledge or stereotype-related associations.The 13-strategy means shift from 0.437 to 0.450 on WMDP, 0.556 to 0.565 on CrowS-Pairs, and 0.567 to 0.585 on WinoGender.Source: J Behavioral Proxies · p. 33, Table 7 · p. 34
  • These behavioral proxies neither establish the underlying causal mechanism nor show that post-compression adjustment improves safety.Source: J Behavioral Proxies · p. 33

Implications

  • Post-compression adjustment is designed for efficiency and capability recovery, so adjusted models still require separate, application-specific safety and alignment evaluation.The paper identifies safety-aware adjustment objectives as future work.Source: J Behavioral Proxies · p. 33

Related papers

Loading 2609.06076v2…