PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Knowledge Forgetting and Refusal Learning in MLLM Unlearning

Haodong Chen, Shengtao Wen, Dong Liang, Lingbing Guo, Xiang Chen

arXiv:2610.14605v1cs.AIcs.CLcs.IRcs.LG

TL;DR

Multimodal unlearning evaluations often focus on whether target answers disappear, while suppression can lead to confabulation and refusal can leave target knowledge intact. FASD separates direct answer-span suppression from target-masked refusal distillation and retain-side supervision. Across three MLLMU-Bench backbones, it reports strong forgetting and refusal quality while preserving utility, with probes finding no evidence of concealed or recoverable knowledge under the evaluated conditions.

  • Problem

    Multimodal unlearning evaluations often focus on stopping target outputs, although suppression can induce hallucinations and refusal training can leave the underlying knowledge intact.

    Source: Introduction · p. 1

  • Method

    FASD separates direct suppression of ground-truth answer spans from target-masked on-policy refusal distillation using an ignorance-prompted frozen model copy, alongside retain-side distillation.

    Source: Introduction · p. 2, Methodology · p. 3

  • Results

    Across three MLLMU-Bench backbones, FASD achieves top-tier classification forgetting and refusal quality while preserving utility, and evaluated probes find no evidence of concealed or recoverable target knowledge.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Under the evaluated probes, FASD’s results support treating unlearning as both removal of target knowledge and alignment of responses with the model’s post-unlearning knowledge state.

    Source: Introduction · p. 2, Conclusion · p. 7

Abstract

from arXiv · show

Current evaluations of multimodal unlearning typically focus on whether a model stops outputting target information. This surface-level criterion, however, masks two critical failure modes. First, knowledge suppression techniques often induce confabulation, leading the model to hallucinate facts about the unlearned subject. Second, refusal training merely alters observable behavior without erasing the underlying knowledge; such models might decline open queries but still identify correct answers under multiple-choice probing. To overcome these limitations, we propose Forget-Aware Self-Distillation (FASD). This method is designed to achieve honest ignorance, a state where the model genuinely loses the target knowledge and learns to acknowledge its absence. FASD separates these goals into two complementary channels. The behavior channel distills appropriate refusals from an ignorance-prompted copy of the model itself. Crucially, target tokens are masked from the teacher distribution to avoid accidentally reteaching the forgotten concepts. Simultaneously, the knowledge channel directly suppresses ground-truth answer spans independently of the model's generated responses. Experiments demonstrate that FASD successfully unlearns target knowledge, prevents information leakage in open-ended tasks, and preserves performance on retained data. The resulting model not only forgets the specified information but accurately recognizes the boundaries of its own knowledge. Code and data will be released upon publication.

Introduction

Multimodal unlearning must remove target information while specifying how the model should respond afterward. FASD aims for honest ignorance by separating knowledge suppression from refusal learning and preserving utility.

  • Existing multimodal unlearning methods suppress target information but do not explicitly specify what the model should say after that knowledge is erased.Source: Introduction · p. 1
  • Suppression-based methods lower target-answer likelihood without specifying a replacement, which can produce plausible fabrications or degenerate text; refusal-based controls alter expressed behavior only.Source: Introduction · pp. 1–2
  • Honest ignorance requires lowering the target answer’s relative advantage without pushing it to the lowest rank, acknowledging unavailable information rather than fabricating, and answering retained or out-of-scope queries as before.Source: Introduction · p. 2
  • FASD uses a frozen model copy for target-masked on-policy refusal distillation and independently suppresses ground-truth answer spans, with a retain channel to preserve non-target utility.Source: Introduction · p. 2
  • FASD operationalizes honest ignorance through target-knowledge suppression, selective refusal, and utility preservation, and reports superior classification forgetting and refusal quality across three MLLMU-Bench backbones while preserving utility.Source: Introduction · p. 2

Related Work

Prior work studies multimodal unlearning and on-policy distillation, but the supplied passages distinguish those research strands rather than detailing a unified approach.

Machine Unlearning and Multimodal Unlearning

  • Multimodal unlearning spans methods for removing visual-textual concepts, benchmark evaluations, and approaches including localized updates, gradient ascent, activation intervention, and reinforcement learning.Source: Machine Unlearning and Multimodal… · p. 2

On-Policy Self-Distillation

  • On-policy self-distillation uses student-generated trajectories for teacher supervision and can condition the same model on privileged information unavailable to the student.Source: On-Policy Self-Distillation · p. 2

Preliminary

The preliminary formulation frames unlearning as removing a designated response from a forget set while preserving retained and general knowledge. It defines honest ignorance through three distinct desiderata.

Problem Definition

  • The formulation assigns designated target responses in a supervised forget set for removal while seeking to preserve utility on retained examples and general out-of-scope knowledge.Source: Problem Definition · p. 2
  • Low forget-set scores alone are underspecified because they conflate genuine forgetting with indiscriminate refusal, random guessing, or catastrophic degradation.Source: Problem Definition · p. 2
  • D1 requires forced-choice rankings that neither favor the target answer nor place it last, with no exploitable signal across option permutations.Source: Problem Definition · p. 3
  • D2 requires high context-appropriate abstention on forgotten knowledge and low abstention on retained knowledge, while D3 requires utility near the original level on retained and real-world out-of-scope data.Source: Problem Definition · p. 3
  • Honest ignorance combines suppression, selective refusal, and utility preservation, jointly optimized using a frozen copy of the original model as teacher and reference.Source: Problem Definition · p. 3

Methodology

FASD uses one frozen model copy in distinct conditioning roles to supervise suppression, refusal, and retention. Its objectives target answer spans directly, teach refusals on student-generated states, and preserve retained-side behavior.

Methodology

  • A frozen branch of the vanilla model serves as teacher and reference alongside a trainable student updated through a lightweight unlearning adapter.Source: Methodology · p. 3

Target-Knowledge Suppression

  • Target-knowledge suppression penalizes only designated answer-bearing spans under ground-truth prefixes, leaving surrounding linguistic structure untouched and remaining active regardless of student output.The fixed reference anchors the objective and mitigates probability drift.Source: Target-Knowledge Suppression · pp. 3–4

Selective Refusal

  • For refusal learning, the student samples its own rollouts and an unknown-prompted frozen teacher scores the visited states; target tokens are masked to prevent restoring forgotten answers.Source: Selective Refusal · p. 4
  • The prefix-dependent blocked set allows continuation tokens only when the generated prefix matches a target alias, avoiding suppression of unrelated vocabulary through subword collisions.Source: Selective Refusal · p. 4

Utility Preservation

  • A retain-side objective reconditions the teacher with the answer prompt and leaves its distribution unmasked, providing dense supervision for factual content, response format, and linguistic quality.Source: Utility Preservation · p. 4

Joint Training Objective

  • The three signals are optimized in parallel from the same initialization, with student-only gradient updates; jointly optimizing the forget-side signals prevents refusal from concealing residual knowledge and suppression from degenerating into unsupported generation.Source: Joint Training Objective · p. 4

Experiments

Across the evaluated backbones and settings, FASD combines strong target forgetting with selective refusal and retained utility. Forced-choice, component, adversarial, and forget–utility analyses further examine whether forgetting is genuine and how the method behaves across conditions.

Experimental Setup

  • Experiments use MLLMU-Bench splits, optimize on Forget- and Retain-set VQA, and compare three backbones with established unlearning baselines.Evaluation includes ACC, ROUGE-L, Contextual Refusal, and Forgetfulness; ASRU results are published rather than locally evaluated.Source: Experimental Setup · pp. 4–5

For Q1: Does FASD Achieve Effective Target Erasure While Preserving Utility?

For Q2: Does Unlearning Induce Selective Refusal Rather Than Hallucination?

  • FASD attains the highest reported Contextual Refusal and Forgetfulness scores across all three backbones, supporting selective abstention rather than fabricated or disclosed answers.These metrics assess coherent acknowledgment of unavailable information and avoidance of target-answer disclosure.Source: Table 1 · p. 5, For Q2: Does Unlearning Induce… · p. 6

Q3: Is Target Erasure Genuine or Merely Masked by Refusal?

  • Refusal-free diagnostics find that FASD disperses target-answer preference, while Steering recovers the target at 45.83% accuracy when refusal is disallowed.FASD’s correct-option mean rank shifts from 1.47 in vanilla to 2.63; the reported probes find no statistically detectable recovery advantage over chance.Source: Q3: Is Target Erasure Genuine or… · p. 6

A1: Is the Ranking Advantage of the Target Eliminated or Reversed?

  • FASD spreads forgotten answers across all four ranks, with no advantage for best-of-four rank rules, least-likely guessing, or position-balanced probing.The four rank frequencies are 15.83%, 30.00%, 30.00%, and 24.17%; position-balanced accuracy is 27.71%, near the 25% chance level.Source: A1: Is the Ranking Advantage of the… · p. 6, Figure 4 · p. 6

A2: What Does Each Component Contribute?

  • Ablations indicate that knowledge suppression, behavior learning, target masking, and retain supervision contribute complementary roles to selective forgetting.Removing knowledge weakens forgetting despite strong refusal; removing behavior degrades contextual refusal, while removing retain supervision causes severe retained-utility collapse.Source: A2: What Does Each Component Contribute? · pp. 6–7, Table 3 · p. 7

A3: Does Target Suppression Withstand Adversarial Elicitation?

  • Across four tested attacks, open-ended leakage and classification accuracy remain at or below unattacked levels, including 1.67% accuracy under answer-prefilling.The protocol includes random prefixes, paraphrases, jailbreak prompts, and answer-prefill attacks.Source: Table 2 · p. 6, A3: Does Target Suppression Withstand… · p. 7

A4: How Does the Forget–Utility Trade-off Change with Forget-Set Size?

  • Across 5%, 10%, and 15% forget sets, FASD outperforms baselines on the forget–utility frontier for classification and generation.The figure’s horizontal axes measure forget-set performance reduction from vanilla, and its vertical axes report retain and real-set utility.Source: A4: How Does the Forget–Utility… · p. 7, Figure 5 · p. 7

Conclusion

The conclusion frames multimodal unlearning as requiring both knowledge removal and behavior aligned with the resulting knowledge state. FASD separates these objectives and reports strong forgetting and refusal quality while preserving utility across three benchmark backbones.

Conclusion

  • FASD combines target-span suppression, target-masked refusal distillation, and retain-side utility anchoring, with strong forgetting and refusal quality across three MLLMU-Bench backbones.Forced-choice, permutation, and adversarial probes find no evidence of concealed or recoverable knowledge under the evaluated conditions.Source: Conclusion · p. 7

A.1 Benchmark and Splits

The appendix describes the MLLMU-Bench data partitions, model initialization, and evaluation metrics used in the experiments.

A.1 Benchmark and Splits

  • MLLMU-Bench contains 500 fictitious profiles, 153 real-celebrity profiles, and 20,754 questions, split into Forget, Retain, Test, and Real Celebrity sets.Profiles pair facial images and biographies with customized multimodal and text-only questions; optimization uses VQA from Forget and Retain only.Source: §A.1 · p. 10

A.2 Models and Initialization

  • Experiments use Qwen3-VL-4B-Instruct, Qwen3-VL-8B-Instruct, and LLaVA-1.5-7B, with methods initialized from backbone-specific vanilla checkpoints.The paired initialization makes post-unlearning differences independent of different pretrained or benchmark-finetuned starting points.Source: §A.2 · p. 10

A.3 Standard Metrics

  • The evaluation reports ACC for multiple-choice classification and ROUGE-L for open-ended generation, with lower Forget and Test scores indicating stronger removal.Higher Retain and Real Celebrity scores indicate better utility preservation.Source: §A.3 · p. 10

A.4 Complete Forget–Utility Results

Table 5 presents the forget–utility measurements for the official 5%, 10%, and 15% forget settings, distinguishing forget-side differences from absolute post-unlearning utility.

A.4 Complete Forget–Utility Results

  • Table 5 covers forget–utility measurements at the official 5%, 10%, and 15% forget settings.The passage does not provide individual table values.Source: §A.4 · p. 10
  • Forget-side differences are calculated as the corresponding Vanilla score minus the post-unlearning score, so larger differences indicate stronger forgetting.Source: §A.4 · p. 10
  • Retain and real-set columns report absolute post-unlearning utility rather than forget-side differences.Source: §A.4 · p. 10

B.1 Judge-Based Response Evaluation

The evaluation combines judged open-ended responses with forced-choice and ranking diagnostics, then tests whether the findings withstand answer-position controls and altered elicitation contexts.

B.1 Judge-Based Response Evaluation

  • GPT-4o-mini is the primary judge because low lexical overlap cannot distinguish appropriate abstention from fluent unsupported substitutes.GPT-5.6-sol independently re-evaluates responses as a consistency check, without replacing or averaging the reported scores.Source: §B.1 · p. 10
  • Contextual Refusal scores appropriate acknowledgment of unavailable information, while Forgetfulness scores avoidance of explicit or implicit target leakage, both on direction-aligned 0–5 scales.The judge also assigns Naturalness as an auxiliary check; an incorrect, non-leaking response can score low on refusal and high on Forgetfulness.Source: §B.1 · p. 10

B.2 Forced-Choice Accuracy and Answer Margin

  • Forced-choice probing requires one A–D label, and the correct-versus-best-wrong margin indicates whether the target outranks every distractor.A positive margin means the correct target remains preferred to every distractor; strict refusal is an exact match to “I don’t know.”Source: §B.2 · p. 10

B.3 Rank and Option-Permutation Controls

  • Correct-answer ranks are compared with a uniform distribution because concentrating targets at Rank 4 would leave them recoverable by a least-likely rule.Four cyclic option permutations and position-offset removal control for answer-position bias; FASD remains close to four-way chance on forget items while retain rankings remain aligned with correct content.Source: §B.3 · pp. 10–11

B.4 Direction-Aligned Diagnostic Profile

  • The seven-axis profile places diagnostics on a 0–100 scale, with higher values indicating stronger evidence of appropriate forgetting or target suppression.Its normalizations avoid rewarding below-chance forced accuracy or systematic Rank-4 concentration as stronger forgetting.Source: §B.4 · pp. 11–12

B.5 Adversarial Elicitation

  • Adversarial elicitation varies context while holding the forget item fixed, testing unrelated prefixes, paraphrases, jailbreak instructions, and answer-oriented prefills.The study uses the epoch-5 FP32-merged checkpoint, fixed few-shot seed and demonstration, and greedy decoding; paraphrases are cached rewrites preserving question meaning and intent.Source: §B.5 · p. 12

C Baseline and Diagnostic-Control Details

The appendix details optimization baselines and inference-time controls, then specifies FASD’s teacher conditioning, target masking, span suppression, and joint training procedure.

C Baseline and Diagnostic-Control Details

  • Local optimization baselines include gradient ascent, retain-regularized variants, negative preference optimization, and forget-relevant parameter localization; ASRU results are quoted from its original paper.The local methods share backbone-specific vanilla checkpoints, forget/retain VQA data, and evaluation pipeline.Source: C Baseline and Diagnostic-Control… · p. 13

C.1 Inference-Time Diagnostic Controls

  • Inference-time controls include a prompt-only exact-refusal instruction and residual steering, testing whether strong refusal behavior alone matches parameter-level suppression’s target-accessibility profile.The steering direction uses paired answer and refusal instructions; the reported L35/S1 point uses final language layer 35 at strength 1.Source: §C.1 · p. 13

D.1 Conditioning Instructions

  • FASD uses unknown-information and answer-informed instructions to condition teacher distributions; these prompts are used only for teacher scoring, never at student inference.Source: §D.1 · pp. 13–14

D.2 Target-Span and Alias Construction

  • Target spans are formed by matching structured target attributes to ground-truth responses with word boundaries, and empty matches still contribute behavior learning but not span loss.Canonical target forms and available aliases are tokenized using the training chat template and whitespace convention.Source: §D.2 · p. 14

D.3 Prefix-Dependent Target Masking

  • Prefix-dependent masking blocks a continuation token only when the generated suffix is already a prefix of a target alias, then masks and renormalizes the teacher distribution.This avoids blocking continuation subwords outside an active target string.Source: §D.3 · p. 14

D.4 Span-Loss Gradient

  • The span-loss gradient lowers target-token probability while its coefficient decays smoothly as the target preference falls below the reference.The coefficient is bounded above by two, and the frozen reference receives no gradient.Source: §D.4 · p. 14

D.5 Joint Training Procedure

  • Joint training interleaves forget and retain instances at a 1:4 ratio, using masked unknown-conditioned supervision and target-span loss for forget items and unmasked answer-informed distillation for retain items.All three objectives update one LoRA adapter while the base model and teacher/reference branches remain frozen.Source: §D.5 · p. 14

D.6 Training Configuration

Table 11 identifies the configuration for the Qwen3-VL-8B experiment with a 5% forget setting.

D.6 Training Configuration

  • Table 11 presents the FASD configuration for the Qwen3-VL-8B 5% forget experiment.Source: Table 11 · p. 15
Loading 2610.14605v1…