PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning

Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong

arXiv:2609.05976v1cs.CLcs.AI

TL;DR

Existing multilingual unlearning evaluations measure cross-language transfer but do not determine whether forgetting follows the requested linguistic boundary. CLLPU benchmarks both universal and language-conditioned forgetting with matched multilingual knowledge probes, and experiments show opposite propagation failures across the two objectives.

  • Problem

    Existing multilingual evaluations focus on cross-language transfer and do not distinguish universal suppression from forgetting confined to a designated language.

    Source: §1 · p. 1

  • Method

    CLLPU benchmarks common-goal and language-conditioned forgetting using topic pairing, schema-matched knowledge units, and dual-anchor multilingual QA translation.

    Source: §3.2.1 · p. 3, §3.3.1 · p. 4

  • Results

    Across six methods, forgetting remains incomplete when universal suppression is required but spreads beyond the intended boundary when language-conditioned confinement is required.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Propagation control and preservation of relation-matched neighbor knowledge are central challenges for multilingual unlearning.

    Source: §1 · p. 2

    Language-conditioned forgetting is a benchmark-specific conditional-access objective, not a unique data-deletion or retraining oracle or a claim of complete knowledge removal.

    Source: §3.1.1 · p. 3

Abstract

from arXiv · show

Large Language Model (LLM) unlearning aims to suppress target knowledge while preserving general capabilities. In multilingual settings, unlearning must additionally propagate within its intended linguistic scope. However, existing evaluations mainly measure cross-lingual transfer and cannot distinguish insufficient from excessive propagation. We introduce CLLPU (Cross-Lingual and Language-Bound Protocol for LLM Unlearning), a multilingual benchmark that formulates this problem through two settings: common-goal forgetting, where target knowledge should be suppressed across all languages, and language-conditioned forgetting, where suppression should remain confined to a designated language. CLLPU combines goal-guided topic pairing, schema-aware relation matching, and dual-anchor multilingual translation to construct 800 matched knowledge-unit pairs and 72,000 QA instances across ten languages. Experiments with six representative methods on Llama-3.1-8B-Instruct reveal opposite failure modes: forgetting remains incomplete when universal suppression is required, yet spreads beyond the intended boundary when language-conditioned confinement is required. We further find that general multilingual utility can conceal damage to neighbor knowledge. These findings establish propagation control as a central challenge for multilingual LLM unlearning. We publicly release CLLPU together with its construction pipeline.

1 Introduction

Existing multilingual evaluations measure whether forgetting transfers across languages, but do not distinguish the universal suppression required by common-goal requests from the language-bound suppression required by language-conditioned requests. CLLPU defines both objectives and tests six methods, finding incomplete forgetting in the former and excessive propagation in the latter.

1 Introduction

  • Reliable unlearning evaluation must assess target-knowledge accessibility while preserving non-target capabilities.The paper situates this goal in practical and regulatory demands including GDPR erasure and CCPA deletion rights.Source: §1 · p. 1
  • Existing multilingual studies assess transfer across semantically equivalent prompts, but cannot establish whether forgetting stays within its intended boundary.Common-goal forgetting requires suppression across all languages; language-conditioned forgetting requires suppression only in the designated language context.Source: §1 · p. 1
  • CLLPU pairs target and neighbor topics, matches knowledge units by schema, and translates QA families using dual anchors to preserve their meaning across languages.The benchmark contains 800 knowledge-unit pairs and 72,000 multilingual QA instances.Source: §1 · pp. 1–2
  • Six methods show opposite propagation failures: forgetting remains incomplete when common-goal requests require universal suppression, but exceeds the intended boundary when language-conditioned requests require confinement.The paper also reports that general multilingual utility can remain near its original level while relation-matched neighbor knowledge is substantially damaged.Source: §1 · p. 2

2 Related Work

Prior unlearning benchmarks cover diverse targets and evaluation dimensions, and some examine preservation of neighboring knowledge. However, they generally do not test whether forgetting follows an intended linguistic scope or whether relation-matched knowledge is preserved.

2.1 LLM Unlearning Benchmarks

  • Existing benchmarks often assess broad utility or unmatched retain knowledge, leaving localized damage to semantically neighboring knowledge potentially unmeasured.Prior work also found syntactically similar neighboring queries vulnerable to degradation, while DUSK tests preservation of knowledge shared between forget and retain data.Source: §2.1 · p. 2

2.2 Multilingual LLM Unlearning

  • Multilingual studies characterize cross-language transfer, but do not explicitly distinguish common-goal from language-conditioned objectives or evaluate relation-matched preservation.Prior results describe cross-lingual forgetting as often incomplete, uneven, and asymmetric.Source: §2.2 · p. 2

3 Benchmark Construction

CLLPU constructs matched multilingual probes by pairing target topics with neighboring topics, aligning their factual units, and translating equivalent QA families across languages. Its design separates the intended forgetting boundary from nearby knowledge that should remain accessible.

3.1 Goal-Guided Topic Pair Construction

  • CLLPU constructs target–neighbor topic pairs so target knowledge lies within the intended forgetting boundary and semantically related neighbor knowledge lies outside it.For language-conditioned forgetting, the neighbor must also remain accessible in the source language, prioritizing boundary precision over broader topic coverage.Source: §3.1 · p. 2, §3.1.2 · p. 3, §3.1.1 · p. 3
  • Topics are selected for post-release, verifiable factual content, then paired within the same domain using comparable abstraction, temporal context, and fact types.Pairs must concern distinct entities or events with different factual answers; parent–child topics, variants of the same event, and superficial matches are excluded.Source: §3.1.1 · p. 2, §3.1.2 · p. 3

3.2 Topic-to-Knowledge-to-QA Transformation

  • Knowledge units encode atomic facts with evidence, and schema-aware matching pairs units with compatible relation types, semantic slots, answer types, and source evidence.Eligible pairs are ranked and selected one-to-one without reuse; unmatched valid units become holdouts, and selected units and matches receive manual review.Source: §3.2.1 · pp. 3–4

3.3 Parallel Multilingual QA Construction

  • Each matched unit becomes a core English QA plus three equivalent surface variants, which are translated into nine other languages using both the structured unit and English QA as anchors.Translations undergo blind back-translation and consistency checks, independent review against both anchors, and human audits of randomly sampled accepted QAs.Source: §3.3.1 · p. 4, §3.2.2 · p. 4, §3.3.2 · p. 4

4 Evaluation Protocol

The evaluation injects multilingual QA knowledge into a shared starting model, then applies unlearning separately for each source language and measures behavior across evaluation languages. It assesses target and matched-retain accessibility alongside membership inference and general multilingual utility.

4.1 LLM Unlearning Pipeline

  • The protocol injects forget and retain core QAs in ten languages into Llama-3.1-8B-Instruct, then unlearns separately for each source language using that language’s QAs.For common-goal forgetting, a retain-only retrained model provides a reference; the paper omits it for language-conditioned forgetting because its construction could conflict with the preservation objective.Source: §4.1 · p. 4

4.2 Multilingual Evaluation

  • The protocol measures answer accessibility with Exact Match, ROUGE-L, and Sentence Similarity, evaluates membership inference with four attacks, and averages zero-shot Belebele accuracy across languages.LLM-as-a-Judge is excluded from the main comparison because evaluator- and language-dependent calibration may confound cross-lingual effects.Source: §4.2.1 · p. 5, §4.2.2 · p. 5
  • Knowledge accessibility is scored by averaging core QAs and three surface variants across units, then macro-averaging diagonal source-language and off-diagonal cross-language results.Common-goal forgetting requires low cross-language forget accessibility; language-conditioned forgetting requires it to remain high, while matched-retain accessibility should remain high in both settings.Source: §4.2.1 · p. 5

5 Experiments

Across six methods, common-goal forgetting transfers across languages but remains incomplete, while language-conditioned forgetting spreads beyond its intended scope. The experiments also examine source-language effects, utility and privacy measures, surface variants, and forced response-language switching.

5.1 Experimental Settings

  • The evaluation uses Llama-3.1-8B-Instruct and six unlearning methods; core QAs are used for injection and unlearning, with variants reserved for robustness evaluation.The authors frame the target model as a task-specific validation setting, not evidence of model-invariant propagation patterns.Source: §5.1.1 · p. 5, §5.1.2 · p. 5

5.2 Overall Performance (RQ1)

  • Across six methods, common-goal forgetting transfers incompletely, while none establishes language-conditioned confinement; stronger forgetting can also damage retained knowledge.The common-goal and language-conditioned results are reported in Tables 1 and 2, respectively.Source: §5.2 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
  • GA’s common-goal Source EM of 0.0070 contrasts with Source SS of 0.3273; relation-matched answers can be semantically similar despite factual differences, so SS need not approach zero for a target fact to be inaccessible.Paired forget–retain answers already have BGE-M3 similarity of approximately 0.48–0.50.Source: §5.2 · p. 6

5.3 Cross-Lingual Transfer Boundaries (RQ2)

  • Cross-lingual effects vary by source language and direction; Arabic-source GA yields average cross accessibility of 0.221, while GD’s Bengali-to-English accessibility is 0.519 versus 0.009 in reverse.In language-conditioned NPO, Arabic retain accessibility is 0.596 in Arabic but 0.030 on average in other languages; GA also damages retain knowledge alongside forgetting.Source: §5.3 · p. 7

5.4 Other Metrics (RQ3)

  • The results show a membership-privacy–utility trade-off: GA has AUCs closest to 0.5 but Belebele accuracy of 0.5597, whereas GD has the highest accuracy, 0.7186, with AUCs farther from 0.5.In language-conditioned runs, even SimNPO’s AUCs are 0.3003–0.3869; GA’s Belebele accuracy of 0.7868 is close to Original’s 0.8086 despite damage to matched retain knowledge.Source: §5.4 · p. 7, Table 3 · p. 7

5.5 Core QAs and Surface Variants (RQ4)

  • Across methods and scopes, forget-set surface-variant gaps range from −3.7 to +1.6 percentage points, while retain Source gaps reach 20.7 points, so core-only evaluation may overestimate retained accessibility.Figure 4 reports mean ROUGE-L across three surface variants and gives variant-minus-core gaps.Source: §5.5 · p. 8, Figure 4 · p. 8

5.6 Forced Response-Language Switching(RQ5)

  • Forced response-language switching lowers average forget-set ROUGE-L accessibility from 0.0658 to 0.0237 in common-goal and from 0.0418 to 0.0198 in language-conditioned evaluation.The concurrent decrease in retain accessibility indicates that constrained generation partly explains the reduction; aligned-language evaluation is the more stringent test.Source: §5.6 · p. 8, Figure 5 · p. 8

6 Conclusion and Future Work

The paper frames multilingual unlearning as controlling how far forgetting propagates: some knowledge should be removed across languages, while other knowledge should be removed only within a designated language context.

6 Conclusion and Future Work

  • CLLPU evaluates knowledge suppression, cross-lingual propagation, and selective preservation under distinct multilingual forgetting objectives.The objectives are global removal and language-conditioned removal.Source: §6 · p. 8
  • Across six methods on Llama-3.1-8B-Instruct, forgetting remains incomplete when global removal is required but exceeds the intended boundary under language-conditioned removal.Source: §6 · p. 8
  • The authors identify explicit control of cross-lingual unlearning propagation as future work.Source: §6 · p. 8

A Benchmark Construction Details

The appendix directs readers to supplementary resources for benchmark construction.

A Benchmark Construction Details

A.1 Stage 1: Goal-Guided Topic Pair Construction

CLLPU constructs matched target–neighbor knowledge units and multilingual QA instances through topic pairing, relation-aware matching, and translation grounded in both the fact and its canonical question.

A.1 Stage 1: Goal-Guided Topic Pair Construction

  • Topic pairs use evidence-backed metadata and comparable domains, abstraction levels, and fact types; human review checks boundary separation and contextual plausibility.The benchmark excludes ambiguous impact boundaries or unsupported rationales, which are not treated as legal determinations or universal cultural judgments.Source: §A.1 · p. 10

A.2 Stage 2: Topic-to-Knowledge-to-QA Transformation

  • Knowledge units encode atomic facts, relations, semantic slots, answer types, and source evidence; matching requires compatible schemas and direct support before one-to-one selection.Coverage-aware selection and manual review yield 500 common-goal and 300 language-conditioned knowledge-unit pairs.Source: §A.2 · pp. 10–11
  • Unmatched units become disjoint holdouts for membership inference, while matched units receive a core QA and three meaning-preserving surface variants.Generated English QA families undergo model and human review for validity, answer consistency, evidence grounding, and leakage.Source: §A.2 · p. 11

A.3 Stage 3: Parallel Multilingual Translation

  • Dual-anchor translation preserves both the structured fact and canonical question intent across nine target languages, with blind back-translation and independent cross-model checks.A human audit confirmed that all 2,000 sampled translated QAs met the stated fidelity and fluency criteria.Source: §A.3 · pp. 11–12

B Evaluation Metrics

The evaluation aggregates reference-model results, membership-inference scores, and multilingual reading-comprehension accuracy using separate protocols.

B.1 Reference-Model Aggregation

  • For source-independent Original and Retrain models, Source and Cross macro-averages are identical under the stated aggregation protocol, unlike language-conditioned results.Source: §B.1 · p. 12

B.2 Membership-Inference Protocol

  • Membership attacks score forget core QAs as members and same-language holdouts as non-members, using likelihood-based, low-probability-token, or compression-calibrated signals.Raw ROC-AUC is macro-averaged across ten source-language runs; values below 0.5 indicate reversed ordering, not necessarily weaker leakage.Source: §B.2 · p. 12

B.3 General Multilingual Utility

  • Belebele utility is zero-shot multiple-choice accuracy on 900 examples per language, macro-averaged across evaluation languages and source-language runs.Higher accuracy indicates better preservation of general multilingual reading-comprehension ability.Source: §B.3 · p. 13

C.1 Experiment Settings

The evaluation combines shared checkpoint-selection criteria with multilingual accessibility analyses. Across heatmap and LLM-as-a-Judge results, methods struggle to control forgetting across language boundaries, while semantic similarity can complicate interpretation.

C.1 Experiment Settings

  • All methods and source languages use shared checkpoint-screening objectives: lower target and higher neighbor weighted ROUGE-L accessibility, without a separate held-out validation split.The paper notes that the fixed knowledge-unit composition and evaluation scope make an independent validation split difficult to introduce.Source: §C.1 · p. 15
  • 0.48–0.50 common-goal and 0.41–0.56 language-conditioned BGE-M3 similarities show paired forget- and retain-set answers can be semantically close, so SS may reflect related retain knowledge.SS measures semantic proximity rather than factual equivalence; Figure 10 shows the paired-answer similarities.Source: §C.1 · p. 15, Figure 10 · p. 15

C.2 More Heatmaps

  • Heatmaps place Source results on the diagonal and Cross results off-diagonal, with target forgetting and neighbor preservation in separate blocks.The common-goal and language-conditioned heatmaps report accessibility using EM and SS.Source: Figure 7 · p. 14, Figure 8 · p. 14, Figure 9 · p. 15
  • Across six methods, language-conditioned heatmaps rarely show the required low-diagonal, high-off-diagonal forgetting pattern, and transfer depends strongly on source language and direction.GA and GD generally forget more strongly but damage neighbor knowledge more; NPO-based methods preserve more utility but forget less consistently across languages.Source: §C.2 · p. 16

C.3 LLM-as-a-Judge Accessibility

  • In the English-source LLM-as-a-Judge diagnostic, no method forgets target knowledge in English while preserving it in other languages.GA and GD damage neighbor knowledge; NPO-based methods preserve more neighbor knowledge but forget less strongly.Source: §C.3 · p. 16

C.4 Per Topic Analyses

Across nine topic pairs, the language-conditioned analysis finds that none of the evaluated methods establishes the intended boundary: forgetting reaches non-source languages, and topic-level results vary substantially.

C.4 Per Topic Analyses

  • 0.045–0.108 Cross target accessibility, versus 0.774 for Original, shows that forgetting propagated beyond the source language.Table 6 reports mean ± standard deviation over nine topic pairs; source target accessibility was also substantially reduced.Source: §C.4 · p. 16
  • Preference-based methods preserved source retain accessibility at 0.829–0.882, but Cross utility decreased to 0.264–0.286, showing that source-language preservation did not generalize across languages.Source: §C.4 · p. 16
  • Large topic-level deviations—0.093 ± 0.131 for GD on Cross target accessibility and 0.098 ± 0.126 for SimNPO on Source target accessibility—reveal substantial heterogeneity across topics that aggregate results conceal.Source: §C.4 · p. 16

Related papers

Loading 2609.05976v1…