PaperlayerView exact revision on arXiv

Source-linked AI summary

When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data

Zijie Liu, Jinhao Duan, Bingqi Shang, Xinming An, Sijia Liu, Tianlong Chen

arXiv:2609.06786v1cs.LG

TL;DR

Tabular LLM unlearning remains underexplored despite the use of tabular prediction in real-world decision pipelines. CAU adjusts retain constraints to address schema-induced overlap, and across sample- and feature-level tasks it more closely matches a retraining oracle while preserving utility and retain-region behavior.

  • Problem

    Unlearning for tabular data remains underexplored even as tabular prediction is common in real-world decision pipelines and records are adapted to language models.

    Source: §1 · p. 1

  • Method

    CAU identifies retained rows near the forget distribution in schema-aligned attribute space and relaxes their preservation constraints while keeping distant rows as strong anchors.

    Source: §1 · p. 2

  • Results

    CAU more closely matches a retraining oracle across sample- and feature-level unlearning, with clearer gains in high-overlap regimes and predictive utility and retain-region behavior maintained.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Reliable tabular LLM unlearning depends on constructing retain constraints with schema-induced forget–retain overlap in mind.

    Source: §1 · p. 2, §6 · p. 9

    CAU may be less beneficial when forget and retain samples are well separated or the schema lacks meaningful shared attribute-level evidence.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Machine unlearning aims to remove the influence of designated training data while preserving model utility, but its behavior on tabular data remains underexplored. This gap is important because tabular prediction is widely used in high-stakes domains and is increasingly adapted to language models through record serialization and schema-aware prompting. We identify a key challenge that distinguishes tabular unlearning from unlearning in free-form text or other modalities: schema-induced forget-retain overlap. In serialized tabular data, records share fixed column-name/value slots, similar attribute ranges, and common output spaces. Consequently, a forget row may have nearby retain rows that rely on the same high-signal attributes, causing retain preservation to oppose the update required for forgetting. Motivated by this failure mode, we propose Conflict-Aware Unlearning (CAU), a schema-aware approach that reduces forget-retain interference by relaxing preservation constraints on retained rows that most conflict with the forget set. Across sample-level and feature-level unlearning on clinical and non-medical tabular tasks, CAU more closely matches a retraining oracle while maintaining predictive utility and retain-region behavior. Our results show that reliable tabular LLM unlearning depends not only on the forgetting objective, but also on how retain constraints are constructed.

1 Introduction

Tabular LLM unlearning is underexplored despite its relevance to real-world decision pipelines. The paper identifies schema-induced overlap as a source of forget–retain interference and proposes CAU, which adjusts retain constraints to improve agreement with retraining.

1 Introduction

  • The underexplored setting matters because tabular prediction is common in real-world decision pipelines, and records are increasingly adapted to language models.The paper focuses on unlearning serialized records or attributes while preserving useful behavior on retained data.Source: §1 · pp. 1–2
  • Shared schemas, repeated column-value slots, and similar attribute patterns can place forget rows near retained rows, creating schema-induced overlap.The paper’s qualitative embedding diagnostic reports more structured local neighborhoods for serialized tabular inputs than for free-form text references.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Preserving nearby retained rows can constrain the same representations that forgetting needs to change, making retain preservation an active source of interference.The paper frames the question as how to unlearn records or attributes while preserving retain behavior under schema-induced overlap.Source: §1 · p. 2
  • CAU relaxes preservation constraints for high-conflict retained rows while keeping the forgetting objective and optimizer unchanged.Rows far from the forget distribution remain strong preservation anchors.Source: §1 · p. 2
  • CAU more closely matches a retraining oracle across sample- and feature-level unlearning, especially in high-overlap regimes where simpler retain-preserving methods may suffice less well.The evaluation also spans model sizes and tabular prediction tasks while maintaining predictive utility and retain-region behavior.Source: §1 · p. 2

2 Related Work

Related work covers machine-unlearning strategies, LLM adaptation to structured data, and the effects of data correlation on unlearning. This paper connects schema-driven representation regularities with forget–retain coupling in serialized tabular prediction.

2 Related Work

3 Motivating Analysis

Serialized tabular records can be locally close under shared schemas and rely on recurring attribute-level evidence. Together, these properties can make retain-preservation constraints oppose the updates needed for forgetting.

3 Motivating Analysis

  • Shared prompt schemas and similar values in high-signal columns can make records close in representation space and produce similar predictive behavior.The paper calls this pattern schema-induced overlap.Source: §3 · p. 3

3.1 Schema-Aligned Predictive Evidence

  • Serialized tabular predictions show strong saliency on recurring column names and values, suggesting nearby rows may depend on overlapping attribute-level evidence.Figure 2 also reports that small input-form perturbations can substantially alter predictions, indicating ties to recurring schema-level evidence.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 From Schema-Induced Overlap to Unlearning Conflict

  • When nearby retain rows rely on the same schema-aligned behavior as forget rows, preserving them equally can oppose forgetting and make the objective ill-conditioned.The paper frames this as a hypothesis for schema-structured tabular settings, not as a claim that nearby retained rows lack evaluation value.Source: §3.2 · p. 4

4 Method: Conflict-Aware Unlearning

CAU reduces forget–retain interference by changing retain constraints according to how close retained rows are to the forget distribution, while leaving the forgetting target unchanged.

4 Method: Conflict-Aware Unlearning

  • CAU adjusts retain-constraint strength by proximity to the forget distribution, downweighting or filtering high-conflict rows while keeping the forgetting target unchanged.The method addresses interference between forgetting and retain preservation without changing the optimizer.Source: §4 · p. 4

4.1 Schema-Aware Distance to the Forget Distribution

  • Forget-Normalized Distance (FND) measures each row’s closeness to the forget distribution in schema-aligned attribute space; smaller scores indicate potentially higher-conflict retain constraints.For all-numerical datasets, FND is the root-mean-square of forget-normalized z-scores across columns.Source: §4.1 · pp. 4–5

4.2 Conflict-Aware Retain Construction

  • CAU treats retain rows whose distance to the forget distribution is below a cutoff as high-conflict constraints, then controls their preservation through filtering or soft downweighting.λ = 0 reduces CAU to hard filtering, removing Ddrop from the retain-preservation loss; 0 < λ < 1 softly downweights high-conflict retain rows.Source: §4.2 · p. 5

4.3 Instantiating CAU in Retain-Preserving Unlearning

  • For sample-level unlearning, CAU targets all input tokens; for feature-level unlearning, it targets selected attribute-name and value spans while preserving full retained rows.CAU changes only the retain constraints; the forgetting loss and target set remain unchanged.Source: §4.3 · p. 5

5 Experiments

The experiments test sample- and feature-level tabular unlearning across overlap regimes, using a retain-trained model as the oracle. CAU more closely approximates that oracle, particularly in high-overlap settings, while maintaining predictive utility.

5 Experiments

  • Sample-level unlearning removes entire records, whereas feature-level unlearning suppresses selected attribute-level evidence spans rather than forgetting every record containing those columns.The latter targets selected attribute-name and attribute-value spans while seeking to preserve overall predictive utility.Source: §5 · p. 6
  • The experiments create high- and low-overlap forget sets by selecting k-means clusters with lower or higher separation from the remaining training data.The held-out test set stays fixed across regimes and methods.Source: §5 · p. 6

5.1 Experimental Setup

  • Evaluation spans QWEN3B and QWEN7B, two clinical tasks and Adult Income, with test accuracy for utility and loss-based MIA AUC for privacy behavior.Comparisons include general unlearning baselines and a retain oracle trained from scratch on retained data.Source: §5.1 · p. 6
  • Under strong overlap, baseline methods often deviate substantially from the retain-trained oracle and show inconsistent utility preservation across sample- and feature-level tasks.The reported instability can involve either over-forgetting with utility degradation or under-forgetting relative to the oracle, across model sizes and clinical and non-clinical tasks.Source: §5.1 · p. 6, Table 1 · p. 7, Figure 4 · p. 8
  • PCA visualizations on Diabetes show clearer separation between the forget set and remaining retain constraints after filtering at percentile 30.Filtering relaxes nearby constraints in both high- and low-overlap regimes.Source: Figure 3 · p. 7, §5.1 · p. 7
  • At sample level, CAU often achieves smaller |∆| from the retain-trained oracle while maintaining competitive test accuracy, especially under high overlap.In lower-overlap settings, simpler retain-preserving methods can sometimes match the oracle closely.Source: §5.1 · pp. 7–8
  • At feature level, CAU achieves reliable oracle agreement while avoiding large utility degradation in most settings.Table 1 reports Retinopathy and Diabetes results under high- and low-overlap regimes.Source: Table 1 · p. 7, §5.1 · p. 8
  • Compared with random or embedding-similarity retain selection, CAU more consistently reduces oracle deviation while maintaining comparable test accuracy.The reported benefit comes from selecting constraints to relax by schema-aware proximity, not merely reducing their number.Source: §5.1 · pp. 8–9, Figure 5 · p. 8

6 Conclusion

The paper studies tabular unlearning under shared-schema forget–retain overlap and proposes CAU, which adjusts retain constraints to reduce interference. Across sample- and feature-level tasks, CAU better matches a retraining oracle while preserving utility and retain-region behavior.

6 Conclusion

  • The study focuses on serialized tabular data, where shared schemas and similar attribute values can induce strong forget–retain overlap.Nearby retained samples may constrain representations that forgetting needs to change.Source: §6 · p. 9
  • Across sample- and feature-level unlearning on clinical and non-medical tasks, CAU better matches a retraining oracle while preserving utility and retain-region behavior.The paper identifies retain-constraint construction as a key component of reliable tabular LLM unlearning.Source: §6 · p. 9
  • CAU relaxes constraints on high-conflict retained rows while keeping distant rows as strong preservation anchors.It changes retain-constraint construction rather than the forgetting objective or optimizer.Source: §6 · p. 9

Limitations

The work evaluates serialized tabular prediction tasks with fixed schemas and measurable forget–retain overlap; CAU may be less beneficial outside those conditions.

Limitations

  • The study is scoped to serialized tabular prediction with fixed schemas and measurable forget–retain overlap.The authors identify broader schemas and regression tasks as future work.Source: Limitations · p. 9
  • CAU may be less beneficial when forget and retain samples are already well separated or shared attribute-level evidence is not meaningful.These are boundaries on when conflict-aware retain construction is expected to help.Source: Limitations · p. 9
  • Because CAU relaxes constraints on samples close to the forget distribution, retain-region evaluation is needed to check high-conflict retain behavior.The authors also propose studying combinations with additional unlearning objectives.Source: Limitations · p. 9

A.1 Additional Tabular Data Analysis

The diagnostics find greater perturbation sensitivity and more regular attention patterns in tabular prompts than in free-form text. These observations support the paper’s hypothesis that schema-structured prompts can produce evidence-concentrated behavior relevant to forget–retain coupling.

A.1 Additional Tabular Data Analysis

  • Across prompt variants, tabular inputs have substantially higher prediction-variance scale and spread than free-form text, suggesting greater sensitivity to localized input changes.The diagnostic measures each sample’s target-logit variance across multiple prompt variants.Source: §A.1 · p. 11
  • Concentrated use of schema-aligned evidence tokens may make small perturbations more consequential and increase the likelihood that unlearning updates affect nearby retained samples.The passage presents this as a possible mechanism consistent with greater forget–retain interference.Source: §A.1 · p. 11
  • Tabular inputs show substantially lower variance in attention entropy across samples, consistent with repeated attention to schema-defined token positions.Mean attention entropy is dataset-dependent; lower entropy indicates more concentrated attention mass.Source: §A.1 · p. 12
  • Tabular prompts show larger, heavier-tailed prediction variance and more consistent attention patterns than free-form text in the evaluated diagnostics.Together, these findings support the motivating hypothesis that structured prompts may increase forget–retain coupling and make stable unlearning more challenging.Source: §A.1 · pp. 11–12

A.1.1 Task Instruction

The instruction specifies a diabetes-outcome prediction task and requires a response containing only the class label.

A.1.1 Task Instruction

  • The prompt asks the model to predict diabetes outcome as 0 or 1 and respond with only that label.The stated labels are 0 for no diabetes and 1 for diabetes.Source: §A.1.1 · p. 12

A.1.2 Structured Tabular Format

The appendix specifies tabular examples, unlearning and evaluation protocols, and reproducibility details. It also situates the approach among prior unlearning methods and discloses the manuscript’s use of LLM editing assistance.

Sample S1

  • The two diabetes examples serialize patient attributes as named values followed by a binary outcome.The first example ends with output 0; the second ends with output 1.Source: Sample S1 · p. 12, Sample S2 · p. 12

A.2 Implementation and Evaluation Details

  • CAU is compared with RMU, NPO, SIMNPO, and GRADDIFF using shared data splits, base checkpoints, and evaluation pipelines.The protocol distinguishes sample-level deletion from feature-level suppression and constructs high- and low-overlap forget sets.Source: §A.2 · p. 12, §A.2.1 · p. 12

A.2.2 Feature-Level Unlearning Setup

  • Feature-level forgetting targets selected attribute name-and-value token spans, while retain losses use the full sequence and test utility uses unmasked inputs.Figure 6 reports prompt-perturbation logit variance and token-level attention entropy across text and tabular inputs.Source: §A.2.2 · pp. 12–13, Figure 6 · p. 13

A.2.3 Unlearning Objective and Hyperparameters

  • Unlearning methods use matched optimizers, learning rates, batch sizes, and update steps; representation-level loss is applied at layer ℓ to marked or full-sequence hidden states.Table 4 is identified as the shared optimization settings and key hyperparameters.Source: §A.2.3 · p. 13, Table 4 · p. 13

A.2.4 Membership Inference Attack Protocol

  • The membership attack scores examples by average per-token ground-truth output NLL, then reports ROC AUC using forget samples as members and disjoint held-out non-members.Member and non-member sampling sizes and the evaluation seed are fixed across methods.Source: §A.2.4 · p. 13

A.2.5 Reproducibility Notes

  • Experiments fix data splits, serialization templates, and feature-token rules, and share checkpoint selection, evaluation scripts, and MIA scoring across methods.Tables 5 and 6 are identified as Adult Income feature-level results for Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct, respectively.Source: §A.2.5 · pp. 13–14, Table 5 · p. 14, Table 6 · p. 14

A.4 Additional Related Work

  • Prior work spans efficient deletion and post-hoc or LLM unlearning, while geometry-aware methods manage forget–retain conflict through optimization rather than CAU’s retain-set filtering.The paper describes its data-level filtering as complementary to existing objectives such as RMU.Source: §A.4 · pp. 14–15

A.5 LLM Usage

  • LLM assistance was limited to editing and polishing; the author states it did not contribute to research ideas, methods, analysis, or experimental design.The author reports responsibility for and review of the edited manuscript content.Source: §A.5 · p. 15

Related papers

Loading 2609.06786v1…