PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-tuning Done Right in Model Editing

Wanli Yang, Rui Tang, Hongyu Zang, Du Su, Qi Cao, Jingang Wang, Huawei Shen, Xueqi Cheng, Fei Sun

arXiv:2509.22072v4cs.CL

TL;DR

Direct fine-tuning has been treated as a weak baseline for model editing, while effective tuning locations remain underexplored. This paper restores breadth-first mini-batch fine-tuning and localizes updates; LocFT-BF exceeds the best baselines by an average of 33.72% in editing success rate while maintaining general capabilities.

  • Problem

    Direct fine-tuning is treated as a weak editing baseline, while evidence on which LLM layers and modules to tune remains limited.

    Source: §1 · pp. 1–2

  • Method

    LocFT-BF combines a breadth-first pipeline, mini-batch gradient aggregation, and localized parameter updates informed by analysis of tuning locations.

    Source: §1 · p. 2

  • Results

    33.72% average improvement in editing success rate over the best baselines, while LocFT-BF consistently maintains general capabilities of edited models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings establish fine-tuning as a powerful and practical technique for model editing, rather than a weak baseline.

    Source: §7 · p. 10

    LocFT-BF can introduce false or harmful information when modifying LLM knowledge, so the authors urge responsible use and caution.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Fine-tuning, a foundational method for adapting large language models, has long been considered ineffective for model editing. Here, we challenge this belief, arguing that the reported failure arises not from the inherent limitation of fine-tuning itself, but from adapting it to the sequential nature of the editing task, a single-pass depth-first pipeline that optimizes each sample to convergence before moving on. While intuitive, this depth-first pipeline coupled with sample-wise updating over-optimizes each edit and induces interference across edits. Our controlled experiments reveal that simply restoring fine-tuning to the standard breadth-first (i.e., epoch-based) pipeline with mini-batch optimization substantially improves its effectiveness for model editing. Moreover, fine-tuning in editing also suffers from suboptimal tuning parameter locations inherited from prior methods. Through systematic analysis of tuning locations, we derive LocFT-BF, a simple and effective localized editing method built on the restored fine-tuning framework. Extensive experiments across diverse LLMs and datasets demonstrate that LocFT-BF outperforms state-of-the-art methods by large margins. Notably, to our knowledge, it is the first to sustain 100K edits and 72B-parameter models,10 x beyond prior practice, without sacrificing general capabilities. By clarifying a long-standing misconception and introducing a principled localized tuning strategy, we advance fine-tuning from an underestimated baseline to a leading method for model editing, establishing a solid foundation for future research.

1 INTRODUCTION

The paper argues that fine-tuning’s poor reputation in model editing stems from depth-first, sample-wise adaptation rather than an inherent limitation. Restoring breadth-first mini-batch training and choosing effective tuning locations yields a simple method with strong editing performance and preserved general capabilities.

  • Restoring the breadth-first pipeline mitigates forgetting, while mini-batch updates reduce degradation of general capabilities caused by sample-wise optimization.The controlled experiments change pipeline and update granularity separately; the depth-first pipeline’s single pass lets later edits overwrite earlier ones, and batch-size-1 updates produce high-variance gradients.Source: §1 · p. 2
  • Tuning later-layer down- or up-projection matrices often achieves near-perfect editing success while preserving general capabilities, even when the configuration is not optimal.Optimal layer and module configurations can vary across models.Source: §1 · p. 2
  • LocFT-BF combines breadth-first training, mini-batch gradient aggregation, and localized updates without the precomputation, extra labeled data, or architectural changes required by other method families.The paper describes the method as easy to implement, efficient to run, and broadly applicable across architectures.Source: §1 · p. 2
  • LocFT-BF exceeds the best baselines by an average of 33.72% in editing success rate while maintaining general capabilities, and sustains 100K edits from 7B to 72B models.The paper identifies it as the first method to sustain 100K sequential edits while preserving general capabilities.Source: §1 · p. 2

2 IMPLEMENTATION MATTERS IN FINE-TUNING

The section argues that fine-tuning’s reported weakness in model editing stems from a mis-specified depth-first, sample-wise pipeline, not an inherent limitation; breadth-first epoch training with mini-batches makes it competitive.

2.2 MIS-SPECIFIED IMPLEMENTATION

  • DF trains each sample to convergence before moving on, risking overwriting earlier edits and unstable gradients; BF instead traverses epochs with mini-batch updates.The paper describes BF as iterating over the dataset across epochs, unlike DF’s sequential, single-pass procedure.Source: §2.2 · p. 3

2.3 IMPACT OF TRAINING PIPELINE

  • Replacing DF with BF improves performance across methods, especially FT-M and AdaLoRA, while FT-L and RoseLoRA show only minor gains.The authors attribute the smaller gains to FT-L optimizing only the target answer’s last token and RoseLoRA limiting trainable parameters.Source: Table 1 · p. 4, §2.3 · p. 4
  • With FT-M on 1,000 ZsRE samples, DF success on earlier shards falls as later shards are edited, whereas BF lifts all shards jointly to near-perfect success.The shard evaluations show DF overwriting earlier edits, while BF improves the shards together across epochs.Source: Figure 3 · p. 4, §2.3 · p. 4

2.4 IMPACT OF GRADIENT AGGREGATION

  • Under BF, larger mini-batches substantially improve FT-M capability enough to surpass AdaLoRA, while reducing generalization, more noticeably for AdaLoRA.AdaLoRA gains less because its low-rank design already regularizes updates; the authors suggest data augmentation to mitigate the generalization loss when each edit uses one QA pair.Source: Figure 4 · p. 5, §2.4 · p. 5

2.5 BREADTH-FIRST PIPELINE FOR SEQUENTIAL EDITING

  • For sequential arrivals, BF can retrain after each edit or buffer requests into small batches; the authors conclude ill-suited prior implementations, not fine-tuning itself, drove reported failures.They describe both updating strategies as feasible for real-world sequential editing.Source: §2.5 · p. 5

3 TAILORING FINE-TUNING FOR MODEL EDITING

The section argues that fine-tuning can be effective for model editing when paired with systematic tuning-location selection, and introduces LocFT-BF as a method combining location choice with breadth-first training and mini-batch optimization.

  • The study independently tested five modules per layer across three LLMs, using 1,000 ZsRE edits and evaluating reliability, generalization, and capability.Full-parameter and multi-layer tuning were excluded after poor performance and disruption of general capabilities.Source: §3 · p. 5
  • Across all models, nearly any layer or module achieved near-perfect reliability, but MLPdown was most stable while whole-layer, attention, and MLP tuning sometimes dipped.The occasional drops occurred at certain layers.Source: Figure 5 · p. 6, §3 · p. 6
  • Generalization favored coarse whole-layer and full-MLP modules over MLPup and MLPdown, peaking in layers 3–8, declining mid-network, then showing a smaller peak around layers 20–25.This pattern was consistent across the three LLMs studied.Source: Figure 5 · p. 6, §3 · p. 6
  • Capability was stronger and more stable with fine-grained MLPup and MLPdown modules and favored later layers, unlike generalization, revealing a location-dependent trade-off.The paper attributes the preservation of general capability to less disruptive localized updates.Source: Figure 5 · p. 6, §3 · p. 6
  • Because reliability is high across most locations, the study prioritizes capability and finds later-layer MLPdown minimizes capability degradation while retaining near-perfect reliability and acceptable generalization.This selection reflects the stated priority of preserving general abilities, while generalization may be improved with data augmentation.Source: §3 · p. 6
  • The later-layer strategy remains effective when not optimal, including for Qwen, and Appendix A.2 reports transfer to MoE models and unseen QAEdit.Source: §3 · p. 7
  • LocFT-BF combines breadth-first training, mini-batch gradient aggregation, and tuning-location selection in a plug-and-play method that avoids locate-then-edit precomputation and meta-learning overhead.The paper describes its location-selection strategy as an initial, relatively stable way to address the need to specify tuning locations.Source: §3 · p. 7

4 BENCHMARKING LOCFT-BF IN LIFELONG EDITING

Across lifelong editing benchmarks, LocFT-BF shows strong overall performance, with the highest reliability and the strongest stability across evaluated models and datasets.

4.1 EXPERIMENTAL SETUP

  • The evaluation compares six editing methods across three open-source LLMs and four datasets, measuring reliability, generalization, general capability, and average edit time using WILD for the first two metrics.The LLMs are LLaMA-3-8B-Instruct, Mistral-7B-v0.1, and Qwen2.5-7B-Instruct; the datasets are ZsRE, COUNTERFACT, WikiBigEdit, and MedEditBench. WILD uses autoregressive decoding to avoid overestimating reliability and generalization.Source: §4.1 · p. 7

4.2 RESULTS & ANALYSIS

  • LocFT-BF achieves the highest reliability on all nine dataset–LLM combinations, gaining 33.72% on average and up to 58.50% over the second-best method.The authors conclude that LocFT-BF performs strongly across all key evaluation dimensions while maintaining stability.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • LocFT-BF has optimal generalization in six of nine combinations; irrelevant-text prefixes make COUNTERFACT especially challenging, but data augmentation lets it surpass MEMIT and AlphaEdit.AlphaEdit and MEMIT have a relative advantage on COUNTERFACT because they explicitly enhance robustness to this prompt noise.Source: §4.2 · pp. 7–8
  • LocFT-BF maintains high capability across all evaluated LLMs and datasets, while AlphaEdit fails to generalize beyond LLaMA3-8B and RLEdit and UltraEdit struggle on Mistral-7B.The authors attribute LocFT-BF’s stability to fine-tuning’s simple, general design, which adapts across architectures.Source: §4.2 · p. 8, Table 3 · p. 8
  • LocFT-BF, RLEdit, and UltraEdit complete each edit within one second, while other baselines are nearly 50 times slower due to added computational procedures.The cited overhead sources include auxiliary-module optimization for WISE and additional matrix calculations for MEMIT, RECT, and AlphaEdit.Source: §4.2 · p. 8, Table 3 · p. 8

5 SCALING TOWARDS REAL-WORLD EDITING

LocFT-BF is evaluated at up to 100K edits and on models as large as 72B, with reported results showing effective knowledge updates and retained general capabilities.

5.2 SCALING TO LARGER MODELS

  • LocFT-BF extends from Qwen2.5-7B to 72B with reliable updates and retained general capabilities, and either heuristic tuning-position strategy avoids exhaustive search.Default Position performs strongly on Qwen2.5-14B, while Proportional Position yields better results on Qwen2.5-32B.Source: Table 4 · p. 9, §5.2 · p. 10

5.1 SCALING TO LARGER DATA VOLUMES

  • LocFT-BF outperforms leading baselines across all metrics at large edit volumes, maintaining near-perfect success and general capabilities at 100K edits while RLEdit and UltraEdit capabilities decline sharply beyond 20K.LocFT-BF generalization declines at this scale, but the authors note it could be improved with data augmentation; prior evaluations were typically limited to about 3,000 edits.Source: §5.1 · pp. 8–9, Figure 6 · p. 9

5.2 SCALING TO LARGER MODELS

  • For larger Qwen2.5 models, LocFT-BF uses either the same absolute Layer6.MLPdown position or a proportional-depth position, such as Layer19.MLPdown for 72B.The proportional strategy preserves relative layer depth; the example maps the 7th of 28 layers to the 20th of 80.Source: §5.2 · p. 9
  • 99.80% reliability and 57.60% robust generalization were achieved by LocFT-BF on Qwen2.5-72B edited with 100K ZsRE samples, retaining 62.54% of general capabilities versus 64.84% originally.The paper presents this as an industrial-scale scenario bridging data and model scaling.Source: §5.2 · p. 10

6 RELATED WORKS

Related work spans several model-editing paradigms and fine-tuning-specific mitigation strategies; this study instead attributes fine-tuning’s reported failures to flawed implementation and tuning locations.

Model Editing Methodologies

Fine-tuning Based Model Editing

  • Direct fine-tuning of located parameters suffers catastrophic forgetting, prompting parameter-efficient fine-tuning and data-augmentation approaches.FT-L (Meng et al., 2022) and FT-M (Zhang et al., 2024) tune located parameters; MELO (Yu et al., 2024) and RoseLoRA (Wang et al., 2024a) use low-rank adaptation, while Gangadhar & Stratos (2024) and Wei et al. (2024b) enrich editing data.Source: §6 · p. 10
  • The study argues that fine-tuning’s reported failure reflects flawed implementation, not unsuitability, and presents corrected fine-tuning with customized tuning locations.To the authors’ knowledge, this is the first study to revisit the failure of fine-tuning-based model editing.Source: §6 · p. 10

7 CONCLUSION

The paper re-examines fine-tuning for model editing, corrects flaws in prior implementations, and proposes LocFT-BF, a localized approach grounded in empirical analysis of tuning positions. Experiments report that it surpasses state-of-the-art methods and scales to 100K samples and models up to 72B parameters.

7 CONCLUSION

  • LocFT-BF significantly surpasses state-of-the-art editing methods and scales to 100K samples and models up to 72B parameters.The authors characterize it as a pure localized fine-tuning approach.Source: §7 · p. 10
  • The paper identifies and corrects critical flaws in prior fine-tuning implementations, challenging the view that fine-tuning is inherently weak for model editing.The authors describe this as a re-examination of a technique long regarded as a weak baseline.Source: §7 · p. 10
  • LocFT-BF builds on comprehensive empirical analysis of tuning positions to provide a pure localized fine-tuning approach.Source: §7 · p. 10

ETHICS STATEMENT

The paper reports that its publicly sourced, nonsensitive data pose no privacy concerns, while urging caution because model editing can introduce harmful information.

ETHICS STATEMENT

  • LocFT-BF is intended to correct misinformation and bias, but could introduce false or harmful information, so the authors urge responsible use and due caution.The paper states that it adheres to the ICLR Code of Ethics and uses only publicly available data without personally identifiable or sensitive information.Source: ETHICS STATEMENT · p. 11

A APPENDIX · A.1 DETAILED EXPERIMENTAL SETUP · A.1.1 EDITING METHODS

The appendix describes the compared editing methods and standardizes their implementations and target-answer formatting for a fair comparison. The methods differ in their update strategies, supervision, and approaches to limiting interference or side effects.

A.1.1 EDITING METHODS

  • FT-L (Meng et al., 2022) supervises only the final answer token, undermining editing success, whereas FT-M (Zhang et al., 2024) uses the full answer and improves editing success.FT-L tunes the MLP of a causally traced layer with l∞ regularization; FT-M applies cross-entropy over the answer while masking the prompt.Source: §A.1.1 · p. 16
  • AdaLoRA (Zhang et al., 2024) allocates LoRA parameters by importance, while RoseLoRA (Wang et al., 2024a) applies row- and column-wise sparsity to low-rank updates.Zhang et al. (2024) report competitive results after adapting AdaLoRA to editing; RoseLoRA selectively updates critical parameters to preserve irrelevant knowledge.Source: §A.1.1 · p. 16
  • WISE (Wang et al., 2024b) separates pretrained and edited knowledge into routed memories, while RLEdit (Li et al., 2025) trains a hypernetwork through sequence-level reinforcement learning.WISE uses a router to direct queries between main and side memories; RLEdit treats target edits and model state as the environment and editing losses as rewards.Source: §A.1.1 · p. 16
  • UltraEdit computes parameter shifts with lightweight linear algebra and uses lifelong normalization to adapt feature statistics while maintaining consistency over time.The framework is training-, subject-, and memory-free.Source: §A.1.1 · p. 16
  • MEMIT (Meng et al., 2023) extends ROME to multi-layer rank-one updates, while RECT (Gu et al., 2024) updates only the top-k% of weights ranked by relative change.MEMIT identifies relevant layers and modules through causal tracing and propagates edits across successive layers; RECT uses selective updates to suppress side effects while preserving editing performance.Source: §A.1.1 · p. 16
  • AlphaEdit projects parameter changes onto the null space of preserved knowledge to keep post-edit outputs unchanged on queries about that knowledge.The passage describes this preservation property as theoretically proven.Source: §A.1.1 · p. 16
  • For fair comparison, implementations retain official algorithmic configurations, use empirically tuned batch sizes, and append EOS tokens to target answers across methods.The paper notes that prior implementations typically omit EOS, which can cause irrelevant or incorrect generation after the answer, whereas standard fine-tuning practice requires it.Source: §A.1.1 · pp. 16–17

A.1.2 EDITED LLMS … A.6 SCALING BASELINE METHODS TO LARGER MODELS

The appendix tests LocFT-BF across new models, datasets, domains, and implementations, and examines its scalability and the causes of its COUNTERFACT generalization gap. These evaluations report robust performance, with prefix augmentation improving COUNTERFACT generalization and competing baselines degrading at 14B.

A.1.2 EDITED LLMS

A.1.3 EDITING DATASETS

A.1.4 REPRESENTATIVE TASKS

  • Capability evaluation replaces locality and uses sampled MMLU, Natural Questions, SST2, WMT16 de–en, and GSM8K test data to assess unrelated knowledge and abilities.MMLU uses 28,500 questions, Natural Questions 3,610 pairs, and GSM8K 1,319 problems.Source: §A.1.4 · pp. 17–18

A.2 APPLYING LOCFT-BF TO MOE MODELS

  • On 1,000 QAEdit samples, LocFT-BF’s default mid-to-late-layer MLP down-projection location generalizes to Qwen1.5-MoE-A2.7B with stable editing and preserved capabilities.The evaluation tests generalization to both an unseen dataset and a heterogeneous MoE architecture.Source: §A.2 · p. 18, Table 5 · p. 18

A.3 EVALUATION ON MEDICAL EDITING

  • On a 930-edit medical benchmark across three LLMs, LocFT-BF injects specialized knowledge while preserving general capability and significantly outperforming RLEdit and UltraEdit.Evaluation focuses on reliability and capability because the benchmark has no rephrased prompts.Source: §A.3 · p. 18, Table 6 · p. 18

A.4 INTERPRETING GENERALIZATION GAP ON COUNTERFACT

  • COUNTERFACT’s prepended irrelevant text makes generalization harder; five-prefix augmentation markedly improves LocFT-BF, placing it ahead of augmented MEMIT and AlphaEdit while preserving general capabilities.The authors attribute the gap to COUNTERFACT’s evaluation design rather than a limitation of fine-tuning.Source: §A.4 · pp. 18–19, Table 7 · p. 19, Figure 7 · p. 19

A.5 IMPLEMENTATION AND VALIDATION ON LLAMAFACTORY

  • LlamaFactory’s LocFT-BF implementation retains significant baseline outperformance and gains in generalization and efficiency; FullFT improves edit reliability and generalization but catastrophically degrades general capabilities.LocFT-BF shows marginal reliability and capability variance across frameworks, attributed to differences such as token-level versus sample-level loss averaging.Source: §A.5 · pp. 19–20, Table 8 · p. 20

A.6 SCALING BASELINE METHODS TO LARGER MODELS

  • At 14B, RLEdit and UltraEdit degrade substantially, while LocFT-BF scales more easily and maintains significantly better performance under default or proportional layer placement.The comparison uses 1,000 ZsRE samples on Qwen2.5-14B.Source: §A.6 · p. 20, Table 9 · p. 20
Loading 2509.22072v4…