How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Ziwen Xu; Haiwen Hong; Linsong Yu; Benglei Cui; Longtao Huang; Hui Xue; Ningyu Zhang

Source-linked AI summary

How LoRA Remembers? A Parametric Memory Law for LLM Finetuning

Ziwen Xu, Haiwen Hong, Linsong Yu, Benglei Cui, Longtao Huang, Hui Xue, Ningyu Zhang

arXiv:2605.30260v1May 28, 2026cs.CLcs.AIcs.CVcs.LG

TL;DR

Exact parametric memory in LoRA lacks quantitative capacity limits and dynamics despite its importance for updating LLM knowledge. The paper probes this memory with LoRA, derives a scaling law and token-level threshold, and shows that MemFT improves exact-recall fidelity and parameter efficiency over standard SFT.

Problem
Existing work largely relies on qualitative downstream evaluations, leaving the capacity limits and dynamics of exact parametric memory insufficiently characterized.
Method
The paper uses LoRA as a controllable latent-space probe, formulates a power-law memory model, and guides training toward tokens below a probability threshold.
Results
Both MemFT variants consistently outperform standard SFT in fidelity and parameter efficiency, with MemFT-OT reaching 100.0% Acc at Llama-r9 and Qwen-r8.
Takeaways & Limitations
The findings identify a power-law capacity relationship and show that p > 0.5 is sufficient for verbatim recall under greedy decoding.
Takeaways & Limitations
The analysis is restricted to 8B-scale models, and the p = 0.5 transition remains unverified for stochastic decoding methods.

Abstract

from arXiv · show

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.

1 Introduction

The paper addresses the under-explored quantitative limits and dynamics of exact parametric memory in LLMs. It introduces the Parametric Memory Law and MemFT, guided by token-level memory-locking behavior.

Motivation: Static pretrained parameters make integrating updated facts, user preferences, and task-specific knowledge a key continual-learning problem.The introduction contrasts this need with models whose knowledge remains largely fixed after deployment.
Research Gap: Existing benchmarks conflate raw information memorization with downstream comprehension and instruction-following, leaving parametric memory’s quantitative capacity boundaries and dynamics under-explored.The paper therefore focuses on exact text reconstruction as a measure of verbatim retention.
Parametric Memory Law: The Parametric Memory Law establishes a power law quantifying exact memory capacity from effective parameters and sequence length.The study scans LoRA rank and memory-sequence configurations and finds stable power-law scaling in LoRA-induced loss reduction ΔL.
Dynamics Mechanism: Token-level competition can hide behind low average loss, while p > 0.5 is a sufficient condition for memory locking under greedy decoding.Lower probabilities are identified as a catalyst for cascade collapse.
MemFT Method: MemFT uses threshold-guided optimization to target stubborn sub-threshold tokens, surpassing standard SFT in memory fidelity and parameter efficiency.Its training strategy reallocates the optimization budget toward tokens below the identified threshold.

2 Preliminary

The paper defines exact parametric memorization as writing answer information into parameter updates, with questions serving only as keys. It evaluates this process using answer-only metrics and uses LoRA rank as a controllable probe of trainable capacity.

Task Setup: Exact memorization uses unique queries as keys and learned parameter increments as the exclusive storage medium for target answers.The formulation decouples memorization from retrieval or contextual comprehension because answer content is inaccessible during inference except through the query.
Answer-only Accounting: All reported sequence-length, loss, and accuracy quantities are computed only over answer tokens, with length determined by each model’s tokenizer.Question tokens provide conditioning context but are excluded from every reported metric.
Evaluation Metrics: The evaluation tracks sequence-averaged loss, token-level accuracy, and exact-match accuracy under greedy decoding.These metrics respectively capture global convergence, fine-grained token dynamics, and strict verbatim recall fidelity.
Evaluation Metrics: Because loss and token accuracy are not monotonically aligned, the study jointly monitors L, Acctok, and AccEM.L measures global convergence trends, Acctok measures token-level behavior, and AccEM measures strict recall fidelity.
LoRA: LoRA rank r provides a monotone capacity knob: with the base model frozen, changes in L or Acc are attributable solely to the low-rank residual update.The residual branch can be fused into the frozen weights without changing the resulting model.

3 The Parametric Memory Law

The Parametric Memory Law quantitatively maps loss reduction to parameter rank and sequence length through a robust power-law relationship. Across semantic densities, model families, and memory tasks, it achieves high explanatory power while describing aggregate rather than token-level memorization dynamics.

Loss reduction is defined as ∆L = Linit − Lfinal, the decrease in cross-entropy loss after applying parametric memory.
∆L increases with rank r and decreases with sequence length ℓ, producing distinct linear trends in log-log space that support a power-law relationship.
The law assigns positive exponents α and β to rank efficiency and length-related memory difficulty, respectively, alongside a positive scaling constant C.α is the Capacity Exponent, while β is the Length Penalty Exponent.
R2 > 0.98 with low MAPE is achieved across pure-semantic, fully random, and short-context PhoneBook settings.For the full 0%–100% random long-context mixture, R2 is 0.987 for Llama3.1-8B-IT and 0.983 for Qwen3-8B-IT.
The law provides a macroscopic mapping among parameter budget, sequence length, and loss reduction, while aggregate metrics leave individual-token memorization dynamics unresolved.The next section analyzes these microscopic token-level dynamics.

4 The Deterministic Phase Transition of Memory

Token-level analysis reveals that average loss can conceal localized bottlenecks that cause autoregressive failure. Under greedy decoding, crossing the Ptarget = 0.5 threshold marks a deterministic phase transition guaranteeing target-token dominance and reliable memory.

Loss-Accuracy Misalignment: Average cross-entropy loss can approach zero while token-level accuracy remains negligible because easy-token confidence masks hard-token errors.Persistently low probabilities at specific positions create invisible bottlenecks despite low global loss.
Loss-Accuracy Misalignment: A single local misprediction corrupts subsequent contexts, triggering error propagation that collapses autoregressive sequence generation.This makes average loss insufficient as a proxy for generation fidelity.
Token-Level Bottlenecks: Stubborn positions remain below p = 0.5 despite LoRA rank increases, with position i = 153 alone accounting for 28% of all failures.The earliest stubborn position tightly bounds the first decoding failure, with Spearman ρ = 0.908.
Deterministic Phase Transition: Under greedy decoding, Ptarget > 0.5 is sufficient to guarantee that the target token is the most probable candidate.Ptarget = 0.5 therefore serves as the critical boundary for deterministic memory success.
Deterministic Phase Transition: The threshold corresponds to a critical loss boundary: L > Lcrit yields disordered memory, whereas L < Lcrit yields ordered memory with guaranteed target-token dominance.The empirical threshold is characterized as Lfinal > 0.69 in Section 3.1.
Deterministic Phase Transition: The deterministic phase transition explains why loss reduction translates into effective accuracy only after crossing the threshold.Lower loss increases the confidence margin, but reliable memory begins with crossing the deterministic boundary.

5 MemFT: Methodology and Empirical Verification

MemFT replaces uniform token-level optimization with threshold-guided weighting that concentrates training on tokens below the deterministic recall threshold, using hard masking and adaptive spatial or temporal curricula. Across long-context and PhoneBook memorization benchmarks, both variants outperform SFT, with MemFT-OT strongest at high rank and MemFT-SW strongest for PhoneBook and low-rank settings.

MemFT: Methodology: MemFT replaces uniform optimization with a normalized token-weighted objective whose dynamic weights prioritize tokens that have not crossed the critical loss threshold.Normalizing by the sum of weights stabilizes gradient scales across samples with different numbers of active tokens.
MemFT: Methodology: MemFT-OT applies a hard threshold mask, concentrating gradients exclusively on tokens below the phase transition without adding hyper-parameters.This avoids over-optimizing tokens that are already easy or memorized.
MemFT: Methodology: MemFT-SW focuses optimization around the first greedy prediction error using exponential spatial decay and expands its window when the anchor remains static.The method preserves base weights upstream of the anchor while prioritizing nearby downstream tokens.
Empirical Verification: 100.0% Acc is reached by MemFT-OT at Llama-r9 and Qwen-r8 in the Long-Context Stress Test, where it surpasses MemFT-SW at high ranks.MemFT-SW consistently leads in low-rank configurations r1 . . . r3, while MemFT-OT accelerates more sharply as rank expands.
Empirical Verification: 100.0% EM accuracy is reached fastest by MemFT-SW at p7 for Llama and p6 for Qwen on PhoneBook, while SFT struggles under lower parameter budgets.MemFT-SW maintains a stable lead across almost all budget scales in this exact key-value memorization benchmark.
Beyond Memorization: Enhanced Generalization: 7%–15% gains over SFT in generalization accuracy are reported across ranks on the Linear Rule Learning benchmark.The benchmark uses f(x, y) = 3x+5y+7 and evaluates both seen and unseen pairs.

6 Related Work

LLM memory methods comprise non-parametric approaches such as ICL and RAG, and parametric approaches that store knowledge in model parameters or modular structures. Existing work, including LoRA-based memory, mainly emphasizes downstream task performance, leaving quantitative capacity patterns and mechanisms underexplored.

LLM Memory: LLM memory strategies divide into non-parametric methods, including ICL and RAG, and parametric methods that store knowledge in model parameters or modular structures.Parametric memory enables persistent storage and retrieval-free reasoning.
LLM Memory: Existing studies primarily evaluate memory through downstream functional tasks, leaving quantitative patterns and mechanisms of parametric memory capacity largely unexplored.
LoRA as Parametric Memory: LoRA is widely used for parameter-efficient fine-tuning and has been adopted as a modular mechanism for encoding new knowledge.Prior work mainly demonstrates LoRA’s effectiveness through downstream task performance improvements.

7 Conclusion

The paper uses LoRA to uncover a Parametric Memory Law linking loss reduction to LoRA rank and sequence length, and identifies token-level decoding collapse from unresolved bottleneck tokens. Based on this mechanism, it proposes MemFT to resolve critical memory bottlenecks.

7 Conclusion: The Parametric Memory Law characterizes loss reduction as a power-law function of LoRA rank and sequence length.LoRA serves as a controllable probe into LLM memory mechanisms within latent space.
7 Conclusion: A deterministic phase transition in token-level loss dynamics shows that unresolved bottleneck tokens can trigger decoding collapse.
7 Conclusion: MemFT is a fine-tuning strategy designed to explicitly resolve critical memory bottlenecks using this mechanistic understanding.

Limitations

The study’s findings are limited by evaluation on 8B-scale models and by the greedy-decoding specificity of the p = 0.5 phase transition. Broader capability trade-offs remain incompletely assessed.

The Parametric Memory Law is evaluated only on 8B-scale models, so its generalization to other scales remains unverified.
The p = 0.5 phase transition is specific to greedy decoding, with robustness under stochastic methods such as nucleus sampling still unverified.
The study provides only a preliminary generalization analysis and lacks a comprehensive assessment of trade-offs with broader capabilities like open-ended

Ethics Statement

The authors acknowledge that memorization techniques have dual-use risks, while limiting this study to mechanistic capacity analysis using non-sensitive benchmarks and synthetic examples. They frame clarifying capacity limits as a contribution to safer model design but emphasize responsible deployment.

Memorization techniques have dual-use potential because they could be misused to encode harmful content.
The study focuses exclusively on model-capacity mechanisms and uses standard benchmarks containing no sensitive personal information.
Examples in Table 4 are synthetic artifacts created for illustration, and the authors link capacity-limit analysis to safer model design and responsible deployment.

A Dataset Construction Details … F Additional Performance Landscapes

The appendices define controlled benchmarks, LoRA configurations, aggregation procedures, and curriculum schedules for measuring exact parametric memorization. They also provide convergence checks and performance-landscape visualizations across rank–length settings.

A Dataset Construction Details: Two controlled benchmarks evaluate exact parametric memory: the Long-Context Memorization Stress Test and the PhoneBook benchmark.The stress test uses synthetic key-value pairs with controlled token corruption, while PhoneBook retains only key-value pairs after removing context.
A Dataset Construction Details: Length buckets count only answer tokens, excluding prompts, chat templates, and keys; PhoneBook bucket boundaries match exactly because phone-number targets are highly regular.Pairs are accumulated until reaching the desired bucket size L.
B LoRA Configuration and Rank Settings: All experiments freeze the base model and train only LoRA adapters, with rank serving as the primary controllable parameter budget.LoRA targets the MLP down_proj module for the stress test and the entire MLP block for PhoneBook, using model-specific layer selections.
C Aggregation Protocol for Main Results: Main-table rank-wise accuracies average performance across multiple memory-length buckets rather than representing a single length.The stress test averages over lengths 4000–10000, while PhoneBook uses answer-only buckets from 1k to 32k.
D PhoneBook Inter-Batch Curriculum Hyperparameters: MemFT-SW PhoneBook experiments use inter-batch temporal curricula with exposure ratios [0.2, 0.4, 0.6, 0.8, 1.0] and epoch boundaries controlling schedule transitions.Length-dependent schedules are reported separately for Qwen3-8B-Instruct and Llama3.1-8B-Instruct because their tokenizations yield different training-pair counts.
E Additional Training Convergence Results: Additional training-loss curves verify convergence across the full rank–length sweep and are intended to rule out under-training as an explanation for accuracy differences.Each subfigure fixes one length–rank configuration and shows the corresponding dataset/model training-loss trajectory.
F Additional Performance Landscapes: Additional performance landscapes visualize results across LoRA ranks, memory lengths, models, and training methods.Subplots group each model by rank and compare methods as memory length increases, complementing averaged main-table results.

G Token-Level Probability Grids Across Data Scenarios

Per-position teacher-forcing probability grids for Qwen3-8B map token probabilities across memory lengths and LoRA ranks in three memorization scenarios. They show that semantically coherent tokens ease memorization, while stubborn positions emerge earlier as random-token content increases.

Grid interpretation: Each grid indexes rows by memory length L and columns by LoRA rank r, with blue curves showing token probabilities and red dots marking p < 0.5 positions.Black dotted vertical lines mark the free-run first-failure position i∗.
Random-token scenarios: Figures 10–11 cover Random 100% across low ranks r ∈{8, 10, 12, 14, 16} and higher ranks r ∈{48, 64, 128, 256, 512}.The low-rank range aligns with LongBench-mixed scenarios for direct comparison.
Random-token scenarios: With 80% semantically coherent LongBench tokens, Random 20% memorizes more easily and stubborn positions appear only at longer lengths.This scenario is shown in Figure 12.
Random-token scenarios: With 40% semantically coherent tokens, Random 60% has intermediate difficulty, and stubborn positions emerge at shorter lengths than in Random 20%.This scenario is shown in Figure 13.