PaperlayerView exact revision on arXiv

Source-linked AI summary

Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Shayan Shahrabi-Farahani, Dara Rahmati

arXiv:2608.18578v1cs.CLcs.LG

TL;DR

Proactive interference can impair retrieval in overwritten LLM states, but it is unknown whether quantization worsens this vulnerability. Across three instruction-tuned models and multiple precisions, the paper finds that bitsandbytes INT4 quantization reduces accuracy under semantic interference, with smaller penalties for INT8 in some models.

  • Problem

    It is unknown whether quantized models’ aggregate-benchmark robustness extends to interference-heavy retrieval, a practical gap for frequently updated, semantically dense applications.

  • Method

    The study evaluates FP16, INT8, and INT4/NF4 bitsandbytes quantization on a key-rebinding retrieval task across three instruction-tuned models and semantic versus numeric distractors.

  • Results

    INT4 reduces interference endurance on word-type attributes in all three models, with penalties of 1.8%, 5.1%, and 5.2%, while INT8 falls between FP16 and INT4.

  • Takeaways & Limitations

    Bitsandbytes INT4 quantization can impose an additional retrieval cost under semantic proactive interference, despite robustness on aggregate benchmarks.

  • Takeaways & Limitations

    The evaluation is specific to bitsandbytes LLM.int8/NF4 quantization, so other quantizers and newer formats remain to be assessed.

Abstract

from arXiv · show

Proactive interference (PI) is a documented failure mode in large language models in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ($p \le 2.6 \times 10^{-6}$) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10^{-7}$). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

1 Introduction

The introduction connects two previously separate concerns: quantization is widely assumed safe based on aggregate benchmarks, while proactive interference already weakens retrieval from repeatedly updated contexts. The paper tests whether quantization disproportionately harms this fragile capability and finds a semantic-specific INT4 effect, with INT8 showing a cliff-like pattern rather than proportional degradation.

  • Motivation: PTQ is the default deployment path for open-weight LLMs, and negligible aggregate-benchmark losses have fostered the assumption that 4-bit quantization is broadly safe.The cited benchmarks include perplexity, MMLU, and standard QA.
  • Background: Proactive interference causes retrieval of repeatedly overwritten values to degrade log-linearly as prior updates accumulate, despite the newest value being nearest the query.The passage illustrates this with a meeting time updated from 2pm to 3pm to 4pm.
  • Research question: The study addresses whether quantization’s aggregate-benchmark robustness extends to interference-heavy retrieval, especially in applications with long-lived, frequently updated states.Examples include multi-turn assistants, document revision tracking, and dialogue systems tracking user preferences.
  • Study and central finding: Across three precision levels and three architecturally distinct open-source models, INT4 specifically degrades retrieval under semantic interference, whereas numeric controls show no such effect.The tested precisions are FP16, INT8, and INT4; semantic attributes include mood, occupation, and favorite color or animal.
  • Quantization pattern: INT8 is statistically indistinguishable from FP16 in two of three models, while INT4 produces a significant, consistent drop, indicating a cliff rather than graceful degradation.This pattern suggests the effect is not simply proportional to quantization aggressiveness.

2 Related Work

Prior work establishes proactive interference as a retrieval failure caused by repeated key rebindings, while quantization research has largely emphasized aggregate benchmark accuracy. Related long-context studies distinguish positional retrieval effects from proactive interference and report disproportionate retrieval damage from NF4 quantization.

  • Proactive interference in LLMs: PI-LLM finds retrieval accuracy declines log-linearly as prior key rebindings accumulate, across model families and despite prompting or chain-of-thought attempts to ignore earlier values.The paradigm repeatedly rebinds a key to new values while querying only the final value.
  • Post-training quantization: 8-bit and 4-bit NF4 post-training quantization preserve aggregate task performance with minimal loss, but evaluation typically overlooks fine-grained behavioral failure modes.The cited literature primarily uses perplexity and MMLU-style question answering, while PTQ-Bench is introduced as a recent large-scale benchmarking effort.
  • Long-context retrieval: Long-context retrieval work identifies position-dependent accuracy, or “lost in the middle,” as distinct from PI’s repeated key-rebinding paradigm.The long-context setting uses one unambiguous target among irrelevant distractors at varying positions.
  • Long-context retrieval: NF4 is reported as disproportionately damaging in retrieval-heavy long-context settings, converging with this study’s focus on bitsandbytes and supporting a quantization-specific retrieval effect.The comparison involves longer contexts and no repeated key-rebinding, so the retrieval paradigm differs from this study’s task.

3 Method

The study applies a key-rebinding retrieval task across multiple interference levels to three instruction-tuned models evaluated at FP16, INT8, and INT4/NF4 precision. Trial-level pairing and complementary statistical analyses test quantization effects while examining error types and potential tokenization confounds.

  • Task and conditions: Models retrieve a subject’s final attribute after k sequential overwrites, with k spanning 1, 2, 4, 8, 16, 32, 64, and 96.The task includes eight attributes divided into two conditions, including four word-type semantic attributes.
  • Models and quantization: Three open-weight instruction-tuned models—Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Phi-3.5-mini-instruct—are evaluated at FP16, INT8, and INT4 NF4 precision.All precisions are loaded through bitsandbytes, with double quantization for INT4 and no explicitly excluded layers, including the lm head.
  • Experimental design: 23,075 unique trials use identical subjects, distractors, and ordering across quantization levels, enabling paired FP16-versus-INT4 and FP16-versus-INT8 comparisons.The passage reports zero mismatches across the shared trial sequences.
  • Statistical analysis: McNemar’s exact test is primary, with Fisher’s exact and pooled Mantel–Haenszel tests retained as more conservative cross-checks.McNemar’s tests are applied at each model’s most-powered interference level, selected to minimize ceiling and floor effects.
  • Robustness and error analysis: A mixed-effects logistic regression uses all eight interference levels, while incorrect word-type responses are classified by intrusion type and numeric comparisons are stratified by token length.The regression includes random intercepts for model, seed, and attribute, plus a random-slopes robustness variant.

4 Results

INT4 quantization produces a high-interference accuracy penalty that aggregate accuracy can obscure, while INT8 shows a smaller effect in two models. The penalty is specific to word-type interference, associated with increased same-key intrusions, and originates in the transformer backbone rather than the output projection.

  • Overall accuracy: 1.1–2.4 percentage points: pooled FP16–INT4 aggregate accuracy differences are small, masking the concentrated high-interference penalty.The effect is not visible from aggregate accuracy alone.
  • Word-type interference: At high interference, INT4 separates from FP16 and INT8 in all three models, with Qwen accuracy dropping from 81.0% to 68.3% at level 64.Qwen remains at ≥96% through level 32 under every precision, so its local 13-point drop is diluted in aggregate IES.
  • INT8 comparison: INT8 shows a smaller but real paired-test effect in two of three models, whereas Qwen remains statistically indistinguishable from FP16.The unpaired Fisher tests missed these effects in all three models.
  • Interference endurance: 1.8%, 5.1%, and 5.2%: INT4 reduces IES relative to FP16 for Qwen2.5-7B, Mistral-7B, and Phi-3.5-mini, respectively.INT8 falls between FP16 and INT4 in every model, following FP16 ≥ INT8 ≥ INT4.
  • Error mechanism: 21.5%, 23.2%, and 24.6%: same-key intrusion rates rise monotonically from FP16 to INT8 to INT4 across all word-type trials.The pooled FP16-versus-INT4 paired test gives OR = 0.84, 95% CI [0.78, 0.90], p = 4.8 × 10^-7.
  • Ablation: Backbone-only INT4 and INT4 with a quantized lm head have indistinguishable accuracy, with paired McNemar tests yielding p = 1.0 for all three models.The ablation therefore identifies the transformer backbone, not the output projection, as the source of the penalty.

5 Discussion

The discussion argues that the INT4 penalty is robust across models and interference levels, specific to semantic interference, and practically relevant for applications tracking frequently updated, semantically dense state. It also identifies INT8 as a meaningfully safer alternative than INT4 for such applications.

  • Robustness and specificity: All three architecturally distinct models show the same effect direction, with every individual seed agreeing with the aggregate direction (15/15).The passage also states that the primary paired McNemar’s test is significant.
  • Robustness and specificity: β = −0.046, p = 5.7 × 10−8, confirms that the INT4 penalty grows with interference across all eight levels, three models, and attributes.Adding random slopes by model and attribute leaves the result essentially unchanged: β = −0.048, p = 1.9×10−8.
  • Robustness and specificity: The effect is specific to semantic word-type interference and reverses direction under numeric interference (β = +0.053, p < 10−16).This specificity indicates that the reported penalty depends on the interference condition rather than appearing uniformly across task variants.
  • Practical implications: Applications using 4-bit quantized models to track frequently updated, semantically dense state may face more retrieval failures than aggregate benchmark scores suggest.Examples include assistants tracking evolving user preferences and systems tracking many similar entities across long contexts.
  • Practical implications: INT8 quantization remains a meaningfully safer choice than INT4 for applications that depend on such retrieval.The passage states that INT8’s accuracy penalty is smaller than INT4’s.

6 Limitations and Future Work

The study establishes a robust effect within a controlled, bitsandbytes-specific scope, while identifying limitations in quantization coverage, control-vocabulary filtering, prompt length, and mechanistic validation. Future work should extend quantization evaluations, strengthen design controls, and probe model internals directly.

  • Quantization scope: The evaluation covers only bitsandbytes LLM.int8/NF4 quantization, motivating tests of additional quantization methods and settings.The reported INT4 effect is localized to the transformer backbone because lm head quantization added no penalty in the forced-quantization ablation.
  • Task and design scope: The numeric control vocabulary was not tokenizer-filtered to single-token items, and prompts were limited to at most a few hundred tokens.Token-length composition was flat across interference levels and the specificity result held in the dominant token-length stratum, reducing but not eliminating concern about the control design.
  • Mechanistic validation: The proposed mechanism is supported behaviorally and by backbone localization, but it has not been tested at the representational level.Direct probing of attention patterns, residual-stream precision, and activation distributions is identified as a needed direction.

7 Conclusion

bitsandbytes INT4 post-training quantization significantly degrades retrieval accuracy under semantic proactive interference across three architecturally distinct open models. The penalty is large under high interference yet largely hidden by aggregate benchmark accuracy, while INT8 shows a smaller penalty in two models.

  • Conclusion: INT4 significantly degrades LLM retrieval accuracy under semantic proactive interference across three architecturally distinct open models.The finding was confirmed by per-model paired McNemar’s tests and a conservative unpaired pooled test.
  • Conclusion: INT8 carries a smaller but real penalty in two of the three models when evaluated with the appropriately matched test.The paired re-analysis revises the paper’s original “cliff” narrative.
  • Conclusion: 1–2 percentage points separates FP16 and INT4 in aggregate benchmark accuracy, versus 10–13 percentage points under high-interference, semantically-confusable conditions.The overall effect is therefore invisible to aggregate benchmark accuracy in the reported data.

A Full Task Design: Prompts, Scoring, and Vocabulary Construction

The task uses chronological overwrite prompts whose final update supplies the only correct answer, with native chat formatting and exact-match scoring. Vocabulary construction controls rarity and token-length confounds for word attributes, while numeric trials use a fixed integer pool and document a tokenizer-related pipeline pitfall.

  • Prompt design: Each trial presents k chronological updates for one subject-attribute pair, with the final update containing the ground-truth value requested by the retrieval question.Prompts are delivered as one user message through each model’s native instruction-chat template, without a system prompt.
  • Scoring: Responses are constrained to one word or number within 12 new tokens, then normalized and compared with the gold value using exact-match criteria.Word responses are lowercased, whitespace-trimmed, optionally stripped of one trailing period, and reduced to the first alphabetic substring.
  • Numeric control: Numeric attributes use approximately 990 unique integers per category from a fixed range, supporting every tested interference level without systematic token-count drift.Integers from 3–998 may occupy one or two tokenizer tokens, creating a documented but level-invariant asymmetry relative to word-type filtering.
  • Methodological pitfall: An earlier arithmetic tokenizer-pool method silently corrupted one tokenizer family’s vocabulary while appearing valid for another, making it a documented methodological pitfall.The pipeline previously subtracted a “blank” baseline encoding, which caused the corruption.

B Reproducibility: Model Loading Configuration · C Trial Budget, Seeds, and Paired Design

The study used a fixed single-GPU loading setup and evaluated each condition across five seeds with a 69,225-trial budget. Quantization comparisons were paired on identical generated trials, while sampling emphasized higher interference levels.

  • B Reproducibility: Model Loading Configuration: All experiments ran on a single NVIDIA RTX 3090 GPU with 24 GB of VRAM under the model-loading configuration summarized in Table 8.No additional dtype autocasting was applied beyond the compute dtype specified for INT4 quantization.
  • B Reproducibility: Model Loading Configuration: Table 8 specifies the model loading and quantization configuration used across all experiments.
  • C Trial Budget, Seeds, and Paired Design: 5 independent random seeds were used for every model × quantization × interference-level × attribute condition.
  • C Trial Budget, Seeds, and Paired Design: 69,225 total trials comprised 28,050 word-type and 41,175 numeric trials across 45 model × quantization × seed runs.
  • C Trial Budget, Seeds, and Paired Design: For each fixed model–seed pair, FP16, INT8, and INT4 used identical generated trial sequences, so comparisons were paired rather than independently sampled.Subject names, distractor values, ordering, attributes, and interference levels were held constant across quantization levels.
  • C Trial Budget, Seeds, and Paired Design: Across seeds, subject names and distractor values were sampled independently from the same fixed, tokenizer-filtered vocabulary pool.
  • C Trial Budget, Seeds, and Paired Design: The trial budget was intentionally weighted toward higher interference levels to reduce the chance that floor and ceiling effects would obscure quantization impacts.Table 9 reports trials per interference level for each model, quantization, seed, and attribute cell.

D Statistical Methodology: Full Details

The analysis combines power-motivated per-model FP16–INT4 comparisons with mixed-effects regression across all interference levels. Additional checks address paired trials, random slopes, error classification, token length, and aggregate endurance.

  • Per-model comparisons: Qwen uses k = 64, Mistral k = 16, and Phi k = 8 as the most statistically informative interference levels for FP16–INT4 comparisons.These levels are selected to avoid ceiling effects at low k and floor effects at very high k.
  • Mixed-effects model: A mixed-effects logistic regression analyzes every interference level, model, and seed using quantization mode × log2(interference level), with FP16 as reference.The model includes random intercepts for model, seed, and attribute and is fit via variational Bayes.
  • Paired significance testing: McNemar’s test verifies paired quantization comparisons across 23,075 unique model–seed–attribute–level–trial combinations.The pairing arises because identical trial content is presented at both quantization levels.
  • Robustness and error analysis: Robustness analyses refit the mixed model with random log2(level) slopes by model and attribute, while deterministic replay enables same-key intrusion classification with zero reconstruction mismatches.The replay reproduces the logged subject name and gold value for all 23,075 unique trials.
  • Interference Endurance Score: IES is normalized raw accuracy-curve AUC over a log2-scaled interference axis, placing endurance on the [0, 1] scale for cross-model comparison.The area is computed by trapezoidal integration and divided by the tested log2 range.

E Additional Results and Robustness Checks

Additional analyses largely support the paper’s conclusions while documenting a Phi non-monotonicity at k = 16, model-level quantization gaps at k = 1, robustness to random slopes, and limitations in the numeric control and IES methodology.

  • Phi non-monotonicity: At k = 16, Phi-3.5-mini-instruct is the one condition where INT8 accuracy falls below INT4, while the key-level comparison uses k = 8.Table 10 reports n = 300 per cell.
  • Mixed-model interpretation: At k = 1, both INT8 and INT4 show sizeable negative main effects relative to FP16 in word-type and numeric attributes.The mixed-model coefficients refer to the lowest interference level because log2(1) = 0 after centering, where accuracy is near ceiling.
  • Random-slope robustness: Refitting the mixed model with random slopes for log2(level) by model and attribute leaves the key interaction essentially unchanged in both panels.The original model included random intercepts for model, seed, and attribute but not these random slopes.
  • Token-length robustness: Numeric token-length composition is nearly flat across interference levels, with 3-token numbers comprising 89.9–92.4% of trials, addressing a possible token-length confound.The numeric pool was not tokenizer-filtered to single tokens, unlike the word-type pools.
  • Numeric control: For Mistral, the INT4 penalty is 5.1% under word-type interference versus 2.6% under numeric interference, whereas Qwen’s penalties are 1.8% versus 0.05%.The numeric-control pattern only partially corroborates the specificity result.
  • Methodological limitation: The IES computation was not independently re-verified for differing tokenizer-filtered vocabulary caps across word-type attributes.For Qwen, mood reaches k = 64 while occupation caps at k = 32, leaving the pooling convention for higher levels unresolved.

F lm head Ablation: Additional Methodological Detail

The ablation established that the original INT4 condition left the lm head unquantized by default, while manually quantizing it changed outputs on only 4.0% of paired trials.

  • F lm head Ablation: Additional Methodological Detail: The original INT4 configuration produced token-for-token identical outputs with or without an lm-head skip-list across 5,050 paired trials.The loaded model showed that the installed library version left the lm head at full precision by default.
  • F lm head Ablation: Additional Methodological Detail: Manual conversion of the lm head to a genuine 4-bit layer changed 203/5,050 paired trials, or 4.0%, relative to the backbone-only INT4 condition.The conversion was confirmed before running trials and matched the backbone’s quantization settings.
Loading 2608.18578v1…