Source-linked AI summary
Faithfulness Is Not Free: Auditing Offline KV-Cache Quantization in Retrieval-Augmented Generation
Atta Ul Asad, Ahsan Bilal, Muhammad Ali, Muhammad Haseeb, Dean F. Hougen
TL;DR
Offline KV-cache quantization can reduce storage in retrieval-augmented generation, but it is unclear whether compressed caches preserve faithfulness to retrieved evidence. This paper audits INT8 and INT4 caches on RGB and HotpotQA using accuracy and three faithfulness signals, finding INT8 near-lossless while INT4 degrades grounding even when accuracy is preserved.
Problem
Prior evaluations measure task accuracy without establishing whether quantized caches preserve faithfulness, meaning responses remain supported by retrieved evidence.
Method
The study uses a controlled offline RAG protocol that caches the full retrieved context in one causal pass and evaluates INT8 and INT4 against a BF16 baseline with three faithfulness signals.
Results
INT8 is near-lossless, whereas INT4 reduces accuracy and causes over 90% of faithfulness changes among accuracy-preserved answers to be negative across RGB and HotpotQA.
Takeaways & Limitations
Faithfulness should be audited alongside accuracy before deploying aggressively compressed offline KV caches, especially under noisy retrieval or greater retrieval depth.
Takeaways & Limitations
The study covers one model family, two QA-style benchmarks, and three retrieval depths, while HHEM and NLI are less reliable for HotpotQA at low K.
Abstract
from arXiv · showhide
Retrieval-augmented generation systems can precompute and store key-value caches of retrieved documents to avoid re-encoding context at every query. Quantizing these caches further reduces storage, but no prior work asks whether compression damages faithfulness, whether responses remain grounded in the retrieved evidence. Faithfulness and accuracy are not equivalent: a model can produce a correct answer that is no longer supported by the context it was given. We evaluate Qwen2.5-7B-Instruct under INT8 and INT4 quantization on RGB and HotpotQA, measuring both accuracy and faithfulness with a hallucination detector, NLI entailment, and an LLM judge. INT8 is near-lossless across both metrics. INT4 reduces accuracy and, more critically, even among answers that remain factually correct, over 90% of faithfulness changes are negative, i.e., accuracy metrics are blind to this regression. The harm grows under noisy retrieval and with more retrieved chunks. Faithfulness must be audited before compressed caches are deployed.
1 Introduction
Offline KV-cache quantization reduces storage costs in RAG, but prior accuracy-only evaluations leave faithfulness unresolved. This audit asks whether compressed caches preserve grounding even when answers remain correct.
- Offline KV caches reduce repeated context processing, but their storage footprint grows with corpus size and retrieval depth.Low-bit quantization is therefore considered as a storage-reduction strategy.
- Prior evaluations report little accuracy degradation from INT8 and INT4, while aggressive compression can harm reasoning on complex tasks.
- Accuracy metrics such as Exact Match and F1 do not verify whether generated answers remain supported by retrieved evidence.
- A correct answer can remain unchanged while becoming less dependent on retrieved evidence, creating a hidden failure mode for compressed RAG.
- The audit isolates cache precision and evaluates faithfulness with three independent signals, finding INT8 near-lossless but INT4 silently degrading grounding on accuracy-preserved examples.More than 90% of faithfulness flips are negative, with McNemar p < 10^-20.
2 Background and Related Work
Related work establishes storage-saving cache methods and faithfulness-evaluation gaps, but does not test whether compressing offline retrieval caches harms grounding.
- KIVI, KVQuant, ZipCache, and KVTuner study low-bit KV-cache compression, including asymmetric, per-channel, token-sensitive, and 4-bit approaches.
- The audit pipeline precomputes and stores a cache, reuses it after dequantization, and evaluates three faithfulness signals alongside accuracy.
- Sub-4-bit compression can cause up to 59% accuracy loss on long-context tasks, motivating variance-normalized and query-aware mixed-precision methods.
- RAG faithfulness benchmarks show that EM and F1 can miss systematic grounding failures, while LLM-as-judge can outperform detector-only evaluation.
- TurboRAG and CacheBlend reduce repeated prefill costs, but neither examines whether compressing stored caches harms faithfulness.
3 Method
The method isolates stored-cache precision by using one position-consistent cache for the full retrieved context, then comparing BF16, INT8, and INT4 under identical decoding conditions.
- The experiment keeps retrieved evidence, prompts, and decoding fixed while varying only stored cache precision.
- Building a unified cache: A single causal prefill pass creates one unified cache for the full retrieved context before compression and storage.
- Building a unified cache: Unified caching avoids positional mismatch from stitching independently cached chunks, ensuring cache bit-width is the only changed variable.
- Quantizing the cache: INT8 uses per-token asymmetric quantization, whereas INT4 uses group-wise quantization over groups of 64 channels with separate scale and zero-point values.
- Quantizing the cache: The cache-size model accounts for bit-width, cached dimensions, token count, group size, and scale/zero-point overhead; BF16 has no overhead term.
- Precision comparisons: C1 BF16 is the cache baseline, while C2 INT8 and C3 INT4 isolate standard and aggressive compression effects against it.
4 Experimental Setup
The evaluation uses Qwen2.5-7B-Instruct on RGB and HotpotQA across three retrieval depths, measuring answer accuracy and faithfulness with complementary signals.
- Qwen2.5-7B-Instruct is a bfloat16-native 7B instruction-tuned decoder with a 32K-token context window.
- RGB contains 300 examples with positive and distractor passages, while HotpotQA contains 300 distractor-split examples requiring evidence from at least two supporting passages.
- Each dataset is evaluated at K ∈ {1, 3, 5} retrieved chunks, producing 3,600 generations per dataset across four cache settings.
- Accuracy uses containment EM and token-level F1, while faithfulness uses HHEM hallucination rate, DeBERTa-v3 NLI entailment, and an LLM judge.The LLM judge serves as the primary faithfulness signal and achieves 92% agreement on 50 handchecked labels.
- The hypotheses test faithfulness changes on accuracy-preserved examples, retrieval-depth effects through a linear slope, and distractor-noise effects.
5 Results
INT8 stays near the BF16 baseline, while INT4 harms both accuracy and faithfulness, including for answers whose accuracy is preserved. The degradation worsens with retrieval difficulty and depth, despite substantial storage savings.
- Accuracy and hidden faithfulness harm: INT4 faithfulness flips remain overwhelmingly negative among accuracy-preserved answers: 231 vs. 24 on RGB and 173 vs. 31 on HotpotQA.The LLM judge reports p<10^-38 on RGB and p<10^-30 on HotpotQA.
- Accuracy and hidden faithfulness harm: INT4 reduces containment-EM and token-F1, with correct→wrong flips exceeding wrong→correct flips on RGB (133 vs. 10) and HotpotQA (117 vs. 24).INT8 remains close to the BF16 cache baseline across accuracy and faithfulness metrics.
- Amplification, storage, and degeneration: INT4–BF16 hallucination gaps increase monotonically from K=1 to K=5 on both benchmarks, although the three-depth slope test is underpowered (p=0.12).The study treats the retrieval-depth trend as suggestive rather than conclusive.
- Amplification, storage, and degeneration: On RGB, the INT4 faithfulness gap grows with distractor fraction independently of retrieval depth (β=0.22), significantly within-K at K=3 and K=5.This supports the reported distractor-related faithfulness hypothesis.
- Amplification, storage, and degeneration: At K=5, INT8 and INT4 shrink cache size by approximately 1.9× and 3.6×, respectively.INT4 falls below nominal 4× because of per-block scale and zero-point overhead.
6 Conclusion
Compressed KV caches can preserve surface answers while weakening their dependence on retrieved evidence. Therefore, practical offline-cache evaluation must audit faithfulness, especially when aggressive quantization provides storage savings.
- KV-cache compression should be treated as a reliability decision, not only a storage optimization.
- Faithfulness is a necessary evaluation target for practical KV-cache compression, especially with aggressive quantization.
Limitations
The study’s scope is limited to one model family, two QA-style benchmarks, and three retrieval depths, constraining how broadly its faithfulness findings can be generalized.
- The evaluation covers one model family, two QA-style benchmarks, and three retrieval depths.The magnitude of faithfulness loss may differ for larger models, other architectures, denser retrieval settings, or production retrievers.
A Ablation Studies
The BF16 hallucination gap increases as more chunks are retrieved on both benchmarks, but the available retrieval-depth sweep is limited.
- The BF16 hallucination gap rises from 0.09 to 0.26 on RGB and from 0.05 to 0.25 on HotpotQA as K increases from 1 to 5.A denser sweep over K ∈{1, 2, 3, 4, 5, 7, 10} is left to future work.
A.1 Fidelity of the Cache Round-Trip (Stage-0 Gate)
The Stage-0 gate verifies that the cache round-trip itself does not degrade the baseline before quantized runs are compared.
- C1 (BF16) exactly reproduced C0 (Oracle) on all 50 held-out examples within floating-point tolerance.This validates the baseline and attributes later C2/C3 differences to quantization bit-width.
A.2 Numerical Stability
The evaluation checks numerical stability, faithfulness-signal complementarity, storage overhead, and the statistical strength of retrieval-depth trends. INT4 shows rare numerical degeneracy, while its faithfulness harm is measured across complementary signals and compression settings.
- Under C3 (INT4), non-finite logits occurred in < 0.1% of examples and were aborted rather than emitted as corrupted tokens.No non-finite logits appeared under C0, C1, or C2.
- HHEM and NLI entailment agree only moderately on RGB (Pearson r=0.44), while the LLM judge agrees with human labels at 92% on RGB and 76% on HotpotQA.The signals capture distinct failure modes, and refusal artifacts add noise on HotpotQA.
- ∼3.6× is the INT4 footprint reduction over BF16 at K=5, versus the nominal 4×, because per-block scale/zero-point pairs consume ≈11% of the cache.INT8 achieves ∼1.9× rather than nominal 2× because per-token overhead consumes ≈5%.
- p=0.12 leaves the H2 slope regression statistically underpowered because it uses only three retrieval depths, so the directional trend is treated as suggestive.The available depths are K ∈{1, 3, 5}.
- Table 1 reports INT8 as safe across the board and INT4 as breaking faithfulness even when accuracy looks fine.Its columns indicate precision-level effects and whether accuracy metrics would detect each problem alone.