PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion

Genglin Wang, Wangsong Yin, Yeerzhati Abudunuer, Haoxuan Xu, Guoliang Xing, Zhenyu Yan

arXiv:2609.35139v1cs.LG

TL;DR

Independently cached RAG chunks omit cross-chunk context, creating a gap in reusing caches for multi-document answering. CacheRepair predicts per-token KV residuals for a frozen target LLM; across twelve model–dataset combinations, it improves the reported quality–latency trade-off.

  • Problem

    Independently precomputed chunk caches omit cross-chunk attention information, leaving the challenge of recovering that context with low online computation.

    Source: §1 · pp. 1–2

  • Method

    CacheRepair uses a lightweight external network to predict the residual between stale and joint KV caches for every document token while keeping the target LLM frozen.

    Source: §1 · p. 2

  • Results

    Across twelve model–dataset combinations, the largest repairers deliver 1.69–4.61× speedups in p50 TTFT over Full Prefill and improve mean F1 by 2.1–26.1 percentage points over direct cache reuse.

    Source: §1 · p. 2

  • Takeaways & Limitations

    A repairer trained on generic retrieval data is reused across four downstream datasets for each target LLM, and contributes a nondominated operating point in eleven of twelve target–workload pairs.

    Source: §1 · p. 2

    The measured single-request TTFT curves use one A800 with warm pinned-host caches, and all-token repair has Θ(N^2) attention cost.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multi-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality. Selective recomputation methods recover the missing cross-chunk context by rerunning the target LLM on selected tokens, incurring substantial online computation. We introduce CacheRepair, a lightweight network that learns the difference between independently computed KV caches and those produced by processing the chunks together. The network combines compressed KV features with token embeddings and uses attention that is bidirectional within each chunk and flows from earlier to later chunks. Each repair block receives the compressed cache features, and the predicted residual is added to every document token's cache. Each repair network is trained for a specific frozen target LLM on a generic retrieval corpus and reused across downstream datasets. Our analysis shows that repair reduces KV errors both near chunk boundaries and throughout chunk interiors. Evaluation across three target LLMs and four downstream datasets places CacheRepair on the measured answer-quality-latency Pareto frontier in eleven of twelve model-dataset combinations. Reported time to first token (TTFT) includes online cache transfer and repair. Across all twelve combinations, the largest repairers achieve 1.69-4.61$\times$ speedups in median TTFT over full prefill and improve mean F1 by 2.1-26.1 percentage points over direct cache reuse.

1 INTRODUCTION

Independent chunk caches enable reuse across different document combinations, but omit cross-chunk context. CacheRepair predicts per-token KV residuals, and evaluation reports improved quality–latency trade-offs across target LLMs and downstream datasets.

Motivation

  • Independent chunk caches omit cross-chunk context, while selective recomputation restores context by rerunning selected document tokens through the target LLM.The paper asks whether a lightweight external network can repair stale KV while preserving the frozen LLM and reusable caches.Source: §1 · pp. 1–2
  • Stale-cache errors include boundary-local hotspots and layer-persistent bands extending into chunk interiors, motivating repair across all document tokens.Stale KV already contains computation performed within each chunk, so the repairer predicts the residual needed to approximate joint KV.Source: §1 · p. 2

CacheRepair

  • CacheRepair predicts the residual between joint and stale KV caches for every document token, replacing selective target-LLM recomputation while keeping the target LLM frozen.Its compressed cache features are combined with projected token embeddings; attention is bidirectional within chunks and flows from earlier to later chunks.Source: §1 · p. 2

Evaluation

  • Across twelve model–dataset combinations, the largest repairers deliver 1.69–4.61× speedups in p50 TTFT over Full Prefill and improve mean F1 by 2.1–26.1 percentage points over direct cache reuse.CacheRepair contributes at least one nondominated operating point in eleven of twelve combinations.Source: §1 · pp. 2–3
  • Each target-specific repairer is trained on a generic retrieval corpus and reused across MuSiQue, HotpotQA, MultiHop-RAG, and TriviaQA.The evaluation spans Qwen2.5-3B, Llama-3.1-8B, and Qwen2.5-14B.Source: §1 · p. 2

2 RELATED WORK AND PROBLEM SETTING

Prior work reuses independently computed chunk caches to avoid repeated prompt processing, but those caches lack cross-chunk attention information. This section frames cache fusion and defines the repair objective under frozen-model and reuse constraints.

2.1 RELATED WORK

  • Independent chunk caches can be reused across different retrieved combinations and orders, but omit attention to preceding chunks in the assembled request.The resulting context mismatch motivates KV cache fusion to recover answer quality.Source: §2.1 · p. 3
  • Existing fusion approaches use selective target-LLM recomputation, reshape attention, adapt the target LLM, or learn reusable cache components offline.CacheRepair instead uses a target-specific external network to predict residual corrections for every document token.Source: §2.1 · p. 3

2.2 PROBLEM SETTING AND OBJECTIVES

  • The stale KV cache concatenates independently prefetched chunk caches after aligning keys to request positions, but remains different from the joint-prefill cache because cross-chunk attention is missing.Position alignment alone cannot recover the missing information.Source: §2.2 · p. 3
  • The repair objective is to approach joint-prefill quality by correcting cached KV directly while keeping the target LLM frozen and retaining offline chunk reuse.The joint-prefill cache is the learning and evaluation target, and repair depends on ordered documents rather than query text.Source: §2.2 · p. 3

3 OBSERVATION

The observation examines where stale KV differs from joint prefill and finds errors at chunk beginnings as well as across interior positions in particular layers. This structure motivates all-token repair, whose online cost must be balanced against selective recomputation.

Observation

  • The analysis compares frozen Qwen2.5-3B-Instruct caches on the first 256 MuSiQue requests, with heatmaps covering 545,276 tokens in chunks after the first.Relative RMSE is grouped by layer and KV head, and by layer across 16 normalized within-chunk position bins.Source: §3 · p. 4
  • Stale-versus-joint KV errors show boundary-local hotspots and layer-persistent bands spanning boundary and interior positions, with magnitudes varying across layers and within-chunk locations.The reported patterns appear in both keys and values; Appendix G shows them across all three target LLMs.Source: §3 · p. 4

Implications for repair

  • Because errors extend into chunk interiors, the analysis motivates correcting every document token; selective recomputation instead reruns only a subset through the target LLM.For N cache tokens and selected fraction r, selective attention costs O(rN^2); repair also has quadratic attention, so cost depends on network dimensions, token budgets, kernels, and cache movement.Source: §3 · p. 4

4 METHOD

CacheRepair repairs stale KV caches by predicting residuals for document tokens from compressed cache features and token embeddings. Its inference pipeline uses chunk-structured attention, while training pairs independently prefetched caches with joint-prefill targets for a target-specific repairer reused across downstream datasets.

4.1 INFERENCE

  • At inference, compressed stale-KV features and frozen token embeddings are fused, then Repair Blocks predict residuals added to the full stale cache.Each block receives stale-feature reinjection; repaired keys are then placed using global RoPE before the frozen target LLM processes the query.Source: §4 · p. 4, Figure 4 · p. 5, §4.1 · pp. 5–6
  • Block-causal attention permits bidirectional interaction within each chunk and passes information only from earlier to later chunks.The mask follows document order, matching the direction of missing cross-chunk context.Source: §4.1 · p. 5

4.2 TRAINING

  • CacheRepair learns residuals between independently prefetched and joint-prefill KV caches from paired examples produced by a frozen target LLM.The resulting target-specific checkpoint, with its normalization statistics, is reused across downstream datasets.Source: §4.2 · p. 6

5 EXPERIMENTS

Across three target LLMs and four datasets, CacheRepair generally advances the measured quality–latency frontier, while its repair reduces KV errors in both chunk boundaries and interiors. Evaluation also shows that results depend on target, workload, and serving conditions.

5.1 EXPERIMENTAL SETUP

  • The evaluation uses three frozen target LLMs, four downstream datasets, and repairers trained for six epochs on a generic retrieval corpus.Each target-specific epoch-six checkpoint and its normalization statistics are reused across all four datasets; each dataset contributes 500 requests.Source: §5.1 · pp. 6–7

5.2 RESULTS

  • At half of Full Prefill p50 TTFT, CacheRepair’s mean F1 exceeds every eligible baseline in nine settings, while no repairer meets the budget on Qwen2.5-3B/HotpotQA.Selection uses latency alone; the highest eligible means on Qwen2.5-3B MultiHop-RAG and TriviaQA belong to KV Packet and EPIC, respectively.Source: §5.2 · p. 7, Table 3 · p. 9
  • CacheRepair contributes frontier points on all four datasets for both Llama-3.1-8B and Qwen2.5-14B; on Qwen2.5-14B/MultiHop-RAG, it reaches 72.71% F1 at 115.74 ms.That 159M repairer is 6.76× faster than CacheBlend 0.6, with a paired F1 difference of +0.16 points and 95% interval [−1.95, 2.26].Source: §5.2 · p. 7, Figure 6 · p. 8
  • 1.69–4.61× p50 TTFT speedups over Full Prefill and 2.1–26.1 percentage-point F1 gains over Stale KV hold across all twelve settings; CacheRepair contributes frontier points in 11 of 12.The largest repairers are also 1.30–6.76× faster than the highest tested recomputation presets, with F1 differences ranging from −3.86 to +4.43 points.Source: §5.2 · p. 7, Figure 5 · p. 7, Figure 6 · p. 8
  • For the largest repairers on MultiHop-RAG, cache transfer takes longer than repair on Llama-3.1-8B and Qwen2.5-14B, but not Qwen2.5-3B.From 3B to 14B, repair time grows 2.11× while Full Prefill TTFT grows 4.13×; the speedup over Full Prefill rises from 2.98× to 4.61× on this workload.Source: §5.2 · p. 8, Table 1 · p. 8
  • Among the evaluated attention variants, block-causal attention achieves the highest mean F1 and enables cross-chunk information exchange while preserving document order.Block-diagonal attention reaches 0.0133 F1, while the full design reaches 0.2462; removing the residual output path reaches 0.0049.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 KV ERROR BEFORE AND AFTER REPAIR

  • Repair lowers K relative RMSE by 53.9% at boundaries and 50.9% in interiors, with V relative RMSE reductions of 24.4% and 16.2%, respectively.On 256 MuSiQue requests, teacher-forced output KL falls from 1.731 to 0.555 versus Stale KV; with document content fixed, increasing chunks from 8 to 16 lowers Stale KV F1 from 0.231 to 0.165 while CacheRepair 30M remains near 33 ms p50 TTFT.Source: §5.3 · pp. 8–9
  • At 16K document tokens, the largest repairers achieve 3.43–6.14× p50 TTFT speedups over Full Prefill while using 2.0–4.0% of its dense matrix MACs.Across measured 4K, 8K, and 16K lengths, every repairer capacity has lower p50 TTFT than every CacheBlend and InfoFlow preset; comparisons with EPIC depend on length and capacity.Source: §5.3 · p. 9

6 DISCUSSION

CacheRepair is deployed as a target-specific, fixed checkpoint, and its evaluation emphasizes single-request serving conditions. Its use therefore has boundaries in model configuration, sequence length, and serving setup.

6 DISCUSSION

  • Each frozen target LLM needs its own repairer and normalization statistics, making the design suited to stable target models and frequent document reuse.Training and statistics costs equal the saved first-token time of 2.02–5.80 million serial requests; changes to model weights, quantization, or RoPE configuration require checking the resulting KV distribution and repairer.Source: §6 · p. 9
  • The main Pareto curves measure single-request TTFT on one A800 with warm pinned-host caches, and all-token repair has Θ(N^2) attention cost.Concurrent-serving measurements use per-request repair; batching repair and pipelining transfer, repair, and KV writes are left for future work.Source: §6 · p. 9
  • Each target-specific checkpoint is trained offline on generic retrieval documents, then reused across four downstream datasets selected under the paper’s evaluation procedure.Under the same two-epoch budget, MSE has the highest mean answer F1 among the three evaluated objectives.Source: §6 · p. 10

7 CONCLUSION

CacheRepair learns all-token KV residual repair for a frozen target LLM and reuses each target-specific repairer across four downstream datasets. Across the evaluated settings, it improves Stale KV quality while reducing TTFT and often adding nondominated quality–latency points.

7 CONCLUSION

  • CacheRepair predicts KV residuals for every document token for a frozen target LLM, and each repairer is reused across four downstream datasets.Source: §7 · p. 10
  • Across twelve settings, the largest repairers achieve 1.69–4.61× p50 TTFT speedups over Full Prefill and improve Stale KV F1 by 2.1–26.1 percentage points.Source: §7 · p. 10
  • CacheRepair contributes nondominated quality–TTFT points in eleven of twelve evaluated model–dataset settings, alongside selective recomputation and learned cache fusion.Source: §7 · p. 10

AI USE DISCLOSURE

The authors disclose AI assistance in research execution and manuscript preparation, while retaining responsibility for the work. The accompanying evaluation details describe the configurations, inputs, timing, and checks used in the experiments.

AI USE DISCLOSURE

  • The authors report AI assistance with drafting, experimental planning, code, analysis, and figures, while stating that they directed the work and take responsibility for the final manuscript and results.Source: AI USE DISCLOSURE · p. 11

A EVALUATION DETAILS

  • Cache fusion is evaluated with fixed target-LLM weights, preserving deployed behavior on requests without document-cache fusion; fine-tuning methods are treated as a complementary setting.Source: A EVALUATION DETAILS · p. 14
  • Online TTFT includes method-specific transfer and processing through first-token generation, while independent chunk compilation and pinned-host preparation occur before timing.Source: A EVALUATION DETAILS · p. 14
  • In four comparisons totaling 4,000 generations, full-token recomputation with CacheBlend or InfoFlow reproduces Full Prefill’s generated tokens exactly.Source: A EVALUATION DETAILS · p. 15

A.1 DATASET SELECTION

The evaluation uses fixed, shared samples selected through content and provenance checks, then examines how answer-F1 comparisons vary across request batches. The appendix also documents what those checks cannot rule out.

A.1 DATASET SELECTION

  • The four 500-request samples come from specified public pools, screened against three shared repair-training manifests containing 488,823 distinct passage texts.The selection uses fixed content matching against the original passage texts referenced by the manifests.Source: §A.1 · p. 15
  • Evaluation screening excludes requests with matching evidence sentences, 50-token context spans, or question provenance linking them to training sources.The checks use normalized text, contiguous matches, source-ID mappings, and resolved component questions.Source: §A.1 · p. 15
  • Each downstream dataset retains 500 evaluation requests and 50 reserves, selected deterministically before generation and shared across methods and target LLMs.The four datasets provide 2,000 distinct normalized questions; selection does not use model predictions.Source: §A.1 · p. 15, Table 5 · p. 16
  • The screening checks do not cover semantic paraphrases or target-LLM pretraining corpora.The code release provides IDs, hashes, source specifications, and reconstruction scripts for the selected evaluation inputs.Source: §A.1 · p. 15

A.2 ANSWER-F1 VARIATION

  • Across the combined 1,000 Qwen2.5-3B/MultiHop-RAG requests, CacheRepair 30M scores 55.20% F1 versus KV Packet’s 54.04%, while KV Packet remains faster.The paired F1 difference is +1.16 points [−1.32, 3.62], and p50 TTFT is 41.29 versus 27.78 ms.Source: §A.2 · p. 16
  • For Qwen2.5-3B/MultiHop-RAG, CacheRepair 30M’s difference from KV Packet shifts from −1.51 F1 points on the original batch to +3.82 on the additional batch.The original and additional batches each contain 500 requests; answer wording also changes token-F1 scores in the reported example.Source: §A.2 · p. 16

B COMMON-BUDGET RESULTS ACROSS TARGET LLMS

The section compares CacheRepair with high-budget recomputation and reports shared-budget results, detailed KV error measurements, and residual-capacity analyses. It also examines how repairer parameter costs scale with the target model.

B COMMON-BUDGET RESULTS ACROSS TARGET LLMS

  • Across all twelve model–dataset settings, CacheRepair has lower p50 TTFT in all 36 comparisons with the highest tested recomputation presets, with 1.30–6.76× speedups and F1 differences from −3.86 to +4.43 points.Table 7 reports paired differences and pointwise 95% intervals for CacheBlend 0.6, EPIC 64, and InfoFlow 0.25.Source: Table 7 · p. 17, B COMMON-BUDGET RESULTS ACROSS TARGET… · p. 17
  • At half the Full Prefill latency budget, CacheRepair’s mean F1 exceeds every eligible baseline in nine panels, with simultaneous paired intervals remaining positive in four.The tables select each method’s largest tested setting meeting the budget; selection uses latency alone.Source: B COMMON-BUDGET RESULTS ACROSS TARGET… · p. 17, Table 8 · p. 18, Table 9 · p. 18

C DETAILED KV REPAIR MEASUREMENTS

  • On 500 Qwen2.5-3B/MuSiQue requests, epoch-6 repair reduces relative RMSE across all document tokens by 51.4% for K and 18.0% for V, including chunk interiors.Figure 7 reports boundary and interior reductions separately; first-chunk relative RMSE is 0.0068/0.0097 for stale/repaired K and 0.0268/0.0294 for V.Source: Figure 7 · p. 19, C DETAILED KV REPAIR MEASUREMENTS · p. 19

D RESIDUAL STRUCTURE AND REPAIR CAPACITY

  • Raw residuals require ranks of 1,990 and 3,298 to retain 95% of fitting energy for Qwen2.5-3B and Qwen2.5-14B, while evaluated widths retain 67.1–84.8%.The corresponding normalized ranks are 3,105 and 4,282.Source: D RESIDUAL STRUCTURE AND REPAIR CAPACITY · p. 19, Figure 8 · p. 20
  • Actual repair’s normalized residual MSE falls with capacity from 0.514 to 0.450 for Qwen2.5-3B and from 0.556 to 0.505 for Qwen2.5-14B.The learned-head projection uses the true validation residual, whereas repair must infer it from stale KV and token features.Source: D RESIDUAL STRUCTURE AND REPAIR CAPACITY · pp. 19–20
  • The output head accounts for 63.01M of the 104M Qwen2.5-14B repairer’s parameters; the measurements establish feasibility through the evaluated 14B target.Sharing head parameters or factorizing the output projection could reduce cost but would change the output space and require validation.Source: D RESIDUAL STRUCTURE AND REPAIR CAPACITY · p. 20, Table 12 · p. 20

E TRAINING OBJECTIVES AND FUNCTIONAL EFFECTS

The authors compare three training objectives under matched two-epoch conditions and measure both answer quality and functional effects. MSE has the highest reported mean F1, while the repairers improve functional metrics over stale KV on the evaluated requests.

E TRAINING OBJECTIVES AND FUNCTIONAL EFFECTS

  • On 256 MuSiQue requests, mean KL falls from 1.731 for Stale KV to 0.555 for the six-epoch repairer and 0.698 for two-epoch MSE.Attention-output relative RMSE falls from 0.598 to 0.328 and 0.353, respectively; paired intervals for both repairers’ reductions in KL, attention error, and normalized KV MSE lie above zero.Source: E TRAINING OBJECTIVES AND FUNCTIONAL… · p. 21, Table 14 · p. 22
  • QK/V and Huber also improve all three functional metrics over Stale KV, while QK/V has higher normalized KV MSE and KL than two-epoch MSE.QK/V uses fixed equal weights for target-scale attention-logit error and residual-RMS-normalized V error; Figure 9 compares optimization using shared normalized KV MSE.Source: E TRAINING OBJECTIVES AND FUNCTIONAL… · p. 21, Figure 9 · p. 22
  • Under matched two-epoch training, MSE has the highest mean F1, exceeding QK/V by 4.78 points and Huber by 1.08 points.The controlled repairers share architecture, examples, initialization, sample order, normalization, optimizer, and learning-rate schedule.Source: E TRAINING OBJECTIVES AND FUNCTIONAL… · p. 21, Table 13 · p. 22

F ATTENTION TOPOLOGY AND WITHIN-CHUNK CONTEXT

The controlled comparisons examine how attention topology and residual prediction affect repair, then test the error patterns across target models and CacheRepair against a larger KV Packet calibration budget.

F ATTENTION TOPOLOGY AND WITHIN-CHUNK CONTEXT

  • A0–A2 and A4 were close in F1, while the larger A3 and A5 differences support cross-chunk information exchange and residual prediction without establishing rankings across seeds.A0–A2 and A4 use block-causal, strict token-causal, fully bidirectional, and stale-feature reinjection variants; all reported variants shared the training setup.Source: F ATTENTION TOPOLOGY AND WITHIN-CHUNK… · p. 23
  • 24.62% F1 for A0 exceeded 1.18% for A5-JR and 0.49% for A5 under the matched two-epoch training budget.A5-JR improved over A5 by 0.69 points, while residual prediction held a 23.44-point advantage over A5-JR.Source: F ATTENTION TOPOLOGY AND WITHIN-CHUNK… · p. 23
  • 26.56 F1 with both K and V repaired compared with 18.81 for K-only and 0.00 for V-only; keeping the first chunk gave 25.89 F1.The first-chunk difference was −0.66 points with paired interval [−2.14, 0.75]; each choice executed the complete repair network.Source: F ATTENTION TOPOLOGY AND WITHIN-CHUNK… · p. 23, Table 15 · p. 24
  • A3 reduced boundary and interior error relative to Stale KV but perturbed the first chunk, whose K/V relative RMSE was 0.115/0.157 versus 0.011/0.030 for A0.The proposed link to A3’s low answer F1 is a plausible contributor, not an isolated cause.Source: F ATTENTION TOPOLOGY AND WITHIN-CHUNK… · p. 23, Table 16 · p. 25

G CROSS-MODEL VALIDATION OF KV-ERROR STRUCTURE

H LEARNED-FUSION BASELINE COMPATIBILITY

  • Increasing KV Packet calibration from 512 to 8,192 examples changed F1 by −0.04 to +2.84 points, with all paired intervals including zero; CacheRepair had higher mean F1 on three datasets.KV Packet retained higher mean F1 on MultiHop-RAG; the main Pareto curves use its 512-example configuration.Source: Table 18 · p. 27, H LEARNED-FUSION BASELINE COMPATIBILITY · p. 27

I SCALING AND AMORTIZATION DETAILS

The section measures quality and serving costs across document lengths, chunk counts, and concurrency, alongside longer-context tests and training-data and amortization details.

I SCALING AND AMORTIZATION DETAILS

  • At 16K tokens, the largest repairers achieved 3.43–6.14× speedups over Full Prefill using 2.0–4.0% of its dense MACs.Across 270 configurations and 12,960 timed runs, 88 of 108 repairer–selective-baseline pairs had lower repair p50 TTFT at all five lengths; the first-token study does not attach answer quality to those operating points.Source: Figure 12 · p. 28, Figure 13 · p. 28, I SCALING AND AMORTIZATION DETAILS · p. 28
  • 46.68/49.02/49.80% F1 at 4K/8K/16K came with 3.36/4.08/4.45× speedups over Full Prefill, testing a checkpoint trained to 2,269 tokens beyond its training range.At 16K, CacheRepair reached 49.80% F1 versus 49.41% for CacheBlend 0.6, while p50 TTFT was 145.8 versus 1,487.4 ms.Source: Table 20 · p. 30, I SCALING AND AMORTIZATION DETAILS · p. 30
  • At 8, 12, and 16 chunks, CacheRepair gained 5.81, 10.36, and 11.20 F1 points over Stale KV.CacheRepair p50 TTFT ranged from 33.29 to 33.97 ms across four chunk counts, versus 68.54 ms for Full Prefill; gains increase with chunk count at fixed content.Source: I SCALING AND AMORTIZATION DETAILS · pp. 30–31, Figure 15 · p. 31
  • Reversing chunks yielded 25.47 F1 and a fixed permutation 26.49, versus 26.56 in retrieval order; paired intervals for changes relative to Full Prefill included zero.Full Prefill also varied with ordering, with F1 ranging from 28.82 to 33.13.Source: I SCALING AND AMORTIZATION DETAILS · p. 31
  • Further serving work is needed to integrate repair with the scheduler, including batching variable-length inputs and pipelining transfer, repair, and KV writes while preserving dependencies.The stated goal is improving throughput and tail latency for concurrent cache fusion.Source: I SCALING AND AMORTIZATION DETAILS · p. 31
  • At concurrency eight, CacheRepair served 2.70× as many requests per second as Full Prefill; KV Packet reached 46.80 requests/s.CacheRepair throughput rose from 11.35 to 30.70 requests/s as concurrency increased from one to eight.Source: I SCALING AND AMORTIZATION DETAILS · p. 31, Table 21 · p. 32
  • The latency-equivalent reuse-count proxy assumes a serial stream on one A800 and counts first-token savings; actual amortization depends on preparation, decoding, batching, and GPU utilization.Nine epoch-six repairers required 65.9–130.5 A800 GPU-hours each; statistics construction is shared across capacities for each target LLM.Source: I SCALING AND AMORTIZATION DETAILS · p. 32, Table 22 · p. 32

J GENERIC REPAIR PRETRAINING CORPUS COMPOSITION

  • The generic repair corpus contains 50,000 examples from six retrieval sources, pairing queries with ordered retrieved passages.It includes 874,000 chunks and 99,539,881 document tokens; within each length band, four sources contribute 20% of examples each and two contribute 10% each.Source: J GENERIC REPAIR PRETRAINING CORPUS… · pp. 32–33, Table 23 · p. 33

Related papers

Loading 2609.35139v1…