Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs

1 INTRODUCTION

• Text communication makes each receiver prefill shared context again, adding redundant computation and latency as context length grows.Cross-family cache reuse must address mismatches in tokenization, model depth, and KV representations while preserving the receiver computation induced by the cache.Source: §1 · p. 1
• Cache reconstruction error alone does not establish preserved receiver behavior: KV Ridge has lower reconstruction error than HeteroFold yet distorts receiver attention.Figure 2 compares K/V reconstruction, attention-weight KL divergence, and attention-output error averaged over six transfer directions on Qasper; lower is better.Source: §1 · p. 1, Figure 2 · p. 2
• Figure 1 illustrates receiver-side question reconstruction after cache transfer, including cross-family Ministral-3-14B→Llama-3.1-8B transfer using Token Alignment on GSM8K.The receiver restates the question without the original text; the figure’s percentages report ordered overlap with the prompt.Source: Figure 1 · p. 2
• Across six transfer directions, HeteroFold outperforms evaluated cache-transfer baselines on all long-context benchmarks and most short-context benchmarks.At 32K context length, Llama-3.1-8B→Ministral-3-14B transfer is 10.7× faster than Native Prefill and 1.18–1.47× faster than KV Ridge and Dense Latent.Source: §1 · p. 2
• HeteroFold uses Token Alignment, cross-family K/V mapping, and receiver-aware calibration while keeping both language models frozen.It aligns token boundaries, combines multi-layer sender features with cross-head mixing, and folds learned corrections into fixed affine maps for direct decoding without receiver prefill.Source: §1 · p. 2
• Figure 3 presents HeteroFold’s sequence of token and layer alignment, separate K/V-map initialization, and receiver attention/output calibration.The corrections fold into affine maps, and both models remain frozen.Source: §1 · p. 2, Figure 3 · p. 3

2 RELATED WORK

• TextMas requires receiver prefill, C2C requires a receiver-generated prompt cache, and RECURSIVEMAS transfers representations that receiver layers still process.These approaches therefore do not provide a decode-ready cache without receiver-side processing.Source: §2 · p. 3
• LATENTMAS assumes cache compatibility, while Dense Latent and KV Ridge avoid receiver prefill but were evaluated within families with compatible tokenization.Those approaches do not address cross-family token correspondence, layer alignment, and KV-representation mismatches.Source: §2 · p. 3
• HeteroFold addresses both limitations by aligning tokenizers and model depths, mapping sender K/V features into receiver space, and preserving receiver behavior without receiver prefill.This enables direct cross-family KV cache transfer.Source: §2 · p. 3

3 METHOD: HETEROFOLD

• HeteroFold maps sender caches into receiver-compatible K/V states while both models remain frozen and the receiver skips prefill.Its pipeline combines token and layer alignment, Recolor KV mapping, and output-aware calibration.Source: §3 · p. 3

3.1 TOKEN AND LAYER ALIGNMENT

• Token Alignment pairs shared character boundaries, while Layer Alignment combines sender-layer features; flattening supports cross-head mapping despite differing head counts.Unmatched receiver boundaries reuse the latest preceding shared boundary; Layer Alignment uses three sender layers at offsets (−4, 0, +4).Source: §3.1 · p. 4

3.2 MOMENT-MATCHED RECOLORING

• Recolor whitens sender and receiver features, Procrustes-aligns paired features, then restores the receiver mean and covariance in the affine initialization.With sufficient sender dimension and nonsingular covariances, the unregularized map matches receiver moments; numerical stabilization makes this approximate in practice.Source: §3.2 · p. 5

3.3 OUTPUT-AWARE CALIBRATION

• Output-aware calibration separately tunes keys to match attention effects and values under fixed attention, then folds corrections into fixed affine maps for inference.The key objective targets attention patterns and their post-projection effect; the value objective refines mapped values without updating keys.Source: §3.3 · pp. 5–6

4 EXPERIMENTAL RESULTS

• Evaluation covers six directed frozen BF16 transfers among Llama, Qwen, and Ministral, with native receiver prefill as the text-based reference.Dense Latent and KV Ridge are evaluated with and without TA; their direct cross-family variants pair tokens by index.Source: §4 · pp. 6–7
• QA evaluation spans four long-context and five short-context benchmarks, alongside multi-round heterogeneous-agent communication on HIDDENBENCH.Calibration uses 1,600 training and 400 held-out prompts; HotpotQA calibration prompts do not overlap with evaluation sets.Source: §4 · p. 7

4.1 RESULTS ON CROSS-FAMILY TRANSFER

• HeteroFold improves over evaluated cache-transfer baselines in most settings, with particularly clear gains on long-context QA, including benchmarks unseen during calibration.TextMas provides the native-prefill reference under lossless text communication.Source: §4.1 · p. 7

4.3 COMPONENT ABLATION

• Replacing TA caused the largest ablation degradation; single-layer and head-local mappings also reduced performance, while Recolor outperformed Ridge.Output-aware calibration substantially recovers Ridge’s performance, but TA + Recolor with output-aware calibration performs better, with key correction contributing most of the long-context gains.Source: Table 3 · p. 8, §4.3 · p. 8

4.2 RESULTS ON HIDDENBENCH: MULTI-AGENT COMMUNICATION

• On HIDDENBENCH, HeteroFold performs comparably to TextMas while outperforming evaluated cache-transfer baselines in multi-round communication.The table reports initial group accuracy and metrics after 15 communication rounds.Source: Table 2 · p. 7, §4.2 · p. 8

5.1 LATENCY MEASUREMENT

• 10.7× faster than Native Prefill at 32K, with speedup rising from about 3.5–3.8× at 4K; HeteroFold has the lowest latency in both measured directions.The measurement includes tokenization, alignment, payload transfer, mapping, cache construction, receiver-side processing, and first-token computation.Source: §5.1 · pp. 8–9

5.2 CROSS-FAMILY TRANSFER ANALYSIS

• Same-index token mismatch grows with context, whereas TA maintains close alignment through shared character boundaries.Figure 5 compares mean absolute character-position differences for matched tokens across six transfer directions and three context lengths.Source: §5.2 · p. 9, Figure 5 · p. 9
• Across all six directions, HeteroFold more closely preserves native receiver next-token distributions, attention weights, and answer choices than both baselines.Figure 6 reports next-token KL, answer agreement, and attention KL by receiver model.Source: §5.2 · p. 9, Figure 6 · p. 9
• With TA shared across methods, the comparison evaluates K/V mapping designs, including HeteroFold’s receiver-aware calibration.Source: §5.2 · p. 10

• HeteroFold enables cross-family KV cache transfer without receiver prefill while keeping both language models frozen.Source: §6 · p. 10
• Across six transfer directions, HeteroFold improves over evaluated cache-transfer baselines, including on all four long-context benchmarks, while reducing receiver-side latency.Source: §6 · p. 10
• The results show KV computation can be reused across model-family boundaries without requiring receiver prefill.The paper presents this as a step toward efficient cache sharing among heterogeneous language-model agents and more general KV interfaces.Source: §6 · p. 10

AI USE STATEMENT

• AI tools were used to improve manuscript clarity, grammar, and readability; the authors reviewed and verified the edits and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

APPENDIX: HETEROFOLD

• The appendix provides evaluation settings, implementation details, additional receiver analyses, and latency measurements.Source: APPENDIX: HETEROFOLD · p. 13
• Appendix sections cover experimental details, alignment and calibration, same-family results, receiver analysis, and system latency.Source: APPENDIX: HETEROFOLD · p. 13

A.1 MODELS AND EVALUATION

• All three evaluated models are frozen; Qwen runs in non-thinking mode, Ministral uses its language component, and the latency setup places sender and receiver on separate H100 GPUs.Each model tokenizes the same prompt independently, with native chat-template tokens handled separately.Source: §A.1 · p. 14
• The evaluation covers short-context QA, four long-context QA datasets, and HIDDENBENCH, using accuracy, token F1, and accuracy or invalid rate as applicable.Long-context sets are Qasper and HotpotQA subsets from LongBench, LoCoMo categories 1–4, and QuALITY development; LoCoMo excludes unanswerable category 5.Source: §A.1 · p. 14

A.2 HIDDENBENCH EVALUATION

• HIDDENBENCH uses 65 Hidden Profile tasks with three- or four-agent groups, 15 communication rounds, and initial and final votes.Invalid votes count as incorrect; the paper reports mean participant accuracy, majority-vote accuracy, and invalid-response rate.Source: §A.2 · p. 14
• For HIDDENBENCH cache transfer, senders map only generated-message-span K/V states to the receiver token grid; Qwen-to-Qwen messages remain text for every method.Private facts and conversation histories transfer only when expressed in a generated message.Source: §A.2 · p. 14

A.3 CALIBRATION AND BASELINE IMPLEMENTATIONS

• Calibration uses 1,600 training and 400 held-out prompts balanced between Open-R1 problem text and HotpotQA passages and questions, excluding gold answers and solutions.Moment statistics come from the training split; output-aware correction is selected using held-out loss.Source: §A.3 · p. 14
• Dense Latent and KV Ridge are reimplemented from their published settings, while direct cross-family controls pair same-index tokens and stop calibration at the shorter sequence.The +TA variants change token correspondence while retaining each baseline’s mapping and layer-selection design.Source: §A.3 · p. 15

B.1 TOKEN ALIGNMENT (TA) IMPLEMENTATION

• Token Alignment pairs nonempty sender and receiver tokens ending at the same character position, using the latest preceding shared boundary when no exact match exists.Before the first shared boundary it uses the first sender token; in many-to-one cases it selects the sender token at the matched boundary rather than averaging states.Source: §B.1 · p. 15
• Figure 5’s token-position mismatch averages character-offset differences across 48 documents, with equal weight per document and no model forward pass.The comparison uses tokenizer offsets only.Source: §B.1 · p. 15
• HeteroFold clips a three-layer sender neighborhood at model boundaries, mapping keys before receiver normalization and RoPE and values after value projection.Mapped K/V features use BF16, and the receiver applies its native key normalization and RoPE once.Source: §B.1 · p. 15

B.2 OUTPUT-AWARE CALIBRATION DETAILS

• Output-aware calibration matches native receiver attention weights and post-output-projection attention outputs at question-span query positions.Key and value corrections are optimized jointly, with each objective updating only its corresponding correction.Source: §B.2 · pp. 15–16
• Receiver-aware calibration objectives decrease across all six transfer directions, using the checkpoint selected by the lowest combined held-out loss.Figure A compares Recolor initialization with the selected checkpoint using attention KL and normalized attention-output error.Source: §B.2 · p. 16
• After calibration, the learned low-rank correction is folded into the affine map, so inference needs no separate correction module.Source: §B.2 · p. 16

B.3 SENSITIVITY TO CALIBRATION CHOICES

• Increasing calibration data from 200/50 to 1,600/400 prompts improves ARC-C from 72.87 to 77.05 and HotpotQA from 26.40 to 49.44.QuALITY changes less between the two larger settings.Source: §B.3 · pp. 16–17
• All nonzero correction ranks improve HotpotQA over Recolor alone, while ARC-C and QuALITY vary within a narrower range; rank 16 is fixed across directions.Source: §B.3 · p. 17, Table C · p. 17
• Three sender layers improve ARC-C from 66.89 to 77.05, HotpotQA from 22.52 to 49.44, and QuALITY from 51.58 to 62.90 versus a single layer.Three- and five-layer settings perform similarly, so three layers are used by default.Source: §B.3 · p. 17, Table D · p. 17
• The equal Open-R1/HotpotQA calibration mixture gives the highest results on all three benchmarks: 75.68 on ARC-C, 41.97 on HotpotQA, and 63.57 on QuALITY.ARC-C and QuALITY are not used for calibration.Source: §B.3 · p. 17

B.4 MAPPER SIZE AND CONSTRUCTION COST

• HeteroFold stores 201–252 million mapper parameters per direction, approximately 25% fewer than Dense Latent and 62% fewer than KV Ridge.Constructing one transfer direction takes approximately two H100 GPU-hours under the default calibration setting; Table F compares stored mapper parameters.Source: §B.4 · p. 18

C ADDITIONAL EXPERIMENTAL RESULTS: SAME-FAMILY

• The same-family comparison follows each baseline’s original same-family token correspondence and mapping procedures, using the same calibration data and train/held-out split as the main experiments.The experiments also serve as a sanity check for baseline implementations in their original setting.Source: C ADDITIONAL EXPERIMENTAL RESULTS:… · pp. 18–19
• Across all four Qwen3 transfer directions, HeteroFold outperforms both evaluated cache-transfer baselines on Qasper, HotpotQA, LoCoMo, and QuALITY while remaining competitive on short-context QA.The reported pattern extends HeteroFold’s gains beyond cross-family tokenizer alignment to same-family transfer.Source: C ADDITIONAL EXPERIMENTAL RESULTS:… · p. 19

D.1 MEASURING RECEIVER BEHAVIOR PRESERVATION

• Receiver-behavior evaluation measures median next-token KL, agreement on the highest-probability A/B/C/D answer token, and mean attention KL against native receiver behavior.Answer-choice agreement measures consistency with the native receiver, not correctness against the gold answer.Source: §D.1 · p. 19

D.2 NATURAL-LANGUAGE CONTENT PRESERVATION

• On 100 GSM8K questions, ordered overlap (word-level longest-common-subsequence F1) was 83.04 for HeteroFold, 76.68 for KV Ridge + TA, and 39.74 for Dense Latent + TA; native was 100.0.Ordered overlap measures lexical reconstruction, not downstream task accuracy.Source: §D.2 · pp. 19–20

D.3 STABILITY UNDER AUTOREGRESSIVE DECODING

• Prompt-only calibration using question-span query positions and no generated answers or reasoning traces showed no sustained rise in prediction divergence during teacher-forced GSM8K decoding.Differences were generally smaller at later positions.Source: §D.3 · p. 20

E.1 MEASUREMENT

• Transfer timing includes tokenization, TA, payload copying, K/V mapping, cache construction, and receiver processing through the first output token, but excludes sender prefill and later decoding.Runs follow matched-input warm-up and report the median of three measurements.Source: §E.1 · p. 21, Table I · p. 22

E.3 LATENCY ACROSS ALL SIX DIRECTIONS

• Across all six directions and 4K–32K contexts, both transfer methods beat Native Prefill in latency; HeteroFold beats KV Ridge in latency and additional peak memory, though not always Native Prefill in memory.Figure D compares speedup and additional peak receiver memory across directions and context lengths.Source: Figure D · p. 22, §E.3 · p. 23

E.2 LATENCY BREAKDOWN

• Most of HeteroFold’s latency advantage comes from its smaller K/V mapping and cache-construction stage.Figure C decomposes stage timings for both directions and all three context lengths.Source: Figure C · p. 21, §E.2 · p. 22

E.4 SENDER-SIDE COST

• If the sender has not already processed the context, sender prefill and payload capture must be added before cache transfer.For transfers to Ministral, the added costs at 4K, 16K, and 32K are 280, 1,328, and 3,381 ms for Llama, and 235, 1,163, and 3,083 ms for Qwen.Source: §E.4 · p. 23