Accelerating Cross-Model LLM Inference via Attention Map Reuse

1 Introduction

• Shared structured contexts recur across turns and operators, making redundant prefix processing a major bottleneck for interactive latency and throughput.Examples include schemas, system constraints, tool specifications, interaction history, and intermediate results.Source: §1 · p. 2
• Existing prefix caching is largely limited to identical models because KV tensors are coupled to model-specific representations, dimensions, structures, and parameters.Translating KV states across heterogeneous architectures risks accuracy loss or adaptation overhead.Source: §1 · p. 2
• MapFlow reuses attention maps because token-dependency patterns can be more structurally similar across models than KV tensors, then applies lightweight linear translation and partial recomputation.The stated setting assumes models share a tokenizer and process the same context.Source: §1 · p. 2
• Block-level top-p compression and a fused retrieval kernel address attention maps’ quadratic size and avoid materializing dense receiver-side intermediates.Prefix-hash indexing and overlapping retrieval with partial recomputation support online reuse.Source: §1 · pp. 2–3
• MapFlow reduces TTFT by 15%–20% and improves end-to-end throughput by up to 25% while maintaining generation quality across heterogeneous models and multi-turn workloads.Benefits are especially pronounced in long-context, multi-turn settings.Source: §1 · p. 3

2 Background

• Decoder-only Transformer models provide the architectural context for attention-based LLM inference [63].A standard block has multi-head attention and a feed-forward network.Source: §2.1 · p. 3

2.1 Transformer and Multi-Head Attention

• Each attention head projects embeddings into queries, keys, and values; scaled dot-product attention weights values before head outputs are concatenated and projected.The attention map encodes token-to-token weights.Source: §2.1 · pp. 3–4

2.2 Inference Phases: Prefill and Decode

• Prefill processes the prompt in parallel and produces the first token, whereas decode generates subsequent tokens autoregressively, one at a time.Prefill latency is TTFT; decode is typically memory-bound.Source: §2.2 · p. 4

2.3 Prefix Caching

• Prefix caching retrieves stored KV tensors to bypass redundant computation for shared prefixes, but traditionally requires identical model architectures and weights [33] [84].Source: §2.3 · p. 4

2.4 Multi-LLM Systems

• Multi-LLM routers assign queries to models suited to their complexity, so identical prompt prefixes may be processed by different models [7] [10].Source: §2.4 · p. 4

2.5 Sparse Attention

• Attention is sparse in practice, with |S| ≪ k enabling near-linear storage with minimal precision loss; MapFlow targets prefill acceleration through sparse attention-map reuse [71] [78].Other methods accelerate prefill by finding sparse patterns or focus on decode after full prefill [28] [34] [42] [52].Source: §2.5 · pp. 4–5

2.6 Cross-model Communication

• Cross-model communication approaches send raw text or reuse KV states through partial recomputation or learned adaptation [73] [16] [75] [32] [13].Raw-text transfer requires full receiver prefill; some learned approaches still require both models to compute KV caches.Source: §2.6 · pp. 5–6

3.1 Terminology

• Table 1 collects the mathematical notation used throughout the paper.The section introduces the notation before discussing cross-model reuse.Source: §3.1 · p. 6, Table 1 · p. 6

3.2 Challenges in Cross-Model Reuse

• Heterogeneous layer and head dimensions defeat homogeneous reuse designs, while learned KV adapters can erase latency gains and KV dependence on model weights and dimensions makes accurate translation difficult.KV distributions can differ even between models fine-tuned from the same checkpoint.Source: §3.2 · p. 6

3.3 Attention Maps as Reuse States

• Attention maps capture token dependencies and may be more structurally similar across models sharing a tokenizer and context than model-specific KV states.The proposed reusable state is the attention map rather than KV tensors.Source: §3.3 · p. 6
• Transferability depends on the model pair and layer mapping, so MapFlow calibrates and validates it offline rather than assuming it holds universally.An appropriate mapping can linearly approximate receiver-layer attention from corresponding sender-layer maps.Source: §3.3 · p. 7
• Simplex-constrained weights form a nonnegative, normalized convex mixture of sender heads, enabling lightweight linear mapping that can be fused with sparse retrieval and value aggregation.Calibration fits weights on a small set to diagnose transferability and guide layer selection; deployed weights are optimized using receiver end-to-end response loss.Source: §3.3 · pp. 7–8
• Early receiver layers fit poorly, so MapFlow recomputes them and reuses later layers; increasing recomputation from 3 to 15 raised TTFT from 0.3205s to 0.3556s without ROUGE-L improvement.The system profiles layer-wise TV distance offline to select the recomputation boundary and uses suffix alignment for layer mapping.Source: §3.3 · p. 8

4 Data Flow in MapFlow

• MapFlow accelerates prefill through attention reuse, organizing its data flow around partial recomputation, layer mapping, and head mapping.These are the system’s three core components.Source: §4 · p. 8
• The receiver computes its first r layers normally, then skips heavy attention computation and reuses cached sender attention maps for subsequent layers.The first r layers provide accurate receiver hidden states.Source: §4 · p. 8
• Layer mapping selects a sender layer for each reused receiver layer, while linear head mapping computes each receiver head from sender attention maps.The formal description identifies learned linear transformation weights for head mapping.Source: §4 · p. 8
• Top-p compression stores attention maps in Block Sparse Row format to address their O(N^2) storage cost.The retained blocks reduce dependence on sequence length from quadratic to near-linear in the reuse path when the average retained positions k are much smaller than L.Source: §4 · p. 9

5 MapFlow System Design

• MapFlow addresses quadratic attention-map storage and costly receiver-side reconstruction through block-wise compression, prefix-hash cache management, and fused retrieval.The fused GPU kernel combines receiver attention reconstruction with value-vector multiplication without intermediate dense materialization.Source: §5 · p. 10

5.1 Offline Calibration and Training

• For each sender–receiver pair, MapFlow trains only head-mapping weights while freezing both LLMs, optimizing them with receiver end-to-end response loss.For Ministral3-8B→Ministral3-3B on Conversation, weights account for less than 0.001% of receiver parameters and converge within two 20-hour epochs on two A800 80GB GPUs.Source: §5.1 · pp. 10–11

5.2 Attention State Compressor

• The compressor applies block-level top-p sparsity and stores retained attention blocks in BSR format, avoiding full dense materialization and reducing memory footprint and transmission payload.Compression runs once after the sender produces the map and does not block its critical path; the compressed map can serve multiple receivers.Source: §5.2 · pp. 11–12

5.3 Block Manager

• Because retained attention blocks have dynamic shapes across sequence positions and layers, the Block Manager uses block-wise indexing and prefix hashes to retrieve cached blocks during receiver partial recomputation.The cache server maps each prefix hash to block indices across sender layers and streams hit blocks to the receiver GPU.Source: §5.3 · p. 12

5.4 Attention Retrieval with Fused Kernel

• A fused GPU kernel reconstructs receiver attention from sparse sender maps and mapping weights, then applies it to receiver values without materializing a dense attention matrix in global memory.Fusion keeps intermediate attention scores on-chip and eliminates the need to instantiate receiver attention in HBM.Source: §5.4 · pp. 13–14

6 Implementation

• The 5K-line Python server has sender, receiver, and cache-server modules; sender and receiver integrate with vLLM, and the fused retrieval kernel is written in Triton.The kernel supports PagedAttention, Chunked Prefill, and Variable Length Sequences Packing.Source: §6 · p. 14
• A preallocated GPU buffer registered for IPC shares tensor data through negotiated offsets, amortizing handle creation and avoiding per-transfer CUDA API overhead.Modules signal tensor availability through lightweight metadata messages.Source: §6 · p. 14
• The cache server reconstructs and compresses attention maps on the GPU once during caching, allowing multiple receiver instances to reuse them without recomputation or recompression.Source: §6 · p. 14
• The evaluated implementation assumes co-located deployment, but the method can also transfer compressed block-sparse maps to a receiver in distributed settings for local fused retrieval.The co-location assumption is used to isolate the computational benefit of attention-map reuse.Source: §6 · pp. 14–15

7.1 Setup

• Evaluation uses Qwen3 and Ministral3 model pairs and compares MapFlow with full prefill, cross-model KV reuse, sparse prefill, and attention-map reuse baselines.DroidSpeak was not compared because its embedding transfers are infeasible for heterogeneous pairs with differing hidden dimensions.Source: §7.1 · p. 15

7.2 Prefill Latency–Generation Quality Trade-off

• 15%–20% lower prefill delay than full prefill across four model pairs and three datasets, without compromising generation quality.Compared with FlexPrefill, MapFlow further reduces prefill delay by 7%–12% with better or equal generation quality; it outperforms IAM on latency and quality in most settings.Source: §7.2 · p. 15, Figure 7 · p. 16
• Performance depends on model pair and task: Ministral3 pairs preserve quality more consistently than Qwen3 pairs, while QA is more sensitive to attention errors than conversation and summarization.The reported trends favor calibrating reuse per model pair and workload rather than enabling it universally.Source: §7.2 · p. 16

7.4 End-to-End Throughput Improvement on Multi-turn Conversation

• Up to 25% higher throughput at the same P90 TTFT, with lower P90 TTFT across arrival rates and reduced tail latency for large-to-small routed turns.The TTFT distribution shifts left across evaluated model pairs, with the gap widening at the upper tail.Source: Figure 8 · p. 17, §7.4 · pp. 17–18

7.3 Long-Context Evaluation

• On 20K RULER contexts, MapFlow achieves better F1 than FlexPrefill, IAM, and MInference for all four model pairs while maintaining competitive prefill latency.Latency speedups are less significant than on short datasets because the long-context setting uses larger top-p values.Source: §7.3 · pp. 16–17, Table 2 · p. 16

7.5.1 Memory Footprint.

• MapFlow’s memory footprint initially increases with sequence length, then plateaus after 12k tokens; memory usage remains stable across all top-p settings in the 12k–24k-token range.Source: §7.5.1 · p. 18

7.5.2 Quality Loss.

• Qwen3 1.7B quality falls from 0.286 to 0.092 as top-p drops from 0.85 to 0.8, so sparsification thresholds require per-model calibration.Quality degrades gracefully at top-p=0.9 for most tested models, but drops sharply below top-p=0.85 for several models.Source: §7.5.2 · p. 19, Table 4 · p. 19

7.6 Generalizability Evaluation

• Transfer weights trained on HotpotQA retain similar generation quality when evaluated zero-shot on 2WikiMultihopQA, without retraining.The experiment suggests the weights capture model-pair-level attention correspondence rather than dataset-specific behavior.Source: §7.6 · p. 19

7.7 Case Study of Different Routing Thresholds

• MapFlow delivers speedups across transition probabilities, with the most pronounced improvements at higher probabilities of large-to-small transitions.Higher transition probability increases the large-model-inherited TTFT while small-model-inherited TTFT stays relatively constant; more frequent large-model routing also raises baseline latency while improving generation quality.Source: Figure 10 · p. 20, §7.7 · p. 20

7.8 Single-Workflow Latency for Three Models

• 0.48–0.73 seconds saved per round, with up to 5.4% end-to-end speedup in a three-model Ministral3 workflow.The workflow routes through 14B, 8B, and 3B models; because MapFlow accelerates prefill rather than decode, its benefit is bounded when decoding dominates latency.Source: Figure 11 · p. 21, §7.8 · p. 21

• LLM-driven database workflows process redundant schemas, system constraints, and multi-turn instructions, motivating cross-model state reuse to reduce prefill costs.Source: §8 · p. 21
• Routing, prefix caching, adapter serving, and heterogeneous model multiplexing address different parts of multi-model serving, but the surveyed frameworks remain unoptimized for shared prompts.Source: §8 · p. 22
• KV-cache paging, prefix sharing, compression, and eviction improve cache management, yet these approaches remain tied to KV tensors and are orthogonal to MapFlow’s attention-map design.Source: §8 · p. 22

Applicability beyond shared tokenization

• Shared-tokenizer, canonicalized-context deployments are MapFlow's most effective setting; mismatched tokenizers, prompt canonicalization, or more divergent architectures remain broader challenges.The paper identifies token-level correspondence recovery and intermediate semantic anchoring as possible alignment strategies.Source: §9 · p. 22

Compact representations for reusable attention state

• Reusable attention states can still occupy nontrivial memory in long-context, high-reuse workloads despite block-level Top-p compression.Proposed directions include quantized sparse or low-rank representations and sharing one sender-layer cache across semantically aligned receiver layers.Source: §9 · pp. 22–23

Overall conclusion

• The stated limitations leave attention maps a promising systems abstraction for heterogeneous prefix reuse.They also point to a broader design space for reusable cross-model execution states.Source: §9 · p. 23

Conclusion

• 15% to 20% reduction in Time-to-First-Token and up to a 25% increase in end-to-end throughput for multi-turn routing, while maintaining generation quality.The conclusion attributes these results to MapFlow's evaluated system.Source: §10 · p. 23
• MapFlow shares semantic attention maps across different LLMs to reuse prefixes and address redundant prompt processing in multi-LLM systems.The system combines a lightweight linear adapter, block-level Top-p sparse compression, and a custom fused kernel.Source: §10 · p. 23
• MapFlow is presented as a scalable, memory-efficient solution for accelerating complex multi-model workflows.The conclusion frames this as the system's practical contribution.Source: §10 · p. 23