PaperlayerView exact revision on arXiv

Source-linked AI summary

See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents

Siyi Chen, Xiaoyan Zhang, Meng Wu, Jonathan Tremblay, Valts Blukis, Stan Birchfield, Rene Vidal, Alvaro Velasquez, Sijia Liu, Qing Qu

arXiv:2606.13594v1cs.MA

TL;DR

Text-based handoffs in multi-agent systems are lossy and expensive, while heterogeneous KV-cache communication still lacks effective cross-model alignment. This paper introduces dense cache alignment with two-phase training, outperforming heterogeneous baselines, matching or exceeding text communication at 2–3× lower compute in context-aware settings, and remaining effective without receiver input.

  • Problem

    Multi-agent systems rely on lossy, expensive text handoffs, while heterogeneous latent communication lacks evidence of cross-model alignment that transfers both source context and reasoning.

  • Method

    The paper combines compressed-sensing analysis with dense heterogeneous KV-cache alignment using per-head transformations, positional disentanglement, gating, and reconstruction-then-generation training.

  • Results

    Across six Qwen3 model directions and six benchmarks, the method surpasses heterogeneous baselines, matches or exceeds text communication at 2–3× lower compute, and remains accurate without receiver input.

  • Takeaways & Limitations

    Context-unaware transfer tests whether dense alignment preserves task-critical knowledge when the receiver has no access to the source context.

  • Takeaways & Limitations

    The method requires one training pass per sender–receiver pair, limiting scalability to open-set pairings and shared transforms across many senders.

Abstract

from arXiv · show

Multi-agent systems communicate mostly through text, paying a lossy and expensive decode and re-encode cost. KV-cache communication is a promising alternative, yet most prior work is homogeneous, using duplicate copies of the same model, and avoids the central challenge of cross-model latent alignment; existing heterogeneous methods are also restrictive, typically assuming shared input and using transferred caches mainly for steering. We study a more fundamental question: can heterogeneous agents be aligned well enough to perform real "mind reading" and transfer both what one agent sees and how it thinks? Our information-structure analysis reveals a duality: context-aware transfer is driven by sparse reasoning signals, while context-unaware transfer, where the receiver sees no input, requires dense contextual knowledge preservation. Motivated by this, we propose dense alignment for heterogeneous KV-cache communication via a lightweight cross-model cache transformation and two-phase training: reconstruction followed by generation. Across all six directions of {Qwen3-4B, 8B, 14B} and six in-domain and out-of-domain benchmarks, our method outperforms prior heterogeneous baselines, matches or exceeds text communication in context-aware settings at roughly 2 to 3 times lower compute, and remains effective in context-unaware transfer where prior methods collapse.

1 Introduction

The paper addresses the difficulty of transferring latent information across heterogeneous agents, where existing KV-cache methods largely assume homogeneous models. It proposes dense alignment to preserve both reasoning and contextual knowledge, enabling efficient transfer in context-aware and context-unaware settings.

  • Existing latent communication is dominated by text, which incurs decoding overhead and limits computation reuse across specialized agents.Latent methods instead exchange embeddings, hidden-state trajectories, or KV caches to reduce decoding overhead and reuse computation.
  • Cross-model KV-cache transfer is difficult because heterogeneous agents differ in layer depth, head structure, channel geometry, and positional encoding.Prior latent communication is largely restricted to homogeneous replicas whose latent representations are naturally aligned.
  • The method supports direct context-unaware transfer, allowing receivers to solve tasks solely from sender latents and reuse transmitted representations without re-encoding the original context.In context-aware communication, it surpasses sparse-steering heterogeneous baselines and is more compute-efficient than text communication.
  • The paper identifies a duality: context-aware transfer needs sparse reasoning signals, whereas context-unaware transfer requires dense contextual knowledge preservation.Context-unaware communication follows a stricter protocol in which the receiver sees no input and must rely on transmitted latents.
  • Dense alignment uses a lightweight cross-model KV-cache adapter with positional disentanglement, fine-grained per-head transformations and selection, and reconstruction-then-generation training.The framework maps sender caches into receiver-compatible caches while preserving reasoning and contextual information.

2 Background and Problem Setup

This section defines latent communication in multi-agent systems and frames heterogeneous cache transfer as an alignment problem. It distinguishes context-aware communication, where caches support reasoning over shared input, from context-unaware communication, where caches must carry source information and reasoning state alone.

  • Multi-Agent Systems: Multi-agent systems coordinate specialized agents such as planners, retrievers, executors, and verifiers to solve tasks beyond a single model’s capabilities.The system is formally represented as a set of distinct agents, each potentially serving a specialized role.
  • Latent Communication: Latent communication transfers intermediate states such as KV caches directly, avoiding text decoding and re-encoding while reusing previously computed attention states.KV caches store key and value tensors across transformer layers and attention heads, summarizing processed context and avoiding O(N^2) historical recomputation during decoding.
  • Heterogeneous Alignment: Direct cache transfer works naturally for homogeneous agents but fails across heterogeneous models because latent representations are tied to incompatible internal coordinate spaces and architectures.Homogeneous agents share architectures, layer configurations, and hidden dimensions; heterogeneous agents may differ in architecture, capabilities, information access, or foundational design.
  • Heterogeneous Alignment: Existing heterogeneous methods remain limited by aligning text embeddings instead of KV caches, assuming shared inputs, or relying on restrictive communication mechanisms.These limitations constrain general communication beyond the settings addressed by prior approaches.
  • Communication Regimes: Context-aware communication combines the original input with transferred caches for generation, whereas context-unaware communication generates solely from caches that must preserve contextual evidence and reasoning state.The context-aware regime treats the cache primarily as a reasoning signal; the context-unaware regime treats it as a self-contained carrier of source-side information.

3 The Information Bottleneck: Sparse Reasoning vs. Dense Knowledge

Compressed-sensing analysis reveals a duality in latent communication: context-aware transfer needs only sparse reasoning signals, whereas context-unaware transfer requires dense cache information to preserve task-critical context. This duality motivates training and transformation mechanisms designed to preserve dense knowledge while prioritizing structurally important reasoning signals.

  • Compressed-sensing analysis: Post-hoc compressed sensing estimates KV-component importance by ablating random sender-head subsets, fitting contribution coefficients, and aggregating them into KV-group scores.The analysis uses homogeneous Qwen3-4B self-communication to isolate information structure without cross-model misalignment.
  • Information duality: Context-aware communication achieves near-full performance with only a small KV-cache fraction because the receiver’s input already supplies core task information.The transmitted cache primarily carries a sparse reasoning signal, and CS-derived rankings outperform uniform random selection.
  • Information duality: Context-unaware communication requires a dense cache fraction because the receiver lacks input context and must preserve contextual knowledge in the transmitted representation.Performance stays at chance or zero under most dense-compression levels before a sharp phase transition as more KV groups are retained.
  • Framework design: The dual information structure motivates two-stage training: cache reconstruction preserves dense contextual and reasoning information, while generation training makes that signal actionable.This design directly addresses the distinct demands of sparse reasoning and dense knowledge preservation.
  • Framework design: Per-head learnable transformations and gating recover structurally important head signals, while positional disentanglement prevents rotary embeddings from interfering with content alignment.Positional information is stripped before transformation and restored afterward.

4 Design of Dense Latent Communication

The method aligns heterogeneous sender caches with a receiver-compatible transformation designed for both context-aware reasoning transfer and context-unaware dense context substitution. It combines reconstruction-then-generation training with position-disentangled, depth- and KV-group-aware parameterization.

  • Dense alignment: Dense alignment addresses complementary information structures: sparse reasoning signals steer receivers with source context, while dense knowledge preserves missing context for input-free receivers.The same cache interface serves both communication regimes, providing reasoning signals alongside the receiver’s input or substituting for unavailable source context.
  • Two-phase training: The adapter 𝒯θ maps a sender cache into a receiver-compatible cache and is trained in two phases: receiver-cache reconstruction followed by generation-oriented optimization.Reconstruction encourages reusable receiver-native latent alignment before downstream generation tuning.
  • Two-phase training: Phase II jointly trains context-aware and context-unaware prompts so the aligned cache functions as either a structured reasoning signal or a dense replacement for missing context.This makes the transformed cache actionable rather than merely similar to the receiver’s internal states.
  • Cache transformation: Position disentanglement separates cache content from model-specific rotary phase rotations before restoring the receiver’s positional convention.This treats position handling as part of the communication interface rather than directly copying the sender’s positional convention.
  • Cache transformation: The transformation aligns differing model depths monotonically and produces receiver KV groups through routed, learnably gated per-group key/value transformations.Structured gates weight reliability within a dense channel: sparse signals may concentrate in selected groups, while context-unaware transfer requires broad knowledge preservation.

5 Experiments

Experiments across six Qwen3 model directions and six in-domain/OOD benchmarks show that dense cache transformation matches or exceeds text communication in context-aware settings while using 2–3× less compute. In context-unaware transfer, it preserves accuracy within 0–10 pp of context-aware performance, unlike prior methods that largely collapse.

  • Experimental Setup: The evaluation covers all six directions among Qwen3-4B, 8B, and 14B on three in-domain tasks and three held-out MCQ benchmarks.Baselines are Receiver-only, T2T text communication, and C2C learned cache transformation; both context-aware and context-unaware regimes are tested.
  • Context-Aware Results: Our cache transformation matches or exceeds T2T on every in-domain task across all six pairs and remains competitive on held-out benchmarks.The gains over T2T are +0.16 to +4.85 pp on GSM8K, +6.00 to +20.40 pp on MATH-500, and +1.34 to +3.94 pp on ARC-C; it is at-or-above T2T on OpenBookQA in 5 of 6 pairs.
  • Efficiency: 2–3× lower TFLOPs make the channel cheaper than T2T, and it is cheaper than Receiver-only in 5 of 6 directions.For 4B→14B, inference costs are 21.5 versus 56.2 TFLOPs; for 8B→14B, they are 21.8 versus 67.1, enabled by zero sender autoregressive reasoning and compact cache transfer.
  • Context-Unaware Results: In context-unaware transfer, T2T falls to 19–57% on GSM8K and MCQ chance on ARC-C, MedQA, and OpenBookQA, whereas dense transformation sustains accuracy within 0–10 pp of context-aware results.The receiver sees only the transferred signal; Ours-context-unaware ranges are GSM8K 81–91, MATH-500 64–82, ARC-C 87–94, MMLU-Redux 55–77, MedQA 53–65, and OpenBookQA 82–90.
  • Geometric Analysis: PCA projections show transformed sender caches overlapping receiver-native manifolds at both early and late layers, while untransformed caches occupy a disjoint region.This supports geometric alignment rather than a brittle decoder-fooling shortcut.

6 Conclusion · A Phase-II Trace Construction

The paper concludes that heterogeneous agents can transfer both observed context and reasoning through dense latent communication, with context-aware transfer relying on sparse reasoning signals and context-unaware transfer requiring dense knowledge preservation. Phase II trains receiver-native cache transformations using guided self-traces across mixed context regimes, while remaining limited by pair-specific training and privacy concerns.

  • 6 Conclusion: The central question is whether one heterogeneous agent can transfer both what it sees and how it thinks to another.
  • 6 Conclusion: Context-aware communication uses sparse reasoning signals, whereas context-unaware communication requires dense knowledge preservation when the receiver sees no input.
  • 6 Conclusion: Across all six Qwen3 directions, the method surpasses sparse-steering heterogeneous baselines and matches or exceeds text communication at 2–3× lower compute in context-aware settings.
  • 6 Conclusion: The method remains accurate in context-unaware settings where prior heterogeneous baselines collapse.
  • 6 Conclusion: One training pass is currently required per sender-receiver pair, making open-set pairings and shared transforms across many senders important future directions.
  • A Phase-II Trace Construction: Phase II uses receiver-self guided traces so the transformation learns receiver-native latent states rather than imitating another model’s wording or reasoning style.
  • A Phase-II Trace Construction: Trace generation uses GSM8K, MATH-algebra, and ARC-Challenge for training, while MATH-500, MMLU-Redux, MedQA, and OpenBookQA are reserved for evaluation.
  • A Phase-II Trace Construction: Receiver models generate step-by-step traces from source questions and gold answers, with pair-specific files and mixed context-aware or context-unaware prompts during training.The receiver prompt is context-aware with probability 0.5 and context-unaware with probability 0.5; Phase II runs 2000 optimization steps from the Phase-I reconstruction checkpoint.

B Efficiency Analysis and Per-Side Breakdown

This section measures system-level compute efficiency under each method’s canonical inference recipe and decomposes FLOPs into four per-side token counts.

  • Compute efficiency is measured at the system level using each method’s canonical inference recipe.
  • The FLOPs column in Tables 1 and 2 is broken into four constituent per-side token counts, including sender prefill ℓ𝑆.

B.1 Measurement recipe

The evaluation uses a shared, single-sample decoding setup with fixed sampling parameters and token budgets, while reporting exact token counts and a separately disclosed bandwidth cost. Results aggregate fixed-seed samples across six benchmarks, and headline TFLOPs use a standard forward-pass estimate that excludes a sub-5% attention contribution.

  • Sample composition: ≈102 samples per cell are pooled from 17 fixed-seed examples per benchmark across six benchmarks.The benchmarks are GSM8K, MATH-500, ARC-Challenge, MMLU-Redux, OpenBookQA, and MedQA; only mean per-token-count fields are reported.
  • Inference recipe: All methods use fixed sampling parameters, the same token budget, single-sample inference, and their strongest canonical configurations for fair comparison.Receiver-only and T2T use standard text-generation settings, while Ours follows its training reasoning style.
  • FLOPs estimate: 2 parameters per token estimates forward-pass FLOPs, with the attention O(Lkv) contribution excluded because it is below 5% of linear-projection cost.Including the attention term does not change headline-table rankings; NS and NR denote sender and receiver non-embedding parameter counts.
  • Bandwidth disclosure: 20–30 MB per sample is the approximate KV-cache payload for Ours, versus hundreds of bytes for equivalent text, and bandwidth is excluded from headline TFLOPs.This disclosure concerns network bandwidth, not compute.

B.2 Per-side breakdown

The per-side breakdown shows that Ours and C2C reduce compute by eliminating sender-side autoregressive reasoning, while differing in receiver decode length. Ours uses longer explicit reasoning than C2C, trading fewer FLOPs for substantially lower accuracy on most benchmarks.

  • Token-count structure: Table 3 reports four per-side token counts plus the unweighted total All tok for every pair, method, and mode cell.These counts expose the structural mechanism behind each method’s compute profile.
  • Sender-side computation: 220 tokens for Ours and 117 for C2C are the sender pre-reasoning lengths in w/ctx mode, both substantially below T2T’s roughly 1,100–1,300 re-encoded sender-text tokens.Both methods eliminate the sender’s autoregressive reasoning.
  • Accuracy–compute tradeoff: This receiver-length difference trades fewer FLOPs for substantially lower accuracy on most benchmarks.The passage directs readers to Tables 1 and ??.

B.3 Structural observations

Cache-based communication removes sender decoding and sharply reduces receiver prefill versus text-to-text communication. Ours maintains stable context-unaware decoding and favorable compute while preserving explicit reasoning in context-aware transfer.

  • Sender decoding: Ours and C2C eliminate sender decoding, whereas T2T spends ∼900–1,080 sender-decode tokens producing a ∼2,048-token sender CoT.Ours encodes the question directly into a sender KV cache with ℓ_S dec = 0; C2C transfers a cache without sender generation.
  • Receiver prefill: Receiver prefill is ∼6× smaller for cache-based methods than T2T, with Ours using ℓ_R pre = 220 (w/ctx) / 63 (context-unaware).C2C uses 117 / 14, while T2T re-encodes ∼1,100–1,300 sender-message tokens.
  • Receiver decoding: In w/ctx mode, Ours produces explicit step-by-step reasoning before answering, with ℓ_R dec ∼80–170.C2C’s lower decode length yields a ∼2–3× smaller TFLOPs total but substantially lower accuracy on most benchmarks.
  • Receiver decoding: In context-unaware mode, C2C’s receiver decode ranges from ℓ_R dec ∈[270, 1,817], while Ours remains stable at ℓ_R dec ≈400 across all six pairs.Ours-context-unaware is the cheapest cell in 5 of 6 pairs.

B.4 Notes

The notes show that Ours generally improves the FLOPs–accuracy tradeoff, while its efficiency depends on transfer direction and context mode. C2C minimizes compute but sacrifices substantial accuracy, whereas Ours matches or exceeds T2T across nearly every reported cell.

  • Efficiency by direction and mode: 9.18 TFLOPs for Receiver-only rises to 10.18 TFLOPs for Ours in 14B→4B w/ctx, while Ours-context-unaware costs 8.90 TFLOPs.In every other direction, Ours is cheaper than Receiver-only.
  • Accuracy / FLOPs Pareto: 70.58% versus 91.13% on 14B→4B GSM8K and 44.20% versus 82.00% on 4B→8B MATH-500 show C2C trailing Ours substantially.C2C achieves the lowest TFLOPs by emitting short answers, but its accuracy trails Receiver-only and Ours on most benchmarks.
  • Accuracy / FLOPs Pareto: Ours lies between C2C and T2T on FLOPs while matching or beating T2T on accuracy in nearly every cell, strictly beating T2T in the Table 1 Pareto frontier.The stated dominance consists of lower FLOPs and higher accuracy than T2T.
  • Per-side efficiency breakdown: Table 3 reports pooled mean per-sample sender and receiver token counts across six benchmarks, with TFLOPs computed using Appendix B.1’s formula.Sender counts are split into prefill and decode, as are receiver counts; all-token totals include every token touching either model.

C Compressed-Sensing Analysis Across Regimes … C.6.3 Takeaway for model design

The compressed-sensing analysis identifies task-relevant sender KV groups and shows a sharp contrast between sparse context-aware transfer and dense context-unaware transfer. These findings motivate preserving dense information while extracting structured sparse signals for heterogeneous communication.

  • C.1 Self-communication setup: Qwen3-4B self-communication ranks 1,152 sender query heads, aggregates four query-head scores per GQA group, and transmits selected groups without changing receiver attention.The cache contains 288 KV groups across 36 layers, with each KV head shared across four query heads.
  • C.2 Stage 1: CS head ranking: The first stage uses 200 stratified 5%-mask ablations over 1,152 query heads and fits Lasso importance scores from accuracy changes on GSM8K, MATH-algebra, and ARC-Challenge.Heads are ranked by the most negative fitted coefficients because masking them most degrades accuracy.
  • C.3 Stage 2: K-sweep on full test: The second stage retains K∈{10, 20, 50, 100, 150, 288} ranked KV groups under context-aware and context-unaware receiver prompts.The two regimes use identical rankings and K levels; only receiver access to the question differs.
  • C.4 Random-filter baseline: Random filtering selects K KV groups uniformly over three seeds, so its gap from CS filtering measures the value of learned head ranking beyond uniform sparsity.The random-filter markers report the mean across the three seeds.
  • C.5 Recovery-limit caveat: At most ∼70 KV groups inherit reliable signal from the Lasso recovery limit, making CS and random schemes converge and slightly invert near K=200.With M=200 and N=1,152, the underlying query-head recovery limit is s_max≈70–80; headline CS results therefore stop at K≤150.
  • C.6.1 Context-aware regime: At K=10, context-aware transfer matches the full-KV ceiling across all three tasks: GSM8K 0.883, ARC-Challenge 0.873, and MATH-algebra 0.699.The receiver’s question supplies most task information, limiting channel lift over K=0 to at most ∼10 pp on MATH-algebra and under 2 pp on ARC-Challenge.
  • C.6.2 Context-unaware regime: Context-unaware accuracy stays near chance through K≤150, rises sharply between K=150 and K=200, and approaches the channel ceiling by K=250.Examples include ARC-Challenge 0.27→0.79 and GSM8K 0.00→0.28 between K=150 and K=200; at K=250, the values are 0.834/0.904/0.686.
  • C.6.3 Takeaway for model design: Context-unaware evaluation is the stricter alignment test because the channel is the only task signal, motivating two-phase training, per-head transformation, and gating.The design dichotomy is sparse reasoning signals for context-aware transfer versus dense information preservation for context-unaware transfer.
Loading 2606.13594v1…