PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling Agents to Communicate Entirely in Latent Space

Zhuoyun Du, Runze Wang, Huiyu Bai, Zouying Cao, Xiaoyong Zhu, Yu Cheng, Bo Zheng, Wei Chen, Haochao Ying

arXiv:2511.09149v5cs.LGcs.AIcs.MA

TL;DR

Natural-language messages can lose information from agents’ rich internal states, motivating communication beyond discrete tokens. Interlat sends last-layer hidden states between agents and learns to compress them in latent space. Across experiments, latent communication improves task performance and communication efficiency, including with heterogeneous models; compressed messages can maintain competitive performance with substantially lower latency.

  • Problem

    Natural-language communication down-samples rich internal states into discrete messages, discarding alternative reasoning paths and contributing to ambiguous, lossy coordination.

    Source: §1 · p. 1

  • Method

    Interlat directly transmits temporally aligned last-layer hidden states as agent messages and uses latent-space reasoning to compress them into compact prefixes.

    Source: §1 · p. 1

  • Results

    Across experiments, latent communication improves task performance and communication efficiency, including across heterogeneous models; compression to as few as 8 tokens maintains competitive performance with up to a 24× reduction in communication latency.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The results support latent-state communication as a viable, efficient complement to language-based interaction in LLM-powered agent systems.

    Source: §7 · p. 10, §6 · p. 10

    Evaluation focuses on controlled two-agent ALFWorld and MATH settings, and the method requires accessible last-layer states, making it primarily applicable to open or inspectable models.

    Source: §7 · p. 10

Abstract

from arXiv · show

While natural language is the de facto communication medium for LLM-based agents, it presents a fundamental constraint. The process of downsampling rich, internal latent states into discrete tokens inherently limits the depth and nuance of information that can be transmitted, thereby hindering collaborative problem-solving. Inspired by telepathy, which bypasses symbolic language in communication, we propose Interlat (Inter-agent Latent Space Communication), a paradigm that leverages the continuous last hidden states of an LLM as a representation of its thought for direct communication (termed latent communication). An additional learned compression process further compresses latent communication via latent space reasoning. Experiments demonstrate that Interlat outperforms both fine-tuned chain-of-thought (CoT) prompting and single-agent baselines, even across heterogeneous models, promoting more exploratory behavior and enabling genuine utilization of latent information. Further compression not only substantially accelerates inference by up to 24 times but also maintains competitive performance through an efficient information-preserving mechanism. We position this work as a feasibility study of entirely latent space inter-agent communication, and our results highlight its potential, offering valuable insights for future research. Our code is available at https://github.com/XiaoDu-flying/Interlat.

1 Introduction

Natural-language communication compresses agents’ rich internal states into discrete, linear messages, potentially losing alternative reasoning paths and task-relevant information. Interlat instead communicates through hidden states and adds a learned process for compressing latent messages; experiments report exploratory behavior and faster communication in a controlled two-agent setting.

Motivation

  • Natural-language messages downsample rich internal states into discrete, often redundant sequences, discarding alternative reasoning paths and contributing to ambiguity and coordination failures.Prior work on hidden-state communication either uses one-shot activation grafting or remains tied to language trajectories, often requiring ad-hoc layer selection.Source: §1 · p. 1

Interlat

  • Interlat transmits temporally aligned last-layer hidden states as representations of an agent’s thoughts, rather than decoded language tokens.A supervised objective encourages the receiver to interpret and use task-relevant latent information, with stochastic token–latent mixing to stabilize training.Source: §1 · pp. 1–2

Latent compression

  • A separate autoregressive reasoning process generates compact latent messages, compressing long trajectories into prefixes while retaining task-critical information for downstream execution.The generation process operates directly in latent space without decoding to language tokens.Source: §1 · p. 2

Experimental findings

  • In a two-agent sender–receiver setting, latent communication is associated with more exploratory behavior; messages as short as 8 tokens maintain competitive performance with up to a 24× reduction in communication latency.The experiments exclude tool use, retrieval, and multi-round debate.Source: §1 · p. 2

2 Related Work

Prior work explores latent reasoning within individual models and communication methods between agents. Interlat shifts the focus to direct, compressed latent communication between agents.

Latent Reasoning in LLMs

  • Latent-reasoning studies move computation beyond discrete language tokens through pause or filler tokens, KV-cache modification, feedback of hidden states, and modular frameworks.The paper positions its focus as extending latent reasoning from single-model computation to inter-agent communication.Source: §2 · p. 2

Multi-agent Communication

  • Prior multi-agent approaches include emergent protocols learned from scratch and methods that transmit tokenizer embeddings or hidden activations (Lazaridou et al., 2016, 2018; Tucker et al., 2021, 2022; Pham et al., 2023; Ramesh and Li, 2025).Other methods communicate latent deltas tied to language trajectories or infer latent thoughts with an autoencoder (Tang et al., 2025; Zheng et al., 2025).Source: §2 · p. 2
  • Interlat directly transmits temporally aligned last hidden states, compresses them for language-free communication, and requires no parameter sharing, memory coupling, or cache synchronization.The paper contrasts this design with prior approaches and describes it as preserving agent autonomy while expanding effective communication bandwidth.Source: §2 · p. 2

3 Interlat

Interlat represents an agent’s generated message with its last-layer hidden states and passes those states directly to another agent. Its training procedure combines task supervision with objectives and a curriculum intended to encourage use of task-relevant latents.

3 Interlat

  • For a prompt and completion sequence, Interlat collects the last-layer hidden states immediately before each completion token is predicted.The resulting state sequence represents the agent’s generated message for inter-agent communication.Source: §3 · p. 2

3.1 Latent Communication

  • Interlat replaces a discrete message with collected hidden states, marked by beginning- and end-of-communication tokens and processed through an adapter before use by the receiving agent.The adapter uses lightweight self-attention and a projection layer to rescale and interpret the latent communication.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 Training Procedure

  • The setup assigns a reasoning agent to send a task-specific plan and hidden states, while an actor agent uses the communication to generate actions.The paper presents this sender–receiver arrangement as a building block for more complex multi-agent systems.Source: §3.2 · p. 3
  • Actor training combines supervised task loss with conditional separation between outputs given matched and mismatched latents, encouraging use of task-relevant information.The task loss is cross-entropy; the separation objective uses weighted Jensen–Shannon divergence.Source: §3.2 · p. 3
  • Plan-aligned regularization constrains predictions conditioned on latents using predictions conditioned on the corresponding language-space plan, while a stochastic token-to-latent curriculum replaces early positions with plan-token embeddings.The curriculum addresses instability when learning to interpret latents from scratch.Source: §3.2 · pp. 3–4

4 Information Compression

Long latent messages introduce substantial communication latency, motivating a learned bottleneck that compresses them while retaining utility for downstream agents. A separate model generates compact messages entirely in latent space and is trained with task, agreement, and geometry objectives.

4 Information Compression

  • Long latent messages can span dozens to hundreds of steps, creating communication latency; the goal is to compress them while retaining downstream utility.Latent states are described as over-parameterized relative to task requirements, with much of their temporal structure potentially redundant.Source: §4 · p. 4
  • A separate reasoning model generates compact latent messages autoregressively, feeding each last hidden state back as the next input embedding rather than truncating or subsampling the original message.The actor and communication adapter remain frozen, and only the compression model is updated, isolating compression from changes in actor behavior.Source: §4 · p. 4
  • The composite training objective includes actor cross-entropy for downstream task utility and uncertainty-weighted agreement that selectively aligns behavior under compressed and full-length communications.The agreement objective emphasizes positions where latents reduce predictive uncertainty while avoiding over-regularization where they are uninformative.Source: §4 · p. 4
  • A latent geometry alignment loss preserves the global semantic orientation of the original communication in the compressed space, alongside the task and agreement objectives.Together, the objectives encourage an information-preserving bottleneck that discards redundant temporal structure while retaining task-critical functional and geometric properties.Source: §4 · p. 5

5 Experiments

Experiments compare Interlat with single-agent and natural-language communication baselines, then examine symbolic reasoning, exploration, compression, training components, and multi-agent topologies. The study uses Alfworld and MATH and evaluates base models to isolate benefits from instruction-tuning priors.

5.1 Main Results

  • Interlat improves task-solving performance over both fine-tuned single-agent baselines and agents trained to communicate in natural language.The comparison is reported in Table 1.Source: §5.1 · p. 6
  • Cross-task latents substantially reduce performance, while perturbations that destroy higher-order structure further impair it, indicating reliance on task-specific latent content and geometry.Qwen-derived latents also yield stronger gains when used with a LLaMA actor, despite distinct latent manifolds.Source: §5.1 · p. 6

5.2 Compression Analysis

  • 24× speed-up reduces end-to-end latency from 9.19 s to 0.39 s with 8-step latents; a lightweight bridge reduces it further to 0.20 s.Compression training maintains higher, more stable success rates across latent lengths from 8 to 128 steps, unlike naive truncation under aggressive shortening.Source: Table 3 · p. 7, §5.2 · p. 7

5.1 Main Results

  • On MATH, Interlat slightly underperforms CoT on simpler problems but surpasses it on the most challenging Level 5 tasks.The authors attribute this pattern to preserving parallel hypotheses for broader latent-space search on complex problems.Source: §5.1 · p. 7

5.2 Compression Analysis

  • Trained latents preserve broader top-token support across reasoning steps, while untrained compression concentrates probability toward top-ranked tokens.The trained model has lower P50(S10), indicating broader support over plausible alternatives.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 Ablation Studies

  • Removing curriculum learning destabilizes actor training, removing separation loss encourages prompt-only shortcuts, and removing the communication adapter causes the largest task-completion drop.The adapter is described as bridging latent spaces and enabling latent interpretation.Source: §5.3 · p. 8, Table 4 · p. 9
  • For compression at K = 128, direction alignment is the most critical reasoning-model loss; removing cross-entropy hurts seen tasks but slightly improves unseen performance.The authors leave this in-distribution versus generalization trade-off for future investigation.Source: §5.3 · p. 8

5.4 Generalization to Complex Topologies.

  • 62.85% vs. 61.19% on seen tasks for the parallel tree: both three-agent latent configurations outperform matched natural-language CoT structures and the original two-agent Interlat baseline.The tree configuration supports parallel reasoning without pair-specific training.Source: §5.4 · p. 9, Table 5 · p. 9

6 Conclusion

Interlat communicates and reasons over latent states, with reported gains in task performance and communication efficiency, including across heterogeneous models. The authors present latent states as a viable, efficient, and generalizable medium for multi-agent systems.

6 Conclusion

  • Directly transmitting and reasoning over latent states improves task performance and communication efficiency, including across heterogeneous models.Latent-space reasoning compresses messages while retaining parallel hypotheses and discarding redundant structure.Source: §6 · p. 9
  • Latent-space reasoning forms compact representations that retain parallel hypotheses while discarding redundant structure.The conclusion describes this as an aggressive compression of full-length latent exchange.Source: §6 · p. 9
  • The authors highlight latent states as a viable, efficient, and generalizable medium for next-generation multi-agent systems.They suggest communication need not be bound to language tokens.Source: §6 · pp. 9–10

7 Limitations

The authors identify limits in evaluation scope, model access, and human interpretability. They frame the findings as preliminary empirical evidence for latent coordination and task-preserving compression.

7 Limitations

  • The evaluation centers on a controlled two-agent setting and two benchmarks, which do not capture the diversity of real-world multi-agent systems.The stated gaps include larger teams, dynamic roles, long-horizon collaboration, tool use, and external memory.Source: §7 · p. 10
  • Interlat assumes access to last-layer hidden states, limiting its current applicability primarily to open or inspectable models.Extending it to restricted-access settings remains an open challenge.Source: §7 · p. 10
  • Latent messages are not directly human-readable, complicating debugging, monitoring, and failure analysis.The authors describe interpretability analysis via PCA as preliminary and call for principled interpretability and controllability tools.Source: §7 · p. 10
  • The authors frame the findings as initial empirical evidence that latent communication supports coordination and compression can preserve task utility.They position it as a complementary paradigm to language-based interaction in LLM-powered agent systems.Source: §7 · p. 10

8 Ethical Considerations

The ethics statement limits the study to simulated experiments and reports that harmful instructions were neither trained nor evaluated. The compression appendix describes objectives that preserve task utility and align compressed latents with full-length latents through the actor’s behavior.

8 Ethical Considerations

  • Experiments used simulated environments and standard dataset splits; no harmful instructions were trained or evaluated, and no harmful actions occurred.The paper notes a potential risk that latent communication could circumvent language-based safety mechanisms.Source: §8 · p. 10

Appendix

  • The appendix provides additional data, explanations, and details beyond the main paper.Source: Appendix · p. 13

A LLM usage

  • ChatGPT1 was used for language-only editing, and generated content was manually reviewed and approved by the authors.The stated uses were spellchecking and paraphrasing, without suggesting new content.Source: A LLM usage · p. 13

B Compression Loss

  • The compression objective aligns compressed and full-length latents through the frozen actor’s behavior, rather than reconstructing latent representations.This supports variable-length messages and abstraction across reasoning steps without assuming latent invertibility or step-by-step correspondence.Source: B Compression Loss · pp. 13–14
  • Actor-side direction alignment preserves representation geometry and improves stability, mitigating mode collapse when the compressed message length K is small.The actor cross-entropy term also requires compressed latents to support correct next-token predictions, discouraging compression that leaves the actor unable to use the message.Source: B Compression Loss · pp. 13–14

B.1 Baselines and settings in Interlat.

The section specifies language-based and no-communication baselines, then describes controlled changes to the latent channel and a cross-family evaluation. It also outlines Alfworld’s sequential household-task setting and a curriculum-removal diagnostic.

B.1 Baselines and settings in Interlat.

  • Full-CoT models receive complete plans and are fine-tuned, while No-CoT models answer directly; the former serves as a language-communication upper-bound comparison.Baselines use the same base models as Interlat, differing in whether and how communication is provided.Source: §B.1 · p. 14
  • Text, No-Comm, and CrossTask variants test language versus latent communication, communication’s contribution, and whether the actor uses task-specific latent content.CrossTask substitutes a latent message sampled from a different task; a substantial degradation indicates reliance on meaningful latent information rather than superficial distributional shortcuts.Source: §B.1 · pp. 14–15
  • Noise, covariance-matched Gaussian, and random-rotation variants probe robustness and whether latent semantics depend on trajectory geometry, higher-order structure, or temporal organization.RandomRot preserves the latent distribution’s first two moments while scrambling geometric orientation and higher-order structure.Source: §B.1 · p. 15
  • Removing curriculum learning produces highly unstable optimization dynamics, as shown in Figure 7.Source: Figure 7 · p. 15
  • Cross-Family evaluation sends Qwen2.5-7B-Instruct latents to a LLaMA3.1-8B-Base actor without shared parameters or tokenizer vocabularies.The setting tests whether latent messages carry task-relevant information across model families.Source: §B.1 · p. 15

C.1 Alfworld

  • Alfworld simulates partially observable, long-horizon household tasks through text, with official train/seen/unseen splits and success measured under a fixed step budget.Agents observe textual scenes and issue commands from a constrained action set.Source: §C.1 · p. 15

D Ablations and Step Analysis

Ablations show that success rate must be read alongside step count: successful full-model trials involve longer trajectories, while some unseen-task ablations take longer and succeed less. Removing curriculum learning destabilizes training; compression results are organized across tasks, ratios, and latent lengths.

D Ablations and Step Analysis

  • Removing curriculum learning destabilizes actor training and substantially degrades latent comprehension, preventing consistent use of communicated information.Figure 7 illustrates the unstable training dynamics.Source: D Ablations and Step Analysis · p. 16
  • On seen tasks, ablations lower overall success despite often taking fewer steps on completed trials; on unseen tasks, some ablations take more steps while achieving lower success.The full model combines higher seen-task success with longer trajectories; the authors interpret these as productive exploration, while longer unseen-task ablation trajectories can reflect unstructured search.Source: D Ablations and Step Analysis · p. 16

E Compression Result

  • Compression results cover seen and unseen accuracy, steps, and latency across tasks, with performance trends also examined by retained ratio and latent length.Latency comparisons use the same machine and decoding policy across rows.Source: E Compression Result · p. 16, Figure 8 · p. 16

F Latent Parallelism Analysis

The parallelism analysis finds that trained latents retain a broader distribution of plausible reasoning paths rather than collapsing toward a single leading hypothesis. The analysis reports this pattern across both six-step and 32-step horizons.

F Latent Parallelism Analysis

G Qualitative Analysis of Latent Communication via PCA

PCA reveals task-dependent structure in latent communications, while the communication adapter shifts representations in template-dependent directions. The actor training template places either latent communication or a natural-language plan before iterative environment interaction.

G Qualitative Analysis of Latent Communication via PCA

  • The PCA analysis projects mean-pooled last-layer reasoning-agent states from 3,119 ALFWorld training samples onto the first two principal components.The samples are grouped by six official task templates.Source: G Qualitative Analysis of Latent… · p. 19
  • PCA shows task-dependent clusters: procedural thermal tasks occupy adjacent but separable regions, perception tasks form localized concentrations, and action-centric tasks cluster centrally.Within-template dispersion also indicates retained variation across instances rather than collapse to one prototype.Source: G Qualitative Analysis of Latent… · p. 19, Figure 11 · p. 21
  • Template-dependent adapter shifts selectively reorganize latent representations, with larger relative shifts for execution-heavy tasks and more stable observation-oriented tasks.The shifts are moderate relative to within-template dispersion, suggesting preservation of overall geometry rather than global rescaling.Source: G Qualitative Analysis of Latent… · p. 20, Figure 12 · p. 21, Table 9 · p. 21

H Training Template

  • The actor training template inserts either latent communication or a natural-language plan after the instruction, then proceeds through actions and environment feedback until the task is completed.Figure H illustrates this training-data structure.Source: H Training Template · p. 20
Loading 2511.09149v5…