PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent Collaboration in Multi-Agent Systems

Jiaru Zou, Ruizhong Qiu, Gaotang Li, Xiyuan Yang, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, James Zou, Mengdi Wang, Ling Yang

arXiv:2511.20639v4cs.CLcs.AIcs.LG

TL;DR

Existing multi-agent systems rely on text to mediate reasoning and communication, leaving open whether agents can collaborate purely in latent space. LatentMAS answers this with training-free latent thought generation and shared working-memory transfer, and it outperforms strong baselines while reducing token use and inference time.

  • Problem

    Existing LLM-based multi-agent systems generally use text as the medium for agents’ internal thoughts and communication, motivating the question of pure latent collaboration.

  • Method

    LatentMAS combines autoregressive last-layer hidden-state thought generation with shared latent working-memory transfer across agents in a training-free framework.

  • Results

    Across 9 benchmarks, LatentMAS outperforms single-agent and text-based MAS baselines, achieving up to 14.6% higher accuracy, 70.8%-83.7% lower output-token usage, and 4×-4.3× faster inference.

  • Takeaways & Limitations

    LatentMAS provides a scalable collaboration paradigm that extends multi-agent reasoning and communication beyond natural-language mediation.

  • Takeaways & Limitations

    The evaluation adopts sequential and hierarchical MAS settings as its experimental bases rather than a single definitive MAS architecture.

Abstract

from arXiv · show

Multi-agent systems (MAS) extend large language models (LLMs) from independent single-model reasoning to coordinative system-level intelligence. While existing LLM agents depend on text-based mediation for reasoning and communication, we take a step forward by enabling models to collaborate directly within the continuous latent space. We introduce LatentMAS, an end-to-end training-free framework that enables pure latent collaboration among LLM agents. In LatentMAS, each agent first performs auto-regressive latent thoughts generation through last-layer hidden embeddings instead of text. Then, a shared latent working memory preserves and transfers each agent's internal representations and latent thoughts, ensuring lossless information exchange without re-encoding. We provide detailed theoretical analyses showing that LatentMAS achieves higher expressiveness and lossless information preservation with lower overall complexity than standard text-based MAS. In addition, empirical evaluations across 9 comprehensive benchmarks spanning math and science reasoning, commonsense understanding, and code generation show that LatentMAS outperforms advanced single agents and text-based MAS baselines, achieving up to 14.6% higher accuracy, reducing output token usage by 70.8%-83.7%, and providing 4$\times$-4.3$\times$ faster end-to-end inference. Code and data are fully open-sourced at https://github.com/Gen-Verse/LatentMAS.

1. Introduction

LatentMAS addresses whether multi-agent systems can collaborate entirely in latent space by combining latent reasoning with lossless latent communication. Across diverse benchmarks, it improves accuracy and efficiency over single-agent and text-based MAS baselines.

  • Motivation: LatentMAS is motivated by the limitations of text as the standard medium for transmitting agents’ internal thoughts.Prior work separately explores latent reasoning and latent information exchange, but does not provide a comprehensive framework unifying both.
  • Framework: LatentMAS is a training-free framework for multi-agent collaboration entirely within continuous latent space.Agents reason and communicate through internal latent representations rather than explicit text.
  • Latent reasoning: Each agent generates latent thoughts autoregressively from last-layer hidden representations instead of decoding tokens.Generated hidden states form continuous latent thoughts for subsequent processing.
  • Latent communication: Shared latent working memory transfers input representations and latent thoughts across agents without re-encoding, enabling lossless communication.The memory uses layer-wise KV caches to preserve the agents’ internal representations.
  • Principles: The framework is theoretically characterized as more expressive and less computationally complex than text-based MAS.Its three principles concern reasoning expressiveness, communication fidelity, and collaboration complexity.
  • Empirical evaluation: LatentMAS consistently outperforms single-model and text-based MAS baselines across 9 benchmarks, improving accuracy by up to 14.6%, reducing output tokens by 70.8%-83.7%, and accelerating inference 4×-4.3×.The evaluations span math and science reasoning, commonsense understanding, and code generation.

2. Preliminary and Notations

The paper formalizes transformer generation, KV-cache working memory, and the sequential and hierarchical MAS settings used to evaluate LatentMAS. These foundations describe how representations are generated, accumulated, and exchanged across agents.

  • Transformer generation: Transformer generation maps input tokens through hidden representations to next-token outputs, while latent generation autoregresses over hidden states instead of tokens.Latent generation bypasses explicit decoding and next-token embedding.
  • KV-cache memory: KV caches accumulate layer-wise key and value representations from prior steps, maintaining a growing working memory during autoregressive generation.New key/value vectors are appended to accumulated matrices at each step.
  • MAS settings: The experiments use sequential and hierarchical MAS architectures because collaboration designs vary across downstream tasks.The sequential setting uses planner, critic, refiner, and solver agents in a chain-of-agents pipeline.

3. LatentMAS

LatentMAS enables agents to reason and communicate entirely in latent space through autoregressive hidden-state generation and shared latent working memory. Its analyses argue that this design increases expressiveness, preserves information, and lowers complexity relative to text-based MAS.

  • LatentMAS combines autoregressive latent-thought generation, layer-wise working-memory transfer, and lower complexity than text-based MAS.Agents generate hidden representations instead of decoded tokens, while shared working memory transfers information across agents.
  • 3.1. Auto-regressive Latent Thoughts Generation: Each agent generates latent thoughts by repeatedly feeding last-layer hidden states back as inputs for m steps.The resulting hidden-state sequence H is defined as the agent’s continuous latent thoughts.
  • 3.1. Auto-regressive Latent Thoughts Generation: The alignment operator maps generated hidden states back toward the learned input-embedding distribution before subsequent latent steps.The projection matrix Wa is computed once and reused, making alignment computationally negligible.
  • 3.1. Auto-regressive Latent Thoughts Generation: Theorem 3.1 formalizes higher latent expressiveness than discrete text, while Qwen3-4B, 8B, and 14B examples reach 235.7, 377.1, and 471.4 times greater efficiency.The theorem’s accompanying remark attributes the efficiency advantage to latent-thought generation relative to text-based reasoning.
  • 3.2. Latent Working Memory Transfer: Layer-wise KV-cache transfer preserves preceding agents’ input context and latent thoughts without re-encoding, and Theorem 3.3 establishes equivalent outputs to explicit exchange.Each successor prepends the predecessor’s cache to its own cache before generating new latent thoughts.
  • 3.3. End-to-End Pipeline with Complexity Analyses: LatentMAS’s per-agent complexity is lower than text-based MAS complexity for achieving the same expressiveness and remains compatible with sequential or hierarchical collaboration.The framework is described as agnostic to the specific collaboration strategy.

4. Empirical Evaluations

LatentMAS is evaluated across nine benchmarks, five backbone scales, and sequential or hierarchical settings. It improves accuracy while substantially reducing token usage and inference time relative to single agents and text-based MAS.

  • 4. Empirical Evaluations: Across nine benchmarks, evaluations span math and science reasoning, commonsense reasoning, and code generation using Qwen3 and Llama 3 backbones.The study uses five off-the-shelf model scales and compares against single-agent and text-based MAS baselines.
  • 4.1. Main Results: LatentMAS improves accuracy by 14.6% and 13.3% over single-model baselines, and by 2.8% and 4.6% over text-based MAS in sequential and hierarchical settings.These averages cover the reported benchmark suite and both MAS architectures.
  • 4.1. Main Results: LatentMAS delivers 4× and 4.3× faster inference than sequential and hierarchical text-based MAS under identical architectures.The evaluation reports accuracy, total output tokens, and end-to-end speed across nine benchmarks.
  • 4.2. Efficiency Analyses on Latent Collaboration: LatentMAS reduces token usage by 70.8% and 83.7% relative to sequential and hierarchical TextMAS.The reduction follows from transferring latent working memory instead of communicating through decoded text.
  • 4.2. Efficiency Analyses on Latent Collaboration: LatentMAS achieves a 2.6×-7× speedup over vLLM-optimized TextMAS, while fewer than 50 latent steps can match or exceed performance on AIME24/25.TextMAS typically uses more than 20K output tokens for full text-based chain-of-thought trajectories.
  • 4.2. Efficiency Analyses on Latent Collaboration: Compared with single-model reasoning, LatentMAS uses 15.0%-60.3% fewer output tokens while distributing the question across collaborating agents.The final agent primarily aggregates preceding latent thoughts and decodes the answer.

5. In-depth Analyses on LatentMAS

The analyses examine whether latent thoughts preserve text-level semantics, how alignment affects representations, and how latent-step depth influences performance. Results support semantic consistency, greater representational diversity, improved accuracy after alignment, and a performance peak around 40-80 steps.

  • Effectiveness of Input-Output Alignment: Input-output alignment realigns generated vectors with the original input-embedding structure and mitigates representation drift across iterative latent steps.The unaligned output vector deviates from the original input distribution, whereas the aligned vector restores geometric and density-level consistency.
  • Effectiveness of Input-Output Alignment: Applying Wa produces consistent accuracy gains of 2.3%-5.3% across three downstream tasks.The comparison evaluates downstream performance before and after alignment.
  • Optimal Latent Step Depth: Increasing latent-step depth generally improves performance, with accuracy on three Qwen3-14B tasks peaking around 40-80 steps.The result links additional latent thoughts with increased collaborative expressiveness within the tested range.
  • Intermediate Error Tracking within LatentMAS: A debug mode converts inner latent thoughts into human-readable text within agents to support intermediate error tracking and interpretation.Implementation details and additional analyses are deferred to appendices.

6. Related Work

Prior LLM-based multi-agent systems primarily coordinate through text, while newer work explores latent-state communication. LatentMAS targets fully latent collaboration without auxiliary alignment training.

  • LatentMAS constructs multi-agent collaboration entirely in latent space for more expressive and efficient coordination.
  • Existing latent collaboration methods communicate through internal states but generally require auxiliary alignment mechanisms and additional training.
  • LatentMAS differs by combining training-free latent reasoning and native latent communication in one framework.

7. Conclusion

LatentMAS is presented as a training-free framework for latent-space multi-agent collaboration. The paper reports gains in accuracy, token usage, and decoding speed across diverse benchmarks compared with strong baselines.

  • LatentMAS consistently improves accuracy, token usage, and decoding speed over strong single-model and text-based MAS baselines.
  • LatentMAS enables latent auto-regressive reasoning and lossless transfer through a latent working-memory mechanism.
  • The framework is positioned as a scalable paradigm for agentic collaboration beyond natural-language constraints.

Impact Statement

The paper analyzes latent representations, alignment, and working-memory transfer through assumptions, optimization objectives, and equivalence proofs. These analyses support claims about distribution alignment, representational expressiveness, and faithful latent computation.

  • The alignment objective minimizes Wasserstein distance between aligned hidden-embedding and token-embedding distributions.
  • Theorem A.1 upper-bounds this distributional gap, and the chosen alignment matrix minimizes the bound.
  • Theoretical analysis assumes hidden embeddings are linear combinations of an underlying semantic basis with ternary coefficients.
  • Under the linear representation hypothesis, lossless text representation of latent thought sequences requires a lower-bounded text length.
  • Theorem B.2 states that latent working-memory inputs produce outputs equivalent to directly inputting preceding agents’ outputs.

B.3. Proof of Theorem 3.4

The proof compares LatentMAS with vanilla text-based MAS, showing that latent collaboration reaches equivalent expressiveness with lower per-agent time complexity.

  • B.3. Proof of Theorem 3.4: LatentMAS requires lower per-agent time complexity than vanilla text-based MAS to achieve the same expressiveness.The theorem states separate complexity bounds for LatentMAS and text-based MAS under the same expressiveness target.
  • B.3. Proof of Theorem 3.4: The analysis derives LatentMAS complexity from self-attention and feed-forward computation over an input-plus-latent-thought sequence.For m latent reasoning steps and input length t, the proof accounts for attention and feed-forward operations across L transformer layers.
  • B.3. Proof of Theorem 3.4: The vanilla text-based MAS analysis separately accounts for hidden-embedding computation, token decoding, and the number of text-based reasoning steps.The proof combines these components into the overall complexity of the text-based baseline.

C.1. Evaluation Details

The evaluation covers diverse reasoning and code-generation benchmarks, using exact-match, numeric-equality, and execution-based correctness protocols.

  • C.1. Evaluation Details: The evaluation spans math and science reasoning, commonsense understanding, and Python code generation benchmarks.Datasets include GSM8K, AIME24, AIME25, GPQA-Diamond, MedQA, ARC-Easy, ARC-Challenge, MBPP-Plus, and HumanEval-Plus.
  • C.1. Evaluation Details: MBPP-Plus and HumanEval-Plus assess Python synthesis by executing generated functions against comprehensive unit-test suites.Both benchmarks use stricter execution-based verification than their original versions.
  • C.1. Evaluation Details: The implementation uses Python, PyTorch, HuggingFace Transformers, official chat templates, and optionally vLLM for fast decoding and tensor-parallel inference.Special chat tokens are included in the backend setup.
  • C.1. Evaluation Details: Non-coding benchmarks score normalized final answers, exact answer letters for multiple choice, and numeric equality for numeric problems.Failed numeric parsing is counted as incorrect.
  • C.1. Evaluation Details: Code-generation predictions are correct only when all sandboxed unit tests pass without runtime errors within a 10-second timeout.The predicted code is combined with benchmark tests before execution.

D. Additional Experiments

Additional experiments show that LatentMAS retains accuracy advantages or parity while reducing token usage and inference time across hierarchical, sequential, and Llama3 settings.

  • D. Additional Experiments: Figure 9 compares system-level reasoning accuracy for LatentMAS against single-model and text-based MAS baselines in sequential and hierarchical settings.The figure reports consistent accuracy gains across both collaboration settings.
  • D. Additional Experiments: LatentMAS achieves higher or comparable accuracy than TextMAS while substantially reducing inference time and total token usage in hierarchical MAS.The trend is reported across additional Qwen3 backbones and six general tasks.
  • D. Additional Experiments: LatentMAS maintains higher or comparable accuracy with fewer tokens and faster inference across sequential and hierarchical Llama3 experiments.These results preserve the qualitative trend observed for Qwen models.
  • D. Additional Experiments: Figure 10 compares end-to-end inference speed and total system token usage for LatentMAS, single models, and TextMAS under hierarchical MAS.The figure highlights faster inference and fewer system-wise tokens for LatentMAS.
  • D. Additional Experiments: LatentMAS reduces final-agent decoding length by an average of 29.1% compared with TextMAS across model scales and tasks.This measurement excludes intermediate agents and isolates the final answering stage.
  • D. Additional Experiments: Input-output alignment restores geometric and density-level consistency for Qwen3-8B latent outputs that otherwise drift from the input embedding distribution.The alignment matrix preserves embedding-space structure and prevents representation drift.
  • D. Additional Experiments: LatentMAS yields less-collapsed and more diverse latent representations, with lower average pairwise cosine similarity than TextMAS across three Qwen3 backbones.Lower cosine similarity indicates greater representational diversity, while Figure 5 and Table 6 jointly support semantic consistency and broader representation space.

E.3. Ablations on Latent Reasoning and Communication in LatentMAS

Hybrid ablations show that full LatentMAS outperforms variants retaining only latent reasoning or only latent communication, supporting the combination of both components.

  • E.3. Ablations on Latent Reasoning and Communication in LatentMAS: Latent reasoning with text communication decodes only the last 128 tokens for communication between agents.This variant preserves latent internal reasoning but uses a compressed text interface for transfer.
  • E.3. Ablations on Latent Reasoning and Communication in LatentMAS: Text reasoning with latent communication uses standard text-based reasoning while transferring intermediate information through latent working memory.This variant isolates the communication contribution while retaining text reasoning.
  • E.3. Ablations on Latent Reasoning and Communication in LatentMAS: Both hybrid variants underperform full LatentMAS across all evaluated tasks, showing that both latent reasoning and latent communication contribute to its gains.The ablations replace one component at a time with its text-based counterpart.

F. How to Debug LatentMAS?

LatentMAS debug mode probes latent reasoning through parallel intermediate text, revealing how intermediate errors relate to final answers. Case studies show both error propagation and successful guidance, while correlation results support the probe’s reliability.

  • Failure case: An incorrect numerical relationship from the Refiner led the final Solver to use an erroneous dependency and produce an incorrect answer.
  • Success case: When intermediate agents produced logically sound outputs, those outputs successfully guided the final agent to derive the correct answer.
  • Debug-mode correlation: 96.2% of intermediate text was valid when final answers were correct, while 90.0% contained errors when final answers were incorrect.The correlation study used 100 Qwen3-14B GSM8K debug text–final answer pairs: 80 correct and 20 incorrect final answers.
  • Additional discussion: LatentMAS currently assumes same-shaped transformer layers, while heterogeneous agents would require representation-alignment adapters.
  • TextMAS versus LatentMAS: The GSM8K case study contrasts lengthy TextMAS exchanges, where early planner errors propagate, with LatentMAS latent working-memory communication.
Loading 2511.20639v4…