PaperlayerView exact revision on arXiv

Source-linked AI summary

Augmenting Multi-Agent Communication with State Delta Trajectory

Yichen Tang, Weihang Su, Yujia Zhou, Yiqun Liu, Min Zhang, Shaoping Ma, Qingyao Ai

arXiv:2506.19209v2cs.CL

TL;DR

Natural-language communication between agents can lose information about reasoning, and token-probability representations have not conveyed all information hidden in inference. SDE augments tokens with token-wise hidden-state changes; across information-asymmetry and symmetry tasks, it outperforms existing protocols, particularly on complex reasoning.

  • Problem

    Natural-language communication can lose reasoning information through token sampling, and token-probability representations alone may not convey important information hidden in inference.

    Source: §1 · pp. 1–2

  • Method

    State Delta Encoding (SDE) augments natural-language tokens with adjacent-token hidden-state differences, which the receiving agent adds to corresponding tokens.

    Source: §1 · p. 2

  • Results

    Across information-asymmetry and symmetry tasks, SDE outperforms existing communication protocols, by up to 17.3% on tasks requiring complex reasoning.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings point to the potential of multi-agent communication beyond natural language, especially for tasks involving complex reasoning rather than simple fact communication.

    Source: §1 · p. 2, §6 · p. 9

    The authors identify an assumption about hidden states as a limitation of SDE.

    Source: §7 · p. 9

Abstract

from arXiv · show

Multi-agent techniques such as role playing or multi-turn debates have been shown to be effective in improving the performance of large language models (LLMs) in downstream tasks. Despite their differences in workflows, existing multi-agent systems constructed from a single base LLM mostly use natural language for agent communication. While this is appealing for its simplicity and interpretability, it also introduces inevitable information loss as one model must down sample its continuous state vectors to discrete tokens before transferring them to the other model. Such losses are particularly significant when the information to transfer is not simple facts, but reasoning logics or abstractive thoughts. To tackle this problem, we propose a new communication protocol that transfers both natural language tokens and token-wise state transition trajectory from one agent to another. Particularly, compared to the actual state value, we find that the sequence of state changes in LLMs after generating each token can better reflect the information hidden behind the inference process. We propose a State Delta Encoding (SDE) method to represent state transition trajectories. The experimental results show that multi-agent systems with SDE achieve SOTA performance compared to other communication protocols, particularly in tasks that involve complex reasoning.

1 Introduction

Natural-language communication can lose information about an agent’s reasoning, and simply transferring token probabilities is not enough to convey it. The paper introduces State Delta Encoding (SDE), which augments tokens with hidden-state changes; experiments report its strongest advantages on complex-reasoning tasks.

1 Introduction

  • Natural-language messages may discard reasoning alternatives during token sampling, and transferring token probabilities or weighted embeddings yields only marginal improvements.The authors identify conveying internal reasoning between agents as an open communication-protocol question.Source: §1 · pp. 1–2
  • SDE combines natural-language tokens with adjacent-token hidden-state differences, which the receiving agent adds to corresponding tokens to recover information lost in sampling.The protocol transfers state changes rather than the full state sequence because agents can have different prompts or local context.Source: §1 · p. 2
  • Token-wise state changes capture reasoning hidden behind output tokens and provide a medium for transferring that information among agents.The paper presents this as a way to extend multi-agent communication beyond natural language.Source: §1 · p. 2
  • SDE achieves state-of-the-art performance across information-asymmetry and symmetry tasks, outperforming prior methods by up to 17.3% on tasks requiring complex reasoning.The reported advantage is particularly strong when tasks involve complex reasoning.Source: §1 · p. 2

2 Related Work

The related work situates the paper in single-LLM agent systems, communication protocols, and inference-time manipulation of latent representations. It distinguishes SDE by using hidden-state dynamics for inter-agent communication.

2.1 LLM based Multi-Agent System

  • Single-LLM-based systems vary agents’ behavior through profiles or distinct information access, using debates and task-specific workflows to improve performance.This paper focuses on optimizing communication within such systems and making better use of each inference step.Source: §2.1 · p. 2

2.2 Multi-agent Communication

  • SDE uses hidden-state dynamics for inter-agent communication, beyond natural-language tokens, probability-weighted embeddings, and prior direct-state transfer.CIPHER uses final-layer token probabilities; another approach restricts transfer to one direction and, for debate, requires heterogeneous models, whereas SDE supports any inter-agent communication.Source: §2.2 · pp. 2–3

2.3 Latent Space Arithmetic

  • Prior latent-space methods manipulate representations within a model, while this work extracts one agent’s states and injects them into another for coordination.The cited steering-vector approaches use hidden-state differences to guide generation toward a desired quality or semantic direction.Source: §2.3 · p. 3

3 Methodology

SDE augments natural-language communication with token-wise changes in hidden states, rather than transferring an agent’s context-dependent state sequence directly. The receiving agent adds these deltas to corresponding hidden states during inference.

3 Methodology

  • SDE augments natural-language messages with token-wise hidden-state changes to convey richer reasoning traces between agents.The method avoids transferring the original state trajectory because agents may have different prompts or local contexts.Source: §3 · p. 3
  • Natural-language communication transfers one sampled token sequence, which may make it difficult for the receiving agent to understand the sender’s full intentions.The agents considered are constructed from the same transformer-based language model.Source: §3 · p. 3
  • Each state delta is the difference between successive hidden states and serves as a context-agnostic trace of reasoning dynamics associated with token generation.The initial state corresponds to the last token of the sender’s input prompt.Source: §3 · pp. 3–4
  • During communication, the receiving agent adds each state delta as a steering vector to the hidden state for the corresponding token before passing it to the next layer.The method is intended to guide the receiver’s understanding without overwriting its own reasoning.Source: §3 · p. 4
  • SDE is applied at a few selected layers that remain fixed for each model across tasks, and the selected layers generalize to evaluated downstream tasks without task-specific selection.The layer choices come from a preliminary experiment and are not guaranteed to be optimal.Source: §3 · p. 5

4 Experimental Setup

The experiments evaluate SDE in settings where agents either have different information or share the same information. Tasks use retrieval, debates, or sequential agent workflows, with several communication baselines and model configurations.

4 Experimental Setup

  • Experiments cover information-asymmetry tasks and information-symmetry settings that include multi-agent debates and agent workflows.In the symmetry setting, agents share information and pass or refine intermediate thoughts through structured communication frameworks.Source: §4 · p. 5, §4.2 · p. 5

4.1 Information Asymmetry (IA) Tasks

  • In information-asymmetry tasks, two agents receive evenly divided private sets of six retrieved documents and can exchange questions and answers for up to five rounds.The passage split does not strictly guarantee disjoint private information; a single-agent baseline reports the better score from either agent using only its own passages. Tasks use Quasar-T, ComplexWebQuestions, and StrategyQA.Source: §4.1 · p. 5

4.2 Information Symmetry (IS) Tasks

  • Information-symmetry tasks test communication and reasoning refinement when all agents share the same information.The two task types are multi-agent debate and agent workflows.Source: §4.2 · p. 5

4.2.1 Multi-agent Debates

  • Debates use two agents over three rounds, beginning with independent answers and then revising responses after seeing peers’ previous-round responses.The reported score per question is the average accuracy of all agents’ responses in the last round.Source: §4.2.1 · pp. 5–6

4.2.2 Agent Workflows

  • Agent workflows adapt ReAct so up to seven agents take sequential turns, passing thoughts and actions while receiving environment observations.The tasks use HotpotQA and StrategyQA for question answering and FEVER for fact verification; evaluation reports HotpotQA EM and F1 and accuracy for StrategyQA and FEVER.Source: §4.2.2 · p. 6

4.3 Baselines

  • The baselines are a single agent, natural-language token communication, and CIPHER’s probability-weighted token embeddings.CIPHER embeds each token’s probability distribution as a weighted embedding sequence; NL and CIPHER use the same implementation across tasks, while Single varies by setting.Source: §4.3 · p. 6

4.4 LLM Selection and Generation Settings

  • Experiments use Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and Qwen2.5-14B-Instruct, with greedy decoding for NL and SDE in information-asymmetry and workflow tasks.Debate initial responses use default sampling and temperature settings, and reported debate results average three independent runs.Source: §4.4 · p. 6

4.5 Layer Selection

  • SDE injection layers are selected in a preliminary 2WikiMultihopQA experiment and held fixed across later experiments for each model.The 2WikiMultihopQA questions are used only for layer selection, not main evaluation; the selected layers are not guaranteed optimal but generalize to the evaluated tasks without additional tuning.Source: §4.5 · p. 6

5 Results

SDE outperforms existing communication protocols across information-asymmetry and information-symmetry tasks, with larger gains reported on complex-reasoning tasks. Ablations favor small numbers of selected layers and token-wise state deltas over unprocessed hidden states.

5.1 Main Experiments

  • SDE outperforms NL and CIPHER on almost all tasks, while multi-agent systems outperform single agents in most cases.The authors report that the improvements are more significant in information-symmetry tasks and suggest SDE also enhances agents’ understanding of reasoning processes.Source: §5.1 · pp. 6, 8
  • SDE improves over the best-performing baseline by 0.3% to 8.9% on most information-asymmetry tasks, 0.3% to 13.7% in debates, and up to 17.3% in workflows.Improvements are generally more significant on multi-hop CWQ and StrategyQA than on Quasar-T; debate gains are notable on MMLU across evaluated models.Source: Table 1 · p. 7, Table 2 · p. 7, Table 3 · p. 7

5.2 Different Layer Selections

  • With Qwen2.5-14B-Instruct, modifying combined top-k layers (k ≤4) performs similarly to top-1 selection but more stably, whereas modifying all layers significantly reduces performance.The authors recommend selecting and modifying only a small number of top-ranking layers, such as 1–3.Source: §5.2 · p. 8, Figure 2 · p. 8

5.3 Ablation Study on State Delta

  • Removing state deltas consistently lowers performance, and sometimes falls below natural-language-only communication, suggesting raw hidden states may introduce noise.The ablation compares full SDE with a variant using original token hidden states instead of their differences.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Multi-agent Debate in Different Settings

  • SDE consistently outperforms NL and CIPHER across the tested numbers of agents and debate rounds, indicating robustness to these configuration changes.The evaluation varies agent counts and round counts on Formal Logic using Qwen2.5-7B-Instruct.Source: §5.4 · p. 8, Table 5 · p. 9

6 Conclusion and Future Work

The paper presents State Delta Encoding (SDE), a single-LLM multi-agent communication protocol that augments natural-language messages with token-wise hidden-state changes. The authors report stronger performance than existing communication protocols, especially on complex-reasoning tasks, and identify specialized agents as a direction for future work.

6 Conclusion and Future Work

  • SDE is a single-LLM multi-agent communication protocol that augments natural-language messages with token-wise state deltas.The method encodes hidden-state changes to capture generation dynamics and reduce interference from irrelevant agent context.Source: §6 · p. 9
  • Across information-asymmetry and information-symmetry tasks, SDE outperforms existing communication protocols, particularly on complex-reasoning tasks.The authors present these results as evidence for communication beyond natural language.Source: §6 · p. 9
  • The study focuses on general multi-agent settings; integrating SDE into specialized search and planning agents remains future work.The authors also propose exploring SDE for tracking query-intent or memory-state evolution in search settings.Source: §6 · p. 9

7 Limitation

SDE has access and communication-cost limitations: it requires hidden-state access and increases bandwidth, especially with long contexts or large models.

7 Limitation

  • SDE requires the generating agent’s hidden states to be extracted and injected into the receiving agent, which may be infeasible for black-box models without internal access.The authors identify selective transmission or state-delta compression as possible ways to reduce the cost.Source: §7 · p. 9
  • Adding hidden states increases communication bandwidth, particularly for long-context communication or large models.Although SDE modifies only a small number of layers, the overhead may still require compression or optimization.Source: §7 · p. 9

A Layer Selection

The authors select a small, fixed set of transformer layers using a preliminary experiment, then use those layers for state-delta capture and injection across subsequent experiments.

A Layer Selection

  • For layer selection, the authors evaluate each transformer layer individually on the first 300 2WikiMultihopQA questions in a preliminary information-asymmetric task, using exact match (EM) and F1 jointly.The dataset is used only for layer selection and is excluded from all evaluations.Source: A Layer Selection · pp. 12–13
  • Top-performing layers generally lie in middle-to-late positions across models, though some earlier layers also perform well.The preliminary ranking uses combined exact match (EM) and F1 on 2WikiMultihopQA; selected layers are underlined in Table 6.Source: A Layer Selection · p. 12, Table 6 · p. 13
  • The selected layers are Layer 22 for Qwen2.5-7B-Instruct and Layers 17 and 20 for Llama3.1-8B-Instruct.The selected layers remain fixed across experiments; sender state deltas are injected into the same layers on the receiver.Source: A Layer Selection · pp. 12–13

B Experimental Details

Experiments were conducted using PyTorch on NVIDIA A100 GPUs with 40GB of memory.

B Experimental Details

B.1 Information Asymmetry (IA) Tasks

The information-asymmetry task uses two agents with private retrieved passages that communicate over multiple rounds; the study compares communication protocols with independent-agent baselines.

B.1 Information Asymmetry (IA) Tasks

  • Two agents receive separate sets of retrieved passages and discuss for up to five rounds to collaboratively answer each factual question.The corpus contains 21 million Wikipedia passages; BM25 retrieves six passages, split by odd and even rank between agents.Source: §B.1 · p. 13
  • In the first round, each agent reasons from private knowledge and asks the other questions; later rounds use prior responses to answer, reason, or ask further questions.Discussion stops when an agent produces an answer in the specified boxed format.Source: §B.1 · p. 13
  • Question scores average all formatted answers in the final round.Source: §B.1 · p. 13
  • The single-agent baseline has each agent answer using only its own private passages, and reports the higher of the two agents’ total scores.The authors omit a single agent given the union of retrieved passages because the study focuses on transmitting private information through communication.Source: §B.1 · p. 13
  • Single, natural-language, and SDE methods use greedy decoding, while CIPHER uses temperature 0; generations are capped at 256 tokens.The authors note that CIPHER at temperature 0 still captures the full vocabulary probability distribution under their implementation.Source: §B.1 · p. 14

B.2 Information Symmetry (IS) Tasks

This section specifies debate and workflow protocols: debate agents revise responses after reading peers’ previous-round answers, while workflow agents pass thoughts, actions, and environmental observations sequentially.

B.2.1 Multi-agent Debate

  • Two agents debate for three rounds, and the score is the proportion of correct final-round responses among all agents.Each agent independently answers in round one, then receives peers’ previous-round responses and revises or refines its answer.Source: §B.2.1 · p. 14
  • The debate baseline uses one direct response; CIPHER varies agent temperatures, while settings are run three times and averaged.The CIPHER scheme sets agent i’s temperature to i/n × Tdefault to encourage varied embeddings without dataset-specific tuning.Source: §B.2.1 · p. 14

B.2.2 Agent Workflow

  • Workflow agents act sequentially on the question and full history, producing Thought and Action outputs that the environment validates and answers with an Observation.Search actions use BM25 retrieval over the Wikipedia corpus split by DPR.Source: §B.2.2 · p. 14
  • A workflow can run for up to seven iterations, and fails if no agent returns a correctly formatted Finish[answer] within that limit.The prompts use ReAct examples; the single-agent baseline receives no retrieved documents for HotpotQA and StrategyQA, but receives candidate answers for FEVER.Source: §B.2.2 · pp. 14–15
  • Workflow prompts interleave Thought, Action, and Observation, with Search, Lookup, and Finish as action types.The single-agent workflow generation limit is 256 tokens, versus 100 tokens per generation for the other methods.Source: §B.2.2 · p. 15

C Different Layer Selections

Layer-selection ablations favor combined top-k modifications for robustness, while modifying all layers degrades performance. The accompanying prompts illustrate task-specific agent interactions.

C Different Layer Selections

  • The authors recommend selecting a small number of combined top-ranking layers, such as top 1–3, rather than using a single layer or all layers.The recommendation is based on the instability of single-layer selection and degradation from modifying all layers.Source: C Different Layer Selections · p. 15
  • Combined Top-k yields more stable performance across k and better robustness and generality across tasks, although isolated single-layer cases can outperform it.The comparisons cover StrategyQA information asymmetry and Formal Logic multi-agent debate on Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and Qwen2.5-14B-Instruct.Source: C Different Layer Selections · p. 15, Figure 3 · p. 16, Figure 4 · p. 16
  • All Combine consistently degrades performance across models and tasks, which the authors attribute likely to disruption of internal representations.The study compares modifying all layers with jointly modifying top-k layers and modifying only the k-th top-ranking layer.Source: C Different Layer Selections · p. 15, Figure 3 · p. 16, Figure 4 · p. 16
  • Only Top-k performs inconsistently across layer ranks and tasks, including a rank-5 increase after declines from rank-1 to rank-4 on Qwen2.5-7B Formal Logic.The authors interpret this as sensitivity of single-layer modifications to task-specific factors.Source: C Different Layer Selections · p. 15, Figure 3 · p. 16, Figure 4 · p. 16

D Prompts

  • The workflow prompt examples interleave Thought, Action, and Observation steps, with Search, Lookup, and Finish as available actions.Search retrieves a Wikipedia paragraph or similar entities; Lookup returns a sentence containing a keyword, and Finish ends the task with an answer.Source: D Prompts · p. 20
Loading 2506.19209v2…