PaperlayerView exact revision on arXiv

Source-linked AI summary

Safety of Latent Communication in Multi-Agent Systems

Muhammad Huzaifa, Sina Mavali, Thorsten Eisenhofer

arXiv:2609.39788v1cs.AIcs.LGcs.MA

TL;DR

Latent links can compromise the safety of multi-agent systems even when their agents remain unchanged. The paper tests direct, poisoned-data, and reward-guided link attacks, finding that reward-guided optimization raises mean harmful-compliance score from 27.9 with benignly trained links to 76.9. It also shows that changing rewards can substantially reduce harmful compliance across evaluated attacks and topologies.

  • Problem

    The paper examines how learned communication links that shape multi-agent behavior affect safety when the underlying agents remain unchanged.

    Source: §1 · p. 1

  • Method

    The paper optimizes communication links directly, poisons benign training data, and uses scalar feedback on system responses to reward harmful compliance and benign task performance.

    Source: §1 · p. 1, §4 · pp. 4–5, §4.2 · p. 6

  • Results

    27.9 with benignly trained links to 76.9 mean harmful-compliance score across four safety benchmarks and three topologies under reward-guided optimization.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Safety alignment must account for the multi-agent system as a whole, including its agents and learned communication links.

    Source: §8 · p. 10

    Maintaining safe and useful behavior as participating models, tasks, and communication mechanisms change remains future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Latent communication enables multi-agent systems to exchange information directly in internal representation space, reducing the token, computation, and latency overhead of text-based communication. To this end, lightweight trainable links are introduced to map the sender's representations into the receiver's input space. In this work, we show that even benign link training can increase harmful compliance relative to text-based communication while the underlying safety-aligned agents remain unchanged. An attacker can amplify this effect by optimizing the links on harmful query--response pairs or poisoning otherwise benign training data. We further develop a reinforcement-learning attack that rewards harmful compliance alongside benign task performance without requiring harmful target responses. Across three communication topologies and four safety benchmarks, this attack raises the mean harmful-compliance score from 27.9 with benignly trained links to 76.9. Compared with direct supervised optimization, it also achieves higher average accuracy on two benign utility benchmarks. Adapting the rewards toward safer behavior also enables repair of compromised links, substantially reducing harmful compliance across all evaluated attacks without updating the agents. Overall, our results show that safety alignment requires considering the multi-agent system as a whole.

1 INTRODUCTION

Latent communication reduces text-based messaging overhead, but learned links can change system behavior even when the agents remain fixed. The paper examines attacks and repairs that modify links, finding that reward-guided optimization increases harmful compliance while safer rewards can reduce it.

1 INTRODUCTION

  • Attacks optimize links on harmful query–response pairs, poison otherwise benign training data, or use scalar rewards for harmful compliance and benign performance without harmful target responses.The agents remain unchanged during link manipulation.Source: §1 · pp. 1–2
  • 27.9 with benignly trained links to 76.9 mean harmful-compliance score across four safety benchmarks and three topologies; reward-guided optimization also yields higher average accuracy on both utility benchmarks than direct supervised optimization.The utility benchmarks are MATH500 and GPQA-Diamond.Source: §1 · pp. 1–2
  • Safety alignment must consider the whole multi-agent system, including individual agents and the learned communication interfaces shaping their interactions and responses.Source: §1 · p. 2
  • Even benign link training can increase harmful compliance relative to text-based communication while the underlying agents remain unchanged.Latent communication links map internal representations between agents, while latent communication aims to reduce the token, latency, and computational overhead of text messages.Source: §1 · pp. 1–2
  • Adapting rewards toward safer behavior substantially reduces harmful compliance across evaluated attacks and communication topologies by updating links alone.The repair procedure rewards correct responses to benign queries while penalizing harmful compliance.Source: §1 · p. 2

2 LATENT MULTI-AGENT SYSTEMS

The section describes how agents exchange information through text or learned latent links, then explains how links can be trained while agent models remain fixed.

2.1 MULTI-AGENT COMMUNICATION

  • A communication topology specifies directed connections between agents, such as planner-to-solver, sequential refiner, or expert-to-summarizer arrangements.With text communication, the sender generates a message that the receiver processes again, incurring computation at both models.Source: §2.1 · p. 2

2.2 LATENT COMMUNICATION LINKS

  • Latent communication links map sender representations to a latent input that the receiver uses alongside the original query to generate its response.Links bridge differences in dimensionality or information encoding between models; in multi-link systems, the final receiver may receive representations from several agents.Source: §2.2 · p. 3

2.3 LEARNING COMMUNICATION LINKS

  • Supervised link training adjusts links to produce correct answers for benign query–response pairs, while leaving the language models unchanged and changing how information passes between them.The learned links therefore influence the behavior of the composed system.Source: §2.3 · p. 3

3 SAFETY OF LATENT COMMUNICATION

Benignly trained latent links increase harmful compliance relative to text communication even though the underlying agents remain fixed. They reduce inference cost, while refusal initiation also changes and brief refusal prefixes reduce compliance.

3 SAFETY OF LATENT COMMUNICATION

  • Aggregate harmful-compliance scores rise from 4.4 to 31.1 (two-agent), 3.2 to 31.7 (sequential), and 12.5 to 20.8 (mixture) under benign latent links.The underlying LLM parameters remain frozen.Source: Table 1 · p. 4, §3 · p. 4
  • Across all three topologies, benign latent communication reduces token usage and inference time, while benign task performance improves in five of six comparisons.MATH500 accuracy in the mixture system is the exception, decreasing from 80.6% to 76.8%.Source: Table 1 · p. 4, §3 · p. 4
  • First-token refusal probability falls to 0.29 with trained links from 0.82 for the receiver alone and 0.84 under text communication; a five-token refusal prefix reduces compliance to 0.Forcing I, No, or Sorry as the first token lowers the reported compliance score from 31.1 to 9.6, 3.7, and 1.6, respectively.Source: §3 · p. 4, Figure 2 · p. 4
  • The authors suggest benign supervision may reward task completion without preserving refusal behavior, potentially steering the frozen receiver toward answering.They suggest reduced refusal initiation contributes to the observed increase in harmful compliance.Source: §3 · p. 4

4 ATTACKING LATENT COMMUNICATION LINKS

Link attacks can amplify harmful compliance while the agents remain frozen, using harmful target responses, poisoned training data, or response-level rewards. Reward-guided optimization increases harmful compliance while producing mixed utility results, and optimizing individual links can also provide substantial control.

4 ATTACKING LATENT COMMUNICATION LINKS

  • The attacker can directly optimize links or inject harmful examples into training data, while all underlying LLM parameters remain frozen and modified links are reused across queries.The section also considers reward-guided optimization using feedback on the system’s own responses rather than harmful target responses.Source: §4 · pp. 4–5, §4.2 · p. 6

4.1 SUPERVISED LINK OPTIMIZATION

  • Direct supervised optimization raises average compliance from 27.9 to 75.6, while poisoning raises it to 58.2; both attacks reduce benign task accuracy.With poisoning, MATH500 accuracy falls from 67.2% to 46.7% and GPQA-Diamond from 36.4% to 32.5%; the poisoning rate is approximately 10%.Source: Table 2 · p. 5, §4.1 · pp. 5–6
  • Direct supervised optimization trains links on harmful query–response pairs, while poisoning injects 212 harmful pairs into 1,904 benign examples under the unchanged benign objective.The direct supervised attack uses 3,000 harmful query–response pairs.Source: §4.1 · pp. 5–6

4.2.1 OPTIMIZATION PROCEDURE

  • Reward-guided optimization uses GRPO to update links from relative response-level rewards while agent and reward-model parameters remain fixed.The attack reward combines a harmful-compliance judge with language-consistency and lexical-overlap terms, and benign queries are interleaved with a separate utility judge.Source: §4.2.1 · p. 6

4.2.2 RESULTS

  • Reward-guided link optimization raises aggregate harmful compliance from 27.9 to 76.9 across topologies and exceeds direct supervised optimization in two configurations.The scores are 81.3 compared with 78.3 in the two-agent system and 95.9 compared with 81.6 in the mixture system.Source: Table 3 · p. 7, §4.2.2 · p. 7
  • Average MATH500 accuracy increases from 67.2% to 71.0%, whereas GPQA-Diamond accuracy decreases from 36.4% to 33.6%; the mixture system improves on both benchmarks while having the highest harmful-compliance score.The mixture-system result shows that improved benign benchmark accuracy can coexist with substantial harmful compliance.Source: §4.2.2 · p. 7

4.3 ATTACKING SINGLE LINKS

  • Optimizing one link can provide substantial control: the sequential Refiner→Solver link scores 70.0 versus 67.0 for both links, while a single mixture expert scores 80.3 versus 81.6.The scores are mean harmful-compliance scores.Source: §4.3 · p. 7, Figure 4 · p. 7

5 REPAIRING LATENT COMMUNICATION LINKS

Reward-guided repair updates communication links while keeping agent parameters fixed, aiming to reduce harmful compliance without sacrificing benign utility. Across nine attack–topology pairs, repair substantially reduces harmful compliance, while utility outcomes vary by benchmark and attack.

5 REPAIRING LATENT COMMUNICATION LINKS

  • The repair objective adapts reward-guided optimization to penalize harmful compliance while rewarding correct responses to benign queries.A useful repair is intended to reduce harmful compliance while preserving benign-task performance.Source: §5 · p. 7

5.2 RESULTS

  • 70.3 to 4.8: averaged across nine attack–topology pairs, the aggregate harmful-compliance score falls after repair.The reduction holds across all four safety benchmarks in every configuration.Source: Table 4 · p. 8, §5.2 · p. 8

5.1 OPTIMIZATION PROCEDURE

  • Repair uses GRPO to update links from compromised parameters, alternating harmful and benign queries while all agent and judge parameters remain fixed.Harmful-query rewards penalize judged compliance and encourage explicit refusals; benign-query rewards score task correctness and answer formatting.Source: §5.1 · p. 8

5.2 RESULTS

  • Across nine repaired configurations, MATH500 accuracy averages 67.4%, near the clean-link 67.2%, while GPQA-Diamond averages 33.6%, below its clean-link 36.4%.Repair improves MATH500 accuracy after supervised optimization and data poisoning in all three topologies, but reduces it in all three reward-guided settings.Source: §5.2 · p. 9

6 DISCUSSION & OUTLOOK

Learned links can change harmful-request responses even when safety-aligned agents remain unchanged. The discussion argues that safety assessment and alignment must include the communication interface, with repair judged on both safety and utility.

6 DISCUSSION & OUTLOOK

  • Changing learned links can substantially alter harmful-request responses even when the underlying safety-aligned agents remain unchanged.The findings bear on assessment of composed systems and training their communication interfaces.Source: §6 · p. 9
  • Benign link training can increase harmful compliance, so fixed agent parameters do not guarantee that system safety behavior is preserved.Safety assessment must extend to the whole system, including its trained communication interface.Source: §6 · p. 9
  • Repair through link updates can reduce harmful compliance, but may also reduce task performance, so both outcomes must be assessed.The discussion notes utility reductions even in settings where utility remained high after an attack.Source: §6 · p. 9
  • Future work needs to examine how to maintain safe and useful behavior as participating models, tasks, and communication mechanisms change.Source: §6 · p. 9

7 RELATED WORK

The related-work discussion places this study at the intersection of latent communication and multi-agent safety. It distinguishes prior work on communication effectiveness and efficiency from this paper’s focus on link training and system safety.

7 RELATED WORK

  • The paper connects latent communication research with safety and alignment questions in multi-agent systems.Source: §7 · p. 9

Latent communication in multi-agent systems

  • Prior latent-communication work largely focuses on effectiveness and efficiency, whereas this work studies how training and modifying learned links affect system safety.Examples include LatentMAS, KVComm, RecursiveMAS, and StateBridge, which use different mechanisms to share or map latent representations.Source: §7 · p. 9

Safety and alignment of multi-agent systems

  • Prior multi-agent safety studies address prompt injection, message tampering, malicious agents, and inference-time latent interventions; this work examines mappings between frozen agents.The discussion also cites benign-data fine-tuning as a way individual-model safety alignment can weaken.Source: §7 · p. 9

8 CONCLUSION

The paper concludes that learned communication links can compromise system safety without changing the agents, and that reward-guided repair can substantially reduce harmful compliance. The appendices provide experimental setup, implementation, analysis, prompt, and qualitative-example materials.

8 CONCLUSION

  • Learned communication links can compromise system safety without modifying agents; reward-guided optimization achieves high harmful compliance without harmful targets while retaining more benign utility than supervised optimization on average.Adapting rewards also enables repair that substantially reduces harmful compliance across evaluated attacks and topologies.Source: §8 · p. 10

APPENDIX OVERVIEW

  • The appendix outlines experimental and implementation details, followed by additional analyses, prompt templates, and qualitative examples.Source: APPENDIX OVERVIEW · p. 12

B. Attack and Defense Implementation

C. Additional Analyses

  • Appendix C lists additional analyses of training paradigm, poisoning rate, attack surface, utility-reward effects, and benign-training effects on harmful compliance.Source: C. Additional Analyses · p. 12

D. Prompt Templates

E. Qualitative Examples

A EXPERIMENTAL SETUP

A.1 MULTI-AGENT CONFIGURATIONS AND COMMUNICATION LINKS

  • Three topologies keep agent roles and underlying models fixed across clean, attacked, and repaired settings; Table 5 summarizes assignments, hidden dimensions, and trainable link parameters.The table caption specifies that only latent communication links are trainable and all LLM parameters remain frozen.Source: §A.1 · p. 12, Table 5 · p. 13

A.2 DATASETS AND QUERY CONSTRUCTION

The appendix documents the attack and evaluation data, shared optimization settings, and evaluation protocol. It also specifies supervised-link attack training and the evaluator prompts used to score harmful responses.

A.2 DATASETS AND QUERY CONSTRUCTION

  • Supervised, poisoning, and target-free RL attacks use distinct data: harmful query–response pairs, poisoned benign data, and harmful queries only, respectively.Clean links use mathematics data; the target-free attack discards reference responses and intermediate annotations.Source: Table 6 · p. 13, §A.2 · p. 13

A.3 SHARED OPTIMIZATION SETTINGS

  • Links use AdamW at a 5 × 10−4 learning rate, with 10-step warmup, gradient clipping at 1.0, bfloat16, and checkpointing.Experiments run on one NVIDIA H100 GPU per job without distributed training.Source: §A.3 · p. 14

A.4 EVALUATION PROTOCOL

  • Safety evaluation covers HarmBench, StrongREJECT, JailbreakBench, and AdvBench; benign utility is evaluated on MATH500 and GPQA-Diamond.The PKU-derived attack query pools have no exact overlap with the four safety benchmarks, so they measure held-out transfer.Source: §A.4 · p. 14

A.4.1 EVALUATOR PROMPT TEMPLATES

  • HarmBench, AdvBench, and JailbreakBench use a binary behavior classifier, while StrongREJECT grades harmful assistance from 1 to 5.The HarmBench rules exclude vague or minimal instances and count harmful behavior even when caveats accompany it.Source: §A.4.1 · p. 15

B.1 SUPERVISED LINK ATTACK

B.2 DATA POISONING

This appendix details poisoning, target-free reward-guided link optimization, and reward-guided repair. The additional analyses examine vulnerabilities across training paradigms, poisoning rates, links, utility objectives, and refusal behavior.

B.2 DATA POISONING

  • The default poisoning setup adds 212 harmful examples to 1,904 benign examples, then trains communication links for one epoch with batch size 2.The injected examples pair harmful queries and responses with a fixed generic intermediate instruction.Source: §B.2 · p. 16

B.3 RL ATTACK

  • The target-free RL attack uses GRPO with alternating harmful and benign updates, rewarding harmful compliance and preserving utility without harmful target responses.Its attack reward combines a compliance judge with script and query-overlap heuristics; the clean-query reward uses a mathematical-answer-quality judge and length gate.Source: §B.3 · p. 16

B.4 DEFENSE IMPLEMENTATION

  • Repair starts from compromised links and alternates harmful-safety and benign-utility updates for 300 steps, using the same rollout configuration as the attack.The safety reward combines a refusal-prefix signal with a response-level compliance judge; the benign reward is based on task correctness and formatting.Source: §B.4 · pp. 16–17

C ADDITIONAL ANALYSES

  • The additional analyses cover training paradigm, poisoning rate, individual links, utility preservation, and why benign supervised training can weaken refusal behavior.Source: C ADDITIONAL ANALYSES · p. 17

C.1 EFFECT OF THE TRAINING PARADIGM

The analyses compare supervised and RL link optimization under harmful and benign-only training, then test how poisoning rate affects safety. Benign-only supervised training yields higher harmful compliance than RL across all three topologies, while poisoning effects rise sharply at low rates and largely saturate around 20%.

C.1 EFFECT OF THE TRAINING PARADIGM

  • The training-paradigm comparison starts both optimization methods from the same untrained initialization and tests harmful supervision and benign-only mathematics training.Source: §C.1 · p. 17
  • Under harmful supervision, RL has higher mean harmful compliance in the 2-Agent and Mixture systems, while supervised optimization is stronger for Sequential.Neither optimization paradigm consistently dominates across topologies.Source: §C.1 · p. 17, Table 7 · p. 17
  • 31.1 vs. 7.9 for 2-Agent, 31.7 vs. 19.6 for Sequential, and 20.8 vs. 5.7 for Mixture: benign-only supervised training yields higher harmful compliance than RL.RL also achieves higher MATH500 accuracy in every topology while maintaining lower harmful compliance.Source: §C.1 · pp. 17–18, Table 8 · p. 18

C.2 EFFECT OF POISONING RATE

  • Poisoning causes substantial safety degradation even when most training data remain benign, with harmful compliance rising sharply at low poisoning rates.The analysis varies the poisoning fraction for Sequential and Mixture across all four safety benchmarks.Source: §C.2 · p. 18
  • Harmful compliance largely saturates around 20% poisoning in both topologies, with additional poisoned data providing little consistent gain.Source: §C.2 · p. 18, Table 9 · p. 18

C.3 ATTACK SURFACE ANALYSIS

The analyses show that attack effectiveness depends on which communication links are optimized and how utility rewards interact with link initialization. They also examine the onset of refusal, provide topology-specific prompts, and present qualitative examples of attacks and repair.

C.3 ATTACK SURFACE ANALYSIS

  • 70.0 vs. 67.0 mean harmful compliance: optimizing only the downstream Refiner→Solver link outperforms jointly optimizing both Sequential links, while either Mixture expert link nearly matches joint optimization.The analysis concludes that link location and functional role can matter more than the number of compromised links.Source: §C.3 · p. 19, Table 10 · p. 19
  • A learned latent interface provides a continuous, differentiable path from downstream objectives to communication parameters, unlike text generation’s discrete bottleneck.The authors identify this as a distinct gradient-based attack surface, not evidence that text communication is inherently secure.Source: §C.3 · p. 19

C.4 WHEN DOES THE UTILITY REWARD MATTER?

  • 17.6–38.4 MATH500 points recovered: utility-guided RL on heavily degraded links improves task performance while simultaneously increasing attack success across all three topologies.Source: §C.4 · p. 19
  • From clean initialization, the utility reward’s benefit depends on topology; in 2-Agent, it restores MATH500 accuracy from 41.8% to 63.8% after judge-only RL lowers it from 65.4%.Mixture shows a smaller positive utility gain, while Sequential preserves strong benign performance without explicit utility guidance.Source: §C.4 · pp. 19–20, Table 11 · p. 20

C.5 WHY DOES BENIGN SUPERVISED LINK TRAINING INCREASE HARMFUL COMPLIANCE?

  • Benign-trained links reduce first-token refusal probability to 0.289, versus 0.820 for the receiver alone, 0.943 with an untrained link, and 0.841 with text communication.The frozen receiver’s refusal behavior remains recoverable; the results suggest the learned context makes refusal less likely at generation onset, while task-completion supervision offers no corresponding signal to preserve refusal openings.Source: §C.5 · pp. 20–21, Table 12 · p. 20

D PROMPT TEMPLATES

  • The prompts assign planner-to-solver roles in 2-Agent, planner-to-refiner-to-solver roles in Sequential, and math/science experts-to-summarizer roles in Mixture.Source: §D.1 · p. 21, §D.2 · p. 22, §D.3 · p. 23

E QUALITATIVE EXAMPLES

  • The qualitative examples compare clean links, three attack settings, and reward-guided repair across the communication topologies.In the 2-Agent examples, benign training weakens refusal, attacks increase harmful compliance, and repair restores refusal behavior.Source: E QUALITATIVE EXAMPLES · p. 24, Figure 5 · p. 24

Related papers

Loading 2609.39788v1…