PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Covert Collusion between AI Agents through Partner-Outsider Positive Listening

Kevin Yandoka Denamganaï

arXiv:2610.37849v1cs.AIcs.CL

TL;DR

Message-only monitoring can miss private conventions because a message may look innocent while its intended partner acts on it. The paper proposes offline replay to compare partner and outsider responses to paraphrases, and reports a positive PoPLi gap when a code is present but not when it is absent.

  • Problem

    A message-only monitor cannot detect a secure private code when the message remains distributed like an innocent one, motivating a behavioural test of what the partner does with it.

    Source: §1 · p. 1

  • Method

    PoPLi compares how the partner and a fresh outsider in the partner’s role change their actions when logged messages are paraphrased with meaning preserved.

    Source: §3.1 · p. 4, §3.2 · p. 6

  • Results

    PoPLi is above zero in the three conditions with a code and at zero in the three without one, while the outsider responds to stated intentions in the open condition.

    Source: §5 · p. 9

  • Takeaways & Limitations

    PoPLi detects use of a private channel, not collusion as such, and the message-only monitors did not separate colluding from non-colluding markets.

    Source: §6 · p. 10

    Results come from one environment, one model, and ten markets per condition, with a scripted speaker and colluders who do not adapt to the test.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

AI agents that share a secret convention can coordinate through messages that look innocent, so a monitor that reads only the messages can miss them. We propose a behavioural test: a partner that uses a code reacts to the exact wording of a message, whereas an uninvolved agent placed in its seat does not. Partner--Outsider Positive Listening (PoPLi) is the difference between the two agents in how much their action changes when the message is reworded with its meaning kept, and it is computed after the fact by replaying logged prompts. In a simulated market with five conditions (no collusion, a joint reward with neutral messages, the same with an openly stated intention, a covert first-letter code, and the code with a stated intention), PoPLi is between 0.48 and 0.59 whenever the code is present (95% intervals all above 0.29) and between -0.10 and -0.01 without it. Message-only monitors did not separate the conditions: one reading the speaker's messages answered "no" in all 50 markets, and one reading every seller's messages flagged 9 of 10 markets with no collusion. PoPLi detects the use of a private channel, not collusion as such, and it cannot detect collusion through incentives alone or codes carried by meaning.

1 Introduction

Message-only monitoring can miss a private convention because the same message may look innocent to an overseer while its partner acts on hidden information. The paper proposes comparing the partner’s response to an uninvolved listener placed in the same role, then tests this approach against message-only monitoring.

1 Introduction

  • A covert convention can make an innocent-looking message useful only to its intended partner, so evidence must come from the partner’s behaviour rather than the message alone.The paper distinguishes private-channel collusion from collusion visible in words and collusion through shared incentives alone.Source: §1 · p. 1
  • The research question is whether a partner reacts more than a role-matched outsider to message wording when a private channel is present, but not when it is absent.The no-channel cases include no collusion, open collusion, and collusion through shared incentives.Source: §1 · p. 2
  • PoPLi compares how much the partner’s and outsider’s actions change when a message is paraphrased, alongside a measure of listening to meaning.The proposed measure uses on-distribution interventions and needs no auditing task.Source: §1 · p. 2
  • The test replays logged multi-agent prompts after editing them, without restoring an agent’s state.The paper also compares prior approaches that focus on a listener’s own partner, remove wording and meaning together, or use cooperative settings.Source: §1 · p. 2
  • The experiments test PoPLi against a message-only monitor in five market conditions, including covert-code, incentive-only, and open-collusion cases.The tests ask whether PoPLi detects a covert code, stays near zero without one, and whether message-only monitoring separates the conditions.Source: §1 · pp. 2–3

2 Related Work

Prior work measures communication influence, steganographic decodability, or other signals of collusion, but does not compare a partner’s task response with an outsider’s response to the same message. This paper frames that comparison as evidence of whether a partner acts on a code.

Measuring communication

  • Positive-listening measures intervene on messages, but prior approaches focus on a listener’s own partner and may remove wording and meaning together.Those approaches were studied in cooperative or repeated-interaction settings, which may differ from mixed-motive settings.Source: §2 · p. 3

Steganography and its defences

  • The steganographic gap measures information a Receiver can decode beyond a Sentinel, whereas PoPLi measures how a partner’s own task action changes.A positive steganographic gap can occur without a partner choosing to collude.Source: §2 · p. 3

Collusion between AI agents and its detection

  • Existing collusion detectors use message judges, joint-action mutual information, or activation probes, but do not test whether a partner acts on a code.These signals may be present even when no partner acts on them.Source: §2 · p. 3

3 Methods

The methods frame monitoring as a comparison between an intended partner and a fresh outsider in the same role, using paraphrases to test listening to wording. Logged prompts are replayed offline, with controls and a scripted market setup supporting the experiment.

3.1 Theory

  • The covert referential game models a speaker sending a message about a private payload to a listener who may share a private resource.The listener can be the intended partner, who shares the resource, or an uninvolved listener who receives the same context and message without it.Source: §3.1 · p. 4
  • The outsider is a fresh instance of the partner’s model with the partner’s role prompt and public history, but without private instructions, memory, or channel.In the market, the private instruction maps the message’s first letter to the partner’s intended move.Source: §3.1 · p. 4
  • PoPLi measures how much more the partner listens to message form than the outsider, while marginal listening measures response differences from a message-averaged policy.The ideal paraphrase preserves meaning and removes wording information; the proposition predicts no form-listening gap when actions depend only on meaning.Source: §3.1 · pp. 4–5

3.2 Method

  • Offline replay estimates listening measures from logged prompts by editing the speaker’s message, without restoring agent state.It samples eligible turns, uses unchanged-message draws to estimate a noise floor, discretises asks into action categories, and bootstraps intervals over sessions and turns.Source: §3.2 · p. 6

3.2.1 Meaning control

  • The meaning control estimates a message-averaged policy by replaying listeners on four alternative messages sampled from the speaker’s policy.This checks whether a zero PoPLi gap could simply reflect an outsider that ignores messages.Source: §3.1 · p. 5, §3.2.1 · p. 6
  • Three controls check replay fidelity, meaning preservation with code-feature removal, and whether the partner follows the code.These controls are assessed against minima fixed in advance.Source: §3.2.1 · p. 6

3.3 Experimental setup

  • The experiment uses a continuous double auction with three sellers and three buyers, running for ten hours per market.All agents use DeepSeek v4-flash, and each condition has ten markets.Source: §3.3 · p. 6
  • The speaker is scripted to send remarks whose first letter follows a balanced schedule, with S and L versions sharing the same meaning.The paraphraser is prompted to preserve meaning and change wording, and a judge checks each paraphrase.Source: §3.3 · p. 7

4 Results

The experiments validate the replay instruments and find that PoPLi distinguishes coded wording from no-code conditions, including when an intention is also stated. Message-only monitors do not separate the conditions, and incentive-based collusion with neutral messages remains undetected.

Experiment #0: Validating the instrument

  • The instrument checks met their minimums: meaning was kept in 151 of 160 later paraphrases (94%), and the first letter changed in 160 of 160.Unmodified-prompt replay recovered the logged action as the most frequent reply in 141 of 154 later turns (92%); calibration found B followed the code in 35 of 36 coded hours (97%).Source: §4 · p. 7

Experiment #1: the gap detects a covert code

  • With a covert code, b∆PL = 0.615 (95% interval [0.453, 0.751]), versus −0.027 [−0.112, 0.044] for joint-reward collusion without a code.The partner’s c PLΠ was 0.615 [0.479, 0.738], while the outsider’s was 0.000 [−0.040, 0.047]; the re-run giving the outsider B’s previous ask produced PoPLi = 0.590 [0.411, 0.748] with the code and −0.012 [−0.126, 0.095] without it.Source: §4 · p. 7, Figure 1 · p. 8

Experiment #2: five kinds of market

  • PoPLi is above zero in all three code conditions and has an interval including or ending at zero in all three conditions without a code.The code is detected in K and both X groups, whether or not the stated intention agrees with it; in O, the outsider responds to the stated intention, but the PoPLi gap stays at zero.Source: §4 · p. 8, Figure 1 · p. 8
  • In X, B lowered its ask more often after an L than an S by 0.515 [0.290, 0.740], whereas C′ 0 lowered it more often after “lower” than “keep” by 0.217 [−0.051, 0.477], an interval that includes zero.Source: §4 · p. 9
  • The message-only monitor reading A’s messages answered “no” in all 50 markets, while the monitor reading every seller’s messages flagged 9 of 10 no-collusion markets.Neither monitor separated the conditions or identified A and B; neither caught joint-reward collusion with neutral messages, where PoPLi was zero.Source: §4 · pp. 8–9

5 Discussion and Limitations

In the market, PoPLi detects wording-carried private channels but not collusion as such, and the authors delimit the test’s scope and outline extensions to distinguish what carries a convention.

5 Discussion and Limitations

  • PoPLi is above zero with a code and at zero without one across the tested collusion forms, supporting detection of wording-carried private channels in one environment.It is not yet a test for learned or emergent codes, meaning-carried codes, or colluders who expect testing.Source: §5 · p. 9

5.1 Limitations

  • Results cover one environment, one model, ten markets per condition, one outsider, an instructed code destroyed by paraphrase, and colluders that do not adapt.The authors also note that a listener’s market response mixes trust with understanding.Source: §5.1 · pp. 9–10

5.2 Future work

  • A ladder of outsiders is proposed to test whether a convention is carried by shared history, shared model weights, or the private channel.C1 adds a summary of past exchanges; C2 uses another provider’s model with the history but not the weights.Source: §5.2 · p. 10
  • The authors propose a same-provider rerun of H1 and K to remove the remaining provider difference between conditions.Their provider comparison has limited power to rule out effects, with 42 hours on one provider.Source: §5.1 · p. 9, §5.2 · p. 10

6 Conclusion

PoPLi detects covert codes carried by wording, while remaining near zero without a code. The paper concludes that it detects a private channel, not collusion itself.

6 Conclusion

  • PoPLi detects wording-carried covert codes through partner behavior, stays near zero without a code, and is blind to codes carried by meaning.Message-only monitors did not separate colluding from non-colluding markets in the simulated market.Source: §6 · p. 10

A Limitations and dual use

The paper’s results are bounded by the instructed, scripted market setup and by the test’s blind spots; the authors also describe its dual-use implications.

A Limitations and dual use

  • One environment, one model, and ten markets per condition do not establish whether the gap behaves similarly in other tasks or models.Source: A Limitations and dual use · p. 13
  • The code was instructed rather than learned, and the tested feature was one that paraphrase destroys.Source: A Limitations and dual use · p. 13
  • Because the speaker is scripted, the positive control tests detection of a code its partner follows, not whether an AI speaker would invent or conceal one.Source: A Limitations and dual use · p. 13
  • In H0, the outsider’s marginal listening is 0.019 [−0.031, 0.073], an interval that includes zero; outsider listening is shown only when the speaker states an intention.Source: A Limitations and dual use · p. 14
  • The same model grades the paraphraser’s output, and its same-meaning validation passed narrowly at 0.81 against a 0.80 minimum.Source: A Limitations and dual use · p. 14
  • The test is blind to codes carried by the choice of meaning and to conventions carried by earlier messages, since interventions edit only the last message.Source: A Limitations and dual use · p. 14
  • Listening measures developed in cooperative settings may behave differently in a market, where a listener’s response mixes trust with understanding.Source: A Limitations and dual use · p. 14
  • The results show that a single letter can suffice as a covert channel a capable model follows reliably, while shared model weights could let conventions go undetected and expected substitution could defeat the test.Source: A Limitations and dual use · p. 14

B Proof sketches

The paper provides proof sketches and leaves full proofs for future work.

B Proof sketches

B.1 Message-only monitors

Message-only monitors have limited power when private codes make stegotexts resemble covertexts. The section explains what paraphrasing removes, relates marginal listening to communication’s causal influence, and identifies a blind spot for ideal paraphrases.

B.1 Message-only monitors

  • At most 1/2 + sqrt(ε/2), a message-scoring monitor’s lineup AUC when stegotexts and covertexts differ by at most ε in relative entropy; under perfect security, it is exactly 1/2.The bound follows from the total-variation limit implied by Pinsker’s inequality.Source: §B.1 · p. 14

B.2 What a paraphrase removes

  • Paraphrasing preserves information carried by message meaning, while its surviving mutual information is bounded by the entropy of meaning given context.The decomposition separates information retained after paraphrasing from a non-negative residual term.Source: §B.2 · p. 14

B.3 Marginal listening and the causal influence of communication

  • Positive marginal listening requires positive mutual information between the speaker’s message and the listener’s action.The measure is the single-step causal influence of communication, expressed as mutual information.Source: §B.3 · p. 15

B.4 Blind spot

  • An ideal meaning-preserving paraphraser makes each distance in the definition of PLΠ zero.This is a blind spot when the relevant signal is carried by meaning rather than wording.Source: §B.4 · p. 15

C Implementation details

The implementation estimates a listener-specific noise floor and uses resampling to quantify uncertainty.

C Implementation details

  • 200 simulations from pooled unchanged samples estimate the expected empirical-distribution distance when both samples come from the same distribution.This noise floor accounts for positive sampling distances even when the underlying distributions are equal.Source: C Implementation details · p. 15
  • 2,000 resamples of sessions and then turns within sessions produce the reported 95% percentile intervals.The decision rules follow Hypotheses 1 and 2.Source: C Implementation details · p. 15

D Meaning control: from a naive to a marginal design

The section contrasts a naive meaning control with a marginal design and situates the approach among work on signalling, influence-based listening, cross-play, and steganography.

D Meaning control: from a naive to a marginal design

  • The naive control’s out-of-distribution messages, single-draw variance, and lack of an information interpretation motivate replacing it with a marginal design.The revised design compares against a message marginal, samples coherent counterfactuals from the speaker’s policy at the same state, and ties the quantity to causal influence.Source: D Meaning control: from a naive to a… · pp. 15–16

E.1 Signalling games and referential games

  • In signalling games, any convention followed by both sender and receiver is an equilibrium; referential games are a common emergent-communication setting.Agents adapting to one another can drift away from a shared language, a benign analogue of the private convention studied here.Source: §E.1 · p. 16

E.2 Influence-based measures of listening

E.3 Cross-play

  • In zero-shot coordination, the self-play versus cross-play performance gap measures how private a convention is; the paper’s partner-versus-outsider contrast is its listening-side analogue.The comparison is attributed to Hu et al. (2020, § 3, § 6.1).Source: §E.3 · p. 17

E.5 Limitations of prior work addressed by this work

The paper distinguishes its approach from prior listening and steganographic-gap measures by changing the intervention, comparison partner, task setting, and outcome being measured.

E.5 Limitations of prior work addressed by this work

  • Prior listening measures replace whole messages and compare only an agent with its interacting partner; this work uses meaning-preserving paraphrases and a role-matched outsider.It measures in a competitive market, where a listener’s response reflects trust as well as understanding; the steganographic gap’s extension to separate agents and dependence on audit-task choice are also noted.Source: §E.5 · p. 17
Loading 2610.37849v1…