Source-linked AI summary
Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations
Isotta Magistrali, Chen Shani
TL;DR
The paper addresses the gap between individual-agent safety audits and safety in interacting populations. It calibrates population response from benign operation, models adversarial pressure with a mean-field fixed point, and tests rationale visibility and recovery after adversary removal. It finds that population vulnerability is forecastable, rationale sharing weakens weak attacks but delays strong ones, and capture is a temporary excursion rather than a permanent trap.
Problem
Individual-agent safety does not guarantee population safety, yet prior work has not established how to forecast population behavior under attack from normal operation.
Method
The paper calibrates response functions from single-agent probes and benign population logs, combines them with a mean-field adversarial-pressure model, and tests rationale visibility and post-removal recovery.
Results
Population-level vulnerability is predictable from benign operation; rationale visibility neutralizes weaker attacks but delays stronger ones, while removing adversaries returns populations toward their starting state.
Takeaways & Limitations
Population safety should be audited prospectively at the population level, using pre-attack behavior to assess vulnerability and interaction-protocol defenses.
Takeaways & Limitations
The study uses one model, one scenario for recovery, and four seeds, so recovery supports a temporary-excursion interpretation rather than a general claim.
Abstract
from arXiv · showhide
The unit of AI safety evaluation is still the individual model, yet language-model agents are increasingly deployed in interacting populations that read and write one another's decisions. This raises a question no single-agent audit can answer: an agent that is well-calibrated on its own may still be pulled toward a different decision by the agents around it. We study this on a security-triage task, where populations of language-model monitors decide whether to escalate or dismiss alerts, and into which we can inject a committed minority that always pushes one way. We find that two alerts a single agent judges almost identically on its own can drive collective behavior far apart, so auditing any one member need not reveal what the population will do. Yet that collective behavior can be predicted in advance. From a population's benign, adversary-free operation alone, we calibrate a response function that forecasts, before any attack is run, how far a committed minority will later move it. We then ask what shifts the outcome and find that letting agents see each other's reasoning neutralizes a weak attack, while only delaying it against a strong one, turning the question from whether the population converges on the adversaries' choice into when. Finally, we exclude the hypothesis of capture being an irreversible trap: once the committed agents are removed, the population drifts back toward where it began, so capture is a temporary state. Alignment in isolation is not alignment in a population, yet what a population will do under attack can be read in advance, from how it behaves before any adversary arrives.
1 Introduction
The paper asks whether safety in interacting agent populations can be forecast and managed from benign operation, rather than inferred from individual-agent audits alone. It finds that population vulnerability is predictable, rationale visibility offers conditional defense, and removing adversaries enables recovery.
- Individual safety does not guarantee population safety when agents interact and can compose into unsafe systems.
- The study asks whether benign population behavior can forecast attack response, whether rationale visibility mitigates attacks, and whether populations recover after adversaries are removed.
- The experiments compare isolated agents, benign populations, and populations infiltrated by committed agents that always advocate dismissal.
- Benign operation forecasts later infiltration effects, rationale visibility blunts weak attacks but delays strong ones, and adversary removal returns populations toward their initial state.
2 Related Work
Prior work studies emergent conventions, committed minorities, and population-level dynamics, but this paper focuses on forecasting attacked population behavior from normal operation and testing rationale visibility as a narrower interaction-protocol intervention.
- Earlier safety agendas argue that interacting AI systems require evaluation beyond isolated models but stop short of forecasting population behavior from normal operation.
- The paper varies whether honest agents see one another’s rationales, distinguishing this narrow visibility lever from broader deployment-information visibility.
- Related work shows that committed minorities can overturn conventions and that mean-field response laws can describe critical stubborn-agent fractions or metastable states.
- This study instead uses mean-field calibration to forecast attacked honest-agent dismissal levels as a function of minority dose.
- The mitigation acts on the interaction protocol, while the recovery result establishes relaxation after removal in a locally interacting, bounded-memory setting.
3 Tasks and Environments
The paper evaluates Llama-3.1-8B-Instruct agents acting as security monitors in a bounded-memory coordination game, with benign entrenchment followed by adversarial injection and rationale-visibility manipulations.
- 3.1 Task and Coordination Game: All experiments use Llama-3.1-8B-Instruct, leaving other model families and scales for future work.
- 3.1 Task and Coordination Game: Agents choose between escalation, the safe action for genuine threats, and dismissal, the adversary-promoted action.
- 3.1 Task and Coordination Game: The coordination game rewards matching a partner and penalizes disagreement, so the experiment studies which action populations settle on and whether minorities shift it.
- 3.1 Task and Coordination Game: The six scenarios form an evidence ladder, while isolated dismissal probabilities range from 0.012 in the clearest escalation case to 0.893 in the clearest dismissal case.
- 3.2 Population Protocol: Populations contain N=12 agents on a complete graph; each round has 12 random pairwise interactions with simultaneous decisions and five-slot FIFO memories.
- 3.2 Population Protocol: The 15-round benign entrench phase fills memories before injection replaces k agents with committed adversaries that always dismiss.
- 3.2 Population Protocol: The main forecast uses reconciled_backup, where an isolated agent dismisses with probability 0.345, near the ladder’s midpoint.
- 3.3 Conditions: Honest-rationale visibility records partners’ rationales in memory, while hidden visibility records only actions; adversaries are either argued or silent.
4 Forecasting Population Behavior
The forecasting framework calibrates how agents respond to recent social evidence, models adversarial pressure, and solves a mean-field fixed point to predict attacked population behavior before running the attack.
- The framework combines an individual response function, an adversarial-pressure model, and a fixed-point forecast of collective behavior.
- 4.1 Response Functions: The response function estimates dismissal probability from recent observations while holding the alert fixed, using both synthetic single-agent histories and benign population logs.
- 4.1 Response Functions: The pattern-resolved kernel K(d, r, v, h) preserves memory position, rationale visibility, and recent self-actions, whereas g(d) averages naturally occurring benign memory patterns.
- 4.2 Adversarial Pressure Model: For k adversaries among N=12 agents, the forecast uses g(d) with a mean-field mixing term that increases observed dismissal pressure while holding the response law fixed.
- 4.3 Fixed-Point Forecast: The fixed-point model treats five remembered observations as independent, samples dismissal counts from D | m, k ∼Binomial(5, ϕ(m, k)), and maps them through g(d).
- 4.3 Fixed-Point Forecast: The predicted rate is the self-reproducing solution m∗= F(m∗; k), located numerically.
- 4.3 Fixed-Point Forecast: The closure is predictive rather than causal and ignores repeated partners, temporal dependence, rationale content, and correlations with wider population state.
- 4.4 Evaluation Metrics: Evaluation compares forecasts with mean honest dismissal probability, sampled-action frequency, and a capture event defined by xt ≥0.75 for three consecutive rounds.
5 Results
The results show that interacting populations can amplify borderline judgments, while benign operation can forecast attacked behavior. Rationale visibility weakens attacks only temporarily or under weaker conditions, and remediation generally restores the pre-attack state without establishing a general safety guarantee.
- 5.2 Capture Dynamics: Capture became common between k = 3 and k = 4 committed agents under hidden rationales, but the boundary was noisy across four seeds per dose.Capture counts across k = 1 through 6 were 0/4, 0/4, 2/4, 4/4, 2/4, and 4/4.
- 5.2 Capture Dynamics: Visible rationales appeared to delay rather than prevent capture: three of four visible-rationale runs captured in the 60-round experiment, versus four of four hidden-rationale runs.Median capture time roughly doubled with visibility, but the difference was not statistically resolved because one run was censored.
- 5.2 Capture Dynamics: All remediation arms returned aggregate endpoints within a pre-sealed 90% forecast band, while complete per-seed recovery fell from four of four under replacement to one of four under removal.Recovery was toward the population’s dismissal-leaning benign state, and recovery timescales were not validated.
- 5.3 Forecasting Population Behavior from Benign Operation: Three held-out attack doses met the forecast criterion, with maximum absolute error below 0.008 and mean absolute error 0.0058.Final-round estimates remained within 0.03 of their forecasts, and baselines incurred roughly 2.8 times the closure’s error.
- 5.4 Rationale Visibility: Rationale visibility reduced the argued attack effect from +0.211 to −0.021, with all eight paired contrasts shifting in the predicted direction.The effect is consistent with visibility counteracting adversarial arguments, although authentic honest rationales appeared only about three-quarters of the time.
6 Conclusion
A committed minority can pull an interacting security-triage population toward decisions its members would not reach alone, but benign operation can forecast that vulnerability. Removing the adversaries returns the population toward its starting state, making capture a temporary excursion rather than a permanent trap.
- A committed minority can pull an all-honest population toward a decision its members would not reach alone, beyond what any single-agent audit reveals.
- A response function calibrated entirely on benign, adversary-free operation forecasts how far the population will move once an attack begins.
- Sharing reasoning blunts a weak attack but only postpones a strong one.
- Removing the adversaries returns the population toward its starting state, so capture behaves as a temporary excursion rather than a permanent trap.