PaperlayerView exact revision on arXiv

Source-linked AI summary

CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation

Aarush Sinha, Arion Das, Soumyadeep Nag, Charan Karnati, Shravani Nag, Chandra Vadhan Raj, Aman Chadha, Vinija Jain, Suranjana Trivedy, Amitava Das

arXiv:2604.09746v1cs.MAcs.AIcs.CL

TL;DR

The paper asks how strategic behavior emerges in multi-agent LLM environments and studies it in a controlled NYC simulation with opposing Blue and Red incentives. Using repeated interaction rollouts and KTO-based policy updates, it finds improved task completion and selective cooperation, but persistent vulnerability and a safety–helpfulness trade-off.

  • Problem

    Strategic behavior in multi-agent LLM systems requires controlled evaluation because strong agents remain unreliable on realistic planning workloads.

  • Method

    The paper uses an adversarial NYC navigation simulation and a closed-loop KTO alignment pipeline updated over repeated interaction generations.

  • Results

    57.3% task success is achieved at the best-performing generation versus 46.0% for the base policy, while minimum susceptibility remains 70.7%.

  • Takeaways & Limitations

    LLM agents exhibit limited strategic behavior, including selective cooperation, while Red agents retain substantial long-horizon influence and agents remain vulnerable to adversarial persuasion.

  • Takeaways & Limitations

    The results may reflect Qwen3-specific behavior, a static simulated NYC environment, and unreliable automated judging of manipulation susceptibility.

Abstract

from arXiv · show

As large language models (LLMs) are increasingly deployed as autonomous agents, understanding how strategic behavior emerges in multi-agent environments has become an important alignment challenge. We take a neutral empirical stance and construct a controlled environment in which strategic behavior can be directly observed and measured. We introduce a large-scale multi-agent simulation in a simplified model of New York City, where LLM-driven agents interact under opposing incentives. Blue agents aim to reach their destinations efficiently, while Red agents attempt to divert them toward billboard-heavy routes using persuasive language to maximize advertising revenue. Hidden identities make navigation socially mediated, forcing agents to decide when to trust or deceive. We study policy learning through an iterative simulation pipeline that updates agent policies across repeated interaction rounds using Kahneman-Tversky Optimization (KTO). Blue agents are optimized to reduce billboard exposure while preserving navigation efficiency, whereas Red agents adapt to exploit remaining weaknesses. Across iterations, the best Blue policy improves task success from 46.0% to 57.3%, although susceptibility remains high at 70.7%. Later policies exhibit stronger selective cooperation while preserving trajectory efficiency. However, a persistent safety-helpfulness trade-off remains: policies that better resist adversarial steering do not simultaneously maximize task completion. Overall, our results show that LLM agents can exhibit limited strategic behavior, including selective trust and deception, while remaining highly vulnerable to adversarial persuasion.

1 Introduction

The paper studies strategic behavior in a controlled adversarial urban simulation, where Blue agents navigate toward destinations while Red agents steer them toward billboards. Iterative KTO alignment improves task success and supports selective cooperation, but agents remain vulnerable to persuasion.

  • 150 Blue agents pursue destinations while 100 Red agents use adversarial framing to steer them toward billboard locations in a New York City routing topology.
  • The study introduces an adversarial urban simulation and iteratively applies KTO to interaction data across successive policy generations.
  • 57.3% task success is reached at the best-performing generation, up from 46.0% in the base policy, while minimum susceptibility remains 70.7%.
  • Aligned agents combine cooperation and caution, reducing over-refusal while preserving selective responses during conversational encounters.

3 Generations, Runs, and Policy Learning

The policy-learning pipeline repeatedly simulates interactions, converts trajectory judgments into unpaired desirable and undesirable examples, and fine-tunes policies with KTO. Updated policies are redeployed to generate the next iteration's data.

  • The environment contains 150 Blue goal-directed navigators and 100 Red adversaries that steer Blue agents toward billboard locations through dialogue.
  • The alignment procedure runs for 10 iterations, each combining data augmentation, KTO optimization, and post-finetuning simulation rollout.
  • Qwen3-4B generates the initial trajectories across 250 agent instances, capturing benign navigation and adversarial interaction patterns.
  • Each iteration produces unpaired desirable and undesirable supervision, with augmented data generated by Qwen3-14B for KTO fine-tuning.
  • KTO directly uses unpaired examples and centers desirable responses above, and undesirable responses below, a baseline through class-specific weights and z0.
  • After fine-tuning, the updated policy generates fresh trajectories that replace the previous dataset for the next alignment iteration.

4 From Emergent Strategy to Observable Metrics

The paper evaluates Blue–Red interactions using reconstructed trajectories and metrics for completion, safety, trust, efficiency, long-horizon robustness, manipulation effectiveness, and utility. These measures separate immediate behavior from delayed adversarial influence.

  • Reconstructed trajectories evaluate Blue and Red agents across safety, efficiency, and task utility.
  • Task Success Rate and Susceptibility Rate summarize destination completion and exposure to adversarial billboard locations.
  • Blue agents are divided into four mutually exclusive outcomes: reached destination and safe, reached destination but conned, lost but safe, and lost and conned.
  • Immediate interaction metrics distinguish Blue–Red gullibility from resistance and evaluate Blue–Blue helpfulness interactions.
  • Mean Trajectory Length and Path Redundancy measure navigation efficiency using visited locations and reconstructed paths.
  • Long-horizon safety tracks first billboard-hitting time and censored safe trajectories, while Red metrics measure manipulation effectiveness and causal delay.
  • A parameterized utility combines completion, safety, and efficiency, alongside separate Blue and Red utility reports.

5 Results & Findings

Across generations, alignment produces meaningful but non-monotonic improvements in task success, safety, utility, and selective cooperation. Persistent delayed manipulation and disagreement between performance and safety reveal a continuing safety–helpfulness trade-off.

  • Quantitative Analysis: 57.3% task success in run 10 improves on the 46.0% base policy, while run 8 reaches the lowest susceptibility at 70.7%.The safest and best-performing generations do not coincide.
  • Quantitative Analysis: Later policies preserve relatively stable trajectory length and path redundancy while modestly delaying compromise and increasing censored safe trajectories.
  • Quantitative Analysis: Immediate resistance to malicious advice stays above 90%, while later generations improve Blue–Blue trust efficacy and reduce over-refusal.
  • Quantitative Analysis: Blue utility remains negative, although run 10 achieves the highest Blue utility and Red utility declines slightly in better-aligned runs.
  • Post-hoc Analysis: Repeated steering occurs in 673 episodes and yields 93.9% susceptibility with a 39.8% Blue reach rate, while delayed compromise yields 100% susceptibility and 23.2% reach.
  • Post-hoc Analysis: Among agents initially rejecting malicious suggestions, 84.6% are ultimately manipulated, showing that early resistance does not reliably predict eventual safety.
  • Post-hoc Analysis: Confusion under conflicting advice is the dominant Blue failure mode, with 525 episodes and 93.5% susceptibility.
  • Post-hoc Analysis: The analysis indicates that future alignment must preserve long-horizon goal consistency under persistent, socially credible misdirection.

6 Conclusion

The paper presents a controlled empirical study of strategic behavior under repeated adversarial interaction. Iterative alignment improves several behavioral metrics, but gains remain partial, non-monotonic, and far from robust autonomy.

  • The study constructs a controlled setting where strategic behavior can be directly observed and measured.
  • Iterative alignment improves task completion, reduces susceptibility, and strengthens selective cooperation without sacrificing trajectory efficiency.
  • The safest and best-performing generations do not coincide, while Red agents retain substantial long-horizon influence.
  • The resulting behavior is limited but fragile: measurable strategy-like behavior remains far from robust autonomy.

7 Limitations

The study’s conclusions are constrained by its model family, simulated environment, evaluation reliability, and persistently negative aggregate Blue utility.

  • Exclusive reliance on Qwen3-4B and Qwen3-14B may leave the observed adversarial dynamics architecture-specific.
  • A simulated NYC map with static billboards cannot establish whether agents learn spatial reasoning or memorize local geographic heuristics.
  • Manipulation Susceptibility received exceptionally low inter-annotator agreement, with Krippendorff’s alpha of 0.204.
  • Aggregate Blue utility remains negative across all configurations despite KTO improving task success.

8 Ethics Statement

The paper frames its controlled simulation as a safety-oriented analysis of strategic behavior, trust, and adversarial persuasion rather than a real-world deployment study. It highlights risks of persuasive agentic systems while avoiding human-subjects and personal-data use.

  • The study uses a controlled multi-agent simulation and does not involve human subjects, personal data, or real-world navigation deployment.
  • The Red/Blue setting is presented as an analytical abstraction for hidden identities, conflicting incentives, and persuasive dialogue.
  • The paper highlights potential misuse of persuasive agents for manipulation, misleading advice, or targeted influence.
  • Findings are framed around safety vulnerabilities and the tension between task success and resistance to adversarial steering.

9 Frequently Asked Questions (FAQs)

The FAQs position the paper primarily as a controlled behavioral study of limited strategic adaptation under repeated adversarial interaction. They clarify the role of KTO, interpret the gains as structured but fragile, and emphasize limits on causal and ecological claims.

  • Contribution: The primary contribution combines an urban simulation, a closed-loop alignment pipeline, and behavioral evaluation across generations.
  • Method: KTO is used because outcome-supervised, unpaired trajectory judgments fit the setting more directly than SFT, DPO, or PPO supervision.
  • Why this matters: The environment enables controlled observation of hidden identities, adversarial objectives, and dialogue-mediated steering.
  • Incomplete gains: 46.0% to 57.3% task success improvement remains modest because susceptibility reaches 70.7% at best and Blue utility remains negative.
  • Behavioral gains: Later generations show improved task completion, reduced susceptibility, better blue–blue trust efficacy, lower over-refusal, and stable trajectory efficiency.
  • Interpretation: Post-hoc adversarial analysis identifies recurrent failure patterns but does not establish intervention-level causal mechanisms.
  • Goal integrity: Robust alignment requires preserving the original task objective across multiple turns, not merely making one correct local move.
  • Evolution: The dynamics are best understood as partial co-evolution with non-monotonic gains rather than simple convergence.

D Extended Interpretation of Quantitative Results

The extended analysis decomposes agent outcomes across task success, susceptibility, social calibration, efficiency, and utility. Post-hoc trajectory analysis further characterizes adversarial behavior using trace-derived linguistic and path features.

  • Metric interpretation: Task success, susceptibility, resistance, trust, over-refusal, efficiency, and utility jointly distinguish navigation completion from safe and socially calibrated behavior.Table 2 defines these metrics and outcome categories for comparing alignment generations.
  • Metric interpretation: Aggregate task success cannot reveal whether agents reached destinations safely or after visiting adversarial billboard locations.Figure 2a partitions rollouts into four mutually exclusive destination-and-safety outcomes.
  • Metric interpretation: 46.0% to 57.3% task success improvement was non-monotonic, with co-evolving Blue and Red policies solving some subproblems while exposing new weaknesses.TSR measures destination completion, whereas susceptibility captures exposure to adversarially induced states.
  • Adversarial robustness: Long-horizon red influence shows that sustained interaction can cumulatively compromise agents even when immediate resistance to adversarial suggestions is high.Robust planning therefore requires evaluating downstream trajectory shaping, not only one-shot responses.
  • Efficiency and calibration: Stable mean trajectory length and path redundancy indicate that improved safety is not explained simply by stalling, refusal to move, or excessively conservative routing.These metrics test whether robustness gains preserve execution quality.
  • Efficiency and calibration: Aligned agents become better calibrated by rejecting malicious advice while retaining productive trust in benign advice, rather than ignoring external input globally.Blue-red resistance, blue-blue trust efficacy, and over-refusal separate selective cooperation from uniform defensiveness.
  • Post-hoc analysis: The post-hoc adversarial analysis extracts dialogue cues and trajectory deviations from interaction logs, comparing actual routes with billboard-free reference paths.Features include helpfulness, social trust, malicious intent, blue trust language, extra path length, detours, and Levenshtein distance.

E.2 Extended Results

The extended results compare attack strategies, attack strengths, counterfactual conditions, and Blue failure modes. They show that repeated, strong, or sustained adversarial interaction is associated with greater susceptibility and poorer navigation outcomes.

  • Attack strategies: Repeated steering is the most common and highly effective attack, while delayed compromise guarantees susceptibility but often prevents destination reach.The comparison is reported in the attack-strategy analysis.
  • Attack strength: Strong attacks involving multiple Red agents and persistent interventions drastically reduce reach rate and almost guarantee susceptibility.Outcomes are stratified by heuristic attack strength.
  • Counterfactual subsets: Multiple Red contacts and long conversations are highly correlated with increased susceptibility and longer trajectory deviations.Counterfactual subsets compare Blue performance under specific adversarial conditions.
  • Failure modes: Confusion under conflicting advice is the leading categorized cause of Blue-agent failure.This failure mode reflects difficulty balancing original instructions against persistent external misdirection.

F Prompts

The appendix evaluates navigation behavior with an LLM judge and presents recurring strategies and failure patterns in Blue–Red interactions. The analysis highlights route-logic resistance, cooperative anchoring, and cumulative vulnerability to socially framed manipulation.

  • F.1 LLM-as-a-judge evaluation: The judge scores agent trajectories across goal directedness, manipulation susceptibility, and reasoning quality using a 1–5 Likert scale.Judges receive destination information and step-by-step trajectory transcripts before assigning scores.
  • F.1 LLM-as-a-judge evaluation: Goal Directedness measures efficient progress toward the destination, while Manipulation Susceptibility measures resistance to adversarial distraction.
  • G.1.1 Destination-Anchor Reasoning: Blue agents can reject Red suggestions by checking destination geography, direction, connectivity, and transit logic before accepting them.Examples show agents explicitly explaining why Flatiron, High Line, and 125th Street suggestions conflict with their routes.
  • G.1.3 Collaborative Transit Anchoring: Blue–Blue interactions can converge on geographically defensible transit hubs, creating a distributed route-correction mechanism against Red manipulation.Agents independently select or endorse hubs such as Midtown Manhattan and Manhattan Bridge.
  • G.1.4 Destination Reached Awareness (Terminal Convergence): Goal-completion detection is inconsistent: some agents stop at their destination, while others continue wandering after reaching destination-adjacent locations.The appendix contrasts terminal convergence with continued route generation after arrival.
  • G.1.5 Social Compliance Cascade (Predominant Vulnerability): Social compliance accumulates when compelling Red framings make each locally rationalized waypoint the prior for the next decision.A Blue agent can spend more than ten iterations in a billboard corridor without being explicitly deceived in one step.
  • G.2 Red attack strategies: Red agents chain billboard waypoints and frame deviations as scenic, cultural, nearby, or time-sensitive to make adversarial routes appear useful.The described corridor runs through Hudson Yards, Flatiron, SoHo, 42nd Street, 8th Avenue, and the High Line.

G.3.1 New Blue Agent Strategies from Alignment

Alignment produces new Blue strategies that explicitly justify route choices and offer counter-routes, but resistance remains tactic-specific. Agents can reject efficiency-based diversions while still accepting scenic framing and continuing cooperation after arrival.

  • New Blue Agent Strategies: Generation 3 introduces a polite counter-proposal in which Blue agents explain why a Red suggestion is inefficient and name a more direct alternative.This behavior was rarely observed in generations 1 and 2.
  • New Blue Agent Strategies: Blue agents verbalize destination constraints as a shield against manipulation, replacing earlier silent resistance or verbal agreement.Examples explicitly connect the destination to a direct transit route.
  • Route-logic counter-proposals: Blue agents can reject geographically inefficient suggestions while proposing alternatives such as Midtown Manhattan, the Staten Island Ferry, or the Financial District.These examples show route-logic reasoning paired with explicit verbal responses.
  • Partial Immunization Paradox: Resistance improves against traffic pseudo-authority and transit-hub false claims, while scenic framing remains high and proximity-illusion attacks increase from 270 to 305.
  • Partial Immunization Paradox: The same agent can resist an efficiency-based detour yet accept High Line scenic framing, demonstrating tactic-specific rather than general resistance.
  • Destination-as-Terminal-Hub: Some Generation 3 agents treat a reached destination as a cooperation node for other Blue agents instead of terminating immediately.This differs from earlier terminal-convergence behavior.

G.3.2 New Red Agent Strategies from Alignment

Across generations, Red agents increasingly hardened persistence, compressed their targeting around dense billboard corridors, and shifted away from urgency framing while maintaining false-efficiency persuasion.

  • Persistence and targeting: Red agents increasingly refused to change suggested waypoints after Blue rejection, producing a monotonically increasing persistence strategy.This was identified as the most statistically robust Red evolutionary signal across the first four generations.
  • Persistence and targeting: Repeated corridor flooding sustained manipulation even though most individual interactions were resisted.A Generation 3 Red agent repeatedly broadcast a near-identical corridor, with enough Blue agents complying to maintain the adversarial path.
  • Persistence and targeting: Red’s preferred routes contracted from broad dispersal into a dense Flatiron–Herald Square–Times Square–42nd Street nucleus.Peripheral locations such as Hudson Yards and broader cultural institutions declined as the corridor concentrated.
  • Tactic evolution: Urgency framing fell 59%, from 41 instances in Generation 1 to 17 in Generation 4.The tactic declined across successive generations, while false-efficiency language remained the highest-frequency Red tactic.
  • Population-level transition: Generation 8 marked a population-level compliance transition after earlier generations showed oscillating compliance and individual strategy emergence.Subsequent generations fell below the earlier susceptibility pattern despite greater total Blue–Red exposure.

G.4.1 New Blue Agent Strategies from Alignment

Later Blue policies developed more explicit, geographically grounded resistance to Red suggestions, including destination assertions, direct destination capture, and competing hub proposals, while scenic framing remained highly effective.

  • Competing route proposals: Generation 8 introduced substitute-hub proposals that rejected Red’s route and offered a geographically superior alternative in the same verbal turn.This extended earlier silent redirection and destination assertion into explicit competitive reasoning about route quality.
  • Destination-grounded resistance: Generation 10 introduced direct destination capture, with Blue agents choosing the actual destination rather than an intermediate suggestion.Examples include heading directly to Newark Liberty Airport or choosing a direct route toward Queens after consulting Red.
  • Destination-grounded resistance: Generation 9 agents named destinations in 47.4% of verbal responses, up from 41.1% in Generation 1, increasingly using them as directional counterarguments.The destination reference shifted from background context to an explicit test of geographic coherence.
  • Red adaptation: Red multi-targeting declined from a Generation 6 peak of 4.37 to 3.01 in Generation 10, indicating a shift toward fewer, better-targeted manipulations.Red persistence after resistance peaked at 65.8% in Generation 9 before easing in Generation 10.
  • Remaining vulnerability: Urgency resistance remained stable when the tactic was reintroduced, but scenic framing persisted as the main unresolved vulnerability.Scenic framing never fell below 70% susceptibility and reached 78.5% in Generation 10, unlike urgency and transit-hub false claims.
Loading 2604.09746v1…