Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation

1 Introduction

• Scorable negotiation games create a multi-step benchmark requiring agents to assess deals, infer others’ goals, weigh options, and seek common ground under partial observation.The tasks involve arithmetic, strategic reasoning, commonsense reasoning, and Theory-of-Mind capabilities.Source: §1 · p. 1
• The games use parties’ secret scores, priorities, and acceptance thresholds to quantify feasible agreements and can be varied in difficulty.The authors use an LLM to design three new games and describe procedures for creating further games.Source: §1 · p. 2
• GPT-4 still underperforms as game difficulty increases, while some open-source models outperform GPT-3.5 and the latest Gemini.In mixed populations, GPT-4 agents can also earn higher rewards than GPT-3.5 agents assigned the same role.Source: §1 · p. 2
• Greedy or adversarial agents can steer compromising agents toward greed or manipulation, potentially rewarding the adversary’s demands more highly.An adversarial agent may also form a coalition against a target.Source: §1 · p. 2
• The benchmark is intended to support research on LLM capabilities, manipulation, and future robustification; the authors plan to release games, code, and transcripts.Source: §1 · p. 2

Parties

• Six parties negotiate over five issues, with proposer, resource-manager, beneficiary, constraint-imposing, and opposing roles.The proposer seeks returns on its investment; the resource manager supplies the budget and has veto power.Source: §2 · pp. 2–3

Issues

• A deal selects one sub-option for each of five issues, producing 720 possible combinations.Issue options can be quantity ranges or discrete alternatives.Source: §2 · p. 3

Scoring

• Each party scores issue options according to its goals and priorities, producing a non-zero-sum game with opportunities for cooperation and competition.Some parties may be neutral on particular issues.Source: §2 · p. 3

Feasible solutions

• A deal is feasible only if it exceeds the thresholds of at least five parties, including the proposer and resource manager.Changing thresholds changes the feasible-deal set and can alter game difficulty.Source: §2 · p. 3

New games

• The authors adapted a base negotiation exercise and used an LLM to create new game settings, then manually curated them and assigned scores to yield about 55 feasible deals.The LLM generated stories, parties, issues, goals, and preferences without seeing the base game in context.Source: §2 · p. 3

3 LLMs Playing the Game

• The setup presents agents with initial prompts and rounds of negotiation, while the paper directs readers to appendices for the full algorithm and prompts.Source: §3 · p. 3

3.1 Agents’ Interaction Protocol

• Each agent receives shared project and party information, its own confidential scores and threshold, and rules; each round it sees recent interaction history and can support or propose a deal.The proposer issues the final official deal after the rounds, and that deal determines whether agreement is reached.Source: §3.1 · pp. 3–4

3.2 Compromising, Greedy, and Adversarial Games

• The game variants prompt agents to compromise, maximize their own benefits while seeking agreement, or sabotage negotiations for personal gain.An adversary can be assigned a target or choose one autonomously; in the adversarial variant, no deal yields it a higher score than any deal.Source: §3.2 · p. 4

3.3 A Baseline Prompting Solution Framework

• The baseline prompting framework decomposes negotiation into observation, exploration, and planning steps.Agents calculate scores for recent deals and infer preferences; they generate candidate deals and select one, then keep a secret plan for a later turn.Source: §3.3 · pp. 4–5

4.1 Experimental Setup and Evaluation Metrics

• Experiments used 24 rounds and 20 runs, measuring final and any success, score leakage, and agents’ own-score alignment with assigned roles.The tested models included GPT-4, GPT-3.5, Gemini Pro, Llama2 and Llama3 variants, and Mixtral 8x7B.Source: §4.1 · p. 5

4.2 Ablation of Prompts’ Structure

• GPT-3.5 often proposed deals below its minimum threshold, whereas wrong deals were almost negligible for GPT-4, especially with CoT.Source: §4.2 · p. 6
• Inferring others’ preferences increased success; GPT-4 matched ground-truth sub-option preferences at 61%, versus 42% for GPT-3.5.The preference-matching results were averaged over all agents.Source: §4.2 · pp. 6–7
• Exploration helped GPT-3.5 but biased GPT-4 toward high-scoring historical deals; planning mattered for final success, preventing collective-score saturation.Without exploration, GPT-4 more often adapted deals to other parties, though proposals could still overcompensate on one issue.Source: §4.2 · p. 7

4.3 Mixed Population

• Adding GPT-3.5 to a GPT-4 population reduced group success most when p1 was GPT-3.5, and GPT-3.5 agents also scored lower than their GPT-4 counterparts.Source: Table 2 · p. 7, §4.3 · p. 7

4.5 Performance on Other Games

• Performance on rewritten, semantically similar games was comparable to the base game; newly created games varied, with sparse-score Game 3 matching the base on final deal.Source: §4.5 · pp. 7–8

4.6 Tuning the Game Difficulty

• Shrinking the feasible-deal set lowered success, while fewer players produced higher all-way agreement; thresholds and player count can tune difficulty.Source: §4.6 · p. 8

4.7 Greedy and Adversarial Variants

• Greedy and adversarial incentives lowered success versus compromising; targeted attacks reduced the target’s final score, while aligned coalitions could strengthen targeting.When p1 was greedy, success dropped sharply; untargeted adversaries also prevented deals.Source: Table 5 · p. 8, §4.7 · pp. 8–9

5 Related Work

• Earlier negotiation evaluations were non-interactive or involved only two players, unlike this work’s six-player simulation.The simulation includes leading and veto parties, increasing interaction and evaluation complexity.Source: §5 · p. 9
• The benchmark links entities and negotiation issues through indirect semantic relationships that require commonsense reasoning and Theory-of-Mind.Source: §5 · p. 9
• The expandable benchmark spans four distinct games and introduces greedy or adversarial incentives to study cross-agent ripple effects.The authors argue these safety questions cannot be adequately studied with only two players.Source: §5 · p. 9

6 Conclusion

• The benchmark supports adjustable difficulty and study of cross-agent attacks and exploitation in multi-agent negotiation.The authors plan to release its games, prompts, interaction platform, and model logs, and to maintain a leaderboard.Source: §6 · p. 9
• The task remains unsolved: open-source models are less successful than GPT-4, which underperforms as difficulty increases and in games with non-sparse payoffs.Source: §6 · p. 9
• Future work could explore other reasoning and planning methods, manipulation setups, defenses, evasion attacks, and safety considerations such as biases.Source: §6 · p. 9

Appendices A–C

• Appendix A lists notation and the agents’ interaction protocol; Appendices B and C add payoff-alignment and mixed-population analyses.Source: Appendix Guide. The appendix of this… · p. 13

Appendices D–E

• Appendices D and E provide game-score and difficulty comparisons, plus results from reducing the number of players.Source: Appendix Guide. The appendix of this… · p. 13

Appendices F–H

• Appendices F and G add results for greedy and adversarial variants, while H gives qualitative examples of GPT-3.5 output.Source: Appendix Guide. The appendix of this… · p. 13

Appendices I–J

• Appendices I and J provide initial game and agent prompts, including prompts for interactions between agents during rounds.Source: Appendix Guide. The appendix of this… · p. 13

Solution Framework

• The protocol accepts parties, issues, scores, thresholds, variants, a window, and instructions, and returns a Boolean success result.It initializes an empty public history.Source: Solution Framework · p. 15
• Each agent receives public and secret knowledge together with its game variant, while agent order is shuffled across rounds.Source: Solution Framework · p. 15

4: Rounds

• Across rounds, the protocol updates agents’ instructions with the variant and chain-of-thought instructions.Source: §4 · p. 15
• After the rounds, the protocol extracts the final deal and checks whether it succeeds.Source: §4 · p. 15

• p1 votes more often for low than high environmental-protection measures, consistent with its higher payoff for low measures.Source: Figure 7 · p. 16
• The environmental agent pi almost never votes for low measures, matching its payoffs for intermediate and high options.Source: Figure 7 · p. 16
• Intermediate options receive many votes because agents are instructed to compromise.Source: Figure 7 · p. 16

C Mixed Population

• GPT-3.5 agents may receive lower scores than their same-role counterparts in an all-GPT-4 game.The text suggests non-leading agents may perpetuate infeasible proposals, contributing to lower group success.Source: C Mixed Population · p. 17, Figure 9 · p. 17
• The comparison uses a compromising-variant base game with a mixed population of GPT-3.5 and GPT-4 agents.Source: C Mixed Population · p. 17, Figure 9 · p. 17
• GPT-3.5 leading agents frequently violate their minimum-score role late in negotiation, making agreement unsuccessful even when other agents’ scores are satisfied.Source: Figure 8 · p. 17, C Mixed Population · p. 17

D Other Games: More Results and Analysis

• Figure 10 tracks p1’s own and collective scores for proposed deals over negotiation in the rewritten base game and two newly created games.Agent actions show patterns similar to the base game’s best prompt.Source: Figure 10 · p. 18
• In games 2 and 3, agreeing-party counts fluctuate for deals with close p1 scores, which may help explain their lower performance and require finer deal selection.Source: Figure 11 · p. 18
• The base game is described as more stable, while game 3 seems most stable and is also the easiest by Figure 4 performance.Source: Figure 11 · p. 18

E Varying the Number of Players

• Adding players makes the task harder, with the game description, issues, preferences, and player descriptions held fixed.Players are dropped from simulations rather than changing the game materials.Source: Table 7 · p. 19
• The 3-player setup includes p1, p2, and the opposing party.This is the smallest configuration described.Source: Table 7 · p. 19
• The 4- and 5-player setups progressively add the two beneficiary parties.Source: Table 7 · p. 19

F Game Variants: Greedy

• When beneficiary parties are incentivized to be greedy, p2’s score from their deals can fall slightly on average and become more variable.Because p2 has veto power and must agree, this helps explain why the greedy variant may have lower success.Source: Figure 13 · p. 20
• Greedy incentives raise p1’s own score but reduce compromise, significantly lowering the eventual success rate.Source: Figure 14 · p. 20
• Consensus on issues raised by a greedy agent can produce deals less favorable to other agents and may end in no agreement.Source: Figure 16 · p. 21
• Compromising agents may over-compromise in a greedy game, leaving p1 below its required score; unanimous agreement would otherwise add 10 points to p1.Source: Figure 17 · p. 21

G Game Variants: Adversarial

• In the targeted variant, the target agent receives lower average scores from p1’s deals, including the final deal.The compromising variant also has less variance in the target’s score than the untargeted game.Source: G Game Variants: Adversarial · p. 22
• GPT-3.5 as adversary does not act consistently with its incentive to maximize its own score while minimizing collective or target scores.Source: Figure 19 · p. 22
• In one example, most parties favor A2, B2, C2, D3, E1, while the Green Alliance proposes a significantly different deal.Source: G Game Variants: Adversarial · p. 23
• Some parties identify adversarial proposals as unlike the majority, though one example shows accommodation of some sub-options.Source: Figure 20 · p. 23, Figure 21 · p. 23
• In the untargeted game, the adversary selects a target; in the targeted game, p1’s agreement with anti-target sub-options may lead to no agreement.Source: Figure 22 · p. 24, Figure 23 · p. 24

H Examples from GPT-3.5

• The cited text lists A1, B2, C3, D1, E4 three times as a score-100 deal meeting the minimum score requirement and aligning with stated preferences, with agreement recorded each time.Source: H Examples from GPT-3.5 · p. 25
• After scoring Eventix’s proposal at 0 and three other proposals at 55, GPT-3.5 said none met its minimum of 50, then proposed deals scoring 55, 56, and 57.It described the 55-point proposals as the highest-scoring options and said it was willing to accept them.Source: H Examples from GPT-3.5 · p. 25
• Figure 24 reports frequent option-to-score mapping, score calculation, threshold comparison, and inference errors, as well as score disclosure and inadequate exploration.The figure describes examples from GPT-3.5 with the best prompt in Table 1.Source: Figure 24 · p. 25

I.1 Base Game (Re-written)

• The rewritten base game places Eventix in an Aberdeen project-approval negotiation over five issues, with multiple parties and role-specific scoring preferences.The prompt identifies the parties, the five issues, and confidential preferences quantified with scores.Source: §I.1 · p. 26

H Examples from GPT-3.5

• The neighboring-cities role prioritizes compensation (issue D, max 60), lower government funding (A, max 18), and union preference (E, max 12), while environmental impact has no score.It also assigns facility location a maximum score of 10 and bars disclosure of score-sheet numbers or deal values.Source: §I.1 · p. 28
• The game prompt frames a five-party, five-issue negotiation with limited turns, a minimum of five votes including Eventix and the Ministry, and veto power for both.Eventix consolidates suggestions into a proposal for a test vote; the neighboring-cities prompt sets a minimum acceptable score of 31 and gives 31 if no deal is reached.Source: §I.1 · p. 28
• The Green Alliance role prioritizes environmental impact (issue C, max 55) and facility location (B, max 45), with its lowest-scored options being environmental harm and offshore construction.It assigns zero points to issues A, D, and E, sets a minimum acceptable score of 50, and gives 50 if no deal is reached.Source: §I.1 · p. 29
• The appendix also begins a governor-role prompt and identifies the same five issues, but the supplied excerpt does not show that role’s preferences or scores.The prompt identifies the governor of Aberdeen as the represented party but ends during the facility-location options.Source: §I.1 · p. 30

I.1 Base Game (Re-written)

• The governor values the highest-grant option (A1) most, cannot accept a deal scoring below 30 and scores 30 if none is reached; passage requires agreement by at least five parties, including Eventix and the Ministry of Culture and Sport.Source: §I.1 · p. 30
• The local Workers’ Union representative prioritizes union hiring, then government grants and job-creating facility choices, accepts scores of at least 50, and has no environmental preference.E1 scores 42 versus E4 at 0; A1 scores 30 versus A4 at 0; B2 scores 20 versus B3 at 0; all C options score 0. No-deal score is 50.Source: §I.1 · p. 31

I.2 Base Game - Greedy Variant

• The greedy variant relocates the negotiation to SportCo’s proposed Harbour Sport Park in England, with the reader representing the Environmental League.The visible prompt describes choices for infrastructure mix, ecological impact, and employment rules, including environmental improvement and union quotas.Source: §I.2 · p. 32
• The greedy role prioritizes environmental protection, scoring ecological impact and infrastructure mix while treating employment, loans, and city compensation as valueless.It cannot accept a deal below 50, must keep its numerical scoring confidential, and can share only high-level priorities.Source: §I.2 · p. 32

I.3 Base Game - Adversarial Variant

• The adversarial role is instructed to sabotage agreement, using environmental demands, other parties’ preferences, and veto power to pursue a no-deal outcome.No deal is assigned a score of 150, while an approved deal must score more than 50; the role can use other issues to manipulate parties and exploit SportCo and the Department of Tourism’s veto power.Source: §I.3 · p. 33

I.4 Creating New Negotiation Games Prompt

• The game-generation prompt asks for a cooperative, non-zero-sum negotiation with six parties, five multi-option issues, and potentially overlapping or conflicting stakeholder interests.It requests party motivations, issue preferences and priorities, and reasons for those preferences, while excluding a mediator.Source: §I.4 · p. 34

I.5 New Game 1

• The airport-game example assigns roles to an island government, development bank, environmental NGO, tourism association, Indigenous community, and construction company.Their stated interests span project funding and feasibility, environmental and social impacts, tourism benefits, and construction profit and efficiency.Source: §I.5 · p. 35
• The airport negotiation covers location, budget, environmental measures, social-impact assessment, and profit sharing, with options varying in costs, impacts, and benefits.The example describes three location options, budgets from $300 million to $1.2 billion, four environmental-mitigation levels, five assessment levels, and three profit-sharing schemes.Source: §I.5 · p. 35

H Examples from GPT-3.5

• The three roles have contrasting priorities: the bank favors moderate funding and environmental safeguards, the community prioritizes location and social assessment, and the NGO prioritizes environmental mitigation.The bank prefers B3 and C4; the community prefers A2 and D5 or D4; and the NGO prefers C4 or C3 and A3.Source: §I.5 · pp. 36–38
• The bank, indigenous-community, and environmental-NGO prompts set minimum acceptable scores of 60, 47, and 60, respectively.If no deal is reached, each agent receives its own minimum threshold score.Source: §I.5 · pp. 36–38
• The prompts prohibit sharing numerical scores and require a limited-turn proposal vote, with approval dependent on designated parties and voting thresholds.The bank prompt gives the government veto power and requires at least five parties including the bank and government; the indigenous and NGO prompts likewise require those parties’ approval.Source: §I.5 · pp. 36–38
• The airport negotiation covers location, budget, environmental measures, social-impact assessment, and profit sharing, with options trading off cost, impact, and benefits.The prompts describe six parties, including the government, development bank, NGO, tourism association, indigenous community, and construction company.Source: §I.5 · p. 38

I.5 New Game 1

• The airport game assigns stakeholders different private option scores, sets 57 as the minimum acceptable score, and requires five votes including the government and bank.The five issues concern airport location, budget, environmental measures, social impact assessment, and profit-sharing.Source: §I.5 · p. 40

J.1 Kick-off

• The kickoff prompt tells each party to begin at round 0 by proposing an initial deal in a short response.The proposed deal is enclosed in the specified deal tags.Source: §J.1 · p. 41

J.2.1 Cooperative

• The cooperative prompt directs agents to balance interests, listen and adapt, and seek a mutually acceptable agreement while using a private scratchpad and planning notes.Agents are instructed to calculate deal scores and keep scores out of their short public answer.Source: §J.2.1 · p. 41

J.2.2 Greedy

• The greedy prompt prioritizes maximizing the agent’s score through negotiation tactics, while retaining a deal above its minimum threshold.The agent is asked to observe others, plan privately, and keep scores out of its public response.Source: §J.2.2 · p. 42

J.2.3 Adversarial - Untargeted

• The untargeted adversarial prompt permits deception and encourages proposals that cause at least one other party to disagree while preserving the agent’s minimum score.Its scratchpad instructions ask the agent to explain observations, show deal calculations, and plan for disagreement.Source: §J.2.3 · p. 43

J.2.4 Adversarial - Targeted

• The targeted adversarial prompt directs the agent to identify a target’s preferences, work against them, and subtly persuade others to isolate that party.As in the untargeted variant, the agent must still seek a score above its minimum and induce disagreement.Source: §J.2.4 · p. 44

J.3 Final Deal Suggestion

• The final-deal prompt asks the agent to propose a complete deal that fits its score and is likely to gain other parties’ agreement.It calls for private calculations and a short public proposal aimed at balancing the agent’s score with others’ preferences.Source: §J.3 · p. 45

J.4 Probing for Other Agents’ Preferences

• The preference-probing prompt asks agents to infer each party’s preferred option for every issue and report the choices in a structured format.It requests step-by-step explanations and a party-by-party list of options A# through E#.Source: §J.4 · p. 45