Evaluating Language Model Agency through Negotiations

1 INTRODUCTION

• The evaluation need grows as language models enter interactive applications, where both performance and alignment require reliable assessment.The introduction notes commercial efforts to build agents for tasks requiring extensive interactive reasoning.Source: §1 · p. 1
• Static benchmarks poorly capture agent behavior and realistic economic constraints, while benchmark exposure, obsolescence, and limited multi-turn coverage complicate evaluation.Keeping benchmarks secret could reduce assessment validity, integrity, and transparency; dynamic tasks are proposed as an alternative.Source: §1 · pp. 1–2
• Language-model outcomes can depend on the context or persona being emulated, and harmful behaviors may appear only occasionally in extended dialogue.The introduction argues that multi-turn tasks can provide more insight into such behavior than single-turn performance tasks.Source: §1 · pp. 1–2
• Cross-model interactions are presented as a more effective way to test machine-to-machine outcomes than static tasks and self-play alone.The authors connect this need to the expected growth of machine-to-machine interactions.Source: §1 · p. 2
• Structured negotiations are proposed to assess language-model alignment and performance through dynamic, multi-turn, cross-model interactions.The authors describe negotiations as realistic, ecologically valid tasks whose complexity can be increased from relatively simple segments.Source: §1 · p. 2

2 DEFINING STRUCTURED NEGOTIATION GAMES

• A structured negotiation combines a setting, Issues, payoff tables, agent roles, and protocol rules; agents exchange private notes and public messages to maximize weighted payoffs.Payoff tables are not shared between agents, and failure to agree on any Issue yields zero total payoff.Source: §2 · p. 3

2.1 TYPES OF NEGOTIATION ISSUES

• Distributive Issues pit opposing interests against each other, while compatible Issues align interests; integrative preferences can create trade-offs that increase combined payoffs.The paper distinguishes non-integrative games, where agents share Issue preferences, from integrative games, where their preference weights differ.Source: Figure 2.1 · p. 3, §2.1 · pp. 3–4

2.2 TASK COMPLEXITY AND CO-EVOLVING BENCHMARKS

• Game complexity can vary with Issue count and negotiation setting, while self-play offers an internal benchmark and cross-play tests how results transport across models.Self-play can overestimate performance against cooperative opponents or underestimate a model that performs poorly against itself but beats weaker models.Source: §2.2 · p. 4

2.3 STATE-OF-MIND CONSISTENCY

• The protocol measures faithfulness by comparing agents’ stated acceptable offers with their own offers and their predictions of opponents’ acceptable offers.Internal unfaithfulness occurs when an agent offers less than it said it would accept; external unfaithfulness occurs when its offer exceeds what it predicted the opponent would accept.Source: §2.3 · pp. 4–5

3.1 CONTROLLING FOR BIAS

• The study balances intra-game biases by having each agent play both sides and starting positions, and reduces persona cues by removing gender references.The authors also describe agents as representatives of their respective negotiating parties.Source: §3.1 · p. 5

3.2 PERFORMANCE FACTORS

• Note and message length constrain planning and communication, while too much generative capacity may invite hallucinations and added costs.The experiments vary available note history rather than adding a separate memory module, partly because models may not fit the full negotiation history in context.Source: §3.2 · p. 5

3.3 BENCHMARKS

• Seven model families were tested, but models had to reach at least one agreement in ten single-issue distributive self-play games to qualify for extensive testing.Cross-play included qualifying models playing against the other qualifying models and was used to examine robustness to opponents’ messages.Source: §3.3 · pp. 5–6

3.4 COMPLETION CRITERIA AND EVALUATION METRICS

• Agreement rates distinguish internal-state alignment from explicit agreement phrases, and the evaluation reports normalized total payoffs, agreement payoffs, and instruction-following metrics.Negotiations end when both agents use the agreement phrase or after a maximum of 10 rounds; alignment measures also include internal and external faithfulness.Source: §3.4 · p. 6

3.5 NEGOTIATION GAMES EVALUATED

• Games increase in complexity by adding Issues, mixing distributive and compatible Issues, and introducing integrative preference weights that permit cooperative bargaining opportunities.In those latter games, both agents can obtain more than U = 0.5.Source: §3.5 · p. 6

4 RESULTS

• Most models advanced after qualification, but LLaMA 2 was excluded from cross-play after qualitative inspection of command-light self-play; gpt-4 averaged just over 15 runs versus 25+ for others.The gpt-4 results are marked with an asterisk because of the lower run count.Source: §4 · p. 6

4.1 SELF-PLAY

• gpt-4 led faithfulness and instruction-following but ranked near the bottom for agreement rate and needed the most rounds; gpt-3.5 was most efficient in self-play.All models followed formatting instructions reasonably well.Source: §4.1 · p. 6, Table 1 · p. 6
• In single-Issue distributive self-play, claude-2 had the highest agreement rate and command the lowest; gpt-4 appeared better at competitive agreements than command.Because these games are zero-sum, the passage identifies agreement rate as the metric of interest.Source: §4.1 · p. 7, Table 2 · p. 7
• In compatible-Issue self-play, command, claude-2, and gpt-3.5 agreed most often but reached mediocre agreements, while gpt-4 agreed less often but achieved near-perfect payoffs when it did.The authors interpret this as gpt-4 maximizing aligned interests when it reaches agreement, unlike the other three models.Source: §4.1 · p. 7, Table 2 · p. 7
• Adding a second Issue reduced agreement rates across models, which struggled to raise integrative distributive payoffs above U*=0.5.Integrative games offer trade-offs through differing Issue preferences but also make theory-of-mind inference more difficult.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 CROSS-PLAY

• Cross-play agreement rates rose for all models except claude-2, while average rounds fell except for a slight increase for gpt-3.5.Strong self-play models such as gpt-4 and chat-bison also appeared to improve lesser models’ instruction-following at some cost to their own performance.Source: §4.2 · pp. 7–8, Table 3 · p. 7
• gpt-3.5 led almost every cross-play category, while claude-2 fell in agreement-rate ranking and gpt-4 remained strong mainly in cooperative agreed-only payoffs.gpt-4 ranked near the bottom in many other categories; chat-bison’s performance was closer to the other models than in self-play.Source: Table 4 · p. 8, §4.2 · p. 8

• Paid APIs limited the number of experiments, reducing the authors’ ability to narrow confidence intervals.Source: §5 · p. 8
• Cross-play researchers may depend on third parties and face prohibitive costs; cheaper opponents plus ELO-style extrapolation are proposed as an alternative.Source: §5 · p. 8
• A shared prompt set may not benefit every model equally, despite using temperature 0.2 and averaging runs to mitigate prompt effects.Source: §5 · p. 8
• The framework may be misused to select agents solely for negotiation ability, although the authors also hope it will help expose risks and advance safety research.Source: §5 · p. 8

• LMs have been used to rank outputs, red-team, and evaluate alignment, and this work uses self- and cross-play negotiations to evaluate LMs themselves.Source: §6 · p. 9
• Prior LM-agent work adds tools, combines specialized agents, simulates societies, or supports robotic perception; this paper evaluates rather than builds or enhances agents.Source: §6 · p. 9
• Earlier negotiation systems often used restricted inputs and outputs, while hybrid RL–LM architectures added optimization structure.Source: §6 · p. 9
• Most recent LM negotiation studies focus on few-shot or single-issue games, whereas this work targets extended, arbitrarily complex games and jointly evaluates alignment and performance.Source: §6 · p. 9

7 CONCLUSION

• The authors propose negotiation games as an alternative to static benchmarks for studying real-world interaction, alignment, performance, and resistance to evaluation data leakage.Source: §7 · p. 9
• Only closed models could complete the tested tasks at the time of writing; models struggled with cooperative bargaining and multi-issue games, while gpt-4 underperformed in negotiation outcomes despite stronger faithfulness and instruction-following.Source: §7 · p. 9
• The authors call for further work on human negotiation biases, external tools, and repeated games, and invite broader contributions to evaluation benchmarks.Source: §7 · p. 9

A.1 DEFAULT SETTINGS

• The parameter sweep varied note and message limits across 32, 64, and 128 words, and varied access to prior notes and messages.Source: §A.1 · p. 16
• The defaults retained full message history, no previous notes when generating a new note, and only the most recent note when generating a message.Limited message history caused divergence; all-note history performed worse than the single-note setting, and no note history yielded the highest agreement rate for note generation.Source: §A.1 · p. 16
• A 32-word limit led to instruction-following breaks and slightly worse agreement; the authors chose 64 words because 64 and 128 had minimal agreement-rate differences and 128 mostly increased token use.Source: §A.1 · p. 17
• The authors could not rigorously test whether word limits should vary with the number of issues in a game.Source: §A.1 · p. 17
• Dialogue-style negotiations raised gpt-3.5’s agreement rate from 0.44 to 0.49, but the authors used transcripts for fair comparison across providers.Source: §A.1 · p. 17
• Showing round numbers increased gpt-4’s agreement rate by almost 50%, while it had little effect on gpt-3.5 convergence.The authors therefore included round numbers in prompts, despite concern that doing so could expose a subgame-perfect Nash equilibrium.Source: §A.1 · p. 17

A.2 SOFT AND HARD AGREEMENTS

• A soft agreement means both agents’ notes indicate they prefer the same acceptable offers, while a hard agreement requires aligned internal states and both agents to use the agreement phrase.Source: §A.2 · p. 18
• Hard agreements depend on instruction-following because both agents must utter the phrase; the authors expect hard metrics to replace soft metrics as LM agency advances.Source: §A.2 · p. 18

A.3 BIAS CONSIDERATIONS

• The persona-mixture analysis compares no persona description with explicit expert or novice negotiator descriptions to test for performance differences.The authors frame LMs as mixtures of personas and note that responses can change with prompt context.Source: §A.3 · p. 18
• To reduce potential RLHF artifacts, the authors format negotiation dialogues as transcripts and use a ‘Human’ message to present the transcript and prompt the next step.They note that traits such as increased honesty may be undesirable for negotiation.Source: §A.3 · p. 18
• Game descriptions may affect perceived difficulty even with identical payoff matrices, but the authors did not systematically test this inter-game bias.They experimented with settings such as corporate mergers and loan agreements.Source: §A.3 · p. 18

A.4 QUALIFIERS

• All tested models except the open-source LLaMA 2 models qualified by reaching at least one agreement in ten self-play games; LLaMA 2 struggled to maintain roles and follow instructions.The models used rental-agreement variants and a temperature of 0.2; the LLaMA 2 behavior worsened as models got smaller.Source: §A.4 · p. 18

A.5 EXTRACTING OFFERS FROM TEXTS

• Offers arise in mental notes and public messages, and the extraction pipeline uses regex first, then GPT-3.5 when regex fails.GPT-3.5 examples are also used to extract varied free-form offers; the formatting metric measures how often LM extraction is needed.Source: §A.5 · p. 19

B.1 NEGOTIATION RULES

• Negotiation rules require complete agreements, prohibit negative-payoff acceptance and side payments, and restrict offers to values in the payoff tables.Total payoff is the sum across issues; partial agreements yield zero.Source: §B.1 · p. 20

B.2 PROMPTS USED

• Prompts ask agents to reason over rules, payoff tables, and the transcript, then propose acceptable offers for each issue and advance at least one issue in a message.Mental notes and messages are each subject to a word limit.Source: §B.2 · p. 20

B.3 GAMES AND ISSUES

• The rental-agreement game assigns landlord and tenant roles across issues including duration, deposit, subletting, and rent, with payoff values specified for each issue.The role descriptions direct each agent to maximize its represented party’s interests using its own payoff information.Source: §B.3 · pp. 21–22

C API CONSIDERATIONS

• Direct-dialogue negotiation requires at least three API roles, and the described format is supported only by OpenAI and Meta among the providers considered.Google’s limited System role does not support it.Source: C API CONSIDERATIONS · p. 22, Table 7 · p. 22

D.1 DISTRIBUTIVE ISSUES

• For distributive issues, the analysis seeks an aggregate game-optimal payoff; equal preferences yield a lower bound of 0.5 per party, while exactly misaligned preferences permit both to reach 1.The allocation vector assigns each issue’s “pie” between the parties.Source: §D.1 · p. 23

D.2 MIXTURE OF DISTRIBUTIVE AND COMPATIBLE ISSUES

• The distributive-issue solution does not apply when compatible issues are present, because both parties benefit from the same allocation direction on those issues.The authors describe the resulting mixed cooperative-bargaining problem as involving trade-offs without a clearly defined optimal score.Source: §D.2 · p. 23

E.1 VISIBILITY AND AGENT STABILITY

• The visibility study used 162 self-play runs of single- to three-issue distributive games across three information settings.The settings varied whether agents saw the other party’s title, payoffs, or stated ability.Source: §E.1 · p. 24
• Expert descriptions corresponded to a subtle performance increase and awful-negotiator descriptions to a similar drop, but standard errors overlapped.The authors present imposed ability as a possible future direction for studying negotiating performance.Source: §E.1 · p. 24
• The appendix identifies Table 9 as reporting average normalized total payoffs by internal description and Table 10 as showing cross-play.The supplied passages do not report the tables’ values.Source: §E.1 · p. 24, Table 9 · p. 24, Table 10 · p. 24

F HEAD-TO-HEAD RESULTS FOR CROSS-PLAY EXPERIMENTS

• Tables 11 and 12 report head-to-head normalized agreed-only payoffs, separately for soft and hard agreements and competitive versus cooperative games.The captions define competitive games as non-integrative distributive games and cooperative games as including compatible issues or integration opportunities.Source: Table 11 · p. 25, Table 12 · p. 25
• Tables 13 and 14 report head-to-head agreement rates for soft and hard agreement games, respectively.The supplied captions do not state the comparative outcomes.Source: Table 13 · p. 25, Table 14 · p. 26

G TRANSCRIPTS AND EXAMPLES

• A command-light transcript example is flagged as potentially affected by API-specific caching because experiments were run in parallel.The appendix also provides example offers and mental notes from negotiations.Source: G TRANSCRIPTS AND EXAMPLES · pp. 27–28