Source-linked AI summary
Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback
Yao Fu, Hao Peng, Tushar Khot, Mirella Lapata
TL;DR
The paper asks whether LLMs can autonomously improve negotiation with minimal human intervention. It has buyer and seller agents bargain across rounds while a critic supplies natural-language feedback used as in-context demonstrations. Only some models improve continuously, and stronger agents face a tradeoff between better prices and deal success, raising oversight concerns.
Problem
The paper asks whether multiple LLMs can improve each other autonomously through game playing and AI feedback, potentially reducing reliance on human intervention.
Method
Buyer and seller LLMs negotiate over products while a critic provides natural-language feedback and prior dialogue is reused as in-context demonstrations across rounds.
Results
Only a subset of models continuously improve from iterative feedback; stronger agents can improve deal prices but face a higher risk of breaking the deal.
Takeaways & Limitations
The bargaining game provides an initial testbed for studying LLM capabilities in rule following, feedback use, role-dependent learning, and continuous improvement.
Takeaways & Limitations
Autonomous multi-agent improvement may be harder to oversee, motivating further alignment and safety research in this setting.
Abstract
from arXiv · showhide
We study whether multiple large language models (LLMs) can autonomously improve each other in a negotiation game by playing, reflecting, and criticizing. We are interested in this question because if LLMs were able to improve each other, it would imply the possibility of creating strong AI agents with minimal human intervention. We ask two LLMs to negotiate with each other, playing the roles of a buyer and a seller, respectively. They aim to reach a deal with the buyer targeting a lower price and the seller a higher one. A third language model, playing the critic, provides feedback to a player to improve the player's negotiation strategies. We let the two agents play multiple rounds, using previous negotiation history and AI feedback as in-context demonstrations to improve the model's negotiation strategy iteratively. We use different LLMs (GPT and Claude) for different roles and use the deal price as the evaluation metric. Our experiments reveal multiple intriguing findings: (1) Only a subset of the language models we consider can self-play and improve the deal price from AI feedback, weaker models either do not understand the game's rules or cannot incorporate AI feedback for further improvement. (2) Models' abilities to learn from the feedback differ when playing different roles. For example, it is harder for Claude-instant to improve as the buyer than as the seller. (3) When unrolling the game to multiple rounds, stronger agents can consistently improve their performance by meaningfully using previous experiences and iterative AI feedback, yet have a higher risk of breaking the deal. We hope our work provides insightful initial explorations of having models autonomously improve each other with game playing and AI feedback.
1 Introduction
The paper investigates whether LLMs can autonomously improve negotiation through self-play and natural-language AI feedback. It introduces a bargaining testbed and finds that only some models can understand the rules, use feedback, and improve continuously.
- Research question: The study asks whether LLMs can improve each other through competitive bargaining with minimal human intervention.Two models negotiate as buyer and seller, while a third model critiques one player's strategy.
- Research question: The bargaining game evaluates negotiation using a clear objective: sellers seek higher prices and buyers seek lower prices.The game is chosen because its textual rules and deal-price objective are well defined.
- Findings: Only gpt-3.5-turbo, gpt-4, and claude-v1.3 continuously improve over multiple rounds in the reported experiments.Other models either fail to understand bargaining rules, fail to incorporate feedback, or improve for only one round.
- Method: The approach, called In-Context Learning from AI Feedback, uses critic feedback and prior dialogue histories as demonstrations for later rounds.Unlike RLHF's scalar reward, the feedback is provided in natural language and is intended to support scalable self-improvement.
- Findings: A worked example shows the seller improving the deal price from $16 to $17 after receiving critic feedback.The seller incorporates the suggestion to emphasize the balloon's rarity and special qualities.
2 Problem Setting
The paper implements bargaining with fixed opening prices, conversational state tracking, and iterative criticism. Multi-round improvement becomes harder as prior gains raise the baseline, and only gpt-4 improves across five rounds.
- Experimental setup: The experiments compare combinations of cohere-command, AI21 jurassic-2, gpt-3.5-turbo, gpt-4, claude-instant-v1.0, and claude-v1.3.The critic uses the same language-model engine as the player it critiques; in one setting, gpt-4 improves as seller against a restarting gpt-3.5-turbo buyer.
- Process of the Game: Negotiations begin at $20 for the seller and $10 for the buyer, so rule-following deals should fall between $10 and $20.The full conversation history conditions each player's subsequent response until DEAL or NO DEAL.
- Critic: After each terminating round, the critic receives prior histories and feedback, then gives three suggestions for improving the player's next strategy.The improved player receives the feedback as text before the next round.
- The Moderator: The moderator classifies negotiations as ON-GOING, DEAL, or NO DEAL using recent conversation rounds and few-shot state examples.Four conversation rounds were empirically sufficient, although recognizing NO DEAL was harder than recognizing DEAL.
- Playing for Multiple Rounds: Only gpt-4 improves over 5 rounds, while other models' improvements may saturate at about 3 rounds.More rounds become progressively harder because each improved price becomes the next round's baseline.
3 Related Work
The work adapts AlphaGo Zero's self-improvement idea to natural-language bargaining while distinguishing itself from other multi-agent and AI-feedback studies through its explicit measurable objective.
- Game Playing and AlphaGo Zero: The setting is inspired by AlphaGo Zero: language-model agents play a competitive game and seek improvement with little human supervision.The paper transfers this self-play framing from Go to natural-language bargaining.
- Game Playing and AlphaGo Zero: Figure 3 illustrates that not all models can bargain: cohere fails to understand bargaining rules, while AI21 Jurassic-2 fails to incorporate critic feedback.The caption timestamps these observations to May 2023 because model versions may change.
- Large Language Models as Generative Agents: Unlike related generative-agent and multi-agent game settings, this work uses a clear objective for evaluating negotiation behavior.The objective is a measurable deal price in bargaining.
- Learning from AI Feedback: Unlike constitutional-AI-style approaches using scalar rewards, the paper provides AI feedback directly in natural language for strategic negotiation.The paper positions its contribution as confirming natural-language feedback's effectiveness in this setting.
4 Experiments
The experiments test whether LLMs can learn negotiation strategies from AI feedback across roles and multiple rounds. Results show substantial differences in rule-following, feedback incorporation, role-dependent improvement, and the tradeoff between deal price and deal success.
- 4.2 Basic Experiments: Only a subset of models can participate reliably: cohere-command fails to understand bargaining rules, while j2-jumbo-instruct does not improve from AI feedback.The remaining models used for multi-round games are gpt-3.5-turbo, claude-instant-v1.0, and claude-v1.3.
- 4.3 Behaviors of Different LLM Backend: For weaker agents, seller improvement is easier than buyer improvement, whereas claude-v1.3 and gpt-4 still improve as buyers after feedback.Average deal-price changes for weaker agents are -0.2 and -0.5 as buyers versus +1.0 and +1.7 as sellers; stronger-agent buyer changes are -1.2 and -3.0.
- 4.2 Basic Experiments: AI feedback induces improvements comparable to randomly selected human feedback, while requiring less human effort and scaling more easily.The comparison uses 500 runs and a predefined pool of ten human suggestions, with three suggestions sampled for each condition.
- 4.3 Behaviors of Different LLM Backend: Single-round gains do not necessarily continue: gpt-3.5-turbo improves across multiple rounds, while claude-instant-v1.0 improves at most once.This pattern appears when the models play both seller and buyer roles.
- 4.4 Towards Continuous Improvements from Iterative AI Feedback: Higher deal prices in multi-round games come with lower deal-success ratios, while gpt-4 achieves a better buyer-side price and deal-rate tradeoff than gpt-3.5-turbo.Claude-instant-v1.0 cannot improve over multiple buyer rounds, claude-v1.3 saturates after three rounds, and gpt-4 and gpt-3.5-turbo continue improving.
- 4.4 Towards Continuous Improvements from Iterative AI Feedback: Iterative feedback increases response length, but verbosity alone does not indicate better strategy: gpt-4 uses fewer words than claude-v1.3 while achieving better deal price and success rate.The authors interpret gpt-4’s increased verbosity as more strategic than claude-v1.3’s.
5 Conclusions
The study finds that some language models can improve through repeated game playing and iterative AI feedback, while better deal prices can increase deal-breaking risk. It also highlights implications for scalable improvement and the need for oversight and safety research.
- Certain models improve through continuous competition games with iterative AI feedback under well-defined rules.
- Better deal prices come with a higher risk of breaking the deal, motivating global optimization across multiple rounds.
- Iterative feedback can produce more effective negotiation language even when the stronger strategy uses fewer words.
- Autonomous improvement may reduce human intervention, but autonomous model behavior makes oversight more challenging and calls for alignment and safety research.