PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method

Tian Xia, Zhiwei He, Tong Ren, Yibo Miao, Zhuosheng Zhang, Yang Yang, Rui Wang

arXiv:2402.15813v3cs.CLcs.GT

TL;DR

The paper asks how to evaluate and improve LLM agents’ bargaining as buyers and sellers. It formalizes bargaining, builds a real-product benchmark, and introduces OG-Narrator, which combines deterministic offer generation with LLM narration. The study finds that buying is harder than selling and that OG-Narrator substantially improves buyer bargaining performance.

  • Problem

    The paper addresses the open problem of evaluating whether LLM agents can bargain effectively as buyers or sellers.

    Source: §1 · pp. 1–2

  • Method

    The paper formalizes bargaining with buyer and seller evaluation metrics, collects AmazonHistoryPrice, and proposes OG-Narrator for buyer enhancement.

    Source: §5 · p. 7, §8 · p. 9

  • Results

    Playing the Buyer is more difficult than playing the Seller, while larger models improve Seller performance but not Buyer performance.

    Source: §5 · pp. 6–7, §8 · p. 9

  • Takeaways & Limitations

    A deterministic Offer Generator paired with an LLM Narrator can improve buyer bargaining ability by controlling offers while preserving natural-language interaction.

    Source: §5 · p. 7

    The data were collected on November 18, 2023, use English and USD, and may bias agents as prices change over time.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Bargaining is an important and unique part of negotiation between humans. As LLM-driven agents learn to negotiate and act like real humans, how to evaluate agents' bargaining abilities remains an open problem. For the first time, we formally described the Bargaining task as an asymmetric incomplete information game, defining the gains of the Buyer and Seller in multiple bargaining processes. It allows us to quantitatively assess an agent's performance in the Bargain task. We collected a real product price dataset, AmazonHistoryPrice, and conducted evaluations of various LLM agents' bargaining abilities. We find that playing a Buyer is much harder than a Seller, and increasing model size can not effectively improve the Buyer's performance. To address the challenge, we propose a novel approach called OG-Narrator that integrates a deterministic Offer Generator to control the price range of Buyer's offers, and an LLM Narrator to create natural language sentences for generated offers. Experimental results show that OG-Narrator improves the buyer's deal rates from 26.67% to 88.88% and brings a ten times multiplication of profits on all baselines, even a model that has not been aligned.

1 Introduction

The paper addresses how to evaluate whether LLM agents can bargain effectively as buyers or sellers. It formalizes the task, introduces a real-product benchmark, and proposes OG-Narrator to improve buyer performance.

1 Introduction

  • The bargaining setting hides each agent’s budget or cost and private thought while transmitting only Talk and Action between buyer and seller.Source: Figure 1 · p. 1
  • Unsuccessful or unreasonable bargaining can cause user losses and unpredictable behavior in autonomous-agent environments.Source: §1 · p. 1
  • Existing bargaining studies used small, domain-limited datasets without clearly defining the bargaining problem or comparing buyer and seller performance.Source: §1 · p. 2
  • 930 real products across 18 categories support a benchmark for quantitatively evaluating LLM bargaining abilities as buyers and sellers.The benchmark includes GPT-4, ChatGPT, Llama-2, Yi, and Mistral-7B.Source: §1 · p. 2
  • OG-Narrator combines a deterministic Offer Generator with an LLM Narrator to dramatically improve buyer bargaining ability.Source: §1 · p. 2

2 Dataset

AmazonHistoryPrice fills the need for a larger real-product bargaining dataset by covering popular products across many categories, prices, and historical records. It also provides textual, visual, and price-history information for each item.

2 Dataset

  • Earlier datasets were smaller or artificially designed, including a single balloon product, two artificial categories, or a 161-item Craigslist test set.Source: §2 · p. 2
  • AmazonHistoryPrice contains 930 popular products across 18 categories with real prices collected from camelcamelcamel.Source: §2 · p. 2
  • Product records include names, descriptions, features, lowest and highest prices, current and list prices, and image links.Source: §2 · p. 2
  • The dataset’s imbalanced category distribution mirrors the real-world distribution of popular online-shopping items.Source: §2 · p. 2
  • Prices range from 0 to 4500 USD, and some product histories extend back to 2009.Source: §2 · p. 2
  • Descriptions, feature introductions, and pictures provide agents with additional textual and visual product context.Source: §2 · pp. 2–3

3 A Benchmark for Bargaining Task

The paper formalizes bargaining between a Buyer and Seller as an asymmetric incomplete-information task, with rational agents choosing actions under private budgets and costs. It introduces normalized profit and share metrics to compare bargaining outcomes across mutual- and conflicting-interest sessions.

3.1 Task Definition

  • The benchmark models bargaining as alternating Buyer–Seller actions under private budgets and costs, ending in agreement or quitting.Agents pursue higher payoffs and avoid negative profits while inferring the counterpart’s private information from dialogue.Source: §3.1 · p. 3
  • Mutual-interest sessions permit agreements, whereas conflicting-interest sessions can make agreement impossible when the Buyer’s budget does not exceed the Seller’s cost.The feasible set consists of deal prices yielding positive profit for both rational parties.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • Normalized Profits assign zero to no-deal sessions and remain comparable across mutual- and conflicting-interest scenarios, including the B = C boundary adjustment.The benchmark aggregates these values with Sum of Normalized Profits across all sessions or scenario subsets.Source: §3.1 · p. 4
  • Valid sessions must terminate in the required output format, and their total normalized profit equals valid MI deals minus valid CI deals.This relationship follows from the benchmark’s normalized-profit construction.Source: §3.1 · p. 4
  • Share measures how the total available profit is divided between Buyer and Seller, with Buyer Share above 50% indicating greater Buyer profit.Share is undefined when combined normalized profit is nonpositive.Source: §3.1 · p. 4

4 Experiments

The experiments evaluate multiple LLMs in a controlled 930-product bargaining benchmark using fixed implementation and initialization choices. Results show that Buyers generally lose profit and underperform Sellers, while GPT-4 is strongest as a Seller and Buyer difficulty is not reliably reduced by model scaling.

4.1 Implementation Details

  • The benchmark uses one single product per session, sets Cost to each product’s historical lowest observed price, sets List Price to the historical highest price, and uses f = 0.8 to keep CI sessions at 5%.All temperatures are set to 0, though OpenAI’s closed-source non-deterministic implementation may cause small reproduction changes.Source: §4.1 · p. 4, §4.2 · p. 5

4.3 Benchmark Results

  • Almost all tested Buyers have negative SNP, indicating average negative profit whenever they close a deal and failure to consistently obey the positive-profit rule.GPT-4 has the least loss, ChatGPT the most valid sessions, and Mistral-7B and Mixtral-8x7B lead among open-source models on different measures.Source: §4.3 · p. 5
  • Table 2 reports various LLMs’ performances as Buyer and Seller in the Bargaining task, sorted by SNP in ALL, with SNP used for cross-model comparisons.Source: Table 2 · p. 6
  • GPT-4 is the strongest Seller, with the highest SNP in both MI and CI sessions and a 100% valid rate across all 930 sessions.ChatGPT, GPT-4, and Mixtral-8x7B-Instruct obtain Seller Shares above 100%, indicating excess profits against their Buyers.Source: §4.3 · p. 6, Table 2 · p. 6

5 Discussion

The benchmark finds that bargaining as a Buyer is harder than as a Seller, while model size alone does not reliably improve Buyer performance. Buyers often start too close to budget, limiting profit, whereas OG-Narrator improves Buyer outcomes and exposes Seller vulnerability.

Buyer and Seller performance

  • Playing Buyer is more difficult than playing Seller, even when agents have relatively equal abilities.ChatGPT as Seller beats all Buyers, but as Buyer loses to GPT-4 and Mixtral-8x7B-Instruct.Source: §5 · p. 6
  • Buyer difficulties include understanding the budget constraint and persuading conservative Sellers who often reject offers.These requirements make Buyer bargaining dependent on both valid price decisions and active persuasion.Source: §5 · pp. 6–7
  • Mixtral-8x7B’s normalized-profit distribution across all sessions has an average slightly below zero.The -0.5 and 0.5 thresholds separate Buyers who gain more than Sellers from those who do not, in CI and MI.Source: Figure 5 · p. 7
  • Different-sized models in the same family show similar Buyer performance, indicating that training and alignment matter more than model size alone.This pattern is reported for Llama-2, Yi, and Mistral models with nearly identical decoder architectures.Source: §5 · p. 7
  • Most Buyers start with offers only slightly below budget, sharply limiting their potential profit.One Buyer offered $30 against a budget just $2 higher, leaving only $2 of profit room.Source: §5 · p. 7

Buyer enhancement

  • OG-Narrator decreases the First Bid Ratio and improves Buyer performance across model types.The method is motivated by the finding that Human Buyers use lower first-bid ratios and achieve higher SNP and Share than LLM Buyers.Source: §5 · p. 7, Table 4 · p. 8

Buyer and Seller performance

  • Seller model size generally improves valid and deal rates, but Mixtral-8x7B and Baichuan2-13B-Chat are exceptions.Mixtral’s extra expert parameters increase Seller Share without improving valid rate or SNP, while Baichuan2-13B-Chat performs worse than its 7B counterpart.Source: §5 · pp. 7–8

6 A Simple Bargaining Method: OG-Narrator

OG-Narrator separates Buyer offer generation from language generation: a deterministic component selects prices while an LLM narrates them. Across models, this controlled strategy improves bargaining metrics, including for an unaligned model, and makes ChatGPT Seller performance vulnerable.

6 A Simple Bargaining Method: OG-Narrator

  • OG-Narrator combines a deterministic Offer Generator with an LLM Narrator to improve Buyer bargaining.The method is designed to reduce the price of the first bid while preserving natural-language interaction.Source: §6 · p. 8, §6.1 · p. 8

6.1 Method

  • The Offer Generator produces escalating factors from 0.5 to 1, applies them to the Buyer’s budget, and selects offer prices each turn.The Buyer proposes another generated price when the Seller’s offer remains above the generated threshold; otherwise, it deals.Source: §6.1 · p. 8
  • The LLM Narrator converts generated Buyer actions into natural-language Talk based on the action and session history.This decouples offer generation from language generation and lets the LLM focus on dialogue.Source: §6.1 · p. 8, Figure 6 · p. 8

6.2 Experiments

  • The experiments compare OG-Narrator with the original Buyer benchmark on selected models, including unaligned Phi-2.Phi-2 is a 2.7B coding model that was not fine-tuned with reinforcement learning from human feedback.Source: §6.2 · pp. 8–9

6.3 Results

  • OG-Narrator controls each Buyer’s FBR to the target 0.5 and significantly improves every tested model, including the unaligned model.It improves valid rate, deal rate, SNP, and Share across models, with especially large SNP gains in CI sessions.Source: §6.3 · p. 9, Figure 7 · p. 9
  • For Phi-2, OG-Narrator increases valid rate 19 times and deal rate 117 times, making the unaligned agent substantially more reliable as a Buyer.Phi-2 otherwise has very low valid and deal rates in the original benchmark.Source: §6.3 · p. 9
  • OG-Narrator enables all tested models to exceed 100% Share against ChatGPT Seller, unlike the negative Shares in the original benchmark.The result indicates that ChatGPT Seller is vulnerable when facing Buyers enhanced by the simple offer-generation strategy.Source: §6.3 · p. 9

7 Related Work

Related work covers autonomous LLM agents and prior bargaining datasets and evaluations. Earlier bargaining studies used limited settings, motivating broader task formulation and multi-session assessment.

AI Agents

  • Research on autonomous agents emphasizes LLM memory, planning, reasoning, and communication for independent social interactions.Generative agents illustrate this direction by assigning autonomous roles within a simulated town.Source: §7 · p. 9

Bargaining

  • Earlier bargaining work used a small second-hand-item dialogue dataset with weak price expectations for both Buyers and Sellers.The cited dataset was limited in scale and realism for evaluating bargaining behavior.Source: §7 · p. 9
  • Another recent study evaluated Buyer and Seller performance using only one item’s final deal price and could not assess multiple bargaining sessions.This limitation distinguishes prior evaluation from the paper’s multi-session focus.Source: §7 · p. 9

8 Conclusion

The paper formalizes bargaining evaluation, finds buyer performance is harder to improve, and proposes OG-Narrator to address this challenge. OG-Narrator enhances buyer performance and exposes vulnerability in the ChatGPT Seller.

  • The paper formally defines bargaining evaluation metrics for quantitatively assessing Buyer and Seller performance.Source: §8 · p. 9
  • Buyer performance is harder to improve than Seller performance, motivating buyer-focused enhancement.Source: §8 · p. 9
  • OG-Narrator makes the ChatGPT Seller vulnerable when facing an enhanced Buyer.Source: §8 · p. 9

Limitations

The paper identifies temporal, linguistic, currency, interpretability, flexibility, and strategy-coverage limitations that constrain how its findings should be applied.

  • Data collected on November 18th, 2023 may differ from later actual prices, while English-only USD data may bias agents.Source: Limitations · p. 10
  • Model implementation differences make it difficult to determine which model or training aspects influence bargaining ability.Source: Limitations · p. 10
  • OG-Narrator uses a simple linear function for offer-price factors, limiting flexibility compared with letting the LLM generate them.Source: Limitations · p. 10
  • The study focuses first on Buyers, leaving potential Seller strategies underexplored because of page limits and differences from real-life psychological bargaining.Source: Limitations · p. 10

Ethics Statement

The paper reports research-use data collection and model usage, ethical checks on products, and supporting materials for the bargaining task and prior research context.

Ethics Statement

  • Product data came from camelcamelcamel and Amazon image links, collected for research purposes only.Source: Ethics Statement · p. 10
  • The authors manually checked products for information identifying individuals or offensive content.Source: Ethics Statement · p. 10
  • Open-source LLMs were used for research under their respective model licenses.Source: Ethics Statement · p. 10
  • A graduate student participated as a Human Buyer in a preliminary study using the same prompts as the LLMs.Source: Ethics Statement · p. 10

A Bargaining Research

  • The paper situates its task against prior bargaining datasets and methods, including CraigslistBargaining, balloon bargaining, allocation negotiation, and NegoCoach.Source: A Bargaining Research · p. 12

B Concept Definitions

E All LLMs in our benchmark

The benchmark uses instruction-tuned or RLHF-aligned LLMs, compares model and human bargaining behavior, and documents budget settings, dialogue examples, actions, and prompting materials.

E All LLMs in our benchmark

I Prompts

F Different Budget Factors

G Human Buyer Performance Compared with ChatGPT and GPT4

System Prompt

Loading 2402.15813v3…