Source-linked AI summary
Decoupling Strategy and Generation in Negotiation Dialogues
He He, Derek Chen, Anusha Balakrishnan, Percy Liang
TL;DR
Negotiation agents must coordinate strategic choices with natural-language realization, but end-to-end models offer limited control and reinforcement learning can yield degenerate language. The paper decouples these functions with coarse dialogue acts, evaluates supervised, reinforcement-learning, and rule-based strategies, and reports stronger, more human-like negotiation with diverse utterances.
Problem
Negotiation agents must choose both a bargaining strategy and its natural-language realization, while end-to-end models are difficult to control and reinforcement learning can produce degenerate utterances.
Method
The paper decouples strategy and generation with coarse dialogue acts, using a manager for strategic acts and a generator conditioned on acts and dialogue history.
Results
The modular model maintains human-like behavior while optimizing objectives, and coarse-act models are stronger negotiators with more diverse utterances than word-based models.
Takeaways & Limitations
Coarse dialogue acts provide strategic control and allow learned policies to be combined with hand-coded rules or different negotiation goals.
Takeaways & Limitations
Because coarse dialogue acts omit full utterance semantics, the parser and generator must retain access to the full dialogue history.
Abstract
from arXiv · showhide
We consider negotiation settings in which two agents use natural language to bargain on goods. Agents need to decide on both high-level strategy (e.g., proposing \$50) and the execution of that strategy (e.g., generating "The bike is brand new. Selling for just \$50."). Recent work on negotiation trains neural models, but their end-to-end nature makes it hard to control their strategy, and reinforcement learning tends to lead to degenerate solutions. In this paper, we propose a modular approach based on coarse di- alogue acts (e.g., propose(price=50)) that decouples strategy and generation. We show that we can flexibly set the strategy using supervised learning, reinforcement learning, or domain-specific knowledge without degeneracy, while our retrieval-based generation can maintain context-awareness and produce diverse utterances. We test our approach on the recently proposed DEALORNODEAL game, and we also collect a richer dataset based on real items on Craigslist. Human evaluation shows that our systems achieve higher task success rate and more human-like negotiation behavior than previous approaches.
1 Introduction
The paper decouples negotiation strategy from language generation with coarse dialogue acts, addressing limited strategy control and reinforcement-learning degeneracy. It evaluates modular systems across negotiation datasets and learning settings, reporting more human-like behavior, stronger negotiation, and greater utterance diversity.
- End-to-end negotiation models are difficult to control and interpret, while reinforcement learning can produce ungrammatical or repetitive utterances.
- The proposed framework separates strategic decisions from language realization using coarse dialogue acts that encode an intent and its arguments.The manager predicts the next act, and the generator conditions on that act and dialogue history.
- Coarse dialogue acts allow different negotiation goals to share the same language generator and support combining learned policies with hand-coded rules.The acts provide strategic control without representing the full meaning of each utterance.
- The paper introduces CRAIGSLISTBARGAIN, a dataset of negotiations over real Craigslist goods collected through buyer-seller interactions on Amazon Mechanical Turk.The scenario is intended to support richer, open-ended negotiation behavior.
- The study evaluates word-based and coarse-act systems trained with supervised learning, reinforcement learning, or domain knowledge on CRAIGSLISTBARGAIN and DEALORNO DEAL.Human workers compare systems using task-specific scores and human-likeness.
- The modular model maintains reasonable human-like behavior while optimizing its objective, and coarse-act models are stronger negotiators with more diverse utterances than word-based models.The reported comparison covers both behavior and utterance diversity.
2 Craigslist Negotiation Dataset
CRAIGSLISTBARGAIN extends negotiation research from closed-domain games to realistic bargaining over Craigslist items. Its scenarios and human dialogues support longer, more diverse interactions involving persuasion and side offers.
- Prior human-human negotiation datasets use fixed objects in closed-domain games, restricting dialogue-domain and language richness.Examples include Settlers of Catan and division of books, hats, and balls.
- CRAIGSLISTBARGAIN assigns buyer and seller roles and asks agents to negotiate an item’s price using its description, photos, and listing price.
- Scenarios were created from Craigslist postings across six categories, with buyer target prices set to 0.5x, 0.7x, or 0.9x the listing price.
- 6682 human-human dialogues were collected on Amazon Mechanical Turk, producing longer dialogues and more diverse utterances than prior datasets.Workers could embellish items and negotiate side offers such as free delivery or pickup.
- The dataset elicits richer negotiation behavior, including embellishment, side offers, appeals to sympathy, information gathering, and persuasion.
3 Approach
The approach separates negotiation strategy from language generation through coarse dialogue acts, while retaining full dialogue history for parsing and response generation. Its modular parser, manager, and generator support multiple controllability regimes and reduce reinforcement-learning degeneracy.
- Overview: The framework decouples strategy from generation by extracting coarse dialogue acts, optimizing decisions in act space, and retrieving responses from full dialogue history.Coarse acts capture high-level strategic moves without representing an utterance’s full semantics.
- Overview: The parser maps each utterance to a coarse dialogue act, the manager predicts the next act, and the generator produces a natural-language response.The parser uses keyword, entity, regular-expression, and if-then rules; the manager conditions on past acts and the scenario.
- Overview: Both parser and generator retain full dialogue-history access because coarse acts cannot capture an utterance’s full meaning.The generator therefore uses context beyond the act, such as a preceding question when responding to an inform act.
- Overview: Restricting the manager to coarse dialogue acts reduces the risk of degeneracy during reinforcement learning.The paper contrasts this with end-to-end models, whose strategies are difficult to control and interpret and whose reinforcement learning can produce ungrammatical or repetitive utterances.
- Manager: The dialogue manager can be learned through supervised learning, reinforcement learning, or a hybrid policy combining learned behavior with hand-coded rules.These options provide increasing controllability over the agent’s strategic decisions.
CRAIGSLISTBARGAIN Rules
The framework represents negotiation strategy with coarse dialogue acts, optimizes policies over those acts, and generates context-aware language by retrieval.
- Dialogue-act representation: Coarse dialogue acts encode intents and arguments, with parser rules detecting entities and keywords such as prices and negotiation phrases.
- Dialogue-act representation: The dialogue manager models sequences of coarse dialogue-act tokens with attention and carries hidden states across the dialogue for history.
- Policy optimization: Reinforcement learning optimizes rewards over complete coarse dialogue-act sequences, including utility, fairness, and dialogue length.
- Policy optimization: Policy-gradient updates use sampled trajectories and final rewards, with a baseline for variance reduction.
- Policy control: A hybrid policy combines a learned intent manager with hand-coded rules that fill arguments such as prices.
- Language generation: Retrieval-based generation conditions on the selected act and dialogue history, ranking matching candidates by context similarity and sampling among top candidates for diversity.
4 Experiments
The experiments compare word-based and coarse-act systems across CRAIGSLISTBARGAIN and DEALORNODEAL using supervised, reinforcement, and rule-based strategies, with human evaluation of task performance and human-likeness.
- Tasks and systems: The study evaluates negotiation on CRAIGSLISTBARGAIN, which involves negotiating an item’s price, and DEALORNODEAL, which involves dividing items using private utilities.
- Tasks and systems: Experiments compare end-to-end word models with modular coarse-act models trained using supervised learning, reinforcement learning, or domain knowledge.
- Evaluation: Systems are evaluated with task-specific scores and human-likeness, the latter indicating deviation from human behavior associated with over-optimization.
- Evaluation: In human evaluation, workers chatted with bots or other workers in randomized pairings and rated whether partners showed reasonable human behavior on a 1–5 Likert scale.
- Results: SL(act) matched human behavior more closely in statistics across metrics, whereas SL(word) produced shorter, generic utterances and lower utility.
- Results: RL(word) became degenerate, with the three most frequent sentences comprising 81.6%, 100%, and 100% of utterances for RLutility(word), RLfairness(word), and RLlength(word).
- Results: RL(act) optimized different negotiation goals while maintaining reasonable human-likeness across both tasks.
- Results: Injecting domain knowledge into SL(act)+rule achieved high utility and human-likeness on both CRAIGSLISTBARGAIN and DEALORNODEAL.
5 Related Work and Discussion
CRAIGSLISTBARGAIN extends negotiation research toward richer, open-ended language while retaining a strategic backbone through coarse dialogue acts. The paper argues for hybrid systems that combine strategic control with open-ended generation.
- CRAIGSLISTBARGAIN uses richer and more diverse language than prior collaborative or competitive negotiation datasets.
- Latent discrete variables can increase utterance diversity but do not provide controllability over dialogue strategy.
- Unlike domain-specific negotiation policies, the model operates in an open-ended natural-language space.
- The modular framework models a rough strategic backbone while supporting open-ended generation.
- The authors identify hybrid approaches for open-ended, goal-oriented settings as a direction for further research.
A CRAIGSLISTBARGAIN Web Interface
The CRAIGSLISTBARGAIN web interface provides the setting in which workers negotiate.
- The web interface is used by workers to conduct negotiations.
B Argument Detection of the Rule-based Parser
The rule-based parser detects prices, items, counts, and agent references to recover structured negotiation arguments from utterances. Its procedures differ between CRAIGSLISTBARGAIN and DEALORNO DEAL.
- On CRAIGSLISTBARGAIN, prices are detected from dollar-sign patterns or contextual word neighborhoods learned from training data.
- A test-time number qualifies as a price when contextual conditions hold and it is not larger than 1.5x the training threshold.
- On DEALOR NODEAL, regular expressions detect pronouns, balls, hats, books, and counts from 1 to 10.
- Object-count pairs are grouped with the most recently referred agent through left-to-right processing and pronoun resolution.
C Example Dialogues
The paper provides human-bot negotiation examples from DEALOR NODEAL and identifies differences between two strategy models in their conversational behavior.
- The evaluation dialogues are available through the Codalab worksheet.
- The CRAIGSLISTBARGAIN chat interface shows how workers negotiate through the paper’s web-based setting.
- Table 9 presents example human-bot dialogues from DEALOR NODEAL, with bot utterances shown in bold.
- SL(word) is quick to concede, whereas SL(act) is generally harder to persuade.