PaperlayerView exact revision on arXiv

Source-linked AI summary

GameTalk: Training LLMs for Strategic Conversation

Victor Conchello Vendrell, Max Ruiz Luyten, Mihaela van der Schaar

arXiv:2601.16276v1cs.CLcs.AIcs.GTcs.LGcs.MA

TL;DR

LLMs are commonly optimized for isolated decisions despite real-world strategic interactions requiring long-term objectives through dialogue. GameTalk adapts fine-tuning methods and whole-conversation reward signals to train strategic multi-turn behavior, outperforming baselines with targeted reward shaping and DPO strongest. The framework remains limited to two-player games against fixed LLM opponents and lacks validation with humans or more complex multi-agent dynamics.

  • Problem

    LLM training often uses single-turn, static objectives, leaving long-term strategic coordination and negotiation through dialogue comparatively unexplored.

  • Method

    GameTalk adapts GRPO, DPO, and STaR with global conversation-level rewards and behavioral signals for training strategic multi-turn interactions.

  • Results

    Across representative games, GameTalk significantly outperforms baseline models, with targeted reward shaping improving results and DPO emerging as most effective, particularly for persuasion.

  • Takeaways & Limitations

    Conversational fine-tuning can train LLMs to use language for negotiation and long-term planning in game-theoretic interactions.

  • Takeaways & Limitations

    Experiments use two-player games against fixed LLM opponents, without validation against human players or more complex multi-agent dynamics.

Abstract

from arXiv · show

Strategic decision-making in multi-agent settings is a key challenge for large language models (LLMs), particularly when coordination and negotiation must unfold over extended conversations. While recent work has explored the use of LLMs in isolated decision tasks, little attention has been given to optimizing long-term objectives through dialogue. We introduce \textbf{GameTalk}, a framework for training LLMs to make strategic decisions via multi-turn interactions. Unlike prior work that focuses on single-turn objectives or static action prediction, we train LLMs to optimize a global objective across full conversations. We achieve this by adapting fine-tuning methods like GRPO, DPO, and STaR to incorporate reward signals that depend on the entire interaction. We evaluate this approach on a suite of increasingly complex games, designed to stress different aspects of reasoning, coordination, and opponent modeling. Our results show that GameTalk significantly outperforms untrained models, especially under reward shaping, with DPO consistently yielding the strongest gains. These findings position conversational fine-tuning as a promising path for LLMs to reason, negotiate, and act in interactive environments.

1 INTRODUCTION

GameTalk addresses the mismatch between single-turn LLM optimization and strategic interactions that require sustained, goal-driven dialogue. It adapts conversational fine-tuning and behavioral reward signals to improve strategic performance across representative games.

  • The work targets applications where negotiation and collaborative decision-making depend on sustained interaction rather than isolated response quality.
  • The framework adapts GRPO, DPO, and STaR to train LLMs to integrate reasoning, coordination, and opponent modeling across conversations.
  • Three behavioral signals—ISE, SRP, and LO—dissect strategic performance and support targeted reward shaping for full-interaction outcomes.ISE evaluates opponent-strategy prediction, SRP evaluates actions relative to beliefs, and LO evaluates influence over opponent behavior.
  • Standard fine-tuning methods struggle with sparse final-outcome rewards, whereas tailored reward shaping significantly improves performance over untrained baselines.
  • GameTalk introduces an RL-based framework for fine-tuning LLMs for strategic multi-turn conversation.
  • Experiments on Rock-Paper-Scissors, Bertrand Competition, and Size-Prize Bargaining Game show consistent improvements over baseline models.The games are chosen to examine different capabilities in strategic dialogue.

2 RELATED WORK

Prior research uses multi-agent games to evaluate negotiation, cooperation, and decision-making, but finds that LLMs often struggle with subjective and contextually strategic reasoning. GameTalk builds on behavioral game theory’s effort to model behavior beyond perfect rationality.

  • Studies of LLMs in multi-agent games evaluate negotiation, cooperation, and decision-making while revealing limitations in strategic behavior.
  • Existing work finds that LLMs can imitate strategic behavior but often struggle with subjective reasoning and contextually strategic responses.
  • Standard game theory’s perfect-rationality assumption can poorly account for how players actually reason and interact in applied settings.
  • Behavioral Game Theory addresses these shortcomings by modeling human non-optimalities as learning processes or modifying utilities to represent social preferences.

3 METHOD

GameTalk trains and analyzes LLMs in multi-turn games by optimizing episode-level utility and measuring strategic capabilities throughout interaction. It adapts several fine-tuning algorithms using conversational rollouts and global rewards.

  • 3.1 GameTalk: GameTalk structures games as setting prompts, private reasoning before each action, multi-turn communication, and final game actions.Agents use private Chain-of-Thought before sending messages or executing game actions; the conversation then ends after the game outcome.
  • 3.1 GameTalk: Training uses one LLM against a fixed opponent and a single utility reward computed from both players’ joint actions at episode end.The same global reward optimizes generated content, including reasoning and dialogue.
  • 3.2 Behavioral Signals for Strategic Analysis: GameTalk evaluates strategic behavior through ISE, SRP, and LO, measuring opponent prediction, belief-relative action effectiveness, and influence over opponent behavior.Higher values indicate better strategic capacities, and the signals can be computed during interaction using prompted action-probability estimates.
  • 3.2 Behavioral Signals for Strategic Analysis: The behavioral signals are theoretically linked to true expected utility, with high utility requiring favorable opponent behavior, accurate understanding, and effective action.The theorem states utility bounds when the metrics use the exact opponent policy and agent belief.
  • 3.3 Adaptation of RL alogrithms: GRPO and DPO branch a root conversation into parallel rollouts, using each branch’s final reward to update the response generated at the branching point.The described GRPO adaptation forks one history into k branches, while DPO extends preference learning to grouped completions through pairs, permutations, or rankings with ties.
  • 3.3 Adaptation of RL alogrithms: STaR learns from a filtered dataset, unlike the comparative GRPO and DPO adaptations.The broader framework adapts GRPO, DPO, and STaR to multi-turn strategic interactions.

4 EXPERIMENTS

GameTalk evaluates reward shaping and adapted training algorithms for strategic dialogue across three games of increasing complexity. The experiments show that auxiliary rewards and algorithm choice materially affect performance, with DPO strongest overall and learned strategies adapting to game demands.

  • Reward Shaping Analysis: In constrained Rock-Paper-Scissors, the Base agent outperforms the Untrained baseline, confirming that the reinforcement-learning framework works in a multi-turn setting.The setup prohibits the trained agent from playing Paper to encourage strategic dialogue rather than a non-conversational strategy.
  • Reward Shaping Analysis: ISE reward improves opponent modeling but harms game performance, whereas LO reward produces the highest win rate by teaching the agent to influence its opponent.Adding a naturalness bonus preserves the high win rate while improving dialogue quality, although opponent-modeling scores remain lower.
  • Experimental Setup: The evaluation spans Rock-Paper-Scissors, Bertrand Competition, and Size-Price Bargaining Game, testing opponent modeling, coordination, and negotiation.The games were selected to analyze distinct strategic capabilities and increase in complexity.
  • Main Results: All trained agents substantially outperform the Untrained baseline and a non-conversational Nash-equilibrium agent across the evaluated games.The Nash-equilibrium agent yields an NRA of 0 and reward 1 in Rock-Paper-Scissors, with near-zero rewards in Bertrand Competition.
  • Main Results: DPO consistently performs best in Bertrand Competition and Size-Price Bargaining, GRPO performs best in Rock-Paper-Scissors, and STaR underperforms in complex settings.The authors attribute DPO’s advantage over GRPO to its richer relational learning signal, while STaR may overfit to an initially successful pattern.
  • Main Results: Qualitative analysis shows that learned strategies adapt to game demands, including direct move-hinting, deceptive pricing dialogue, and more favorable bargaining outcomes.DPO agents are described as the most effective negotiators in the bargaining game.

5 DISCUSSION

GameTalk trains LLMs for strategic reasoning and action across multi-turn dialogues, outperforming baselines with further gains from targeted reward shaping. The main limitation is a disconnect between persuasive performance and accurate opponent modeling, alongside evaluation restricted to fixed-opponent two-player games.

  • 5 DISCUSSION: GameTalk adapts GRPO, DPO, and STaR to optimize policies over entire conversations in game-theoretic settings.The framework treats language as a tool for negotiation and long-term planning.
  • 5 DISCUSSION: GameTalk significantly outperforms baseline models, with performance further boosted by targeted reward shaping based on behavioral signals.DPO is reported as the most effective algorithm, particularly for persuasion tasks.
  • 5 DISCUSSION: Learned conversational strategies are critical for influencing opponent behavior and achieving better outcomes.
  • 5 DISCUSSION: GameTalk addresses the gap between static, single-turn LLM training and dynamic, goal-oriented real-world interactions.
  • 5 DISCUSSION: Agents can learn effective persuasive policies without necessarily forming accurate internal models of their opponents.The authors identify this performance–opponent-modeling disconnect as the most significant limitation.
  • 5 DISCUSSION: Experiments are limited to two-player games against a fixed LLM opponent, without validation against human players or more complex multi-agent dynamics.Future work proposes more explicit rewards for reasoning about others’ intentions and beliefs.

Checklist

The checklist records affirmative coverage of assumptions, theoretical details, empirical reporting, reproducibility materials, asset information, and human-subjects documentation where applicable.

  • Checklist: The submission marks mathematical-set, algorithm-complexity, and model explanations as included.
  • Checklist: The submission marks assumptions and complete proofs as included for theoretical results.
  • Checklist: The submission marks measure definitions and error-bar reporting as addressed, with error bars listed as not applicable.
  • Checklist: The submission states that code, data, source-code URLs, training details, and computing infrastructure information will be provided.
  • Checklist: The checklist includes a comment identifying an unrelated dataset and code repository alongside cited prior work.
  • Checklist: Asset creators are cited, while license information is marked not applicable.
  • Checklist: Consent, sensible-content discussion, and human-participant documentation are marked not applicable.
  • Checklist: The checklist states that participant risks, institutional review approval, and compensation details are not applicable.

A HYPERPARAMETERS AND COMPUTE DETAILS

The experiments use constrained compute and a compact Llama-3 setup, while the formal strategy framework represents beliefs, actions, and messages as components of deficiency-aware decision-making.

  • A HYPERPARAMETERS AND COMPUTE DETAILS: Experiments use one Nvidia RTX 6000 ADA GPU with 48 GB of VRAM, and each run takes 1.5–3.5 days.
  • A HYPERPARAMETERS AND COMPUTE DETAILS: The experiments use 3-billion-parameter Llama-3 for both the trained agent and its conversation counterpart, with LoRA for efficient fine-tuning.
  • A HYPERPARAMETERS AND COMPUTE DETAILS: Memory issues in Bertrand Competition are addressed by computing and backpropagating each generation’s loss separately before the gradient update.
  • A HYPERPARAMETERS AND COMPUTE DETAILS: Interactions are capped at 5 per game because of computational constraints, with the cap reached most often in Bertrand Competition.
  • A HYPERPARAMETERS AND COMPUTE DETAILS: The trained model consistently plays second, and serves as the buyer in bargaining because the untrained seller initially had greater bargaining power.
  • B MODELING OTHER PLAYERS: A player’s strategy is represented by message and action policies conditioned on conversation history and private information.
  • B.1 Deficiency-Aware strategy: A deficiency-aware strategy maintains an internal state and uses it to select actions and messages.Its components include a prior, state updates, and message and action policies.
  • B.1 Deficiency-Aware strategy: The framework separates belief modeling, game policy, and conversational strategy, including dialogue that can improve beliefs and influence opponents.

B.2 Formal Derivation of Utility Bounds

The appendix derives utility bounds from ISE, SRP, and LO, then specifies prompts and rules for the evaluated games. Together, these materials connect behavioral signals to utility while documenting the experimental interaction settings.

  • B.2 Formal Derivation of Utility Bounds: The formal derivation shows that ISE, SRP, and LO collectively form a sandwich bound around true expected utility.
  • B.2 Formal Derivation of Utility Bounds: The theorem assumes ISE, SRP, and LO are computed using the exact opponent policy and the agent’s belief policy, with C denoting the utility range.
  • B.2 Formal Derivation of Utility Bounds: The lower utility bound relates true utility to belief-based utility through total variation distance, Pinsker’s inequality, and ISE.
  • B.2 Formal Derivation of Utility Bounds: SRP substitutes belief-based utility into the lower-bound inequality, while LO supplies the upper bound as maximum achievable utility against the true opponent policy.
  • C SETTING PROMPTS: Setting prompts define placeholders such as max_interact, my_name, and other_name, with max_interact fixed at 5.
  • Initial prompt: Rock-Paper-Scissors prompts specify the game rules, conversation-before-action protocol, and an experiment where the trained agent cannot play paper.
  • Initial prompt: Mid-conversation prompts require the responding player to format its reasoning and move using think and play tags after the opponent acts.
  • C.3 Size-Prize Bargaining Game: Bertrand Competition prompts define price competition, sales, and earnings, while Size-Price Bargaining prompts define unit-price negotiation and the buyer’s expected-benefit objective.

D ADDITIONAL TRAINING ABLATIONS

Exploration incentives produced a trade-off between reward metrics and strategic conversational capabilities. Lowering the KL penalty increased rewards but weakened opponent modeling, while entropy regularization preserved conversational metrics without improving rewards.

  • Exploration incentives: Lower β sharply reduced ISE and SRP, indicating weaker ability to model or exploit the opponent despite higher rewards.The resulting trade-off conflicts with the goal of developing strategic conversational abilities.
  • Exploration incentives: Adding an entropy term preserved ISE, SRP, and LO near base-model levels but failed to improve rewards and sometimes lowered R and NRA.The entropy coefficient was fixed at γ = 10^-1.
  • Behavioral analysis: Models trained with smaller KL coefficients converged toward playing scissors, largely ignoring dialogue and reasoning channels.The β = 10^-3 model always chose scissors, while β = 10^-2 preferred scissors more than the base model.
  • Behavioral analysis: Entropy regularization produced longer, more varied conversations without strategic depth, and avoided the default tendency to play scissors.The model no longer defaulted to scissors under entropy regularization.

D.2 Model structure

The model-structure ablations examined alternative architectures and LoRA ranks. Memory and generation constraints prevented practical use of the tested alternative models, while rank 32 was retained because rank 8 underperformed and rank 64 offered no consistent gain.

  • Alternative architectures: The tested Llama-3 8B and Qwen-3 4B models encountered significant memory or training-cost constraints.Llama-3 left insufficient memory for generation and training, while Qwen-3 produced excessively long responses that made training prohibitively expensive.
  • LoRA rank: LoRA rank 8 performed substantially worse across most metrics than higher ranks.The comparison evaluated ranks 8, 32, and 64.
  • LoRA rank: Increasing LoRA rank from 32 to 64 produced no consistent improvement, with notable differences only in ISE and SRP and in opposite directions.The metrics were otherwise very similar.
  • LoRA rank: The subsequent experiments used LoRA rank 32 based on the rank comparison.

E GAMES ANALYSIS

GameTalk evaluates strategic conversation across three games chosen to test different capabilities and interaction structures. The environments include a simultaneous zero-sum game, repeated price competition, and quantity-price bargaining with asymmetric negotiation power.

  • Game suite: The three environments are Rock-Paper-Scissors, Bertrand Competition, and Size-Price Bargaining Game.They were selected to analyze different capabilities and increasing interaction complexity.
  • Rock-Paper-Scissors: Rock-Paper-Scissors tests situation understanding and anticipation in a simultaneous three-action zero-sum game.Players receive 2 for a win, 1 for a tie, and 0 for a loss.
  • Rock-Paper-Scissors: Conversation can help a player identify and exploit deviations from the mixed Nash equilibrium in Rock-Paper-Scissors.For example, playing Paper more often is a best response when the opponent chooses Rock with probability above 1/3.
  • Bertrand Competition: Bertrand Competition models repeated price-setting by two firms selling a homogeneous good, with the lower-priced firm capturing the market.Performance is measured by normalized earnings, defined as total profit divided by maximum possible earnings and bounded between 0 and 1.
  • Bertrand Competition: Bertrand game instances vary cost c, demand d, maximum price pmax, and product name, with 200 scenarios generated and manually reviewed.Actions are restricted to natural-number prices.
  • Size-Price Bargaining: Size-Price Bargaining requires agreement on both purchase quantity and unit price between a seller and buyer.Actions are proposals represented by quantity and price pairs or acceptance of the opponent’s last offer.
  • Size-Price Bargaining: Because bargaining powers are asymmetric, negotiation skills can shift outcomes, measured through the agent’s bargaining power α1.Values closer to 1 indicate stronger negotiation capacity, while 0.5 represents balanced bargaining.

F EVALUATION OF DPO VARIANTS

The appendix compares DPO-pairs and DPO-permutations across the three game environments and provides example trained conversations. Both variants perform comparably, while the examples illustrate terse or strategically explicit dialogue behavior.

  • DPO variants: Both DPO variants exhibit comparable performance across all three games.The Ranking with ties variant was not tested because continuous reward shaping makes ties in the final reward signal highly unlikely.
  • DPO variants: The authors suggest Ranking with ties may be useful when rewards are discrete or ties are common.
  • DPO variants: DPO-pairs and DPO-permutations are evaluated against the Untrained baseline across Rock-Paper-Scissors, Bertrand Competition, and Size-Price Bargaining.
  • Example conversations: An example trained only with LO-reward contains unnatural, terse responses from the trained second player.
  • Example conversations: The example dialogue explicitly uses deception and counterplay, including claiming Rock before selecting Paper against an opponent’s Rock.

G.2 Final reward shaped agend

The final reward-shaped agents use multi-turn dialogue to infer opponents’ strategies, manipulate their beliefs, and select actions that improve their position in Rock-Paper-Scissors and Bertrand Competition.

  • Rock-Paper-Scissors: In Rock-Paper-Scissors, the trained agent identifies an opponent’s preference for paper, feints rock, and then plays scissors to win.The example is described as more natural than the preceding conversation.
  • Rock-Paper-Scissors: The agents update their move choices from the opponent’s prior actions, using the game rules to counter paper with scissors.The dialogue shows repeated strategic reevaluation after each move.
  • Rock-Paper-Scissors: The Rock-Paper-Scissors dialogue combines probability, game theory, hidden patterns, and psychological probing to avoid predictable mirroring.The agents explicitly mix mirrored and unmirrored moves while trying to expose the opponent’s reasoning.
  • Bertrand Competition: In Bertrand Competition, the agent seeks the opponent’s minimum acceptable price before choosing a price that maximizes its own earnings and reduces the opponent’s earnings.The strategy uses the sales relation (300 - lowest_price) / 0.2 to connect revealed price information with possible sales.
  • Bertrand Competition: The Bertrand dialogue uses a $135 offer above the opponent’s proposed $130 price to elicit information while preserving room to adjust the strategy.The agents discuss whether $135 is optimal and consider the trade-off between lower prices, competitiveness, and sales.
Loading 2601.16276v1…