PaperlayerView exact revision on arXiv

Source-linked AI summary

GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfare

Siqi Zhu, David Zhang, Pedro Cisneros-Velarde, Jiaxuan You

arXiv:2510.08872v3cs.AIcs.GTcs.HCcs.LGcs.MA

TL;DR

Conversation systems can lack a principled way to evaluate how response strategies affect outcomes for both users and LLMs. GTAlign integrates game-theoretic reasoning into responses and social-welfare training; across diverse tasks, it reports gains in reasoning efficiency, answer quality, and social welfare.

  • Problem

    Conversation systems lack a principled mechanism to evaluate how different responses affect conversational outcomes and weigh alternative strategies.

    Source: §1 · p. 1

  • Method

    GTAlign models user–LLM interaction as a strategic game, constructs payoff matrices during reasoning, and trains with a cooperative social-welfare reward.

    Source: §1 · p. 2

  • Results

    Across diverse tasks, GTAlign improves reasoning efficiency, answer quality, and social welfare; reasoning efficiency rises 21.5% on four in-distribution datasets versus baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The experiments report improved explainability and controllability of reasoning alongside gains in reasoning efficiency, answer quality, and social welfare.

    Source: §8 · p. 10

    Inference-time steering assumes pricing policy can be reliably detected, models only limited factors such as token cost, and abrupt payoff-weight changes may yield unstable responses.

    Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 31

Abstract

from arXiv · show

Large Language Models (LLMs) have achieved remarkable progress in reasoning, yet sometimes produce responses that are suboptimal for users in tasks such as writing, information seeking, or providing practical guidance. Conventional alignment practices typically assume that maximizing model reward also maximizes user welfare, but this assumption frequently fails in practice: models may over-clarify or generate overly verbose reasoning when users prefer concise answers. Such behaviors resemble the prisoner's dilemma, where individually rational choices lead to socially suboptimal outcomes. The fundamental challenge is the lack of a principled decision making mechanism that mutually benefits both the LLM and the user. We propose Game-Theoretic Alignment (GTAlign), an alignment framework that integrates game-theoretic decision making into both reasoning and training. During reasoning, the model explicitly treats user-LLM interaction as a strategic game: it constructs payoff matrices within its reasoning chain to estimate welfare for both itself and the user, and then selects actions that are mutually beneficial. During training, we introduce a social welfare reward that reinforces cooperative responses, aligning model behavior with socially efficient outcomes. In addition, we introduce an inference technique that leverages game-theoretic reasoning to dynamically adapt LLM's response when pricing policies of LLM service change. Extensive experiments demonstrate that GTAlign substantially improves reasoning efficiency, answer quality, and social welfare compared to baselines across diverse tasks. The code is available at https://github.com/ulab-uiuc/GTAlign .

1 INTRODUCTION

Conversation systems can lack a principled way to evaluate how response strategies affect conversational outcomes. GTAlign combines game-theoretic reasoning and social-welfare training, and the paper reports gains across tasks.

1 INTRODUCTION

  • Existing methods often use task-specific rewards or behavioral imitation without explicitly reasoning over alternative strategies, leaving LLMs without a principled mechanism to assess conversational outcomes.The introduction identifies deliberate search of action spaces and strategic trade-off evaluation as an open problem.Source: §1 · p. 1
  • GTAlign models user–LLM interaction as a strategic game, adds payoff-matrix reasoning and a cooperative social-welfare reward, and steers pricing behavior by changing payoffs without fine-tuning.The framework jointly maximizes user and LLM rewards during training.Source: §1 · p. 2
  • Across four in-distribution datasets, GTAlign improves reasoning efficiency by 21.5%, answer quality by 4.9%, and social welfare by 7.2% versus baselines.On three out-of-domain datasets, it increases social welfare by 10.5% and answer quality by 7.4%; a user study reports an 11.3% improvement in satisfaction.Source: §1 · p. 2

2 PROBLEM FORMULATION

The section models user–LLM conversation as a sequential strategic game with distinct actions and payoffs. Its example shows how individually rational choices can produce a socially suboptimal equilibrium.

2 PROBLEM FORMULATION

  • Figure 1 contrasts a default Prisoner’s Dilemma interaction with the preferred jointly optimal outcome guided by the LLM.The section describes the default equilibrium as socially suboptimal despite the existence of a mutually beneficial state.Source: Figure 1 · p. 2, §2 · p. 3
  • The user can choose a vague or detailed question, while the LLM can provide a direct answer, ask a clarifying question, or answer and ask a follow-up.The payoff function represents utility for both the user and the LLM, and the matrix considers possible user actions on the next turn.Source: §2 · pp. 2–3
  • A vague question followed by a direct answer is the individually rational equilibrium at (1,1), although the interaction admits a jointly optimal payoff of (3,3).The example's cooperative path begins with the LLM asking a clarifying question, which encourages a detailed query before a high-quality answer with follow-up questions.Source: Figure 1 · p. 2, §2 · p. 3

3 GAME-THEORETIC ALIGNMENT (GTALIGN) FRAMEWORK

GTAlign responds to unstable training on classic games and uncertain transfer by embedding game-theoretic analysis in reasoning and optimizing social welfare. It comprises reasoning, welfare design, and inference-time steering.

3 GAME-THEORETIC ALIGNMENT (GTALIGN) FRAMEWORK

  • Direct RL on classic sequential games shows high variance and no convergence, while transfer to math reasoning is limited and non-systematic; GTAlign embeds game-theoretic analysis in reasoning and optimizes social welfare.The framework is intended to support rational, adaptive decisions in multi-turn interactions across domains.Source: Figure 3 · p. 4, Table 2 · p. 4, §3 · p. 4

3.1 GAME-THEORETIC REASONING

  • The game-theoretic reasoning chain assigns payoffs, constructs a JSON payoff matrix, identifies joint actions maximizing the geometric mean of user and model utilities, and produces the user-facing response.The model solves the matrix itself rather than relying on an external solver, and a reward evaluates whether its solution is correct.Source: §3.1 · p. 4

3.2 SOCIAL WELFARE DESIGN

  • Social welfare design weights measurable factors including answer quality, user and LLM costs, and game-theoretic reasoning, then aggregates user and LLM welfare with a Cobb-Douglas function.The aggregation yields zero if either party's welfare vanishes and has diminishing returns that direct optimization toward the worse-off party; increases correlate with higher user ratings.Source: §3.2 · pp. 4–5

3.3 STEERING LLM BEHAVIOR DURING INFERENCE

  • At inference time, modifying the payoff matrix can steer behavior when pricing policies change, including applying a token-cost penalty to LLM utility for subscription pricing.The passage reports that behavior can be steered without retraining.Source: Figure 4 · p. 5, §3.3 · p. 6

4 EXPERIMENT SETUP3

Experiments use four in-distribution datasets for training and testing, plus three out-of-distribution datasets for evaluation. The setup specifies Qwen2.5-3B-Instruct as the policy model and defines welfare, answer-quality, and baseline comparisons.

4 EXPERIMENT SETUP3

  • The training and test data cover Medium, Math, Ambig-QA, and WildGuard, split 9:1; OOD evaluation uses Minerva-Math, Ambig-CoQA, and AdvBench.The curated sets contain 1,000 Medium, 2,000 Math, 3,000 Ambig-QA, and 3,000 WildGuard questions; OOD sets contain 272, 1,060, and 520 questions, respectively.Source: §4 · p. 6
  • Qwen2.5-3B-Instruct is the policy backbone for SFT and RL, with Qwen3-32B used as the LLM judge.gpt-oss-20b and Qwen3-32B provide reference answers for SFT.Source: §4 · p. 6
  • User welfare weights answer quality at 0.5 and efficiency costs at 0.5, while LLM welfare weights answer quality at 0.4, efficiency at 0.2, and game-theoretic reasoning at 0.4.User efficiency costs comprise 0.2 response-length regularization and 0.3 reasoning latency; LLM efficiency is response-length regularization, and game-theoretic reasoning combines format correctness and payoff-matrix quality at 0.2 each.Source: §4 · p. 6
  • Answer quality is scored by task: safety behavior, math accuracy, writing similarity or clarification, and ambiguity handling; RL baselines use LLM welfare, user welfare, or an equal-weight combination.Table 3 reports format score, answer score, answer per token, and reasoning-to-total response length ratio.Source: §4 · p. 6, Table 3 · p. 7

5 MAIN RESULTS

GTAlign improves reasoning performance and efficiency, and its Cobb-Douglas reward yields higher social welfare and Pareto efficiency than alternative reward designs. The results also report gains in safety and ambiguity handling.

5.1 GTALIGN DEMONSTRATES SUPERIOR REASONING ABILITY

  • GTAlign consistently surpasses 3B-scale baselines across four datasets in answer score and achieves a near-perfect format score; RL also outperforms SFT and larger Qwen3-32B.The results indicate stronger answer quality and format consistency than SFT, while outperforming the larger model.Source: §5.1 · p. 7
  • 31% improvement in reasoning efficiency (Ans./Token) over the best baseline on Math.High answer scores and R/T scores further support cost-effective reasoning chains.Source: §5.1 · p. 7
  • GTAlign reaches total accuracies of 97.33% on safety-sensitive WildGuard tasks and 93.00% on Ambig-QA ambiguity handling, outperforming baselines on both.WildGuard evaluation distinguishes safe alternatives for harmful queries from helpful answers to harmless queries; Ambig-QA evaluates helpful answers and ambiguity handling.Source: §5.1 · pp. 7–8

5.2 GTALIGN IMPROVES WELFARE AND PARETO EFFICIENCY

  • Cobb-Douglas reward achieves the highest social welfare among four reward designs, with an average 7.2% improvement across tasks while maintaining strong user and LLM welfare.The comparison includes rewards optimizing user welfare, LLM welfare, their linear combination, and Cobb-Douglas aggregation.Source: §5.2 · p. 8, Table 5 · p. 8

6 ABLATION STUDIES

GTAlign generalizes to three out-of-domain datasets, demonstrates payoff-matrix reasoning, and receives higher human satisfaction ratings than the base and SFT models. Human ratings also correlate positively with social welfare.

6.1 GTALIGN GENERALIZES WELL TO OOD TASKS

  • On Minerva-Math, AdvBench, and Ambig-CoQA, GTAlign exceeds the best-performing baseline by 10.5% in social welfare and 7.4% in answer quality on average.The result is reported across three out-of-domain datasets.Source: §6.1 · p. 8, Table 7 · p. 9

6.2 GTALIGN SHOWS PAYOFF MATRIX REASONING ABILITY

  • An LLM judge compares GTAlign’s payoff-matrix reasoning with programmatically computed ground-truth mutually beneficial actions and finds robust reasoning quality.Table 7 reports out-of-domain performance alongside payoff-matrix reasoning ability.Source: §6.2 · p. 8, Table 7 · p. 9

6.3 USER STUDY ON GTALIGN

  • Human annotators rate GTAlign above the base and SFT models across four test sets, with an average satisfaction improvement of 11.3%; satisfaction ratings strongly correlate with social welfare.Three annotators rated responses on a 1–5 scale, and the study sampled 20 questions from each test set.Source: §6.3 · p. 9

7 RELATED WORK

Related work covers human-centered dialogue, reasoning efficiency, game theory for LLMs, and multi-objective alignment. GTAlign combines game-theoretic reasoning and reward modeling to target Pareto-efficient responses under explicit utility trade-offs.

7 RELATED WORK

  • Human-centered LLM work addresses clarification, persuasion, and social simulation; GTAlign emphasizes generalizable QA with payoff awareness, interactivity, and safety.Prior approaches include prompt engineering and training for clarification, and adapting persuasive strategies.Source: §7 · p. 9
  • Reasoning-LLM research studies accuracy-computation trade-offs through prompting, length-penalized RL, and certainty-based methods; GTAlign makes decisions interpretable through game-theoretic reasoning.It also frames reasoning as human-centered communication across interactive settings.Source: §7 · pp. 9–10
  • Game-theoretic LLM work includes game-based evaluation and training games; GTAlign applies game principles to both reasoning and reward modeling for Pareto-efficient responses.Its approach predicts multiple strategic payoffs in one turn under explicit utility trade-offs.Source: §7 · p. 10
  • Multi-objective alignment commonly combines rewards with fixed-weight linear scalarization, while GTAlign defines social welfare through a non-linear function of user and LLM utilities.The utilities themselves are formulated as linear combinations of measurable factors.Source: §7 · p. 10

8 CONCLUSION

GTALIGN models user–LLM interactions as strategic games and incorporates welfare considerations into reasoning and training. The experiments show gains in reasoning efficiency, answer quality, and social welfare, while the authors identify further directions for extending the framework.

8 CONCLUSION

  • GTALIGN models user–LLM interactions as strategic games to incorporate welfare considerations into both training and reasoning.The authors describe this as a principled lens for studying alignment.Source: §8 · p. 10
  • Experiments demonstrate gains in reasoning efficiency, answer quality, and social welfare, alongside improved explainability and controllability of reasoning.Source: §8 · p. 10
  • The authors identify complex payoff matrices, multi-agent interactions, and rigorous social-welfare definitions for real-world scenarios as future research directions.They characterize the work as a first step toward a broader design space of game-theoretic alignment.Source: §8 · p. 10

ETHICS STATEMENT

The paper frames GTALIGN as supporting rational, cooperative LLMs aligned with human welfare, while acknowledging that its inference-time steering can be used to manipulate incentives. It describes transparency and the Cobb-Douglas utility function as mitigating factors and limits safety-dataset use to controlled research.

ETHICS STATEMENT

  • The work aims to advance LLM systems that are more rational, cooperative, and aligned with human welfare.It seeks a principled foundation for decisions that foster socially efficient and mutually beneficial outcomes.Source: ETHICS STATEMENT · pp. 10–11
  • GTALIGN’s inference-time steering can let service providers alter payoff matrices to favor their objectives over user welfare.The paper identifies this as a manipulation risk.Source: ETHICS STATEMENT · p. 11
  • The authors cite interpretable payoff matrices and a Cobb-Douglas utility function as safeguards against neglecting user welfare.They state that social welfare approaches zero if user welfare is neglected.Source: ETHICS STATEMENT · p. 11
  • WildGuard and AdvBench were used only in controlled research to train and evaluate identifying and refusing malicious requests.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the experimental models, dataset availability, training configurations, and evaluation materials. The appendices provide training configurations and evaluation definitions, while the paper also lists language-editing uses of LLMs.

REPRODUCIBILITY STATEMENT

  • The statement identifies Qwen2.5-3B-Instruct as the experimental base model and Qwen3-32B as the quality-evaluation judge.It says the datasets are publicly available and points to Section 4 and Appendix B.2 for dataset details and citations.Source: REPRODUCIBILITY STATEMENT · p. 11
  • The statement directs readers to formal definitions of welfare and Pareto-efficiency metrics and to the exact LLM-judge prompts in the appendices.Source: REPRODUCIBILITY STATEMENT · p. 11

A THE USE OF LARGE LANGUAGE MODELS

B.1 DETAILS ABOUT GTALIGN TRAINING

  • SFT uses deepspeed under Table 9 with eight NVIDIA H20 GPUs; RL uses verl under Table 10 with four NVIDIA H20 GPUs, plus four additional H20 GPUs to deploy the Qwen3-32B judge via SGLang.Source: §B.1 · p. 19

B.2 DATASET

This section describes the study’s training and evaluation data, social-welfare rewards, Pareto metrics, and game-theoretic prompting and judging procedures. It also outlines dataset-specific judge rubrics for factual accuracy, safety and helpfulness, writing assistance, and payoff-matrix action consistency.

B.2 DATASET

  • OOD evaluation tests mathematical reasoning, adversarial safety, and multi-turn ambiguity beyond the corresponding training tasks.The benchmarks are Minerva-Math, AdvBench, and Ambig-CoQA; in-distribution data include Math-Hard and Medium, Ambig-QA, and WildGuard.Source: §B.2 · p. 19

B.3 SFT DATA SYNTHESIS

  • SFT data are generated sequentially across four Game-Theoretic CoT components, then nine candidate responses per question are screened for highest social welfare.Qwen3-32B generates thinking, analysis, and response, while gpt-oss-20b generates the payoff matrix; each component conditions on preceding components.Source: §B.3 · p. 20

B.4 REWARD DESIGN

  • Sensitivity analysis varying welfare reward-component weights by ±0.1 from the chosen baseline found the selected configuration on a stable performance plateau, measured by Pareto Coverage against the SFT baseline.User welfare combines answer quality, length regularization, and reasoning score with weights 0.5, 0.2, and 0.3; LLM welfare combines format correctness, payoff alignment, answer quality, and length regularization with weights 0.2, 0.2, 0.4, and 0.2.Source: §B.4 · pp. 20–21, Table 12 · p. 21, Table 13 · p. 21
  • Reward instantiation varies by dataset: ambiguity judging for Ambig-QA, exact-match correctness for Math, harm avoidance for WildGuard, and BLEU-or-judge adequacy for Medium.Length regularization favors 100–1,000 tokens for user welfare and 500–1,500 for LLM welfare, with a user-side penalty for unnecessarily long reasoning.Source: §B.4 · p. 20

C.1 METRICS TO EVALUATE PARETO EFFICIENCY

  • Pareto evaluation counts instance-level dominance and assesses global efficiency through frontier coverage, hypervolume, and average regret.Larger hypervolume indicates a frontier covering more utility space, while smaller regret indicates solutions closer to Pareto optimality.Source: §C.1 · p. 22, §C.1.1 · p. 22, §C.1.2 · p. 22

C.2 COBB-DOUGLAS UTILITY FUNCTION; C.3 A JUSTIFICATION OF SOCIAL WELFARE DESIGN

  • Cobb–Douglas welfare combines user and LLM outcomes multiplicatively, while the CES family spans efficiency-focused, balanced, and fairness-focused trade-offs.The design seeks symmetry, monotonicity, and zero-dominance, motivating GTAlign’s adoption of Cobb–Douglas welfare.Source: §C.2 · p. 23, §C.3 · pp. 23–24

D.1 SYSTEM PROMPT DURING RL TRAINING

  • The RL prompt compares direct-answer, clarification-only, and answer-plus-follow-up strategies, derives their Pareto frontier, and selects a strategy by joint welfare with user-first tie-breaking.It classifies questions as definite or vague and instructs the model to implement the chosen strategy in tagged output blocks.Source: §D.1 · pp. 24–25

D.2 LLM JUDGE PROMPT FOR AMBIG-QA DATASET

  • The Ambig-QA judge rewards clarification or multiple-interpretation answers for ambiguous questions and direct relevant answers otherwise, using a binary score.The prompt defines 1 as good ambiguity handling and 0 as poor handling.Source: §D.2 · p. 26

D.3 LLM JUDGE PROMPT FOR MATH DATASET

  • For Math, the judge assigns binary accuracy based on whether the model’s final answer is factually correct and consistent with the provided ground truth.A score of 1 means the response matches the ground truth; 0 means it contradicts or misses it.Source: §D.3 · p. 26

D.4 LLM JUDGE PROMPT FOR WILDGUARD DATASET

  • For WildGuard, harmful-request responses that avoid compliance and offer a safe, constructive alternative receive the preferred 0.5 score, while refusal alone scores 1; harmless requests are judged for helpfulness.The rubric also treats a response to a harmless request as unhelpful if it refuses unnecessarily, hallucinates, or fails to assist.Source: §D.4 · p. 27

D.5 LLM JUDGE PROMPT FOR MEDIUM DATASET

  • For Medium writing-related queries, the judge scores responses from 0 to 4 for helpfulness and alignment with intent, treating thoughtful clarification as helpful when intent is unclear or multiple interpretations are possible.The prompt supplies the user intent, query, ideal answer, and assistant response as evaluation context.Source: §D.5 · pp. 28–29

D.6 LLM JUDGE PROMPT FOR EVALUATING PAYOFF MATRIX REASONING ABILITY

  • For payoff-matrix reasoning, the judge scores whether the explanation identifies a model action on the ground-truth Pareto frontier, using 1 for explicit recognition, 0.5 for a vague or overshadowed match, and 0 for no correct action.Extra or missing actions are tolerated if at least one correct frontier action is clearly identified.Source: §D.6 · p. 29

E LIMITATIONS AND FUTURE DIRECTIONS

The paper identifies implementation and evaluation boundaries for GTAlign, including payoff-matrix tooling, model coverage, and inference-time assumptions. These remain areas for future work or caution.

E LIMITATIONS AND FUTURE DIRECTIONS

  • GTAlign’s current implementation analyzes payoff matrices within its reasoning chain but does not use external tools to solve them.Adding external tool usage is left for future work and would require additional engineering.Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 31
  • The study primarily evaluates Qwen2.5-3B-Instruct as its base model and Qwen3-32B as its LLM judge.The authors describe broader model-family evaluation as future work, while saying the present setup is sufficient to substantiate their claims.Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 31
  • Inference-time payoff adaptation assumes pricing policies can be reliably detected, considers a limited set of factors, and may respond unstably to abrupt payoff-weight changes.The authors suggest smoothing payoff weights across dialogue turns as a response to the instability caveat.Source: E LIMITATIONS AND FUTURE DIRECTIONS · p. 31
Loading 2510.08872v3…