Source-linked AI summary
Words or Deeds? How LLMs Predict Users' Decisions in Human-Agent Dialogue
Baiqiao Zhang, Shuai Ma, Zhifeng Liao, Chao Zhou, Jianshu Wang, Yuhan Zhao, Chaoyue Zhang, Tongjun Si, Yuying Tang, Xiangxian Li, Yong-Jin Liu, Yulong Bian
TL;DR
In mixed-motive interactions, statements can express plans while also seeking to influence the other side, and no study links statements to users’ earlier decisions, counterparts, and later actions. The paper evaluates eight open-weight LLMs on recorded repeated games, finding that decision records predicted cooperation well but dialogue brought no detectable overall gain and obscured important prediction errors.
Problem
No study links users’ statements to their earlier decisions, the counterpart they face, and their later actions, though statements can both express plans and influence the other side in mixed-motive interactions.
Source: §1 · p. 2Method
Results
Takeaways & Limitations
Predictors should assess users’ statements against their earlier decisions, particularly when the statements conflict with those decisions.
Source: §1 · p. 2The study used 42 Chinese university students in short, low-stakes games and eight zero-shot predictors; generalization to other populations, longer relationships, and Web tasks needs validation.
Source: §5.4 · p. 9
Abstract
from arXiv · showhide
Web systems increasingly predict users' decisions from what users say, yet when interests partly conflict, users may promise to cooperate to win the other side's cooperation and then not do so. Sales or service agents that rely on such promises may grant concessions to users who then do not keep their promises, at a cost to the party they represent. We test how LLMs combine users' statements with decision records to predict users' decisions, using 1,260 rounds in which 42 users played a repeated prisoner's dilemma-style game with five agents, each prompted to express a dominant Big Five trait, and could talk with the agent before every decision. Across eight open-weight LLMs, predictions from decision records separated cooperation from defection well (AUC = .80), and adding dialogue brought no detectable gain. With dialogue, predictions agreed with users' statements more often, and errors concentrated on rounds in which users broke their statements: accuracy on rounds with a statement barely changed (.73 to .74), while it rose on kept statements (.76 to .95) and fell sharply on broken ones (.65 to .17). Predictions weighted statements more heavily than users' behavior warranted: when users stated a decision they had not yet made in that game, the predicted probability of keeping it rose from .13 to .69, whereas users kept .40 of such statements. Users' decisions also varied with the agent they faced, but predictions hardly reflected this: users cooperated more with the Agreeableness agent than with the Extraversion agent (70.6% vs. 54.5%), whereas predicted cooperation was nearly identical (64.5% vs. 63.1%). Users also cooperated more with an agent when they themselves scored higher on that agent's trait, whereas predicted cooperation barely changed with these scores. Dialogue-based predictions should therefore weigh users' statements by their earlier decisions and the agent they face.
1 Introduction
The introduction asks whether LLMs can combine users’ statements with behavioral records without over-relying on promises, and whether their predictions track differences across agents and user-agent pairings.
1 Introduction
- Because promises can both express plans and seek to influence a counterpart, predictions should weigh statements in light of earlier decisions and the agent faced.The introduction notes that promises raise cooperation but about a quarter are broken, and that pooled accuracy can conceal gains on kept statements and losses on broken ones.Source: §1 · pp. 2–3
- The study links separately recorded statements to earlier decisions, the agent faced, and the subsequent action, addressing a gap in existing prediction and user-simulator studies.Those studies lacked one or more of repeated behavioral histories, statements of the predicted action, or later actions that could contradict simulated conversation.Source: §1 · p. 2
- The researchers used a repeated prisoner’s-dilemma-style game with 42 users, five trait-prompted agents, and eight open-weight LLMs predicting decisions from histories and dialogue.Predictions were assessed by whether statements were kept or broken, whether they conflicted with earlier decisions, and which agent the user faced.Source: §1 · p. 2
- Decision records achieved AUC = .80, while adding dialogue brought no detectable gain and sharply reduced accuracy on broken statements.On rounds with a stated decision, accuracy changed from .73 to .74; on kept statements it rose from .76 to .95, while on broken statements it fell from .65 to .17.Source: §1 · p. 2
- For decisions users had not yet made in that game, dialogue raised predicted word-keeping from .13 to .69, although users kept only .40 of those statements.The predictions therefore moved beyond the observed rate of keeping these statements.Source: §1 · p. 2
- Users cooperated more with the Agreeableness than the Extraversion agent (70.6% vs. 54.5%), while predicted cooperation was nearly unchanged (64.5% vs. 63.1%).Users also cooperated more as trait compatibility rose, whereas predictions barely changed and increasingly fell below users’ cooperation.Source: §1 · p. 2
2 Related Work
Related work examines prediction from dialogue and behavioral history, the reliability of stated intentions, and how counterpart and user-agent differences shape behavior and forecasts.
2.1 Prediction from Dialogue and History
- Repeated-game prediction work uses game structure and prior actions, while language-based studies examine whether dialogue or generated data improve forecasts; one-shot settings lack a history for assessing current statements.Prior work also considers conversations that provide choice context without stating the intended choice.Source: §2.1 · p. 3
2.2 Stated Decisions as Behavioral Evidence
- Stated decisions can predict behavior without guaranteeing it, and prior research found that human observers overestimated promise keeping and struggled to distinguish kept from broken promises.This motivates separating whether statements are informative from whether forecasters’ confidence matches observed behavior.Source: §2.2 · p. 3
2.3 Counterparts and User-Agent Pairing
- Agent conduct and persona, as well as user-agent personality pairing, can shape behavior; this paper compares forecasts with actions across agents and pairings.Prior findings include persona-related variation in cooperation and agreement keeping, and mixed evidence about personality similarity versus complementarity.Source: §2.3 · p. 3
3 Experiment Design
The study evaluates offline LLM forecasts of decisions in repeated games, where users converse with trait-prompted agents before deciding. It defines measures of users’ behavior and predictions, tests 19 information conditions, and uses user-level bootstrap inference.
3 Experiment Design
- Predictions are evaluated offline: each round’s predictor receives pre-decision material and forecasts whether the user will cooperate.The predictor is separate from the agent that played with the user.Source: §3 · p. 3
3.1 Users, Agents and Game
- Forty-two participants played six rounds with each of five trait-prompted GPT-4o agents, with conversation before each decision and payoffs based on cooperation or defection.The five agents expressed Openness, Conscientiousness, Extraversion, Agreeableness, or Neuroticism; dialogue was empty in 276 of 1,260 rounds.Source: §3.1 · p. 3
3.2 Stated Decisions
- A stated decision is a user’s expressed choice for that round; in games 2–5, 96% announced cooperation, and 27.6% of stated cooperation was followed by defection.Statements were independently labeled by two researchers, with κ = .97 agreement.Source: §3.2 · p. 3
3.3 Quantities
- The measures track cooperation and word-keeping by agent, with relative rates subtracting each user’s own mean across games, and trait compatibility standardizing the user’s score on that agent’s defining trait.Word-keeping compares stated decisions with the actions taken in rounds where a decision was stated.Source: §3.3 · p. 4
3.4 Predictors and Information Conditions
- Eight open-weight LLMs received 19 combinations of decision records and dialogue; predictions used games 2–5, while behavior comparisons across agents used all five games.The information conditions varied which prior decisions and dialogue were supplied to the predictors.Source: §3.4 · p. 4, Figure 2 · p. 4
3.5 Statistical Analysis
- Uncertainty was estimated by bootstrapping users, with Holm correction within test families; RQ3 used an uncorrected least-squares slope of game-level outcomes on compatibility.Each of 50,000 bootstrap draws sampled 42 users with replacement and retained all their rounds.Source: §3.5 · p. 5
4 Results
Decision records predicted cooperation better than dialogue, but adding dialogue made predictions follow users’ statements even when they were broken. Predictions also underrepresented variation associated with agent persona and user–agent trait compatibility.
4.1 RQ1: Words and Deeds
- .80 AUC for decision records exceeded .70 for dialogue alone, while combining both reached .79 and adding dialogue brought no detectable gain.Adding a decision record to dialogue raised AUC in all 12 comparisons.Source: §4.1 · p. 5
- With current-round dialogue, accuracy on stated-decision rounds barely changed (.73 to .74), rising on kept statements (.76 to .95) and falling on broken ones (.65 to .17).The accuracy gains on kept statements offset losses on broken statements.Source: §4.1 · p. 5, Figure 4 · p. 6
- For decisions users had not yet made in that game, dialogue raised predicted keeping from .13 to .69, against an observed rate of .40.After an earlier broken statement, dialogue lowered predicted cooperation from .60 to .49, while observed cooperation was .29.Source: §4.1 · p. 6
- One inserted sentence shifted agreement from .75 to .98 when matching the recorded decision and to .08 when conflicting; model responses to statements exceeded their association with users’ actions.The difference between model sensitivity and behavioral association was positive for all eight models.Source: §4.1 · p. 6
4.2 RQ2: Agent Persona
- In games 2–5, users cooperated with the Agreeableness agent more than the Extraversion agent (70.6% vs. 54.5%), while predictions were nearly identical (64.5% vs. 63.1%).AUC did not differ detectably across personas, but persona-specific prediction biases differed by up to 14.6 points despite near-zero pooled bias.Source: §4.2 · pp. 6–7, Figure 5 · p. 7
4.3 RQ3: User Personality and Compatibility
- Users’ relative cooperation rose 3.0 points per standard deviation of trait compatibility, while predictions increasingly underrepresented cooperation as compatibility rose.Adding personality or compatibility information reduced the estimated bias slope, but most of the bias remained.Source: §4.3 · pp. 7–8
5 Discussion
The discussion argues for grounding stated intentions in prior statement–action records and evaluating predictions by counterpart and outcome group. It also cautions that the study’s setting limits generalization.
5.1 Grounding Statements in Behavior
- Linking each earlier statement to its outcome corrected most overestimation of promise keeping, supporting user-level records of what people said and then did.For inconsistent statements, predicted keeping fell from .69 to .40, matching the observed rate; sales or service agents could weigh new promises against that record.Source: §5.1 · p. 8
5.2 Modeling Users with Their Counterparts
- Because cooperation varied with agent persona and was associated with trait compatibility, user representations should retain the counterpart and interaction history rather than assume a stable general tendency.Compatibility showed no detected association with statement fulfillment, distinguishing willingness to cooperate from reliability of stated decisions.Source: §5.2 · p. 8
5.3 Evaluating Errors That Averages Conceal
- Opposing errors across fulfillment groups and personas can cancel in pooled measures, so evaluation should report accuracy for kept and broken statements and for each agent.AUC measures discrimination, not agreement between predicted probabilities and observed rates; pooled cooperation rates alone provide little evidence of counterpart-specific responses.Source: §5.3 · p. 8
5.4 Limitations and Future Work
- The evidence comes from 42 Chinese university students in short, low-stakes games with eight zero-shot predictors and GPT-4o counterparts, so broader settings need validation.The study also notes that persona prompts confound personality expression with behavioral reciprocity and that statement–action associations do not identify causal effects.Source: §5.4 · p. 9
6 Conclusion
The paper evaluates how LLMs use dialogue and behavioral history to predict decisions in mixed-motive games. Predictors overestimated fulfillment of stated decisions and understated behavior differences associated with agent persona and trait compatibility, motivating evaluation tied to subsequent actions and interaction context.
6 Conclusion
- Across recorded mixed-motive games, eight open-weight predictors overestimated fulfillment of stated decisions and understated behavioral variation associated with agent persona and trait compatibility.Nearly unchanged accuracy on stated-decision rounds and near-zero pooled bias across agents obscured these mismatches; explicit personality information modestly reduced the estimated compatibility-related bias slope.Source: §6 · p. 9
A Predictor Models and Inference
The appendix describes predictor inference and the translated prompt structure for a repeated human–AI cooperation game. Prompts supply game context and earlier decisions, then ask for the player’s current choice.
A Predictor Models and Inference
- Predictor inference used one greedy output token and read log-probabilities for the two answer letters, with an exact prompt-log-probability fallback when needed.Reasoning was disabled where supported; inference ran on NVIDIA L20 and RTX A6000 GPUs.Source: A Predictor Models and Inference · p. 10
B Prompt Template
- The prompt describes six rounds of talk followed by hidden cooperate-or-defect choices, with specified payoffs and the agent’s current choice concealed from the player.The prompt text is presented as an English translation of fixed parts that were written in Chinese.Source: B Prompt Template · p. 10
- Prompt blocks can include earlier games’ decisions and dialogue, the current round number, earlier rounds’ dialogue and results, and a one-letter choice question.Blocks absent from a condition are left out.Source: B Prompt Template · p. 10
C Personality Information in the Prompt
The appendix describes prompt conditions that added users’ personality information or a record of their earlier stated and actual decisions. These conditions were tested on games 2–5 in both letter orders.
C Personality Information in the Prompt
- The two personality-information conditions added either the user’s BFI-44 scores or a compatibility score to the prompt, and were tested on 1,008 rounds from games 2–5 in both letter orders.Both blocks were placed after the game rules.Source: C Personality Information in the Prompt · p. 10
- Users’ self-reported Big Five dimensions were scored from 1 to 5; mean scores were Openness 3.47, Conscientiousness 3.13, Extraversion 3.10, Agreeableness 3.76, and Neuroticism 2.95.Higher scores indicate a stronger trait.Source: C Personality Information in the Prompt · p. 10
- The agent was prompted to be extremely high on one Big Five dimension, and the prompt included the user’s score on that same dimension.The user score was reported as a self-report on a 1–5 scale and as higher than a percentage of study participants.Source: C Personality Information in the Prompt · pp. 10–11
- Table 2 lists the rounds and users included in each analysis subset.The caption states the table’s scope but gives no subset values in the supplied passage.Source: Table 2 · p. 11
D Word-Keeping Record in the Prompt
- A further condition added each user’s earlier stated decisions alongside the decisions they actually made; it was tested on 1,008 rounds from games 2–5 in both letter orders.The block followed earlier-round results and preceded current-round dialogue; round 1 had no such record.Source: D Word-Keeping Record in the Prompt · p. 11
- For statements consistent with earlier decisions and users with no broken statement, the record changed predictions by less than .05 compared with earlier-round dialogue.The comparison used the groups in Figure 4b and c, with the same user bootstrap, Holm correction, and letter-order rule as that figure.Source: D Word-Keeping Record in the Prompt · p. 11
E Stated-Decision Labels
Researchers independently labeled users’ statements about their intended decisions without seeing the decisions themselves. The labels were resolved through discussion, with high agreement after categories were collapsed.
E Stated-Decision Labels
- Two researchers independently coded each round’s dialogue without seeing the user’s decision, using the agent’s words only to understand the user’s response.When users changed their statement, the last statement counted.Source: E Stated-Decision Labels · p. 11
- The five labels were explicit cooperation, implicit cooperation, defection, vague, and none; across 1,260 rounds, counts were 594, 197, 30, 116, and 323, respectively.Explicit and implicit cooperation were collapsed into stated cooperation; vague and none were merged as no stated decision.Source: E Stated-Decision Labels · p. 11
- Inter-rater agreement was κ = .81 for five classes and κ = .97 after collapsing them into stated cooperation, stated defection, and no stated decision.All disagreements were resolved through discussion.Source: E Stated-Decision Labels · p. 11
F Descriptive Statistics
Users cooperated in 64.5% of rounds, and overall kept 74.3% of their stated decisions. Cooperation rates also differed across rounds and agent personas.
F Descriptive Statistics
- Users cooperated in 64.5% of 1,260 rounds, declining from 82.4% in round 1 to 42.4% in round 6.Agents cooperated in 77.7% of rounds overall, with persona-specific rates from 67.5% to 89.3%.Source: F Descriptive Statistics · p. 11
- Users kept 74.3% of stated decisions overall: 74.0% of stated cooperation and 83.3% of stated defection.After decision-submission records were removed, dialogue was empty in 276 rounds; users spoke 2.4 turns on average in the other 984 rounds.Source: F Descriptive Statistics · p. 11
- Mean user BFI-44 scores on the 1–5 scale were Openness 3.47, Conscientiousness 3.13, Extraversion 3.10, Agreeableness 3.76, and Neuroticism 2.95.The number of users meeting each persona in games 2–5 ranged from 30 for Openness and Agreeableness to 39 for Conscientiousness.Source: F Descriptive Statistics · p. 11
G Analysis Subsets
Analysis subsets differed by purpose: prediction analyses used games 2–5, while user-behavior analyses across agents used all five games. Comparisons with earlier rounds excluded round 1.
G Analysis Subsets
- Prediction analyses used games 2–5, and comparisons of user behavior with predictions used the same rounds.These analyses required materials from another game to be available.Source: G Analysis Subsets · p. 11
- Analyses of user behavior across agents used all five games because relative rates were calculated from each user’s five games.Source: G Analysis Subsets · p. 11
- Analyses comparing a round with earlier rounds of its game excluded round 1.These were the analyses shown in Figure 4b–d.Source: G Analysis Subsets · p. 11
H Test Families and Weights of the Inserted Sentence
The section compares how much inserted cooperation statements influence predictions with how strongly prior decisions influence users’ behavior. It also describes the statistical correction procedures and ethical safeguards and risks.
H Test Families and Weights of the Inserted Sentence
- Holm correction was applied within each defined test family, including Figure 3 and Figure 4 panels, persona pairs, prediction-bias tests, and inserted-sentence tests.The inserted-sentence tests covered weight-ratio differences, changes in the previous-decision effect, and changes in AUC.Source: H Test Families and Weights of the… · pp. 11–12
- Model weight ratios ranged from .63–2.60, all above the behavioral ratio of .35.The reported model ratio is the mean across eight models; the behavioral ratio compares stated cooperation with no statement against the user’s previous decision.Source: H Test Families and Weights of the… · p. 12
- A second cooperation-only wording, “I’ll pick cooperate this round,” had a similar effect.Source: H Test Families and Weights of the… · p. 12
I Ethical Considerations
- Unfulfilled statements may reflect changed decisions rather than dishonesty; the authors warn that prediction and persona selection also risk manipulative targeting.They recommend disclosing agent roles and data use, preserving users’ ability to revise choices, and obtaining separate authorization for forecasts.Source: I Ethical Considerations · p. 12