Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization

1 Introduction

• Most LLM agents rely on task-specific, sophisticated prompts, while complex imperfect-information tasks require strategic behavior and agents that can learn from interaction.The introduction frames learning and elevating behavioral strategies through interaction as an under-explored question.Source: §1 · pp. 1–2
• Agent-Pro forms beliefs about itself and the world, then reflects on past experiences to calibrate beliefs and optimize its behavior policy without parameter tuning.Its decisions draw on those beliefs, and its reflection turns experience into useful prompt instructions for a new policy.Source: §1 · p. 2
• After evolution, Agent-Pro defeats vanilla LLMs and specialized models in Blackjack and Texas Hold’em, improving game payoffs.The paper also reports learned skills such as cutting losses, bluffing, and disguising intent.Source: §1 · pp. 2–3
• Policy-level reflection updates long-horizon strategies from sequences of actions and delayed feedback, rather than correcting a single action.The process includes belief calibration, refinement of behavioral guidelines and world modeling, and policy verification.Source: §1 · p. 2
• Agent-Pro turns long-term interactions into an optimizable policy of Behavioral Guidelines and World Models, including objectives, rules, strategies, and demonstrative trajectories.This contrasts with approaches that store experience as verbal long-term memory for text-based reasoning.Source: §1 · p. 2

2 Problem Definition

• The study focuses on multiplayer imperfect-information games, where agents have access to their own states and public information but not others’ states.In Texas Hold’em, for example, players cannot observe opponents’ cards.Source: §2 · p. 3
• Dynamic interaction means multiple agents can influence the environment, making the effects of one agent’s actions unpredictable to others.Source: §2 · p. 3
• Blackjack and multiplayer Limit Texas Hold’em serve as testbeds for scenarios that can also be abstracted in competition, company negotiations, and security.Source: §2 · p. 3

3 Methods

• Agent-Pro turns policy learning into gradient-free prompt optimization, updating instructions from exploration experience to consolidate high-reward strategies.The approach uses in-context learning rather than training the LLM’s parameters.Source: §3 · p. 3

3.1 Belief-aware Decision-Making Process

• The agent updates self- and world-beliefs before choosing actions, aiming for coherent decisions in dynamic, imperfect-information scenarios.Beliefs are expressed in natural language and updated during each decision cycle.Source: §3 · p. 3, §3.1 · p. 4

3.2 Policy-Level Reflection

• For long-horizon tasks with delayed feedback, Agent-Pro reflects on failed trajectories and the rationality of underlying beliefs rather than correcting individual actions.It summarizes the reflection into Behavioral Guideline and World Modeling instructions, then verifies them by replaying the same game; instructions are retained if the score improves, with failed candidates discarded after three retries.Source: §3.2 · p. 5

3.3 DFS-based Policy Evolution

• DFS-based policy evolution evaluates updated policies on novel scenarios and searches candidate branches for a policy with higher gains than the strongest opponent.Evaluation swaps players’ hands and playing order to mitigate those sources of randomness; unsuccessful branches can be rejected and the search can backtrack to alternatives.Source: §3.3 · pp. 5–6

4 Game: Blackjack

• The Blackjack experiments use RLCard and report each agent’s win rate against the dealer over 900 games.Agents compete independently against the dealer.Source: §4 · p. 6, §4.1 · p. 6, Table 1 · p. 6

4.1 Results

• +4% average advantage over baseline agents; on Qwen-72B and Llama2-70B, Agent-Pro exceeds Reflexion by +3.9% and +11%, respectively.For GPT-4, the strategies have similar win rates; Table 1 reports win rates against the dealer over 900 games.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Analyisis

• Agent-Pro’s hit rate differs between low- and high-card dealer scenarios, tending to stand against high cards and hit against low cards.The analysis attributes this behavior to its belief that a dealer with high cards is more likely to bust, making the player’s own risk less worthwhile.Source: Figure 3 · p. 6, §4.2 · p. 7

5 Game: Limit Texas Hold’em

• Across 167 training hands and 20 evaluation hands, the study evaluates policies on 1,600 test games formed by swapping hands and player positions.The four-player matches include DQN, DMC, GPT-3.5, and the evaluated agent.Source: §5 · p. 7

5.1 Results

• Agent-Pro consistently outperforms RL-based agents such as DMC and surpasses other LLM agents; reported advantages include +2 points on average and +3.2 points over DMC on GPT-4.It also exceeds other LLM-based agents by more than 2.0 points.Source: §5.1 · p. 7, Table 2 · p. 8
• Without policy-level learning, dynamic beliefs improve Vanilla LLM performance by +0.9 points overall, while few-shot examples help less than the evolution strategy and action-sequence reflection is unstable.The reported improvements are +1 point on GPT-3.5 and +1.3 points on GPT-4; dynamic beliefs track community cards and opponent-strategy changes.Source: §5.1 · p. 7

5.2 Analysis on Learning Process

• GPT-4- and GPT-3.5-based agents improve rapidly early (+2.1 and +2.3 chips maximum increases), while Llama2-70B first declines and then gains +0.6 chips.Their strategies differ: GPT-4 may bluff to probe opponents, GPT-3.5 becomes cautious and folds more, and Llama-based Agent-Pro adopts a risk-averse approach.Source: §5.2 · pp. 7–8, Figure 4 · p. 8

5.3 Analysis on Policy Evolution

• As learning proceeds, Preflop folding decreases and raising rebounds across all four stages; Agent-Pro shifts from cautious early behavior toward flexible, proactive play.Compared with Vanilla LLM, Agent-Pro wins more chips with strong hands, loses fewer with weak hands, and improves performance by more than 80% with medium-strength hands.Source: §5.3 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 9

• Texas Hold’em can have up to 10^162 information sets, and Agent-Pro uses belief-aware decision-making to address this challenge.Source: §6 · p. 9
• Because the environment changes with opponents’ actions, Agent-Pro uses self-belief and world-belief to represent the dynamic environment.Source: §6 · p. 9
• With opponents’ states unobservable and bluffing common, Agent-Pro uses policy-level reflection and optimization to learn strategies and surpass trained RL-based agents.Source: §6 · p. 9

• The paper frames Agent-Pro as an agent capable of learning and evolving in complex interactive tasks.Source: §7 · p. 9
• Agent-Pro first constructs dynamic beliefs for uncertain decisions, then reflects on experience, corrects irrational beliefs, and summarizes findings as behavioral guidelines and world descriptions for a new policy.Source: §7 · p. 9
• After learning from historical experiences, Agent-Pro’s decision-making capabilities significantly improve across the two evaluated zero-sum games.Source: §7 · p. 9

• Agent-Pro’s learning process depends heavily on the foundational model’s reasoning and reflection abilities.Source: Limitations · p. 9
• GPT-3.5- and Llama2-70B-based Agent-Pro remain weaker than DMC despite notable improvements, while GPT-4-based Agent-Pro approaches human-player level and surpasses DMC.Source: Limitations · p. 9
• The authors note a possible significant gap from state-of-the-art algorithms such as CFR+ in gaming scenarios.Source: Limitations · p. 9

A.1 LLMs; A.2 Baselines

• The evaluation uses GPT-3.5-Turbo-0613, GPT4-0613, Llama2-Chat-70B, and Qwen-72B, compares five agent baselines, and averages five runs per test.Baselines include Vanilla LLM, ReAct, Reflexion, Self-Consistency, and Multiagent Debate.Source: §A.1 · p. 14, §A.2 · p. 14

A.3 Setups For Blackjack

• Blackjack learning uses 50 failed games and evaluation on 900 newly sampled games, omitting Verification and DFS-based Policy Evolution.The authors omit those components because Blackjack has a small state space.Source: §A.3 · p. 14

A.4 Detailed Setup For Texas Hold’em

• Texas Hold’em learning selects 167 challenging hands for training and 20 for development from 500 allocated hands, using Exploration, Reflection, and Evolution.Evolution evaluates sampled development hands and stops when policy improvement ends or all samples have been explored.Source: §A.4 · p. 14

B Introduction of Two Games; B.1 Blackjack; B.2 Limit Texas Hold’em; B.3 The Challenging of two Games

• Blackjack tests hit-or-stand decisions under hidden-card uncertainty, while Limit Texas Hold’em adds private cards, community cards, and four betting actions.The two games are presented as testbeds for uncertainty and dynamic interaction.Source: §B.1 · p. 14, §B.2 · p. 14, §B.3 · p. 14

B.3 The Challenging of two Games

• The games span different complexity levels: Blackjack has 1,000 information sets, while multiplayer Limit Texas Hold’em reaches 10^14.The paper uses the contrast to examine learning in intricate scenarios.Source: §B.3 · p. 15

C.1 Whether Few-shot Learning Can Handle Such Complex Interaction

• Few-shot winning trajectories change performance by -0.5, while losing trajectories improve it by +0.6 in the reported comparison.The authors suggest winning demonstrations may add little because the vanilla agent can already win those games.Source: §C.1 · p. 15, Table B2 · p. 15

C.2 Policy-Level Reflection without Belief

• Removing dynamic belief lowers Agent-Pro’s chips from 4.3 to 2.5, though it remains +1.1 over Vanilla LLM.The ablation also produced vaguer, more verbose instructions than Agent-Pro with belief.Source: §C.2 · p. 15, Table B2 · p. 15

C.2 Policy-Level Reflection without Belief; A Learned Instruction From Agent-Pro; A Similar Instruction From Agent-Pro w/o Belief

• With belief, learned guidance is more specific and concise; one example advises conservative play on a weak hand and folding if the flop does not improve it.The ablation’s corresponding instruction is described as vague and verbose.Source: §C.2 · p. 15, A Learned Instruction From Agent-Pro · p. 15, A Similar Instruction From Agent-Pro… · p. 15

C.3 Detailed Analysis Experiments

• The detailed analysis selects 20 challenging four-player hands with significant hand differences to compare Agent-Pro and Vanilla LLM.Table C4 identifies card suits and ranks, including T for 10.Source: §C.3 · p. 15, Table C4 · p. 16

D.1 LLM-based Application

• Unlike task-specific agents that rely on manually specified behavior protocols, Agent-Pro learns task understanding and behavioral strategies through environmental interaction.The paper describes this as enabling more effective task completion.Source: §D.1 · p. 16

D.2 LLMs For Interactive Scenarios

• Prior interactive-agent work spans reasoning-action-observation systems, generative agents, negotiation, experiential learning, and large-scale games.The cited examples include StarCraft, Minecraft, Leduc Hold’em, and strategy-based gaming.Source: §D.2 · p. 16

D.3 Improving the Quality of LLM Responses

• Unlike action-level self-correction strategies, Agent-Pro reflects at the policy level, while supervised fine-tuning depends on human-annotated data.The related-work discussion also covers reasoning architectures and output-refinement methods.Source: §D.3 · p. 17

D.4 Automatic Prompt Optimization

• Agent-Pro extends prompt optimization to interactive gaming through policy-level reflection and search, dynamically adjusting prompts to environmental changes and opponent styles.The cited prior approaches include prompt evaluation, gradient-like optimization, sequence-to-sequence translation, and reinforcement learning.Source: §D.4 · p. 17

D.5 Benchmarking LLM-based Agent

• The authors aim to evaluate Agent-Pro on broader agent benchmarks, including SmartPlay, Clembench, and AGENTBENCH.They describe these benchmarks as covering diverse games or environments.Source: §D.5 · p. 17

E Detailed Prompts

• Appendix E provides detailed prompt designs for the baselines and Agent-Pro across both games.The prompts are organized in Appendices E.1–E.4.Source: E Detailed Prompts · p. 17

F Case Study

• The case study reports improved rule understanding, technique mastery, and handling of uncertainty after learning across Blackjack and Texas Hold’em cases.The cases compare solving steps between Agent-Pro and ReAct.Source: F Case Study · p. 17

E.1 Baseline’s Prompts For Blackjack

• Blackjack baseline prompts range from vanilla and aggressive instructions to ReAct reasoning and Reflexion’s response-checking step.The prompts share game rules and current-card information.Source: §E.1 · p. 18

E.2 Agent-Pro’s Prompt For Blackjack

• The Blackjack Agent-Pro prompt adds behavioral guidelines, world modeling, and self-belief and world-belief before selecting an action.Its reflection prompt reviews belief correctness, consistency, and reasons for losing, then distills guidance and world modeling.Source: §E.2 · p. 19

E.3 Baseline’s Prompts For Limited Texas Hold’em

• Texas Hold’em baseline prompts include vanilla, aggressive, conservative, ReAct, and reflection variants over shared game information.Reflection asks the model to check and revise its thought and action.Source: §E.3 · p. 20

E.4 Agent-Pro’s Prompt For Limited Texas Hold’em

• The Texas Hold’em Agent-Pro prompt combines behavioral guidelines and world modeling of opponents with self-belief and world-belief before acting.Reflection checks belief correctness, consistency, and rationality against opponents’ intentions and the final result.Source: §E.4 · p. 21

F.1 Cases For Blackjack

• In the Qwen-72B Blackjack case, ReAct is described as too aggressive, while Agent-Pro guesses the dealer’s next move and makes a reasoned decision.The figure caption frames this as a comparison of their Blackjack cases.Source: Figure F1 · p. 22

F.2 Learning Process For Blackjack

• After interaction and learning, Agent-Pro’s initially ill-considered beliefs become more reasonable, alongside accurate world modeling and behavioral guidelines.The figure revisits the same card game before and after learning.Source: Figure F2 · p. 22
• ReAct is characterized as aggressive, while Agent-Pro is described as more rational in the Blackjack cases.The case captions also show Agent-Pro guessing the dealer’s next move and making a reasoned decision.Source: §F.2 · p. 23
• The cases contrast ReAct’s failure to consider bust risk and misunderstanding of “A” with Agent-Pro’s better rule understanding.The figures identify “A” as a rule-understanding issue and describe Agent-Pro as accurately understanding the rules.Source: Figure F3 · p. 23, Figure F4 · p. 23, §F.2 · pp. 23–24

F.3 Cases For Texas Hold’em

• With a King-high but weakly connected hand, Agent-Pro weighs an opponent’s repeated raises as strength or bluff, then raises to unsettle him; the opponent folds.Source: §F.3 · p. 25
• 2.5 points: Agent-Pro wins this hand while the baseline loses by 13.0 chip(s).Source: §F.3 · p. 25
• Facing a raise with King–2 of hearts, Agent-Pro treats the raiser’s strength as uncertain and calls to await the flop.Source: §F.3 · p. 26
• 50.0 points: Agent-Pro wins this hand while the vanilla LLM baseline loses by 2.0 points.Source: §F.3 · p. 26
• 50.5 points: Agent-Pro wins this hand while the vanilla LLM baseline loses by 1.0 points.Source: §F.3 · p. 27
• Agent-Pro loses 8.0 points in another hand, while the vanilla LLM baseline loses by 15.0 points.Source: §F.3 · p. 28