Source-linked AI summary
Solving Large Imperfect Information Games Using CFR+
Oskari Tammelin
TL;DR
Approximate Nash equilibrium methods for imperfect-information games such as poker are based on CFR and its variants, motivating faster algorithms. The paper introduces CFR+, which changes regret matching and empirically achieves more than an order of magnitude improvement in convergence time over CFR.
Problem
CFR and its variants are established approaches for solving imperfect-information games, motivating improved algorithms for approximate Nash equilibrium computation.
Method
CFR+ replaces CFR's regret-matching algorithm with regret-matching+ and uses a vector-form, alternating-update CFR design.
Results
More than an order of magnitude improvement in convergence time is reported for CFR+ compared with CFR.
Takeaways & Limitations
CFR+ can empirically converge to an approximate Nash equilibrium through its current strategy profile without requiring an averaging step.
Abstract
from arXiv · showhide
Counterfactual Regret Minimization and variants (e.g. Public Chance Sampling CFR and Pure CFR) have been known as the best approaches for creating approximate Nash equilibrium solutions for imperfect information games such as poker. This paper introduces CFR$^+$, a new algorithm that typically outperforms the previously known algorithms by an order of magnitude or more in terms of computation time while also potentially requiring less memory.
1 Introduction
CFR is a technique for solving extensive-form games with imperfect information, and CFR variants use regret-matching to choose strategies from positive cumulative regrets. The paper introduces a new CFR-like algorithm whose regret-matching variant is empirically faster by an order of magnitude or more.
- CFR solves extensive-form games with imperfect information using regret-matching-based strategies.Regret-matching chooses strategies proportional to positive cumulative regrets.
- The paper introduces a new CFR-like algorithm that replaces the prior regret-matching variant and is empirically faster by an order of magnitude or more.
2 Algorithm
The algorithm section positions CFR+ as an alternating, vector-form CFR variant that replaces CFR's regret matching with regret-matching+. Its current strategy profile empirically approaches an approximate Nash equilibrium without required averaging, while weighted averaging can accelerate convergence.
- 2 Algorithm: Vanilla CFR traverses the entire game tree at every iteration without sampling.
- 2 Algorithm: CFR+ is a vector-form algorithm with alternating regret updates.
- 2 Algorithm: CFR+ replaces CFR's regret-matching algorithm with regret-matching+ to produce new strategies from cumulative counterfactual regret+.
- 2 Algorithm: CFR+ empirically converges, or almost converges, to an approximate Nash equilibrium using its current strategy profile, so averaging is unnecessary.Weighted averaging remains available, with weights wT = max {T −d, 0}, where d is the averaging delay, and may accelerate convergence.
3 Compressibility
CFR+ produces many zero cumulative regret values, unlike CFR's indefinitely accumulating negative regrets, reducing data entropy and potentially enabling high compression ratios in large poker games.
- CFR+ has many zero cumulative regret values, whereas CFR's negative regrets continue accumulating indefinitely.
- Lower regret-data entropy can support high compression ratios using fixed-point arithmetic, prediction, context modeling, and arithmetic coding.The paper states that compression details are outside its scope and will be discussed later.
4 Results
The experiments compare CFR+ with vector-form vanilla CFR using alternating regret updates in one-card poker and a No Limit Texas Hold’em subgame. CFR+ average strategy converges more than an order of magnitude faster than CFR, although its current strategy converges slowly for small decks.
- 4 Results: The comparison uses vector-form vanilla CFR with alternating regret updates, matching the form used by CFR+.
- 4 Results: Figure 1 measures iterations needed to reduce exploitability below one millibets per hand across one-card poker deck sizes.
- 4 Results: CFR+ average strategy converges more than an order of magnitude faster than CFR.
- 4 Results: CFR+ current strategy converges slowly when the one-card poker deck size is small.
- 4 Results: In a No Limit Texas Hold’em subgame with a 5-milliblind exploitability target and three bet sizes, CFR+ shows a similar performance difference.
5 Conclusion
CFR+ was compared with CFR and showed more than an order-of-magnitude improvement in convergence time.
- More than an order of magnitude improvement in convergence time was observed for CFR+ compared with CFR.