PaperlayerView exact revision on arXiv

Source-linked AI summary

Monopoly Deal: A Benchmark Environment for Bounded One-Sided Response Games

Will Wolf

arXiv:2510.25080v2cs.GTcs.AIcs.LG

TL;DR

Well-studied game-interaction patterns do not fully represent the range of structures relevant to decision-making under uncertainty. This paper formalizes bounded one-sided response in a modified Monopoly Deal environment and applies CFR. The agent achieves competitive play against strong baselines within 1,000 games, about 19 minutes of training.

  • Problem

    Well-studied game-interaction patterns do not fully represent the range of structures relevant to decisions under uncertainty.

    Source: §1 · pp. 1–2

  • Method

    The paper formalizes BORGs in a modified Monopoly Deal environment and applies CFR with a compact representation that accommodates response phases.

    Source: §1 · p. 2, §9 · p. 17

  • Results

    Within 1,000 games, approximately 19 minutes of training, the agent achieves competitive play against strong baselines.

    Source: §8 · p. 17

  • Takeaways & Limitations

    The ϱ-operator integrates finite, turn-interrupting interactions into the game state without requiring modifications to standard CFR.

    Source: §8 · p. 17

    Response-phase action order does not affect the outcome, so the current task is equivalent to selecting a combination of cards rather than modeling sequential decisions within that phase.

    Source: §8 · p. 17

Abstract

from arXiv · show

Card games are widely used to study sequential decision-making under uncertainty, with real-world analogues in negotiation, finance, and cybersecurity. These games typically fall into three categories based on the flow of control: strictly sequential (players alternate single actions), deterministic response (some actions trigger a fixed outcome), and unbounded reciprocal response (alternating counterplays are permitted). A less-explored but strategically rich structure is the bounded one-sided response, where a player's action briefly transfers control to the opponent, who must satisfy a fixed condition through one or more moves before the turn resolves. We term games featuring this mechanism Bounded One-Sided Response Games (BORGs). We introduce a modified version of Monopoly Deal as a benchmark environment that isolates this dynamic, where a Rent action forces the opponent to choose payment assets. The gold-standard algorithm, Counterfactual Regret Minimization (CFR), converges on effective strategies without novel algorithmic extensions. A lightweight full-stack research platform unifies the environment, a parallelized CFR runtime, and a human-playable web interface. The trained CFR agent and source code are available at https://monopolydeal.ai.

1 Introduction

The introduction identifies a gap in familiar game-interaction models and presents modified Monopoly Deal as a benchmark for bounded one-sided response. It applies CFR and provides a lightweight platform for studying this setting.

1 Introduction

  • Existing game-interaction categories do not capture a common structure in which an opponent makes a bounded, non-reciprocal sequence of moves before play resumes.The paper names games with this structure Bounded One-Sided Response Games (BORGs).Source: §1 · p. 2
  • Modified Monopoly Deal isolates the BORG dynamic: playing Rent transfers control to an opponent who must satisfy the debt before the turn concludes.The response may involve selecting Cash or Property cards or canceling the demand with Just Say No.Source: §1 · p. 2
  • CFR applies directly to this environment and converges reliably without novel algorithmic extensions.The implementation uses a compact state representation that accommodates response phases.Source: §1 · p. 2
  • The full-stack platform runs on a single workstation and supports training, policy inspection, and interaction with learned models.Its stated aim is to lower the barrier to studying BORGs.Source: §1 · p. 2
  • The work formalizes a reproducible BORG benchmark, demonstrates direct CFR applicability, and presents a lightweight research platform.The platform combines the game environment, parallelized CFR runtime, and web interface.Source: §1 · p. 3

2 Background and Notation

The section sets out an extensive-form, imperfect-information game framework, introduces a bounded-response operator, and reviews CFR and its Monte Carlo variants.

2 Background and Notation

  • The game is modeled as a two-player, zero-sum imperfect-information extensive-form game, with information sets grouping states indistinguishable from a player's private observations.The notation defines reachable states, terminal states, available actions, and player utilities.Source: §2 · p. 3

Bounded One-Sided Response Operator

  • The bounded-response operator ϱ gives the opponent exclusive control in a finite response subgraph until a stopping condition, then deterministically concludes the phase and returns play.This non-reciprocal subphase generalizes strictly sequential play and differs from fixed outcomes or arbitrarily deep reciprocal counterplay.Source: Bounded One-Sided Response Operator · pp. 3–4

Counterfactual Regret Minimization

  • CFR accumulates action regrets and uses regret matching; minimizing maximum expected regret makes the average strategy approach a Nash equilibrium as iterations increase.The section defines counterfactual reach probability using the action probabilities of other players and chance along the path to a state.Source: Counterfactual Regret Minimization · p. 4
  • Monte Carlo CFR uses sampling to estimate utilities and regrets while matching full-traversal CFR updates in expectation.The implementation uses external sampling with action-based rollouts and omits counterfactual reach probability from instantaneous regret calculations.Source: Counterfactual Regret Minimization · p. 5

3 Modified Monopoly Deal Environment

The environment is a simplified two-player Monopoly Deal variant designed to preserve strategic structure while isolating bounded one-sided responses to Rent.

3 Modified Monopoly Deal Environment

  • Players seek to complete property sets by acquiring properties, collecting rent, and managing cash under imperfect information.Winning requires completing a fixed number of sets; an exhausted deck otherwise ends the game in a draw.Source: §3 · pp. 5–6
  • Rent cards charge amounts based on owned properties of the matching color and trigger a response phase when the player owns at least one.The Green rent progression is $2, $4, and $7 for one, two, and three owned properties, respectively.Source: §3 · p. 5
  • During a Rent response, only the opponent acts, paying with cash or property, canceling with Just Say No, or yielding if unable to pay.Play resumes deterministically once rent is settled or blocked, and the implementation disallows the initiating player's counter-cancel of Just Say No.Source: §3 · p. 6
  • The variant reduces property types, removes most action cards, limits cash denominations, and removes the original hand-discard rule to keep the testbed tractable.These simplifications are described as preserving strategic depth while isolating the BORG dynamic.Source: §3 · p. 6

4 Related Work

Related work spans Monopoly Deal agents, game benchmarks, scalable algorithms, and reproducible research platforms; this paper adds a BORG domain and studies it with tabular CFR.

4 Related Work

  • Earlier Monopoly Deal agents used heuristic-driven personalities, hand-coded priorities, and graph search, whereas this work derives a policy through CFR and intent-based abstraction.Similar heuristic agents serve as baseline opponents.Source: §4 · p. 6
  • OpenSpiel, RLCard, PettingZoo, and Hanabi Learning Environment support broader game experimentation, while this work adds modified Monopoly Deal as a BORG benchmark.The cited platforms emphasize modularity, standardized interfaces, and reproducibility.Source: §4 · pp. 6–7
  • Where complex-game systems use function approximation and large-scale infrastructure, this work uses classic tabular CFR in a modest abstracted state space.The platform follows a local-execution tradition, supporting transparent experimentation on a single workstation.Source: §4 · p. 7

5 System Design and Architecture

The system combines a simplified Monopoly Deal environment, CFR training, and human–model interaction in a reproducible research platform. Its design targets fast convergence and single-workstation experimentation while preserving tools for inspection and replay.

5.1 Design Goals

  • Stable CFR convergence is achieved within twenty minutes on a single workstation through compact state abstraction and an efficient parallel runtime.The design combines a transparent environment, parallelized runtime, and accessible interface for high-fidelity experimentation.Source: §5.1 · pp. 7–8

5.2 Game Environment

  • The two-player game uses a reduced card set and restricted property types and set requirements to keep information sets and lookup tables manageable.This simplified setting focuses on the bounded response dynamic and keeps regret and policy tables within single-workstation RAM.Source: §5.2 · p. 8

5.3 Architecture Overview

  • Training self-play exports JSON checkpoints to a serving stack that loads policies for human–model interaction through a web interface.The training stack logs metrics and checkpoints; the serving stack uses FastAPI, a database, and a Next.js frontend.Source: §5.3 · p. 8, Figure 1 · p. 9

5.4 Data Model and State Representation

  • Immutable information-set states separate the acting player’s private hand from the opponent’s public holdings and track bounded-response context and legal actions.Deterministic serialization and hashing support consistent indexing and checkpoint recovery.Source: §5.4 · pp. 9–10

5.5 Runtime and Reproducibility Model

  • Three runtime modes trade determinism against speed, while seeded trajectories, checkpointing, and logged metrics support reproducibility and resumption.Unordered parallel updates are fastest but nondeterministic; ordered batch updates are deterministic but increase convergence time.Source: §5.5 · pp. 10–11

5.6 Human Interaction and Policy Evaluation

  • The web interface supports human play, policy-probability and state inspection, and exact reconstruction of logged games using the training engine.The integrated platform supports training, evaluation, visualization, and examination in one environment.Source: §5.6 · p. 11

6 Methods

The methods adapt tabular Monte Carlo CFR to the compact Monopoly Deal environment using action-based rollouts and an intent-based abstraction. Practical choices reduce variance, stabilize action selection, and provide an averaged policy for deployment.

6.1 Monte Carlo CFR with Variance Reduction

  • The intent-based abstraction maps each information set to its available abstract actions and streak index, yielding roughly one hundred information sets in practice.It omits quantitative features such as hand size, property counts, and opponent statistics.Source: §6.1 · p. 12
  • Response states are distinguished by their restricted valid-action sets, allowing CFR to handle BORG phases without modifying the algorithm.The implementation uses a custom non-recursive, action-based rollout strategy with compact state abstraction.Source: §6.1 · p. 12
  • At each visited information set, the learner estimates each action’s counterfactual value by averaging sampled terminal outcomes after that action and subsequent policy play.Rollouts enumerate available actions and single-sample subsequent actions for N trajectories.Source: §6.1 · p. 12
  • The regret estimator omits counterfactual reach-probability weighting, introducing small bias relative to exact MCCFR while reducing variance and retaining competitive, fast-converging policies.Regret matching updates cumulative regret from estimated action values.Source: §6.1 · p. 12
  • When a card-playing action has positive accumulated regret, clamping non-card actions to nonpositive regret restricts passive delay unless it is the only positively weighted option.Exploration samples actions ε-greedily with ε = 0.1; deployment uses a uniform average over a configurable buffer of recent policies, default N = 10.Source: §6.1 · p. 13

7 Experiments

The experiments examine self-play convergence, baseline win rates, information-set update counts, and learned action probabilities in the modified Monopoly Deal environment. Training reaches a low-regret state within 1,000 games, with strong results against both tested baselines.

7 Experiments

  • The self-play experiments assess convergence speed, policy stability, and interpretability under the intent-based abstraction.The evaluation uses the modified Monopoly Deal environment.Source: §7 · p. 13

7.1 Experimental Setup

  • On a single Apple M1 workstation with 8 logical cores and 16 GB RAM, training converged in approximately 19 minutes.Training used Monte Carlo CFR with action-based rollouts, a fixed ϵ-greedy exploration rate, and games capped at 250 turns.Source: §7 · p. 13, §7.1 · p. 13

7.2 Regret Convergence

  • Maximum expected regret declines early and reaches a relatively stable, low-regret state within 1,000 games.This corresponds to 19 minutes of wall-clock training time.Source: §7.2 · p. 14, Figure 2 · p. 15

7.3 Win Rate Evaluation

  • The agent reaches nearly 100% win rate against RandomSelector and roughly 75% against RiskAwareSelector.Figure 3 reports these outcomes under agent-first and randomized-start conditions.Source: Figure 3 · p. 15, §7.3 · p. 15

7.4 Update Dynamics

  • Information-set update counts range from a median of roughly 50 to a maximum of approximately 2,000.Figure 4 presents the update-count quantiles on a logarithmic scale.Source: Figure 4 · p. 16, §7.4 · p. 16

7.5 Policy Evolution

  • During response phases, the policy favors JustSayNo and giving cash over giving property; in normal play, it favors adding to or completing property sets over cash.Figure 5 tracks median probabilities of abstract actions across relevant information sets.Source: §7.5 · p. 16, Figure 5 · p. 17

8 Discussion

The experiments support deriving a practical, high-performing policy from a minimal intent-based abstraction using Monte Carlo CFR. The discussion identifies the response phase’s order-insensitive multi-set structure as a limitation to address for sequential decision-making.

8 Discussion

  • Within 1,000 games (approximately 19 minutes), the agent achieves competitive play against strong baselines.The authors interpret this as validating the abstraction’s soundness and the training runtime’s efficiency.Source: §8 · p. 17
  • The ϱ-operator formalizes finite, turn-interrupting interactions within the game state without modifying standard CFR.The authors position this structure as an intermediate testbed between purely sequential and fully reciprocal, recursive games.Source: §8 · p. 17
  • Response-phase actions form an order-insensitive multi-set, so the current task selects card combinations rather than modeling sequential choices within that phase.The authors identify modeling that sequence as necessary to realize a sequential decision process within the response phase.Source: §8 · p. 17

9 Conclusion

The paper formalizes BORGs through a modified Monopoly Deal system and reports that classical Monte Carlo CFR applies directly and converges reliably on modest hardware. It also presents a research platform and outlines extensions to response-phase modeling, policy learning, and training scale.

9 Conclusion

  • Classical Monte Carlo CFR applies directly to the BORG setting and converges reliably on modest hardware.The platform integrates the environment, parallelized CFR runtime, and web interface to support reproducibility and introspection.Source: §9 · pp. 17–18
  • The platform is intended as an accessible, transparent framework for reproducible BORG research and future extensions.The conclusion identifies broader platform utility as the intended outcome of the proposed extensions.Source: §9 · p. 18
  • Future work targets sequential dependencies in response phases, more granular policies using modern reinforcement learning, and distributed training as policy complexity grows.The current response phase uses a multi-set decision structure; larger state spaces may require moving beyond tabular methods and potentially removing intent-based abstraction.Source: §9 · p. 18

A Information Set Key Structure and Serialization

  • An InfoSet key deterministically identifies each decision point by serializing an abstract state as canonical JSON and hashing it with MD5.The key combines the player index, abstraction class name, and abstraction key, supporting consistent state recovery and policy lookup.Source: A Information Set Key Structure and… · pp. 20–21

B Baseline Opponent Policies

The evaluation compares the CFR agent with two fixed, non-learning policies under agent-first and alternating-start conditions. The baselines range from uniform random action selection to a heuristic balancing property acquisition and cash caution.

B Baseline Opponent Policies

  • RandomSelector samples uniformly from all legal abstract actions at each information set, serving as the statistical baseline.Source: B Baseline Opponent Policies · p. 21
  • RiskAwareSelector balances a fixed aggressiveness parameter by favoring property acquisition over banking cash, with temperature controlling preference sensitivity.The passage characterizes this fixed strategy as more competitive than pure random play.Source: B Baseline Opponent Policies · p. 21
  • The CFR agent is evaluated against both fixed baselines when it always goes first and when starting order alternates.Both baselines use the same abstract action space and deterministic GreedyActionResolver.Source: B Baseline Opponent Policies · p. 21

C Checkpoint Structure

The checkpoint stores configuration metadata and three components of CFR learner state in a human-readable JSON snapshot. The accompanying web interface lets researchers play against trained CFR policies while viewing game state and action history.

C Checkpoint Structure

  • A human-readable checkpoint supports resuming or replaying an experiment from a saved index.The complete learner state is serialized after a configurable number of games.Source: C Checkpoint Structure · p. 21
  • The JSON checkpoint captures game configuration and metadata, policy vectors, regrets and visit counts, and counterfactual reach probabilities.The reach probabilities are currently used to compute maximum expected regret during training.Source: C Checkpoint Structure · pp. 21–22

D Interactive Web Interface

  • The interactive web interface renders game state and action history in real time and lets a human compete against the trained CFR agent.Figure 7 depicts the interface.Source: Figure 7 · p. 24
Loading 2510.25080v2…