Source-linked AI summary
Learning Market Competition in Shared Spectrum: A Multi-Agent Reinforcement Learning Approach
Qixuan Zai, Randall Berry
TL;DR
Shared-spectrum market models typically assume providers know demand, while MARL studies of tacit collusion have largely omitted congestion. This paper models providers learning unknown demand under Bertrand and Cournot competition; it reports that Cournot approaches theoretical Nash equilibria while Bertrand fosters tacit collusion.
Problem
Prior shared-spectrum market models generally assume known demand, while MARL collusion research has focused largely on markets without congestion.
Source: §I · pp. 1–2Method
The paper models Bertrand and Cournot competition over shared spectrum as MARL problems in which providers learn pricing or quantity strategies through repeated interaction.
Source: §I · p. 2Results
Cournot competition reliably converges toward theoretical Nash equilibria, whereas Bertrand competition bypasses the classical zero-price paradox and fosters tacit algorithmic collusion.
Source: §V · p. 6Takeaways & Limitations
Congestion can strengthen or weaken collusive behavior depending on the competition model and learning algorithm, informing the study of wireless market design and learning-enabled decision-making.
Source: §I · p. 2
Abstract
from arXiv · showhide
This paper investigates market competition among wireless service providers (SPs) that serve customers using shared spectrum. Prior work has analyzed such markets through models of competition with congestible resources, capturing both the congestion-sensitive nature of wireless spectrum and the effects of spectrum sharing on service quality. These models typically assume that the market demand function is known, enabling SPs to optimize pricing or quantity decisions under either Bertrand or Cournot competition. In contrast, we consider a setting in which the demand function is initially unknown and must be learned over time. We model this learning process using multi-agent reinforcement learning (MARL), allowing competing SPs to learn market dynamics while adapting their competitive strategies. Although MARL has shown strong performance in a variety of economic settings, recent work has demonstrated that it can also give rise to tacit collusion among self-interested agents. We therefore examine whether similar collusive behavior emerges in shared-spectrum markets and how its prevalence depends on the mode of competition (price versus quantity) and the choice of MARL algorithm. Our results provide insight into the interaction between learning dynamics, market structure, and spectrum sharing, with implications for both wireless market design and the deployment of learning-enabled decision-making systems.
I. INTRODUCTION
Shared-spectrum markets couple provider competition with congestion, while prior models generally assume providers know demand. This paper uses multi-agent reinforcement learning to study how congestion, competition mode, and learning algorithms relate to tacit collusion.
I. INTRODUCTION
- Existing shared-spectrum models capture congestion-sensitive service quality and have studied equilibrium prices, market shares, profits, consumer surplus, and welfare under price and quantity competition.These models generally assume that market demand is known to providers.Source: §I · p. 1
- The study asks how congestion-induced interactions in shared-spectrum markets affect tacit collusion and market outcomes when providers learn under unknown demand.Prior work largely studied learning-driven collusion without congestion, while spectrum-market analyses generally assumed known demand.Source: §I · pp. 1–2
- The paper formulates repeated Bertrand price competition and Cournot quantity competition over shared spectrum as multi-agent reinforcement-learning problems.Agents adapt strategies through repeated interaction without being given the demand model.Source: §I · p. 2
- The paper compares MARL algorithms to characterize differences in convergence, market efficiency, and tacit collusion.The stated implications concern wireless market design and learning-enabled decision-making systems.Source: §I · p. 2
- Congestion can strengthen or weaken collusive behavior depending on the competition model and learning algorithm.The paper argues that congestion changes the learning dynamics and strategic behavior found in conventional MARL pricing models.Source: §I · p. 2
II. SYSTEM MODEL
The model studies an incumbent with licensed spectrum and entrants sharing a common band, with customers responding to delivered prices that include congestion. Providers do not know demand and learn through observed market outcomes.
II. SYSTEM MODEL
- The market has one incumbent with licensed bandwidth and N entrants sharing a common spectrum band; the one-incumbent setup is chosen for simplicity.The authors note that the model can be extended to multiple incumbents.Source: §II · p. 2
- Customers face delivered prices combining each provider’s service price with congestion costs, and active providers’ delivered prices satisfy Wardrop equilibrium conditions.The model assumes linear congestion costs and characterizes demand through an unknown inverse demand function.Source: §II · pp. 2–3
- Providers observe market outcomes such as revenue but do not observe the underlying demand function or competitors’ private decisions.A linear inverse demand function is used for analytical benchmarks, not revealed to learning agents.Source: §II · p. 3
- The framework evaluates both Cournot competition, where providers choose quantities, and Bertrand competition, where they choose prices.Both modes use the same shared-spectrum market framework.Source: §II · pp. 2–3
A. Cournot Congestion Competition
In the Cournot model, providers choose quantities and market prices follow from Wardrop equilibrium. The known-demand congestion game supplies a theoretical benchmark, while MARL agents learn quantity strategies without knowing demand.
A. Cournot Congestion Competition
- Under Cournot competition, providers choose user quantities, and prices are determined endogenously by Wardrop equilibrium and congestion across licensed and shared bands.The model represents providers competing by controlling market share rather than directly setting prices.Source: A. Cournot Congestion Competition · p. 3
- With complete information, the Cournot congestion game is a potential game with a pure-strategy Nash equilibrium and favorable best-response convergence properties.The paper instead studies incomplete information, where providers repeatedly observe outcomes and learn quantity strategies using MARL.Source: A. Cournot Congestion Competition · p. 3
- The complete-information equilibrium is the benchmark for learned policies, derived using linear inverse demand and the linear congestion model.The learning agents are not given the underlying demand function.Source: A. Cournot Congestion Competition · p. 3
B. Bertrand Competition
In the Bertrand model, providers choose prices and customer allocation follows Wardrop equilibrium. The paper uses the complete-information equilibrium as a benchmark for assessing learned pricing and collusion.
B. Bertrand Competition
- Under Bertrand competition, providers choose service prices while Wardrop equilibrium determines customer allocation through prices, congestion costs, and total demand.Customers select the provider offering the lowest delivered price.Source: B. Bertrand Competition · p. 3
- Unlike the classical prediction that homogeneous-product prices fall to marginal cost,, reinforcement-learning agents may learn tacitly collusive prices without communication,.The paper examines how congestion and spectrum sharing influence this learned pricing behavior.Source: B. Bertrand Competition · p. 3
- The complete-information equilibrium is the analytical benchmark, and persistent prices above the competitive equilibrium indicate tacit collusion.The benchmark uses linear inverse demand and linear congestion costs; comparison with Cournot is used to assess congestion’s influence on pricing and efficiency.Source: B. Bertrand Competition · p. 3
C. Analytical Benchmarks
The analytical benchmarks summarize complete-information Nash outcomes under linear demand and congestion, alongside a perfectly coordinated revenue-maximizing outcome. The Price of Collusion compares competitive and collusive welfare, while a stated lower bound depends on entrant bandwidth.
C. Analytical Benchmarks
- The complete-information Nash benchmarks assume linear inverse demand and linear congestion costs, with all service providers serving customers.Cases where some providers do not serve customers are omitted for simplicity.Source: C. Analytical Benchmarks · p. 3
- The collusive benchmark maximizes total revenue across firms and represents perfect coordination, not a prediction of rational strategic behavior.It serves as an upper bound for evaluating tacit collusion learned by MARL agents.Source: C. Analytical Benchmarks · p. 4
- The Price of Collusion is competitive-Nash social welfare divided by collusive-equilibrium welfare; values above 1 quantify proportional consumer-welfare degradation.It is introduced to quantify deviations from the competitive benchmark.Source: C. Analytical Benchmarks · p. 4
- Under the Bertrand congestion model with entrants only, the stated lower bound for the Price of Collusion lies between 0 and 1, depending on entrant bandwidth.Source: C. Analytical Benchmarks · p. 4
D. Revenue Expectations Under Learning
Learning makes prices and quantities stochastic, so revenue generally cannot be factored into expected price times expected quantity. The section establishes conditions for that identity under Cournot and Bertrand competition and identifies profiles where it fails.
D. Revenue Expectations Under Learning
- Learning-induced correlation between stochastic prices and quantities generally prevents expected revenue from equaling expected price times expected quantity.The section uses repeated learning trajectories to interpret empirical market outcomes.Source: D. Revenue Expectations Under Learning · p. 4
- Under Cournot competition, the revenue identity requires fixed incumbent supply or, for entrants, independent incumbent choices and zero entrant supply variance.The theorem assumes active firms maintain nonzero prices and allocations; the benchmark uses a continuous action space, with possible slight deviations in discrete settings.Source: D. Revenue Expectations Under Learning · p. 4
- Under Bertrand competition, the identity holds when all entrants maintain the minimum market price and remain active.Source: D. Revenue Expectations Under Learning · p. 4
- When some entrants announce prices above the minimum, the identity breaks down because some providers serve no customers.Source: D. Revenue Expectations Under Learning · p. 4
III. MULTI-AGENT REINFORCEMENT LEARNING
The paper models repeated shared-spectrum competition as a Markov game in which independent providers learn through market interaction without knowing demand. Agents receive revenue rewards, and DQN is the primary learning method, with actor–critic used for comparison.
III. MULTI-AGENT REINFORCEMENT LEARNING
- Providers repeatedly choose prices or quantities, observe market outcomes, receive realized revenue as reward, and update policies without knowing the demand function.Allocated demand is determined by the Wardrop equilibrium.Source: §III · p. 4
- Each agent observes the joint actions from the previous K rounds and chooses from discrete prices or quantities.Source: §III · p. 4
- Independent DQN agents use their own value networks and replay buffers, while an actor–critic method provides an algorithm comparison.Source: §III · pp. 4–5
- Training uses ϵ-greedy actions, stores transitions after market clearing, updates networks from mini-batches, and continues until behavior converges.Source: §III · p. 5
A. Neural Network Architecture
The implementation uses a shared two-hidden-layer network architecture across the markets and describes an independent multi-agent DQN procedure for learning in congested markets.
A. Neural Network Architecture
- Each agent uses an independent multilayer perceptron with two hidden layers of 64 neurons, with input size set by history length and output size by the discrete action count.Experiments use the same architecture and hyperparameters across both markets unless stated otherwise.Source: A. Neural Network Architecture · p. 5
B. Learning in Market with congestion
- The congestion-market DQN procedure initializes agent networks and replay buffers, selects actions with ϵ-greedy exploration, and updates networks from sampled transitions.Agents act simultaneously, receive revenues after market execution, and update target networks periodically.Source: B. Learning in Market with congestion · p. 5
A. Benchmark Validation
The benchmark shows that collusion’s welfare effect depends on price: lower collusive prices can reduce congestion, while higher prices reduce participation and increase efficiency loss. More history is associated with higher learned Bertrand prices, while entrant behavior differs under Bertrand and Cournot competition.
A. Benchmark Validation
- Low collusive prices keep allocation near the competitive equilibrium and reduce congestion, whereas higher prices cut participation and increase efficiency loss.The number of entrant providers has essentially no effect on the Price of Collusion (PoC).Source: A. Benchmark Validation · p. 5, Figure 1 · p. 5
- The PoC captures both the existence of collusion and its impact on overall market efficiency, providing a benchmark for interpreting learning results.Source: A. Benchmark Validation · p. 5
B. Impact of State Representation
- With K = 0, agents converge to the competitive Nash equilibrium; as history length increases, they learn higher prices corresponding to tacit collusion.The K = 0 setting effectively reduces the learning problem to a contextual bandit.Source: B. Impact of State Representation · p. 6
C. Impact of Market Structure
- The incumbent approaches the theoretical Nash equilibrium under both competition models, but entrant behavior differs: Cournot stays near equilibrium while Bertrand produces more collusive prices.The text attributes this difference partly to Cournot’s potential-game structure and the absence of the same convergence guarantees for Bertrand.Source: C. Impact of Market Structure · p. 6
D. Impact of Learning Algorithm
The two MARL methods reach similar market outcomes, although actor–critic learning is smoother and more stable than DQN. In the entrant-only Bertrand setting, the learned equilibrium has a PoC of approximately 0.886, with relatively low prices and reduced congestion.
D. Impact of Learning Algorithm
- Both MARL algorithms converge to similar market outcomes, while actor–critic has smoother and more stable learning trajectories than DQN.The authors suggest that tacit collusion is driven more by market structure than by the specific learning algorithm.Source: D. Impact of Learning Algorithm · p. 6
E. Impact of the Incumbent
- Empirical average realized revenue equals the product of empirical average price and average quantity under the conditions established by Theorem 2.Source: E. Impact of the Incumbent · p. 6
- With only entrant providers under Bertrand competition, the learned equilibrium has a PoC of approximately 0.886, slightly improving social welfare relative to the competitive benchmark.The authors attribute this to relatively low learned collusive prices that reduce congestion.Source: E. Impact of the Incumbent · p. 6
- The experiments validate the theoretical revenue identities and illustrate that market composition influences the emergence of collusive behavior.Source: E. Impact of the Incumbent · p. 6
F. Impact of Congestion
Learned Cournot strategies remain close to the Nash equilibrium, whereas learned Bertrand strategies lie between the Nash and collusive benchmarks, indicating partial tacit collusion. The conclusion emphasizes that congestion can reinforce competitive behavior or facilitate collusion depending on the competition model.
F. Impact of Congestion
- Learned Cournot strategies remain close to the Nash equilibrium, while Bertrand strategies fall between the Nash and collusive benchmarks, indicating partial tacit collusion.The comparison is with theoretical benchmarks in the absence of congestion.Source: F. Impact of Congestion · p. 6, Table II · p. 7, Table III · p. 7
V. CONCLUSION
- Congestion can reinforce competitive behavior or facilitate tacit collusion, depending on the underlying competition model.The conclusion states that Cournot reliably converges toward theoretical Nash equilibria, while Bertrand fosters tacit algorithmic collusion.Source: F. Impact of Congestion · p. 6, §V · p. 6