Source-linked AI summary
Foundation Model-Aided Multi-Agent Reinforcement Learning for Wireless Random Access Network Optimization
Myeung Suk Oh, Zhiyao Zhang, Alvaro Velasquez, Nathaniel D. Bastian, Jia Liu
TL;DR
Random-access MARL can require substantial task-specific training, while centralized training raises deployment and communication concerns. This paper uses a pretrained foundation model in a consensus-based decentralized actor–critic framework, reporting faster learning across RA tasks, with at least 55% faster convergence than both baselines. The analysis concerns a saturated-traffic setting.
Problem
RA optimization must manage contention and collisions, while MARL policy learning incurs task-specific training overhead and centralized approaches have deployment and communication limitations.
Source: §1 · pp. 1–2Method
A pretrained, reward-agnostic foundation model serves as the critic backbone in a consensus-based, decentralized actor–critic MARL framework.
Source: §1 · p. 2Results
Across all tasks, FM-aided learning improves convergence speed by at least 55% against end-to-end MARL and conventional decentralized MARL.
Source: §5.2 · p. 8Takeaways & Limitations
The approach achieves faster convergence while optimizing throughput and AoI across diverse objectives, according to the authors.
Source: §6 · p. 9The modeled network assumes saturated traffic, with packet arrivals sufficient to keep all devices supplied.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Random access (RA) is one of the most foundational medium access control (MAC) layer scheduling schemes for handling unpredictable data traffic from multiple terminals. While multi-agent reinforcement learning (MARL) has been explored to optimize RA-based wireless networks, its reliance on experience-driven, distributed policy learning incurs significant training overhead for each optimization task, limiting its feasibility in real-world applications. In this work, we propose to leverage a foundation model (FM) to improve MARL efficiency across diverse RA network optimization tasks. Specifically, we design an FM-aided actor-critic algorithm within a consensus-based decentralized MARL architecture and provide its convergence analysis under local reward exchanges and nonlinear value function approximations to show that our algorithm achieves the same convergence order as the conventional MARL with critic model exchanges and linear approximations. Our numerical results show that our FM-based approach significantly enhances MARL speed for RA network optimization.
1 Introduction
Random access networks need efficient contention control, while existing optimization methods face performance-guarantee, centralization, and task-adaptation challenges. This work proposes a fully decentralized, consensus-based FM-aided MARL framework and reports faster learning across multiple RA tasks.
Background and Motivations
- Existing RA approaches either use heuristics without performance guarantees or idealized analytical models that may not capture real-world complexity.Excessive contention also wastes communication resources and diminishes decentralized access benefits.Source: §1 · p. 1
Foundation Model-Aided Multi-Agent Reinforcement Learning
- Centralized-training MARL depends on a central entity, scales poorly in communication, and can introduce single points of failure and privacy or security concerns.Conventional MARL can also overfit to specific objectives, requiring retraining or architectural changes when metrics change.Source: §1 · p. 2
Our Approach and Contributions
- The framework combines decentralized training and execution with average consensus and a pretrained, reward-agnostic foundation-model backbone for the critic.A task-specific head uses the backbone’s self-supervised system-dynamics representations to support online policy learning.Source: §1 · p. 2
- The proposed framework provides finite-time convergence analysis for consensus with local reward exchanges and nonlinear state-value approximation.The authors state it achieves the same convergence order as conventional MARL with critic exchanges and linear approximations.Source: §1 · p. 2
- Numerical simulations show faster learning than end-to-end MARL across multiple random-access optimization tasks.The FM is pretrained on reward-independent data and used as the critic’s reward-agnostic backbone.Source: §1 · p. 2
2 Related Work
Prior RA optimization spans heuristics and learning-based approaches, with MARL methods often retaining centralized training and task-specific designs. This paper positions a reward-agnostic FM backbone in a decentralized actor–critic framework to address those limitations.
Traditional RA Network Optimization
- Traditional sensing-free ALOHA is simple but collision-prone, while sensing-based CSMA/CA uses random backoff yet can incur delays and suboptimal channel use.Theoretical guarantees for these heuristic approaches remain unclear, while queue-length-based CSMA results often rely on idealized models.Source: §2 · p. 2
Learning-driven RA Network Optimization
- Early RL work tunes CSMA/CA parameters, while MARL methods use approaches such as deep Q-networks, federated policy training, QMIX, and MAPPO [43].These MARL approaches rely on centralized training and decentralized execution, and are often task-specific, with objective changes requiring modification and retraining.Source: §2 · p. 2
Foundation Models for Wireless Networks
- Foundation models have been explored for wireless PHY and MAC tasks, including multi-task prediction such as channel estimation and traffic forecasting [9] [14] [38] [37] [31].This paper instead uses an FM as a reward-agnostic backbone within an actor–critic framework for diverse RA objectives without centralized coordination.Source: §2 · pp. 2–3
3 System Model and Problem Formulation
The system model describes contending devices, their channel and transmission behavior, and three distinct optimization objectives. Its operating assumptions include saturated traffic, slotted access, and local device communication.
3.1 Network System and the RA Protocol
- The network has K single-antenna devices with queued packets transmitting to an N_r-antenna access point under saturated traffic.Packets are served first-come, first-served, and arrivals are sufficient to keep every device supplied with data.Source: §3.1 · p. 3
- Devices use slotted listen-before-talk access: after clear-channel assessment, each chooses whether to transmit a packet over τ slots.Successful reception is acknowledged; a collision yields no acknowledgment and involved devices wait for another opportunity to retransmit.Source: §3.1 · p. 3
- Decentralized operation assumes no global controller; devices exchange information with neighbors over links in an undirected graph.The network channel gain combines distance-dependent large-scale fading with an i.i.d. complex Gaussian small-scale fading vector.Source: §3.1 · p. 3, §3.2 · p. 3
3.2 Data Transmission Model
- The data-transmission model uses a SIMO channel gain with small-scale fading and a distance-dependent large-scale fading factor.The model can use indoor ITU-R or urban-micro 3GPP pathloss scenarios.Source: §3.2 · p. 3
- With block fading, the channel remains static during each packet transmission, and the model derives successfully transmitted data from the achievable rate.The rate expression uses bandwidth, device transmit power, and noise spectral density.Source: §3.2 · p. 3
- The amount of data m_k,t is proportional to the device’s rate and informs transmission decisions, since a low-data transmission can block devices with better channel conditions.Devices therefore consider channel quality when choosing whether to transmit.Source: §3.2 · p. 3
3.3 RA Network Optimization Tasks
- The fair-AoI task minimizes worst-case information age, defined by elapsed slots since each device’s latest successful transmission.Its minimax objective focuses on fairness in transmission frequency, not channel quality or data rates.Source: §3.3 · p. 3
- The max-rate task maximizes total throughput subject to a minimum rate requirement for each device, while fair-rate maximizes the minimum device throughput.The former guards against high-rate devices dominating access; the latter prevents one device’s poor throughput from becoming a bottleneck.Source: §3.3 · pp. 3–4
4 Foundation Model-Aided Decentralized MARL for Wireless RA Network Optimization
The section formulates wireless random access optimization as decentralized MARL and develops a reward-agnostic foundation-model critic within a consensus-based actor–critic framework. It analyzes nonlinear value approximation with local reward exchange and reports convergence to a neighborhood of the global optimum.
4.2 MDP Formulation for MARL-based Wireless RA Network Optimization
- The RA problem is modeled as an MDP with global states, device-specific actions, shared transitions, and locally computable rewards.Devices choose whether to wait or transmit; the reward formulations address fair AoI, maximum rate, and fair rate.Source: §4.1 · p. 4, §4.2 · p. 4
4.1 Actor-Critic Framework for MARL Problem
- The actor–critic framework optimizes decentralized device policies for expected long-term discounted reward, using critic value estimates to form the actor’s advantage-based update.The MDP advances at decision time slots, when devices take actions.Source: §4.1 · p. 4
4.3 Foundation Model Architecture Design
- The self-supervised foundation model predicts next-state embeddings from state–action histories, creating reward-agnostic representations transferable across downstream tasks.Its architecture uses encoders, positional embeddings, transformer blocks, and a prediction head; pretraining minimizes the difference between predicted and actual next-state embeddings.Source: §4.3 · p. 5
4.4 Foundation Model-Aided Decentralized MARL Algorithm Design
- Each device trains its actor and critic from local experience, exchanging scalar rewards with neighbors while keeping the pretrained foundation-model backbone fixed.The fixed backbone supplies dynamics representations to a reward-specific value head, reducing online backbone updates and communication relative to exchanging critic parameters.Source: §4.4 · p. 5
4.5 Theoretical Performance Analysis
- The convergence bound is more sensitive to network depth than width, and includes critic estimation and approximation errors from foundation-model pretraining.The latter errors reflect the quality of the pretrained reward-agnostic representations and scale with the number of agents.Source: §4.5 · p. 7
- O(1/NT) convergence to a neighborhood of the global optimum matches actor–critic methods using linear approximation and full critic exchange.The analysis also characterizes convergence toward global optimality, unlike cited prior guarantees limited to stationary points.Source: §4.5 · p. 7
5 Numerical Evaluation
The evaluation tests end-to-end learning across three RA objectives and compares FM-aided MARL with conventional and end-to-end baselines. FM-aided MARL learns faster while achieving comparable final rate and AoI performance.
5.2 Experimental Results and Discussion
- For K=2, end-to-end learning balances AoI, maximizes one device’s rate, or jointly improves both devices’ throughput while preserving fairness, depending on the task.The three tasks are Fair-AoI, Max-rate, and Fair-rate.Source: Figure 4 · p. 8, §5.2 · p. 8
- Fair-AoI reduces collisions fastest, while Fair-rate retains the highest collision counts after many episodes; normalized reward sum rises during learning.The Fair-rate reward imposes stronger penalties for rate unfairness, and collision reduction emerges later in training.Source: Figure 5 · p. 8, §5.2 · p. 8
5.1 Experimental Settings
- Each device’s FM is pretrained on more than 100,000 reward-independent samples, and reported results average at least 20 independent runs.The evaluation measures throughput, AoI, and normalized fairness gap.Source: §5.1 · p. 8
5.2 Experimental Results and Discussion
- FM-aided learning improves convergence speed by at least 55% across tasks versus end-to-end and conventional MARL.The conventional MARL baseline performs full critic exchanges.Source: §5.2 · p. 8, Figure 6 · p. 9
- E-E and FM-MARL outperform CSMA-based baselines; round-robin is near-optimal for Fair-AoI but degrades under Max-rate.Under Max-rate, learned methods prioritize devices with more favorable channel conditions.Source: Figure 7 · p. 9, §5.2 · p. 9
- FM-aided MARL reaches its 95%-optimal performance level substantially faster than end-to-end training, with comparable final rate and AoI performance.The comparison is summarized in Table 1.Source: §5.2 · p. 9
6 Conclusion
The conclusion presents an FM-aided decentralized MARL framework for RA optimization and reports faster convergence than end-to-end training across diverse objectives. It identifies larger action spaces, including transmit power control, as future work.
6 Conclusion
- The FM-aided algorithm leverages self-supervised knowledge of network dynamics through a reward-agnostic critic backbone, improving learning efficiency and converging faster than end-to-end training across throughput and AoI objectives.Table 1 compares E-E and FM-aided MARL for K=4 (top) and K=8 (bottom); AoI is measured for fair-AoI, and rate for max-rate and fair-rate tasks.Source: §6 · p. 9, Table 1 · p. 9
- The framework uses an FM as a reward-agnostic critic backbone to leverage self-supervised knowledge of network dynamics and improve learning efficiency.Source: §6 · p. 9
- Future work will extend the framework to more complex RA scenarios with larger action spaces, such as transmit power control.Source: §6 · p. 9