PaperlayerView exact revision on arXiv

Source-linked AI summary

Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Management

Yiming Zhang, Kun Yang, Cong Shen, Dongning Guo

arXiv:2608.28878v1eess.SYcs.AI

TL;DR

Wireless resource management needs distributed policies that operate with local information despite the performance limits of heuristics and the deployment barriers of centralized optimization. The paper develops a hybrid offline-online multi-agent decision-transformer framework that combines trajectory imitation with critic-guided online fine-tuning and coordination mechanisms. Across joint scheduling and power allocation and coordinated beamforming, learned policies achieve QoS comparable to centralized methods, while fine-tuning can surpass policies that generated lower-quality offline data.

  • Problem

    Wireless resource management needs distributed QoS control under local observations, because centralized optimization depends on impractical global CSI and computation while local heuristics can perform substantially worse.

  • Method

    The framework pretrains multi-agent decision transformers offline on existing-policy trajectories, then fine-tunes online with critic-guided gradients, return-weighted replay, neighbor-conditioned critics, and correlated exploration.

  • Results

    Across joint scheduling and power allocation and coordinated beamforming, learned policies achieve QoS comparable to genie-aided centralized methods using local measurements and limited neighborhood exchange.

  • Takeaways & Limitations

    Online fine-tuning can improve policies beyond the data-generating policies, including when offline pretraining uses suboptimal data.

Abstract

from arXiv · show

This paper develops a hybrid offline-online multi-agent reinforcement learning framework based on decision transformers. The policy is first pretrained offline via supervised sequence modeling of trajectories generated by existing policies, providing a safe and sample-efficient initialization. It is then fine-tuned online using a hybrid objective that incorporates critic-guided gradients, enabling performance improvements beyond the offline policy. To facilitate stable offline-to-online transfer and effective multi-agent coordination, the framework incorporates return-weighted sampling, a critic conditioned on neighbors' actions, and neighborhood-correlated exploration. The approach is fully distributed: both training and execution rely only on local observations and limited information exchange among neighboring agents. Evaluations with dynamic traffic arrivals in two settings: (i) joint scheduling and power allocation and (ii) coordinated beamforming, show that the proposed method achieves quality-of-service (QoS) performance comparable to centralized methods. Moreover, when pretrained on lower-quality datasets, online fine-tuning is also observed to surpass the initial offline policy. These results demonstrate a promising learning-based alternative for wireless resource management.

I. Introduction

The paper asks whether distributed wireless resource management can use only local information while approaching centralized QoS, and proposes a hybrid offline-online MARL framework based on decision transformers. It addresses partial observability, coordination, and deployment constraints across multiple wireless settings.

  • Motivation: Distributed wireless resource management must balance QoS against the impracticality of global CSI and centralized computation.Existing centralized optimization is near-optimal but difficult to deploy, while local heuristics can substantially degrade performance.
  • Policy architecture: The framework uses decision transformers because return-conditioned sequence modeling supports supervised offline pretraining for dynamic wireless resource allocation.The architecture models evolving CSI, queues, actions, and returns over time.
  • Hybrid framework: The framework combines offline imitation of existing policies with online critic-guided fine-tuning to provide safe initialization and improvement beyond the offline policy.Offline pretraining uses generated trajectories, while online learning adapts the policy through subsequent interaction.
  • Coordination challenges: Return-weighted replay, neighbor-action-conditioned critics, and neighborhood-correlated exploration address nonstationarity, credit assignment, and coordinated exploration.These mechanisms use local observations and neighborhood information exchange.
  • Applications: The approach is fully distributed and applies to joint scheduling with power allocation and coordinated beamforming by redefining per-agent observations, actions, and rewards.Both settings evaluate packet delay as the QoS metric under dynamic traffic arrivals.

B. Offline Pretraining with Decision Transformers

The offline pretraining stage trains a shared decision-transformer policy to predict actions from local and neighborhood histories, returns-to-go, and temporal context. At execution, the policy targets high-return behavior from the offline dataset while acting independently at each agent.

  • Policy inputs: Each agent’s decision transformer uses its own and neighbors’ observations, past actions, returns-to-go, and a within-episode slot embedding.The transformer maps a length-L history of aggregate information to an autoregressive action distribution.
  • Offline benefits: Offline pretraining provides safe initialization by imitating proven policies and improves efficiency by avoiding online exploration during initial training.Prior decision-transformer results indicate convergence toward the performance of the behavior policies generating the data.
  • Dataset construction: The offline dataset records agents’ local observations, actions, rewards, and returns-to-go from simulated baseline-policy trajectories.Length-L trajectory windows are sampled for supervised training over all agents and time slots.
  • Execution: At execution, the target return starts at the offline dataset’s mean top-quartile episode return and decreases with each realized reward.A causal attention mask prevents predictions from using future tokens, and each agent executes independently.

C. Online Fine-Tuning

Online fine-tuning augments supervised imitation with critic-guided policy improvement, allowing the decision-transformer policy to surpass its offline training data. Twin critics and stabilized target updates provide the advantage estimates used for this refinement.

  • Purely offline policies can fail under distribution shift and remain limited by low-reward training trajectories.
  • The actor combines cross-entropy regularization, reinforcement-learning improvement, and entropy-based exploration during online fine-tuning.The supervised term preserves proven behavior, while entropy encourages continued exploration.
  • Twin critics estimate per-action values from aggregate information, with the second critic reducing overestimation in bootstrap targets.The critics are lightweight, memoryless networks rather than transformer critics.
  • The RL loss increases the likelihood of actions whose estimated advantage is above average.Advantage normalization makes the gradient scale less sensitive to reward magnitude and smoothly reweights the action distribution.
  • Quasi-static target critics and a target actor are updated by Polyak averaging to stabilize temporal-difference learning.The target uses the minimum of two target critics to counteract value overestimation.
  • Online iterations collect trajectories, train critics repeatedly from replay data, then update the actor and target networks.The replay buffer retains high-quality offline trajectories alongside newly collected online data.

D. Stability and Coordination in Online Fine-Tuning

The framework stabilizes offline-to-online transfer and addresses coordination challenges caused by partial observability, nonstationarity, and neighborhood-dependent rewards. Its mechanisms trade added implementation complexity for coordinated behavior, especially in tightly coupled systems.

  • Partially observable multi-agent fine-tuning creates nonstationary targets, credit-assignment problems, and coordination demands that independent exploration may miss.
  • Stabilizing the offline-to-online transition: Critic warm-up, gradual activation of the RL term, anchored replay, and return-weighted sampling stabilize the transition from offline training.The replay buffer mixes high-return offline anchors with online trajectories.
  • Stabilizing the offline-to-online transition: Using the running maximum of observed episode returns prevents poor exploratory episodes from lowering the target return for data collection.
  • Conditioning the critic on neighbors’ actions: Because an agent’s reward depends on neighbors’ concurrent actions, the critic is conditioned on neighbors’ intended actions during training.This lets the critic evaluate an action in the context of neighboring behavior.
  • Temporally-extended, neighborhood-correlated exploration: Neighborhood-correlated exploration coordinates actions across agents and consecutive slots, addressing the limits of independent epsilon-greedy exploration.
  • The coordination mechanisms add implementation complexity but may be unnecessary in smaller or weakly coupled systems.In those settings, a per-agent critic and standard action sampling may suffice.

III. Wireless Model and Problem Formulation

The paper instantiates its distributed MARL framework in wireless networks, using offline pretraining followed by online refinement while preserving distributed execution. The formulation covers channel dynamics, queues, interference-defined neighborhoods, and power-control variables.

  • III. Wireless Model and Problem Formulation: The three-phase pipeline synthesizes or collects data, trains an initial decision-transformer policy offline, then fine-tunes it online through environmental interaction.Online refinement is intended to improve beyond the original dataset while preserving distributed execution.
  • A. Joint Scheduling and Power Allocation: The joint scheduling and power-allocation setting models K transmitter-receiver links, each treated as an agent communicating with interference-coupled neighbors.Neighbor communication can use low-latency interfaces such as the Xn interface.
  • 1) System Model:: Each link uses a single-antenna flat-fading model with channel power gains determined by large-scale path loss and small-scale fading.
  • 1) System Model:: Temporal channel correlation is modeled through a first-order complex Gauss-Markov process with correlation coefficient ρ.The fading innovations are independent, identically distributed circularly symmetric complex Gaussian variables with unit variance.
  • 1) System Model:: Agent neighborhoods are defined by a relative path-loss threshold indicating nonnegligible interference, with typically at most six neighbors in a hexagonal deployment.
  • 1) System Model:: Each transmitter selects a power, and the global power-allocation vector collects the K per-link powers under additive white Gaussian noise.The resulting spectral efficiency is defined for each link.
  • 1) System Model:: Traffic-driven links maintain FIFO queues whose packet arrivals and queue lengths evolve over time.Queue length is measured in bits at the end of each slot, while local observations are formed at the beginning of the slot.

2) Agent Design:

Each agent forms decisions from locally available and neighborhood-shared information, selecting transmission behavior and discrete power levels. Queue length serves as a QoS surrogate, while neighbor utilities are included to discourage interference-inducing selfish actions.

  • The queue backlog after new arrivals is part of the agent’s decision information.
  • Previous-slot spectral efficiency and related transmission quantities are available with a one-slot delay through neighborhood information exchange.
  • Agent k decides whether to transmit and, if transmitting, selects a power level.
  • The discrete power action set includes zero and levels equally spaced in dB between minimum and maximum nonzero powers.
  • Queue length is used as a surrogate QoS objective because Little’s law links shorter time-averaged queues with shorter delays.
  • Rewards also include neighboring agents’ utilities to discourage reducing one’s backlog at the cost of greater interference and larger neighboring queues.

B. Beamforming

The beamforming instantiation models six mutually interfering AP agents with inward-facing sectors, local observations, neighbor exchange, and discrete beam-selection actions. Traffic-driven delay minimization couples each AP’s rate with the interference it creates for neighboring cells.

  • Deployment: Six APs form a ring cluster with inward-facing 120° sectors, each serving one device while neighboring sectors create strong interference coupling.
  • Interference and QoS: Beam selections trade each AP’s own transmission rate against the interference imposed on other cells.
  • Traffic and QoS: Poisson arrivals accumulate in FIFO queues, and the setting uses traffic-driven delay minimization as its QoS objective.
  • Agent design: Each AP either remains silent or selects one of B beams spanning its sector.
  • Observations: The local observation includes direct and neighboring beam gains, with realized transmission quantities becoming available after a one-slot delay.

IV. Performance Evaluation

The evaluation tests the hybrid reinforcement-learning framework in two wireless resource-allocation settings. The joint scheduling and power-allocation task is abbreviated as power allocation throughout the evaluation.

  • The experiments evaluate the hybrid-RL framework on both wireless resource-allocation settings introduced earlier.
  • The joint scheduling and power-allocation task is referred to as power allocation for brevity.

A. Power Allocation

The power-allocation evaluation uses a 19-device, 19-cell network, multiple distributed and centralized baselines, and expert or medium offline datasets. Training uses 40 offline episodes followed by 200 online exploratory episodes, with held-out common evaluation conditions.

  • Simulation setup: The simulation contains 19 devices in 19 hexagonal cells, with neighborhoods determined by interference coupling and neighbor observations ordered by local index.
  • Baselines: The comparison includes full-power, random-power, ITLinQ, ideal WMMSE, and delayed WMMSE baselines.
  • Offline data: Offline datasets contain 40 episodes and 40,000 transitions, generated by ideal WMMSE or delayed WMMSE behavior policies.
  • Evaluation protocol: Online fine-tuning collects one exploratory episode per iteration for 200 iterations, while evaluation uses a disjoint held-out channel and traffic realization.
  • Metrics: The training curves report average episode return alongside mean end-to-end delay because the negative queueing-cost reward makes them almost perfectly affinely related.

2) Training dynamics and convergence:

Offline pretraining rapidly imitates the behavior datasets, while online fine-tuning initially dips under exploration before recovering. Medium-quality pretraining benefits most: it improves delay beyond both its dataset and centralized teacher, whereas expert pretraining has little headroom.

  • Training dynamics and convergence: Within roughly 20 epochs, offline pretraining approaches the dataset-average return using only 40 episodes and no environment interaction.
  • Training dynamics and convergence: Online fine-tuning shows a transient return drop during the first ∼25 iterations because committed exploration and initially uninformative critics depress early updates.
  • Training dynamics and convergence: Expert-pretrained fine-tuning returns to the dataset-average level, while medium-pretrained fine-tuning crosses that average around iteration 70 and settles above it.
  • QoS performance: The expert-pretrained policy achieves 31.8 ms mean delay versus 30.5 ms for ideal WMMSE while operating fully distributedly.
  • QoS performance: Fine-tuning the medium-pretrained policy reduces mean delay from 44.8 to 37.0 ms, a 17.5% improvement that outperforms delayed WMMSE.
  • QoS performance: Learned policies outperform full-power, ITLinQ, and random-power references, whose 99th-percentile delay tails exceed one second.
  • Discussion: The distributed policy uses local measurements and limited neighborhood exchange, with per-agent computation and signaling that do not grow with network size.
  • Discussion: Synthetic data from unrealizable ideal WMMSE can distill into a distributed policy attaining nearly the same QoS performance with local information.

1) Simulation setup:

The beamforming experiment evaluates offline pretraining and online fine-tuning under Poisson traffic, using per-packet delay and comparisons with distributed heuristics and centralized coordination. Fine-tuning improves performance beyond medium-quality behavior data, while gains remain concentrated in interference-limited settings and are bounded by local information.

  • Simulation setup: Beamforming uses Poisson traffic and per-packet delay, comparing random selection, distributed greedy max-gain selection, and a queue-weighted coordinatedescent expert.The expert performs Gauss–Seidel descent over the discrete joint beam space.
  • Training dynamics and convergence: Offline pretraining imitates medium-dataset max-gain behavior within roughly 15 epochs using only 10 episodes.The evaluation return rises from a large negative value to just below the dataset average.
  • Training dynamics and convergence: Online fine-tuning crosses the medium dataset average within about 20 iterations and settles well above it.Exploratory episodes are initially depressed by committed exploration, whereas the evaluation return improves beyond the behavior policy.
  • QoS performance: On the medium dataset, fine-tuning reduces mean packet delay by 40%, from the pretrained policy’s 82.7 ms to 49.5 ms, outperforming its teacher.The pretrained policy’s delay CDF nearly coincides with the max-gain teacher’s CDF before fine-tuning.
  • Broader implications: The framework transfers across scheduling and power allocation and coordinated beamforming while requiring only local observations and limited neighborhood exchange.The beamforming setting changes action semantics, deployment, agent granularity, and model and data budgets.
  • Scope and limitations: Beamforming gains are concentrated in interference-limited regimes, while partial interferer visibility leaves an imitation gap to the centralized expert.With light inter-cell interference, greedy max-gain already approaches expert delay statistics, leaving little room for learning-based coordination.
Loading 2608.28878v1…