PaperlayerView exact revision on arXiv

Source-linked AI summary

Pervasive Machine Learning for Smart Radio Environments Enabled by Reconfigurable Intelligent Surfaces

George C. Alexandropoulos, Kyriakos Stylianopoulos, Chongwen Huang, Chau Yuen, Mehdi Bennis, Mérouane Debbah

arXiv:2205.03793v1cs.ITcs.ETcs.LGeess.SPmath.IT

TL;DR

Efficient orchestration of multiple RISs is challenging amid channel-estimation overhead and limited device computing. The paper surveys DRL approaches and proposes contextual bandits for online sum-rate optimization; in tested cases, its Neural ϵ-greedy method matches DQN and RL methods outperform random selection.

  • Problem

    Orchestrating multiple RISs requires efficient configuration, while channel estimation of direct and RIS-linked channels can impose substantial overhead.

    Source: §I · p. 2

  • Method

    The paper surveys DRL orchestration and formulates online sum-rate maximization as contextual bandits by exploiting independence between current configurations and future channel states.

    Source: §VI · p. 19, §VII · p. 25

  • Results

    Across considered action-space cases, Neural ϵ-greedy and DQN perform identically, both exceed about 80% of optimal rates, and all RL methods far outperform random selection.

    Source: C. Sum-Rate Results versus the Size of… · p. 22

  • Takeaways & Limitations

    When RIS and precoder choices affect only the current frame, the orchestration objective is myopic and can be treated as contextual bandits.

    Source: §VI · p. 19

    The formulation uses a centralized controller as the sole agent; distributed UE or multiple-controller setups are left outside the paper’s scope.

    Source: A. Extensions with Practical… · p. 18

Abstract

from arXiv · show

The emerging technology of Reconfigurable Intelligent Surfaces (RISs) is provisioned as an enabler of smart wireless environments, offering a highly scalable, low-cost, hardware-efficient, and almost energy-neutral solution for dynamic control of the propagation of electromagnetic signals over the wireless medium, ultimately providing increased environmental intelligence for diverse operation objectives. One of the major challenges with the envisioned dense deployment of RISs in such reconfigurable radio environments is the efficient configuration of multiple metasurfaces with limited, or even the absence of, computing hardware. In this paper, we consider multi-user and multi-RIS-empowered wireless systems, and present a thorough survey of the online machine learning approaches for the orchestration of their various tunable components. Focusing on the sum-rate maximization as a representative design objective, we present a comprehensive problem formulation based on Deep Reinforcement Learning (DRL). We detail the correspondences among the parameters of the wireless system and the DRL terminology, and devise generic algorithmic steps for the artificial neural network training and deployment, while discussing their implementation details. Further practical considerations for multi-RIS-empowered wireless communications in the sixth Generation (6G) era are presented along with some key open research challenges. Differently from the DRL-based status quo, we leverage the independence between the configuration of the system design parameters and the future states of the wireless environment, and present efficient multi-armed bandits approaches, whose resulting sum-rate performances are numerically shown to outperform random configurations, while being sufficiently close to the conventional Deep Q-Network (DQN) algorithm, but with lower implementation complexity.

I. INTRODUCTION

The introduction frames RISs as a way to make wireless propagation reconfigurable for 6G, while noting practical orchestration and channel-estimation challenges. It motivates online reinforcement learning and outlines a survey of DRL alongside a multi-armed bandit alternative for multi-RIS systems.

I. INTRODUCTION

  • 6G targets higher data rates, extended coverage, 1000× network capacity, 10× energy and cost efficiency, and cm-level positioning accuracy.The cited consensus gives peak data rates up to 1 Tbps.Source: §I · p. 1
  • RISs can reconfigure EM-wave propagation through controlled meta-atom states, but their reflective operation and limited computing and storage create network-design challenges.The surfaces are described as almost passive and lacking power amplification.Source: §I · pp. 1–2
  • Channel estimation must cover direct BS–UE, RIS–BS, and RIS–UE channels, and its high-dimensional overhead motivates investigation of faster RIS phase-profile management.Source: §I · p. 2
  • Prior RIS-focused deep-learning work spans physical- and MAC-layer applications, while this survey favors online RL for adapting in non-stationary settings without a separate data-collection stage.The introduction contrasts RL with supervised learning, whose models require data collection before deployment.Source: §I · p. 3
  • The paper models a multi-antenna BS, multiple single-antenna UEs, and shared RISs, then maps a QoS-constrained sum-rate objective to DRL terminology and generic algorithmic steps.Source: §I · p. 3
  • Beyond RL theory and taxonomy, the survey reviews DRL applications and practical challenges, and proposes a simpler multi-armed bandit formulation with numerical comparisons to benchmark algorithms.Source: §I · p. 3

II. THE CONSIDERED SMART RADIO ENVIRONMENT

The section models joint control of multiple RIS reflections and BS precoding in a multi-user downlink. It describes discrete RIS hardware and the channel, power, and propagation assumptions used for the system model.

II. THE CONSIDERED SMART RADIO ENVIRONMENT

  • The optimization formulation jointly selects RIS reflections and BS precoding, with equal power allocated among the K user signals.Multi-user power control is an additional design freedom left for future work.Source: §II · p. 4, B. System Model · p. 6

A. Abstract Modeling of RISs

  • RISs are modeled as planar arrays of digital meta-atoms, each with b-bit phase resolution and a reflection coefficient controlling its reflected electromagnetic wave.The model uses a transmission-line circuit representation; practical reflection amplitude can depend on phase and incident-wave angle, and the response is frequency dependent.Source: A. Abstract Modeling of RISs · p. 5

B. System Model

  • The system model jointly orchestrates multiple RISs and BS precoding for downlink communication between a multi-antenna BS and K single-antenna UEs.The RISs share a controller, and are placed near the UEs, where direct BS communication is highly attenuated.Source: B. System Model · p. 5, Figure 1 · p. 6
  • The end-to-end channel model uses a cascaded BS–RIS–UE representation; applying the framework to physics-inspired non-cascaded channel models is left for future work.Reflections involving more than one RIS are neglected because their multiplicative pathloss is assumed to make them highly attenuated.Source: B. System Model · p. 6

C. Channel Models

The channel models combine Ricean fading for RIS links with Rayleigh fading for direct BS–UE links. They also specify propagation assumptions and identify extensions for inter-RIS reflections, mutual coupling, and temporal correlation.

C. Channel Models

  • RIS–BS and RIS–UE links use Ricean fading, while direct BS–UE links are Rayleigh faded with a common attenuation factor ν.The Ricean factors are the power ratios of the specular components to the random components; ν=0 denotes absent direct links and ν=1 non-attenuated direct links.Source: C. Channel Models · pp. 6–7
  • Reflections involving more than one RIS are neglected because their multiplicative pathloss is assumed to make them highly attenuated.Source: C. Channel Models · p. 6
  • The diagonal RIS response model can be extended to account for mutual coupling, and correlated fading can be included in the channel matrices.For meta-atom spacing much smaller than λ/2, the response matrix may be full or banded and incorporates a mutual-coupling matrix.Source: C. Channel Models · p. 6
  • A finite-state Markov channel model can capture temporal correlation, an identified direction for reducing RIS-optimization overhead.The paper assumes channels change independently between transmission time intervals.Source: C. Channel Models · p. 6
  • RIS-link models include line-of-sight steering components and non-line-of-sight scattering, while the direct links are modeled as non-line-of-sight channels.The scattering components are modeled with complex Gaussian distributions.Source: C. Channel Models · pp. 6–7

D. Orchestration of Smart Radio Environments

This section frames RIS orchestration as a difficult discrete optimization problem and uses sum-rate maximization as a representative objective for RL-based solutions. It also situates the paper as an RL-focused survey that introduces RL concepts and taxonomies before discussing its application to RIS systems.

D. Orchestration of Smart Radio Environments

  • The RIS phase responses and BS precoding can be optimized for utilities including rate, energy efficiency, coverage, localization, sensing, radio mapping, secrecy, and low EM-field exposure.The paper uses achievable sum rate as its exemplary objective while noting the formulation applies to other utilities expressible as functions of RIS-element responses.Source: D. Orchestration of Smart Radio… · p. 7
  • OP1 maximizes users’ achievable sum rate subject to discrete RIS and precoding constraints, optionally including each UE’s minimum-rate request.The paper treats the problem with and without individual rate constraints and notes that even with perfect CSI, the discrete non-convex optimization is computationally exhaustive.Source: D. Orchestration of Smart Radio… · p. 7

III. OVERVIEW OF REINFORCEMENT LEARNING

  • The survey focuses on RL approaches, assuming broad familiarity with ANNs and deep learning rather than introducing them.It provides an RL theory introduction, a taxonomy of RL methods, and discussion of DRL approximations and evaluation methodology.Source: §III · pp. 7–8

A. Markov Decision Processes

An MDP models sequential decisions through states, actions, transition probabilities, and rewards, with an agent learning a policy to maximize expected return. The section introduces value-function optimization and its practical limitations, then outlines generic DRL training.

A. Markov Decision Processes

  • An MDP is a tuple of state and action spaces, transition probabilities, and a reward function; at each step, the agent observes a state, chooses an action, receives reward, and the environment transitions.The Markov property makes the next state depend only on the current state-action pair, not earlier history.Source: A. Markov Decision Processes · p. 9, Figure 3 · p. 9
  • A policy maps states to actions, while the RL objective is to find a policy maximizing expected return; return is the discounted sum of future rewards.The discount factor makes the reward sum finite in the infinite-horizon case and weights immediate versus future rewards.Source: A. Markov Decision Processes · p. 9
  • Under partial observability, the agent sees noisy observations rather than the true state, making value-function learning harder; DRL may use learned representations and memory units to address this.The text notes that MDP algorithms are also applied to POMDPs when noisy observations are assumed to retain enough useful information.Source: A. Markov Decision Processes · p. 9

B. Value Functions

  • State-value and action-value functions estimate expected return under a policy, and Bellman equations provide recursive relations for deriving their optimal forms.Once optimal values are obtained, the optimal policy selects the action maximizing expected return via the Q-function.Source: B. Value Functions · p. 10
  • The exact dynamic-programming solution assumes known transition probabilities and requires iterating over all states or state-action pairs, which is intractable for many practical problems.These are the two stated drawbacks of the optimal MDP solution.Source: B. Value Functions · p. 10
  • Generic DRL training repeatedly collects state, action, next-state, and reward experience, then updates ANN parameters to optimize an objective at intervals.The learned policy and collection policy are updated from the optimization result.Source: B. Value Functions · p. 10

C. Deep Reinforcement Learning Algorithms

DRL approximates MDP solution functions using ANNs and trains from agent-environment experience. The section classifies DRL approaches and explains DQN’s value-based learning procedure and stabilization measures.

C. Deep Reinforcement Learning Algorithms

  • DRL methods commonly use a model-free strategy that estimates functions from agent experience, with policies parameterized directly or induced by value functions.The survey groups methods into value-based, policy-gradient, and actor-critic approaches.Source: C. Deep Reinforcement Learning… · p. 10
  • During DRL training, an agent collects observations, actions, and rewards, then periodically applies gradient-based updates to ANN parameters using the collected data.Collection may be on-policy or off-policy, depending on whether the collection policy matches the policy being learned.Source: C. Deep Reinforcement Learning… · p. 10

1) Value-Based Approaches:

  • DQN [162] estimates Q-values for discrete actions with a neural network, sampling stored experience tuples in mini-batches and updating estimates using temporal-difference error.The replay buffer stores state, action, reward, and next-state tuples; the network outputs one predicted Q-value per action.Source: 1) Value-Based Approaches · pp. 10–11
  • DQN uses an off-policy learned action maximizing predicted Q-values, while ϵ-greedy exploration occasionally selects random actions to balance exploration and exploitation.The text gives typical ϵ values of [0.01, 0.2] and notes that training often starts higher before decreasing it.Source: 1) Value-Based Approaches · p. 11
  • DQN training can be unstable because frequent updates make the same network generate target Q-values and update current estimates; a slower-updated target network offers increased stability when state estimates change slowly enough.The target network uses separate parameters and is updated less frequently than the original Q-network.Source: 1) Value-Based Approaches · p. 11

2) Policy-Gradient Approaches:

Policy-gradient methods directly optimize a parameterized policy, with REINFORCE using Monte Carlo sampling to estimate gradients. Actor-critic methods pair a policy-learning actor with a value-estimating critic, with PPO and DDPG addressing distinct update and action-space challenges.

2) Policy-Gradient Approaches:

  • Policy-gradient methods use a parameterized policy to maximize expected episodic reward, updating policy parameters by gradient ascent.The policy-gradient theorem gives a stochastic gradient that can be approximated through Monte Carlo sampling in REINFORCE.Source: 2) Policy-Gradient Approaches · p. 11

3) Actor-Critic Approaches:

  • Monte Carlo gradient estimates can have high variance; actor-critic methods use an advantage function and separate actor and critic networks to estimate policy and state value.The critic’s estimated state value supports substituting the advantage estimate into the policy-gradient update.Source: 3) Actor-Critic Approaches · pp. 11–12
  • PPO [168] clips gradient updates within a trust region to discourage large changes in action likelihood under the new policy.The text describes PPO as relatively straightforward to implement and reports its performance in multi-agent training scenarios [169].Source: 3) Actor-Critic Approaches · p. 12
  • DDPG [170] is an actor-critic method for continuous actions, using a separate actor to select actions and a critic to estimate Q-values.The networks are trained concurrently off-policy, with target networks used to stabilize training.Source: 3) Actor-Critic Approaches · p. 12
  • Value-based methods can be simpler and more interpretable but scale poorly in large discrete action spaces; policy gradients handle such spaces but are usually less sample efficient.Actor-critic methods combine value and policy networks, at the cost of increased computational needs and complexity.Source: 3) Actor-Critic Approaches · p. 12

D. Multi-Armed Bandits

Multi-armed bandits select actions to maximize expected reward when actions do not affect future environment states. The section covers reward estimation and exploration strategies, then extends the framework to contextual bandits and neural methods.

D. Multi-Armed Bandits

  • In multi-armed bandits, actions do not affect future states, and the agent seeks the most rewarding action without a state or policy.Each action has a separate payoff distribution, and the agent receives a reward sampled from the selected action’s distribution.Source: D. Multi-Armed Bandits · p. 12
  • Bandit algorithms estimate each action’s expected reward with a running average and combine it with an exploration strategy.The section describes ϵ-greedy and Upper Confidence Bound (UCB) strategies; UCB incorporates uncertainty into action selection.Source: D. Multi-Armed Bandits · pp. 12–13
  • For non-stationary rewards, recent observations can receive greater weight through an exponential recency-weighted average.The section notes that this can replace the ordinary running average.Source: D. Multi-Armed Bandits · p. 13
  • Contextual bandits use an observation before each action, but lack state-transition probabilities and are simpler than full MDPs.The observation is related to reward values but need not fully represent the system’s internal state.Source: D. Multi-Armed Bandits · p. 13
  • LinUCB models expected rewards linearly from context, while neural linear bandits learn the coefficient representation with an encoding network.LinUCB estimates coefficients by ridge regression and uses UCB; neural linear bandits use a learned representation and Thompson sampling in the described approach.Source: D. Multi-Armed Bandits · p. 13
  • A reward-prediction ANN can estimate each action’s expected reward from context, with training error computed only for the selected action.The section names this approach Neural ϵ-greedy and describes training on observation-reward tuples.Source: D. Multi-Armed Bandits · p. 13
  • Bandits offer more intuitive and explainable algorithms than DRL, with convergence properties that are easier to study theoretically.The section also notes that their effectiveness can be demonstrated empirically.Source: D. Multi-Armed Bandits · p. 13

E. Algorithmic Evaluation

The paper presents a DRL formulation for RIS-empowered environments and discusses its use for sum-rate maximization and other objectives. It also outlines algorithmic evaluation practices and the training and deployment workflow for the formulation.

E. Algorithmic Evaluation

  • Convergence guarantees for DRL methods may be asymptotic or limited to particular MDP families and function constraints.The section attributes the difficulty of general results to the minimal assumptions of the MDP formalism and application-dependent convergence behavior.Source: E. Algorithmic Evaluation · p. 14
  • Algorithmic evaluation can combine cumulative reward with the number of samples needed to reach a performance threshold as a measure of sample efficiency.Off-policy methods are generally expected to be more sample-efficient because they can reuse prior experiences.Source: E. Algorithmic Evaluation · p. 14
  • DRL execution cost is treated as constant per iteration, with forward-pass and gradient-update complexity tied to network parameters and action selection.For discrete actions, the forward pass adds a Θ(card(A)) argmax cost; training is comparable under O(1) mini-batch sizes.Source: E. Algorithmic Evaluation · p. 14

IV. DRL-ORCHESTRATED SMART RADIO ENVIRONMENTS

  • Section IV presents a DRL formulation for RIS-empowered environments and surveys available DRL approaches for wireless communication systems including RISs.The formulation is introduced for the design objective from Section II.Source: §IV · p. 14

A. DRL Formulation

  • The controller acts as the agent, choosing RIS phase profiles and BS precoding from the wireless-system design parameters.The proposed formulation maps the considered system and sum-rate objective to DRL ingredients.Source: A. DRL Formulation · p. 14
  • The training loop estimates channels, forms the state and action, gathers UE SINR feedback, computes rewards, stores experience, and updates the ANN.After training, the learned policy configures the environment without requiring SINR collection for training.Source: A. DRL Formulation · p. 15
  • With IID channel realizations and actions that do not affect future channel states, the DRL formulation has state-action-independent transitions and aligns with bandits.For time-correlated channels, the section describes bandit updates that emphasize recent observations as a possible approach.Source: A. DRL Formulation · p. 15
  • The formulation can use objectives that directly depend on current channel coefficients and RIS phases, including energy efficiency, secrecy rate, and localization accuracy.It can also extend beyond CSI observations and standalone RIS control to such inputs as UE positions and controls such as power allocation or UAVs with mounted RISs.Source: A. DRL Formulation · p. 16

B. Literature Overview

The section surveys DRL approaches for RIS-empowered wireless systems, including their formulations, objectives, and system settings. It notes that many prior models with IID states and actions that do not affect future states fit a bandit formalism.

B. Literature Overview

  • Most surveyed approaches train DRL agents on CSI observations to configure RIS phase shifts, sometimes alongside other system parameters.The overview says many studies use MISO systems with one RIS and multiple UEs, often adding BS processing or power allocation to the action space.Source: B. Literature Overview · pp. 16, 18
  • Prior work applies DDPG to continuous RIS phases and precoding, including a multi-RIS THz system where it outperformed an alternate beamforming benchmark in total throughput.The overview also describes DQN in a system combining mobility estimation, UE clustering, and RIS configuration for sum-rate optimization.Source: B. Literature Overview · p. 16
  • Some studies address limited CSI through partial observations, reward feedback, or local channel states, and evaluate DRL under imperfect CSI.The described examples include hybrid RIS channel sensing and a mobile-UE setting that uses past actions and rewards as observations.Source: B. Literature Overview · pp. 16–17
  • Table IV provides a taxonomy of prior RIS-empowered DRL approaches by their reinforcement-learning formulations.The overview notes that many models with IID channel realizations and actions that do not affect future states adhere to a bandits formalism despite being studied as MDPs.Source: Table IV · p. 17, B. Literature Overview · p. 18
  • The surveyed objectives include energy efficiency and secrecy rate, with RL action spaces also covering RIS switching, UAV power allocation, and RIS displacement.The overview describes DQN, PPO, DDPG, and D3QN applications across these settings.Source: B. Literature Overview · p. 17
  • Some more elaborate settings fall outside the presented general formulation, including mobile RIS orchestration and sequential RIS sensing for object identification.The cited examples use traditional optimization for some mobile-RIS problems and combine policy-gradient control with supervised object detection for sensing.Source: B. Literature Overview · p. 17

V. PRACTICAL CONSIDERATIONS AND OPEN CHALLENGES

The section introduces practical considerations and open challenges arising from the paper’s DRL design formulation, while noting that its discussion is not exhaustive.

V. PRACTICAL CONSIDERATIONS AND OPEN CHALLENGES

  • The section offers a non-exhaustive discussion of practical considerations and open challenges arising from the proposed design formulation.It frames these considerations in relation to the DRL formulation described earlier.Source: §V · p. 18

A. Extensions with Practical Considerations

The paper discusses practical limits and open challenges for multi-RIS orchestration, then presents a contextual bandit formulation and a reward-prediction ANN for online sum-rate maximization.

A. Extensions with Practical Considerations

  • Time-varying channels make each next state depend on the current or earlier states, so DRL must store and train on transitions longer than one step.The discussion contrasts this with IID channel realizations, for which transitions of length one suffice.Source: A. Extensions with Practical… · p. 18
  • The formulation is centralized, with one controller as the sole agent serving all UEs; UE agents or multiple RIS controllers are left for future study.Those distributed cases may involve conflicting UE goals and lead to multi-agent learning or game-theoretic formulations.Source: A. Extensions with Practical… · p. 18

B. Open Research Challenges

  • RL methods lack theoretical performance guarantees, and the paper identifies several further open problems for their application.The authors frame the listed challenges as important research directions.Source: B. Open Research Challenges · p. 18

1) Environmental Observations and Control Overhead:

  • RL assumes access to observations and rewards, but sensing, CSI estimation, control-channel exchange, and their overheads can affect achievable performance.The authors identify quantifying control and observation overheads, often treated as negligible, as a critical issue, especially for multi-RIS systems.Source: 1) Environmental Observations and… · p. 18

2) Simulations versus Real-World Deployment:

  • Policies learned in simulation require cautious transfer to physical RIS systems because channel models may not capture all important physical aspects.The authors note that RIS electromagnetic models are not yet well understood.Source: 2) Simulations versus Real-World… · p. 18

3) Combinatorially Large Action Spaces:

  • The action space grows exponentially with the number of RIS meta-atoms, making optimal performance intractable or training-intensive at large scales.Proposed remedies include combining domain-specific optimization with DRL or using algorithms tailored to large or binary-vector action spaces.Source: 3) Combinatorially Large Action Spaces · pp. 18–19

4) RIS Operation Capabilities and Computational Requirements:

  • ANN training may be offloaded to an edge server, while integrating the controller into a sensing-capable RIS could reduce control exchanges at added computational and sensing cost.The overheads of these alternatives are not yet extensively studied for multi-RIS deployments.Source: 4) RIS Operation Capabilities and… · p. 19

A. Proposed and Benchmark Methods

  • Because system design choices affect only the current time frame under IID channels, the orchestration problem reduces to contextual reward prediction, addressed with Neural ϵ-greedy.The ANN maps a CSI observation to expected rewards for available actions and is updated on recent experience using the observed reward for the selected action.Source: §VI · p. 19, 1) Neural ϵ-Greedy · pp. 19–20

2) DQN:

DQN serves as an MDP-based benchmark for testing whether its added complexity benefits the multi-armed bandit approach; random and exhaustive-search policies provide a baseline and upper bound.

2) DQN:

  • DQN is used as a representative MDP-based DRL benchmark to test whether its added ANN and tuning complexity benefits performance over multi-armed bandits.The comparison is motivated by the formulation's contextual-bandit structure; bandits are also special cases of MDPs with horizon T = 1.Source: 2) DQN · p. 20
  • For a fair complexity comparison, DQN uses a Q-network identical to Neural ϵ-greedy’s reward-prediction ANN, but maintains two independent copies.The parameter values were selected by Bayesian optimization over candidate values in a simple scenario.Source: 2) DQN · p. 20

3) Random and Optimal Schemes:

  • The random policy chooses RIS profiles and BS precoders randomly, while the optimal policy exhaustively searches all available actions for maximum sum rate.Source: 3) Random and Optimal Schemes · p. 21

4) UCB:

The results compare Neural ϵ-greedy, DQN, and UCB against random and optimal schemes across action-space sizes and transmit powers. UCB improves over random, while the DRL methods remain close to optimal in the tested settings.

4) UCB:

  • UCB selects the action with the highest upper-bound estimate of average gain, using running action averages initialized randomly and a single confidence-interval parameter.The parameter c was empirically set to 0.6 for the experiments.Source: 4) UCB · p. 21

B. Simulation Setup

  • The simulations use two RISs, four BS antennas, and two single-antenna UEs; RIS meta-atoms are controlled in groups of 16 to reduce action-space dimensionality.The grouping is expected to slightly reduce performance but makes ANN training computationally tractable.Source: B. Simulation Setup · p. 21

C. Sum-Rate Results versus the Size of the Action Space

  • Across action-space cases, both DRL methods exceed about 80% of optimal rates, while UCB beats random and stays reasonably close without CSI; relative performance declines as action spaces grow.Neural ϵ-greedy and DQN have identical performance and converge to optimal RIS phase configurations in the simplest 32-meta-atom-per-RIS case.Source: Figure 7 · p. 22, Figure 8 · p. 22, C. Sum-Rate Results versus the Size of… · p. 22

D. Sum-Rate Results versus the Transmit Power

  • Around 75% of optimum performance is achieved by both DRL methods nearly irrespective of transmit power; UCB rises from around 45% at 10 dBm to around 70% at 50 dBm.At 50 dBm, UCB attains almost equal sum rate to both DRL methods; all three RL methods outperform random selection, which remains below 45% of optimum.Source: Figure 9 · p. 23, D. Sum-Rate Results versus the… · p. 23

E. Reward Results versus the Size of the Action Space

The section compares reward and sum-rate performance across constrained and partial-CSI settings, then evaluates execution time and summarizes the paper’s bandit contribution.

E. Reward Results versus the Size of the Action Space

F. Orchestration with Partial CSI Observability

G. Methods’ Execution Time Comparison

VII. CONCLUSIONS

  • The paper surveys multi-RIS DRL orchestration and concludes bandits can approach DQN with simpler implementation and faster execution.The bandit formulation leverages the independence of system design parameters from the next environment state.Source: §VII · p. 25
Loading 2205.03793v1…