PaperlayerView exact revision on arXiv

Source-linked AI summary

A Reinforcement Learning Framework for Optimizing Age-of-Information in RF-powered Communication Systems

Mohamed A. Abd-Elmagid, Harpreet S. Dhillon, Nikolaos Pappas

arXiv:1908.06367v1cs.ITcs.NI

TL;DR

The paper addresses online freshness optimization in multi-source RF-powered monitoring systems, where WET and update scheduling jointly determine weighted sum-AoI. It models the problem as an average-cost MDP, develops a computationally efficient DRL algorithm, and analytically characterizes policy structure. The resulting age-optimal policy is threshold-based in process AoI, while age-optimal and throughput-optimal policies have different structures.

  • Problem

    The paper asks how to jointly optimize WET and update scheduling to minimize long-term average weighted sum-AoI in multi-source RF-powered monitoring systems.

  • Method

    The authors formulate an average-cost MDP, develop a computationally efficient DRL algorithm, and analytically characterize age-optimal and throughput-optimal policy structures.

  • Results

    The age-optimal policy has a threshold-based structure with respect to process AoI, and its structure differs from the throughput-optimal policy.

  • Takeaways & Limitations

    The analysis and numerical results provide policy-structure and system-design insights for RF-powered freshness-aware communication systems.

Abstract

from arXiv · show

In this paper, we study a real-time monitoring system in which multiple source nodes are responsible for sending update packets to a common destination node in order to maintain the freshness of information at the destination. Since it may not always be feasible to replace or recharge batteries in all source nodes, we consider that the nodes are powered through wireless energy transfer (WET) by the destination. For this system setup, we investigate the optimal online sampling policy (referred to as the age-optimal policy) that jointly optimizes WET and scheduling of update packet transmissions with the objective of minimizing the long-term average weighted sum of Age-of-Information (AoI) values for different physical processes (observed by the source nodes) at the destination node, referred to as the sum-AoI. To solve this optimization problem, we first model this setup as an average cost Markov decision process (MDP). Due to the extreme curse of dimensionality in the state space of the formulated MDP, classical reinforcement learning algorithms are no longer applicable to our problem. Motivated by this, we propose a deep reinforcement learning (DRL) algorithm that can learn the age-optimal policy in a computationally-efficient manner. We further characterize the structural properties of the age-optimal policy analytically, and demonstrate that it has a threshold-based structure with respect to the AoI values for different processes. We extend our analysis to characterize the structural properties of the policy that maximizes average throughput for our system setup, referred to as the throughput-optimal policy. Afterwards, we analytically demonstrate that the structures of the age-optimal and throughput-optimal policies are different. We also numerically demonstrate these structures as well as the impact of system design parameters on the optimal achievable average weighted sum-AoI.

I. INTRODUCTION

The paper studies freshness-aware transmission in RF-powered monitoring systems, where energy constraints and channel variation complicate online AoI minimization. It develops reinforcement-learning and analytical tools to characterize age-optimal and throughput-optimal policies.

  • Motivation: RF-powered source nodes must deliver timely updates despite limited energy budgets and wireless-channel pathloss.Energy harvesting is motivated as an alternative to replacing or recharging batteries, especially for difficult-to-reach devices.
  • Problem: The objective is to schedule transmissions under energy-causality constraints while minimizing average AoI at the destination.AoI quantifies information freshness, and the paper focuses on freshness-aware RF-powered communication systems.
  • Contributions: The paper proposes computationally efficient numerical characterization of the age-optimal policy and analytical comparison with the throughput-optimal policy.The contributions include structural analyses of both policies rather than only numerical optimization.
  • Related work: Prior energy-harvesting studies use varied battery, transmission-time, channel, and harvesting assumptions, limiting direct applicability to RF-powered systems.Here, harvested energy depends on time-varying channel-state information, requiring CSI statistics in decision-making.
  • Gap and scope: Unlike prior single-pair or simplified models, this paper considers multiple RF-powered sources sensing potentially different physical processes.The model jointly accounts for battery dynamics, process-specific AoI values, and CSI.

B. Contributions

The paper formulates joint WET and update scheduling for minimizing weighted sum-AoI, then combines deep reinforcement learning with structural analysis. Its results characterize threshold policies and distinguish age optimization from throughput optimization.

  • Problem formulation: The problem jointly optimizes WET and update-packet scheduling to minimize long-term average weighted sum-AoI.The objective aggregates AoI values for different physical processes using importance weights.
  • MDP formulation: The system is modeled as an average-cost MDP whose decisions depend on source energy, process AoI, and CSI.Each time slot is allocated either to WET or to transmission from a source node.
  • DRL approach: Extreme MDP state-space dimensionality makes classical reinforcement-learning algorithms computationally infeasible for the age-optimal policy.The paper addresses this limitation with a DRL algorithm designed for computationally efficient policy learning.
  • Age-optimal policy: The age-optimal policy has a threshold-based structure with respect to AoI values for different processes.The analysis establishes this through monotonicity of the value function associated with the MDP.
  • Single-pair structure: For a single source-destination pair, the age-optimal policy is threshold-based in battery level, AoI, and channel power gains.This structure supports analytical comparison with throughput-optimal scheduling.
  • System insights: Age-optimal and throughput-optimal policies differ most at large AoI values, while their structural differences vanish as AoI decreases.Numerical results also show weighted sum-AoI decreases with battery capacity and increases with update-packet size.

C. Organization

The paper proceeds from system modeling and MDP formulation to DRL optimization, structural analysis, numerical verification, and conclusions. The model specifies RF harvesting, AoI evolution, channel knowledge, and slot-level actions.

  • Paper organization: Sections II–III present the system model and formulate weighted sum-AoI minimization before introducing the DRL solution.The organization then turns to policy-structure analysis and numerical evaluation.
  • Network model: Multiple source nodes observe physical processes and send status updates to a common destination powered by a stable energy source.Sources harvest RF energy from destination broadcasts as their only energy source.
  • State dynamics: The system uses unit-length slots, finite batteries, bounded AoI states, and quasi-static fading on downlink and uplink channels.Channel gains remain constant within a slot and change independently between slots.
  • Operational assumptions: The destination has perfect current-slot CSI and statistical knowledge of future slots, while each source cannot harvest and transmit simultaneously.These assumptions define the information available for online decisions.
  • State and action spaces: The system state contains each source’s battery level, AoI, and downlink and uplink channel gains, while actions select WET or one source transmission.The action set is {H, T1, T2, ..., TN}, with H denoting WET.
  • Slot evolution: A WET action charges source batteries, whereas a transmission action sends a generated update packet subject to the source’s energy condition.Successful transmission resets the corresponding AoI to one; otherwise AoI increases subject to its cap.

III. PROBLEM FORMULATION AND PROPOSED SOLUTION

The paper formulates weighted sum-AoI minimization as an average-cost MDP whose actions jointly select wireless energy transfer or update transmission. Because the resulting state space grows explosively, it motivates DRL and analytical policy characterization.

  • A. Problem Statement: The objective is to find a policy minimizing long-term average weighted sum-AoI across processes at the destination.Process weights θ_i represent their relative importance.
  • A. Problem Statement: Each slot selects either WET or an update transmission according to AoI, battery level, and channel gains.The action space includes H for WET and T_i for transmission from source i.
  • B. MDP Formulation: The discretized system is modeled as a finite-state, finite-action MDP with state variables for batteries, AoI values, and channel gains.Continuous channel supports are divided into equiprobable intervals before forming the discrete state space.
  • B. MDP Formulation: The average-cost Bellman equations characterize an optimal stationary deterministic policy and its achievable average AoI.The value function and Q-function support the policy characterization.
  • B. MDP Formulation: With 10 values per state variable and three sources, the state space reaches 10^12 states, making classical methods computationally infeasible.Complexity increases with either finer discretization or more source nodes.
  • B. MDP Formulation: The paper proposes DRL to obtain the age-optimal policy numerically and derives structural properties analytically.These contributions address the large-scale policy-characterization problem.

C. Deep Reinforcement Learning for Optimizing AoI

The proposed DRL method approximates the Q-function with an ANN to handle the extremely large MDP state space. It combines Q-learning with feature extraction, replay memory, and batch updates.

  • Deep Q Network: The method uses an ANN to approximate Q(s, a|β) and extract useful state features in a smaller representation.A fully connected layer performs the state-space abstraction.
  • Q-Learning: The reinforcement component retains Q-learning updates while selecting actions through the learned Q-function.The average-cost update uses the immediate cost and the minimum next-state Q-value.
  • Exploration and Exploitation: An ϵ-greedy policy explores state-action pairs while gradually reducing exploration to emphasize exploitation.Random actions are selected with probability 0 < ϵ < 1 during learning.
  • Motivation: Q-learning alone is insufficient when the state space is extremely large because storing or visiting all state-action pairs becomes impractical.The limitation worsens as state supports or the number of sources grow.
  • Training: Replay memory stores past experiences, from which random batches are sampled to train the ANN.The paper states that batch updates and replay memory improve DRL convergence.
  • Algorithm: Algorithm 1 iterates through observing states, selecting actions, recording costs and next states, sampling experiences, and updating network weights until convergence.The stopping condition is convergence to a value of average weighted sum-AoI.

IV. STRUCTURAL PROPERTIES OF THE AGE-OPTIMAL POLICY

The paper analytically establishes monotonicity properties of the value function and a threshold-based structure for the age-optimal policy. The threshold result is general in AoI variables but more specialized for other state variables.

  • Value-Function Monotonicity: The value function is non-increasing in battery level and downlink or uplink channel gain, but non-decreasing in AoI.These monotonicity properties are established through value-iteration arguments.
  • Multiple Sources: For multiple sources, the age-optimal policy has a threshold structure with respect to each process’s AoI.If transmitting process j is optimal at threshold A_th,j, it remains optimal for larger A_j with other variables fixed.
  • Multiple Sources: The threshold interpretation reflects that sufficiently large AoI makes sending a new update packet optimal for that process.This consequence is stated for the destination’s information status.
  • Multiple Sources: For N > 1, the policy does not generally have a threshold structure in battery levels or channel power gains.The paper contrasts this limitation with the AoI threshold result.
  • Single Source: For N = 1, the policy has threshold-based structure over the full state variables, including battery, AoI, and both channel gains.The single-source result applies to the classical source-destination pair model.
  • Connection to Throughput: The single-source structural results are used to compare age-optimal and throughput-optimal policies.The paper identifies the single-source case as more tractable for this comparison.

V. AGE-OPTIMAL POLICY VS. THROUGHPUT-OPTIMAL POLICY

The paper compares age-optimal and throughput-optimal policies analytically in the single source-destination setting. It formulates throughput maximization and uses its policy structure to identify differences between the objectives.

  • Comparison Setup: The comparison focuses on the single source-destination model because its structure is more tractable analytically.The paper first formulates average throughput maximization for N = 1.
  • Policy Comparison: The analysis characterizes structural properties of the throughput-optimal policy and highlights differences from the age-optimal policy.The comparison is based on the respective policy structures.

A. Average Throughput Maximization Formulation and Proposed Solution

The throughput-maximization problem is formulated as a finite-state, finite-action MDP with harvesting or transmission actions, then solved through a Bellman equation and value iteration.

  • The single-source throughput state comprises battery level and downlink and uplink channel gains, while AoI is excluded.
  • Each slot offers harvesting or transmission of a packet of size S, with battery evolution governed by the system dynamics.
  • The problem is a finite-state, finite-action MDP with an optimal stationary deterministic policy.
  • The throughput objective assigns reward S only to slots allocated for data transmission and maximizes long-term average throughput.
  • The optimal policy is obtained by solving the Bellman equation using value iteration, with transition probabilities computed from the system model.

B. Structural Properties of the Throughput-optimal Policy

The throughput-optimal policy has a threshold-based structure over system states, but its structure differs from the age-optimal policy even for states sharing battery and channel conditions.

  • The associated value-function analysis establishes structural properties for the throughput-optimal policy.
  • The policy structure is monotone with respect to battery level, downlink channel gain, and uplink channel gain.
  • The throughput-optimal policy has a threshold-based structure over the state set Sth,r.
  • The age-optimal and throughput-optimal policies have different structures, as demonstrated analytically and verified numerically.
  • For states with the same battery and channel-gain combination, the two policies can still choose different actions.

VI. NUMERICAL RESULTS

The numerical section verifies the analytical results and evaluates the proposed DRL algorithm’s achievable average weighted sum-AoI across system design parameters.

  • The experiments verify analytical results and examine performance under different system parameters.
  • Channel power gains are modeled using distance-dependent path loss and exponentially distributed small-scale fading.
  • The channel gains are discretized into Gi and Hi possible values for the numerical model.
  • The numerical setup uses W = 1 MHz, P = 37 dBm, η = 0.5, and σ2 = −95 dBm unless otherwise specified.

A. Verification of Analytical Results

Numerical results verify the threshold structures of the age-optimal and throughput-optimal policies and show that their actions diverge depending on AoI and battery conditions.

  • VI. Numerical Results: The plotted states encode optimal actions, with T1, T2, and H represented by blue squares, red circles, and black diamonds.
  • A. Verification of Analytical Results: The numerical age-optimal policies satisfy the analytical structural properties established in Theorems 1 and 2.
  • A. Verification of Analytical Results: When the action at (2, 1) is H, the action at (1, 1) is also H, confirming the stated age-policy structure.
  • B. Comparison of the Structures of the Age-optimal and Throughput-optimal Policies: The throughput-optimal policy’s analytical structure is also verified numerically through its threshold-region actions.
  • B. Comparison of the Structures of the Age-optimal and Throughput-optimal Policies: For small AoI, the age-optimal and throughput-optimal policies are similar because fresh information creates no urgency to transmit.
  • B. Comparison of the Structures of the Age-optimal and Throughput-optimal Policies: When A1 > 1, the age-optimal policy always selects T1 regardless of available battery energy, preserving energy for later updates as AoI grows.

C. Impact of System Design Parameters on Optimal Average Weighted Sum-AoI

The DRL algorithm approaches the optimal average AoI and shows how system design parameters affect achievable weighted sum-AoI. Smaller update packets and larger batteries reduce achievable average weighted sum-AoI.

  • Evaluation scope: The study evaluates DRL convergence against classical reinforcement learning for small-scale settings and examines parameter impacts for N = 3.The classical optimum can only be evaluated numerically when N and each state variable’s discrete support are small.
  • Impact of system design parameters: For N = 3, achievable average weighted sum-AoI decreases as update-packet size decreases and battery capacity increases.Smaller packets require less transmission energy, while larger batteries store more harvested energy for transmissions when AoI is large.
  • Convergence of the DRL algorithm: The DRL algorithm approaches the optimal average AoI in the N = 1 setting.The comparison uses the optimal value obtained by the RVIA; the remaining gap is attributed to ϵ-greedy exploration during learning.
  • Convergence of the DRL algorithm: The optimal average AoI can be achieved after reducing ϵ to zero once the algorithm converges.This removes continued exploration and exploits the learned policy.
  • Impact of system design parameters: Greater battery capacity increases the likelihood that a source has enough energy for an update transmission when its process AoI is large.This contributes to reducing achievable average weighted sum-AoI.
Loading 1908.06367v1…