Source-linked AI summary
Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems
Alex S. Leong, Arunselvan Ramaswamy, Daniel E. Quevedo, Holger Karl, Ling Shi
TL;DR
The paper studies how to schedule transmissions from multiple sensors over fewer shared wireless channels so a gateway can estimate remote dynamical processes. It formulates scheduling as an MDP and solves it with a Deep Q-Network that is scalable and model-free with respect to channel parameters. Numerical studies show consistent and significant improvement over other suboptimal scheduling policies, while future work considers model-based learning when channel knowledge is available.
Problem
The problem is scheduling multiple sensors over limited shared wireless channels to support remote state estimation.
Method
The paper formulates scheduling as an MDP and uses deep reinforcement learning with neural-network function approximation to solve it.
Results
The resulting scheduling algorithm consistently and significantly outperforms other suboptimal sensor scheduling policies in numerical studies.
Takeaways & Limitations
The algorithm can run online and is model-free with respect to wireless channel parameters.
Takeaways & Limitations
The scheduler does not use channel parameters; future work considers model-based learning when additional channel knowledge is available.
Abstract
from arXiv · showhide
In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios.
1 Introduction
The paper addresses scheduling sensor transmissions for remote estimation in large cyber-physical systems, where shared wireless channels and estimation requirements complicate access decisions. It formulates the scheduling problem as an MDP and motivates deep reinforcement learning for scalable, model-free solution.
- System setting: Sensors transmit readings over a shared wireless network to a gateway that estimates multiple remote processes.The network has M wireless channels, and channel volatility complicates transmission scheduling.
- Scheduling challenge: Scheduling must balance estimation and control requirements against communications objectives such as waiting times and throughput.Wireless channel diversity can increase the dynamic scheduling difficulty.
- Scheduling challenge: Typically, fewer wireless channels are available than processes, requiring the scheduler to select which sensors transmit at each time.The gateway selects a subset of sensors for communication and estimation updates.
- Solution motivation: Large state and action spaces make traditional Q-learning impractical because of Bellman’s curse of dimensionality.The paper identifies function approximation as a way to address this scalability problem.
- Solution motivation: The paper formulates the scheduling problem as an MDP and applies a scalable, model-free Deep Q-Network approach.The paper compares the resulting scheduler with established scheduling policies in numerical studies.
2 System Model
The system consists of multiple independent linear processes observed by sensors and estimated at a gateway through a shared wireless network. Sensors compute local estimates, while channel assignments and packet drops determine which information reaches the gateway.
- Process and sensing model: The model considers N independent linear discrete-time processes with Gaussian process noise and sensor measurements subject to Gaussian measurement noise.Each process follows a state equation and each sensor produces a corresponding measurement.
- Process and sensing model: Each sensor can run a local Kalman filter and compute local state estimates and estimation error covariance matrices.The model assumes observability and controllability conditions for the process pairs.
- Scheduling and channels: At most M sensors transmit simultaneously because the gateway allocates M shared wireless channels to different sensors.The intended setting is resource-limited, with M much smaller than N in typical applications.
- Scheduling and channels: Wireless transmissions may be lost, with each independent channel modeled by a Gilbert-Elliott process.The channel parameters p_m and q_m represent failure and recovery rates, respectively, and are unknown to the scheduler.
- Scheduling and channels: Scheduling decisions are made at the gateway, fed back to sensors, followed by uplink transmission of their local state estimates.The model assumes the gateway-to-sensor signaling transmission is error-free.
- Gateway estimation: The gateway computes process estimates and estimation error covariances from received transmissions and known process parameters.These quantities support the remote estimation procedure at each time step.
3 Problem Description
The paper formulates sensor scheduling as an MDP that minimizes the long-run average estimation error while accounting for channel outcomes. It derives a stability condition and uses deep reinforcement learning because the problem may have large spaces and unknown channel parameters.
- MDP formulation: The objective is to minimize the long-run average sum of estimation-error covariance traces across all processes and times.The gateway seeks a scheduling policy for this average-cost problem.
- MDP formulation: The MDP state uses previous estimation covariances and channel outcomes, with an equivalent lower-dimensional representation based on sensor holding times and channel states.Holding time records the time since a sensor’s last successful transmission.
- MDP formulation: The action space contains all assignments of M distinct channels to distinct sensors, totaling N!/(N−M)! possible allocations.The single-stage cost is defined within this scheduling MDP.
- Stability condition: Under Assumption 1, the optimal MDP solution has bounded average cost.The assumption is expressed using the largest process spectral radius and largest channel recovery rate.
- Stability condition: The stability condition generalizes the single-process, single-channel case and relates bounded cost to the best channel serving the most unstable process.The relevant channel is the one with the largest recovery rate, while instability is characterized by the largest spectral radius.
- Computational issues: Exact dynamic programming is infeasible for practical problem sizes, and unknown channel parameters prevent standard dynamic programming methods.The paper therefore uses deep neural-network function approximation and reinforcement learning.
4 Sensor Scheduling Using Deep Reinforcement Learning
The paper formulates sensor scheduling as an MDP and approximates its discounted solution with model-free deep Q-learning, addressing large state and action spaces. The resulting policy can be run online without channel-parameter knowledge and is linked theoretically to the average-cost objective.
- Motivation: Large state and action spaces make exact dynamic programming and conventional Q-learning impractical for the scheduling MDP.The paper motivates function approximation to overcome this computational limitation.
- Deep reinforcement learning: The scheduler’s Q-factor is approximated by a deep neural network, with the policy selecting the action that maximizes Q(s, a; θ).The approach uses experience replay and periodic target-network fixing to stabilize learning.
- MDP formulation: The scheduling state can include previous holding times and channel outcomes, with an augmented state used in simulations.This state representation incorporates information available from recent scheduling and transmission events.
- Practical properties: The method is model-free with respect to channel parameters and can be run online.The scheduler therefore does not require knowledge of the channel failure and recovery probabilities.
- Deep reinforcement learning: The algorithm uses replayed transitions and target-network updates while training from observed rewards and next states.Algorithm 1 combines ε-greedy action selection, replay memory, minibatch updates, and a discounted target.
- Average-cost relationship: The discounted-cost solution is ϵ-optimal for the average-cost problem, with ϵ made arbitrarily small as the discount factor approaches one.Because single-stage costs are unbounded, the simulations use a discount factor close to one.
5 Numerical Studies
Numerical studies evaluate the deep Q-network on randomly generated sensor and channel scenarios against random, round-robin, and greedy policies. The learned policy achieves lower empirical average cost and consistently outperforms the alternatives after training.
- Experimental setup: The main experiment uses N = 6 sensors, M = 3 channels, two-dimensional processes, and scalar measurements with randomly generated parameters.The process and channel parameters are sampled for the simulation scenario.
- Experimental setup: The deep Q-network receives a 15-node augmented-state input and outputs 120 action values for the N!/(N −M)! possible allocations.Training uses two 1024-node hidden layers, δ = 0.95, and replay memory size K = 20000.
- Training procedure: Each training episode resets the process after T = 500 steps, and training runs for 200 episodes.On the reported CPU configuration, each episode takes around 30 seconds.
- Results: 17.8 is the trained policy’s empirical average cost over 50000 simulation steps.This value is compared with random, round-robin, and greedy scheduling policies.
- Results: 62.7, 42.7, 31.3, and 22.4 are the empirical average costs for random, round-robin, holding-time-greedy, and error-covariance-greedy policies, respectively.The deep reinforcement learning approach consistently outperforms these policies after around 40-50 training episodes.
- Results: Across 10 randomly generated parameter sets, deep reinforcement learning performs significantly better than every other considered policy.Random and round-robin policies generally perform poorly, while error-covariance greedy generally beats holding-time greedy.
6 Conclusion
The paper applies deep reinforcement learning to wireless channel allocation for remote estimation of multiple dynamical systems. Its scheduling algorithm is online, model-free with respect to channel parameters, and consistently outperforms other suboptimal policies, while future work considers model-based learning.
- Contribution: The studied problem allocates wireless channels among sensors to support remote state estimation of multiple dynamical systems.The setting concerns shared wireless resources and geographically distributed processes.
- Contribution: The proposed deep-reinforcement-learning scheduler is intended to handle larger problems than previous approaches.The conclusion presents scalability as the motivation for using modern deep reinforcement learning ideas.
- Practical properties: The scheduling algorithm can run online and is model-free with respect to wireless channel parameters.The stated model-free scope concerns the channel parameters, not the process parameters.
- Results: Numerical results show that the approach consistently and significantly outperforms other suboptimal sensor scheduling policies.The conclusion summarizes the comparative evaluation across the reported scenarios.
- Future work: Future work will study model-based reinforcement learning when additional channel-parameter knowledge is available.The stated goal is to possibly improve learning speed.
A Proof of Theorem 1
The proof establishes bounded average cost by analyzing a single-channel policy based on the channel with the largest recovery rate. It extends the probability argument to Markovian packet drops and concludes that additional channels do not increase the optimal average cost.
- Single-channel policy: The proof first restricts attention to the single channel m* and selects the sensor with the largest holding time when that time exceeds L > 2N.Here m* denotes the channel used in the single-channel analysis, and the policy is suboptimal but sufficient for establishing bounded average cost.
- Transmission-probability bound: The probability term P_L is interpreted as the probability of n successful transmissions in L time steps.This quantity supports the subsequent bound on successful and failed transmission patterns.
- Markovian packet drops: For Markovian packet drops, among L − n failed transmissions, at most n follow successful transmissions, so at least L − 2n follow failed transmissions.This observation adapts the argument from an i.i.d. packet-drop setting to the Markovian channel model.
- Markovian packet drops: The resulting bound uses transition-rate factors to upper-bound the probabilities of successful transmissions and failed transmissions with preceding failures.The second inequality applies for all n < N when L > 2N, after which limits are taken in (A.2).
- Bounded-cost conclusion: Under the stated condition, the single-channel suboptimal policy has bounded average cost, and therefore the MDP using only channel m* has bounded optimal average cost.Because adding channels cannot increase the optimal average cost, the same boundedness conclusion holds for the full system.