PaperlayerView exact revision on arXiv

Source-linked AI summary

A Learning Theoretic Approach to Energy Harvesting Communication System Optimization

Pol Blasco, Deniz Gündüz, Mischa Dohler

arXiv:1208.4290v2cs.LGcs.NI

TL;DR

The paper studies expected data transmission in an energy-harvesting link when the transmitter has differing levels of knowledge about stochastic arrivals and channel states. It develops offline, online, and learning-based optimizations, showing that learning approaches can approach online performance and substantially outperform greedy transmission in the reported settings.

  • Problem

    EH transmitters must schedule delay-limited data with stochastic energy and data arrivals, but process statistics may change or be unavailable before deployment.

  • Method

    The paper analyzes offline, online, and learning theoretic optimization for a Markov-modeled EH communication system, using policy iteration, mixed integer programming, and Q-learning.

  • Results

    90% of online performance is reached after a reasonably small number of learning iterations, while smart energy-aware policies reach 60%–90% of offline performance versus greedy transmission.

  • Takeaways & Limitations

    Learning-based transmission can provide practically relevant performance without prior statistical knowledge, approaching online optimization as learning progresses.

Abstract

from arXiv · show

A point-to-point wireless communication system in which the transmitter is equipped with an energy harvesting device and a rechargeable battery, is studied. Both the energy and the data arrivals at the transmitter are modeled as Markov processes. Delay-limited communication is considered assuming that the underlying channel is block fading with memory, and the instantaneous channel state information is available at both the transmitter and the receiver. The expected total transmitted data during the transmitter's activation time is maximized under three different sets of assumptions regarding the information available at the transmitter about the underlying stochastic processes. A learning theoretic approach is introduced, which does not assume any a priori information on the Markov processes governing the communication system. In addition, online and offline optimization problems are studied for the same setting. Full statistical knowledge and causal information on the realizations of the underlying stochastic processes are assumed in the online optimization problem, while the offline optimization problem assumes non-causal knowledge of the realizations in advance. Comparing the optimal solutions in all three frameworks, the performance loss due to the lack of the transmitter's information regarding the behaviors of the underlying Markov processes is quantified.

I. INTRODUCTION

The paper addresses EH communication when arrival-process statistics may be unavailable or unreliable, comparing offline, online, and learning-based transmission optimization. It models stochastic arrivals and channel variation while enforcing strict packet deadlines and studies algorithms for each information setting.

  • Information settings: Offline optimization assumes non-causal arrival knowledge, whereas online optimization uses process statistics and causal realizations.The paper argues that changing process characteristics and unavailable statistics make these assumptions impractical in many deployments.
  • System and motivation: The system has Markov-modeled data, energy, and channel processes, with strict deadlines causing untransmitted packets to be lost.The transmitter uses a finite rechargeable battery and may terminate operation probabilistically during its activation time.
  • Learning approach: The learning theoretic approach operates without prior knowledge of the Markov-process statistics and learns from actions, observed states, and immediate rewards.The transmitter learns an optimal policy over time under causal observations.
  • Learning approach: Q-learning performance converges to that of the optimal online transmission policy as learning time increases.This provides a learning-based counterpart to the fully informed online solution.
  • Optimization methods: The paper solves online optimization with policy iteration and formulates offline optimization as a mixed integer linear program solved using branch-and-bound, alongside a linear-program relaxation.Numerical results compare learning theoretic, online, and offline solutions.

II. RELATED WORK

Prior EH optimization work is organized mainly around offline and online information assumptions. Related studies also consider forecasting and historical data to optimize transmission or duty cycles when process knowledge is limited.

  • Existing frameworks: Offline studies assume non-causal knowledge of exact data and energy arrivals, while online studies use process statistics and causal realizations.The literature includes point-to-point, multi-user, fading-channel, and MIMO settings.
  • Limited statistical information: Related work under milder statistical assumptions uses periodic forecasting or historical data to estimate energy arrivals and optimize duty cycles.These approaches motivate alternatives for systems whose stochastic processes are not fully known beforehand.

III. SYSTEM MODEL

The system is a finite-state, time-slotted EH link with strict one-slot data deadlines, finite battery storage, Markov arrivals and channel states, and a probabilistically finite activation horizon. Its objective is expected total transmitted data.

  • System model: Data and energy arrive in packets each time slot, while the channel remains constant within a slot and changes across slots.Harvested energy becomes available for transmission in the following slot.
  • Stochastic processes: Data and energy packet sizes and channel states are modeled as first-order Markov processes.The channel realization is known to the receiver, and packet transmission energy depends on packet size and channel state.
  • Transmission decisions: At each slot, the transmitter observes battery, arriving packet, and channel state, then chooses whether to transmit or drop the packet.Transmission rate and power are fixed at the slot start, and energy expenditure cannot exceed available energy.
  • Objective: The objective is to maximize expected total transmitted data during a randomly terminated activation period, called the ETD-problem.The main analysis assumes 0 ≤ γ < 1; γ = 1 yields an infinite-operation case unsuitable for this performance measure.
  • MDP formulation: The communication model forms a finite-state discrete-time MDP whose state includes energy arrival, data packet, channel, and battery components.Actions are binary, rewards equal transmitted packet data, and the Markov property enables dynamic programming and reinforcement learning.
  • Information frameworks: The three optimization frameworks differ by transmitter information: online uses known transition and reward models, learning lacks them, and offline knows future realizations non-causally.The offline solution can serve as an upper bound for the other approaches.

IV. ONLINE OPTIMIZATION

The online problem is solved as a finite-state MDP using policy iteration. Iterative policy evaluation and improvement continue until the policy is stable, yielding the optimal transmission policy for 0 ≤ γ < 1.

  • Policy iteration: Policy iteration computes the optimal online transmission policy using dynamic programming under finite state and action spaces.Convergence is guaranteed under bounded stationary rewards when 0 ≤ γ < 1.
  • Policy evaluation: During policy evaluation, an iterative value estimate converges to the policy value as the evaluation iteration count tends to infinity.The value function measures expected discounted reward for each state.
  • Policy improvement: During policy improvement, the algorithm selects a policy whose action choice is greedy with respect to the current action-value function.The policy improvement theorem establishes that the resulting policy is at least as good as the evaluated policy.
  • Convergence: Policy iteration alternates evaluation and improvement until the same policy appears in consecutive iterations.The algorithm’s complexity depends on the number of states and actions.

V. LEARNING THEORETIC APPROACH

The learning-theoretic approach uses Q-learning without prior knowledge of transition probabilities or rewards, while balancing exploration and exploitation. Under standard conditions, its estimates converge to the optimal action values, and its performance approaches the optimal online policy as learning proceeds.

  • Learning setup: Q-learning finds the optimal transmission policy without knowing transition probabilities or the immediate reward function.It assumes only an MDP model and observes the next state and reward after each action.
  • Convergence: With nonzero action-selection probabilities, 0 ≤ γ < 1, and suitable learning-rate constraints, Q_n(s_j, x_i) converges to Q^π∗(s_j, x_i) with probability 1.The convergence holds as the number of learning iterations tends to infinity.
  • Learning setup: The transmitter uses ε-greedy selection to explore random actions with probability ε and exploit the greedy policy otherwise.This balances discovering better policies with using currently known actions.
  • Convergence: The convergence rate decreases as the numbers of actions and states, and the discount factor γ, increase.Learning-rate choice also affects the convergence rate.
  • Learning algorithm: Q-learning updates each state-action estimate from the immediate reward plus the discounted value of the next state.The update uses the observed next state and reward after performing an action.
  • Performance: As learning iterations increase, the Q-learning performance converges to that of the optimal online transmission policy.The paper evaluates this performance against other approaches in its numerical comparison.

VI. OFFLINE OPTIMIZATION

The offline problem assumes non-causal knowledge of future arrivals and channel variations, then maximizes transmitted data over a finite realization. Its binary transmission formulation is solved as a MILP using LP relaxation and branch-and-bound methods.

  • Offline formulation: Offline optimization assumes the transmitter knows future data and energy arrivals and channel variations non-causally before transmission begins.It optimizes a finite sequence of state transitions for the realized MDP.
  • Offline formulation: The offline solution provides an upper bound on online and learning-theoretic performance because it uses future system information.The performance difference measures the value of non-causal knowledge.
  • Optimization structure: The complete offline problem is a mixed integer linear program with affine objective and constraints and binary transmission variables X_n.Relaxing 0 ≤ X_n ≤ 1 yields an LP in which partial packet transmission is allowed.
  • Optimization structure: The LP-relaxation supplies an upper bound on the binary complete problem, and an integral LP solution is also optimal for the complete problem.The paper shows that the relaxed formulation remains equivalent under the stated feasibility argument.
  • Branch-and-bound: Branch-and-bound partitions the binary feasible set into subsets, solving LP relaxations and pruning subsets whose bounds cannot improve the incumbent.Branching fixes a fractional variable to either X_n = 0 or X_n = 1.
  • Complexity: Branch-and-bound has worst-case complexity O(2^N), matching exhaustive search, although its average complexity is usually much lower.The average complexity is polynomial under certain conditions.

VII. NUMERICAL RESULTS

The numerical study evaluates the three optimization approaches in an IEEE 802.15.4e-based energy-harvesting scenario. It specifies channel, energy, and transmission parameters and compares expected performance using dynamic programming, learning, and offline optimization procedures.

  • Performance comparison: The comparison evaluates expected performance for the optimal online policy, learning-based optimization, and offline optimization.The online benchmark is obtained using dynamic programming or averaging over starting states.
  • Evaluation procedure: The offline numerical procedure uses branch-and-bound steps that evaluate LP relaxations, prune subproblems, and branch fractional solutions.Confidence intervals are used in the performance analysis, with computation details placed in the appendix.
  • Scenario: The numerical scenario uses 10 ms time slots, 5 ms transmission time, and 2 MHz bandwidth based on IEEE 802.15.4e.These parameters define the communication-system timing and bandwidth assumptions.
  • Scenario: The fundamental energy unit is 2.5 µJ, and each slot harvests either two energy units or none.The harvested-energy process is modeled with discrete outcomes.
  • Scenario: The channel-state transition model uses a 0.9 self-transition probability for both channel states.Thus, ph(h1, h1) = ph(h2, h2) = 0.9 in the stated scenario.
  • Transmission model: Packet transmission energy is modeled using Shannon capacity and a low-power approximation, with energy levels Eu = {1, 2, 4}.These levels correspond to transmit powers of 0.5, 1, and 2 mW.

A. ETD-problem

The ETD evaluation compares offline, online, learning theoretic, and greedy policies. Learning performance improves with iterations and approaches online performance, while greedy transmission remains inferior.

  • Evaluation setup: The evaluation averages 2,000 realizations of 100 random state transitions for γ = 0.9.It compares LP-relaxed offline optimization, offline branch-and-bound, online policy iteration, Q-learning, and a greedy algorithm.
  • Performance ordering: The performance ordering is LP-relaxed offline, offline, online, learning theoretic, then greedy.The LP relaxation upper-bounds offline optimization, offline upper-bounds online optimization, and online upper-bounds learning and greedy approaches.
  • Learning iterations: 99% of online performance is reached by the learning theoretic approach for NL > 2 · 10^5 TSs, compared with 85% for NL > 200 TSs.The learning configuration uses ϵ = 0.07.
  • Performance ordering: The greedy approach reaches only 60% of offline performance in the pH comparison.Its performance is described as notably inferior to the other approaches.
  • Harvested-energy and battery effects: For increasing pH, all approaches transmit more expected data; learning reaches 90% of online performance at pH = 0.5 and 99% at pH = 0.9.The online approach reaches 97% of offline performance at pH = 0.5 and 99% at pH = 0.9.
  • Harvested-energy and battery effects: With NL = 10^4 TSs, expected transmitted data increases with Bmax for all proposed algorithms except greedy, while learning reaches at least 91% of online performance.Online performance is approximately 99% of offline performance across the battery-size comparison.

B. TM-problem

The TM evaluation addresses average-reward optimization, using relative value iteration online and R-learning without established convergence guarantees. Throughput comparisons show learning approaching online performance as harvested energy increases, while greedy performance remains lower.

  • TM formulation: For γ = 1, the TM-problem replaces infinite expected total data with expected throughput maximization.The ETD criterion is impractical when operation can continue indefinitely and policies transmit infinite expected data.
  • Optimization methods: R-learning is used for the learning theoretic TM solution because Q-learning requires a finite horizon or absorbing states.The convergence properties of average-reward methods are not well understood.
  • Optimization methods: Relative value iteration is used to find the optimal online policy because policy evaluation in policy iteration is not guaranteed to converge for γ = 1.The TM evaluation compares LP-relaxed offline, branch-and-bound offline, RVI online, R-learning, and greedy algorithms over 2,000 realizations of 100 transitions.
  • Learning iterations: For Bmax = 5, learning reaches 95% of online throughput after NL > 200 TSs and 98% after NL > 2 · 10^5 TSs, without converging to online performance.The greedy algorithm is notably inferior to the other approaches.
  • Harvested-energy effects: Across pH values, online throughput averages 95% of offline throughput, while learning reaches 91% of online throughput at pH = 0.5 and 98% at pH = 0.9.Learning and online performance relative to offline performance increase as the Markov processes become less random.

VIII. CONCLUSIONS

The paper analyzes strict-deadline communication with Markov data, energy, and channel processes under offline, online, and learning-theoretic optimization. Q-learning approaches online performance asymptotically, while numerical results show substantial gains over greedy transmission.

  • VIII. CONCLUSIONS: The system model combines limited-capacity rechargeable energy harvesting, strict deadlines, stochastic Markov arrivals, and a time-varying channel.The objective is the ETD-problem: maximizing expected total transmitted data during the transmitter’s activation time.
  • VIII. CONCLUSIONS: Offline, online, and learning-theoretic optimization are studied under different assumptions about process information.The online and learning-theoretic formulations use MDPs to identify optimal transmission policies.
  • VIII. CONCLUSIONS: Q-learning reaches the optimal online performance as learning time tends to infinity, while online optimization is solved using policy iteration.The offline problem is characterized as a mixed integer linear program and solved using branch-and-bound and linear-program relaxation.
  • VIII. CONCLUSIONS: 90%: the learning-theoretic approach reaches 90% of online performance after a reasonably small number of learning iterations.This result is reported for practically relevant system parameters.
  • VIII. CONCLUSIONS: 60% up to 90%: smart, energy-aware policies achieve this range of offline performance compared with greedy transmission.Similar observations are reported for the throughput maximization problem, despite lacking theoretical convergence results.

APPENDIX

The appendix quantifies estimation errors and confidence intervals for discounted-sum data performance over finite horizons and finite numbers of MDP realizations. It also explains the throughput-problem analogue under stationarity.

  • APPENDIX: The discounted-sum data analysis estimates the expected performance quantity using transmitter actions and packet sizes across time slots.The action X_n is generated by offline, online, or learning-theoretic optimization, while D_n is the packet size.
  • APPENDIX: An upper bound is obtained by assuming maximum-size future packets, sufficient energy, and a discount factor satisfying 0 ≤ γ < 1.Under these assumptions, the finite-horizon error decreases exponentially with N.
  • APPENDIX: Finite-horizon error and finite-realization estimation error are treated separately before constructing confidence intervals.The sample mean over T MDP realizations is analyzed using the Central Limit Theorem and Tchebycheff inequality.
  • APPENDIX: For δ = 0.9, the numerical analysis computes confidence intervals for the discounted-sum data estimate.The interval uses a Student-t percentile and an estimated variance.
  • APPENDIX: For throughput maximization, stationarity permits treating sufficiently long-horizon expected throughput as equivalent to infinite-horizon throughput.Setting ε_N to zero makes its confidence-interval computation analogous to the discounted-sum data problem.
Loading 1208.4290v2…