PaperlayerView exact revision on arXiv

Source-linked AI summary

A Mobile Agent-Based Hierarchical Reinforcement Learning Framework for Energy-Balanced Data Collection and Wireless Charging in WSN

Ali Heidaripour, Nastooh Taheri Javan

arXiv:2610.00264v1cs.NI

TL;DR

Uneven energy use near the base station and limited adaptability in existing mobile-agent strategies motivate this work. It proposes hierarchical reinforcement learning for sequential data collection and charging, and reports a 15% average network-lifetime increase over existing methods.

  • Problem

    Energy imbalance near the base station and fixed-rule mobile-agent strategies that adapt poorly to changing network conditions are central challenges.

    Source: §II · pp. 2–3

  • Method

    DCR-HRL uses hierarchical Q-learning for adaptive service scheduling and movement, with separate Q-tables updated through agent actions and rewards.

    Source: §I · p. 2, C. Learning Model · p. 14

  • Results

    15% average network-lifetime increase over existing methods is reported for the proposed solution.

    Source: §I · p. 2

  • Takeaways & Limitations

    Simulations report that the approach improves throughput, energy balance, and network lifetime.

    Source: §VII · pp. 24–25

    The single-agent framework demonstrates strong performance in small-to-medium networks; extending it to multi-agent deployments presents scalability and coordination challenges.

    Source: §VII · p. 25

Abstract

from arXiv · show

Energy imbalance remains a key challenge in Wireless Sensor Networks (WSNs), as nodes near the base station deplete their energy faster due to heavy forwarding loads. While mobile agents (MAs) have been employed for either data collection or sensor charging, existing approaches lack adaptability and fail to integrate both functions under realistic hardware constraints. This paper introduces a unified mobile agent framework that performs both data collection and wireless charging sequentially under single-antenna limitations. The agent's decision-making is formulated as a two-layer Hierarchical Reinforcement Learning (HRL) problem, where the upper layer optimizes movement planning and the lower layer determines the appropriate service based on real-time network states. This hierarchical structure enables the agent to learn adaptive task scheduling policies without predefined rules. Extensive simulations demonstrate that the proposed method achieves up to 15% longer network lifetime and more balanced energy distribution compared with state-of-the-art mobile agent and deep RL approaches.

I. Introduction

The paper addresses uneven energy depletion and limited adaptability in mobile-agent WSN strategies by proposing a hierarchical framework for sequential data collection and charging. Simulations report longer network lifetime, while the related-work discussion contrasts the approach with fixed-rule and specialized-agent methods.

I. Introduction

  • DCR-HRL uses two coordinated MDP layers: service selection responds to local energy and buffer states, while movement planning considers adjacent subdomains.The agent performs data collection and charging sequentially under single-antenna constraints.Source: §I · p. 2
  • 15% average network-lifetime increase over existing methods is reported for the proposed solution.The evaluation used extensive simulations against recent mobile-agent-based approaches.Source: §I · p. 2

II. Preliminary Exploration and Motivation

  • Nodes near the base station deplete energy faster under forwarding loads, and the inner-to-outer remaining-energy difference exceeds 40%.The paper defines network lifetime as the time until the first node exhausts its energy.Source: §II · pp. 2–3
  • A mobile agent can reduce multi-hop transmissions by collecting remote data and recharge heavily consumed nodes, motivating adaptive service scheduling.The agent is intended to choose between these tasks according to real-time network conditions.Source: §II · p. 3
  • Single-antenna systems cannot perform charging and data collection concurrently in the described setting, so practical service scheduling must be sequential.The paper attributes this constraint to interference and signal-separation difficulties.Source: §II · p. 3

III. Related Works

  • Prior mobile-agent work often addresses only collection or charging, while some integrated approaches use multiple specialized agents or classical optimization rather than adaptive single-agent scheduling.The paper also identifies predefined paths and service rules as limiting adaptation to changing network conditions.Source: §III · pp. 4–5

IV. System Model and Assumptions

The system model divides the network into subdomains where the mobile agent can stop and serve nearby sensors. Sensor communication uses multi-hop forwarding with energy, buffer, and range constraints.

A. Network Model

  • Each subdomain is a possible mobile-agent stationing location, with uniformly distributed nodes and a 10-meter service radius for up to five nodes.The model uses square-shaped subdomains.Source: A. Network Model · p. 5
  • Sensors remain static and forward packets through neighbors selected by proximity to the base station, falling back to the next listed neighbor if a hop dies.Each node generates 1–3 packets per timestep and has buffer capacity for 100 packets.Source: A. Network Model · p. 6

B. Communication Model

  • Transmission requires a reachable next hop with sufficient energy and buffer space; direct transmission to the mobile agent is usually more efficient when in range.A sender transmits until its buffer empties or its energy or the receiver's buffer becomes insufficient.Source: B. Communication Model · pp. 6–7

C. Energy Model

The energy model accounts for transmission, reception, and wireless charging costs, then updates sensor energy, buffers, and mobile-agent energy over time. These dynamics incorporate distance-dependent charging and stochastic energy harvesting.

C. Energy Model

  • Transmission energy is modeled as circuitry cost plus distance-dependent amplification, with path-loss exponent n depending on propagation conditions [50].The model gives n = 2 for free-space conditions and n > 2 for obstructed environments.Source: C. Energy Model · p. 7
  • Charging energy depends on transfer efficiency, agent power, distance, charging duration, and path loss.The reference distance d0 is typically 1 meter, and the charging duration is 8 seconds.Source: C. Energy Model · p. 7

D. Time-Dependent System State Updates

  • Sensor energy updates combine charging with consumption, while buffer updates account for collection and generated data during each interval.The energy update uses charging status and effective power based on distance and the number of simultaneously serviced nodes; the buffer update uses collection status, generated data, and per-node collection rate.Source: D. Time-Dependent System State Updates · p. 7
  • Mobile-agent energy changes with stochastic harvesting arrivals, service-operation costs, and distance-based movement costs.Harvest arrivals follow a Poisson process with rate λh and deliver fixed energy quanta.Source: D. Time-Dependent System State Updates · p. 7

E. Mobile Agent Model

The mobile agent alternates between charging and data collection under single-antenna constraints, with service capacity, coverage, and timing specified by the model. Each visit permits one service action, and system evolution accounts for service and movement time.

E. Mobile Agent Model

  • Under a single-antenna architecture, the agent performs charging and data collection sequentially, never both at one location.The paper states that simultaneous energy and data transmission can cause interference and signal collisions; sequential operation is used to avoid these problems.Source: E. Mobile Agent Model · pp. 7–8

1) Service Capacity and Coverage Specifications

  • The agent serves up to 5 nodes within a 10-meter radius, with 20×20 m² subdomains averaging 4 nodes and containing at most 7.Charging lasts 8 seconds per subdomain; collection lasts 6 seconds, with an 80 kbps aggregate collection rate.Source: 1) Service Capacity and Coverage… · p. 8
  • The model assumes environmental energy harvesting for the agent, with stochastic arrival timing and up to 12 joules transferred per charging operation.Data collection consumes approximately 0.8 joules per subdomain, and the agent may also return to the base station for recharging when necessary.Source: 1) Service Capacity and Coverage… · p. 8

2) Time Domain Operation Model

  • At each visit the agent chooses exactly one action; broadcast-based service affects all nodes in the subdomain, and simulation time counts movement and service durations.Charging takes 8 seconds, collection takes 6 seconds, and movement between adjacent subdomains takes 5 seconds.Source: 2) Time Domain Operation Model · pp. 8–9

F. Sensor Node Model

Sensors buffer packets for mobile-agent collection to reduce per-packet transmission and multi-hop costs. The paper models the agent's hierarchical decisions and notes limits to single-agent scaling in larger grids.

F. Sensor Node Model

  • Sensors delay transmission until a buffer threshold so the mobile agent can collect batches directly, reducing amplifier wake-up and multi-hop energy costs.The figure contrasts immediate transmission with buffered transmission, which batches packets and reduces multi-hop forwarding overhead.Source: F. Sensor Node Model · p. 9, Figure 2 · p. 9

G. Scalability Considerations

  • A single agent may not scale to large networks because greater distances between subdomains can lead to buffer overflow and energy depletion before visits.One agent covered the tested 5 × 5 grid with 100–500 nodes; multi-agent expansion could improve exploration but introduces coordination overhead, task conflicts, and load imbalance.Source: G. Scalability Considerations · p. 9

V. Analytical Models

  • The analytical model frames the agent's service and movement decisions as two interconnected subproblems solved with hierarchical Q-learning.The agent is described as collecting data from busy areas and charging low-energy sensors.Source: §V · pp. 9–10

A. Markov Decision Process Model

  • The HMDP organizes decision-making into sub-MDPs whose coordinated policies maximize cumulative reward; the paper's Q-learning solution is model-free and learns through interaction.The two levels choose whether to charge or collect data and where the agent should move next.Source: A. Markov Decision Process Model · p. 10

1) Decision Epochs

  • Decisions alternate between selecting a destination at the upper layer and choosing charging or collection after arrival at the lower layer.Each decision belongs to a separate phase of the hierarchy.Source: 1) Decision Epochs · p. 10

2) State Space

  • The upper-layer state represents the agent's location and neighboring-grid energy and buffer status, obtained from lightweight reports appended to forwarded packets.The lower-layer state aggregates current-grid average sensor energy and buffer occupancy; the reports incur 2.32 × 10^-6 J, or 0.8% of a 4000-bit packet, independently of link distance.Source: 2) State Space · pp. 10–11

3) Action Space

The action space separates movement from mutually exclusive charging or collection choices. Rewards combine service benefits and movement costs, with dynamic weighting coordinating the two decision layers.

3) Action Space

  • The upper layer selects among four movement directions, while the lower layer chooses exactly one of charging or data collection at each grid location.Charging prioritizes sensors with low energy; collection prioritizes sensors with higher buffer occupancy.Source: 3) Action Space · p. 11

4) Reward function

  • The upper-layer reward combines charging potential, collection potential, and movement cost, while the lower-layer reward accounts for energy charged, data collected, and penalties.The reward distinguishes charging near the base from collection far from the base, and applies a penalty to data collection near the base.Source: 4) Reward function · pp. 11–12
  • A global reward weights service and movement rewards, dynamically increasing service priority when energy is critical or buffers are high.Otherwise, the movement weight increases to give more importance to movement actions.Source: 4) Reward function · p. 12

B. System Dynamics

Q-learning learns the system dynamics implicitly through interaction rather than requiring explicit transition probabilities.

  • The mobile agent learns system dynamics through interaction with the environment, without requiring explicit transition probabilities.The paper describes the dynamics for completeness because Q-learning does not need them explicitly.Source: B. System Dynamics · p. 13
  • Q-learning’s model-free learning avoids the need to specify transition probabilities.Source: B. System Dynamics · p. 13
  • The described system dynamics are learned through agent-environment interaction.Source: B. System Dynamics · p. 13

1) State Evolution

The framework models changing energy, buffers, and sequential service time, then optimizes a composite policy across service and movement decisions.

1) State Evolution

  • Energy falls with transmission and rises with charging, buffers fill with generated data and empty upon collection, and charging and collection take sequential time.Source: 1) State Evolution · p. 13

2) Optimal Policy

  • The optimal composite policy pairs service and movement policies to maximize expected cumulative global reward.Source: 2) Optimal Policy · p. 13
  • The global reward combines phase-specific rewards using dynamic weights β1 and β2.Source: 2) Optimal Policy · p. 13
  • Alternating decision epochs couple the phases: each phase’s optimal policy depends on the other phase’s value function.Source: 2) Optimal Policy · p. 13
  • The coupled hierarchical policy jointly optimizes service and movement, adapting through β1 and β2 to balance energy provisioning and data collection.Source: 2) Optimal Policy · p. 13

3) Inter-Layer Coupling: Formal Analysis

The two decision layers are mutually coupled through their value functions, while dynamic reward weights separately set the global objective; the learning model uses two Q-tables and model-free updates.

3) Inter-Layer Coupling: Formal Analysis

C. Learning Model

  • Model-based hierarchical methods may require system information that is difficult to obtain, such as sensor shutdown or data-generation probabilities.Source: C. Learning Model · p. 14
  • Q-learning updates values from observed transitions and rewards without an explicit transition model, which suits uncertain environments with possible node failures.Source: C. Learning Model · p. 14
  • The agent maintains separate upper- and lower-level Q-tables, using neighboring energy and buffer states for movement and average levels for service choices.Upper-level actions are four cardinal movements; lower-level actions are charging or data collection.Source: C. Learning Model · p. 14
  • Exploration uses an ε-greedy policy whose ε decreases over time, with γ = 0.9 and α = 0.1 selected for long-term emphasis and stable convergence.The paper describes γ as emphasizing long-term network lifetime and α as supporting stable convergence while retaining adaptability.Source: C. Learning Model · pp. 14–15

1) Q-table Specification and Algorithm Scalability

The tabular design discretizes normalized energy and buffer states; the agent alternates service and movement decisions based on learned values and current network needs.

1) Q-table Specification and Algorithm Scalability

  • Energy and buffer variables are normalized to [0,1] and discretized into five bins, while positions remain discrete across 25 locations in a 5 × 5 grid.Source: 1) Q-table Specification and Algorithm… · p. 15
  • 13,750 Q-table entries require 110 KB, and the paper reports tabular Q-learning as tractable for this network size.The upper table has 12,500 entries and the lower table 1,250; larger deployments are identified as better suited to DQN or Actor-Critic methods.Source: 1) Q-table Specification and Algorithm… · p. 15

D. Mobile Agent Operation Algorithm

VI. Performance Evaluation

The evaluation uses simulations to assess DCR-HRL across network conditions and compare it with mobile-agent strategies. The setup holds the environment and performance metrics constant across methods while comparing distinct decision architectures.

VI. Performance Evaluation

  • The evaluation compares mobile-agent strategies across network conditions using network lifetime, throughput, end-to-end delay, and energy efficiency.Scenarios span 100–500 nodes and 25–125 KB/s data rates.Source: §VI · p. 15, 1) Compared Methods · p. 17

A. Simulation Setup

  • The simulation uses a 100×100 meter area divided into 25 service cells, with uniformly placed sensors and modeled energy, communication, and agent constraints.Scenarios are repeated with different random seeds.Source: A. Simulation Setup · pp. 15–16

1) Compared Methods

  • DCR-HRL’s hierarchical workflow alternates service selection and movement decisions, updating their respective Q-tables as the agent interacts with the network.The algorithm returns the trained Q-functions after the learning loop.Source: Figure 3 · p. 16, 1) Compared Methods · p. 17
  • Baselines include rule-based sequential scheduling, two specialized agents, deep-RL methods, and collect-only or charge-only policies.DQN-MAS jointly represents movement and service decisions, whereas DCR-HRL uses two coordinated tabular Q-tables; both share the environment, topology, metrics, and network-state information.Source: 1) Compared Methods · pp. 16–17

2) Simulation Parameters

The evaluation varies deployment and traffic conditions, defines lifetime and operational metrics, and examines learning convergence, discretization, computational cost, and scalability. Five-bin discretization converges fastest in the tested settings, while larger grids expose a single-agent coverage boundary.

2) Simulation Parameters

  • Simulation parameters vary network density from 100 to 500 nodes and data generation rates from 25 KB/s to 125 KB/s to test differing deployment and traffic conditions.The 5×5 grid provides service locations for agent navigation.Source: 2) Simulation Parameters · p. 17

3) Evaluation Metrics

  • Network lifetime is assessed with FND, HND, and LND, with LND defined at the point when only 30% of nodes remain alive.Throughput, end-to-end delay, and Energy Per Bit (EPB) measure delivery rate, delivery time, and energy per delivered bit, respectively.Source: 3) Evaluation Metrics · p. 18

1) Convergence

  • The convergence assessment tracks average reward over time; reward stabilizing indicates that the learned policy has converged and further training yields diminishing improvement.Reward comprises points earned through successful data collection, sensor charging, and network coverage.Source: 1) Convergence · p. 18, Figure 4 · p. 18

2) Discretization Sensitivity Analysis

  • Five bins converge fastest at episode 10, while 3- and 4-bin settings converge only at episodes 996 and 993; FND stays within a narrow band across converged configurations.Higher bin counts take progressively longer to converge without a lifetime benefit, and five bins are reported as the best trade-off at the evaluated network scale.Source: Table 3 · p. 18, 2) Discretization Sensitivity Analysis · p. 18

3) Comparison with Baseline Computational Cost

  • DCR-HRL trains in 78.0 s versus 94.8 s for DQN-MAS and converges in 10 versus 176 episodes, but inference takes 22.4 µs versus 4.5 µs and memory is 110 KB versus 4.05 KB.The comparison identifies a storage-for-stability trade-off; Table 4 compares training, inference, and memory across methods.Source: Table 4 · p. 19, 3) Comparison with Baseline… · p. 19

4) Scalability Beyond the 5×5 Setting

  • Q-table size grows from 26,950 to 220,000 entries across the tested 7×7–20×20 grids, while convergence episodes rise from 23.0 to 60.0.Training time rises roughly 10%, from 111.9 s to 123.2 s, despite an 8× increase in state-space size.Source: 4) Scalability Beyond the 5×5 Setting · p. 19
  • As grid size increases from 7×7 to 20×20, average FND declines from 233.4 s at 7×7 with 500 nodes to 101.5 s at 20×20 with 2,500 nodes.The paper attributes this decline to longer revisit times for a single mobile agent as the grid expands.Source: 4) Scalability Beyond the 5×5 Setting · p. 19
  • The paper recommends a 5×5 grid with 100–500 nodes for the single-agent architecture, with spatial coverage becoming the main limitation beyond that evaluated range.For 1,000 nodes, increasing grid size from 7×7 to 10×10 reduces average FND from 230.9 s to 115.3 s.Source: 4) Scalability Beyond the 5×5 Setting · p. 19

5) Energy Harvesting Rate Sensitivity Analysis

The section evaluates harvesting-rate sensitivity, single-antenna operation, network lifetime, delay, and throughput. Results describe how DCR-HRL performs across network conditions and identify boundaries in harvesting and spatial coverage.

5) Energy Harvesting Rate Sensitivity Analysis

  • At harvesting rates below 0.3 J/s, skipped services and critical-low agent energy are frequent; at 0.3 J/s and above, both diagnostics drop to zero.FND shows no detectable harvesting-rate effect in the tested 5×5, 300-node setting, while HND and LND improve up to 0.3 J/s.Source: 5) Energy Harvesting Rate Sensitivity… · p. 19, Table 6 · p. 20

6) Empirical Validation of the Single-Antenna Constraint

  • Sequential operation yields 79–99 dB SINR versus −22.7 to −2.7 dB simultaneously; at a 0 dB threshold, packet delivery is 0.45% versus error-free sequential reception.The reported SINR penalty for simultaneous operation is 101.8 dB, and delivery falls to zero at a 6 dB decoding threshold.Source: Table 7 · p. 20, 6) Empirical Validation of the… · p. 20

7) Network Lifetime

  • DCR-HRL outperforms Sequential, Multi-Agent, DQN-MAS, and Collect-Only in FND across node densities and traffic loads, while Charge-Only and MDDRL-JSSTC lead FND.DCR-HRL records 584.3±48 s at 100 nodes and 279.0±41 s at 500 nodes; across 25–125 KB/s it records 520.0 ± 32 s to 335.0 ± 12 s.Source: 7) Network Lifetime · p. 21, Figure 5 · p. 22
  • Increasing node density from 100 to 500 reduces network lifetime across all metrics as heavier forwarding accelerates energy depletion near the base station.Figure 5 compares FND, HND, and LND across node densities and data-generation rates.Source: 7) Network Lifetime · pp. 20–21, Figure 5 · p. 22
  • As data-generation rates rise from 25 to 125 KB/s, DCR-HRL remains ahead of Sequential, Multi-Agent, DQN-MAS, and Collect-Only in FND.Higher traffic increases transmission and forwarding activity; the reported DCR-HRL FND falls from 520.0 ± 32 s to 335.0 ± 12 s.Source: 7) Network Lifetime · p. 21, Figure 5 · p. 22

8) End-to-End Delay

  • DCR-HRL delay is U-shaped across node densities, reaching 6.9 ± 0.5 ms at 300 nodes, then rising to 27.1 ± 0.6 ms at 500 nodes.Across traffic loads, delay rises to 47.5 ± 0.8 ms at 125 KB/s, versus 33.4 ± 0.3 ms for Charge-Only and 32.9 ± 0.2 ms for MDDRL-JSSTC.Source: 8) End-to-End Delay · p. 21, Figure 6 · p. 22

9) Throughput

  • At 500 nodes, DCR-HRL has the highest throughput at 360 pps; its throughput peaks at 370 pps at 100 KB/s before declining to 365 pps at 125 KB/s.The paper attributes the later decline to congestion, faster energy depletion, and increased packet losses.Source: 9) Throughput · p. 23

10) Energy Consumption

Energy-per-bit results vary with network conditions, and DCR-HRL trades energy efficiency for sustained operation. The paper reports a joint throughput, energy-balance, and lifetime outcome while identifying scale and real-world evaluation as boundaries for future work.

11) Joint Optimization

  • DCR-HRL achieves the highest throughput among evaluated methods while remaining competitive, rather than dominant, on lifetime, delay, and energy efficiency.Charge-Only has the highest lifetime and lowest delay but no direct throughput from the agent.Source: 11) Joint Optimization · p. 24

10) Energy Consumption

  • At 200 nodes, DCR-HRL reaches its minimum EPB of 3.2e-5 J/bit, while MDDRL-JSSTC and Multi-Agent report 2.4e-5 to 4.4e-5 J/bit across densities.EPB rises beyond 200 nodes as congestion dominates; DCR-HRL’s higher EPB reflects a trade-off for network longevity.Source: 10) Energy Consumption · pp. 23–24, Figure 8 · p. 25
  • At 125 KB/s, DCR-HRL reaches 6.4e-5 J/bit, while MDDRL-JSSTC records the lowest EPB at 4.8e-5 J/bit; EPB increases with traffic for all methods.At 25 KB/s, MDDRL-JSSTC records 2.7e-5 J/bit, followed by Charge-Only at 2.8e-5 J/bit and Multi-Agent at 2.9e-5 J/bit.Source: 10) Energy Consumption · p. 24, Figure 8 · p. 25

VII. Conclusion and Future Work

  • The conclusion reports that DCR-HRL outperforms traditional models in throughput, balanced energy distribution, and network lifetime.The approach uses a multi-functional mobile agent that adaptively selects charging or data collection based on real-time network conditions.Source: §VII · pp. 24–25
  • DCR-HRL demonstrates strong performance in small-to-medium networks, while real-world evaluation across heterogeneous sensors, disturbances, and hardware limits remains future work.The conclusion also identifies multi-agent deployment and more advanced reinforcement-learning techniques as research directions.Source: §VII · p. 25
Loading 2610.00264v1…