Movable-Antenna Relay Enhanced Multiuser Communications: Reinforcement Learning-Based Antenna Positioning and Power Allocation

I. Introduction W

• Prior MA relay studies primarily considered single transceiver pairs, leaving multi-pair interference insufficiently addressed by existing solutions.The paper targets this gap by exploiting spatial reconfigurability in a multiuser relay system.Source: §I · p. 2
• The proposed system combines a half-duplex DF relay with relay-mounted MAs and a shared buffer to improve end-to-end transmission across multiple user pairs.The optimization maximizes the minimum average end-to-end achievable rate by jointly optimizing MA positions, power allocation, and beamforming matrices.Source: §I · p. 2
• The proposed DRL algorithm uses TD3 to optimize MA positions and power allocation, with beamforming matrices derived in closed form each slot.Source: §I · p. 2
• Simulations report higher average achievable rate than conventional FPA-based systems and improved performance from the buffer-aided relay scheme.Source: §I · p. 2

II. System Model and Problem Formulation

• The system uses a shared relay for K source-destination pairs; source and destination users have one fixed-position antenna each, while the relay has M movable antennas.Communication proceeds through relay reception and decoding in a multi-access stage, followed by re-encoding and broadcast.Source: §II · p. 3
• The transmission period is discretized into time slots, each divided into antenna movement, MAC, and BC subslots.A sufficiently large number of slots lets discrete MA positions approximate continuous trajectories.Source: §II · p. 3

A. Channel Model

• Quasi-static S-R and R-D channel vectors vary only with the MA positions, and the field-response model represents channels as superpositions of multiple paths.Source: A. Channel Model · p. 3
• The S-R receive field-response vector is defined from path arrival angles and the propagation distance difference between each relay MA and a reference point.The carrier wavelength is used in the distance-dependent response.Source: A. Channel Model · pp. 3–4
• The S-R transmit response and R-D transmit and receive responses are likewise constructed from path angles, user positions, and MA-dependent propagation distances.Source: A. Channel Model · p. 4
• The per-user field responses and path-response matrices form the S-R and R-D channel vectors, which are assembled into channel matrices for each slot.Source: A. Channel Model · p. 4

B. Signal Model

• During MAC, K source users transmit with a diagonal power allocation, and the relay receives their signals with complex AWGN.Source: B. Signal Model · p. 4
• The relay applies a receive combining matrix to the received signal to obtain user-specific combining outputs.Source: B. Signal Model · p. 4
• Under DF, the relay regenerates the decoded source signals and transmits them using a beamforming matrix during BC.The model assumes perfect decoding at the relay.Source: B. Signal Model · p. 4
• The model defines SINRs and achievable rates for both the S-R and R-D links.Source: B. Signal Model · p. 5

C. Relay Model

• Each slot first moves the MAs, then receives and decodes source data into the relay buffer, and finally forwards buffered data to destinations.Source: C. Relay Model · p. 5
• Source users transmit at their maximum rate when buffer space suffices; otherwise, they reduce rates to avoid exceeding remaining capacity.Source: C. Relay Model · p. 5
• The buffer occupancy is updated after MAC and again after BC, and each pair's end-to-end rate over the period is determined by its effective BC-stage transmission rate.Source: C. Relay Model · p. 5
• During BC, the relay transmits at the R-D achievable rate only when enough data is buffered; otherwise, it forwards all stored information.Source: C. Relay Model · p. 5

D. Energy Model

• Each antenna’s movement distance is the sum of its absolute displacements along the X-, Y-, and Z-axes.The model assumes stepper motors aligned with those three axes.Source: D. Energy Model · p. 5
• Movement energy is modeled as proportional to antenna movement distance, with η denoting energy consumed per unit distance.The model assumes the MAs move at constant speed vMA.Source: D. Energy Model · p. 5
• The relay allocates transmit power to destination users in the BC stage, and the corresponding transmission energy is included in the model.Source: D. Energy Model · p. 6

E. Problem Formulation

• The objective is to maximize the minimum average achievable rate among user pairs to ensure fairness.Source: E. Problem Formulation · p. 6
• The per-slot design jointly optimizes antenna positions, transmit and receive beamforming matrices, and power allocation under an energy trade-off between movement and transmission.Source: E. Problem Formulation · p. 6
• The constraints bound antenna positions and displacement, require at least half-wavelength inter-antenna spacing, limit source and relay transmit power, and impose a total movement-and-transmission energy budget over N slots.Source: E. Problem Formulation · p. 6
• Because the variables are highly coupled and the problem is non-convex, the authors propose a TD3-based two-layer algorithm for a suboptimal solution.The passage also identifies the long-term energy constraint and sequential decisions as challenges for traditional bio-inspired optimization.Source: E. Problem Formulation · p. 6

III. Proposed Solution

• The outer layer uses TD3 to optimize antenna positions and power allocation over N slots, while the inner layer obtains transmit and receive beamforming solutions in closed form per slot.Source: §III · p. 6

A. Partially Observable Markov Decision Process

• The DRL framework is represented as an MDP with state and action spaces, a reward, state-transition probabilities, and discount factor γ.Source: A. Partially Observable Markov… · p. 6
• The action includes MA movement and uplink/downlink power allocation, while the state includes MA positions, prior user rates, channel information, and available energy.Source: A. Partially Observable Markov… · p. 6
• The reward incorporates penalties for spacing and total-energy constraints that cannot be directly enforced through the action space; movement actions are clipped to satisfy the predefined-region constraint.Source: A. Partially Observable Markov… · p. 6
• The energy penalty permits temporary budget exceedance when movement can improve end-to-end performance, while encouraging energy savings when further movement yields limited rate improvement.Source: A. Partially Observable Markov… · p. 7

B. TD3-based Algorithm

• TD3 uses an actor and two critic networks, each with a target network, and stores state, action, and reward transitions in a replay buffer.The actor maps the state to an action, while the critics evaluate state-action pairs; target networks provide stable update targets.Source: B. TD3-based Algorithm · p. 7
• Gaussian exploration noise is added to the actor’s action and the result is clipped to the feasible action bounds for antenna movement and power allocation.Source: B. TD3-based Algorithm · pp. 7–8
• The critic networks are trained by minimizing squared temporal-difference errors, while the actor maximizes the first critic’s expected Q-value.Source: B. TD3-based Algorithm · p. 8
• The algorithm samples transitions from the replay buffer to update both critics, then updates the actor only after Ndelay critic updates and soft-updates the target networks.Source: B. TD3-based Algorithm · p. 8

• Buffer-aided relaying separates beamforming into S-R and R-D subproblems, corresponding to data arrival at the buffer and delivery to destination users.Source: C. Closed-form Beamforming · pp. 8–9
• For S-R receive combining, the paper adopts an MMSE receiver to maximize user-link rates and improve data arrival at the relay buffer.Source: C. Closed-form Beamforming · p. 9
• For R-D transmission, the paper uses an optimal beamforming structure but sets its Lagrange coefficients equally to relay power divided by user count as a heuristic.Source: C. Closed-form Beamforming · p. 9

• Environment interaction costs O{Fa + KM(Lsr + Lrd) + 4KM^2 + 2M^3} per time slot, covering actor propagation, channel calculation, and closed-form beamforming.Source: D. Computational Complexity Analysis · p. 9
• Per-slot TD3 learning includes target-network and critic updates, plus delayed actor updates, with the latter incurred every Ndelay slots.The passage gives the critic/target cost as O{Nbatch(Fa + 6Fc)} and the delayed actor-update cost as O{Nbatch(2Fa + 2Fc)}.Source: D. Computational Complexity Analysis · p. 9
• Training spans Nepisode episodes with N time slots per episode, and total computational complexity combines environment interaction and learning costs.Source: D. Computational Complexity Analysis · p. 9

A. Simulation Setup and Benchmark Schemes

• The setup uses a geometry channel model with distinct transmit and receive path components and defaults to three paths per link.Source: A. Simulation Setup and Benchmark… · pp. 9–10
• The default setting has four user pairs, six relay MAs, a 6λ cubic moving region, a 6 J total energy budget, and a 400-bit buffer.The relay array's maximum transmission power is 9 dBm; actor and critic networks use four and three fully connected layers, respectively.Source: A. Simulation Setup and Benchmark… · p. 10
• Benchmarks include sparse- and dense-UPA fixed-position arrays, fixed MAs placed using PSO, and a bufferless MA relay whose rate is limited by its bottleneck link.Source: A. Simulation Setup and Benchmark… · p. 10

B. Numerical Results

• The reward fluctuates during the first 200 episodes before converging to a stable value; critic Q-value loss and actor policy loss also eventually stabilize.The critic loss initially rises as the critic networks learn more accurate Q-values.Source: B. Numerical Results · p. 10, Figure 4 · p. 10
• The proposed MA scheme’s average achievable rate increases with normalized movement-region size and exceeds all benchmark schemes.The authors attribute the gains to antenna-position optimization and buffer-aided rate balancing, as well as reduced channel correlation across user pairs.Source: B. Numerical Results · p. 10, Figure 5 · p. 10
• The proposed MA method outperforms all benchmarks as the number of paths varies, and its performance gap over Fixed MA widens with more paths.The authors associate this widening gap with more effective channel reconfiguration in rich multipath environments.Source: B. Numerical Results · p. 11, Figure 6 · p. 11
• All schemes improve as relay antenna count increases, but the proposed scheme’s gain becomes less pronounced at sufficiently large counts, such as M = 10.At large counts, mobility energy consumption is a rate-energy tradeoff; the MA-without-buffer scheme can fall below Fixed MA and sparse and dense UPA benchmarks.Source: B. Numerical Results · p. 11, Figure 7 · p. 11
• The proposed scheme remains best across user-pair counts, although its achievable rate falls by 27.6% relative to the two-pair case.The passage attributes the decrease mainly to growing inter-user interference and notes that increasing rate imbalance makes buffer coordination more challenging.Source: B. Numerical Results · p. 11, Figure 8 · p. 11
• MA position optimization significantly reduces and balances user-pair channel correlations on both S-R and R-D links, mitigating interference and improving spatial multiplexing.Fig. 9 compares the initial fixed relay positions with correlations averaged over 1000 time slots under the proposed optimization.Source: B. Numerical Results · p. 11, Figure 9 · p. 12

V. Conclusion

• Simulations show the MA relay system outperforms conventional FPA relays and other MA benchmarks, while buffering alleviates rate imbalance between the S-R and R-D links.The system uses a half-duplex DF relay with an MA array and buffer.Source: §V · p. 11, Figure 9 · p. 12