PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Agent Reinforcement Learning for Movable Antenna-aided Cell-Free Massive MIMO Systems

Bokai Xu, Jiayi Zhang, Shuaifei Chen, Ziheng Liu, Huahua Xiao, Derrick Wing Kwan Ng, Bo Ai

arXiv:2610.08387v1cs.ITmath.IT

TL;DR

Non-convex minimum-separation constraints make joint movable-antenna positioning and transmission optimization difficult in cell-free MIMO. The paper proposes GLIIR-HAPPO, a heterogeneous MARL framework combining role-specialized optimization, geometric constraint handling, and coordinated learning. Simulations report faster convergence than distributed baselines and lower learning communication overhead with federated distillation.

  • Problem

    Joint antenna positioning and power control must address non-convex minimum-separation constraints in cell-free MIMO.

    Source: §I · pp. 1–2

  • Method

    GLIIR-HAPPO decomposes the joint design into heterogeneous agent subtasks, using penalty-based geometric handling, a dynamic-interaction graph critic, and role-conditioned federated distillation.

    Source: §I · p. 2

  • Results

    GLIIR-HAPPO converges faster and reaches higher steady-state reward than MAPPO and HAPPO, while federated distillation reduces per-round communication overhead by more than 80% relative to its non-FD counterpart.

    Source: §V · pp. 13–14

  • Takeaways & Limitations

    The proposed framework achieves performance comparable to centralized CPPO with decentralized online execution, and federated distillation lowers learning-related communication overhead.

    Source: §V · pp. 13–14

    The extrinsic reward omits movement delay, actuation energy, and CSI/control overhead, so these practical factors are not optimized by the reward formulation.

    Source: B. Heterogeneous Coordination Network · p. 7

Abstract

from arXiv · show

The inherent non-convex minimum-separation constraints introduced by movable antennas present a formidable challenge to the joint optimization of antenna positions and transmission strategies, rendering conventional methods computationally infeasible, particularly in large-scale cell-free massive multiple-input multiple-output (MIMO). In this work, we propose the graph-based learning individual intrinsic reward heterogeneous-agent proximal policy optimization (GLIIR-HAPPO) algorithm, a novel heterogeneous multi-agent reinforcement learning (MARL) framework that fundamentally overcomes this impasse by systematically decomposing the original coupled optimization into coordinated subproblems. To ensure tractability, we embed the non-convex geometric constraints into a penalty-augmented reward structure and develop a specialized geometric solver that enables the positioning agents to efficiently navigate the high-dimensional action space. Specifically, we propose an architecture featuring a dynamic-interaction graph critic for adaptive cross-role coordination, together with role-conditioned federated distillation that synchronizes same-role policies through compact actor-output statistics. Beyond architectural design, we establish a rigorous theoretical analysis that derives monotonic performance improvement bounds and establishes convergence guarantees for the proposed bi-level optimization. Numerical simulations demonstrate that our framework yields significant sum-rate improvements over state-of-the-art MARL schemes. Moreover, the performance of our advanced architecture closely approaches its fully centralized counterpart, while drastically reducing communication overhead.

I. INTRODUCTION

The paper targets joint movable-antenna positioning and power control in cell-free MIMO, where geometric constraints and coordination demands challenge scalable optimization. It proposes a heterogeneous MARL framework with role-specialized agents, geometric constraint handling, and coordinated learning.

I. INTRODUCTION

  • Cell-free networks coordinate distributed access points to improve network-wide throughput and user experience, but can require substantial fronthaul signaling and processing.Movable antennas offer flexible spatial positioning, while fixed-position antenna alternatives lack that flexibility.Source: §I · pp. 1–2
  • Prior movable-antenna optimization faces coupled, non-convex constraints and can impose computational or centralized information-exchange burdens.Centralized designs require CSI sharing among base stations, which can strain fronthaul bandwidth and latency.Source: §I · p. 2
  • The paper asks how a heterogeneous multi-agent framework can jointly handle antenna-positioning geometry and power-control interference.The cited motivation also identifies non-convex geometry, communication overhead, and learning instability as implementation challenges.Source: §I · p. 2
  • GLIIR-HAPPO decomposes joint antenna-position and transmission optimization into positioning and power-control agent subtasks, with centralized training and decentralized execution.The framework is intended to avoid monolithic centralized optimization.Source: §I · p. 2
  • The method addresses minimum-distance constraints with a penalty-augmented reward and a geometric solver for feasible, collision-free antenna locations.The solver helps positioning agents navigate a high-dimensional action space.Source: §I · p. 2
  • A dynamic-interaction graph critic and role-conditioned federated distillation support cross-role coordination and compact same-role policy exchange.The paper also reports trust-region monotonic-improvement and bilevel intrinsic-reward convergence results under stated assumptions.Source: §I · pp. 2–3

II. SYSTEM MODEL AND JOINT OPTIMIZATION PROBLEM

The paper models a cell-free MIMO downlink with movable antennas at distributed access points and formulates joint power-allocation and antenna-position optimization. The nominal objective is sum rate, subject to transmit-power, movement-region, and minimum-separation constraints.

II. SYSTEM MODEL AND JOINT OPTIMIZATION PROBLEM

  • Each of L distributed access points has N movable antennas and serves K single-antenna users over the downlink.The access points connect to a central processing unit through fronthaul links.Source: §II · p. 3
  • The position-dependent multipath channel determines array responses, while all access-point antennas jointly transmit to users on shared time-frequency resources.The precoder is treated as a fixed deterministic module rather than an additional MARL action.Source: §II · p. 3
  • The nominal design maximizes network sum rate by jointly optimizing downlink power allocation and movable-antenna positions under power, movement-region, and minimum-separation constraints.The paper identifies variable coupling and non-convex geometric constraints as challenges for conventional optimization.Source: §II · p. 4
  • Movement delay, CSI reacquisition, actuation energy, control power, positioning uncertainty, and finite movement resolution enter supplementary practical-effect models and sensitivity analysis.Effective rate and energy efficiency are supplementary metrics, not terms in the nominal sum-rate objective or MARL reward.Source: §II · p. 4

III. PROBLEM RECONSTRUCTION AND HETEROGENEOUS MULTI-AGENT MODELING

The section introduces the centralized-training, decentralized-execution heterogeneous multi-agent framework and previews its MARL formulation and design components.

III. PROBLEM RECONSTRUCTION AND HETEROGENEOUS MULTI-AGENT MODELING

  • The framework transforms the joint optimization into a MARL-compatible form and specifies agent structure, states, actions, and rewards.It uses centralized training with decentralized online execution.Source: §III · p. 4

A. Problem Reconstruction

The reconstruction separates the minimum-distance geometry from the master optimization using auxiliary positions and a quadratic penalty. Its geometric subproblem admits a finite candidate search based on boundary projections and circle intersections.

A. Problem Reconstruction

  • Auxiliary antenna positions and a quadratic penalty reformulate the original minimum-distance constraints within the optimization.The penalty coefficient is adjusted between episodes according to average minimum-distance violation, and increased when violation exceeds a feasibility threshold.Source: A. Problem Reconstruction · p. 5
  • Each access-point geometric subproblem is independent and can be solved in parallel, using an unconstrained physical-position update and auxiliary variables enforcing minimum separation.The resulting feasible region is the exterior of multiple exclusion disks.Source: A. Problem Reconstruction · pp. 5–6
  • The solver constructs candidate points on active exclusion-circle boundaries, including radial stationary points and intersections with other constraint circles.The candidate sets are built from the currently violated distance constraints and relevant circle boundaries.Source: A. Problem Reconstruction · pp. 5–6
  • The geometric optimum lies on the feasible-region boundary, reducing continuous optimization to radial projections and circle-circle intersections.If a radial projection is infeasible, the nearest feasible candidate occurs at an intersection endpoint on a remaining feasible arc.Source: A. Problem Reconstruction · pp. 5–6

B. Heterogeneous Coordination Network

GLIIR-HAPPO coordinates role-specialized AP agents through a dynamic graph critic and distinct intrinsic rewards. It combines decentralized online policies with sequential updates and role-conditioned federated distillation.

B. Heterogeneous Coordination Network

  • A dynamic-interaction graph critic coordinates agents, while a connected backbone prevents sparse attention pruning from isolating an agent during training.Top-κ pruning suppresses redundant non-backbone links while retaining weak connectivity.Source: B. Heterogeneous Coordination Network · pp. 6–7
  • The alternating optimization procedure solves auxiliary geometric variables first, then optimizes power allocations and antenna positions with those variables fixed.The second step holds the auxiliary variables constant while optimizing the original variables.Source: B. Heterogeneous Coordination Network · p. 6
  • Each AP uses power-control and positioning agents to learn task-specific policies from local observations while jointly optimizing a global objective.The two roles control downlink power allocation and movable-antenna locations.Source: B. Heterogeneous Coordination Network · p. 6
  • The learning scheme combines extrinsic team reward with each agent’s intrinsic reward and uses proxy advantages for sequential HAPPO policy updates.The extrinsic reward combines nominal sum rate with a geometric feasibility penalty.Source: B. Heterogeneous Coordination Network · pp. 6–8
  • The analysis gives a KL-constrained monotonic-improvement guarantee and sufficient conditions for two-timescale convergence, while treating PPO-Clip as covered only when its step meets KL budgets.The analysis is a KL-controlled stochastic-approximation abstraction, not a line-by-line proof of every implementation detail.Source: B. Heterogeneous Coordination Network · p. 8
  • Role-conditioned federated distillation synchronizes policies by exchanging compact actor-output statistics rather than full actor parameters or raw observations.Cross-role coordination remains through the shared extrinsic reward and dynamic graph critic.Source: B. Heterogeneous Coordination Network · p. 9

IV. THEORETICAL ANALYSIS OF THE PROPOSED FRAMEWORK

The analysis establishes a KL-controlled extrinsic improvement bound and convergence conditions for the graph-FD, bilevel learning procedure. Supporting results characterize proxy advantages, policy-gradient updates, and the LIIR meta-gradient.

Policy-gradient identity

  • The prefix-marginalized policy-gradient identity gives each agent’s update using its proxy return while marginalizing over other agents’ actions.The derivation conditions on other-agent actions before subtracting a baseline uncorrelated with the agent’s score function.Source: §IV · p. 10

Joint KL decomposition

  • For factorized multi-agent policies, the joint KL divergence decomposes additively into per-agent conditional KL divergences.The result also holds after taking expectation over any state distribution.Source: §IV · p. 10

Proxy advantages and shaping

  • Potential-based intrinsic shaping contributes only an action-independent baseline, so it does not bias the agent’s policy update.The proxy–extrinsic advantage gap is bounded by the discounted contribution of the intrinsic reward.Source: §IV · p. 11

Theorem 2: Extrinsic improvement

  • Theorem 2 bounds extrinsic performance improvement for KL-controlled HAPPO updates using graph-estimated proxy advantages and an FD-regularized direction.For PPO-Clip, the result applies to realized updates only when the per-agent KL budgets are met through monitoring and early stopping.Source: §IV · pp. 9, 11

LIIR meta-gradient

  • The LIIR meta-gradient is correct when trajectories come from the post-update policy or are reweighted by exact importance sampling.The outer objective evaluates extrinsic return under policies after an inner HAPPO pass.Source: §IV · p. 11

Theorem 3: Two-time-scale convergence

  • Under the stated stochastic-approximation conditions, the graph-FD iterates converge to stationary points of the limiting bilevel objective.With summable FD perturbations the limits are the original HAPPO-LIIR limits; with fixed FD regularization they are the FD-regularized limits.Source: §IV · p. 12

V. SIMULATION RESULTS

Simulations compare GLIIR-HAPPO and benchmark schemes across channel and network settings, evaluating convergence, sum rate, communication cost, and practical sensitivities. The results show advantages over FPA and AS, near-centralized performance, and reduced communication overhead with federated distillation.

Simulation setup

  • The simulation uses six APs and ten UEs, with four MAs per AP, ten propagation paths, and shared RZF precoding across benchmarks.Using the same deterministic RZF rule makes reported performance differences attributable to optimized power allocation and MA positioning.Source: §V · p. 12

Convergence

  • GLIIR-HAPPO converges faster and reaches higher steady-state reward than MAPPO and HAPPO under perfect CSI, while CPPO finishes slightly higher.GLIIR-HAPPO achieves comparable performance with decentralized online execution.Source: §V · p. 13
  • Under imperfect CSI, GLIIR-HAPPO maintains stable convergence, and federated distillation yields the highest average reward among distributed schemes.The reported channel-estimation error is ϵ = 0.05.Source: §V · p. 13

System sum-rate comparisons

  • Across four system configurations, MA methods outperform FPA and AS; their sum rate rises with more antennas or APs, while all schemes saturate as UE count grows.With increasing noise variance, all methods lose throughput, while CPPO maintains a clear performance margin.Source: Figure 5 · p. 13, §V · p. 14

Communication overhead

  • Adding federated distillation reduces GLIIR-HAPPO’s per-round communication overhead by more than 80% versus its non-FD counterpart and keeps it below MAPPO/HAPPO.FD exchanges compact policy knowledge rather than raw observations or high-dimensional critic messages.Source: Figure 6 · p. 13, §V · p. 14

Learned interaction graph

  • As training progresses, attention concentrates on a sparse AP-interaction backbone; a fixed connected backbone preserves connectivity while adaptive non-backbone links are pruned.The figure’s edge-count reduction represents pruning of adaptive non-backbone links.Source: §V · p. 13

Computational complexity

  • GLIIR-HAPPO inference scales as O(MPactor) for fixed actor size, while training retains O(M^2dk) attention cost; the AO power-update cost is O((LK)^3) per iteration.The inference estimate excludes channel acquisition, precoding, feasibility processing, signaling, and physical antenna movement.Source: Table III · p. 14, §V · p. 14

Practical sensitivity

  • Base-aware and EE-aware MA achieve 78.20 and 77.92 bit/s/Hz, respectively, versus 72.16 bit/s/Hz for the fixed array; robust MA has a 16.1% performance loss under pessimistic conditions.Reduced actuator speed, increased CSI/control overhead, and greater positioning error lower effective rate; CSI/control overhead also lowers energy efficiency.Source: Figure 7 · p. 14, §V · p. 14

VI. CONCLUSION

The paper concludes that role-specialized MARL makes joint antenna-placement and transmission optimization tractable under geometric constraints. It reports sum-rate gains and lower communication overhead, while identifying hardware-aware objectives and validation as future work.

Conclusion

  • The framework decomposes joint antenna placement and transmission into role-specialized subproblems, using variable splitting, quadratic penalties, and feasibility-aware reward shaping for minimum-distance constraints.A dynamic-interaction graph critic and role-conditioned federated distillation coordinate decisions with low communication overhead.Source: §VI · p. 14
  • Numerical results show substantial sum-rate gains over conventional FPA systems and near-centralized performance with significantly reduced communication overhead.The paper supports its stability claims with trust-region analysis and two-timescale meta-gradient convergence arguments.Source: §VI · pp. 14–15

Future work

  • Future work will incorporate movement delay, actuation energy, positioning uncertainty, finite resolution, and CSI/control overhead into hardware-aware objectives, then calibrate and validate through OTA or hardware experiments.Source: §VI · p. 15

A. Proof of Lemma 3

The proof characterizes which portions of a separation circle violate a neighboring distance constraint and identifies the relevant boundary points. The geometry reduces the cases to a single removed arc, its intersection endpoints, or no removal.

Arc removal

  • For 0 < d < 2D, the violated portion of a separation circle is one open arc; for d ≥ 2D or d = 0, no arc is removed.Here d is the distance between the two circle centers, and D is the minimum separation.Source: A. Proof of Lemma 3 · p. 15

Arc endpoints

  • The removed arc’s endpoints are intersections of the two circles and lie on the perpendicular bisector of their centers.The proof derives this by subtracting the squared circle equations.Source: A. Proof of Lemma 3 · p. 15

Boundary cases

  • At d = 2D the two intersection points coalesce at a tangent point; for d > 2D there is no intersection, while d = 0 follows the stated no-endpoints convention.Source: A. Proof of Lemma 3 · p. 15

B. Proof of Proposition 1

The proof establishes that the graph-attention critic’s per-agent values transform consistently when agents are permuted. Summing those values therefore preserves the global team value.

Permutation equivariance

  • Permuting agents permutes the row-stacked embeddings, while shared feature maps preserve the corresponding transformation through the attention computation.The proof applies permutation equivariance of row-wise TopK selection and softmax to the attention matrix and aggregated messages.Source: B. Proof of Proposition 1 · p. 15
  • The update layer preserves the permutation transformation by applying its weight matrix to row-stacked inputs and broadcasting the bias across agents.Source: B. Proof of Proposition 1 · p. 15

Value invariance

C. Proof of Lemma 5

The proof decomposes the performance difference into increments between partially marginalized value functions. It identifies each increment with a prefix-marginalized advantage and sums them to prove the claim.

  • The proof expresses the performance difference as a telescoping sum of increments between partially marginalized value functions.Source: C. Proof of Lemma 5 · p. 15
  • Each increment is the i-th prefix-marginalized advantage: fixing the first i actions and marginalizing the rest matches prefix sets Pi∪{i} versus Pi.Source: C. Proof of Lemma 5 · p. 15
  • Summing the increments over i proves the claim using conditional-expectation linearity and boundedness to interchange expectations under the regularity assumptions.Source: C. Proof of Lemma 5 · p. 16

D. Proof of Theorem 1

The proof reduces a joint expectation to marginal expectations by using that each KL-divergence summand depends only on one agent’s action. Summing over agents and averaging over states establishes the stated identities.

Loading 2610.08387v1…