PaperlayerView exact revision on arXiv

Source-linked AI summary

Decision Transformer for UAV-Mounted RIS-Assisted Dynamic D2D Communications

Yaxuan Liu

arXiv:2609.09885v1cs.AIcs.RO

TL;DR

The paper addresses joint control of a UAV-mounted RIS-assisted D2D system with stochastic links and changing geometric and channel conditions. It combines DRL experts with offline Decision Transformer pretraining and online adaptation, finding that zero-shot DT outperforms transferred DDPG while fine-tuning approaches expert performance with fewer interactions.

  • Problem

    The study addresses joint optimization of UAV trajectory, three-dimensional attitude, and RIS phases under stochastic links, time-varying Rician angles, and angle-dependent reflection.

  • Method

    A Decision Transformer is pretrained offline on high-quality DRL trajectories from multiple scenarios and fine-tuned online for unseen UAV-RIS deployments.

  • Results

    Zero-shot DT consistently outperforms DDPG policies transferred from neighboring scenarios, while fine-tuned DT achieves nearly the scenario-specific DDPG expert performance.

  • Takeaways & Limitations

    Multi-scenario offline training provides cross-scenario decision-making capability, and selected online fine-tuning preserves offline knowledge while adapting to new scenarios.

Abstract

from arXiv · show

This paper studies unmanned aerial vehicle (UAV)-mouted reconfigurable intelligent surface (RIS)-assisted device-to-device (D2D) communication with stochastic link activation. It models UAV motion and attitude, time-varying Rician angles, and angle-dependent RIS reflection. A joint optimization of UAV trajectory, attitude, and RIS phases is formulated to maximize average sum rate under mobility, energy, and hardware constraints. The problem is addressed using deep reinforcement learning and a Decision Transformer trained on expert trajectories from multiple scenarios. Results demonstrate effective cross-scenario generalization, with zero-shot transfer outperforming direct DRL transfer and online fine-tuning achieving competitive performance with fewer interactions.

I. INTRODUCTION

The paper models dynamic indoor D2D communication assisted by a UAV-mounted RIS, addressing limitations of fixed RIS placement and conventional DRL under changing environments. It jointly considers UAV motion, three-dimensional attitude, angle-dependent RIS responses, stochastic links, and cross-scenario Decision Transformer control.

  • Fixed RIS deployment limits adaptability to time-varying channels and link blockage in dynamic communication scenarios.
  • Prior studies often optimize UAV trajectories and RIS phases while simplifying UAV attitude, RIS geometry, or angle-dependent reflection effects.Existing models either assume stationary planar RISs, treat attitude as random jitter, or use simplified array and channel representations.
  • Conventional DRL policies can degrade in unseen communication environments and require costly online interactions for adaptation.
  • The proposed system jointly optimizes UAV trajectory, three-dimensional attitude, and RIS phases with stochastic D2D link activation and time-varying Rician angles.
  • A Decision Transformer is pretrained on high-quality DRL trajectories from multiple scenarios to support zero-shot control and efficient adaptation in unseen deployments.The framework accounts for angle-dependent RIS responses and the coupling between UAV pose, channels, and local incident angles.

B. Angle Dependent RIS Reflection Model

The practical RIS model converts preset element reflection properties into angle-dependent coefficients, producing a phase matrix that reflects propagation-specific incident angles.

  • The RIS phase-shift matrix is defined before applying the practical angle-dependent reflection model.
  • Each RIS element has a preset complex reflection coefficient that is transformed according to the incident angle.
  • The model captures variation in reflection amplitude and phase with electromagnetic incidence angle and outputs the actual coefficient for a propagation scenario.

C. Signal Model

The signal model represents received D2D signals through RIS-assisted transmission with concurrent interference, controllable transmit activity, and additive noise, enabling achievable-rate calculation.

  • The received signal at each D2D user is formulated from the established channel and RIS reflection models.
  • The binary coefficient β_j indicates whether D2U_j transmits in the current time slot, while users share identical transmit power P.
  • Concurrent transmissions create interference, with x_j and x_k denoting transmitted signals and n_i denoting receiver noise.
  • The achievable rate at D2D user DUi is formulated using the received signal model and noise power σ^2.

D. UAV Motion Model

The model updates UAV position using uniformly accelerated motion and represents attitude through yaw, pitch, and roll rotations that determine the RIS’s true electromagnetic incidence angles.

  • The UAV state includes three-dimensional position, velocity, and acceleration vectors for motion modeling.
  • q_U(t+1) = q_U(t) + v_U(t)Δt + 0.5a_U(t)Δt^2 updates position after one time slot.
  • UAV attitude is modeled with yaw ζ_y, pitch ζ_p, and roll ζ_r using three-dimensional rotation matrices.
  • The RIS normal and in-plane axes are rotated according to the UAV attitude to update their orientations.
  • The resulting incident angle is computed from the transmitter and UAV geometry and used to update Rician channels and practical RIS reflection coefficients.

III. PROBLEM FORMULATION

The paper formulates sum-rate maximization for UAV-mounted RIS-assisted D2D communications by jointly optimizing UAV motion, attitude, RIS reflection, and stochastic link activation under practical constraints.

  • The objective maximizes the sum achievable rate of multiple D2D pairs by jointly optimizing UAV trajectories, UAV attitudes, and RIS reflection coefficients.
  • The optimization includes UAV position, velocity, acceleration, three-axis attitude, RIS amplitude and phase limits, and binary D2D-link activation.β_k(t)=1 indicates that D2D link k is served; β_k(t)=0 indicates inactivity.
  • The formulation constrains total UAV propulsion energy to E_max while allowing flexible trajectory and attitude control across T time slots.

IV. DECISION TRANSFORMER OPTIMIZATION FRAMEWORK

The framework addresses the joint optimization problem with a generalizable Decision Transformer designed to adapt across UAV-RIS deployment scenarios.

  • The proposed framework combines MDP formulation, DRL-based expert selection, offline Decision Transformer pre-training, and online fine-tuning for unseen scenarios.

A. MDP Formulation

The system is modeled as an MDP whose states capture UAV, channel, and D2D conditions, whose actions control motion and RIS behavior, and whose reward drives cumulative-return optimization.

  • A. MDP Formulation: The MDP is defined by state space S, action space A, transition probability P, and reward function R.
  • A. MDP Formulation: The state contains UAV position and velocity, three attitude angles, the Rician fading matrix, and the D2D pair-state matrix.
  • A. MDP Formulation: The action controls UAV movement, RIS phase shifts, and incremental changes in the three UAV rotation angles.
  • A. MDP Formulation: After action A_t is applied in state S_t, the system transitions to S_{t+1} according to P(S_{t+1} | S_t, A_t).
  • A. MDP Formulation: The reward uses discounted future returns, and the optimal policy maximizes expected cumulative reward.
  • B. DRL-Based Expert Algorithm Selection: The framework selects a DRL expert by comparing PPO, DDPG, SAC, and TD3 under identical simulation settings.Convergence, average sum-rate performance, and policy stability are compared before selecting the best-performing algorithm.

C. Offline Pre-Training

Offline pre-training builds a multi-scenario Decision Transformer dataset from expert DRL trajectories and trains the model to predict expert actions from historical context and target returns.

  • The selected DRL algorithm is independently trained in M UAV-RIS scenarios to generate high-quality expert policies and trajectories.
  • Each expert trajectory includes a sequence of states, actions, and return-to-go values.The return-to-go at time t is denoted by R-hat and is calculated from future rewards.
  • During offline training, the Decision Transformer uses historical states, actions, and return-to-go values to predict the current action.
  • The model minimizes mean squared error between predicted actions and expert actions.
  • Multi-scenario pre-training gives the Transformer cross-scenario decision-making capability by learning links among states, target returns, and high-quality control actions.

D. Online Fine-Tuning

Online fine-tuning adapts a pre-trained Decision Transformer to an unseen UAV–RIS scenario by collecting candidate trajectories, retaining the highest-return trajectories, and minimizing action-prediction error on this elite set.

  • Zero-Shot Deployment: The pre-trained DT generates a zero-shot policy in a new scenario from the target return, observed state, updated RTG, and historical actions.Gaussian exploration noise is added to encourage exploration during online adaptation.
  • Elite Trajectory Selection: Candidate trajectories are evaluated by cumulative return, and only the top-K trajectories are retained as the elite fine-tuning set.Lower-performing trajectories are discarded rather than used for adaptation.
  • Overall Procedure: The framework pre-trains DT on DRL expert trajectories from multiple offline scenarios before performing online fine-tuning in an unseen scenario.Algorithm 1 separates offline pre-training from iterative online adaptation using candidate and elite trajectory sets.
  • Fine-Tuning Objective: The DT is fine-tuned by minimizing action-prediction error between recorded actions and DT-predicted actions on selected trajectories.The selected trajectories provide high-quality references for adapting the pre-trained model.
  • Adaptation Outcome: Repeated trajectory collection, quality-based selection, and fine-tuning adapt the model while retaining knowledge learned from the offline dataset.The procedure is intended to avoid degradation from low-quality online trajectories.

V. NUMERICAL RESULTS

The numerical study compares DRL algorithms for expert selection and evaluates DT generalization on unseen UAV–RIS initial positions. DDPG is selected as the expert, while zero-shot and fine-tuned DT outperform or approach transferred and scenario-specific baselines.

  • DRL Expert Selection: 29.5 bps/Hz: DDPG achieves the highest average sum rate, exceeding PPO at 25.0, TD3 at 20.5, and SAC at 17.0 bps/Hz.DDPG stabilizes after approximately 25,000 episodes in the compared training environment.
  • DRL Expert Selection: DDPG is selected as the DT expert after comparison with PPO, SAC, and TD3, and generates 500 trajectories in each of 8 training scenarios.The scenarios vary the UAV-mounted RIS initial horizontal position while fixing altitude at z = 4.5 m.
  • Unseen-Scenario Evaluation: The unseen-scenario evaluation uses initial UAV-mounted RIS positions [15, 15, 4.5] and [25, 25, 4.5], excluded from the offline DT training dataset.The horizontal axis reports evaluation episodes rather than training progress.
  • Generalization Results: Fine-tuned DT achieves nearly the same performance as the scenario-specific DDPG expert in the unseen scenario.The comparison evaluates policies at the two unseen initial positions.
  • Generalization Results: Zero-shot DT achieves a high average sum rate and consistently outperforms DDPG policies transferred from neighboring scenarios.Zero-shot DT and transferred DDPG show larger fluctuations because they are not directly optimized for the target scene.

VI. CONCLUSIONS

The paper formulates UAV-mounted RIS-assisted D2D control with stochastic link activation and develops an expert-data-driven Decision Transformer for generalizable adaptation. Zero-shot DT outperforms direct DDPG transfer, while fine-tuning approaches scenario-specific DDPG performance with fewer interactions.

  • System and Optimization: The system jointly models UAV motion, three-dimensional attitude, time-varying Rician LoS angles, and incident-angle-dependent RIS responses.The optimization controls UAV trajectory, attitude, and RIS phases to maximize average sum rate.
  • Decision Transformer Framework: A Decision Transformer is pre-trained on DDPG expert trajectories from multiple scenarios and adapted to unseen scenarios through zero-shot deployment and online fine-tuning.The framework uses expert trajectories as offline training data for generalizable control.
  • Main Findings: Zero-shot DT outperforms direct DDPG transfer, while fine-tuned DT approaches scenario-specific DDPG performance with fewer interactions.These results support expert-data-driven sequence modeling for generalizable and efficient UAV–RIS control.
Loading 2609.09885v1…