Quantifying the Reality Gap for RL-Based UAV Placement at mmWave and Sub-THz

I. Introduction

• Prior RL placement studies often train and evaluate within one simulator, leaving disagreement with physics-grade channels—and its effect on transfer—unmeasured.Earlier placement analyses typically assume frequency-flat channels, which do not extend cleanly to sub-THz absorption structured by atmospheric resonances.Source: §I · p. 1
• The benchmark quantifies reality gaps for an analytically trained RL placement policy across four carriers and four altitudes on a real urban map.It measures bias, RMSE, Jensen–Shannon divergence, optimum-deployment displacement, and linear-domain regret.Source: §I · pp. 1–2
• A deterministic-LoS pipeline reuses Sionna’s mesh under closed-form path gain to separate Monte-Carlo sampling effects from physics disagreement.The study characterizes transfer as sampling-dominated at 28/140 GHz, absorption-model dominated at 183 GHz, and structurally different despite small stochastic bias at 300 GHz.Source: §I · p. 2

II. System Model

• A single UAV at fixed altitude serves uniformly distributed ground users with isotropic single-antenna terminals, optimizing aggregate signal quality separately for each carrier.The placement objective is fairness-agnostic, and the optimal hover position can vary with carrier because blockage and absorption are carrier-dependent.Source: §II · p. 2

A. Geometry and Building Model

• The deployment region is central Doha, discretized into 10 m cells, with a building-height map defining locations where the UAV can safely hover.The building map is derived from a triangular mesh preprocessed into merged axis-aligned cuboids; a location is deployable only when its building height is below the UAV altitude.Source: A. Geometry and Building Model · p. 2

B. Three Channel Pipelines

• The analytical pipeline combines free-space path loss, molecular absorption, and cuboid-based LoS blockage, assigning zero gain to non-LoS cells.Its absorption uses the Kokkoniemi–Lehtomäki fit, while its LoS test uses merged axis-aligned cuboids.Source: 1) Analytical pipeline (M) · p. 3
• Stochastic Sionna RT traces Monte-Carlo paths on the closed mesh with ITU-R P.676-13 absorption, while the deterministic-LoS pipeline reuses that geometry without sampling variance or multi-bounce reflections.The deterministic pipeline provides an ablation point for separating geometry/physics disagreement from sampling effects.Source: 3) Deterministic-LoS pipeline (L) · p. 3, 2) Stochastic ray-traced pipeline (S) · p. 3
• The per-position aggregate path loss represents mean linear gain to uniformly distributed users and is calibrated identically across pipelines.Because the aggregate path loss is invariant to transmit power, pipeline disagreement reflects channel modeling rather than link-budget choices.Source: 4) Per-position aggregate path loss · p. 3

C. Per-Position SNR and Deployment Distribution

• The same position-distributed SNR formulation is applied across channel pipelines to compare their spatial signal-quality distributions.The per-position receive SNR is defined for each pipeline.Source: C. Per-Position SNR and Deployment… · p. 3
• Per-position receive SNR combines transmit and antenna gains, aggregate path loss, and the carrier-dependent noise floor.The noise floor is thermal noise plus bandwidth and receiver noise figure.Source: C. Per-Position SNR and Deployment… · p. 3
• Treating UAV position as uniform over the safe deployment set produces an empirical position-distributed SNR for the reality-gap analysis.This distribution is the analysis’s central object.Source: C. Per-Position SNR and Deployment… · p. 3

D. UAV Deployment as a Markov Decision Process

• Deployment is a finite-horizon MDP on a discretized grid, with deterministic transitions and discount factor γrl = 0.995.The state includes normalized UAV position, four-direction occlusion distances, local signal quality, and remaining time.Source: D. UAV Deployment as a Markov Decision… · p. 3
• The seven actions are hover and movement along ±x, ±y, and ±z.The action set has cardinality 7.Source: D. UAV Deployment as a Markov Decision… · p. 3
• The reward combines weighted signal and coverage quality with movement shaping and a terminal quality bonus.The quality proxy is qt = 0.6st + 0.4ct; the shaping and terminal weights are α = 0.5 and β = 5, with horizon T = 500.Source: D. UAV Deployment as a Markov Decision… · p. 3

A. RL Training

• Each carrier has an independent PPO agent trained on the analytical channel using Stable-Baselines3.Training uses eight parallel environments, 3×10^6 environment steps, and a 2×128-tanh MLP policy.Source: A. RL Training · p. 3
• Training uses a linearly annealed learning rate from 3×10^-4 to 1×10^-5, GAE λ = 0.95, clip range 0.2, and mini-batches of 512.The entropy coefficient is 0.01.Source: A. RL Training · p. 3
• The policies are evaluated under all three channel pipelines without further fine-tuning.Each policy converges in approximately four hours on a single RTX-class GPU.Source: A. RL Training · p. 4

B. Sionna Pre-computation

• Sionna maps are precomputed on a stride-5 sub-grid and bilinearly interpolated, yielding about 1.6×10^4 deployable positions per altitude.ITU-R P.676-13 absorption is applied per ground-cell distance after ray tracing; precomputation takes approximately three hours per carrier on a Colab L4 GPU.Source: B. Sionna Pre-computation · p. 4

C. Sampling-Induced Bias and the Deterministic-LoS Construct

• At 100 m altitude, the sampling model gives a critical distance of about 430 m; the well-sampled disk covers about 7% of the served area.Empirical LoS coverage is approximately 0.06 for stochastic Sionna versus 0.70 for the analytical model; deterministic LoS tests each ground-cell center against the scene and builds path gain without Monte-Carlo variance.Source: C. Sampling-Induced Bias and the… · p. 4
• Four reality-gap metrics measure mean dB bias, position-wise RMSE, Jensen–Shannon divergence, and displacement between each pipeline's optimum deployments.A linear-domain regret ratio converts optimum-deployment displacement into an operational SNR cost.Source: D. Reality-Gap Framework · p. 4
• The dB metrics and regret are invariant to the link-budget offset, so pipeline comparisons reflect channel modeling rather than link-budget choices.Optimum-deployment displacement is measured in metres.Source: D. Reality-Gap Framework · p. 4

E. Decomposition into Sampling and Physics Components

• The bias decomposition separates a sampling component from residual disagreement between the analytical model and deterministic ray-traced reference.The sampling term reflects Monte-Carlo undersampling because the stochastic and deterministic targets share geometry and absorption physics, differing in sampling and reflections.Source: E. Decomposition into Sampling and… · p. 4
• This decomposition diagnoses whether a carrier calls for a sampling upgrade or a physics upgrade.The distinction is measured from the empirical S–L comparison.Source: E. Decomposition into Sampling and… · p. 4

• Unless stated otherwise, results use altitude z = 100 m, where |Pz| ≈ 1.39×10^4.Source: A. Setup · p. 4
• The link budget assumes Pt = 30 dBm, Gt = Gr = 0 dBi, η = 8 dB, and T0 = 290 K.Source: A. Setup · p. 4
• Bias, RMSE, DJS, and ρ are link-budget-invariant; only absolute SNR values shift with the link-budget offset.Bandwidth B(fc) is 200, 1000, 2000, and 5000 MHz at 28, 140, 183, and 300 GHz, respectively.Source: A. Setup · p. 4

B. LoS Coverage

• At 100 m, stochastic Sionna's mean LoS fraction is 0.057 versus 0.700 for the analytical model; deterministic-LoS raises it to 0.283 but remains below the analytical value.The remaining coverage gap reflects geometric differences between MATLAB's merged-cuboid model and Sionna's closed mesh; the corresponding 28 GHz path-loss bias is only about 1.7 dB because affected LoS cells are distant.Source: B. LoS Coverage · pp. 4–5

C. Position-Distributed SNR

• At 100 m, Fig. 2 compares empirical SNR CDFs across carriers, while Fig. 3 maps spatial SNR and marks each pipeline's argmax.Source: C. Position-Distributed SNR · p. 5
• At 28/140 GHz, ΦS is ≈5–6 dB below ΦM, while ΦL is only ≈1.5–1.7 dB below, indicating most of the apparent gap is sampling-induced.Source: C. Position-Distributed SNR · p. 5
• At 183 GHz, ΦS and ΦL are −13.2 and −9.2 dB relative to ΦM, respectively, with JS divergence at ln 2 ≈ 0.693 for both.The saturated divergence indicates practically disjoint SNR distributions.Source: C. Position-Distributed SNR · p. 5
• At 300 GHz, ΦS has bias +0.5 dB and DJS = 0.046, while ΦL exposes a +3.8 dB structural offset and DJS = 0.62.The stochastic agreement with the analytical model is described as coincidental rather than physical.Source: C. Position-Distributed SNR · p. 5

D. Reality-Gap Quantification

• At 28/140 GHz, biasS is −5.58/−4.80 dB and deterministic-LoS mitigation yields biasL of −1.69/−1.47 dB.The decomposition attributes about 3.9/3.3 dB to Monte-Carlo undersampling, with about 1.5 dB residual geometric/material difference; at 28 GHz, displacement contracts from 768 m to 230 m and regret improves from 0.850 to 0.956.Source: Table II · p. 5, D. Reality-Gap Quantification · p. 5
• At 183 GHz, biasL remains −9.2 dB and JS divergence remains saturated at ln 2 after mitigation, as the analytical absorption line under-predicts the ITU-R P.676-13 resonance peak and wings.The two SNR distributions are practically disjoint; optimum displacement contracts only from 864 m to 768 m.Source: Table II · p. 5, D. Reality-Gap Quantification · p. 5
• At 300 GHz, stochastic Sionna’s +0.50 dB bias and JS = 0.046 mask a +3.84 dB deterministic-LoS bias and JS = 0.615.The apparent agreement reflects cancellation between a −3.3 dB sampling component and a +3.8 dB physics component; spatial maps show speckle in M absent from both ray-traced pipelines.Source: Table II · p. 5, D. Reality-Gap Quantification · p. 5

E. Implications for RL Policy Transfer

• Across carriers, ρL is at least 0.93 while optimum displacement ranges from 198 m to 864 m, so deployment value transfers more reliably than location.The analytical optimum costs no more than about 7% in linear SNR, or 0.3 dB, relative to the deterministic ray-traced optimum.Source: E. Implications for RL Policy Transfer · p. 5
• For placement, the analytical channel is sufficient at 28/140 GHz, an absorption-grade upgrade is needed at 183 GHz, and 300 GHz remains adequate despite structural inequivalence.At 28/140 GHz, residual bias is about 1.5 dB and regret is at least 0.95; at 183 GHz, LoS modeling alone does not close the gap; at 300 GHz, regret is 0.967.Source: E. Implications for RL Policy Transfer · pp. 5–6

V. Conclusion

• At 28/140 GHz, deterministic-LoS recomputation removes the dominant Monte-Carlo undersampling bias; at 183 GHz absorption-line bias persists, while at 300 GHz stochastic agreement conceals a structural offset.The optimum-deployment displacement is hundreds of metres, but linear-domain regret remains at least 0.93; the authors say policies trained on simplified channels remain useful but warrant ray-tracing fine-tuning for line-adjacent carriers.Source: §V · p. 6
• The analysis quantifies reality gaps for RL UAV placement across four carriers spanning mmWave to sub-THz on a real urban map using a reproducible Sionna pipeline.The study is limited to one city, single-seed stochastic runs, static single-UAV placement, isotropic antennas, a fairness-agnostic mean-gain objective, and D=1 specular ray tracing without diffraction; future work targets multi-UAV settings and differentiable fine-tuning.Source: §V · p. 6