PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepTelecom: A Differentiable Multimodal Framework for Channel Twinning in Dynamic Heterogeneous Wireless Environments

Bohao Wang, Tianen Yuan, William Yen, Zehua Jiang, Chongwen Huang, Li Wei, Chen Zhu, Jiguang He, Hang Zou, Lina Bariah, Brahim Mefgouda, Anis Bara, Yongliang Shen, Zhaoyang Zhang, Merouane Debbah

arXiv:2610.14545v1cs.ITmath.IT

TL;DR

Existing datasets provide limited reusable high-fidelity scenes, scalable differentiable ray tracing, and synchronized multimodal data, while dense real-world channel measurements are costly and labor-intensive. DeepTelecom constructs material-aware LoD3 twins from coarse scenes and uses differentiable GPU ray tracing, sparse-measurement calibration, and synchronized multimodal generation for dynamic wireless settings. The 90th-percentile error decreased from 31.05 m to 2.43 m.

  • Problem

    Existing datasets provide limited reusable high-fidelity scenes, scalable differentiable ray tracing, and synchronized multimodal data, while dense real-world channel measurements are costly and labor-intensive.

    Source: §I · pp. 1–2

  • Method

    DeepTelecom constructs material-aware LoD3 twins from coarse scenes and uses differentiable GPU ray tracing, sparse-measurement calibration, and synchronized multimodal generation for dynamic wireless settings.

    Source: §I · p. 2

  • Results

    The 90th-percentile error decreased from 31.05 m to 2.43 m.

    Source: §X · p. 12

  • Takeaways & Limitations

    Users can generate customized data for their own scenes and system configurations for applications including UAV identification, localization, tracking, beam prediction, and channel estimation.

    Source: §X · p. 12

Abstract

from arXiv · show

Large-scale multimodal data are a key enabler for integrating large generative artificial intelligence models into wireless networks. However, existing wireless datasets offer limited support for heterogeneous environments, particularly in terms of reusable high-fidelity scenes, scalable differentiable ray tracing, and synchronized multimodal data. These limitations are more pronounced in mobile and aerial scenarios with time-varying environments and channels. To address these limitations, we propose DeepTelecom, a material-aware, high-fidelity multimodal channel dataset and reproducible generation framework for networks involving unmanned aerial vehicles (UAVs) and reconfigurable intelligent surfaces. DeepTelecom refines coarse indoor/outdoor reconstructions and uses a large language model to map segmented surfaces to frequency-dependent electromagnetic parameters, yielding material-aware Level of Detail 3 digital twins. Then, a differentiable GPU-accelerated Sionna ray tracer simulates propagation across the sub-6 GHz/millimeter-wave/sub-terahertz bands, and specifically captures rotor-induced micro-Doppler signatures of stationary or moving UAVs at different pitch angles and translational velocities. The framework synchronously produces channel tensors, coverage maps, transmission videos, multi-view images, and weather-conditioned renderings, and supports differentiable material calibration from sparse channel anchors. The released dataset contains 10 material-aware indoor/outdoor scenes and 50,000 synchronized UAV samples; the framework supports scalable generation of additional data for user-defined scenes and system configurations. In 3D UAV localization, DeepTelecom reduces the 90th-percentile error from 31.05 m to 2.43 m compared with schemes considering only channel state information, demonstrating its significance for artificial-intelligence-enabled wireless systems.

I. INTRODUCTION

AI-driven wireless tasks need large-scale datasets that jointly represent radio channels and physical environments, but existing resources remain fragmented in scene fidelity, scalable calibration, and reusable synchronized data. DeepTelecom combines material-aware indoor and outdoor digital twins, differentiable ray tracing, dynamic UAV modeling, and synchronized multimodal products.

I. INTRODUCTION

  • Existing datasets lack reusable material-segmented LoD3 indoor/outdoor twins, synchronized visual and channel data, and scalable surface-level electromagnetic semantics.Reconstructing LoD3 scenes and manually assigning their electromagnetic properties is labor-intensive and requires expert knowledge.Source: §I · pp. 1–2
  • Differentiable calibration and scalable ray tracing remain fragmented: GPU methods largely target site-specific tasks, while CPU and commercial workflows constrain large-scale generation.These constraints limit differentiably calibrated channel generation across collections of LoD3 scenes.Source: §I · p. 2
  • DeepTelecom turns heterogeneous 3D sources into reusable material-aware LoD3 twins with editable, frequency-dependent surface parameters.An LLM maps segmented surfaces to electromagnetic properties across sub-6 GHz, mmWave, and sub-THz bands.Source: §I · p. 2
  • Its differentiable GPU ray tracer supports RISs, mobile transmitters, and UAV motion, including rotor-induced micro-Doppler; sparse-measurement calibration tunes material parameters.The calibration loop is fixed-topology, robust, and identifiability-aware, and improves path-loss and delay-spread agreement on held-out synthetic anchors.Source: §I · p. 2
  • 31.05 m to 2.43 m: fusing CSI with camera and path-loss images reduces the 90th-percentile UAV 3D localization error relative to CSI alone.The dataset synchronizes channel tensors, path-resolved tables, coverage maps, propagation videos, multi-view images, and weather-conditioned renderings.Source: §I · p. 2

II. FRAMEWORK OVERVIEW

DeepTelecom builds channel datasets through six stages, from scene initialization and material parameterization to dynamic scene elements, ray tracing, calibration, and visualization. Outdoor scenes begin with open geospatial models that are refined into editable terrain and buildings.

II. FRAMEWORK OVERVIEW

  • The six-stage framework initializes coarse indoor/outdoor geometry, assigns frequency-dependent surface materials, adds RIS, mobile-transmitter, and UAV dynamics, then performs ray tracing, calibration, and visualization.The final products are organized under a unified scene schema.Source: §II · p. 3
  • GPU rendering produces snapshots, transmission videos, coverage maps, and weather-conditioned renderings after optional calibration.Calibration minimizes robust simulated-versus-measured path-loss loss and backpropagates through the field computation while holding ray topology fixed.Source: §II · p. 3

III. INITIALIZING THE DIGITAL TWIN: COARSE SCENE ACQUISITION

  • Indoor and outdoor coarse geometries share an interface for refinement and material parameterization in the next stage.The outdoor pipeline uses open geospatial data, while the indoor pipeline reconstructs from LiDAR scans.Source: §III · p. 3

A. Outdoor Coarse-Scene Reconstruction

  • Google 3D Tiles models are exported for SketchUp refinement, where key structures are rebuilt, categorized, and assigned realistic material names.The resulting outdoor twin includes terrain and buildings and is reconstructed from GPS coordinates.Source: A. Outdoor Coarse-Scene Reconstruction · p. 3

B. Indoor Coarse-Scene Reconstruction

Indoor reconstruction converts scans into a shared-coordinate scene model, while the material-aware stage adds surface semantics to coarse geometry. Blender cleans and labels the resulting models before they are parsed by the material and ray-tracing modules.

B. Indoor Coarse-Scene Reconstruction

  • Indoor scenes are reconstructed at LoD3 granularity, with scans aligned into a common coordinate system and aggregated to reveal structures such as walls and furniture.Point clouds are converted into polygonal meshes, simplified while preserving essential detail, and exported in standard formats.Source: B. Indoor Coarse-Scene Reconstruction · pp. 3–4

IV. PARAMETERIZING THE DIGITAL TWIN: FROM COARSE GEOMETRY TO MATERIAL-AWARE SURFACES

  • The material-aware stage segments coarse geometry into named surfaces and assigns frequency-dependent electromagnetic parameters, differentiating it from LoD1 corpora with limited material semantics.Source: §IV · p. 4

A. Region Segmentation and Labeling via Blender

  • Blender repairs and simplifies geometry, assigns descriptive material names, preserves textures, and exports a structured XML scene representation parsed by both the LLM material module and Sionna ray tracer.Source: A. Region Segmentation and Labeling… · p. 4

B. LLM-Assisted Material Assignment

The LLM-assisted assignment process maps scene object labels to frequency-dependent electromagnetic material properties and checks their consistency. DeepTelecom also configures RIS panels, moving transmitters, and rotating UAVs before ray tracing to shape propagation.

B. LLM-Assisted Material Assignment

  • The LLM maps object names to material-database entries, assigns frequency-dependent permittivity, permeability, conductivity, and roughness, and checks for inconsistent neighboring materials.The database is compiled from IEEE standards and literature, and complex materials can use multilayer models.Source: B. LLM-Assisted Material Assignment · pp. 4–5
  • Material properties determine complex permittivity and Fresnel reflection, while rough surfaces attenuate coherent specular reflection using a Rayleigh roughness factor.The ray tracer's effective reflection coefficient incorporates the roughness adjustment.Source: B. LLM-Assisted Material Assignment · p. 5

V. SUPPLEMENTARY SCENE ELEMENTS: RIS, MOBILITY, AND UAV

  • RIS panels add controlled reflection, mobile transmitters add time variation, and rotating UAVs add micro-Doppler modulation before ray tracing.Source: §V · p. 5

A. RIS

The framework adds RIS beam steering and moving-terminal channel dynamics, then models UAV echoes to capture rotor-induced micro-Doppler alongside translational Doppler.

A. RIS

  • RIS panels support single-beam focusing through a path-length-based phase gradient and multibeam codebooks optimized to approximate a desired radiation pattern.The phase gradient steers toward a target direction; multibeam optimization is intended for serving multiple users simultaneously.Source: A. RIS · p. 5

B. MT Mobility

  • Moving-terminal velocity varies the channel response and produces a time-varying CIR, while each path receives a Doppler shift determined by relative speed, wavelength, and path angle.The per-path shift captures translation, not the faster modulation from a spinning rotor.Source: B. MT Mobility · p. 5

C. UAV Modeling and Micro-Doppler

  • Rotor snapshots update UAV body and blade scatterers; weighted round-trip channel coefficients plus noise form an echo whose short-time Fourier transform yields a scene-aware micro-Doppler signature.Pitch changes the rotor-plane geometry, while rotor frequency determines periodic blade motion.Source: C. UAV Modeling and Micro-Doppler · pp. 5–6

VI. RT ENGINE AND CHANNEL EXTRACTION

A differentiable Sionna RT engine traces signals through populated 3D scenes and extracts channel products. Its path records support CIR, CFR, dispersion statistics, and coverage grids.

VI. RT ENGINE AND CHANNEL EXTRACTION

  • Sionna RT combines high-performance ray tracing with differentiability, enabling gradient-based scene-parameter optimization while tracing devices, RIS panels, and mobile terminals together.The engine models propagation through the detailed 3D scene and captures multipath effects in a single pass.Source: §VI · p. 6
  • Fibonacci sphere sampling and adaptive importance sampling distribute rays toward receiver-likely regions, while GPU-parallel OptiX tracing sustains tens of millions of rays per scene.Rays terminate at a maximum interaction depth or power threshold; retained-path field computation is differentiated with respect to continuous scene parameters.Source: §VI · p. 6

B. Channel Data Products

  • Each transmitter–receiver path record stores vertices, interaction types, material identifiers, complex gain, delay, arrival and departure angles, and Doppler shift.The retained path set is used to synthesize standard channel responses, including the time-varying MIMO CIR and its Fourier-transform CFR.Source: B. Channel Data Products · pp. 6–7
  • The channel products include RMS delay spread, angular and Doppler spread, and a rasterized coverage grid whose received power follows a link-budget expression.The grid aggregates path loss from paths reaching each location.Source: B. Channel Data Products · p. 7

VII. CALIBRATION

DeepTelecom calibrates material parameters against sparse path-loss anchors through a robust, differentiable inverse problem. It discovers paths first, then differentiates field replay on fixed path topology.

VII. CALIBRATION

  • Calibration minimizes robust mismatch between sparse target path-loss anchors and material-dependent fields replayed on retained propagation paths.The anchors may be coregistered measurements when available.Source: §VII · p. 7

A. Fixed-Topology Differentiable Forward Model

  • Material parameters alter the material-dependent Fresnel coefficients, and the predicted path loss uses a normalized path-gain observable with a fixed anchor-specific offset.The offset converts the adopted channel-gain normalization to the target path-loss convention and is constant with respect to material parameters.Source: A. Fixed-Topology Differentiable… · p. 7
  • The method first discovers discrete paths for each anchor, then differentiates only the continuous field replay while holding path topology fixed.With geometry and topology fixed, delays remain unchanged while material interactions reweight path coefficients.Source: A. Fixed-Topology Differentiable… · p. 7

B. Robust Objective and Physical Reparameterization

The calibration uses bounded permittivity and positive conductivity coordinates, with a Huber data objective and weak prior regularization for poorly constrained directions.

B. Robust Objective and Physical Reparameterization

  • Unconstrained variables map to bounded relative permittivity and positive conductivity, using a fixed dimensionless conductivity coordinate based on σref = 1 S/m.The reported experiment calibrates marble as its sole material class.Source: B. Robust Objective and Physical… · pp. 7–8
  • The Huber data objective is quadratic near the optimum and linear for large anchor errors, limiting the influence of large residuals.A weak prior regularizes directions that the anchors constrain poorly.Source: B. Robust Objective and Physical… · p. 8
  • Optimization starts from the inverse image of the material-table initialization and updates the unconstrained variables using Adam.The mapping and fixed-topology field replay both participate in gradient propagation.Source: B. Robust Objective and Physical… · p. 8

C. Gradient Verification and Identifiability

The section verifies reverse-mode gradients against finite differences and uses the training-anchor Jacobian to assess which material parameters the anchors constrain locally.

C. Gradient Verification and Identifiability

  • The reverse-mode gradient check compares automatic-differentiation and finite-difference gradients and defines their relative discrepancy egrad(z).The finite-difference comparison uses a dimensionless step size at the same z.Source: C. Gradient Verification and… · p. 8
  • The training-anchor Jacobian diagnoses local material sensitivity: untouched materials yield zero columns, while nearly collinear columns signal parameters that cannot be separated locally.Material coverage, Jacobian rank, and conditioning indicate whether anchors constrain the selected variables.Source: C. Gradient Verification and… · p. 8
  • These Jacobian diagnostics describe local sensitivity, not global uniqueness; changing fixed normalization changes singular values and condition number but not rank.Source: C. Gradient Verification and… · p. 8

VIII. DATASET FORMAT AND PACKAGING

DeepTelecom packages aligned dataset modalities under a shared schema and demonstrates scene generation and UAV simulations across propagation, motion, and weather conditions.

VIII. DATASET FORMAT AND PACKAGING

  • Each scenario’s unified directory schema keeps geometry, configuration, channels, and visualizations together, and supports adding modalities through new subdirectories.The listed directories include models, configuration, snapshots, paths, heatmaps, channel tensors, and scenario metadata.Source: §VIII · p. 8

IX. SIMULATION AND ANALYSIS

  • DeepTelecom is evaluated across RIS-aware scene generation, UAV micro-Doppler modeling, held-out-anchor material calibration, and multimodal 3D localization.Source: §IX · p. 8

A. Scene Generation and RIS-Assisted Propagation

  • Indoor and outdoor LoD3 scenes use consistent BS, MT, and RIS interfaces, while their building geometries produce different multipath structures.The indoor scene includes detailed walls, furniture, and partitions; both scenes represent RIS-assisted propagation.Source: A. Scene Generation and RIS-Assisted… · p. 8, Figure 4 · p. 9

B. UAV Simulation

  • The outdoor UAV study uses a rooftop base station at 28 GHz, a 10 m/s Bézier trajectory, and 0°, 30°, and 45° pitch angles.Rotor frequency and blade radius vary randomly within ±10% of nominal values.Source: B. UAV Simulation · p. 9
  • Camera frames and UAV/radio samples correspond one-to-one at a shared output rate, while user-set rotor snapshots resolve micro-Doppler within a coherence interval.Keeping snapshots within one coherence interval leaves macroscopic propagation geometry locally quasi-static while resolving rotor-induced phase variation.Source: B. UAV Simulation · p. 9
  • Rain applies modeled RF attenuation that changes echo amplitude but not micro-Doppler frequency; snow and sunny conditions provide visual variation only.The current experiment applies no RF snow-attenuation term.Source: B. UAV Simulation · p. 9
  • At 28 GHz, pitch changes the sideband envelope while bulk Doppler follows line-of-sight velocity projection; full UAV models use 25 scatterers, unlike the single-blade reference.The single-blade reference retains one blade-tip point to isolate its elementary trajectory; the two-blade sideband spacing is approximately 50 Hz at nominal rotor frequency.Source: B. UAV Simulation · p. 9, Figure 6 · p. 10, Figure 7 · p. 10

C. Calibration Results

Fixed-topology calibration is evaluated on held-out outdoor UAV anchors and improves path-loss and delay-spread agreement. The results also assess convergence and local observability of the calibrated material parameters.

C. Calibration Results

  • On 12 held-out anchors, path-loss MAE falls from 3.429 dB to 0.343 dB and RMSE from 3.601 dB to 0.435 dB; mean RMS-delay-spread RAE falls from 0.194 to 0.066.The held-out channel-gain ALE and RMS-delay-spread RAE distributions also shift toward synthetic ground truth, with a pronounced contraction of the channel-gain error tail.Source: Table II · p. 10, C. Calibration Results · p. 10
  • Calibration uses 24 training path-loss anchors to estimate marble relative permittivity and conductivity, with other materials, scattering, and link-budget terms held fixed.The study uses 48 low-altitude NLoS UAV anchors at 28 GHz in an outdoor scene: 24 training, 12 validation, and 12 held out.Source: C. Calibration Results · p. 10
  • Across 200 iterations, path-loss MAE decreases steadily and validation and held-out curves follow the training trend, indicating stable convergence and consistent generalization.The 24 × 2 training-anchor Jacobian has rank two, with singular values 399.94 and 6.36 and condition number 62.9; the variables are locally observable, but this does not establish a unique global solution.Source: C. Calibration Results · p. 10
  • Learned material parameters reweight retained propagation paths, changing their relative contributions to channel dispersion and improving delay-spread error despite its exclusion from the loss.The validation and held-out error distributions are assessed alongside aggregate errors.Source: C. Calibration Results · p. 10

D. Downstream Application

The downstream application tests multimodal UAV localization and reports lower errors than CSI-only input within a synthetic scene. The conclusion presents DeepTelecom as a customizable material-aware framework and identifies field validation and sim-to-real transfer as future work.

D. Downstream Application

  • The localization task estimates a passive UAV's 3D position from monostatic echoes, using 6,620 samples across four motion classes and an 8:1:1 train-validation-test split.Each sample includes CSI, a camera image, and a path-loss image; the multimodal model uses separate encoders and feature fusion.Source: D. Downstream Application · p. 11

X. CONCLUSION AND FUTURE WORK

  • DeepTelecom combines editable material-aware LoD3 twins with differentiable GPU ray tracing to generate synchronized channel, visual, and weather-conditioned products for RISs, MTs, and dynamic UAVs.The UAV model captures translational velocity, pitch angle, and rotor-induced micro-Doppler.Source: §X · p. 11

D. Downstream Application

  • Adding camera and path-loss images reduces the 90th-percentile localization error from 31.05 to 2.43 m and the median from 6.96 to 0.72 m versus CSI alone.At epoch 50, multimodal validation loss is 0.160 versus 0.318 for CSI alone; field validation and sim-to-real evaluation remain future work.Source: D. Downstream Application · p. 11, Figure 10 · p. 12

X. CONCLUSION AND FUTURE WORK

  • The framework supports customized data generation for user-defined scenes and configurations, with demonstrated applications including UAV identification, localization, tracking, beam prediction, and channel estimation.In the reported outdoor 28 GHz study, calibration reduces held-out path-loss MAE from 3.429 dB to 0.343 dB and RMS-delay-spread RAE from 0.194 to 0.066.Source: §X · pp. 11–12
  • Future work includes field validation and sim-to-real transfer through a bidirectional loop between DeepTelecom and its physical counterpart.The proposed work also includes richer sensing modalities and improved scene geometry, environmental rendering, and object-dynamics models.Source: §X · p. 12

Related papers

Loading 2610.14545v1…