PaperlayerView exact revision on arXiv

Source-linked AI summary

Coupled Control and Wireless World Models for Resilient Remote Robotic Control

H. P. Madushanka, Sumudu Samarakoon, Mehdi Bennis

arXiv:2609.04851v1cs.ROcs.LG

TL;DR

Remote robotic control must remain reliable despite limited communication, changing wireless channels, and perception disturbances, while continuous sensory transmission is costly. The paper couples control and wireless JEPA world models with structured RF inputs and adaptive resilience in a Gazebo–ROS–Sionna simulation, achieving improved communication efficiency, robustness, and resilience while maintaining navigation performance.

  • Problem

    Remote control is challenged by costly high-dimensional uplink sensing, wireless disturbances, and visual-environment changes that can degrade reliable operation.

  • Method

    The framework couples C-JEPA and W-JEPA world models, uses transfer learning and structured RF representations, and adapts perception embeddings when prediction discrepancies arise.

  • Results

    The approach consistently outperforms DQN, PID, and raw CSI-based methods in communication efficiency, robustness, resilience, and predictive wireless modeling while maintaining navigation performance.

  • Takeaways & Limitations

    Predictive joint modeling can reduce unnecessary uplink transmissions while supporting reliable navigation under wireless and visual perturbations.

  • Takeaways & Limitations

    The study is limited to one robot, one wireless link, simulation, and predefined navigation scenarios.

Abstract

from arXiv · show

Remote robotic systems operating over wireless networks must maintain reliable control despite limited communication resources, changing channel conditions, and environmental disturbances.However, continuously transmitting high-dimensional sensory observations, such as camera images, increases communication overhead and energy consumption while reducing robustness under unreliable connectivity.To address these challenges, this paper proposes a resilient communication-aware remote robotic control framework based on coupled control and wireless Joint Embedding Predictive Architecture (JEPA) world models that jointly capture robot dynamics and wireless channel evolution from visual observations and a combination of raw and structured radio frequency (RF) representations based on spectrograms and Persistence Images(PIs).The learned latent representations enable predictive communication scheduling by jointly forecasting future robot states and wireless conditions, thereby reducing unnecessary uplink transmissions while maintaining reliable control performance.Furthermore, an adaptive resilience mechanism detects latent prediction discrepancies and efficiently adapts perception embeddings to accommodate wireless and visual environmental changes without retraining the complete control policy.The proposed framework is evaluated in a synchronized Gazebo-Robot Operating System (ROS)-Sionna robot-wireless simulation environment under diverse wireless propagation and perception perturbations.Experimental results demonstrate significant improvements in communication efficiency, robustness, and resilience while maintaining navigation performance compared with conventional Proportional Integral Derivative (PID), model-free Deep Q-Network (DQN), and predictive approaches based on Vision Transformers(ViTs).

I. INTRODUCTION

The paper addresses reliable remote robotic control under communication constraints, changing wireless conditions, and environmental disturbances. It extends coupled latent world modeling into a realistic synchronized simulation framework and evaluates robustness, structured RF representations, and adaptive resilience.

  • Continuous transmission of high-dimensional sensory observations increases communication overhead, latency, and energy consumption.
  • Wireless channel variations, packet losses, and environmental disturbances can degrade remote robotic control performance.
  • World models and JEPAs learn compact predictive latent representations for forecasting system states from sensory observations.
  • The paper extends coupled C-JEPA and W-JEPA modeling from a lightweight Gym environment to a realistic Gazebo–ROS–Sionna simulation using transfer learning.
  • The framework is evaluated under wireless disturbances including scatterer changes, access-point handover, carrier-frequency shifts, and intermittent uplink failures.
  • The study investigates raw and structured wireless representations and examines communication-aware latent world models in realistic robotic environments.

III. SYSTEM ARCHITECTURE, METHODOLOGY, AND IMPLEMENTATION

The architecture forms a closed-loop remote control system in which visual observations and wireless measurements support coupled prediction, scheduling, and navigation. Its objectives are reliable control, lower communication overhead, improved resource use, and resilience to changing conditions.

  • A mobile robot sends visual observations over an uplink, while a remote controller returns control commands through a downlink.
  • The controller uses uplink CSI or alternative RF-derived representations as wireless observations alongside visual observations.
  • A binary scheduling variable ρ_t determines whether the robot transmits an observation or omits the uplink transmission.
  • The framework aims to maintain reliable task performance, reduce communication overhead, exploit favorable channel conditions, and adapt to wireless and environmental changes.
  • The coupled C-JEPA and W-JEPA jointly model robot movement and wireless-condition dynamics in a unified latent representation space.

A. Joint C-JEPA and W-JEPA Learning

The C-JEPA encodes visual observations into recurrent latent robot dynamics, while the W-JEPA predicts future wireless representations and transmission power. Their predictions support communication-aware uplink scheduling and continued state prediction when observations are not transmitted.

  • The C-JEPA encodes visual observations and uses an RSSM to represent robot dynamics with deterministic memory and stochastic uncertainty states.
  • The RSSM recurrent transition, prior, and posterior models update deterministic states and infer observation-conditioned stochastic states.
  • The learned latent states drive reward, termination, and control-policy models that generate robot actions.
  • The C-JEPA objective includes KL-divergence, reward-prediction, and terminal-prediction losses.
  • The W-JEPA encodes raw CSI, spectrogram images, or persistence images and predicts future wireless latent representations conditioned on control latents.
  • The W-JEPA predicts uplink transmission power, which informs scheduling decisions for transmitting current observations or continuing C-JEPA state prediction.

B. Structured RF Representations for W-JEPA

The W-JEPA compares raw CSI with two structured RF representations derived from historical wireless observations: spectrogram images and persistence images. These alternatives are intended to expose temporal-frequency or topological structure for predictive wireless modeling.

  • Historical wireless observations are transformed over a window of length τ to improve identification of temporal changes in wireless conditions.
  • Spectrogram construction applies windowing and FFT to create a time-frequency image from wireless observations.
  • Persistence images map time-delay-embedded wireless observations into point clouds and encode topological features through persistent homology.
  • The W-JEPA accepts raw CSI, spectrogram, or persistence-image inputs, whose performance impact is analyzed experimentally.

C. Adaptive Resilience Mechanism

The framework detects latent perception inconsistencies caused by illumination changes or local visual obstruction, then adapts incoming observations without retraining the complete world model.

  • Runtime changes considered include global illumination variation and local visual obstruction such as mud patches.
  • Perception degradation is detected when the weighted latent prediction discrepancy over K future steps reaches the empirical threshold δ.
  • For illumination changes, the system selects a predefined HSV transformation that minimizes latent inconsistency and applies it to incoming observations.
  • For local obstruction, a lightweight VAE reconstructs degraded observations using a subset of previously collected data before C-JEPA re-encoding.
  • The adaptation ratio r and optimization epochs E trade off adaptation time against reconstruction performance.

D. Sionna–Gazebo Integration

The experimental environment synchronizes Gazebo robot simulation with Sionna RT wireless ray tracing through ROS, enabling matched visual and channel observations in a shared physical scene.

  • Gazebo supplies robot dynamics, perception, and navigation, while Sionna RT provides physics-based wireless channel modeling through electromagnetic ray tracing.
  • Because Gazebo and Sionna RT use different scene formats, Blender assigns radio materials and converts the environment for Sionna RT.
  • Figure 4 depicts the robot, sensing uplink, control downlink, Gazebo scene, and synchronized Sionna RT scene.
  • The JetBot URDF and RGB camera are integrated into Gazebo, while ROS supplies robot pose to Sionna RT for CSI generation and visual input to C-JEPA.

E. Experimental Setup

Experiments use synchronized robot and wireless observations from Gazebo and Sionna RT, with a transferred C-JEPA navigation model adapted to Gazebo observations and a progress-based control objective.

  • At each simulation step, ROS synchronizes robot pose and velocity so RGB observations and Sionna-generated CSI represent the same robot state.
  • The evaluation uses a mobile RGB-camera robot navigating a closed-loop road while maintaining sensing uplink and control downlink communication.
  • C-JEPA is pretrained in Gym CarRacing because training directly in Gazebo requires several days of continuous interaction before convergence.
  • A major transfer challenge is the mismatch between Gym CarRacing and Gazebo observation spaces, addressed by transforming Gazebo RGB images into Gym-style observations.
  • The Gazebo reward encourages road progress and efficient navigation through crossing-marker rewards and a per-step penalty.
  • Episodes terminate for road deviation, prolonged stopping, overturning, or sufficiently low cumulative reward, with an additional failure penalty.
  • A discrete-to-continuous mapping converts C-JEPA actions into the JetBot's linear and angular velocity commands.

IV. PERFORMANCE EVALUATION

The evaluation uses synchronized Gazebo–Sionna experiments to test communication-aware control under wireless and environmental changes. Scenarios vary propagation, connectivity, and perception while preserving a common control trajectory.

  • Experimental setup: The framework is evaluated in synchronized Gazebo–Sionna experiments comparing navigation and communication performance under limited wireless connectivity.The evaluation examines reliable remote control while reducing communication resource utilization.
  • Experimental setup: All experiments use the same predefined closed-loop trajectory while selectively modifying wireless communication or visual perception conditions.
  • Wireless perturbations: E2–E4 test robustness to added scatterers, access-point handover, and carrier-frequency shifts from 2.14 GHz to 1.9 GHz or 2.6 GHz.
  • Representation and perception tests: E5 compares raw CSI with spectrogram- and PI-based RF representations, while E6–E8 assess consecutive connectivity failures, lighting changes, and visual obstructions.

B. Baseline Methods

The proposed framework is compared with communication-unaware DQN and PID baselines under matched simulation conditions. It maintains navigation and communication performance more effectively across wireless perturbations and handover events.

  • Baseline design: DQN and PID provide representative navigation baselines implemented in the same Gazebo–Sionna environment under identical conditions.DQN uses the proposed transfer-learning strategy, while PID directly follows the reference trajectory from onboard camera observations.
  • Baseline design: Both baselines continuously transmit RGB observations and lack predictive wireless modeling, transmission scheduling, and wireless power prediction.
  • Overall performance: 57.1 s completion time, 775.4 cumulative average reward, 0.48 ms uplink latency, and 4.74 J uplink energy were achieved by the proposed framework in E1.DQN and PID used 9.21 J and 11.19 J, respectively, while PID required 77.3 s and achieved 716.1 average reward.
  • Wireless robustness: The proposed framework shows only minor completion-time variation and the lowest uplink energy across carrier-frequency and scatterer perturbations.DQN degrades more under these wireless changes, whereas PID remains relatively unaffected by its non-learned communication behavior.
  • Wireless robustness: 100% navigation completion under access-point handover exceeded DQN’s 56.48% and PID’s 58.99% lap completion.

D. Analysis of The Structured RF Representations

Structured RF representations improve wireless prediction over raw CSI, with PIs achieving the lowest errors but higher computational cost. Predictive latent embeddings also reduce communication payload and tolerate more consecutive uplink losses.

  • Prediction quality: Spectrogram- and PI-based W-JEPA representations consistently reduce prediction error versus raw CSI across all prediction horizons.The improvement becomes more evident at longer horizons, with PI-based W-JEPA achieving the lowest error distribution.
  • Prediction quality: Longer temporal windows generally produce lower prediction errors and reduced variability for both structured RF representations.
  • Computational tradeoffs: Representation parameters trade prediction accuracy against generation time, while selected spectrogram and PI settings remain computationally efficient for online operation.The spectrogram uses STFT segment length L, whereas the PI uses time-delay embedding dimension m.
  • Computational tradeoffs: Raw CSI has lower preprocessing cost, spectrograms offer a favorable accuracy–complexity balance, and PIs provide the lowest errors at higher generation cost.
  • Communication efficiency: The proposed 400-dimensional latent embedding reduces communication payload by about 62-fold relative to the ViT baseline without sacrificing control performance.The ViT baseline requires at least 128 × 196 tokens for comparable performance.
  • Communication resilience: The proposed framework sustains stable reward for approximately 13 consecutive missing transmissions, compared with approximately 8 for the ViT baseline.

F. Adaptive Resilience to Visual Disturbances

The adaptive resilience mechanism restores latent perceptual consistency under global illumination changes and local visual obstruction, supporting continued navigation without retraining the complete control policy. Recovery speed trades off against longer-term sustainability as adaptation computation increases.

  • Global perceptual degradation: Illumination changes can cause perceptual inconsistency and controller failure, whereas online latent adaptation restores consistency and allows navigation to continue with slight interruption.The comparison is made under global perceptual degradation in E7.
  • Local visual obstruction: Recovery from local visual obstruction depends on the historical-data ratio r and adaptation epochs ϵ, both of which increase adaptation time when raised.These parameters govern the computation used for online adaptation in E8.
  • Recovery–sustainability tradeoff: Faster recovery with lower r and ϵ is not always sustainable, creating a tradeoff between recovery speed and sustained performance.Maintaining performance for a longer duration requires significant computation in one or both adaptation parameters.
  • Overall resilience: The mechanism recovers from global perceptual degradation and local visual obstruction without retraining the complete control policy.Online restoration of latent consistency supports stable navigation and operational continuity under challenging visual conditions.
  • Overall resilience: The framework jointly models robot and wireless dynamics to schedule communication predictively while maintaining navigation performance under perturbations.Structured RF representations improve wireless prediction accuracy, and the framework outperforms DQN, PID, and raw CSI-based methods across several evaluation dimensions.
  • Scope boundary: The study remains bounded by a single simulated robot, single wireless link, and predefined navigation scenarios.Physical-platform validation and broader multi-robot, network, and environmental settings are identified as future extensions.

APPENDIX A IMPLEMENTATION DETAILS

The appendix identifies the implementation-configuration tables covering hardware, software and wireless settings, and hyperparameters.

  • Configuration tables: Hardware configuration is documented in Table IV.
  • Configuration tables: Software and wireless configurations are documented in Table V.
  • Configuration tables: Hyperparameters are documented in Table VI.
Loading 2609.04851v1…