PaperlayerView exact revision on arXiv

Source-linked AI summary

SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno

arXiv:2606.23444v2cs.ROcs.LG

TL;DR

SkyJEPA addresses the challenge of stable, interpretable, and generalizable long-horizon quadrotor dynamics prediction for real-time control. It combines JEPA-style latent dynamics, a physics-inspired state prober, sampling-based control, and domain-randomized simulation, with experiments demonstrating accurate prediction and robust zero-shot sim-to-real transfer.

  • Problem

    Existing neural dynamics models struggle with long-horizon forecasting, while analytical models require extensive identification and may not capture platform-specific effects across operating regimes.

  • Method

    The framework predicts future latent representations, probes them into physically meaningful states, integrates them with sampling-based control, and trains on domain-randomized simulation data.

  • Results

    Experiments demonstrate improved long-horizon prediction, interpretable state recovery, real-time embedded control, zero-shot sim-to-real transfer, and generalization across operating conditions.

  • Takeaways & Limitations

    A JEPA-style latent model combined with physical probing and diverse simulated data can support real-time quadrotor control without task-specific real-world fine-tuning.

Abstract

from arXiv · show

Accurate dynamics models are critical for informed decision-making in robotic systems, particularly for agile aerial vehicles operating under uncertainty. Neural network dynamics models are attractive for capturing complex nonlinear effects, but existing predictive approaches struggle with long-horizon forecasting because their autoregressive rollout mechanism amplifies errors over time. Joint Embedding Predictive Architectures (JEPAs) offer a compelling alternative by modeling dynamics in latent space, yet prior JEPA-style methods for robot navigation have been studied primarily for kinematic-level planning, with limited investigation in high-frequency control. In this work, we introduce the JEPA-style model for real-time quadrotor control. The proposed approach combines a latent dynamics model with a novel physics-inspired prober that maps frozen latents to interpretable state, enabling physically grounded long-horizon prediction. Additionally, we combine the learned model with a sampling-based optimal control solution to take advantage of its predictive capabilities for real-time control on embedded hardware. Finally, to reduce the dependence on expensive and unsafe real-world data collection, we develop a structured pipeline for automated dataset generation. Extensive open-loop and outdoor closed-loop experiments demonstrate accurate prediction, robust zero-shot sim-to-real transfer, and strong generalization across diverse operating conditions.

I. INTRODUCTION

Quadrotor world models must support accurate, interpretable, real-time, and task-generalizable prediction, but existing analytical and autoregressive learned models face practical limitations. SkyJEPA addresses these gaps with latent prediction, physics-grounded probing, sampling-based control, and domain-randomized simulation.

  • Analytical quadrotor models encode useful structure but require extensive system identification and tuning because platform-specific effects vary across operating regimes.
  • Autoregressive dynamics models accumulate one-step errors during rollout, causing drift, instability, and physically implausible long-horizon trajectories.
  • SkyJEPA predicts future representations in latent space instead of reconstructing future states, while a physics-inspired prober maps frozen latents to meaningful state variables.
  • The framework integrates latent dynamics with sampling-based optimization for real-time embedded control and robust zero-shot sim-to-real transfer.
  • Domain-randomized simulation automates diverse dataset generation, reducing dependence on extensive and potentially unsafe real-world collection.
  • Open-loop prediction and outdoor closed-loop experiments evaluate long-horizon accuracy, control robustness, and generalization across trajectories and platform variations.

B. Joint Embedding Predictive Architectures

The paper motivates latent predictive modeling by showing how recursive state-space rollouts amplify errors over time. Its JEPA-style alternative predicts future embeddings and uses a physics-inspired prober to recover physically meaningful quadrotor states.

  • JEPA-style methods predict future embeddings rather than reconstructing full observations, but prior robotics applications have mainly focused on vision-centric or lower-level settings.
  • The quadrotor state includes position, velocity, attitude, angular velocity, and motor forces, providing the physical variables used to describe system dynamics.
  • Recursive state prediction feeds imperfect outputs back into the model, allowing small one-step errors to grow into drift and physically implausible trajectories.
  • SkyJEPA encodes state-action transitions into latent representations, predicts future latent states, and trains rollouts to remain consistent with encoded future trajectories.

IV. METHODOLOGY

The methodology conditions latent dynamics prediction on recent histories of states and actions, then recursively rolls the predictor forward over future action sequences. Training aligns predicted latent trajectories with encoded future trajectories rather than reconstructing future states directly.

  • State and action histories are encoded into latent representations, and a latent predictor estimates the next latent state from those representations.
  • History conditioning provides recent motion and actuation context, helping represent actuator delay, drag, sensor noise, and platform-dependent dynamics.
  • The predictor is recursively unrolled using encoded future actions to generate a multi-step latent rollout.
  • Training enforces consistency between predicted latent states and encoded future trajectories, avoiding direct reconstruction while preserving information needed for long-horizon dynamics.

B. Training Objective

The training objective learns multi-step latent predictions while preventing representation collapse, then uses frozen latents to recover physically meaningful state trajectories through a structured prober.

  • B. Training Objective: Multi-step prediction consistency trains latent representations to remain predictive across the full rollout horizon.The objective compares rolled-out latent predictions with encoded future states.
  • B. Training Objective: SIGReg regularization prevents degenerate constant embeddings by encouraging the latent distribution toward an isotropic Gaussian.It uses random one-dimensional projections and averages the resulting discrepancy across time.
  • C. Physics-Inspired Probing Mechanism: The two-stage pipeline freezes the encoder and predictor before training a physics-inspired prober to map latent rollouts to metric-state trajectories.This separation prevents supervised state recovery from modifying the learned latent dynamics.
  • C. Physics-Inspired Probing Mechanism: The prober predicts residual translational and angular corrections that are integrated through a structured kinematic model.The integrator preserves attitude geometry through the SO(3) exponential map while correcting unmodeled dynamics.
  • C. Physics-Inspired Probing Mechanism: Frozen latent predictions combined with a differentiable integrator produce physically meaningful rollouts suitable for control.The probing network is trained with supervised rollout loss while latent predictions receive stop-gradient treatment.

D. Sampling-Based Control with Learned Dynamics

The controller integrates latent dynamics and physics-based state probing into MPPI, sampling bounded action sequences and updating them from trajectory costs in a receding-horizon loop.

  • D. Sampling-Based Control with Learned Dynamics: MPPI optimizes future control sequences by Monte Carlo sampling with the learned dynamics model inside the rollout loop.Each sampled action sequence is propagated through latent dynamics and mapped to physically grounded states.
  • D. Sampling-Based Control with Learned Dynamics: Actions are sampled under covariance Σ and projected onto admissible lower and upper limits by element-wise clamping.The bounds are denoted amin and amax.
  • D. Sampling-Based Control with Learned Dynamics: Sampled trajectories are evaluated using weighted quadratic tracking errors and control effort relative to reference states.Qx and Qa are positive diagonal weight matrices for state and control costs.
  • D. Sampling-Based Control with Learned Dynamics: Importance weights based on each trajectory cost update the nominal action sequence, with temperature λ controlling the weighting.The minimum sampled cost is used in the importance-weight computation.
  • D. Sampling-Based Control with Learned Dynamics: Only the first updated action is executed before the optimization repeats in receding-horizon fashion.This closes the loop while continually recomputing the future action sequence.

E. Automated Data Synthesis

The automated synthesis pipeline generates diverse smooth references, tracks them with complementary controllers, and varies plausible simulator parameters to expose the model to broad flight and system conditions.

  • E. Automated Data Synthesis: Gaussian-process references generate randomized, smooth trajectories spanning diverse positions, velocities, accelerations, attitudes, and angular velocities.Periodic kernels with multiple scales provide both slow global motion and faster local variation.
  • E. Automated Data Synthesis: Simulation tracking converts generated references into physically feasible state–action trajectories rather than using references directly as training data.A tracking controller computes actions from the current state and a local reference horizon.
  • E. Automated Data Synthesis: NMPC and MPPI expose the dataset to complementary action distributions, combining smooth optimized commands with broader local sampling variation.This covers control inputs encountered during real-time sampling-based optimization.
  • E. Automated Data Synthesis: Domain-randomized simulation varies mass, drag, inertia, motor delay, thrust and torque coefficients, and arm length across rollouts.The randomized models include aerodynamic drag and first-order motor delay rather than relying on one nominal simulator.
  • E. Automated Data Synthesis: Sampling each trajectory under a different plausible dynamics realization exposes the model to a family of systems and operating conditions.The representation is encouraged to remain consistent across variation in mass, drag, actuation, and inertial properties.

A. Platform Setup

The platform setup combines embedded quadrotor hardware, automated domain-randomized simulation data, and an MPPI control stack designed for real-time onboard evaluation.

  • Platform Setup: Experiments use a 1.3-kg quadrotor with a 4:1 thrust-to-weight ratio, NVIDIA Orin NX computation, Pixracer Pro/PX4 low-level control, ROS2, and GPS-based state estimation.
  • Dataset Generation: The training dataset spans 500 independently randomized quadrotor domains covering inertial properties, actuation, motor response, and aerodynamic drag.
  • Platform Setup: Closed-loop evaluation tests nominal trajectory tracking, payload variation, and propeller switching on the real platform.
  • Latent Dynamics Modeling: The latent dynamics model is recursively trained for T = 20 steps, corresponding to 1.0 s at 20 Hz, using a 0.5 s history at 20 Hz.
  • MPPI Controller: TensorRT optimization and C++ MPPI enable high-frequency rollout evaluation and closed-loop control on embedded hardware.
  • MPPI Controller: Control runtime increases with unroll length and MPPI sample count, while the selected horizon is kept below the 10 ms real-time budget.

A. Latent Dynamics Modeling

Latent-space dynamics modeling reduces recursive error accumulation, produces smoother temporal representations, and improves long-horizon prediction and robustness to corrupted observations.

  • Compounding Error Analysis: At long horizons, the predictive baseline’s compounding ratio reaches approximately 2.4 by k = 60, whereas the proposed method remains closer to 1 and grows more gradually.The compounding ratio compares recursive rollout error with teacher-forced error; values above 1 indicate accumulation from recursion.
  • Compounding Error Analysis: Latent-space dynamics modeling substantially mitigates recursive error accumulation and narrows the gap between teacher-forced and recursive predictions.
  • Open-Loop Rollout Fidelity: Open-loop RMSE grows more slowly for latent dynamics models than for direct predictive models, while the physics-inspired prober improves recovery of position, velocity, and attitude.
  • Temporal Straightening: The JEPA-based model attains a mean temporal straightening score of about 0.75, indicating smoother latent evolution than direct predictive modeling.The reconstruction model achieves the highest straightening score, but the JEPA-based model remains strongly positive.
  • Temporal Straightening: Both latent models learn temporally smoother latent trajectories than direct predictive modeling, making temporal smoothness an emergent property of latent dynamics learning.
  • Robustness to Noise: With zero observation noise, the method reduces median state RMSE by approximately 55%, and at moderate noise achieves roughly a 25–30% reduction versus the predictive baseline.At the highest noise level, it retains about a 10% improvement and produces more concentrated error distributions.

B. Physics Interpretability

The study evaluates latent dynamics models and shows that JEPA-style prediction combined with a physics-inspired prober substantially improves recovery of physically meaningful position and attitude trajectories.

  • Physics Structure in NN Design: Latent dynamics outperform direct autoregressive prediction, while adding physics regularization alone provides only modest improvement.The predictive baseline reaches 8.80 m position RMSE and 53.4° attitude error, whereas physics regularization reduces these to 7.12 m and 49.1°; latent baselines perform better.
  • Training Objective: 1.43 m position RMSE and 4.71° attitude error are achieved by the full JEPA-plus-physics-prober method, with sharply reduced position and attitude variance.Replacing the generic prober reduces JEPA position RMSE from 5.56 to 1.43 m and attitude error from 40.2° to 4.71°; variances fall from 1.31 to 0.10 and 9.30° to 0.50°.
  • Training Objective: JEPA-style latent training improves over reconstruction-based latent training before probing, reducing position RMSE from 6.82 to 5.56 m and attitude error from 45.2° to 40.2°.The comparison indicates that predictive latent representations are stronger than reconstruction-trained representations, while the largest gains arise after introducing the physics-inspired prober.
  • Physics Interpretability: The physics-inspired prober maps latent rollouts through differentiable kinematic structure into interpretable metric-state trajectories for control.This enables recovery of quantities needed for tracking costs, state constraints, and actuator limits.

C. Zero-Shot Sim-to-Real Transfer

The framework is evaluated for real-world trajectory tracking using a dynamics model trained entirely in domain-randomized simulation, without task-specific real-world fine-tuning.

  • Experimental Setup: The closed-loop comparison evaluates the proposed controller against predictive and physics-regularized predictive baselines on real-world trajectories.Each controller uses a different learned dynamics model, testing whether open-loop benefits transfer to real deployment.
  • Zero-Shot Transfer: The proposed controller tracks diverse real-world reference trajectories using a simulation-trained dynamics model without task-specific fine-tuning.Figure 10 shows executed trajectories overlaid on desired references, with tracking error encoded in centimeters.
  • Results: The experiments support zero-shot sim-to-real transfer across multiple real-world reference trajectories using the proposed latent dynamics framework.The reported qualitative tracking remains generally concentrated along the reference paths rather than diverging.

D. Robustness

Robustness is assessed under platform changes and varying dataset quality, showing strong tracking under model mismatch and an inverse relationship between trajectory-distribution quality and prediction error.

  • Platform Variation: The robustness tests alter actuation or vehicle inertial properties using propeller switching and payload transportation not covered during data collection.Trajectories were executed at an average velocity of 2 ms^-1 and repeated five times.
  • Platform Variation: The proposed method achieves the best tracking performance under both propeller switching and payload transportation without retraining.Under propeller switching, position error is about 1.3× lower than MPPI (Pred.) and 1.2× lower than MPPI (Pred.+Phy.); attitude error is about 1.3× and 1.25× lower, respectively.
  • Platform Variation: Lower variance across position and attitude metrics indicates more repeatable tracking under platform mismatch.Qualitative rollouts remain close to reference paths under both non-nominal deployment conditions.
  • Data Quality: TDQ measures whether training data broadly covers state-action features, local transitions, and randomized simulator parameters.Its entropy-coverage components penalize poorly represented regions, while the harmonic mean requires strength across all components.
  • Data Quality: Higher TDQ corresponds to lower state prediction error as dataset size increases, with TDQ rising from 0.01 to 0.94 while state RMSE decreases from 5.4 to 1.4.TDQ combines coverage, transition richness, and simulator-parameter robustness through a harmonic mean.

VII. DISCUSSION

The discussion argues that the proposed latent framework satisfies the main requirements of a quadrotor world model and that structured simulation data can support robust zero-shot deployment.

  • Discussion: The framework combines improved long-horizon prediction, interpretable state rollouts, embedded real-time control, and task-agnostic deployment across trajectories and platform conditions.These properties are supported by the open-loop, onboard MPPI, and closed-loop experiments.
  • Data Generation: Higher-quality simulation distributions improve dynamics prediction, while gains begin to saturate as coverage increases.The discussion attributes remaining limits to model capacity, optimization, or residual dynamics complexity.
  • Training Objective: SIGReg uses a compact anti-collapse objective with one main regularization weight, simplifying practical tuning relative to more complex representation-learning objectives.The discussion contrasts this with methods requiring multiple loss terms, target networks, stop-gradient choices, or reconstruction weights.
  • Overall Findings: The complete method consistently outperforms direct predictive baselines across open-loop prediction, noise robustness, sim-to-real transfer, and platform variation.Real-world deployment occurs without task-specific fine-tuning.
  • Future Work: Future work extends the framework from low-dimensional state inputs to RGB and RGB-D observations and incorporates safety-relevant structures into representation and planning.The proposed directions include visual navigation, obstacle avoidance, visibility, uncertainty, and recoverability.
Loading 2606.23444v2…