PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control

Fanxing Li, Yuxiang Huang, Shengyang Wang, Fangyu Sun, Shuyu Wu, Tianbao Zhang, Dexin Zuo, Yan Yufei, Danping Zou, Wenxian Yu

arXiv:2610.31617v1cs.RO

TL;DR

Agile quadrotor control must anticipate external motion while maintaining precise control, yet learned rollouts can entangle environmental and robot dynamics. DiffDreamer couples latent world-model imagination with differentiable simulation and separates task-reward from motion-reward pathways. It improves convergence and final rewards over baselines on both tasks, with proof-of-concept real-world flight validation.

  • Problem

    Agile quadrotor tasks require predicting external motion for precise control, while learned rollouts can entangle partially observed environmental dynamics with fast robot dynamics.

    Source: §I · p. 1

  • Method

    DiffDreamer synchronizes learned latent rollouts for external dynamics with differentiable simulation of the robot’s physical state and splits actor optimization between task and motion rewards.

    Source: §I · p. 1

  • Results

    DiffDreamer converges faster and achieves higher final rewards than baselines on both tasks, with larger gains in dynamic-gate racing; tracking mean distance error is 0.381 m versus 1.003 m for DreamerV3.

    Source: C. Training Results · p. 6, D. Experimental Performance · p. 7

  • Takeaways & Limitations

    Proof-of-concept real-world aggressive tracking and dynamic-gate racing flights validate the framework in quadrotor scenarios.

    Source: §I · p. 1

    Evaluation is limited to quadrotors; real-world validation relies on motion-capture feedback, leaving onboard perception and robustness to perception noise untested.

    Source: §VI · p. 8

Abstract

from arXiv · show

Tasks such as quadrotor dynamic-gate racing require agents to anticipate external motion while maintaining precise high-frequency control. World-model-based reinforcement learning provides a natural solution by learning latent dynamics for future imagination, yet it often entangles environmental evolution and the robot's own dynamics in a single learned rollout, where model bias can affect policy optimization efficiency. We propose DiffDreamer, a differentiable reinforcement learning framework that tightly couples a learned world model with differentiable simulation for actor optimization. In one optimization step, the learned world model propagates latent states that capture external dynamics and history-dependent features, while the differentiable simulator propagates the robot's explicit physical state. We further decompose the actor objective into a critic-driven pathway for non-differentiable task rewards and a simulator-driven pathway for differentiable motion rewards. Experiments on aggressive tracking and dynamic-gate racing show that DiffDreamer improves training convergence over the model-free, model-based, and differentiable-RL baselines, and final performance over these baselines and classical planning and control methods. Additional experiments assess reward decomposition and the odometry-conditioned relative-state representation through ablations. We also provide proof-of-concept real-world flight experiments for validation.

I. INTRODUCTION

Agile quadrotor tasks require predicting external motion while controlling the vehicle precisely, but learned rollouts can entangle external and robot dynamics, while differentiable RL struggles with partially observed motion and non-differentiable task events. DiffDreamer couples a learned world model with differentiable simulation and separates task-reward and motion-reward optimization pathways.

I. INTRODUCTION

  • A learned rollout may entangle partially observed, history-dependent external dynamics with smooth robot dynamics available through physical modeling, reducing gradient accuracy for precise control.DiffDreamer uses differentiable simulation for robot motion so the world model can focus on external dynamics.Source: §I · p. 1
  • Differentiable RL can provide low-variance gradients for smooth robotic systems, but typically assumes differentiable, explicitly available transitions and rewards.Those assumptions make partially observable external motion and non-differentiable gate-crossing rewards difficult to handle.Source: §I · p. 1
  • DiffDreamer synchronizes latent world-model updates for external dynamics with simulated physical-state propagation under the same action, and separates critic-driven task rewards from simulator-driven motion rewards.The actor receives both latent and explicit physical states.Source: §I · p. 1
  • The paper reports proof-of-concept real-world validation for aggressive tracking and dynamic-gate racing.Ablations assess reward decomposition and odometry-conditioned relative-state representation.Source: §I · pp. 1–2

A. Agile Quadrotor Flight and Modular Pipelines

This section introduces modular quadrotor pipelines and model-based reinforcement learning, then establishes the control formulation and world-model imagination used as context for DiffDreamer.

A. Agile Quadrotor Flight and Modular Pipelines

  • Modular quadrotor pipelines combine perception, state estimation, planning, and control, but separately designed modules can pass prediction or planning errors downstream.Examples include learned perception with trajectory generation and tracking, and prediction with visibility-aware trajectory optimization.Source: A. Agile Quadrotor Flight and Modular… · p. 2

B. Model-based Reinforcement Learning

  • Model-based RL uses learned or known dynamics for planning and policy optimization, but remains sensitive to model fidelity in high-accuracy, high-frequency robotic control.Agile quadrotor rollouts may need to capture both partially observable external motion and fast rigid-body dynamics.Source: B. Model-based Reinforcement Learning · p. 2

A. Preliminaries

  • The control problem is formulated as a POMDP in which the agent acts from noisy or partial observations and receives rewards and episode-continuation signals.The actor infers actions from partial observations, while the critic estimates expected long-term return.Source: A. Preliminaries · p. 2

B. Model-based RL from a World Model

  • DreamerV3’s RSSM encodes observations into deterministic and stochastic latent features, then uses recurrent dynamics to predict future latent states, rewards, and continuation flags.Its actor and critic train on imagined latent trajectories; bootstrapped λ-returns mix predicted rewards with learned values to mitigate truncation bias.Source: B. Model-based RL from a World Model · pp. 2–3

C. Differentiable Reinforcement Learning

Differentiable RL optimizes policies by backpropagating reward gradients through physical dynamics, making it useful for smooth robotic motion while relying on differentiable transitions and rewards.

C. Differentiable Reinforcement Learning

  • Differentiable RL directly optimizes a policy by backpropagating reward gradients through a differentiable physical model.The approach reduces reliance on a learned critic for dense motion optimization when dynamics and rewards are smooth and accurately modeled.Source: C. Differentiable Reinforcement Learning · p. 3
  • The differentiable-RL formulation assumes deterministic transitions and rewards defined over the robot’s explicit state and action.For the quadrotor, the explicit state includes position, velocity, orientation, and angular velocity, propagated with nonlinear rigid-body dynamics.Source: C. Differentiable Reinforcement Learning · p. 3
  • Unrolling differentiable dynamics links each timestep’s reward to actor parameters, allowing cumulative return gradients to flow through preceding actions by the chain rule.The simulator supplies the state-transition Jacobian.Source: C. Differentiable Reinforcement Learning · p. 3

D. DiffDreamer: Learning Behaviors from Imagination and Prediction

DiffDreamer couples learned latent imagination of external dynamics with differentiable simulation of robot motion. Its actor optimization separates non-differentiable task rewards from differentiable motion rewards.

D. DiffDreamer: Learning Behaviors from Imagination and Prediction

  • External prediction modules optimized independently of control can contribute to prediction-control mismatch and error propagation at high speeds.Source: D. DiffDreamer: Learning Behaviors… · p. 4
  • DiffDreamer synchronizes learned latent imagination of history-dependent external dynamics with differentiable physical rollouts of the robot under shared actions.The world model predicts external features while the simulator propagates the robot’s explicit state.Source: D. DiffDreamer: Learning Behaviors… · p. 4
  • The RSSM latent state encodes history-dependent external motion, while explicit state tracks position, velocity, attitude, and angular velocity.The same actor-sampled action updates both states at each rollout step.Source: D. DiffDreamer: Learning Behaviors… · p. 4
  • The critic estimates returns for non-differentiable rewards, while differentiable motion rewards are optimized directly through the simulator.This separates discrete task milestones from continuous physical-motion regulation.Source: D. DiffDreamer: Learning Behaviors… · p. 4
  • During actor optimization, latent inputs are detached, explicit-state inputs remain differentiable, and shared actions keep both rollouts synchronized.World-model and critic parameters are held fixed during this optimization.Source: D. DiffDreamer: Learning Behaviors… · p. 4
  • Separating dynamics can reduce the world model’s learning burden and extend first-order policy optimization to partially observable environments and non-differentiable task events.The simulator supplies gradients for controllable robot dynamics, while the world model predicts external dynamics.Source: D. DiffDreamer: Learning Behaviors… · p. 4

IV. EXPERIMENTS

The experiments use aggressive tracking and dynamic-gate racing to evaluate dynamic-environment perception and control, including proof-of-concept physical flight.

IV. EXPERIMENTS

  • Quadrotors’ smooth, complex dynamics make them a platform for evaluating algorithms that integrate differentiable physics.Source: §IV · p. 4
  • Real-world experiments demonstrate aggressive tracking and dynamic-gate racing, with tracking trajectories and frames and gate-position anticipation shown.Source: Figure 3 · p. 5
  • Aggressive tracking and dynamic-gate racing are designed as end-to-end benchmarks for evaluating perception of dynamic environments and control performance.Source: §IV · p. 5

A. Experimental Setup

The setup implements two quadrotor benchmarks in VisFly and uses full rigid-body dynamics with end-to-end control commands.

A. Experimental Setup

  • VisFly implements aggressive tracking of a high-speed target and sequential traversal of moving gates in the shortest possible time.Source: A. Experimental Setup · p. 5
  • The quadrotor uses full nonlinear rigid-body dynamics, and the policy outputs collective-thrust-and-body-rate commands.The implementation uses an explicit physical state rather than a point-mass model driven by position commands.Source: A. Experimental Setup · p. 5
  • Target and gate trajectories are cubic curves with randomly sampled control points, and external-motion prediction is compared with a 3D constant-acceleration EKF.Source: A. Experimental Setup · p. 5

B. Task Definition

The policy combines learned latent features with explicit physical states, using simulator or motion-capture observations.

B. Task Definition

  • DreamerV3 encodes observations into its RSSM state, while DiffDreamer combines learned latent features with explicit physical states.Source: B. Task Definition · p. 5

1) Aggressive Tracking:

Aggressive tracking combines target-relative position with quadrotor odometry to track an agile target while maintaining camera visibility and distance. Dynamic-gate racing observes the nearest two moving gates and rewards successful passage alongside directed motion and stable control.

1) Aggressive Tracking:

  • Tracking uses relative target position and quadrotor odometry, rewarding desired distance and camera centering while penalizing aggressive, unstable control.The body-frame x-axis aligns with the camera viewing direction.Source: 1) Aggressive Tracking · p. 5

2) Dynamic-gate racing:

  • Racing observes the nearest two gates and combines a gate-passage reward with rewards for approaching gates and penalties for moving away or controlling aggressively.The gate-passage term is a large positive constant.Source: 2) Dynamic-gate racing · p. 5
  • Fig. 4 shows target and gate forecasts over two seconds, target-location heatmaps against the camera field of view, and average gates passed.The heatmap’s more intense colors indicate higher target spatial density.Source: Figure 4 · p. 6

C. Training Results

DiffDreamer converges faster and reaches higher final rewards than the baselines on both tasks, with larger gains in dynamic-gate racing. Evaluations also report more precise motion prediction, stronger tracking results, and more gates passed across tested gate speeds.

C. Training Results

  • Five-seed training compared DiffDreamer with DreamerV3, PPO, and BPTT; classical baselines were Elastic Tracker for tracking and policysearch for racing.Methods shared observations, rewards, and randomized target trajectories, speeds, and initial states, with fixed identified physics parameters.Source: C. Training Results · p. 6, D. Experimental Performance · p. 6
  • DiffDreamer converges faster and achieves higher final rewards than the baselines on both tasks, with larger gains in dynamic-gate racing.Fig. 5 reports training reward curves with means and standard deviations across five seeds.Source: C. Training Results · p. 6, Figure 5 · p. 6

D. Experimental Performance

  • In open-loop forecasts, DiffDreamer predicts agile target and gate trajectories more precisely than the EKF.The EKF’s fixed-motion model cannot anticipate later acceleration or maneuver changes, and its errors grow rapidly during agile maneuvers.Source: D. Experimental Performance · p. 6
  • DiffDreamer’s target distribution is more compact than the baselines’, especially in the XZ plane, suggesting greater robustness to abrupt vertical target motion.Concentration around the desired relative position indicates stable standoff distance and keeping the target within the camera field of view.Source: D. Experimental Performance · pp. 6–7
  • 0.381 m mean distance error for DiffDreamer compares with 0.793 m for Elastic Tracker, 0.835 m for BPTT, and 1.003 m for DreamerV3.The paper reports DiffDreamer has the lowest mean errors on all three tracking metrics.Source: D. Experimental Performance · p. 7
  • 16, 12, and 8 mean gates passed for DiffDreamer at 1.0, 1.5, and 2.0 m/s, versus MPC’s 12, 4, and 1.DiffDreamer outperforms all baselines across gate speeds; its smaller decline relative to MPC suggests greater robustness to faster gate motion.Source: D. Experimental Performance · p. 7

E. Ablations

The ablations examine the odometry-conditioned relative-state representation and the separate reward pathways. The absolute-position variant has broader reconstruction errors, while removing either reward component reduces gates passed at all tested speeds.

1) Odometry and relative-state representation:

  • Removing odometry and reconstructing absolute target position produces a broader reconstruction-error distribution and a more dispersed target distribution in the quadrotor frame during tracking.The comparison is between the relative-state formulation, which combines quadrotor odometry with target-relative observations, and the absolute-position variant.Source: 1) Odometry and relative-state… · p. 7

2) Reward decomposition:

  • Dynamic-gate racing performance is reported as gates passed within 512 simulation steps, with higher counts better.The evaluation notes report mean ± standard deviation over random trajectories and time steps.Source: 2) Reward decomposition · p. 7, Table II · p. 7
  • At 1.0, 1.5, and 2.0 m/s, gate counts fall from 16, 12, and 8 to 2, 2, and 0 without gate-passing reward, and to 7, 4, and 2 without differentiable motion reward.Removing either reward component reduces the number of gates passed at all three speeds.Source: 2) Reward decomposition · p. 7

F. Real-World Experiments

DiffDreamer improves training convergence over model-free, model-based, and differentiable-RL baselines, and achieves higher final performance than those baselines and classical planning and control methods. Real-world tests provide proof-of-concept flight validation under calibrated dynamics and motion-capture state feedback, while onboard perception remains untested.

F. Real-World Experiments

  • Trained policies ran onboard at over 100 Hz, and successful trajectories provided proof-of-concept transfer to physical flight under calibrated dynamics and motion-capture state feedback.Tests used bounded thrust commands in a controlled indoor environment.Source: F. Real-World Experiments · pp. 7–8

V. CONCLUSION

  • Experiments on aggressive tracking and dynamic-gate racing show improved training convergence over model-free, model-based, and differentiable-RL baselines, and higher final performance than these baselines and classical planning and control methods.The conclusion also reports ablations of reward decomposition and the odometry-conditioned relative-state representation.Source: §V · p. 8

VI. LIMITATIONS AND DISCUSSION

  • Real-world validation relies on motion-capture feedback for the vehicle and external objects; fully onboard perception and robustness to perception noise remain untested.The evaluation is limited to quadrotors; extending the framework to other embodiments requires suitable differentiable models and additional task-specific engineering.Source: §VI · p. 8

Related papers

Loading 2610.31617v1…