PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Visuomotor Quadrotor Flight Through Dynamic Narrow Gaps via Differentiable Simulation

Riccardo Bianco, Yunfan Ren, Jiaxu Xing, Davide Scaramuzza

arXiv:2610.37557v1cs.RO

TL;DR

Narrow moving gaps demand precise alignment and timing, yet end-to-end learning without privileged state or a teacher remains open. The paper trains a visual-feature-to-command policy using differentiable simulation and a time-varying gap objective. It reports high simulated success and successful transfer to real quadrotor flight.

  • Problem

    Moving-gap traversal requires precise spatial and temporal control, while prior methods often rely on privileged state, teachers, planners, or distillation.

    Source: §1 · p. 2

  • Method

    A differentiable moving-gap objective gates centering, alignment, and smooth collision costs by signed distance to the time-varying gap plane for a policy trained from random initialization.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    99.3% success over 1,000 simulated rollouts and 80% success over 10 real-world trials were reported for moving-gap traversal.

    Source: §1 · p. 2, §4 · p. 6

  • Takeaways & Limitations

    The visual-inertial policy performs closed-loop moving-gap traversal on a real quadrotor using onboard sensing.

    Source: §1 · p. 2

Abstract

from arXiv · show

Threading a quadrotor through a moving narrow gap is a demanding test of agile flight, requiring centimeter-level spatial alignment and precise temporal execution. We address this challenge using a control policy that operates directly on onboard visual and inertial measurements to output collective thrust and body-rate commands. The task is difficult because success lies on a thin feasible manifold: small errors in timing, position, attitude, or visual observability cause collision. Prior planning and learning methods rely on privileged state, teachers, planners, or distillation; learning dynamic narrow-gap traversal directly, from scratch, remains open. We introduce a differentiable moving-gap formulation that gates centering, alignment, and smooth collision terms by signed distance to a time-varying gap plane, turning a sparse traversal event into a dense pathwise objective. We integrate this formulation into a visual-feature-to-command policy trained in one stage from random initialization, without a teacher, planner, or distillation. We validate our approach in both synthetic and real-world experiments. In controlled benchmarks, we demonstrate that pathwise gradients significantly outperform score-function gradients for precision-critical tasks, yielding a 99.3% success rate in simulation after only a few minutes of training on a single GPU. Furthermore, the policy transfers directly to a real quadrotor, successfully crossing a dynamic narrow gap from varied initial conditions using onboard sensing and inertial measurements, achieving an 80% success rate over 10 physical trials.

1 Introduction

The paper studies end-to-end visual-inertial control for crossing static and moving narrow gaps, where success requires precise position, attitude, and timing. It introduces a differentiable moving-gap objective and evaluates a policy trained from random initialization through simulation and real-world flight.

1 Introduction

  • Moving-gap traversal requires precise geometric alignment and timing, while prior methods often use static apertures, privileged geometry, or teacher and planner pipelines.The paper frames learning this task end-to-end from random initialization without a planner, teacher, or distillation as an open question.Source: §1 · p. 2
  • Pathwise gradients through smooth closed-loop dynamics provide failed rollouts with guidance for adjusting centering, attitude, and timing, unlike noisy score-function estimates on thin-manifold tasks.The authors note that pathwise gradients can also become unreliable with stiffness, discontinuities, contacts, or chaotic dynamics.Source: §1 · p. 2
  • A signed-distance-gated moving-gap objective supplies dense pathwise costs to a visual-feature-to-command policy trained from random initialization without demonstrations, a teacher, a planner, or distillation.The formulation gates centering, alignment, and smooth collision terms by distance to the time-varying gap plane.Source: §1 · p. 2
  • 80% success over 10 real-world trials demonstrates moving-gap traversal by a visual-inertial policy on a physical quadrotor.In simulation, the visual-feature-based moving-gap policy reached 99.3% success over 1,000 rollouts.Source: §1 · p. 2

2 Related Work

Prior narrow-gap and visuomotor flight methods often depend on known or privileged geometry, expert guidance, or large sample budgets. This work contrasts with those approaches by training moving-gap traversal end-to-end from random initialization and examining pathwise gradients.

Narrow-Gap Quadrotor Flight

  • Earlier narrow-gap methods include trajectory optimization, privileged-pose learning, and planner bootstrapping; this work targets moving gaps from random initialization without a teacher, planner, or distillation.The paper distinguishes its treatment of moving gaps as a trained task from prior work where they were a reactive byproduct.Source: §2 · p. 3

End-to-End Visuomotor Quadrotor Policies

  • Vision-conditioned quadrotor policies have used privileged planners, demonstrations, or large RL budgets, whereas the authors aim to avoid those requirements with pathwise gradients.The cited examples include forest flight by planner distillation and racing via deep RL with high-frequency state estimation.Source: §2 · p. 3

Differentiable Simulation for Robot Learning

  • Differentiable simulation has supported analytic-gradient control across robotics, while concurrent work applied it to irregular static gaps; this paper targets static and dynamic gaps and compares against PPO.The comparison is described as using matched parallelism.Source: §2 · p. 3

3 Methodology

The method formulates static and moving gap traversal as closed-loop quadrotor control, using either privileged state or onboard visual-inertial features. It optimizes a policy through differentiable dynamics with costs shaped around the time-varying gap plane and trains with randomized batched rollouts.

3.1 Problem Formulation

  • A traversal succeeds when the quadrotor’s ellipsoid projection lies fully within the rectangular aperture as it crosses the gap plane.The task includes static gaps and moving gaps with time-varying pose.Source: §3.1 · p. 4
  • The visual observation combines onboard inertial quantities, projected gap-corner features, temporal differences, and action history without global pose information.The policy maps observations to collective-thrust and body-rate commands, applied to simulated dynamics with a one-step delay.Source: §3.1 · p. 4

3.2 Differentiable Simulation and Objective

  • Backpropagation through differentiable closed-loop rollouts optimizes the policy using per-step costs for progress, centering, alignment, collision avoidance, stabilization, and camera visibility.Centering, alignment, and smooth collision costs are modulated by signed distance to the time-varying gap plane; stabilization is relaxed during crossing.Source: §3.2 · p. 5

3.3 Training Setup

  • Training uses randomized initial conditions and domain parameters, batched BPTT rollouts, and a two-layer MLP optimized with Adam and a warmup–cosine schedule.The setup uses 100 parallel environments for up to 20k epochs and selects the best validation checkpoint.Source: §3.3 · p. 5

4 Experiments and Results

The experiments compare pathwise and score-function gradients, then evaluate differentiable-simulation policies across state- and vision-based static and moving-gap tasks. They report strong simulated performance, reduced transfer from static training, and real-world traversal, alongside comparisons with PPO and analysis of visual-history windows.

Pathwise vs. Score-Function Gradients

  • Pathwise gradients solved slit navigation with one rollout per update, while score-function gradients failed with up to 4,096; their LQR variance scaled as O(T) versus O(T^2).The rollout ratio applies to the controlled slit task, not quadrotor PPO transitions.Source: §4 · p. 5, Figure 4 · p. 6

Efficiency and Performance

  • Differentiable simulation achieved near-perfect success and centimeter-level crossing accuracy across task variants, with slightly lower performance and slower convergence under vision-based observations.Table 1 reports success, crossing error, and convergence epochs; Fig. 5 shows qualitative rollouts across task variants.Source: §4 · p. 6, Table 1 · p. 6, Figure 5 · p. 7

Static-to-Moving Transfer

  • 31.5% state-policy and 6.6% vision-policy success after static-only training rose to 100% and 99.3%, respectively, after moving-task training.Because the state and vision tasks also differ in gap-pose distribution, the transfer experiment does not isolate timing.Source: §4 · p. 6

Transfer and Real-World Deployment

  • 99.3% simulated success for the vision-based moving-gap policy fell to 81% in high-fidelity simulation, while 8/10 verifiable real flights succeeded.Transfer evaluations were not used for training or checkpoint selection; the authors caution that ten trials cannot isolate failure causes.Source: §4 · pp. 6–7, Table 2 · p. 6

Comparison with PPO

  • On moving gaps, PPO used 19× more transitions and 45.5× more training time than BPTT across converged-seed means.The comparison uses tuned PPO with a curriculum, so the full pipeline difference cannot be attributed solely to the estimator; epochs are not compute-equivalent.Source: §4 · p. 8, Table 3 · p. 8

Partial Observability in Visual-Feature-Based Policies

  • Larger visual-history windows converged earlier, while final success varied by only 0.7 points; K = 1 still retained R, ω, and the four-command queue.The study did not compare an RNN, which may help during longer dropouts.Source: §4 · p. 8

5 Discussion and Limitations

Differentiable simulation achieves near-perfect simulated success and centimeter-level accuracy across static and planar pendulum-driven moving gaps. The proof of concept remains bounded by controlled sensing and motion conditions, with broader gap motions and marker-free perception left for future work.

5 Discussion and Limitations

  • Near-perfect simulated success and centimeter-level accuracy are achieved across the tested static and planar pendulum-driven moving gaps.The complete BPTT pipeline also uses fewer transitions and less training time than tuned PPO, though the pipeline difference cannot be attributed solely to the estimator.Source: K Success (%) Convergence epoch · p. 8
  • Training uses privileged simulator geometry and collision data, while visual input uses engineered features and physical evaluation uses LED-based corners and planar pendulum motion.Natural openings, abrupt or non-periodic motion, severe or prolonged visual dropout, hard contact, and long-horizon instability remain untested; ten scored flights leave broad statistical uncertainty.Source: K Success (%) Convergence epoch · p. 9
  • Future work includes marker-free perception, recurrent policies for longer visual dropout, broader gap-motion evaluation, and learnable residual or more accurate differentiable dynamics.Short-horizon pathwise methods with bootstrap values may suit longer tasks better than full-horizon BPTT.Source: K Success (%) Convergence epoch · p. 9
  • Differentiable simulation offers fast, stable optimization for high-precision flight in regimes where model-free RL is limited by gradient variance and sample complexity.Table 4 reports vision-moving performance as temporal window length K varies, but the supplied passage gives no cell values.Source: Table 4 · p. 8, K Success (%) Convergence epoch · p. 9

A.1 Gradient Analysis and Properties of Pathwise Estimators

This analysis compares pathwise and score-function gradient estimators and examines variance dependence on closed-loop stability and rollout count. It uses full-horizon APG/BPTT for a smooth 150-step rollout rather than proposing a new estimator or optimizer.

A.1 Gradient Analysis and Properties of Pathwise Estimators

  • The analysis compares estimator variance against closed-loop stability margin and examines score-function behavior as parallel rollout count changes in narrow-passage navigation.Experiments use PyTorch and are reproducible with released scripts.Source: §A.1 · p. 13
  • Full-horizon APG/BPTT is used for the smooth 150-step rollout; the section proposes neither a new gradient estimator nor a new optimizer.Short-horizon pathwise methods with a bootstrap value may suit longer horizons, and other pathwise methods were not compared.Source: §A.1 · p. 13
  • The analysis probes estimator variance across a controlled dynamical system and score-function performance across parallel rollout counts in narrow-passage navigation.The section describes these as additional analyses complementing Fig. 4.Source: §A.1 · p. 13

A.2 Common Setup and Hyperparameters

The setup compares pathwise and score-function estimators under matched optimization settings, using a controlled linear system and a planar narrow-passage task. The navigation experiment evaluates success under perturbed starts.

A.2 Common Setup and Hyperparameters

  • The pathwise estimator differentiates the loss through a deterministic rollout, while the score-function estimator uses REINFORCE with a leave-one-out baseline and N independent rollouts per update.The comparison matches optimizer, learning rate, and gradient-update count; PPO is noted as potentially improving optimization stability without changing the underlying credit-assignment mechanism.Source: §A.2 · p. 13
  • The controlled-system experiment uses a stochastic linear policy on a one-dimensional linear system with quadratic loss, evaluated at θ = −0.5 unless stated otherwise.Default parameters are a = 1.0, b = 1.0, λ = 0.1, σw = 0.1, σπ = 0.2, and s0 = 1.Source: §A.2 · p. 13
  • Narrow-passage navigation uses a planar control problem with start (−1, 0), goal (1, 0), horizon T = 40, and bounded controls; both estimators optimize the same reward.The MLP is 2 →16 →2 with tanh activations; PW uses one deterministic rollout, while SF is tested with N ∈{4, 16, 64, 256, 1024, 4096}, using Adam at 5 × 10−3 across eight seeds. Success is evaluated over 100 perturbed-start rollouts and requires crossing x = 0.3 and ending within radius 0.25 of the goal.Source: §A.2 · p. 13

A.3 Variance Scaling: Horizon and Stability Regime

The analysis reports different horizon-variance scaling for score-function and pathwise estimators in the stable regime, while showing that pathwise advantages diminish near instability. Its conclusions are scoped to smooth, locally stable dynamics.

A.3 Variance Scaling: Horizon and Stability Regime

  • At T = 20, pathwise gradient estimates have standard deviation 2.38 versus 27.93 for score-function estimates, a per-trajectory variance ratio of ∼140×.Both estimators are unbiased around the true gradient in the LQR analysis; pathwise retains a >102× variance advantage up to the stability bifurcation, after which both saturate.Source: Figure 9 · p. 14
  • In the stable regime, score-function variance scales as Θ(T^2), while pathwise variance scales as Θ(T).The score-function scaling follows from a linearly growing trajectory return multiplying a sum of T score terms with residual variance Θ(T); bounded pathwise Jacobian norms yield Θ(T).Source: §A.3 · pp. 13–14
  • As the closed-loop pole a + bθ approaches unity, both estimators degrade; pathwise advantage holds up to the bifurcation at a = 1.5 in the stated sweep.The analysis presents this as a regime-specific advantage for smooth, locally stable, moderately long-horizon dynamics, not a new stability boundary.Source: §A.3 · p. 14, Figure 9 · p. 14

A.4 Signal vs. Variance in Narrow-Passage Problems

Increasing sample count reduces estimator variance but does not resolve the near-zero reward signal before a successful passage is discovered. Pathwise updates can use analytic spatial gradients before success, while score-function trajectories remain outside the success region.

A.4 Signal vs. Variance in Narrow-Passage Problems

  • All score-function configurations plateau at ≈0.10–0.12 success, even with N = 1,024 and 4,096, because the pre-discovery reward signal remains near-zero.Under Gaussian exploration, successful trajectories are effectively never sampled, leaving the reward uninformative regardless of N.Source: §A.4 · p. 14
  • Pathwise updates use the analytic spatial gradient before success, progressively steering trajectories toward the feasible manifold; score-function trajectories remain outside the success region.The score-function credit signal is weak and dominated by sampling noise.Source: §A.4 · p. 14

A.5.1 State-Based Observations

The appendix details state-based and visual-feature-based observations, the differentiable trajectory costs and quadrotor model, and the randomization and training procedures used in the experiments.

A.5.1 State-Based Observations

  • State-based observations combine the full quadrotor state, gap pose, goal waypoint, and a history queue of commanded thrust and body-rate actions.The queue spans the maximum perception–actuation delay; with the stated nominal settings, it contains four past actions.Source: §A.5.1 · p. 15

A.5.2 Visual-Feature-Based Observations

  • Visual-feature observations combine inertial measurements with projected gap-corner features, derived geometry, temporal differences, and action history; when all retained frames are invalid, only inertial data and action history remain.The experiments use a four-frame window, and any invalid corner collapses that frame’s geometric features to zero.Source: §A.5.2 · pp. 15–16

A.6 Perception Pipeline

  • Real-world gap corners are extracted from LED-bright regions by HSV thresholding, morphological cleanup, and quadrilateral fitting, under strong assumptions about visual structure.The detected corners define the visual features supplied to the policy.Source: §A.6 · pp. 16–17, Figure 10 · p. 17

A.7 Reward Details

  • The trajectory objective combines progress, centering, alignment, smooth collision penalties, stabilization, and camera-view costs, with most terms shaped by signed distance to the gap plane.The collision formulation smoothly approximates geometric constraints and remains differentiable for gradient-based optimization.Source: §A.7 · pp. 17, 19

A.8 Quadrotor Dynamics

  • The dynamics model represents position, velocity, orientation, and body angular velocity, updates orientation by the exponential map, and supplies the differentiable transition used for BPTT.The orientation update is exact for constant angular velocity over an integration step.Source: §A.8 · pp. 19–20

A.9 Domain Randomization Distributions

  • Domain randomization varies initial states, gap motion, perception noise and dropouts, perception–actuation delay, and thrust to emulate deployment uncertainties.Moving-gap episodes randomize pendulum amplitude, phase, and length; the delay range is chosen from measured deployment latency.Source: §A.9 · pp. 20–21

A.10 Training Details

  • BPTT uses Adam and a warmup–cosine schedule, while PPO uses tuned hyperparameters and reward-weight annealing; without that curriculum PPO fails to learn traversal under the full objective.BPTT uses fixed task weights and does not require a curriculum because pathwise gradients provide signal to optimize the cost components jointly from the start.Source: §A.10 · p. 21, Table 6 · p. 21

Related papers

Loading 2610.37557v1…