PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic

Nuthasith Gerdpratoom, Tianchen Sun, Yichao Gao, Lin Zhao

arXiv:2609.30696v1cs.ROcs.SYeess.SY

TL;DR

Narrow-gap traversal challenges visual quadrotor policies that must infer geometry from observations, while existing approaches can rely on behavior cloning or full-rollout BPTT. The paper trains a privileged expert and critic before warm-starting critic-guided learning of a visual policy with QPG. The authors report improved visual-policy learning, traversal of unseen gap shapes, and real-world experiments, while noting that the masks were rendered from motion-capture poses.

Abstract

from arXiv · show

Traversing narrow gaps is challenging for autonomous quadrotors, especially when control commands come directly from high-dimensional visual observations. Existing end-to-end methods often rely on behavior cloning or full-rollout backpropagation through time (BPTT) via differentiable simulation, which can limit policy performance or incur high training costs. We propose a two-stage reinforcement learning framework for more efficient ego-centric visuomotor gap-traversal policy training, leveraging quasi-analytical policy gradients (QPG) via differentiable simulation and critic warm-starting. The framework utilizes QPG to avoid backpropagation through visual rendering, reducing computation and memory costs while improving sample efficiency. In the first stage, an expert actor and critic are trained using privileged observations, including gap geometry. Unlike prior gap-traversal approaches, our training utilizing QPG does not require resetting the agent along optimized reference trajectories. In the second stage, a visual policy is trained using binary gap masks from two ego-centric cameras and low-dimensional observations, while its privileged critic is warm-started from the first stage. This substantially improves training efficiency and traversal success compared with cold-starting the critic or using full-rollout BPTT. Our framework does not require retraining the expert actor when system parameters change, enabling more efficient generalization across drone platforms than state-of-the-art visual gap-traversal methods based on action supervision. The learned visual policy also generalizes to gaps with unseen shapes. Extensive real-world experiments further demonstrate robust gap traversal using binary masks rendered online. Beyond gap traversal, the proposed framework is generic and can be extended to other visuomotor robot learning tasks.

I. INTRODUCTION

Agile gap traversal requires precise attitude control, while visual policies must infer gap geometry from high-dimensional cues. The paper proposes two-stage QPG training with a privileged critic warm-start to train a visual policy without trajectory resets or expert retraining for plant changes.

I. INTRODUCTION

  • Visual gap traversal requires inferring geometry from high-dimensional cues coupled to agile motion, and the setup allows only 6 cm lateral clearance per side and 23° roll misalignment.The cited prior approaches distill experts trained with known gap geometry, and their action supervision is optimal only for the training plant.Source: §I · p. 1
  • QPG avoids gradients through visual observations, but its reliance on the critic motivates pretraining that critic before visual-policy learning.The cited discussion contrasts this with full-episode BPTT, which can cause exploding gradients, and a randomly initialized critic that initially offers no guidance.Source: §I · p. 2
  • The framework trains a privileged expert and critic with QPG, then warm-starts the critic to train a visual policy from two ego-centric binary masks and low-dimensional inputs.The authors report generalization to unseen gap shapes and real-world traversal with masks rendered online.Source: §I · p. 2

A. Agile Gap Traversal for Quadrotors

  • Prior quadrotor work includes geometric trajectory planning and learning from demonstrations or interaction; this policy instead uses binary gate masks and privileged critic inputs without trajectory optimization or distillation.The related-work passage also notes that prior distilled students are trained from experts reset along optimal trajectories.Source: A. Agile Gap Traversal for Quadrotors · p. 2

B. Policy Optimization via Differentiable Simulator

The paper formulates agile traversal as collision-free passage through a gate followed by stable hovering at a goal. It motivates differentiable-simulator learning with short-horizon, critic-supported gradients and uses binary masks rather than depth images.

B. Policy Optimization via Differentiable Simulator

  • Differentiable-simulator policy gradients can be sample-efficient, but long-horizon BPTT can produce chaotic gradients; short-horizon, decoupled formulations motivate the paper’s QPG approach.The paper uses binary masks instead of depth images, which omit direct gap geometry and therefore pose an additional learning challenge.Source: B. Policy Optimization via… · p. 2

A. Problem Formulation

  • The task requires the quadrotor to pass through the gate without collision and then hover stably at the goal; success requires clean traversal and subsequent stabilization.Clean traversal requires the vehicle’s bounding-box centroid to pass through the opening without intersecting the gate frame at any time step.Source: A. Problem Formulation · pp. 2–3
  • The gate is modeled as a thin planar frame with a rectangular opening, while the quadrotor’s collision geometry is an oriented bounding box.Fig. 2 visualizes the collision box and opening dimensions in Isaac Sim.Source: A. Problem Formulation · p. 2, Figure 2 · p. 3
  • The deployed policy maps a gate binary mask and low-dimensional observations to a stochastic control action, which is sent as thrust and body-rate commands to an autopilot.The listed low-dimensional inputs include roll, pitch, accelerometer readings, and the goal position in body frame.Source: A. Problem Formulation · p. 3

B. Quadrotor Dynamics

The simulator models rigid-body motion, attitude, and actuator response to represent quadrotor dynamics. It discretizes the actuator models before integrating the rigid-body state.

B. Quadrotor Dynamics

  • The dynamics passage defines world-frame position and velocity, body-to-world quaternion attitude, body-frame thrust force, mass, and gravity.The attitude evolves under quaternion kinematics.Source: B. Quadrotor Dynamics · p. 3
  • Achieved body rates are represented in the body frame, and the autopilot’s closed-loop body-rate response is approximated by a second-order low-pass filter.The filter has internal states for body-rate acceleration and body rate.Source: B. Quadrotor Dynamics · p. 3
  • Thrust is modeled with a first-order filter to represent actuation delay, parameterized by a time constant.Source: B. Quadrotor Dynamics · p. 3
  • At each physics substep, the model updates thrust and body rate using discretized inner-loop dynamics, then integrates the rigid-body dynamics.The passage also specifies maximum thrust and body-rate scaling for control inputs.Source: B. Quadrotor Dynamics · p. 3

C. Policy Optimization via Differentiable Simulation

The policy-learning method uses QPG with a short-horizon actor loss and a privileged critic to account for returns beyond the rollout. A pretrained critic is transferred to guide visual-policy learning from its first update.

C. Policy Optimization via Differentiable Simulation

  • QPG minimizes an actor loss combining discounted rewards over a short horizon with a terminal value estimate from privileged observations.The rollout includes the sensor model, policy, and dynamics; the recurrent policy state is present only in stage two.Source: C. Policy Optimization via… · p. 3
  • QPG treats observations as constants during differentiation, removing the observation-feedback term while retaining a first-order policy gradient.The critic contributes beyond the short horizon through the gradient of its value with respect to privileged observations.Source: C. Policy Optimization via… · pp. 3–4
  • The two-stage method trains an expert actor and critic with privileged state and gate geometry, then copies the critic to warm-start visual-policy learning.The visual actor uses binary masks from two ego-centric cameras and low-dimensional inputs; the critic is trained asymmetrically with privileged observations.Source: C. Policy Optimization via… · p. 4

D. Two-Stage Training with Warm-Started Value Prior

The method trains a privileged expert before learning a visual policy, warm-starting the latter with the expert’s critic. Its reward combines progress, gate alignment, clearance, hovering, and action regularization.

D. Two-Stage Training with Warm-Started Value Prior

  • The visual policy uses two ego-centric camera masks and low-dimensional inputs, while its value function is warm-started from the privileged expert’s critic.The expert has access to gate geometry and quadrotor states; transferring the learnable critic provides a task-shaped terminal adjoint from the first policy update.Source: D. Two-Stage Training with… · p. 4
  • The high-level reward combines progress, alignment, clearance shaping, hovering, and action regularization.Progress targets the gate centroid before traversal and the goal afterward; alignment encourages matching attitude with the gate during traversal.Source: D. Two-Stage Training with… · p. 4
  • Clearance shaping rewards each box corner’s clearance and penalizes corners protruding outside the aperture.The clearance term uses the distance from a box corner to the nearest aperture edge.Source: D. Two-Stage Training with… · p. 4
  • The hovering term encourages approach to the goal and stable hovering, while action regularization penalizes actions and their variation.Action regularization is intended to promote energy-efficient trajectories and prevent actuator degradation.Source: D. Two-Stage Training with… · p. 5

E. Simplified Jacobian

A simplified surrogate model is used for gradient computation to avoid expensive and potentially destabilizing gradients through the full dynamics. State alignment preserves the accurate model’s forward-pass value while gradients flow through the simplified dynamics.

E. Simplified Jacobian

  • The full model’s inner-loop dynamics produce stiff gradients, and its physics substeps make gradient computation expensive.The surrogate assumes instantaneous body-rate tracking and omits thrust lag.Source: E. Simplified Jacobian · p. 5
  • The simplified surrogate computes gradients with semi-implicit Euler integration and no physics substeps at the control interval.State alignment makes the forward-pass value agree with the accurate state, while backpropagation uses the simplified dynamics.Source: E. Simplified Jacobian · p. 5
  • Using the simplified model enhances training without sacrificing performance compared with using the full model for gradients.Figure 4 compares gradient computation time and success rate for the simplified and full models.Source: E. Simplified Jacobian · p. 5

F. Visual Observation Renderer

The renderer produces binary gate-aperture masks by projecting gate corners into each ego-centric camera and rasterizing the aperture edges. The two camera masks are concatenated as the visual observation.

F. Visual Observation Renderer

  • The gate’s binary mask is used instead of photorealistic or depth imagery, retaining gap geometry while bypassing texture, lighting, and motion blur.The mask is analytic and can be vectorized with the simulator.Source: F. Visual Observation Renderer · p. 5
  • Gate corners are projected into pixel coordinates, and aperture edges are constructed cyclically between successive projected corners.The renderer tests pixel centers against the aperture edges to define the mask.Source: F. Visual Observation Renderer · p. 5
  • When a gate corner lies behind the camera, the renderer slides the edge endpoint to the camera plane before projecting it.The process is repeated for the second camera, and the two masks are concatenated for the policy.Source: F. Visual Observation Renderer · p. 5

G. Training Details

Training uses 500-step episodes at 100 Hz, a 64-step short horizon, and 1,024 parallel environments. The policy and critic are optimized for 25,000 iterations with linearly decayed learning rates.

G. Training Details

  • Episodes run for 500 steps at 100 Hz, with a short-horizon length of 64.Both stages use a discount factor of 0.99 and TD-λ critic learning with λ = 0.95.Source: G. Training Details · p. 5
  • The policy is optimized with Adam for 25,000 iterations across 1,024 parallel environments on NVIDIA A40 GPUs.Policy and critic learning rates start at 5×10−4 and 2 × 10−3, respectively, and decay linearly.Source: G. Training Details · p. 5

IV. EXPERIMENTS

The experiments evaluate QPG pretraining against PPO, full-episode BPTT, and a no-value-gradient ablation. QPG reaches high success much sooner than BPTT, while removing the value gradient leaves success substantially lower.

IV. EXPERIMENTS

  • The evaluation compares QPG pretraining with full-episode BPTT, PPO, and QPG without the critic gradient, using three random seeds for reported results.The section also evaluates warm-started visual learning, unseen gap shapes, real-world performance, and comparison with a two-stage reinforcement–imitation approach.Source: §IV · p. 5, Figure 5 · p. 6

A. Pretraining a Task-Informed Critic

  • QPG reaches 0.8 success in 12 minutes and saturates near 1.0, while full-episode BPTT reaches similar success only after 16000 epochs.PPO remains at zero for the first 19,500 epochs in the reported settings; QPG without the value gradient plateaus at 0.38.Source: A. Pretraining a Task-Informed Critic · p. 6
  • QPG backpropagates through 64 steps and uses the critic gradient for long-term credit assignment, whereas BPTT differentiates through 500-step dynamics Jacobians.Without the value gradient, the update accounts for only 64 steps ahead, and the success curve plateaus.Source: A. Pretraining a Task-Informed Critic · p. 6

B. Vision-based Policy Learning

The visual policy combines two ego-centric binary-mask cameras with low-dimensional observations, while a pretrained critic warm-starts its training. It achieves a peak success rate of 0.893, traverses unseen gap shapes, and can be retrained for revised plant dynamics without retraining the expert.

B. Vision-based Policy Learning

  • The policy processes two ego-centric binary masks with low-dimensional observations through a CNN, GRU, and MLP, fusing information across camera views early.The GRU carries temporal context across the rollout.Source: B. Vision-based Policy Learning · p. 6
  • The visual policy achieves a peak success rate of 0.893; cold-start QPG outperforms BPTT but not the two-stage approach, while a frozen critic remains below 0.25.The two outward-splayed cameras improve performance slightly over one camera, and the same rectangular-trained policy completes elliptical, trapezoidal, and triangular gap traversals.Source: B. Vision-based Policy Learning · p. 6, Figure 7 · p. 7
  • A pretrained critic supplies task-shaped guidance from the first epoch and can be adapted to the visual policy’s state distribution, unlike a frozen critic.A cold-start critic initially supplies an arbitrary gradient direction while the policy is learning.Source: B. Vision-based Policy Learning · p. 6

C. Transfer Under a Plant Revision

  • Under revised plant parameters, the two-stage approach reuses the old pretrained critic and retains performance without retraining the expert policy.Behavior cloning from the old expert degrades significantly when plant parameters differ from those used to train that expert.Source: C. Transfer Under a Plant Revision · p. 7

D. Real-world Experiments

Real-world trials executed the visual policy with online-rendered binary masks at four gate roll angles, reaching an average maximum speed of 3.5 m/s. The conclusion reports critic warm-starting, unseen-shape traversal, and adaptation to changed plant parameters without retraining the expert policy, while noting that masks came from motion-capture poses.

D. Real-world Experiments

  • The visual policy ran on a 1.279 kg quadrotor, with an NVIDIA Jetson Orin NX onboard computer and a Pixhawk 6X Mini flight controller.The policy was exported to ONNX and ran through its Runtime C++ on the Jetson.Source: D. Real-world Experiments · p. 7
  • The real-world policy traversed a rectangular gate at 45°, 60°, 70°, and 80° roll angles, with an average maximum speed of 3.5 m/s.The gate mask was rendered online from motion-capture poses in a confined 4 × 4 m area with a 2 m operating height.Source: D. Real-world Experiments · p. 7, Figure 9 · p. 7

V. CONCLUSION

  • The conclusion reports that warm-starting the privileged critic improves visual-policy training, and that the learned policy traverses gaps with unseen shapes.It also reports an advantage over the prior RL-imitation method when plant parameters change: the expert policy need not be retrained.Source: §V · pp. 7–8
  • When plant parameters change, the approach reruns only the second training stage with the old pretrained critic, avoiding expert-policy retraining.The comparison uses behavior cloning with revised-plant experience and action labels queried from the old expert.Source: §V · p. 8
  • In the real-world experiment, the binary gate mask came from motion-capture poses rather than onboard cameras, so perception flaws were neglected.The paper identifies a robust gate predictor or an end-to-end raw-image policy as ways to eliminate online mask synthesis.Source: §V · p. 8

Related papers

Loading 2609.30696v1…