PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural-Primitive: An Efficient End-to-end Local Planner with Primitive-based Imitation Learning for Autonomous Flight

Zhitao Liu, Guangtong Xu, Zihan Wang, Jialiang Hou, Chao Xu, Fei Gao

arXiv:2608.20948v1cs.ROcs.AI

TL;DR

Autonomous flight in unknown cluttered environments requires trajectories that are fast, dynamically feasible, collision-free, and target-reaching under limited onboard computation and memory. Neural-Primitive learns from customized offline trajectory primitives and directly predicts polynomial coefficients from sensory inputs. The planner generates executable trajectories without back-end solving, while benchmarks and zero-shot real-world experiments support its efficiency and robustness.

  • Problem

    Fast generation of dynamically feasible, collision-free, and efficiently target-reaching trajectories remains open for autonomous flight in unknown cluttered environments with onboard resource constraints.

    Source: §I · pp. 1–2

  • Method

    Neural-Primitive uses imitation learning of a customized offline primitive strategy and a compact network that maps sensory inputs directly to polynomial coefficients.

    Source: §I · p. 2, 1) Policy Input-Output Design · p. 5

  • Results

    The planner generates smooth, empirically collision-free, dynamically feasible trajectories without back-end solving and demonstrates consistent superiority across computation time, success rate, flight length, and flight time.

    Source: §I · p. 2, §VI · p. 10

  • Takeaways & Limitations

    Zero-shot deployment in indoor and outdoor real-world experiments validates the proposed planner’s robustness and efficiency.

    Source: §VI · p. 11

    The framework is primarily designed for static obstacles and may struggle with large local traps because it relies on instantaneous limited Field-of-View inputs.

    Source: §VI · p. 11

Abstract

from arXiv · show

Autonomous flight in unknown cluttered environments is hindered by the computation-quality-memory trilemma of onboard trajectory generation. In this paper, we propose an efficient end-to-end local planner via imitation learning. A lightweight offline-primitive-based dataset collection framework is designed to produce safe and high-quality trajectory primitives in non-convex environments. A compact neural network directly maps sensory inputs to polynomial coefficients that inherently encode higher-order dynamical information. The learned policy generates smooth, empirically collision-free and dynamically feasible trajectories in real time without back-end solving. It achieves ultra-fast computation (below 1ms on a standard desktop and average 3.68ms during onboard flight), while maintaining low onboard memory requirements (less than 1.5MiB). Extensive simulation benchmarks demonstrate superiority in both planning latency and target-reaching progress quality. Zero-shot deployment in real-world experiments further validates the robust sim-to-real transfer capability of the proposed method.

I. INTRODUCTION

Neural-Primitive addresses the difficulty of generating fast, dynamically feasible, collision-free, target-reaching trajectories under onboard computation and memory constraints. It combines imitation learning with an offline primitive strategy and directly predicts executable polynomial trajectories.

Motivation

  • Jointly optimizing obstacle avoidance and other factors can produce trade-offs, local minima, or intractable problems, whereas primitive methods decouple obstacle avoidance through staged selection.Primitive-based performance depends heavily on the quality of the primitive library.Source: §I · p. 1
  • The computation-quality-memory trilemma makes simultaneous fast computation, near-optimal target-reaching trajectories, and low onboard memory difficult to achieve.Existing approaches may require online optimization, closed-form solving, or projection after network inference.Source: §I · p. 2

Approach

  • Neural-Primitive uses imitation learning of a customized offline primitive strategy to produce high-quality trajectory primitives in non-convex environments.The framework is designed to mitigate covariate shift without resource-intensive DAgger training.Source: §I · p. 2

Evaluation

  • Extensive simulations evaluate computation time, success rate, flight length, and flight time, while unseen-map and real-world experiments assess generalization and sim-to-real capability.The paper reports superior performance across these evaluation dimensions.Source: §I · p. 2

Approach

  • The compact policy directly predicts polynomial coefficients that encode high-order dynamical information and produce controller-executable trajectories without back-end solving.“Without back-end solving” excludes additional optimization, quadratic programming, or boundary-value problem solving after inference.Source: §I · p. 2

B. Learning-based Motion Planning for UAV

The proposed learning-based planner replaces conventional multi-stage trajectory generation with a three-stage pipeline: simulation data collection, coefficient prediction, and zero-shot onboard deployment. Polynomial outputs preserve dynamical information and avoid post-inference trajectory solving.

Background

  • Existing learning-based planners retain hierarchical planning or require online optimization, closed-form solving, or projection to obtain executable trajectories.Waypoint outputs contain geometric information without dynamical properties and require real-time projection.Source: B. Learning-based Motion Planning for… · pp. 2–3

Pipeline

  • Training datasets are collected entirely in simulation using a customized primitive strategy, with polynomial coefficients recorded instead of only geometric path points.The coefficients inherently encode high-order dynamical information.Source: §III · p. 3, Figure 2 · p. 3
  • An MLP policy is trained offline to generate continuous primitives from discrete samples by predicting polynomial coefficients from sensory inputs.This addresses velocity or acceleration discontinuities in traditional offline primitive-library planning.Source: §III · p. 3

Deployment

  • The learned policy is zero-shot deployed onboard in previously unseen environments without fine-tuning, mapping sensory data directly to executable trajectories.The trajectories are sent to the low-level controller in a receding-horizon fashion.Source: §III · p. 3

1) Collection Procedure:

The collection procedure uses closed-loop simulation with randomized maps and navigation tasks, regenerating and selecting primitives during replanning. Only trajectories from successfully completed tasks are retained for imitation learning.

Collection Procedure

  • The full dataset is collected before training to address covariate shift without relying on resource-intensive DAgger.Data are collected in a closed-loop manner from successful randomly initialized navigation tasks.Source: 1) Collection Procedure · pp. 3–4
  • At each replanning step, candidate terminal states are sampled, primitives are solved, traversed voxels are evaluated, and a valid primitive is selected.The algorithm regenerates the trajectory library conditioned on the current state.Source: 1) Collection Procedure · p. 4
  • Each simulation trial generates a random map and runs up to nthrd navigation tasks in parallel with randomized target positions and drone states.The initial target direction d0 is normalized and computed from the randomized configuration.Source: 1) Collection Procedure · p. 4
  • Selected primitives are stored with velocity, acceleration, target direction, point clouds, and polynomial coefficients, then partially executed to update the drone state.This creates sequential closed-loop training data rather than isolated state-action pairs.Source: 1) Collection Procedure · p. 4
  • Temporary data are appended to the dataset only after the drone reaches the target within a predefined tolerance.Unsuccessful task trajectories are not committed to the final dataset.Source: 1) Collection Procedure · p. 4

2) Trajectory Library with Occupancy Relations:

The trajectory library discretizes terminal states and generates spatially diverse, dynamically feasible polynomial primitives offline. Each primitive is obtained through minimum-control-effort optimization with boundary and dynamical constraints.

Trajectory Library

  • The library uses state-lattice discretization extending to acceleration, while sampling only terminal states because the current initial state is already known.This significantly reduces library size and permits multi-constraint optimization without latency concerns.Source: 2) Trajectory Library with Occupancy… · p. 4

Trajectory Representation

  • Each trajectory primitive is a single-segment polynomial vector function in three-dimensional space, with coefficient vectors for x, y, and z.The polynomial basis contains powers of time through the chosen order.Source: 2) Trajectory Library with Occupancy… · p. 4
  • The position trajectory uses coefficient matrix c and fixed duration τ, with x(t), y(t), and z(t) representing drone positions.The polynomial order is n and the coefficient matrix has dimensions R^(n+1)×3.Source: 2) Trajectory Library with Occupancy… · p. 4

Optimization

  • Primitive coefficients are solved by a convex quadratic program that minimizes control effort under boundary and dynamical constraints.The formulation relies on quadrotor differential flatness.Source: 2) Trajectory Library with Occupancy… · p. 4
  • The constraints encode boundary conditions and dynamical limits, with limits specified by Θlib and dynamical checks performed at uniformly sampled time points.The implementation uses n = 5, Nb = 3, Ns = 20, and τ = 2s for minimum-jerk trajectories.Source: 2) Trajectory Library with Occupancy… · pp. 4–5

Reference Frame

  • Primitives are expressed in a velocity-aligned frame whose x-axis follows velocity and whose z-axis opposes gravity.All primitives share the same origin, and occupancy relations support fast collision checking.Source: 2) Trajectory Library with Occupancy… · p. 5

3) Collision Labeling and Best Primitive Selection:

Perceived point clouds are voxelized for rapid collision labeling, after which safe primitives are ranked by target-reaching and length costs. The expert strategy combines collision removal with constrained minimum-jerk primitive generation and task-oriented selection.

Collision Labeling

  • Fixed-size point clouds are mapped into a bounding box to obtain voxel indices for primitive collision labeling.Unsafe primitives are removed when any indexed point occupies their voxel relations; hash lookup avoids repeated ESDF queries.Source: 3) Collision Labeling and Best… · p. 5

Best Primitive Selection

  • Safe candidate primitives are selected using weighted target-approach and primitive-length costs.The target cost promotes forward progress or prevents overshoot, while the length cost favors direct trajectories over curved or weaving ones.Source: 3) Collision Labeling and Best… · p. 5
  • The target position is expressed in the velocity-aligned frame FV, with primitive costs evaluated at discrete samples ti = iτ/N.N denotes the discrete sample number for each primitive.Source: 3) Collision Labeling and Best… · p. 5

Expert Strategy

  • Each candidate primitive is optimal for a constrained minimum-jerk QP under conditioned boundary states, dynamical limits, and fixed duration.This is minimum-control-effort optimality rather than time-optimality; library selection separately pursues spatial near-optimality through sampling, collision removal, and selection.Source: 3) Collision Labeling and Best… · p. 5

B. Fast End-to-end Trajectory Planning Framework

The fast end-to-end framework integrates sensory mapping, front-end path searching, and back-end trajectory solving into one policy that produces executable trajectories from onboard sensing.

Fast End-to-end Trajectory Planning Framework

  • The end-to-end policy unifies mapping, path searching, and trajectory solving to eliminate intermodule latency and compounding errors.Its outputs are designed to preserve state continuity and target-reaching progress quality while remaining executable.Source: B. Fast End-to-end Trajectory Planning… · p. 5

1) Policy Input-Output Design:

The policy maps current motion state, target direction, and point-cloud observations directly to learned polynomial coefficients. This design preserves state continuity while avoiding further trajectory solving.

Policy Input-Output Design

  • The policy takes velocity magnitude, acceleration, target direction, and 3D point clouds as inputs and outputs nine learned coefficients ca.The learned coefficients encode obstacle avoidance and target-reaching quality, while expert imitation reinforces empirical dynamical feasibility.Source: 1) Policy Input-Output Design · p. 5
  • The coefficient block cb is determined directly from current velocity and acceleration to govern state continuity.The complementary block ca is learned from perceived point clouds and target direction.Source: 1) Policy Input-Output Design · p. 5

Network Architecture

  • Npe2eNet uses a point-cloud encoder, feature fusion module, and coefficient prediction head.It preprocesses arbitrary-sized point clouds to 666 points, extracts a global descriptor, and concatenates it with the 7-D state.Source: 2) Network Architecture · p. 6
  • 0.355M parameters and 1.38MiB of storage characterize the compact network.The network requires approximately 34M FLOPs; reported GPU inference memory is 0.65MiB incremental and 11.49MiB total.Source: 2) Network Architecture · p. 6

3) Training Method:

Training uses simulation-based point-cloud and state perturbations to improve robustness to sensory mismatch, with coefficient regression trained on expert outputs. The implementation supports onboard deployment and event-triggered replanning.

Training Augmentation

  • Simulation training addresses the sim-to-real gap caused by differences in real-flight point-cloud density, spatial distribution, and noisy state measurements.Point-cloud preprocessing and domain-randomization strategies are used to improve zero-shot transfer and generalization.Source: 3) Training Method · p. 6

Loss Function

  • The network minimizes mean squared error between expert coefficients c*_a and predicted coefficients ĉ_a.The coefficient vector contains M = 9 elements.Source: 4) Loss Function · p. 6

Implementation Details

  • The system uses LiDAR sensing, an NVIDIA Jetson Orin NX for end-to-end planning and tracking, and a PX45 flight controller for attitude control.Experiments use randomized maps containing cylinders, rectangular columns, and rings, with one million valid samples collected.Source: A. Implementation Details · p. 6
  • Replanning is triggered by a 0.5m travel threshold, a 2τ/3 execution limit, or detected potential collision.This event-triggered receding-horizon mechanism adapts execution time to flight speed within [0, τ].Source: A. Implementation Details · p. 6

B. Ablation Study

The ablation study evaluates domain randomization, point-cloud preprocessing, and input-size choices, identifying configurations that balance robustness, reliability, latency, and memory.

Domain randomization

  • A2 selects rσs=0.03 and rσp=0.04 for domain randomization, achieving a success rate of 0.936.Adding moderate state and point-coordinate noise improves robustness, whereas excessive noise distorts observations and degrades performance.Source: B. Ablation Study · p. 7

Point-cloud preprocessing

  • Point-cloud preprocessing further improves success rates over the A2-derived B0 baseline in complex scenarios.The module exposes the network to broader point-density variation, while over-sparsification can erode obstacle geometric fidelity.Source: B. Ablation Study · p. 7
  • B2 provides the balanced point-cloud preprocessing configuration after varying preprocessing frequency and minimum density.The supplied passage identifies B2 as balanced but truncates its minimum-density value.Source: B. Ablation Study · p. 7

Point-cloud input size

  • 333 points reduce latency and memory but lower success rate to 0.916, while 1024 and 2048 points add memory cost with marginal success-rate changes.Peak incremental memory rises by 54.3% and 207.8% for 1024 and 2048, respectively.Source: B. Ablation Study · p. 7
  • Input size 666 offers the favorable balance among planning reliability, computational efficiency, and memory cost.The point-cloud encoder operates point-wise, so latency and dynamic memory scale with input size.Source: B. Ablation Study · p. 7

C. Acceleration Continuity Analysis

The paper analyzes acceleration continuity and benchmark performance across obstacle densities and speeds. Neural-Primitive combines smooth higher-order behavior with fast, direct, and robust target-reaching trajectories.

Acceleration continuity

  • Acceleration jumps occur throughout Primitive-swarm flights because its library cannot cover all acceleration states within onboard memory and latency limits.Uncovered initial accelerations during online selection create discontinuities, degrading trajectory quality and increasing motor load.Source: C. Acceleration Continuity Analysis · pp. 7–8
  • Neural-Primitive explicitly incorporates higher-order state continuity into network inputs and outputs, addressing the discontinuity problem.The method produces high-order continuous polynomial trajectories without back-end solving.Source: C. Acceleration Continuity Analysis · p. 8

Computation

  • 0.70ms in sparse and 0.68ms in dense environments is Neural-Primitive’s average computation time, about 7 to 60 times faster than baselines.Its efficiency remains stable across obstacle densities because planning is integrated end-to-end.Source: D. Benchmarks · p. 9

Trajectory quality

  • 64.51m in dense and 64.39m in sparse environments are Neural-Primitive’s near-direct flight distances, with the shortest flight time to the target.The customized primitive strategy separates collision removal from task-oriented selection.Source: D. Benchmarks · p. 9

Energy cost

  • Lower accumulated squared-jerk than Ego and Super is achieved by Neural-Primitive while retaining more direct target-reaching motion.Fast and Yopo obtain lower energy cost at the expense of substantially longer detours.Source: D. Benchmarks · p. 9

Speed robustness

  • Above 0.80 success rates at average speeds around 6m/s are maintained by Super and Neural-Primitive, unlike Fast, Ego, and Yopo.Neural-Primitive matches Super’s robustness while avoiding extensive hand-crafted rules and pronounced detours.Source: D. Benchmarks · p. 9

E. Generalization Evaluations

Generalization is evaluated without fine-tuning in unseen forest and unstructured terrain maps, alongside onboard planning-time statistics. The learned policy retains substantial traversal success and real-time efficiency.

Unseen environments

  • 0.907 and 0.833 success rates are achieved in unseen forest and cave-like mountainous maps, respectively, without fine-tuning.Each map type uses 150 trials with randomly initialized start-target separations of 70∼80m.Source: E. Generalization Evaluations · p. 9
  • The forest test includes trunks, branches, vegetation, and spatially distributed noise points from a real-world point-cloud map.The map is cropped to 60×50×20m.Source: E. Generalization Evaluations · p. 9, Figure 6 · p. 10
  • The simulated terrain test uses irregular obstacles with varying shapes and densities that differ from the training datasets.The map resembles caves and mountainous terrain and measures 60×50×10m.Source: E. Generalization Evaluations · p. 9, Figure 6 · p. 10

Onboard efficiency

  • 3.68ms is the mean onboard planning time throughout the flight.The distribution characterizes the proposed planner’s onboard real-time efficiency.Source: Figure 7 · p. 10

F. Real-world Experiments

Real-world tests used previously unseen outdoor and indoor environments, with training performed solely in simulation and no real-world fine-tuning. The planner achieved collision-free forest traversal, low onboard planning latency, and safe operation through narrow passages.

Experimental Setup

  • Simulation-only training transferred to previously unseen outdoor forests and indoor cluttered environments without real-world fine-tuning.The experiments covered both outdoor unknown forests and indoor environments with cluttered obstacles.Source: F. Real-world Experiments · p. 10

Outdoor Navigation

  • 60m forest traversal completed within 12s without collisions, reaching the target within 1m under wind and sensor noise.The quadrotor reached a maximum speed of 6.10m/s while maintaining a nearly straight trajectory.Source: F. Real-world Experiments · p. 10

Onboard Efficiency

  • 3.68ms mean onboard planning time was measured throughout the entire flight, with 2.71ms–5.94ms values and ±0.77ms standard deviation.The median planning time was 3.43ms.Source: F. Real-world Experiments · p. 10

Cluttered Environments

  • Safe traversal remained possible through thick foliage with a narrowest passage under 0.8m, despite the quadrotor diameter being approximately 0.3m.Indoor experiments additionally required sequential visits to multiple preset targets within 0.5m tolerance.Source: F. Real-world Experiments · p. 10, Figure 8 · p. 10

Planner Design

  • The planner directly outputs controller-executable polynomial trajectories without back-end solving, forming an integrated end-to-end planning scheme.The conclusion links this design to inherited offline-primitive trajectory quality and online-network continuity.Source: §VI · p. 10

Limitations

  • The framework is primarily limited to static obstacles and may struggle with large local traps under instantaneous limited Field-of-View inputs.Highly dynamic obstacles, dead ends, and U-shaped obstacles remain identified future-work targets.Source: §VI · p. 11

Related papers

Loading 2608.20948v1…