Neural-Primitive: An Efficient End-to-end Local Planner with Primitive-based Imitation Learning for Autonomous Flight

Zhitao Liu; Guangtong Xu; Zihan Wang; Jialiang Hou; Chao Xu; Fei Gao

Source-linked AI summary

Neural-Primitive: An Efficient End-to-end Local Planner with Primitive-based Imitation Learning for Autonomous Flight

Zhitao Liu, Guangtong Xu, Zihan Wang, Jialiang Hou, Chao Xu, Fei Gao

arXiv:2608.20948v1Aug 21, 2026cs.ROcs.AI

TL;DR

Autonomous flight in unknown cluttered environments requires trajectories that are fast, dynamically feasible, collision-free, and target-reaching under limited onboard computation and memory. Neural-Primitive learns from customized offline trajectory primitives and directly predicts polynomial coefficients from sensory inputs. The planner generates executable trajectories without back-end solving, while benchmarks and zero-shot real-world experiments support its efficiency and robustness.

Problem
Fast generation of dynamically feasible, collision-free, and efficiently target-reaching trajectories remains open for autonomous flight in unknown cluttered environments with onboard resource constraints.
Method
Neural-Primitive uses imitation learning of a customized offline primitive strategy and a compact network that maps sensory inputs directly to polynomial coefficients.
Results
The planner generates smooth, empirically collision-free, dynamically feasible trajectories without back-end solving and demonstrates consistent superiority across computation time, success rate, flight length, and flight time.
Takeaways & Limitations
Zero-shot deployment in indoor and outdoor real-world experiments validates the proposed planner’s robustness and efficiency.
Takeaways & Limitations
The framework is primarily designed for static obstacles and may struggle with large local traps because it relies on instantaneous limited Field-of-View inputs.

Abstract

from arXiv · show

Autonomous flight in unknown cluttered environments is hindered by the computation-quality-memory trilemma of onboard trajectory generation. In this paper, we propose an efficient end-to-end local planner via imitation learning. A lightweight offline-primitive-based dataset collection framework is designed to produce safe and high-quality trajectory primitives in non-convex environments. A compact neural network directly maps sensory inputs to polynomial coefficients that inherently encode higher-order dynamical information. The learned policy generates smooth, empirically collision-free and dynamically feasible trajectories in real time without back-end solving. It achieves ultra-fast computation (below 1ms on a standard desktop and average 3.68ms during onboard flight), while maintaining low onboard memory requirements (less than 1.5MiB). Extensive simulation benchmarks demonstrate superiority in both planning latency and target-reaching progress quality. Zero-shot deployment in real-world experiments further validates the robust sim-to-real transfer capability of the proposed method.

I. INTRODUCTION

Neural-Primitive addresses the difficulty of generating fast, dynamically feasible, collision-free, target-reaching trajectories under onboard computation and memory constraints. It combines imitation learning with an offline primitive strategy and directly predicts executable polynomial trajectories.

Motivation: Jointly optimizing obstacle avoidance and other factors can produce trade-offs, local minima, or intractable problems, whereas primitive methods decouple obstacle avoidance through staged selection.Primitive-based performance depends heavily on the quality of the primitive library.
Motivation: The computation-quality-memory trilemma makes simultaneous fast computation, near-optimal target-reaching trajectories, and low onboard memory difficult to achieve.Existing approaches may require online optimization, closed-form solving, or projection after network inference.
Approach: Neural-Primitive uses imitation learning of a customized offline primitive strategy to produce high-quality trajectory primitives in non-convex environments.The framework is designed to mitigate covariate shift without resource-intensive DAgger training.
Evaluation: Extensive simulations evaluate computation time, success rate, flight length, and flight time, while unseen-map and real-world experiments assess generalization and sim-to-real capability.The paper reports superior performance across these evaluation dimensions.
Approach: The compact policy directly predicts polynomial coefficients that encode high-order dynamical information and produce controller-executable trajectories without back-end solving.“Without back-end solving” excludes additional optimization, quadratic programming, or boundary-value problem solving after inference.

B. Learning-based Motion Planning for UAV

The proposed learning-based planner replaces conventional multi-stage trajectory generation with a three-stage pipeline: simulation data collection, coefficient prediction, and zero-shot onboard deployment. Polynomial outputs preserve dynamical information and avoid post-inference trajectory solving.

Background: Existing learning-based planners retain hierarchical planning or require online optimization, closed-form solving, or projection to obtain executable trajectories.Waypoint outputs contain geometric information without dynamical properties and require real-time projection.
Pipeline: Training datasets are collected entirely in simulation using a customized primitive strategy, with polynomial coefficients recorded instead of only geometric path points.The coefficients inherently encode high-order dynamical information.
Pipeline: An MLP policy is trained offline to generate continuous primitives from discrete samples by predicting polynomial coefficients from sensory inputs.This addresses velocity or acceleration discontinuities in traditional offline primitive-library planning.
Deployment: The learned policy is zero-shot deployed onboard in previously unseen environments without fine-tuning, mapping sensory data directly to executable trajectories.The trajectories are sent to the low-level controller in a receding-horizon fashion.

1) Collection Procedure:

The collection procedure uses closed-loop simulation with randomized maps and navigation tasks, regenerating and selecting primitives during replanning. Only trajectories from successfully completed tasks are retained for imitation learning.

Collection Procedure: The full dataset is collected before training to address covariate shift without relying on resource-intensive DAgger.Data are collected in a closed-loop manner from successful randomly initialized navigation tasks.
Collection Procedure: At each replanning step, candidate terminal states are sampled, primitives are solved, traversed voxels are evaluated, and a valid primitive is selected.The algorithm regenerates the trajectory library conditioned on the current state.
Collection Procedure: Each simulation trial generates a random map and runs up to nthrd navigation tasks in parallel with randomized target positions and drone states.The initial target direction d0 is normalized and computed from the randomized configuration.
Collection Procedure: Selected primitives are stored with velocity, acceleration, target direction, point clouds, and polynomial coefficients, then partially executed to update the drone state.This creates sequential closed-loop training data rather than isolated state-action pairs.
Collection Procedure: Temporary data are appended to the dataset only after the drone reaches the target within a predefined tolerance.Unsuccessful task trajectories are not committed to the final dataset.

2) Trajectory Library with Occupancy Relations:

The trajectory library discretizes terminal states and generates spatially diverse, dynamically feasible polynomial primitives offline. Each primitive is obtained through minimum-control-effort optimization with boundary and dynamical constraints.

Trajectory Library: The library uses state-lattice discretization extending to acceleration, while sampling only terminal states because the current initial state is already known.This significantly reduces library size and permits multi-constraint optimization without latency concerns.
Trajectory Representation: Each trajectory primitive is a single-segment polynomial vector function in three-dimensional space, with coefficient vectors for x, y, and z.The polynomial basis contains powers of time through the chosen order.
Trajectory Representation: The position trajectory uses coefficient matrix c and fixed duration τ, with x(t), y(t), and z(t) representing drone positions.The polynomial order is n and the coefficient matrix has dimensions R^(n+1)×3.
Optimization: Primitive coefficients are solved by a convex quadratic program that minimizes control effort under boundary and dynamical constraints.The formulation relies on quadrotor differential flatness.
Optimization: The constraints encode boundary conditions and dynamical limits, with limits specified by Θlib and dynamical checks performed at uniformly sampled time points.The implementation uses n = 5, Nb = 3, Ns = 20, and τ = 2s for minimum-jerk trajectories.
Reference Frame: Primitives are expressed in a velocity-aligned frame whose x-axis follows velocity and whose z-axis opposes gravity.All primitives share the same origin, and occupancy relations support fast collision checking.

3) Collision Labeling and Best Primitive Selection:

Perceived point clouds are voxelized for rapid collision labeling, after which safe primitives are ranked by target-reaching and length costs. The expert strategy combines collision removal with constrained minimum-jerk primitive generation and task-oriented selection.

Collision Labeling: Fixed-size point clouds are mapped into a bounding box to obtain voxel indices for primitive collision labeling.Unsafe primitives are removed when any indexed point occupies their voxel relations; hash lookup avoids repeated ESDF queries.
Best Primitive Selection: Safe candidate primitives are selected using weighted target-approach and primitive-length costs.The target cost promotes forward progress or prevents overshoot, while the length cost favors direct trajectories over curved or weaving ones.
Best Primitive Selection: The target position is expressed in the velocity-aligned frame FV, with primitive costs evaluated at discrete samples ti = iτ/N.N denotes the discrete sample number for each primitive.
Expert Strategy: Each candidate primitive is optimal for a constrained minimum-jerk QP under conditioned boundary states, dynamical limits, and fixed duration.This is minimum-control-effort optimality rather than time-optimality; library selection separately pursues spatial near-optimality through sampling, collision removal, and selection.

B. Fast End-to-end Trajectory Planning Framework

The fast end-to-end framework integrates sensory mapping, front-end path searching, and back-end trajectory solving into one policy that produces executable trajectories from onboard sensing.

Fast End-to-end Trajectory Planning Framework: The end-to-end policy unifies mapping, path searching, and trajectory solving to eliminate intermodule latency and compounding errors.Its outputs are designed to preserve state continuity and target-reaching progress quality while remaining executable.

1) Policy Input-Output Design:

The policy maps current motion state, target direction, and point-cloud observations directly to learned polynomial coefficients. This design preserves state continuity while avoiding further trajectory solving.

Policy Input-Output Design: The policy takes velocity magnitude, acceleration, target direction, and 3D point clouds as inputs and outputs nine learned coefficients ca.The learned coefficients encode obstacle avoidance and target-reaching quality, while expert imitation reinforces empirical dynamical feasibility.
Policy Input-Output Design: The coefficient block cb is determined directly from current velocity and acceleration to govern state continuity.The complementary block ca is learned from perceived point clouds and target direction.
Network Architecture: Npe2eNet uses a point-cloud encoder, feature fusion module, and coefficient prediction head.It preprocesses arbitrary-sized point clouds to 666 points, extracts a global descriptor, and concatenates it with the 7-D state.
Network Architecture: 0.355M parameters and 1.38MiB of storage characterize the compact network.The network requires approximately 34M FLOPs; reported GPU inference memory is 0.65MiB incremental and 11.49MiB total.

3) Training Method:

Training uses simulation-based point-cloud and state perturbations to improve robustness to sensory mismatch, with coefficient regression trained on expert outputs. The implementation supports onboard deployment and event-triggered replanning.

Training Augmentation: Simulation training addresses the sim-to-real gap caused by differences in real-flight point-cloud density, spatial distribution, and noisy state measurements.Point-cloud preprocessing and domain-randomization strategies are used to improve zero-shot transfer and generalization.
Loss Function: The network minimizes mean squared error between expert coefficients c*_a and predicted coefficients ĉ_a.The coefficient vector contains M = 9 elements.
Implementation Details: The system uses LiDAR sensing, an NVIDIA Jetson Orin NX for end-to-end planning and tracking, and a PX45 flight controller for attitude control.Experiments use randomized maps containing cylinders, rectangular columns, and rings, with one million valid samples collected.
Implementation Details: Replanning is triggered by a 0.5m travel threshold, a 2τ/3 execution limit, or detected potential collision.This event-triggered receding-horizon mechanism adapts execution time to flight speed within [0, τ].

B. Ablation Study

The ablation study evaluates domain randomization, point-cloud preprocessing, and input-size choices, identifying configurations that balance robustness, reliability, latency, and memory.

Domain randomization: A2 selects rσs=0.03 and rσp=0.04 for domain randomization, achieving a success rate of 0.936.Adding moderate state and point-coordinate noise improves robustness, whereas excessive noise distorts observations and degrades performance.
Point-cloud preprocessing: Point-cloud preprocessing further improves success rates over the A2-derived B0 baseline in complex scenarios.The module exposes the network to broader point-density variation, while over-sparsification can erode obstacle geometric fidelity.
Point-cloud preprocessing: B2 provides the balanced point-cloud preprocessing configuration after varying preprocessing frequency and minimum density.The supplied passage identifies B2 as balanced but truncates its minimum-density value.
Point-cloud input size: 333 points reduce latency and memory but lower success rate to 0.916, while 1024 and 2048 points add memory cost with marginal success-rate changes.Peak incremental memory rises by 54.3% and 207.8% for 1024 and 2048, respectively.
Point-cloud input size: Input size 666 offers the favorable balance among planning reliability, computational efficiency, and memory cost.The point-cloud encoder operates point-wise, so latency and dynamic memory scale with input size.

C. Acceleration Continuity Analysis

The paper analyzes acceleration continuity and benchmark performance across obstacle densities and speeds. Neural-Primitive combines smooth higher-order behavior with fast, direct, and robust target-reaching trajectories.

Acceleration continuity: Acceleration jumps occur throughout Primitive-swarm flights because its library cannot cover all acceleration states within onboard memory and latency limits.Uncovered initial accelerations during online selection create discontinuities, degrading trajectory quality and increasing motor load.
Acceleration continuity: Neural-Primitive explicitly incorporates higher-order state continuity into network inputs and outputs, addressing the discontinuity problem.The method produces high-order continuous polynomial trajectories without back-end solving.
Computation: 0.70ms in sparse and 0.68ms in dense environments is Neural-Primitive’s average computation time, about 7 to 60 times faster than baselines.Its efficiency remains stable across obstacle densities because planning is integrated end-to-end.
Trajectory quality: 64.51m in dense and 64.39m in sparse environments are Neural-Primitive’s near-direct flight distances, with the shortest flight time to the target.The customized primitive strategy separates collision removal from task-oriented selection.
Energy cost: Lower accumulated squared-jerk than Ego and Super is achieved by Neural-Primitive while retaining more direct target-reaching motion.Fast and Yopo obtain lower energy cost at the expense of substantially longer detours.
Speed robustness: Above 0.80 success rates at average speeds around 6m/s are maintained by Super and Neural-Primitive, unlike Fast, Ego, and Yopo.Neural-Primitive matches Super’s robustness while avoiding extensive hand-crafted rules and pronounced detours.

E. Generalization Evaluations

Generalization is evaluated without fine-tuning in unseen forest and unstructured terrain maps, alongside onboard planning-time statistics. The learned policy retains substantial traversal success and real-time efficiency.

Unseen environments: 0.907 and 0.833 success rates are achieved in unseen forest and cave-like mountainous maps, respectively, without fine-tuning.Each map type uses 150 trials with randomly initialized start-target separations of 70∼80m.
Unseen environments: The forest test includes trunks, branches, vegetation, and spatially distributed noise points from a real-world point-cloud map.The map is cropped to 60×50×20m.
Unseen environments: The simulated terrain test uses irregular obstacles with varying shapes and densities that differ from the training datasets.The map resembles caves and mountainous terrain and measures 60×50×10m.
Onboard efficiency: 3.68ms is the mean onboard planning time throughout the flight.The distribution characterizes the proposed planner’s onboard real-time efficiency.

F. Real-world Experiments

Real-world tests used previously unseen outdoor and indoor environments, with training performed solely in simulation and no real-world fine-tuning. The planner achieved collision-free forest traversal, low onboard planning latency, and safe operation through narrow passages.

Experimental Setup: Simulation-only training transferred to previously unseen outdoor forests and indoor cluttered environments without real-world fine-tuning.The experiments covered both outdoor unknown forests and indoor environments with cluttered obstacles.
Outdoor Navigation: 60m forest traversal completed within 12s without collisions, reaching the target within 1m under wind and sensor noise.The quadrotor reached a maximum speed of 6.10m/s while maintaining a nearly straight trajectory.
Onboard Efficiency: 3.68ms mean onboard planning time was measured throughout the entire flight, with 2.71ms–5.94ms values and ±0.77ms standard deviation.The median planning time was 3.43ms.
Cluttered Environments: Safe traversal remained possible through thick foliage with a narrowest passage under 0.8m, despite the quadrotor diameter being approximately 0.3m.Indoor experiments additionally required sequential visits to multiple preset targets within 0.5m tolerance.
Planner Design: The planner directly outputs controller-executable polynomial trajectories without back-end solving, forming an integrated end-to-end planning scheme.The conclusion links this design to inherited offline-primitive trajectory quality and online-network continuity.
Limitations: The framework is primarily limited to static obstacles and may struggle with large local traps under instantaneous limited Field-of-View inputs.Highly dynamic obstacles, dead ends, and U-shaped obstacles remain identified future-work targets.