Source-linked AI summary
PAMPC: Perception-Aware Model Predictive Control for Quadrotors
Davide Falanga, Philipp Foehn, Peng Lu, Davide Scaramuzza
TL;DR
Quadrotor navigation must reconcile action objectives with perception requirements because camera motion can affect sensing quality. The paper introduces PAMPC, which jointly optimizes dynamics, feasible inputs, visibility, and image-plane motion in a receding-horizon framework. It runs onboard in real time and is validated in experiments including difficult lighting conditions.
Problem
Action and perception objectives can conflict because quadrotor motion needed for trajectory tracking may reduce the visibility of points of interest.
Method
PAMPC uses nonlinear model predictive optimization to compute dynamically feasible trajectories while optimizing point visibility and image-plane projection velocity.
Results
3.53 ms average execution time, with maximum execution time below 5 ms, enables PAMPC to run at a 100 Hz control frequency on the onboard ARM computer.
Takeaways & Limitations
The framework jointly couples perception and action for real-time onboard quadrotor control and demonstrates behavior in extremely challenging lighting conditions.
Abstract
from arXiv · showhide
We present the first perception-aware model predictive control framework for quadrotors that unifies control and planning with respect to action and perception objectives. Our framework leverages numerical optimization to compute trajectories that satisfy the system dynamics and require control inputs within the limits of the platform. Simultaneously, it optimizes perception objectives for robust and reliable sens- ing by maximizing the visibility of a point of interest and minimizing its velocity in the image plane. Considering both perception and action objectives for motion planning and control is challenging due to the possible conflicts arising from their respective requirements. For example, for a quadrotor to track a reference trajectory, it needs to rotate to align its thrust with the direction of the desired acceleration. However, the perception objective might require to minimize such rotation to maximize the visibility of a point of interest. A model-based optimization framework, able to consider both perception and action objectives and couple them through the system dynamics, is therefore necessary. Our perception-aware model predictive control framework works in a receding-horizon fashion by iteratively solving a non-linear optimization problem. It is capable of running in real-time, fully onboard our lightweight, small-scale quadrotor using a low-power ARM computer, to- gether with a visual-inertial odometry pipeline. We validate our approach in experiments demonstrating (I) the contradiction between perception and action objectives, and (II) improved behavior in extremely challenging lighting conditions.
SUPPLEMENTARY MATERIAL
The paper motivates jointly optimizing quadrotor action and perception because camera-based sensing is affected by environmental conditions and robot motion. PAMPC addresses conflicting objectives through optimization that couples dynamics, feasible inputs, and perception quality.
- Vision-based sensing offers low weight, cost, size, power consumption, and wide field of view, but its accuracy depends strongly on texture, lighting, and robot motion.
- Onboard vision cannot always replace motion capture because camera motion can degrade estimation and constrain robot agility.
- Perception can benefit from planned motion when the robot keeps task-relevant features visible or directs the camera toward texture-rich regions.
- PAMPC jointly optimizes action and perception objectives while satisfying robot dynamics and input saturations.
- Perception objectives are included as cost-function components, allowing the optimizer to trade off perception and action rather than enforce perception as hard constraints.
- Prior approaches variously used parameterized trajectories, hard visibility constraints, near-hover assumptions, offline optimization, or separate trajectory generation and tracking.
C. Structure of the Paper
The paper formulates autonomous navigation as a coupled perception-and-action problem because camera motion affects sensing quality. It models perception state as dependent on robot state and inputs, then optimizes both costs under shared constraints.
- Autonomous navigation requires perception and action, but the two components are rarely treated as a joint problem.
- Camera motion can degrade vision through insufficient image information or blur, while planned motion can expose textured regions and reduce motion blur.
- The formulation uses robot state x, input u, perception state z, and parameters σ, with z coupled to robot state and inputs through perception dynamics.
- Action and perception are represented by separate costs, La(x,u) and Lp(z), within one optimization problem.
- Equality and inequality constraints r(x,u,z) and h(x,u,z) enforce requirements for perception, action, or both simultaneously.
III. METHODOLOGY
The methodology models how a quadrotor-mounted camera projects a world-frame point into the image plane. It establishes world, body, and camera frames and uses quaternion-based rigid-body notation for the coupled perception model.
- Computer-vision navigation algorithms require task-relevant points to remain visible, motivating visibility and image-plane projection-velocity objectives.
- The camera’s projection geometry uses the body-to-camera extrinsic transformation and maps a world-frame feature to image-plane coordinates.
- The framework expresses the dynamics of a single point’s image projection as a function of quadrotor state and inputs.
- The world frame W, body frame B, and camera frame C define the reference frames used to describe the quadrotor and its measurements.
- Quaternion notation represents rigid-body orientation, with its time derivative determined by angular velocity through a skew-symmetric matrix.
- The operator ⊙ rotates a vector by the rotation induced by a quaternion.
B. Quadrotor Dynamics
The quadrotor dynamics use position, velocity, orientation, and thrust-related inputs in world and body frames. Gravity and rotated thrust determine translational acceleration, while motor thrust defines the mass-normalized thrust vector.
- The model state includes body position pW B, world-frame velocity vW B, and body orientation qW B.
- Angular velocity ΩB is expressed in the body frame, while position and linear velocity are expressed with respect to the world frame.
- The mass-normalized thrust vector is c=(0,0,c)⊺, where c=(f1+f2+f3+f4)/m and fi denotes the thrust from motor i.
- The translational dynamics are ṗW B=vW B and v̇W B=W g+qW B⊙c, combining velocity, gravity, and quaternion-rotated thrust.
- Gravity is W g=(0,0,−g)⊺ with g=9.81 m/s2.
C. Perception Objectives
The controller models a landmark’s camera projection and projection velocity to improve robust vision-based perception. It favors image-center visibility because central projections provide larger disturbance margins and avoid peripheral distortion.
- A world-frame landmark is transformed into the camera frame using fixed camera extrinsic position and orientation parameters.
- The camera-frame landmark is projected onto image coordinates s = (u, v)ᵀ using a classical pinhole camera model.The focal lengths fx and fy correspond to pixel rows and columns, respectively.
- Keeping the landmark projection near the image center increases safety margins against disturbances and reduces sensitivity to peripheral distortion.Distortion compensation is imperfect and can degrade estimate accuracy.
- The framework also minimizes the landmark’s image-plane velocity, assuming a static point while noting that analogous reasoning applies to moving points.Projection velocity is obtained by differentiating the projection model with respect to time and expressing it through the quadrotor state and inputs.
D. Action Objectives
Quadrotor action objectives require trajectories that respect bounded motor thrust and account for underactuation. Translational motion therefore must be achieved by coupling position and attitude through the system dynamics.
- Planned trajectories must keep the control input within the platform’s admissible set U at every time.Motor thrust has both upper and lower bounds, limiting the available input vector.
- Because the rotors generate acceleration primarily along the body zB axis, the quadrotor is underactuated in three-dimensional motion.
- Following a desired spatial trajectory requires exploiting the dynamics to couple translational and rotational motion.
E. Challenges
Perception and action objectives can conflict because trajectory tracking requires attitude changes that may reduce landmark visibility. The framework resolves this tension through real-time nonlinear MPC that jointly optimizes costs while enforcing dynamics and input limits.
- Challenges: Tracking a reference trajectory may require rotation to align thrust with desired acceleration, while perception may favor minimizing that rotation.
- Challenges: A model-based optimization framework is needed to couple perception and action objectives through the quadrotor dynamics.
- Optimization: The optimization can satisfy underactuated dynamics and actuator boundaries while minimizing predicted costs, unlike the cited classical PID and LQR schemes.
- Optimization: The nonlinear program is approximated by sequential quadratic programming and used in receding-horizon MPC after discretizing dynamics over a finite horizon.The formulation uses time-varying state, perception, and input cost matrices, with z = [s, ṡ] as the perception function.
- Optimization: Perception references target the image center and zero projection velocity, while state and input references come from a target pose or precomputed trajectory.
- Constraints: Inputs and velocity are constrained by platform-specific bounds, and multiple shooting with Runge-Kutta integration transcribes the dynamics.
- Implementation: At each control loop, one SQP iteration uses the latest onboard VIO state estimate and the previous solution as the next initialization trajectory.
V. EXPERIMENTS
The approach was evaluated onboard a small, vision-based autonomous quadrotor. The experiments were intended to demonstrate the potential of perception-aware model predictive control.
- Experiments ran the perception-aware model predictive controller onboard a small, vision-based autonomous quadrotor.
A. Experimental Setup
The experiments use a lightweight quadrotor and evaluate perception-aware flight during circular trajectories, including visibility-centered maneuvers around tracked features.
- The platform weighs 420 g, has a thrust-to-weight ratio of approximately 2, and uses a 220 mm motor-to-motor diagonal.
- Three experiments test circular-flight visibility, hover-to-hover motion, and vision-based flight in challenging conditions.
- 1) Circular Flight:: The circular-flight experiment uses boxes as visual features and their centroid as the point of interest while the quadrotor follows a circular reference trajectory.
- 1) Circular Flight:: At 3 m/s, the quadrotor keeps the point of interest visible during an agile maneuver with large deviations from hover conditions.
- 1) Circular Flight:: The executed trajectory records quadrotor heading while an arrow indicates the direction toward the point of interest.
2) Hover-To-Hover Flight:
In hover-to-hover flight, the controller trades altitude and pitch to preserve feature visibility, while the darkness experiment adjusts heading to retain scarce visual features.
- 2) Hover-To-Hover Flight:: The hover-to-hover maneuver requests a same-height pose jump from p1 to p2, creating a conflict between acceleration and maintaining point-of-interest visibility.
- 2) Hover-To-Hover Flight:: The quadrotor accelerates upward during the horizontal maneuver, reducing pitch and helping move features toward the image center through increased altitude.
- 2) Hover-To-Hover Flight:: Without perception objectives, the trajectory would not require a height change and could produce poorer point-of-interest visibility.
- 3) Darkness Scenario:: In darkness, the quadrotor follows rectangular waypoints without a heading reference and adjusts heading to keep illuminated spots within view for visual-inertial odometry.
- B. Convexity of the problem: Positive camera-point depth makes the perception projection costs convex after constraining the denominator to the positive half-plane.
C. Choice of point of interest
The section describes the perception-aware controller’s point-of-interest representation, parameter choices, real-time execution, and joint treatment of perception and action objectives.
- Detected features are represented by their centroid because equally weighted features yield the same optimal solution as individual optimization.
- The controller uses dt = 0.1 s and a time horizon of th = 2 s, balancing discretization quality against computation time.Longer horizons and shorter steps increase computation time by roughly O(N^2).
- The optimization requires on average 3.53 ms and remains below 5 ms even while running onboard VIO and the full control pipeline.The controller runs on a quad-core ARM processor, with one core available for PAMPC.
- PAMPC computes trajectories satisfying dynamics and input limits while optimizing visibility and image-plane projection velocity.Perception objectives are included in the optimization alongside action objectives.